Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a new method for rebalancing of class probabilities for longtailed learning under the setting of partial label learning. No others followed the idea of logit adjustment, which factories P(y|x) as P(x|y) P(y). The second component is estimated on a per class basis in the training algorithm.

The authors show their experimental results on CIFAR-100 LT and CIFAR 10 LT data sets. 
strength_and_weaknesses: Strength:
- Proposed dynamic re-balancing technique is interesting.
- The problem formulation is theoretically interesting.

Weaknesses:
- The experimental results are not convincing. The main issue with this paper is that it does not compare its performance with the state of the art methods for longtailed learning. For example the following paper reports in excess of 52% accuracy on CIFAR-100 LT. 

Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, Yu-Gang Jiang. Balanced Contrastive Learning for Long-Tailed Visual Recognition. CVPR 2022

- Another major problem is about the practical motivation for the proposed problem. Authors almost assume that LT+PLL is an important problem setting to learning with without giving any motivating examples. This is also reflected in the experimental set up on benchmark data sets where for the PLL setting the authors simply use uniform probability to generate partial label sets from CIFAR 100 LT. This might generate more partial labels for the tail classes.


- Finally, the paper is very difficult to read. This is because, the exposition in the paper assumes that the author is familiar with the literature on long-tailed learning and partial label learning. However many of these settings are not mainstream and the approaches should be introduced properly.

For example, prior to equation 2 the authors do not define p_uni(y) but refer to a previous paper. It is hard for a reader not familiar with the literature to understand the definition from the previous paper.

clarity,_quality,_novelty_and_reproducibility: Please see above.
summary_of_the_review: Overall the paper is well written and dwells on topical problems. However the current problem is poorly motivated and seems almost like a combination of two other problems. The proposed method is logical and simple. However the exposition of the proposed methods in my opinion needs to be more lucid. Finally while lot of experiments have been performed showing improvement over existing PLL techniques, the baselines for long tail learning have not been incorporated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the combination of long-tailed learning and partial label learning.

Applying existing methods from long-tailed learning to this regime result in poor performance because of ambiguous labels. This paper proposes dynamic rebalancing as a solution.
strength_and_weaknesses: Strengths

 - The motivation behind the setting and the problems with existing solutions are described well.
 - Dynamic rebalancing is clearly explained and the toy example is a clear example of where it works well.
 - Dynamic rebalancing can be applied to many methods as a drop-in replacement.

Weaknesses

 - The baselines are primarily focused on long-tailed learning, so it is natural that the baselines would not handle the partial labels properly.
 - Dynamic rebalancing is a relatively obvious solution (possibly the only simpler solution is to just use the previous epoch to estimate the class distribution, and it's not clear that momentum to average would outperform this).
clarity,_quality,_novelty_and_reproducibility: The paper is relatively clearly written.

The problem setting is novel, although the method is mostly using existing techniques.

The paper says that code will be available, but it is not currently available yet.
summary_of_the_review: The paper combines the long-tailed learning and partial label learning settings. Dynamic rebalancing improves upon the baselines, but mostly combines existing techniques together.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider a problem where data exhibit both a long-tail (LT), where there is major class imbalance that we do not expect at test time, and PLL (partially label learning), where there exists a true label but at training time we only observe a superset that contains the true label. The authors show that standard LT adjustment interact poorly with PLL, even if the oracle class distribution is known. The authors propose a solution that applies a dynamic LT adjustment that is more consistent with the model's current predictions at each step of training. This dynamic adjustment can be used with any PLL algorithm. The authors show that this scheme will lead to an LT adjustment that converges to the oracle, but argue that the dynamic nature of the adjustment allows the model to train better. This last claim is substantiated with several empirical experiments that consider how different LT rebalancing schemes compare with the same PLL algorithm, as well as some studies exploring and probing representations, and a study examining the effect of the momentum tuning parameter that controls the dynamics of the class adjustment.
strength_and_weaknesses: Strengths:
 + The paper does a nice job setting up the LT + PLL problem, and showing that naive combinations of existing methods tend to fail.
 + Each of the LT and PLL problems are well-motivated, and prior work seems to be well-cited.
 + The method is explained well and empirical results are compelling, with appropriate ablations and comparisons.
 + The authors construct a new benchmark dataset, CIFAR-100-LT-NU

Weaknesses (and questions):
- It is still somewhat mysterious why using the oracle class probabilities fails so badly with PLL methods. Is the main pathology that the model simply doesn't train well at all? E.g., if one had an identically distributed test set with the same class distribution as training, would PLL methods using constant rebalancing behave poorly there too? Would "tempering" in the oracle logit adjustment (i.e., temperature scaling P(y) so that you start close to uniform and eventually adjusting for P(y)) lead to a similar result as the dynamic rebalancing? This question doesn't undercut the method in the paper (since it requires no access to the oracle), but it would be useful to shed a bit more light on how the oracle LA is disrupting training dynamics. The linear probing analysis does some of this.
- Most of Theorem 1 seems to be a restatement of results from Liu and Dietterich. Specifying exactly what is added here would be useful. For example, it seems like one could assume that the Liu and Dietterich error bound holds, and Theorem 1 would be a relatively straightforward corollary (Note: the straightforwardness would not be a weakness!). Perhaps this is wrong, but some explicit discussion would be useful to understand what is novel.
 - Switching the softmax and expectation in (4) seems questionable as an approximation, especially in the long-tailed context where we expect some rare or common class logits to lie outside of the region where the softmax is approximately linear. The justification given in the text could at least use a citation if not some elaboration (say in an appendix). Alternatively, could this be formulated as a form of stabilization or regularization via Jensen's inequality?
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear, especially in the essential parts, although some issues with Theorem 1 were noted above. The intuition-building and simple examples are particularly compelling in the presentation. To my knowledge this seems to be new, although I am not particularly steeped in this corner of the literature. Code is not provided with the submission, but experiments seem straightforwardly reproducible.
summary_of_the_review: This is a nice, thorough treatment of this problem, with a method that appears to be quite successful. The authors do a nice job of motivating their work, and showing how their method fits modularly with current PLL methods. The experimental comparisons are generally compelling. The theory could stand to be better-modularized, but that is a minor concern. The paper could potentially be improved by showcasing a "killer app" where the LT-PLL problem arises.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on a more challenging but practically real-world data scenario, which couples the label ambiguity and heavy imbalance. This paper first demonstrates the infeasibility of the straightforward combination of the current long-tailed learning and partial label learning. Then it proposes a dynamic rebalancing method, termed RECORDS, to rebalance the training process. Experimental results on several benchmark datasets show the significant gain of RECORDS compared with a range of baselines. 
strength_and_weaknesses: Strength:
1. Well-motivated. The study of complex label distributions of data is a hot research area. This paper introduces a more difficult but common LT_PLL scenario by combining long-tailed and partial label learning.
2. Well-written. This paper is clearly organized and easy to follow.
3. The proposed method is reasonable and effective, as shown in experimental results.

Weakness:
1. There are other work focusing on similar scenarios, e.g. [1], so this paper is not the first attempt. It would be better to include some discussion and experimental comparison.
2. There are some grammars. For example, in the last paragraph of Page 1 , “In Figure 1, we trace the average prediction of a PLL model PRODEN Lv et al. (2020) “, \citep should be used here, i.e., “PRODEN Lv et al. (2020)” should be PRODEN (Lv et al. , 2020). There are many such mis-uses of \cite in this paper.
“Works” should be replaced by studies
3. Please explain how the approximation in Equation (4) is obtained and what is the gap between these two items. Also explain how the second equation is obtained.
4. The toy data are not a good testbed for the long-tailed and partial labeling setting, as there are no partial labels there.

Ref:
[1] Liu W, Wang L, Chen J, et al. A Partial Label Metric Learning Algorithm for Class Imbalanced Data[C]//Asian Conference on Machine Learning. PMLR, 2021: 1413-1428.

clarity,_quality,_novelty_and_reproducibility: 1. Despite the good results, this paper is incremental compared to the LA.
2. No transparency and reproducibility. The authors have not submitted their code, which is a red light to me. It's 2022 and submitting the code along with the paper should be the norm.

summary_of_the_review: The proposed method is well-motivated and seems to be effective, and experimental results and analysis are quite adequate. However, The experimental evaluation requires improvement, and code should be submitted along with the paper or made publicly available.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) The paper provides a new method to overcome class imbalance in learning from data where among multiple training labels, many may be incorrect (PLL).

(b) Strengths: The work is thorough and the authors responded to the reviewers carefully.

(c) The work is incremental. It is a typical ML paper in the sense that it proposes a new algorithm that is a combination of known components, is heuristic, and is somewhat complicated. The experiments are merely on known datasets, with no new real-world application or insight.
justification_for_why_not_higher_score: Experimental results are only on a small number of artificially manipulated standard datasets. There is no indication of value on a specific real-world application.

As the paper says, it "does not involve any human subjects, ... data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimination/bias/fairness concerns, privacy and security issues, legal compliance, and research integrity issues." The real-world importance is hypothetical, and the mathematical contribution is slight.
justification_for_why_not_lower_score: The core idea of this paper is in Sections 3.3 and 4. The new method is based on a simple but effective and guaranteed previous method for dealing with imbalance, namely logit adjustment (3.3). The authors identify why this does not apply directly to partial label learning, and design a well-motivated extension (4).
Decision: Accept: poster