Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an optimal transport framework Solar for imbalanced partial label learning where each data point has multiple candidate labels and the latent label distribution is skewed. The key of Solar is to refine the pseudo-labels by a constraint of within-candidate allocation and class prior matching. In parallel, a moving-average-based class prior algorithm is proposed to generate class prior for the aforementioned constraints. Experiment results on CIFAR-LT and CUB-200 datasets validate the effectiveness of Solar in improving few-shot accuracy.
strengths_and_weaknesses: Strengths:

1.The long-tailed learning problem is important but is generally overlooked in the PLL problem. This work systematically investigated the imbalanced PLL problem by identifying the failure of existing PLL methods and a new Solar framework, which can inspire future research in this field. 

2.Solid theoretical analysis is given. The overall objective guarantees to recover the true classifier at a population level. The detailed derivation of the Sinkhorn-Knopp algorithm is also provided.

3.The experiments are sufficient. The ablation study strongly supports the effectiveness of the label refinery procedure. Reproducibility is ensured as the source code was submitted.

Weaknesses:


1.In experiments, the PRODEN method also uses mixup and consistency training techniques for fair comparisons. What about other competitive baselines? I'd like to see how much the strong CC method could benefit from the representation training technique.

2.It is not clear why the proposed sample selection mechanism helps preserve the label distribution.

3.In App. B.2, a relaxed solution of Sinkhorn-Knopp algorithm is proposed. Why the relaxed problem guarantees to converge?Does Solar always run this relaxed version of Sinkhorn-Knopp?

4.How is gamma in the Sinknhorn-Knopp affect the performance?

5.How does the class distribution estimate for PRODEN in Figure 1?
questions: Please see my main review.
limitations: Societal Impacts: The main negative impact is lower annotation costs may decrease the requirement for annotator employment. 

Limitations: The experiments need to be further improved.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work approaches the class-imbalanced partial-label learning problem where each training example is assigned a set of candidate labels. Efforts are made to address two technical challenges. First, even the best-performing PLL method fails to identify the true labels from the candidate labels, because the pseudo-labels are largely biased toward head labels. To address this problem, an optimal transport-based objective is proposed to enforce constraints on pseudo-labels to match the class prior. Second, this work proposes a new class prior estimation algorithm along with a sample selection mechanism for practical training. Experiments validate the effectiveness of the proposed method.
strengths_and_weaknesses: Pros:

(1) Most studies about PLL assume the actual label distribution is balanced, which may not hold in practice. This work reveals the leading cause of performance degradation of the SOTA PLL methods long-tailed PLL problem and proposes a principled OT-based framework to resolve this. It is a serious attempt that makes PLL more suitable for real-world applications. 

(2) The iterative class prior estimation technique is novel, as estimating the true class prior without an additional validation set is intractable for any weakly-supervised learning problems. Other imbalanced weakly-supervised learning algorithms can potentially benefit from this technique.

(3) Theoretical results show the consistency of the proposed OT objective.

(4) Experiments are thorough and the proposed SoLar algorithm shows impressive performance. 



Cons:

(1) The main weakness is the evaluation was conducted on CIFAR datasets. With long-tailed sampling, the resulting data examples are typically sparse. Can SoLar get similar performance improvement on real-world long-tailed datasets?

(2) Typos: 'Goup of Labels' in Figures. 

(3) Wrong figure caption in Figure 3, CIFAR-10-LT rather than CIFAR100-H-LT.

questions: I noticed that the class prior vector r is initialized as uniform. But, why can't we count the number of candidate labels as a rough estimation of the class prior? It is typically close to the true class prior and may perform better.
limitations: It would be appreciated if experimental results on large-scale LTL datasets are provided.

Potential negative societal impacts are discussed in the Appendix and the PLL framework may cause potential employment destruction.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This paper concentrates on how to perform disambiguation in partial label learning with the long-tailed label distribution considered. The authors propose an Optimal Transport-based framework called SoLar that allows to refine the disambiguated labels towards matching the marginal class prior distribution. They conduct experiments on several benchmarks to demonstrate the superiority of the proposed method.
strengths_and_weaknesses: Strengths: 
The paper is well-written and easy to follow. The proposed method seems to be technically reasonable. 

Weaknesses: 
The motivation is not convincing. On the one hand, the paper claims that "existing PLL methods have been commonly driven by the assumption that training data consists of class-balanced distribution", which is not true as many SOTA PLL methods do not have the assumption, and they can deal with the imbalanced-class problem natually embedded in the disambiguation. On the other hand, the paper claims that "we find that current best-performing PLL methods display degenerated performance in the long-tailed setting. This happens because the predictions of pseudo-labeling—a core component that PLL methods rely on—can be largely biased towards the head and majority classes." However, many SOTA methods are not pseudo-labeling based methods, and the authors only present one PLL mehtod, i. e., PRODEN, to display the degenerated performance in the long-tailed setting, which is not convincing. 
questions: The authors should compare more SOTA PLL methods except for some specific pseudo-labeling-based method to verify their point that the performance will degenerate in the long-tailed setting. The authors should also conduct experiments to show that their motivation works on benchmark PLL datasets, such as lost/BirdSong/Soccer Player/Yahoo!News. 


limitations: The motivation is not convincing as stated in the weakness. 
The novelty and the techquie seem to be limited.
The experiments have some limitations. For example, experiments on many PLL benchmark datasets (such as lost/BirdSong/Soccer Player/Yahoo!News ) are not conducted, and many SOTA methods are not compared, for example, Instance-Dependent Partial Label Learning, NeurIPS 2021.
review_title: null

Meta-review: metareview: Most approaches to partial-label learning (PLL) tasks assume the label distribution is balanced, which may not hold in practice. This paper provides a principled optimal transport-based framework to resolve the issues with performance degradation caused by skewed label distributions in PLL. The reviewers found that the work makes a theoretically solid and experimentally comprehensive attempt towards solving this practical and under-addressed problem.
confidence: Less certain
Decision: Accept