Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a new technique for partial label learning (PLL). PLL is the task where the labels for each instance include both the ground truth label and a randomly sampled set of distractor labels, and during training the model learns a latent decision for which among this set is the ground truth. The technique presented by the authors uses a combination of momentum (in the representation) and contrastive learning (to augment the label set) that leads to improved PLL results, reaching nearly fully supervised performance.
main_review: Strengths
- Incredibly strong PLL performance close to supervised learning and substantially stronger than baselines.
- Helpful ablations and results on various datasets.
- Provides theoretical connection of the proposed method to EM.

Weaknesses
- More should be said about the closeness of PLL performance and supervised learning performance. This comment may seem counterintuitive, but I do not think this closeness can be so easily glossed over, especially when it is one of the main results.
- (minor) Treatment of clustering is simplistic. K-means is a very specific type of clustering algorithm, and only one of multiple classic clustering techniques.

Questions and comments:

- How is the “novel” prototype-based label disambiguation so different from using a softmax layer? A softmax layer will also have a prototype vector for each class.
- I can see the convenience of momentum style updating of prototype vectors, but re-computing them every iteration (or every N iterations) is not so expensive and at most increases training by 2x.

summary_of_the_review: This seems like a particularly strong result. More could be said about the implications of their results, but otherwise seems like a good paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an innovative approach to partial label learning, where an instance is assigned with some false positive labels besides its true label (due to difficulty in labelling). The approach blends contrastive learning with prototype learning: (i) the former helps to form good clusters for the latter to learn prototype representations (ii) in return the latter helps to select positive samples for the former. Theoretically, the authors prove that the two of them work collaboratively in the EM fashion. 

The paper demonstrates the effectiveness of the proposed approach using a typical setting for the task. Specifically, CIFAR 10 and CIFAR 100 were used and each instance was randomly assigned false labels with some probabilities. The proposed approach achieved impressive results, substantially outperforming five recent models in the literature. Moreover, its performance strongly approaches the one of supervised learning. 

The paper also presents a much stricter setting when false positive labels are semantically correlated with true labels. The proposed approach also gained impressive results on CUB-200 and CIFAR-100-H datasets. Moreover, the paper shows several in-depth analyses.


main_review: ## Strengths 

The problem that the paper tackles, namely partial label learning, is important in the real life where labelling is difficult due to semantic ambiguity (e.g. husky vs malamute). The problem has several connections especially with weakly supervised learning. 

The approach proposed in the paper is very well motivated. The idea of making contrastive learning and prototype learning is sound as they can work collaboratively in the EM fashion (proved in the paper). 

The approach is backed by very impressive empirical results. The presented analyses support the motivation of the approach, e.g. cluster visualisation shows that the clusters are well formed with few classification errors. 

The paper is well written with a clear structure. Reproduction doesn't seem difficult. 


## Weaknesses 

As the approach is EM-like, it also inherits some cons from EM. This is however not presented in the paper. For instance, in which case the learning will get stuck in a bad optimum? What is the consequence? Is there any way to avoid that?

It is unclear how the models were fine-tuned. Did the authors use clean dev sets which don't have any false positive labels? If this is the case, would it be more realistic to fine-tune the models on noisy dev sets instead? One concern is that clean dev sets can provide lots of information for removing labelling ambiguity. For instance, one can achieve reasonably good prototype representations just from a few clean labelled instances. 

It seems that the paper mentions too little about prototype learning literature. 


summary_of_the_review: This is a strong paper tackling an important problem. The approach is interesting and the technique is sound. The claims are well backed by impressive empirical result and theory. Therefore, overall, the contributions of the paper will have a certain impact to the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work approaches the partial label learning problem where each training example is annotated with multiple candidate labels, in contrast to the conventional supervised learning setup that the ground-truth label is provided. The proposed framework comprises two key components: (1) a contrastive learning module that uses the classifier output to select positive pairs and (2) a label disambiguation method that uses the contrastive prototypes to update the pseudo targets in a moving-average style. The experimental results look quite strong, where the performance of PiCO nearly approaches the fully-supervised results.
main_review: I have the following comments.
Pros:
1.	The structure of this paper is well-written and easy to follow. The motivation is clear, and the solution is simple but effective.
2.	I think the theoretical justification of the proposed method from the expectation-maximization perspective is very interesting. It is a generic result and potentially help the community understand the property of contrastive learning.
3.	The experimental results are quite strong and the ablation study also looks good. I especially appreciate the results on new fine-grained datasets to test the performance of PLL methods.
Cons:
1.	Experiments are mainly conducted on datasets with uniformly generated candidates, it is not clear how PiCO performs on non-uniform datasets, such as in the label-dependent setting where the probability of label flipping depends on its ground-truth label.
2.	Why does the second equality in Eq. (9) hold? It would be appreciated if a clear explanation is provided.
3.	There is a redundant space notation in Table 1 (the method CC with q=0.5 on CIFAR10).

summary_of_the_review: This paper is clear and technically solid. The theoretical justification is interesting.
review_title: null

Meta-review: This paper presents PiCO, a novel approach for partial label learning, which achieves very strong performance close to that of fully supervised learning and outperforms PPL baselines. The experiments are extensive with very impressive results and the analysis are thorough.
Decision: Accept (Oral)