Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an objective for contrastive training of representations based on noisy cluster information. In particular, the objective encourages samples from the same noisy cluster to have similar representations and representations from distinct clusters to have dissimilar representations. The authors show that representations learned with this approach often outperform representations learned with only self-supervision (and no noisy clusters), and also that the proposed approach to using noisy cluster information outperforms other baselines using noisy cluster information. The authors further show that they are able to alternate representation learning and clustering in order to learn better representations that self-supervised approaches even without initial noisy clusters.

main_review: Strengths:
- The paper is very interesting.
- The authors conduct a large number of experiments, and nearly all of them seem to support the authors' main claims.
- The results in Figure 4 are particularly impressive.

Weaknesses:
- Fairly minor:
  - Some of the presentation could be a bit clearer. For example, Eqn (1) could make clearer how the negatives are drawn.
  - In 4.3 the authors mention that their approach is more stable/robust to hyperparameters than the cross-ent baseline, and it would be nice to have a bit more detail about what exactly the authors observed.

Additional comment/question:
I'm wondering if the authors have an intuition about why exactly the K-Means+Cl-InfoNCE approach is helpful. Should we think of the initial pass through f and the subsequent clustering as clustering the data based on random features? If so, is it actually necessary to alternate K-means and Cl-InfoNCE, or can we just use the initial random clustering?
summary_of_the_review: This is an interesting paper with results that do an impressive job of supporting the paper's claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a weakly-supervised contrastive representation by using the auxiliary clustering information. Data are clustered with auxiliary tags and the clustering InfoNCE loss is introduced to optimize the system. The authors prove that maximizing the Cl-InfoNCE learns to include the clustering information.
main_review: Strengths：
- The idea is clear and simple, and the authors give theoretical definitions to support their assumptions and experiments.
- The authors prove that Cl-InfoNCE maximization learns to include the clustering information. And thus can use I(Z; T) and H(Z|T) to characterize the goodness of the learned representations. The authors also present experiments to validate it.
- The authors conduct experiments with different supervision levels and show that CL-InfoNCE can better bridge the gap with the supervised learned representations by using auxiliary information.
- The proposed approach can also be applied without auxiliary information with K-means and EM optimization.

Weaknesses:
- In Figure 4, the authors observe that the best performing clusters happen at the intersection between I(Z;T) and negative H(Z|T), could you present proves on this?
- Comparison with related work is required, such as [1]

[1] Weakly Supervised Contrastive Learning. ICCV 2021.
summary_of_the_review: Overall, this paper proposes an easy but effective method for weakly-supervised contrastive learning with auxiliary information. The authors provide clear mathematical proof and good results with adequate experiments. However, comparisons with related work should be included to make to final results more convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a two-stage weakly supervised contrastive learning approach, where it first clusters the data according to the available auxiliary information (or data -driven clusters if no auxiliary information available), and then generates similar representations for the intra-cluster samples and dissimilar representations for inter-cluster samples via a clustering InfoNCE objective. Theoretically, the objective is proved to include the clustering information in the learned representation, and can serve as an interpolation between supervised and self-supervised representation learning. Experiments on four datasets with and without auxiliary information validate the effectiveness of the proposed method. 

main_review: STRENGTHS: 

++ Interesting idea on leveraging the structure information hidden from the data distribution for learning better representation. 

++ Thorough theoretical analysis on the value of clustering information in the proposed objective function.

++ Well written paper. 

WEAKNESSES:

-- Lack of novelty. The proposed Cl-InfoNCE is simply a cluster aggregated NCE loss. The way to construct the cluster is either through top-k attributes or k-means. These methods are straightforward and common tricks in representation learning literature. 

-- Weakly-supervised methods refer to those using noise labels for learning certain tasks. In the context of this paper, it is essentially the using the auxiliary information as structure information to guide the learning process, and the labels utilized are the accurate ground truth. In my opinion, it is misleading to use the term “weakly supervised”, and the method itself is more of  like “structure aware representation learning”.

-- Some key details are not clarified. The attribute-determined clusters contain samples with the same values on a set of attributes, which inevitably produce some sparse clusters with a very small number of samples. How does that affect the model performance ? Is overfitting an issue in such a situation? How to handle the sample imbalance across different clusters ? Though not mentioned, such practical issues would greatly affect the model performance. 

-- Experiments are not sufficient. i). There are other similar existing works with higher performance. For example, Zheng et al. Weakly Supervised Contrastive Learning, ICCV 2021. (79.7% vs 77.9 (this paper) on ImageNet-100). ii) How does the proposed method work compared with other structure-aware representation learning methods such as “Yaling Tao, et al. Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation, ICLR 2021”. 

summary_of_the_review: This paper proposes a structure-aware constructive loss with theoretical justification for representation learning. The theoretical analysis is very interesting and insightful. However, there are some issues in terms of novelty, clarity in the technical details, rationality on the terminology as well as the soundness in the experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a weakly-supervised contrastive loss based on using existing auxiliary information from datasets (such as shoe attributes for classifying shoe categories (UT-zappos50K)). The auxiliary information is used in a contrastive loss formulation similar to the unsupervised contrastive loss of (Chen et al., 20) and the supervised contrastive loss formulation of (Khosla et al., 2020), but instead using clustering information from auxiliary information (Eq (1) of the paper). Experiments are across 4 image datasets (UT-zappos50K, Wider Attribute, CUB-200-2011, ImageNet-100) and shows that representations learned from the proposed weakly-supervised loss performs between unsupervised & fully supervised contrastive loss. There are additional experiments comparing to a few other clustering-based contrastive loss approaches.
main_review: Strengths:
- The experiments are well-organized and are clearly presented. The sub-sections make it clear which experiments refer to which claims by the authors.
- The experiments are on a wide-set of image datasets, and on both weakly-supervised and unsupervised versions of the proposed method, which helps show the generalization ability of the method. (I'd like to see more investigation of the labels though, see below.)

Weaknesses:
- My main concern is that the paper does not sufficiently address other relevant works using weakly supervised contrastive loss, and so the contribution of this work is not clear. More specifically, there are other works that also use auxiliary information/weak-supervision with contrastive loss [A] (the proposed Eq (1) in this paper seems similar to the weakly supervised contrastive loss in Eq (4) of [A], both using additional weak supervision on contrastive loss (in contrast to fully supervised contrastive loss as in (Khosla et al., 2020))). Another set of works are other clustering-based contrastive loss methods [B] (On ImageNet100, looks like the contrastive loss proposed in [B] performs better than this method. Table 2 in [B] = 79.77, Table 5 here = 77.9). Comparing against these as existing baselines would be best, or at least there should be a discussion to contrast with such existing works.
- The performance of this method seems limited relative to existing works - Table 2 shows that CMC is similar in performance to the proposed method. The authors explain that "Cl-InfoNCE performs better with less informative auxiliary information": it would make the paper stronger if the authors specify more concretely when their method should be used over CMC. Also [B] seems to perform better than this method on ImageNet100.
- What if the auxiliary information is incorrect or noisy? Would this method perform worse than SimCLR, or will the method learn to ignore the incorrect label information?
- Minor: The paper would be stronger if it discussed other works using auxiliary information or weak supervision on data, since using auxiliary information is a major part of this paper. Adding a paragraph discussing this would be great (ex: works like [C]) so that the contribution of this paper is more clear.
- Minor: Minor issues with formatting - for example, the plot in Figure 5 looks squished. Also, the writing could be more clear in places (ex: "For instance, if having continuous attributes as auxiliary information, binning or quantization cannot be avoided when constructing the clusters." my understanding is that the authors are saying, the proposed method cannot work with continuous attribute labels?). The Appendix also appears to be submitted with the main PDF (instead of being in supp materials).

[A] Sun, Jennifer J., et al. "Task programming: Learning data efficient behavior representations." CVPR 2021.

[B] Zheng, Mingkai, et al. "Weakly Supervised Contrastive Learning." ICCV 2021.

[C] Tan, Reuben, et al. "Learning similarity conditions without explicit supervision." CVPR 2019.
summary_of_the_review: Overall, the experimental section of the paper was clear and included a good variety of image classification datasets. However, given the lack of discussion/comparison to relevant existing works and limited performance of the proposed method, the contribution of the proposed method is not clear to me at this point.
review_title: null

Meta-review: The paper proposes a weakly supervised contrastive learning, using auxiliary cluster information, for representation learning. Their method generates similar representations for the intra-cluster samples and dissimilar representations for inter-cluster samples via a clustering InfoNCE objective. Their approach is evaluated thoroughly on three image classification task.

The reviewers agree that the paper is well written, presenting interesting theoretical analysis (Reviewer h3zd,  a8kw) and solid experimetal results (Reviewer RhYi, 1ziy). The core idea of the paper is relatively simple and well motivated (Reviewer h3zd). While the focus is using the clustering with auxiliary labels, the method can be applied without auxiliary labels with K-means. 
There were some concerns from the reviewers:  the overlap with a concurrent work [1]. The authors have provided detailed discussions on conceptual (concurrent work focuses on unsupervised cases where this work focuses on weakly-supervised setting) and emprical comparisons. Accordingly, reviewer a8kw and 1ziy had some issues with the novelty of the paper, as it can be interpreted as slight modification from previously explored idea (vanilla InfoNCE loss). 

Despite some overlap with existing approaches, the paper presents an interesting and well conducted study of integrating clustering information for learning representation, so I vote for acceptance. 

[1] Weakly Supervised Contrastive Learning. ICCV 2021.
Decision: Accept (Poster)