Learning Weakly-supervised Contrastive Representations
Authors: Yao-Hung Hubert Tsai
Tianqin Li
Weixin Liu
Peiyuan Liao
Ruslan Salakhutdinov
Louis-Philippe Morency
Venue: ICLR 2022 Poster (2022)
Keywords: Self-supervised Learning
Weakly Supervised Learning
Learning with Auxiliary Information
Clustering-based Representation Learning

Abstract: We argue that a form of the valuable information provided by the auxiliary information is its implied data clustering information. For instance, considering hashtags as auxiliary information, we can hypothesize that an Instagram image will be semantically more similar with the same hashtags. With this intuition, we present a two-stage weakly-supervised contrastive learning approach. The first stage is to cluster data according to its auxiliary information. The second stage is to learn similar representations within the same cluster and dissimilar representations for data from different clusters. Our empirical experiments suggest the following three contributions. First, compared to conventional self-supervised representations, the auxiliary-information-infused representations bring the performance closer to the supervised representations, which use direct downstream labels as supervision signals. Second, our approach performs the best in most cases, when comparing our approach with other baseline representation learning methods that also leverage auxiliary data information. Third, we show that our approach also works well with unsupervised constructed clusters (e.g., no auxiliary information), resulting in a strong unsupervised representation learning approach. 

Published as a conference paper at ICLR 2022
LEARNING WEAKLY-SUPERVISED CONTRASTIVE REP- RESENTATIONS
Yao-Hung Hubert Tsai', Tianqin Li Weixin Liu', Peiyuan Liao', Ruslan Salakhutdinov's Louiuddiooo,Lous-Piiiiiliili Louis-Philippe Morency' 'Carnegie Mellon University (yaohungt, tianqinl, weixinli, peiyuanl, rsalakhu, morency) les cmu edu
ABSTRACT
We argue that form of the valuable information provided by the auxiliary informa- tion is its implied data clustering information. For instance, considering hashtags auxiliary information, we can hypothesize that an Instagram image will be semantically more similar with the same hashtags. With this intuition, we present a two-stage supeervsed contrastive learning approach. The first stage is cluster data according to its auxiliary information. The second stage is to learn similar representations within the same cluster and dissimilar representations for data from different clusters. Our empirical experiments suggest the following three contributions. First, compared to conventional sulf-supervised representations, the representations bring the performance closer to the supervised representations, which use direct downstream labels as supervision signals. Second, approach performs the best in most cases, when comparing our approach with other baseline representation learning methods that also leverage auxiliary data information. Third, show that our approach also works well with unsupervised constructed clusters (e.g., no auxiliary information), resulting in a strong unsupervised representation learning approach.
INTRODUCTION
Self-supervised learning (SSL) designs learning objectives that use data's 's self-information but not labels. As a result, SSL empowers us to leverage a large amount of unlabeled data to learn good representations, and its applications span computer vision (Chen et al., 2020; He et al., 2020), natural language processing (Peters et al., 2018; Devlin et al., 2018) and speech processing (Schneider et al., 2019; Baevski al., 2020). More than leveraging only data's self-information, this paper is interested in weeakly-supervise data, such as the hashtags auxiliary attributes information for Instagram images. The auxiliary information can provide valuable but often noisy information. Hence, it raises a research challenge of how we can effectively leveraging useful information from auxiliary information.
setting by assuming access to additional sources as auxiliary information for
We argue that form of the valuable information provided by the auxiliary information is its implied data clustering information. For example, we can expect an Instagram image be semantically more similar to the image with the same hashtags than those with different hashtags. Hence, our first inn.iiiatioondeeeeeeeeeee
clusters. Specifically, we build data clusters such that the data from the same cluster have similar auxiliary information, such as having the same data auxiliary attributes. Then, our second step is to minimize the intra-cluster difference of the representations. Particularly, we present a contrastive approach the clustering InfoNCE (CI- InfoNCE) objective to learn similar representations for augmented variants data within the same cluster and dissimilar representations for data from different clusters. To conclude, the presented two- stage approach leverages the structural information from the auxiliary information, then integrating the structural information into contrastive representation learning process. See Figure for an overview of our approach.
step constructing
We provide the following analysis and observations to better understand our approach. First, we characterize the goodness of the CI-nnfoNCE-leamee representations via the statistical relationships between the constructed clusters and the downstream labels. A resulting implication is that we can
Equal contribution. Code available http://gggtttt.c///rraaaaaaeaee/ InfoNCK.

---

Published as a conference paper at ICLR 2022
Seff-supervision
Auxiliary Information
Our Method
Cl-nfoNCE (contrastive learning)
apphing image augmentation (different data)
data attributes
beak cone shaped? yes the upper-tail brown' yes the breast solid colored? no the breast striped? yes the throat white? yes
Y
augmentation & feature encoder
Data
D
Label: Henslow's Sparrow
Z
chuster
Auxiliary Information
label not seen during training
construction
Figure I: Left: Self-supervision. Supf-super learning (SSL) uses self- supervision (the supervision from the data itself) for learning representations. An example of sulf-supp is the augmented variant of the original data. Middle: Auxiliary Information. This paper aims to leverage auxiliary information of data for weakly-supervised representation learning. We consider data attributes (e.g.. binary indicators of attributes) as auxiliary information. Right: Our Weakly-supervisee Contrastive Learning Method. We first construct data clusters according to auxiliary information. We argue the formed clusters can provide valuable structural information of data for learning better representations. Second, we present contrastive learning approach the clustering InfoNCE (CI-InfoNCE) objective to leverage the constructed clusters.
expect better downstream performance for our weakly-superviiee representations when having i) higher mutual information between the labels and the ii) lower conditional entropy of the clusters given the labels. Second, CI-InfoNCE generalizes recent contrastive learning objectives by changing the way to construct the clusters. In particular, when each cluster contains only one data point, CI-InfoNCE becomes a conventional self-suuprvised contrastive objective (e.g., the InfoNCE objective (Oord et al., 2018)). When the clusters are built using directly the labels, CI-InfoNCE becomes a supervised contrastive objective (e.g.. the objective considered by Khosla et al. (2020)). These generalizations imply that our approach (auxiliry-inffomaaii determined clusters + Cl-InfoNCE) interpolates between conventional supe-supe and supervised representation learning.
clusters and
We conduct experiments on learning visual representations using UT-ZappoS50K (Yu & Grauman, 2014), CUB-200-2011 (Wah et al., 2011), Wider Attribute (Li et al., 2016) and ImageNet- 100 (Rus- sakovsky et al., 2015) datasets. For the first set of experiments, we shall see how much improvement can the auxiliary information bring to us. We consider the derivative auxiliary information, which means the auxiliary information comes from the datasets: the discrete attributes from UT-Zappos50K, CUB-200-2011 1, and Wider Attribute. We show that the muxiliary-informaaiie -inooonnnaionn-nneedddde weakly- supervised representations, compared conventional self-supervised representation, have much better performance on downstream tasks. We consider two basclines that also leverage auxiliary information: i) predicting the wnrlllar--iiffmmaiionnnnnde
clusters with cross-entropy loss and ii) adopting the contrastive multi-view coding (CMC) (Tian et al., 2020) method when treating auxiliary information as another view of data. Our approach consistently outperforms the cross-entropy method and performs better than the CMC method in most cases. For the second set of experiments, we focus on the analysis of CI-InfoNCE to study how well it works with unsupervised constructed clusters (K- (e-means clusters). We find it achieves better performance comparing to the mustering-basee self-supervised learning approaches, such as the Prototypical Contrastive Learning (PCL) (Li et al., 2020) method. The result suggests that the K-means method CI-InfoNCE can be strong baseline for the conventional self-supervised learning setting.
2 RELATED WORK
Self-supervised Learning. Self-supervised learning (SSL) defines a pretext task as pre-training step and uses the pre-trained features for wide range of downstream tasks, such as object detection and segmentation in computer vision (Chen et al., 2020; He et al., 2020), question answering, and language understanding in natural language processing (Peters et al., 2018; Devlin et al., 2018) and automatic speech recognition in speech processing (Schneider al., 2019; Baevski et al., 2020). In this paper, we focus on discussing two types of pretext tasks: clustering approaches (Caron et al., 2018; 2020) and contrastive approaches (Chen et al., 2020; He et 2020).
The clustering approaches jointly learn the networks' parameters and the cluster assignments of the resulting features. For example, the cluster assignments can be obtained through unsupervised clustering methods such as k-means (Caron et al., 2018), or the optimal transportation algorithms such as Sinkhorn algorithm (Caron et al., 2020). It is worth noting that the clustering approaches
2

---

Published as a conference paper at ICLR 2022
enforce consistency between cluster assignments for different augmentations of the same data. The contrastive approaches learn similar representations for augmented variants of a data and dissimilar representations for different data. Examples of contrastive approaches include the InfoNCE objective (Oord et al., 2018; Chen et al., 2020; He et al., 2020), Wasserstein Predictive Coding (Ozair et al., 2019), and Relative Predictive Coding (Tsai et al., 2021a). Both the clustering and the contrastive approaches aim to learn representations that are invariant to data augmentations.
There another line of work combining clustering and contrastive approaches, such as HUBERT (Hsu et al., 2020), Prototypical Contrastive Learning (Li et al., 2020) and Wav2Vec (Schneider et al., 2019; Baevski al., 2020). They first construct (unsupervised) clusters from the data. Then, they perform a contrastive approach to learn similar representations for the data within the same cluster. Our approach relates to these work with two differences: 1) we construct the clusters from the auxiliary information; and 2) we present C1-InfoNCH as a new contrastive approach and characterize the goodness for the resulting representations. Recent works like IDFD (Tao et al., 2021) aim to achieve better unsupervised clustering by using contrastive learning representations. However, Tao et al. (2021) differs from our work in that they don't directly incorporate auxiliary information into contrastive objectives.
Weaaly-superriseed Learning with Auxiliary Information. Our study relates to work on predic- tion using auxiliary information, by treating the auxiliary information as weak labels (Sun al., 2017; Mahajan et al., 2018; Wen et al., 2018; Radford et al., 2021; Tan et al., 2019). The weak labels can be hashtags of Instagram images (Mahajan et al., 2018), metadata such as identity and nationality of a person (Wen et al., 2018) or corresponding textual descriptions for images (Radford et al., 2021). Compared to normal labels, the weak labels are noisy but require much less human annotation work. Surprisingly, it has been shown that the network learned with weakly supervised pre-training tasks can generalize well to various downstream tasks, including object detection and segmentation, cross-modality matching, and action recognition (Mahajan et al., 2018; Radford et al., 2021). The main difference between these works and ours is that our approach does not consider a prediction objective but contrastive learning objective (i.e., the CI-InfoNCE objective). An independent and concurrent work (Zheng et al., 2021) also incorporates weak labels into the contrastive learning objective. However, our method differs from Zheng et al. (2021) by the the way construct the weak labels. We perform clustering on the annotative attributes or unsupervised k-means to obtain weak labels whereas they employ connected components labeling process. Task-wise, (Zheng et al., 2021) focuses on unsupervised (no access to data labels) and semi-supervised (access to a few data labels) representation learning, and ours focuses on weakly-supervised (access to side information such as data attributes) and unsupervised representation learning. For the common unsupervised representation learning part, we include a comparison with their method in the Appendix.
Another way to learn from auxiliary information is using multi-view contrastive coding (CMC) (Tian et al., 2020) where auxiliary information treated as another view of the data. Specifically, CMC learns representations that can capture the joint information between the data and the accompanying auxiliary information. The main difference between CMC and our approach is that CMC leverages auxiliary information directly and CI-InfoNCE leverages it indirectly (i.e., our approach pre-processes auxiliary information by clustering it).
METHOD
We present a two-stage approach to leverage the structural information from the auxiliary information for weakly-supervised representation learning. The first step (Section 3.1) clusters data according to auxiliary information, which we consider discrete attributes as the auxiliary information'. The second step (Section 3.2) presents our clustering InfoNCE (CI-InfoNCE) objective, a contrastive- learning- based approach, to leverage the constructed clusters. We discuss the mathematical intuitions of our approach and include an information iiformatiion-hhoretticaa characterization of the goodness of our learned representations. We also show that CI-InfoNCE can specialize to recent self celf-supervised and supervised contrastive approaches. For notations, we use the upper case (e.g., X) letter to denote the random variable and the lower case (e.g., x) to denote the outcome from the random variable.
Our approach generalizes to different types of auxiliary information. To help with clarity of the explanations, the paper focuses primarily on discrete attributes, but more details about other auxiliary information can be found in Appendix.
3

---

Published as a conference paper at ICLR 2022
Step by entropy
Is the beak shaped? Is the upper-tall brown? Is the breast solid colored? Is breast striped? Step Is the throar white? select top-k
Step 3r
cluster data such that data within the same cluster have the sume values of the selected attrihutes
the beak cone-shaped? no the upper-tal tail brown? the breast solid colored? no Figure 2: Cluster construction according to auxiliary information (c.g., . the data attributes).
Is the beak cone shapel? yes Is the upper-tail brown? yes the breast solid colored? no
ranked artribures
3.1 CLUSTER CONSTRUCTION FOR DISCRETE ATTRIBUTES
We consider discrete attributes as the auxiliary information. An example of such auxiliary information binary indicators of attributes, such as "short/long hair", "with/without sunglasses" or "short/long sleeves", for human photos. We construct the clusters such that data within each cluster will have the same values for set of attributes. In our running example, selecting hair and sunglasses the set of attributes, the human photos with "long hair" and "with sunglasses" will form a cluster. Then, how we determine the set of attributes? First, we rank each attribute according to its entropy the dataset. Note that if an attribute has high entropy, it means this attribute is distributed diversely. Then, we select the attributes with top-k highest entropy, where & is hyper-parameter. The reason for this selection process is to make sure the selected attributes are informative. See Figure 2 for illustration.
3.2 CLUSTERING INFONCE (CL-INFONCE) OBJECTIVE
This section presents how we integrate the clustering information of data into the representation learning process. Recently, the contrastive approaches (Chen et al., 2020; Caron et al., 2020) have attracted lots of attention for self-supervised and supervised representation learning. The goal is to learn similar representations for correlated data and dissimilar representations for uncorrelated data. To be more specific, the self-supervised setting (e.g., the InfoNCE objective (Oord et al., 2018)) regards different views of the same data correlated and distinct data uncorrelated; the supervised setting (c.g.. the supervised contrastive objective (Khosla et al., 2020)) regards the data with the same downstream label as correlated and the data with distinct labels uncorrelated. Inspired by these methods, when performing weakly-supervised representation learning, we present to learn similar representations for data within the same cluster assignment, and vice versa. To this end, we extend from the self-supervised InfoNCE objective and introduce the clustering InfoNCE (CI-InfoNCE) objective that takes the data clustering information into account. With the alphabets X and Y denoting the representations from augmented data:
X = Feature Encoder Augmentation 1 1(Data 1)) and Y Feature Encoder AAgmentation 2(Data 2))
and the alphabet Z denoting the constructed clusters, we formulate CI-InfoNCE as Definition 3.1 (Clustering- based InfoNCE (CI-InfONCE).
n
Cl InfoNCE sup E (a,n))FEwwP [Px Pvis
n Σ
log
(1)
S==1 ef(
fæz y) is any function that returns scalar from the input (I, y). As suggested by prior work (Chen et al., 2020; He et al., 2020), we choose f(r, y) = cosine (g(z), /T to be the cosine similarity between non-linear projected g(x) and (gy). g(-) is a neural network (also known as the projection head (Chen et al., 2020; He et al., 2020)) and is the temperature are independent copies of (z, y) ~ Pz [rlx) where it first samples cluster ~ Pz and then samples (z, v) pair with x ~ Px|= and y ~ Py Furthermore, we call (x,, U:) as the positively-paieed data (x; and y/ have the same cluster assignment) and (It, y3) (i * j) as the negatively-pairee data (z, and yy have independent cluster assignment). Note that, in practice, the expectation in equation 1 is replaced by the empirical mean of a batch of samples.
{(T,, 1/))
Mathematical Intuitions. Our objective is learning the representations X and Y (by updating the parameters the Feature Encooderr to maximize CI-InfoNCE. At a colloquial level, the maximiza- tion pulls towards the representations of the augmented data within the same cluster and push away the representations of the augmented data from different clusters. At a information taarrieiicaeretccalll level, we present the following:
4

---

Published as a conference paper at ICLR 2022
Theorem 3.2 (informal, CI-InfoNCE maximization learns to include the clustering information).
CI - InfoNCE & DKL. (Erz [x| Priz] |I Px PxPx - H(Z)
(2)
and the equality holds only when H( = (( 0, where HZ) is the entropy of Z and (2 (or H()))) are the conditional entropy of Z given X (or Y). Please find detailed derivations and proofs in Appendix.
The theorem suggests that CI-IffNNCE has an upper bound DKL. which measures the distribution divergence between the product of clusterin--conditonal marginal distributions (i.c., Bpz [x| Pr)z ) and the product of marginal distributions (i.e., Px Py). We give an intuition for DKL [Ep [ Priz] II PxP if DKL (Epz [Px]z Priz] Px Px is high,
ll PxPP))
then we can easily tell whether (z, y) have the same cluster assignment or not. The theorem also suggests that maximizing CI-InfoNCE results in the representations X and including the clustering information Z (:: H(Z[X)= ( = 0).
Goodness of the Learned Representations. In Theorem 3.2, we show that maximizing Cl- InfoNCE learns the representations (X and Y) to include the clustering (Z) information. Therefore, to characterize how good is the learned representations by maximizing CI-InfoNCE or to perform cross validation, we can instead study the relations between Z and the downstream labels (denoting metrics such as the mutual information T(;; T) and the conditional entropy H (Z characterize the goodness of the learned representa- tions. T(Z:7) T) measures how relevant the clusters and the labels, and H (Z(T) measures how much redundant information in the clusters that are irrelevant to the labels. For instance, we can expect good downstream performance for our
by T). In particular, we can use inormatiiontteoretienn
high mutual information and low conditional entropy between the representations when having iattaar--iinommatii-deeeeeeeeeee clusters and the labels. It is worth noting that, when Z and T are both discrete variables, computing I(Z; T) and (Z)T) would be much easier than computing [(X;7) T) and (X
Generalization of Recent Self-supervised and Supervised Contrastive Approaches. Cl- InfoNCE (equation 1) serves as an objective that generalizes to different levels of supervision according to how we construct the clusters (Z). When Z instance id (i.e., each cluster only PxIzPriz| specializes to Pxy and CI-InfoNCE specializes to the InfoNCE objective (Oord et al., 2018), which aims to learn similar representations for augmented variants of the same data and dissimilar representations for different data. InfoNCE is the most popular used self-supervised contrastive learning objective (Chen et al., 2020; He et al., 2020; Tsai et al.. 2021b). When Z = downstream labels, Cl-InfoNCE specializes to the objective described in Supervised Contrastive Learning (Khosla al., 2020), which aims lear similar representations for data that are from the same downstream labels and vice versa. In our paper, the clusters Z are determined by the auxiliary information, and we aim to learn similar representations for data sharing the same auxiliary information and vice versa. This process can be understood as weakly supervised contrastive learning. To conclude, CI-InfoNCE is a clustering- objective. By differing its cluster construction, CI-InfoNCE interpolates among unsupervised, weakly supervised, and supervised representation learning.
contains one instance), Epz, Px|z PY|z|
contrastive learning
EXPERIMENTS
We given an overview of our experimental section. Section 4.1 discusses the datasets. We consider discrete attribute information as auxiliary information for data. Next, in Section 4.2, we explain the methodology that will be used in the experiments. Section 4.3 presents the first set of the experiments, under
setting, to manifest the effectiveness of our approach and the benefits of taking the auxiliary information into account. Last, to study the effect of CI-InfoNCE alone, Section 4.4 presents the second set of the experiments under unsupervised setting. We also conduct comparison experiments with another independent and concurrent weakly supervised contrastive learning work (Zheng et al., 2021) in the Appendix.
4.1 DATASETS
We consider the following datasets. UT-zappos50K (Yu & Grauman, 2014): It contains 50, 025 shoes images along with 7 discrete attributes as auxiliary information. Each attribute follows binomial
5

---

Published as a conference paper at ICLR 2022
distribution, and we convert each attribute into set of Bernuulli attributes, resulting in total of 126 binary attributes. There are 21 shoe categories. Wider Attribute (Li et al., 2016): It contains 13, 789 images, and there are several bounding boxes in an image. The attributes are annotated per bounding box. We perform OR operation on attributes from different bounding boxes in an image, resulting in 14 binary attributes per image the auxiliary information. There are 30 scene categories. CUB-200-2011 (Wah et al., 201 I): It contains 11, 788 bird images with 200 binary attributes as the auxiliary information. There are 200 bird species. For the second set of the experiments, we further consider the ImageNet-100 (Russakovsky et al., 2015) dataset. It is a subset of the ImageNet- Ik object recognition dataset (Russakovsky et al., 2015), where we select 100 categories out of 1, 000, resulting in around 0.12 million images.
4.2 METHODOLOGY
Following Chen et al. (2020), we conduct experiments on pre-training visual representations and then evaluating the learned representations using the linear evaluation protocol. In other words, after the pre-training stage, we fix the pre-trained feature encoder and then catego- rize test images by linear classification results. We select ResNet-50 (He et al., 2016) as our feature encoder across all settings. Note that our goal is learning representations (i.e. X and Y) for maximizing the CI-InfoNCE objective (equation equation 1). Within CI-InfoNCE, the positively- paiered representations (x.y") ~ Ez~Pz [ are the learned representa- rep- PxPy are the representations from arbitrary two images. We leave the network designs, the optimizer choices, and more details for the datasets in Appendix. Before delving into the experiments, we like to recall that, in Section 3.2, we discussed using the mutual in- formation TT and the conditional entropy H (Z ) between the clusters (Z) and the labels (T) to charac- terize the goodness of CI-InfoNCE's learned represen- tations. To prove this concept, on UT-Kappps50K, we synthetically construct clusters for various (Z;T) and H (Z (T)) followed by applying CI-InfONCE. We present the results in the right figure. Our empirical results are in accordance with the statements that the clusters with higher T(Z;T) and lower H (Z)T) will lead to higher downstream performance. In later experiments, we will also discuss these two cntommation-theortiic. metrics.
tions from augmented images from the same cluster 2 ~ Pz and the negatively-patie resentations (z,y")
88
87.5
86
84
82
80
75.0
78
((21))
2
8 10 H(Z)T)
Figure 3: I(Z;T) represents how relevant the clusters and the labels; higher better. H (Z represents the redundant information in the clusters for the labels; lower is better.
4.3 EXPERIMENT I: MEKILIARY-INFORMANN D CLUSTERS + CL-INFONCE
We like understand how well CI-InfoNCE can be combined with the auxiliary information. For this purpose, we select the data discrete attributes as the auxiliary information, construct the clusters (Z) using the discrete attributes (see Section 3.1 and Figure 2), and then adopt attributes-determined clusters for CI-InfoNCE. Recall our construction of -ettrrbbuutesddeemnnnedd clusters: we select the attributes with top-k highest entropy and then construct the clusters such that the data within a cluster will have the same values over the selected attributes. it is the hyper-parameter Note that our method considers a weakly supervised setting since the data attributes can be seen as the data's weak supervision.
We dissect the experiments into three parts. First, we like to study the effect of the hyper-parameter and select its optimal value. Note that different choices of k result in different constructed clusters Z. Our study is based on the informattion-theortiicll metrics (i.c., I (Z; T') and H HZ2(T) between the constructed clusters (Z) and the labels (T)) and their relations with the downstream performance of the learned representations. Second, we perform comparisons between different levels of supervision. In particular, we include the comparisons with the supervised (Z downstream labels T) and the conventional self-supervised (Z = instance ID) setting for our method. We show in Section 3.2, the supervised setting equivalent to the Supervised Contrastive Learning objective (Khosla et al., 2020) and the conventional self-supervised setting is equivalent to SimCLR (Chen et al., 2020). Third, we include baselines that leverage the auxiliary information: i) learning to predict the clusters assignments using cross-entropy loss and ii) treating auxiliary information as another view of data when using the contrastive multi-view coding (CMC) (Tian et al., 2020).

---

Published as a conference paper at ICLR 2022
0.0
1.0
0.0
1.0
0.0
1.0
-0.5
(2:TT
.HIZITI
0.5
N2:11
+12211
-0.5
HIZIT)
(12:1)
-0.5
(02:1)
-H(2(T)
0.5
112:11
50
100
-10
2.5
5.0
7.5
16.0
12.5
-1.0
50
100
150
200
Number attributes selected (k)
Numbar attributes selected (k)
Number attributes selected (k)
Accurarcy
Top3 accuraccy
Topa Accurarcy
20
44
36
42
35
Top1 Acturarcy
Top1 Accurarcy
Top1 Acturarcy
50 Number ottributes selected
100
2.5
12.5 Numbar attributes selected (K)
5.0
7.5
10.0
50 100 150 200
Number attributes selected
(a) UT-Zappos50K
(b) Wider Attribute
CUB-200-2011
Figure 4: Experimental results for attributes-ettemmined clusters + CI-InfoNCE: by tuning the nyper-parament k when constructing the clusters. Note that we select attributes with top-k highest entropy. and we construct the clusters such that the data within a cluster would have the same values for the selected attributes. Z are the constructed clusters. and T' the downstream labels. We find the intersection between the re-scaled [(Z; T) and the re-scaled (Z)T) gives us the best downstream performance.
UT-Zappos50K
Wider Attribute Top-5 Acc.
CUB-200-2011
Method
Top-I Acc. Top-5 Ace.
Top-1 Acc.
Top-1 Acc. Top-5 Acc.
(Supervised) Labels + CI-InfoNCE
SupCon (Khosla et al., 2020)
89.010.4
99.41 0.3
49.910.8
76,210.2
59.910.7
78.81 0.3
(Weadty-sperrvieed) Attributes- Attibuute--ettrmineed Clusters CI-InfoNCE
Ours
84.6.10.4
99.1
45.510.2
75.410.2
20.6.1 0.5
0..
(Self-supervisee)) Instance- ID + CI-IffoNCE
SimCLR (Chen et al., 2020)
77.8
97.910.8
40.210.9
73.020.3
14.1 0.7
35.2=0.6
Table 1: Experimental results for different supervision levels with the presented CI-InfoNCE objective. For clusters, report the best results by tuning
weakly-supervised methods that consider the hyper-parameter k. The results suggest that, with the help of auxiliary information, we can better close the performance gap between supervised and self supervueee representations.
Part 1 Effect of the Syper-paramelte k. To better understand the effeect of the hyper-paramedt for constructing the detibbtes-ddtermined clusters, we study the information-theoretiee metrics between 2 and T and report in Figure 4. Note that, to ensure the same scales for / (Z;T) and H (Z(T) across different datasets, we normalize I I(Z; T) and HZT using
(esn))minn?((;17 I(Z:T)+ max minz/(Z;1])
HZ min max; (Z min (2
and H(Z
As k increases, the mutual information I(Z:T) increases but the conditional entropy (Z(T)) also increases. Hence, although considering more attributes leads the clusters that more correlated to the downstream labels, the clusters may also contain more downstream waatttrcaam-irreeaan information. This is in accord with our second observation that, as k increases, the downstream performance first increases then decreases. Therefore, we only need partial set of the most informative attributes (those with high entropy) to determine the clusters. Next, we observe that the best performing clusters happen at the intersection between I(Z: and negative H (Z) This observation helps us study the trf between I (Z; and HZ))T) and suggests an empirical way to select the optimal & that achieves the best performance. It is worth noting that the above process of determining the optimal k does not require directly evaluating the learned representations.
Part II - Interpolation between Different Supervision Levels. In Section 3.2, we discussed that, by altering the designs of the clusters, our presented approach specializes to the conventional self- supervised contrastive method SimCLR (Oord et al., 2018) and the supervised contrastive method- SupCon (Khosla et al., 2020). In particular, our approach specializes to SimCLR when considering augmented variants of each instance cluster and specializes to SupCon when considering instances with the same downstream label as a cluster. Hence, we can interpolate different supervision levels of approach and study how auxiliary information of data can help improve representation learning.
7

---

Published as a conference paper at ICLR 2022
Method
Top-1 1 Acc. Top-5 Acc.
UT-ZappusS0K
Top-1 Acc.
Wider Attribute
Top Acc.
Top-1 Acc. Top Acc.
CUB-200-2011
Self supervised Representation Learning
MoCo (Ile et al., 2020) SimCLR (Chen et al,, 2020)
83.4.0.2
99.110.3
41.010.7
74.0.104 73.0.1..
13810.7
36.5.10.5
77.8.1.5
97.9.088
40,240.9
14.14 0.7
35.24...6
Wookly superriied Representation Leorning
Contrastive Multi-viow Coding (CMC) (Tian et al., 2020)
83.7=0.55
09.2:00.3
34.11112
65.3=007
32.720.3
61,820,5
Attrbuute- Altibutes-ettermnee Determined Clusters Eotso-E Loss
82.7=0.77
0930:0.33
39.410.6
68.6:02
17.52:00
46.020.8
Attributes Determined Clusters Cl-InfoNCE (Ours)
84.6:1044
99.1:00.2
45.510.2
75.4:0.2
20.61 0.5
47.010.5
Table 2: Experimental results for seakly-supervised representation methods that leverage auxiliary information and supf-sup representation methods. Best results are highlighted in bold. The results suggest that method outperforms the supervised baselines in most cases with the exception that the CMC method performs better than our method on the CUB-200-2011 dataset.
We present the results in Table 1 with different cluster constructions along with CI-InfONCE. We use the top-1 accuracy on Wider Attribute for discussions. We find the performance grows from low to high when having the clusters as instance ID (40.2), attributes-determindd clusters (45.5) to labels (49.9). This result suggests that CL-InfoNCE can better bridge the gap with the supervised learned representations by using auxiliary information.
Part III Comparisons with Baselines that Leverage Auxiliary Information. In the last part, we see that CI-InfoNCE can leverage auxiliary information to achieve a closer performance to supervised representations than self-supervised representations. Nonetheless, two questions still remain: 1) is there another way to leverage auxiliary information other than our method (attributes- determined clusters Cl-InfoNCE), and 2) is the weakly-supervisee methods (that leverages auxiliary information) always better than self-supervised methods? To answer these two questions, in Table 2, we include the comparisons among weakly-supervise representation learning baselines that leverage auxiliary information (Atributes-determinnd clusters + cross-entropy loss and Contrastive Multi-view Coding (CMC) (Tian et al., 2020) when treating auxiliary information as another view of data) and self-supervised baselines (SimCLR (Oord et al., 2018) and MoCo (He et al., 2020)).
First, we find that using auxiliary information does not always guarantee better performance than not using it. For instance, for top-1 acc. on Wider Attribute dataset, predicting the surriutes-determingg clusters using the cross- cross-yntropy loss (39.4) or treating auxiliary information as another view of data then using CMC (34.1) perform worse than the SimCLR method (40.2), which does not utilize the auxiliary information. The result suggests that, although auxiliary information can provide useful information, how can effectively leverage the auxiliary information is even more crucial.
Second, we observe that our method constantly outperforms the baseline Attributes-Determine Clusters + Cross-Entropy loss. For instance, on ZT-Zappos50K, our method achieves 84.6 top-1 accuracy while the baseline achieves 82.7 top-1 accuracy. Note that both our method and the baseline consider constructing clusters according to auxiliary information. The difference is that our method adopts the contrastive approach- CI-InfoNCE, and the baseline considers to adopt cross-entropy loss on an additional classifier between the representations and the clusters. Our observation is in accordance with the observation from prior work (Khosla et al., 2020). It shows that, compared to the cross- cross-entropy loss, the contrastive objective (e.g., our presented CI-InfoNCE) more robust to natural corruptions of data, stable to hyper-parameters and optimizers settings, and enjoying better performance.
I Last, we compare our method with the CMC method. We see that although our method performs better on UT-zappos50K (84.6 over 83.7) and Wider Attributes (45.5 over 34.1) dataset, CMC achieves significantly better results differences, we recall that 1) the CMC method leverages the auxiliary information directly, while method leverages the auxiliary information indirectly (we the structural information implied from the auxiliary information); and 2) the auxiliary information used in UT-zappos50K and Wider Attributes contains relatively little information (i.e., consisting of less than 20 discrete attributes), and the auxiliary information used in CUB-200-201 contains much more information (i.e., consisting of 312 discrete attributes). We argue that since CMC leverages the auxiliary information directly, shall perform better with more informative auxiliary information. On the other hand, CI-InfoNCE performs better with less informative auxiliary information.
(32.7 over 20.6) dataset. To explain such
8

---

Published as a conference paper at ICLR 2022
Method
Top-1 (Accuracy)
UT- S0K
Wider Attribute Top-1 (Accuracy
1op-1 (Accuracy)
CUB-2000 200-2011
Top-1 (Accuracy)
ImageNett-
112-4)
Non-cluttring-eee Seff-oxperviod Appenaches
Ictaa Cletii CHaRINE
R (Chen al. 2020)
77.8+1.5
40.2+009
14.1-07
58.2+
McCo (Ho et 2020))
43.420.5
41.0=0077
13
$9.4
Cluttering dused Seff-axpervisee Approsctts cluotorr 1// 18/2.50)
PCL. (Li al, 2020)
82.4-0.5
41.0=0.4
14,4-0.5
68.9-007
0,53
K-msans + CI-IofoNCE (onas)
84.5=0.4
43.6-0.4
17.6=00.
77,9-0.7
8603
100 125
150
Training Epachs
Figure 5: Experimental results under conventional self- superviee setting (pre-training using no label su- pervision and no auxiliary information). Left: We compare our method (K KK-eaas clusters + CI-InfONCE) with self-supervised approaches that leverage and do not consider unsupervised clustering. The downstream performance is reported using the linear evaluation protocal (Chen et al., 2020). Right: For our method and Prototypical Contrastive Learning (PCL), we plot the mutual information (T(Z; T)) and the conditional entropy (ZT)))) versus training epochs. Z are the unsupervised clusters, and T are the downstream labels. The number of clusters is determined via grid search over (500, 1, 000, 5, 000, 10, 000).
4.4 EXPERIMENT II: K-MEANS CLUSTERS CL.IINFONCE
So far, we see how we can combine auxillar-informatio--deeeee to learn good weakly-supervised representations. Now, we would like to show that Cl-InfoNCE can also learn good self-supervised representations without auxiliary information. To this end, we construct unsupervised clusters (e.g.. k-means clusters on top of the learned representations) for CI-InfOCEE. Similar to the EM algorithm, we iteratively perform the k-means clustering to determine the clusters for the representations, and then we adopt CI-InfoNCE leverage the k-means clusters to update the representations. We select thet Prototypical Contrastive Learning (PCL) (Li et al., 2020) as the baseline of the clustering- saased self-supervisede approach. In particular, PCL. performs data log-likelihood maximization by assuming data are generated from isotropic Gaussians. It considers the MLE objective, where the author makes a connection with contrastive approaches (Chen et al., 2020; He et al., 2020). The clusters in PCL are determined via MAP estimation. For the sake of the completeness of the experiments, we also include the non-clustering-bee based self-supervised approaches, including SimCLR (Chen et al., 2020) and MoCo (He et al., 2020). Note that this set of experiments considers the conventional self-supervised setting, in which we can leverage the information neither from labels nor from auxiliary information.
clusters and CI-InfoNCE
Results. We first look at the left table in Figure 5. We observe that, except for ImageNet- there is no obvious performance difference hetween the non-clustering-basee (i.e., SimCLR and MoCo) and the clustering-basede baseline (i.e., PCL.. Since ImageNet- is more complex dataset comparing to the other three datasets, we argue that, when performing self-supervised learning, discovering latent structures in data (via unsupervised clustering) may best benefit larger-sized datasets. Additionally, among all the approaches, our method reaches the best performance. The result suggests our method can be as competitive as other conventional self supervisdd approaches.
Next. we look at the right plot in Figure 5. We study the mutual information IZZ and the conditional entropy H (2 between the unsupervised constructed clusters Z and the downstream labels T. We select our method and PCL., providing the plot of the two metrics versus the training epoch. We find that, as the number of training epochs increases, both methods can construct unsupervised clusters that are more relevant (higher I (Z;TT) and contain less redundant information (lower (2 (T)) about the downstream label. This result suggests that the cesstrinn-based self-supervised approaches are discovering the latent structures that are more useful for the downstream tasks. It is worth noting that our method consistently has higher I(Z;T) and lower H (Z(T) comparing to PCL.
CONCLUSION AND DISCUSSIONS
In this paper, we introduce the clustering InfoNCE (CI-InfoNCE)) objective that leverages the implied data clustering information from auxiliary information or data itself for learning weakly-supervised representations. Our method effectively brings the performance closer to the supervised learned rep- resentations compared to the conventional self-supervised learning approaches, therefore improving pretraining quality when limited information is at hand. In terms of limitation, our approach requires clustering based on auxiliary information data itself. This process sometimes could pose additional computational cost. In addition, clustering on auxiliary information or data will also lose precision. Tackling these problems would be our further research direction.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
All authors of this work have read the ICLR code of ethics and commit to adhering to it. There is no ethical concern in this work.
REPRODUCIBILITY STATEMENT
The code for reproducing our results in the experiment section can be found at com/Crazy
ACKNOWLEDGEMENTS
The authors would like thank the anonymous reviewers for helpful comments and suggestions. This work is partially supported by the National Science Foundation IIS 163562,, IARPA DI D17PC00340, ONR Grant N0001418 11261111161, Facebook PhD Fellowship, BMW, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards ROIMHI25740 ROIMH096951 and U 16925. Any opinions, findings, conclusions, or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.
REFERENCES
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-super learning of speech representations. arXiv preprint arXiv:2006. 1477, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 098828, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300, 2013.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXhv:1810.04805. 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the 1EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Wei-Ning Hsu, Yao- Hao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can a bad teacher benefit asr pre-training. In Neural Information Processing Systems Workshop Self- Sepe-Super Learning for Speech and Audio Processing Workshop, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. /1362, 2020.
10

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:200S. 000.20005.0496,,, 2020.
Yining Li, Chen Huang, Chen Change Loy, and Xiaoou Tang. Human attribute recognition by deep hicrarchical contexts. In European Conference Computer Vision, 2016.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits weakly supervised pretraining. In Proceedings of the European Conference Computer Vision (ECCV), pp. 181-196, 2018.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39 41, 1995.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. arXiv preprint 11770 2019.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 2tt.::211300002200 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Steffen Schneider, Alexci Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint arXiv: wwt.: 05862, 2019.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. arXiv preprint arXiv:2007.09852. 2020.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference 2017.
computer vision, pp.
Reuben Tan, Mariya I Vasileva, Kate Saenko, and Bryan A Plummer. Learning similarity conditions without explicit supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10373-10382, 2019.
Yaling Tao, Kentaro Takagi, and Kouta Nakata. cesterinn--rienddi representation learning via instance discrimination and feature decorrelation. arXiv preprint arXiv:21066001311 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Eision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang. Han Zhao, Louis-Philippe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. arXiv preprint 103. 1225, 2021a.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Moreney, Self-supervised learning from a multi-view perspective. In ICLR, 2021b.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd binds-200-2011 dataset. 2011.
11

---

Published as a conference paper at ICLR 2022
Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh. Disjoint mapping network for cross-modaa matching of voices and faces. arXiv preprint 1rXiv1187.004836, 2018.
Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 192-199, 2014.
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang. Xiaogang Wang, and Chang Xu.
Weakly supervised contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10042-1005 2021.
12

---

Published as a conference paper at ICLR 2022
APPENDIX A COMPARISON WITH OTHERS RELATED IN LITERATURE
A.1 EXPERIMENTS: COMPARISON WITH WEAKLY SUPERVISED CONTRASTIVE LEARNING(ZIIENO ET AL., 2021)
We would like to point out that a concurrent work Zheng et al. (2021) presented a similar idea on reakly-supervisee contrastive learning in ICCV 2021. We would like to point out the reason is a concurrent work with ours. Zheng et al. (2021) is made publicly available on 10/05/2021, which is the same day as the the paper submission deadline for ICLR'22. To be more precise, ICCV publicly released this paper on 10/05/2021, and the paper's 's arxiv version and code are available on 10/10/2021. The zero time overlap suggests that our two works are independent and concurrent.
Similarity and Difference We acknowledge that the two works share the similar idea of utilizing weak labels of data in contrastive learning. Zheng et al. (2021) motivates by preventing class collision during instance-wise contrastive learning (random data that belongs to the same category will possibly get falsely pushed away in instance-wise contrastive learning), and ours motivates by exploring the structural information of data within contrastive learning. followed by providing analysis to explain how different structural information can affect the learned representations. Task- wise Zheng et (2021) focuses on unsupervised (no access to data labels) and semi-supervised (access to a few data labels) representation learning, and ours focuses on weakly supervised (access to side information such as data attributes) and unsupervised representation learning. For the common unsupervised representation learning part, Zheng et al. (2021) presents to generate weak labels using connected components labeling process, and ours generates weak labels using K-means clustering.
intormation-thoreee
Empirical Results We observed that the performance ImageNet- 100 reported in [1] looks better than ours (79.77 Zheng et al. (2021) v.s. ours 77.9 Figure 5). However, the experimental settings differ lot. First, the datasets are different despite the same name: Zheng et al. (2021) considers ImageNet-- 100 by selecting the first 100 class of the ILSVRC 2012 challenge, and we select a different set of 100 classes (details shown in the Appendix E). Second, the batch size different: Zheng et al. (2021) considers 2048 and ours considers 1 128. Third, the projection heads in architecture are different: Zheng al. (2021) uses projection heads (each with 4096 hidden units) with two objectives, one for InfoNCE and the other is for the proposed Weakly Supervised Contrastive Learning loss; whereas ours uses one projection head with 2048 hidden units for CI-InfoNCE objective only. Although our main experiments have demonstrated that CI- CI-InfoNCE alone can achieve competitive performance, we acknowledge that adding InfoNCE objective with an additional linear projection head would further improve the learned representation.
To fairly compare our CI-InfONCE loss with their proposed Weakly Supervised Contrastive objective, add an additional head trained with InfoNCE along with CI-InfoNCE objective. Experiments are conducted on our version of ImageNet 00 with the controlled set up: same network architecture of resnet50, same batch size of 384, same training epochs of 200, same projection head (2048-2048 128), the same optimizer and linear evaluation protocols, etc. Our Kmeans cluster number K is chosen to be 2500 via a grid search from {100, 1000, 2500, 5000, 10, 0003. The results are shown below Table 3.
Method
Top-1 Accuracy (%)
Weakly Supervised Contrastive Learning Zheng ct al. (2021)
82.2 0.4
CL-InfoNCE Kmeans (Ours)
82.6 0.2
Table 3: Results on ImageNet-100 (Russakovsky al., 2015) compare with concurrent and independent work Zheng et al. (2021).
From the results, are done independently and concurrently, and both works allow a broader understanding of weakly supervised contrastive learning.
that the two methods' performances are similar. Our work and theirs [1]
13

---

Published as a conference paper at ICLR 2022
A.2 EXPERIMENTS: COMPARISON WITH IDFD (TAO ET AL., 2021)
IDFD (Tao et al., 2021) presents to learn representations that are clustering friendly (from a spectral clustering viewpoint) during the instance discrimination (ID) contrastive learning process. Although it includes both ideas of clustering and contrastive learning, IDFD (Tao et al., 2021) differs from our paper fundementally because they does not utilize the constructed clusters as weak labels to train contrastive objective. However, IDFD (Tao et al., 2021) can still be considered representation learning method, hence we perform experiments to compare our unsupervised setting (CI-InfoNCE: + Kmeans method) with their proposed IIFD on CIFAR1O 10 Dataset (Krizhevsky et al., 2009). To provide a fair comparison with IDFD (Tao et al., 2021), we stick to the training paradigm of IDFD where they replaces Resnet-50 with Resnet-18. 18. The batch size of 128 is used following their report. Since IDFD (Tao et al., 2021) was focusing on clustering quality and didn't I report the linear evaluation protocol, we use the released code of IDFD (Tao et al., 2021) re-train the model meanwhile using both the cluster accuracy and the linear evaluation protocal as evaluation metrics. We train both methods for 1000 epochs for a fair comparison. The results are presented in Table 4.
self- self-supervised
550 Epechs
750 Epochs
1000 Epochs
Methods
Lineur Eva.
Clustering Acc.
Linear Eva.
Clustering Acc.
Lineur Eva.
Clustering Acc.
IDFD (Tho et ul., 2021)
81.7 + 0.9
66.9 + 1.4 78.3
82.1 = 1.2
70.1 1.7
84.4 1.1
77.7 1.5
CI- CI-InfoNCC Kmeans (ours)
89.5
90.2 0.7
79.1 2.1
90.7
81.1
Table 4: Comparison with IDFD (Tao et al., 2021) on CIFARIO dataset (Krizhevsky et al., 2009). Two evaluation metrics, Linear evaluation and clustering accuracy are measured during the training epochs. The Kmeans hyperparameter K determined followed by a grid search from {10, 100, 1000, 2500}.
Note that (Tao al., 2021) proposed 2 methods (IDFD and IDFO), we choose the compare with IDFD because (i) IDFO is very unstable, (ii) IDED/IDFO perform at-par for the best performance based Figure2 in (Tao et al., 2021) and (iii) (Tao et al., 2021) only officially releases code for IDFD. We can observe that our method exceeds IDFD on in terms of top-1 classification accuracy during linear evaluation and also improve the raw clustering accuracy score, indicating integrating weak labels from unsupervised clustering with contrastive objectives would help both representation learning and the unsupervised clustering task.
APPENDIX B DATA'S HIERARCHY INFORMATION AS AUXILIARY INFORMATION
In the main text, we select the discrete attributes as the auxiliary infor- mation of data, then presenting data cluster construction according to the discrete attributes. We combine the constructed clusters and the presented CI-InfoNCE objective together for learning weakly- supervised representations. In this section, we study an alternative type of the auxiliary information data labels' hierarchy information, more specifically, the WordNet hierarchy (Miller, 1995), illustrated in the right figure. In the example, we present the WordNet hier- archy of the label "Henslow's 's Sparrow", where only the WordNet hierarchy would be seen during training but not the label.
(WordNet) hierarchy information
animal
aquatic vertebrate
vertebrate
bird
mammall
fish
Label: Henslow's 's Sparrow
label not seen during training
B. CLUSTER CONSTRUCTION FOR WORDNET HIERARCHY
How do we construct the data clusters according to the WordNet hierarchy? In the above ex- ample, "vertebrate" and "bird" can be seen as the coarse labels of data. We then construct the clusters such that data within each cluster will have the same coarse label. Now, we explain how determine which coarse labels for the data. First, we represent the Word- Net hierarchy into a tree structure (each children node has only one parent node). Then, we choose the coarse labels to be the nodes in the level in the WordNet tree hierarchy (the
14

---

Published as a conference paper at ICLR 2022
CL. re.representss CHnnoNCE and CE represents learning to predict the cluster assignments using Eross-Entropy lossi
10
TopI Accuracy
85-
80
75-
Amsinnlak Fy
HIZ T
6
70
Maredy damm
3
dhevrrran Mack
4-
dontatooniie (dde))
2-
8
10
11 12 13
7
8
10 11 12 13
11 12
Hlerarchical Level (f)
Hierarchical Level (f)
Hierarchical Leve (f)
Figure 6: Experimental results on ImageNet- 100 for CI-InfoNCE under supervised (clusters downstream
labels 7), weakly supervised (Z = hierarchy clusters) and conventional self-supervised (Z = instance ID) setting. We also consider the baseline- learning predict the clustering assignment using the cross-entropy loss. Note that construct the clusters such that the data within cluster have the same parent node in the level f in the data's s WordNet tree hierarchy. Under this construction, root node of level 1, and the downstream labels are of the level 14. I(ZTT) is the mutual information, and (2 is the conditional entropy.
root node is level 1). I is
We illustrate the process in the below figure.
Step select nodes
animal
level-e
Step 3:
represent hierarchy
cluster data such that data within the same cluster have the node values
tree structure
vertebrate
aquatic vertehrate
hird
mammal
fish
vertebrate
aquatic vertebrate
B.2 EXPERIMENTS: HEEA--HERARCHH-DAA MINER CLUSTERS + CI.-INFONCE
The experimental setup and the comparing baselines are similar to Section 4.3 in the main text, but now we consider the WordNet (Miller, 1995) hierarchy as the auxiliary information. As discussed in prior subsection, we construct the clusters Z such that the data within a cluster have the same parent node in the level l in the data's WordNet tree hierarchy. / is the hyper-parameter".
Results. Figure 6 presents our results. First, we look at the leftmost plot, and we have several similar observations when having the data attributes as the auxiliary information. One of them is that intor--noooooooneteeeeee
our approach consistently outperforms the loss. Another of them is that the weakly supervised representations better close the gap with the supervised representations. Second, as discussed in prior subsection, the WordNet data hierarchy clusters can be regarded as the coarse labels of the data. Hence, when increasing the hierarchy level 1, we can observe the performance improvement (see the leftmost plot) and the increasing mutual information (Z;TT) (see the middle plot) between the clusters Z and the labels T. Note that H(Z T") remains zero (see the rightmost plot) since the coarse labels (the intermediate nodes) can be determined by the downstream labels (the leaf nodes) under the tree hierarchy structure. Third, we discuss the conventional self-supe setting with the special case when Z instanced IDD. Z as the instance ID has the highest T(Z: 7) (see the middle plot) but also the highest (Z (see the rightmost plot). And we observe that the conventional self-supervised representations perform the worse (see the leftmost plot). We conclude that, when using clustering-bue based representation learning approaches, we shall not rely purely on the mutual information between the data clusters and the downstream labels to determine the goodness of the learned representations. We shall also take the redundant information in the clusters into account.
clusters + cross-entropp
APPENDIX C THEORETICAL ANALYSIS
In this section, we provide theoretical analysis on the presented CI-InfoNCE objective. We recall the definition of CI-InfoNCE and our presented theorem:
Definition C.1 (Clustering-base based InfoNCE (CI-InfoNCE), restating Definition 3.1 in the main text).
ef (m)a))
Cl InfoNCE sup E (m..)) [Px en log
E (
Note that do not compare with the CMC method for fair comparisons with other method. The reason is that the CMC method will leverage the entire tree hierarchy, instead of certain level in the tree hierarchy.
15

---

Published as a conference paper at ICLR 2022
Theorem C.2 (informal, CI-InfoNCE maximization learns include the clustering information, restating Theorem 3.2 in the main text).
Cl InfoNCE & DKL (Erz PxizPriz] PxPr) H(Z) and the equality holds only when H(Z[X) - H(Z
Our goal is to prove Theorem C.2. For better presentation flow, we split the proof into three parts:
Proving C1 - InfoNCE & DKL (Epz PPx2 PxPy) in Section C.1
Proving DKL. (Epz Pxi2Pr/2) PxP)) < H() in Section C.2
Proving C1 InfoNCE maximizes (Z) when H(Z[X) H(ZIY) Sec- tion C.3
C.I PART - PROVING Cl InfoNCE & DKL (Epz PPiz 1
The proof requires the following lemma.
Lemma C.3 (Theorem by Song & Ermon (2020)). Let X and y be the sample spaces for X and Y, f be any function: (X' x )) + R, and P and Q be the probability measures on X x Y Then,
(z.01)
sup (a.n))>P Po 1) log 1 EE= (x.us) f
DKL (P112).
Now, we are ready prove the following lemma:
log ((FLNN) I 2 (Fsiy)))
Lemma C.4 (Proof Part I). C1 InfoNCE sup E
f (#...))Bawrg Px]x
8n
-i=1
DKL Epg [PxizPriz
Px Py)
Proof. By defining P = Epp [x]z Priz] and Q = PxPy, we have
(((m)))
f
log
n プド= (((ss)))
= (#..E) [Pxp
Σ
log
DØ= (w)
1=1
Plug in this result into Lemma C.3 and we conclude the proof.
C.2 PART II PROVING DKL [Epz [x|z (x (H(Z)
The proof requires the following lemma:
Lemma C.5. DKI. EEz [Px12 Priz]
PxPr) < min (MI(2
16

---

Published as a conference paper at ICLR 2022
Proof.
MI(Z;X)- DKL (Epz Pxi2Pr12 Pr12) [
=s. { p(z) p(x)2) log p(z(z) drd= p(2)) [ p(oo)) / p(ulz) log Se ((=p())((a ) (((ll2))) drdydz
p(z)
p(z)p(y)
- p(2) p(z)z) log p(z(z) dædz p(z)
p(2) p(x))) /. p(()2) log [= dzdydz
p(z)
S p(2) p(x1z) / p(u1=) log pl p(2 ()p())))) drdydz
p(0(22
J. p(z) J. plokk) p(Ulz) log
Ser ((a)) da'
drdydz
p(x|z)
J. p(z) / p((r))) / p(ulz) (L.:(e"ll))(xse")
-1
-1) dxdyyd (::log (t-1)
p(rlz))
=0.
Hence, MI(Z; X)
DEL [P Priz] PxPr)
Likewise, MI(Z;Y)
DKL Ep3 Px|z Priz] Px Pv). We complete the proof by combining the two results.
Now, we are ready prove the following lemma:
Lemma C.6 (Proof Part II). DKL [Epz [PxzzPrizt | PxPx) s H()).
Proof. Combining Lemma C.5 and the fact that min (I S HZ),, complete
the proof. Note that we consider as the clustering assignment, which is discrete but not continuous. And the inequality holds for the discrete Z, but may not hold for the continuous Z.
C.3 PART III PROVING Cl InfoNCE maximizes H(Z) when H(ZIX)= HZ
We directly provide the following lemma:
Lemma C.7 (Proof Part III). Cl InfoNCE max. at H(Z) when H(Z H(Z
Proof. When (Z( = 0, 00,( y) is Dirac. The objective
DRt. [8 Priz] 1|
p(=) I plal)) / p(ulz) log Je (((="))((==2)(((1le drdydz
p(x)p())
[ p(=)) S. plal=) p(u)=) log
[x*x('2) drdydz
p(r)
[ p(z) S. p((l2) p(ulz) log (xx=')(((==')((l/// dadydæ
p(z)p())
f p(z) / p(ale) / p(ul=) log p(z)
p(+|2) dædydæ = MI xx:)..
The second-last equality comes with the fact that: when p(Z = y) is Dirac, p(z/ (8) = 1 22 = and p(='ly) Vz' # 2. Combining with the fact that MT [2 H(Z) when H(7 =
we know DKT. EEz Px12 P112 II PxPv) H()) when H(ZIX) = H(()Y) = 0.
Furthermore, by Lemma C.4 and Lemma C.6, we complete the proof.
17

---

Published as a conference paper at ICLR 2022
C.4 BRINGING EVERYTHING TOGETHER
We bring Lemmas C.4, C.6, and C.7 together and complete the proof of Theorem C.2.
APPENDIX ALGORIHTMM
In this section, we provide algorithms for our experiments. We consider two sets of the experiments. The first one K-means clusters + Cl-InfoNCE (see Section 4.4 in the main text), where the clusters involved in CI-InfoNCE are iteratively obtained via K-means clustering top of data representations. The second one is main text and Section B.2), where the clusters involved in CI-InfoNCE are pre-determined accordingly to data attributes (see Section 4.3 in the main text) or data hierarchy information (see Section B.2).
clusters + CI-InfoNCE (see Section 4.3 in the
K-means clusters + CI-InfoNCE We present here the algorithm for K-means clusters + Cl- InfoNCE. At each iteration in our algorithm, we perform K-means Clustering algorithm on top of data representations for obtaining cluster assignments. The cluster assignment will then be used in our CI-InfONEE objective.
Algorithm 1: K-means Clusters Cl-InfoNCE
Result:
Pretrained Encoder fo(.)
fo(.) f Base Encoder Network;
Aug (-) Obtaining Two Variants of Augmented Data via Augmentation Functions;
Embedding + Gathering data representations by passing data through fe(-);
Clusters EetmecansccusstrrrnnEEEeediiiiie
for epoch in . do
for batch I,2.....M do
datal, data2 +- aag(data_batch)) featurel, feature2 t- (ddtta)), (ddat22));
LCI-infoNCE t- Cl-nfoNCEE(icturrll, feature2, Clusters);
fo + fo ir 38 La-infoNCE;
end
Embedding +- gather embeddings for all data through fo(-);
Clusters E.-mmann--lusteriigggnmeeeee
end
Auxiliary information determined clusters + CI-InfoNCE We present the algorithm combine -etiuttiiondetemmnnedd clusters with CI-InfONCE. We select data attributes or data hierar- chy information as the auxiliary information, and we present their clustering determining steps in Section 3.1 in the main text for discrete attributes and Section B. for data hierarchy information.
Algorithm 2: Pre-Determined Clusters CI-InfoNCE
Result: Pretrained Encoder fe(.)
fo(.) Base Encoder Network:
Aug (.) - Obtaining Two Variants of Augmented Data via Augmentation Functions;
Clusters < Pre--eetemminnngg Data Clusters from Auxiliary Information;
for epoch in 1,2....N do
for batch in 1, do
datal, data2 + Aug(data batch);
featurel, feature2 fo(datal), fo(atta2);
LCI LeLinioNCE + CI-InfoNCEffeaturell feature2, Clusters); fo + fo Ir a 80 LeHatONCE:
end
end
18

---

Published as a conference paper at ICLR 2022
APPENDIX E EXPERIMENTAL DETAILS
The following content describes our experiments settings in details. For reference, our code is available at https://githibt com wwt.:///cccaa---aaeaa--aaataaaea mdd
E.1 UT-ZAPPOS50K
The following section describes the experiments we performed on UT-Zappos50KK dataset in Section 4 in the main text.
Accessiblity
The dataset is attributed to (Yu & Grauman, 2014) and available at the link: http://
vision. cs utaxas edue prr pwwojjecttt weeets/fiineee finegrained// ut zap55k. The dataset is for non- commercial only.
Data Processing The dataset contains images of shoe from Zappos.com, We rescale the images to 32 x 32. The official dataset has 4 large categories following 21 sub-categories. We utilize the 21 subcategories for all our classification tasks. The dataset comes with 7 attributes information. We binarize the 7 discrete attributes into 126 binary attributes. We rank the binarized attributes based on their entropy and use the top-k binary attributes form clusters. Note that different k result in different data clusters (see Figure 4 (a) in the main text).
auxiliary
Training and Test Split: We randomly split train-validation images by 7 : ratio, resulting in 35, 017 train data and 13, 008 validation dataset.
Network Design We use ResNet-50 architecture to serve as backbone for encoder. To compensate the 32x32 image size, we change the first 7x7 2D convolution to 3x3 2D convolution and remove the first max pooling layer in the normal ResNet-50 (See code for detail). This allows finer grain of information processing. After using the modified ResNet-50 as encoder, we include 2048-2048- 128
Multi-Layer Perceptron (MLP) the projection head (i.e.. g(-) in f(-:) equation (1) in the main
text) for CI-InfoNCE. During evaluation, we discard the projection head and train a linear layer on top of the encoder's output. For both K-means clusters + Cl-InfoNCE and determined clusters + CI-InfoNCE, we adopt the same network architecture, including the same encoder, the same MLP projection head and the same linear evaluation protocol. In the K-means + CI-InfoNCE settings, the number of the K-means clusters is 1, 000. Kmeans clustering is performed every epoch during training. We find performing Kmeans for every epoch benefits the performance. For fair comparsion, we use the same network architecture and cluster number for PCL.
Optimization We choose SGD with momentum of 0.95 for optimizer with a weight decay of 0.0001 prevent network over-fitting To allow stable training, we employ linear warm-up and cosine decay scheduler for learning rate. For experiments shown in Figure 4 (a) in the main text, the learning rate set to be 0.17 and the temperature chosen be 0.07 in CI-InfoNCE. And for experiments shown in Figure 5 in the main text, learning rate is set to be 0.1 and the temperature is chosen be 0.1 in CI-InfONCE.
Computational Resource We conduct experiments on machines with 4 NVIDIA Tesla P100. takes about 16 hours to run 1000 epochs of training with batch size 128 for both auxiliary information aided and unsupervised CI-InfoNCE.
E.2 WIDER ATTRIBUTES
The following section describes the experiments we performed on Wider Attributes dataset in Section 4 in the main text.
Accessiblity The dataset is credited to (Li al., 2016) and be downloaded from the link: hwwhh///eeee http///ojcctssoooo wwwoojectt/IIDDRREEE... wwDEEREtt html. The dataset is
http:/ //mmlab ab. ie. cuhk, edu. www..k//ppp.eeeeee for public and non-commereial usage.
19

---

Published as a conference paper at ICLR 2022
Data Processing The dataset contains 13, 789 images with multiple semantic bounding boxes attached to each image. Each bounding is annotated with 14 binary attributes, and different bounding boxes in an image may have different attributes. Here, we perform the OR operation among the attributes in the bounding boxes in an image. Hence, each image is linked to 14 binary attributes. We rank the 14 attributes by their entropy and use the top-k of them when performing experiments in Figure 4 (b) in the main text. We consider a classification task consisting of 30 scene categories.
Training and Test Split: The dataset comes with its training, validation, and test split. Due small number of data, we combine the original training and validation set as our training set and use the original test set our validation set. The resulting training set contains 6, 871 images and the validation set contains 6, 918 images.
Computational Resource To speed up computation, on Wider Attribute dataset we use batch size of 40, resulting in 16-hour computation in a single NVIDIA Tesla P100 GPU for 1, 000 epochs training.
Network Design and Optimization We use ResNet-50 architecture encoder for Wider Attributed dataset. We choose 2048-2048- 128 MLP as the projection head (i..., g(-) in f(..-)
equation (1) in the main text) for CI-InfONCE. The MLP projection head is discarded during the linear evaluation protocol. Particularly, during the linear evaluation protocol, the encoder is frozen and linear layer top of the encoder is fine-tuned with downstream labels. For Kmeans + CI-InfoNCE and Auxiliary information + CI-InfoNCE, we consider the same architectures for the encoder, the MLP head and the linear evaluation classifier. For K-means CI-InfoNCE, consider 1, 000 K-means clusters. For fair comparsion, the same network architecture and cluster number is used for experiments with PCL.
For Optimization, we use SGD with momentum of 0.95. Additionally, 0.0001 weight decay is adopted in the network prevent over-fitting We use a learning rate 0.1 and temperature of 0.1 in CI-InfoNCE for all experiments. A linear warm-up following a cosine decay is used for the learning rate scheduling. providing a more stable learning process.
E.3 CUB-200-2011
The following section describes the experiments we performed on CUB-200-2011 dataset in Section 4 in the main text.
Accessiblity CUB-200-2011 is created by Wah al. (2011) and is fine-grained dataset for bird species. It can be downloaded from the link: http:/ http://wwwwwwv v in cal calle beec..ch.................... edu/ Aisipedia/COn -CC--00------- 2011. html. The usage restricted to non-commercial research and educational purposes.
The original dataset contains 200 birds categories over 11, 788 images with 312 binary attributes attached to each image. We utilize those attributes and rank them based on their entropy, excluding the last 112 of them (resulting in 200 attributes), because including these 112 attributes will not change the number of the clusters than not including them. In Figure 4 (c), use the top-k k of those attributes to constrcut clusters with which we perform in CI-InfoNCE. The image is rescaled to 224 x 224.
Data Processing
Train Test Split: We follow the original train-validation split, resulting in 5, 994 train images and 5, 794 validation images.
Computational Resource It takes about 8 hours to train for 1000 epochs with 128 batch size on 4 NVIDIA Tesla P100 GPUs.
Network Design and Optimization
We choose ResNet-50 for CUB-200-2011 as the encoder.
After extracting features from the encoder, 2048-2048- 128 MLP projection head (i.e., g(-) in f(...)
equation (1) in the main text) is used for CI-InfoNCE. During the linear evaluation protocal, the MLP
20

---

Published as a conference paper at ICLR 2022
projection head is removed and the features extracted from the pre-trained encoder is fed into linear classifier layer. The linear classifier layer is fine-tuned with the downstream labels. The network architectures remain the same for both K-means clusters Cl-InfoNCE and auxiliary-informatinn determined clusters + CI-InfoNCR settings. In the K-means clusters + CI-InfoNCE: settings, we consider 1, 000 K-means clusters. For fair comparsion, the same network architecture and cluster number is used for experiments with PCL.
SGD with momentum of 0.95 is used during the optimization. We select linear warm-up following a cosine decay learning rate scheduler. The peak learning rate is chosen to be 0.1 and the lemperature set to be 0.1 for both K-means Cl-InfoNCE and Auxiliary information + CI-InfoNCE settings.
E.4 IMAGENET-I00 100
The following section describes the experiments we performed on ImageNet-100 100 dataset Section in the main text.
Accessibility This dataset is subset of ImageNet-1K dataset, which comes from the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 (Russakovsky et al., 2015). ILSVRC is for non-commercial research and educational purposes and we refer to the ImageNet official site www. image net . org / download. php.
for more information: https:
Data Processing
Section 4 the main text and Section B, we select 100 classes from ImageNet-IK to conduct experiments (the selected categories can be Jaak/CC-ac/CCiIiaaaaaaa
data casta__prceesingg 100 classes. .xt))..tt.................. We also conduct slight pre-processing (via pruning a small number of edges in the WordNet graph) on the WordNet hierarchy structure ensure it admits tree structure. Specifically, each of the selected cate- gories and their ancestors only have one path to the root. We refer the pruning proce- dure in https://githii: cwwccraaye/--/eee Jaw.//ll-ckaaaaaaa wwweoo/c//attapoceessssieeeeeeee processing/ 1magenet100/hieraccct pro.esssiiggggg/oooo imagenet hwwrarchyy.... py (line 222 to 251).
found in pttps://githube com/ mmagenet100/selectieo
We cluster data according their common ancestor in the pruned tree structure and determine the level / of each cluster by the step needed to traverse from root to that node in the pruned tree. Therefore, the larger the I, the closer the common ancestor is the real class labels, hence more accurate clusters will be formed. Particularly, the real class labels is at level 14.
Training and Test Split: Please refer the following file for the training and validation split.
training:
validation: processin /rnc:ess/nn///gggnmnoo/////ee/ee
The training split contains 128, 783 images and the test split contains 5, 000 images. The images are rescaled to size 224 x 224.
Computational Resource It takes 48-hour training for 200 epochs with batch size 128 using 4 NVIDIA Tesla P100 machines. All the experiments on ImageNet- 100 trained with the same batch size and number of epochs.
Network Design and Optimization Hyper-parameters We use conventional ResNet-50 as the backbone for the encoder. 2048-20445 2048- 128 MLP layer and 12 normalization layer used after the encoder during training and discarded in the linear evaluation protocal. We maintain the same architecture for Kmeans + CI-InfoNCE and auxiliary information aided CI-InfoNCE. For Kmeans + CI-InfoNCE, we choose 2500 the cluster number. For fair comparsion, the same network architecture and cluster number is used for experiments with PCL. The Optimizer is SGD with 0.95 momentum. For K-means + CI-InfoNCE used in Figure 5 in the main text, we the learning rate of 0.03 and the temperature of 0.2. We use the learning rate of 0.1 and temperature 0.1 for auxiliary information CI-InfOCCE in Figure 6. A linear warm-up and cosine decay is used for the learning
21

---

Published as a conference paper at ICLR 2022
rate scheduling. To stablize the training and reduce overfitting, adopt 0.0001 weight decay for the encoder network.
APPENDIX F COMPARISONS WITH SWAPPING CLUSTERING ASSIGNMENTS BETWEEN VIEWS
In this section, we provide additional comparisons between Kmeans + CI-InfoNCE and Swapping Clustering Assignments between Views (SwAV) (Caron et al., 2020). The experiment is performed on ImageNet- 100 dataset. SwAV is recent art for blustering-baser sulf-supervised approach. In particular, Sw adopts Sinkhorn algorithm (Cuturi, 2013) to determine the data clustering assignments for batch of data samples, and SwAV also ensures augmented views of samples will have the same clustering assignments. We present the results in Table 5, where we see SWAV has similar performance with the Prototypical Contrastive Learning method (Li al., 2020) and has worse performance than our method (i.e., K-means +CI-InfoNCH).
Method
Top-1 Accuracy (%)
Non-clustering--aaeed Self-supervised Approaches
SimCLR (Chen et al., 2020)
58.2+1.7
MoCo (Ile al., 2020)
19.411.6
Chastering-based Self supervised Approuches (# clusters 2.5K)
SwAV (Caron et al., 2020)
68.5+1.0
PCL (Li ct al., 2020)
68.9=0.7
K-means CI-InfoNCE (ours)
17.
Table 5: Additional Comparsion with SwAV (Caron et 2020) showing its similar performance as PCL on ImageNet- 100 dataset.
APPENDIX G PRELIMINARY RESULTS ON IMAGENET-IK WITH CL-INFONCE
We have performed experiments on ImageNet- dataset, which a subset of the ImageNet- IK dataset (Russakovsky et al., 2015). We the batch size of 024 for all the methods and consider 100 training epochs. We present the comparisons among Supervised Contrastive Learning (Khosla al., 2020), our method (i.e.,
clusters CI-InfoNCE), and SimCLR (Chen et al., 2020). We select the level-12 nodes in the WordNet tree hierarchy structures as our Meearchy-ddtermnneee clusters for CI-InfoNCE. We report the results in Table 6. We find that our method (i.e., hierarchy-determined determined clusters + Cl-InfoNCE) performs in between the supervised representations and conventional sule-supe representations.
Method
Top-1 I Accuracy (%)
Supervised Representaatin Learning (7 downstream labels
SupCon (Khosla et al., 2020)
1.71
Weakly Supervised Representation Learning (Z -evell 12 WordNer hierarchy labels)
Hlicrarchy-Clusters + CI-InfoNCE (ours)
67.9+1.5
Self-supervised Representation Learning (Z instance
SimCLR (Chen et 2020)
62.9+112
Table 6: Preliminary results for NordNeerhiennccyy-deermmee clusters + CI-InfoNCE on ImageNet- IK.
22

---

Published as a conference paper at ICLR 2022
APPENDIX H SYNTHETICALLY CONSTRUCTED CLUSTERS IN SECTION 4.2 IN THE MAIN TEXT
In Section 4.2 in the main text, the UT-Zappos50K dataset, synthesize clusters Z for various (( T) and (( with being the downstream labels. There are 86 configurations of Z total. Note that the configuration process has no access data's auxiliary information and among the 86 configurations we consider the special cases for the supervised (2 = T`) and the unsupervised setting (Z instance ID). In specific, when Z = T. [(2;T) T reaches its maximum at (T)) and ( (T) reaches its minimum O; when Z instance ID, both T(; T) (to be H()) and H TT) (to be (instance ID)) reaches their maximum. The code for generating these 86 configurations can be found in lines 177-299 in https: / /ttpth/bttbt/tt www/Craay-aaaaaaaaaaaa /Cl InfoNCE/ processing/UTS UTT
23


Reviews:
[{"review_id": "Q7IbjY_RZx", "reviewer": "Reviewer_a8kw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS: \n\n++ Interesting idea on leveraging the structure information hidden from the data distribution for learning better representation. \n\n++ Thorough theoretical analysis on the value of clustering information in the proposed objective function.\n\n++ Well written paper. \n\nWEAKNESSES:\n\n-- Lack of novelty. The proposed Cl-InfoNCE is simply a cluster aggregated NCE loss. The way to construct the cluster is either through top-k attributes or k-means. These methods are straightforward and common tricks in representation learning literature. \n\n-- Weakly-supervised methods refer to those using noise labels for learning certain tasks. In the context of this paper, it is essentially the using the auxiliary information as structure information to guide the learning process, and the labels utilized are the accurate ground truth. In my opinion, it is misleading to use the term “weakly supervised”, and the method itself is more of  like “structure aware representation learning”.\n\n-- Some key details are not clarified. The attribute-determined clusters contain samples with the same values on a set of attributes, which inevitably produce some sparse clusters with a very small number of samples. How does that affect the model performance ? Is overfitting an issue in such a situation? How to handle the sample imbalance across different clusters ? Though not mentioned, such practical issues would greatly affect the model performance. \n\n-- Experiments are not sufficient. i). There are other similar existing works with higher performance. For example, Zheng et al. Weakly Supervised Contrastive Learning, ICCV 2021. (79.7% vs 77.9 (this paper) on ImageNet-100). ii) How does the proposed method work compared with other structure-aware representation learning methods such as “Yaling Tao, et al. Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation, ICLR 2021”. \n", "summary_of_the_paper": "This paper proposes a two-stage weakly supervised contrastive learning approach, where it first clusters the data according to the available auxiliary information (or data -driven clusters if no auxiliary information available), and then generates similar representations for the intra-cluster samples and dissimilar representations for inter-cluster samples via a clustering InfoNCE objective. Theoretically, the objective is proved to include the clustering information in the learned representation, and can serve as an interpolation between supervised and self-supervised representation learning. Experiments on four datasets with and without auxiliary information validate the effectiveness of the proposed method. \n", "summary_of_the_review": "This paper proposes a structure-aware constructive loss with theoretical justification for representation learning. The theoretical analysis is very interesting and insightful. However, there are some issues in terms of novelty, clarity in the technical details, rationality on the terminology as well as the soundness in the experiments.     \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NoZKzslmqx", "reviewer": "Reviewer_RhYi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very interesting.\n- The authors conduct a large number of experiments, and nearly all of them seem to support the authors' main claims.\n- The results in Figure 4 are particularly impressive.\n\nWeaknesses:\n- Fairly minor:\n  - Some of the presentation could be a bit clearer. For example, Eqn (1) could make clearer how the negatives are drawn.\n  - In 4.3 the authors mention that their approach is more stable/robust to hyperparameters than the cross-ent baseline, and it would be nice to have a bit more detail about what exactly the authors observed.\n\nAdditional comment/question:\nI'm wondering if the authors have an intuition about why exactly the K-Means+Cl-InfoNCE approach is helpful. Should we think of the initial pass through f and the subsequent clustering as clustering the data based on random features? If so, is it actually necessary to alternate K-means and Cl-InfoNCE, or can we just use the initial random clustering?", "summary_of_the_paper": "The paper proposes an objective for contrastive training of representations based on noisy cluster information. In particular, the objective encourages samples from the same noisy cluster to have similar representations and representations from distinct clusters to have dissimilar representations. The authors show that representations learned with this approach often outperform representations learned with only self-supervision (and no noisy clusters), and also that the proposed approach to using noisy cluster information outperforms other baselines using noisy cluster information. The authors further show that they are able to alternate representation learning and clustering in order to learn better representations that self-supervised approaches even without initial noisy clusters.\n", "summary_of_the_review": "This is an interesting paper with results that do an impressive job of supporting the paper's claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mwv74YtUK-f", "reviewer": "Reviewer_1ziy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The experiments are well-organized and are clearly presented. The sub-sections make it clear which experiments refer to which claims by the authors.\n- The experiments are on a wide-set of image datasets, and on both weakly-supervised and unsupervised versions of the proposed method, which helps show the generalization ability of the method. (I'd like to see more investigation of the labels though, see below.)\n\nWeaknesses:\n- My main concern is that the paper does not sufficiently address other relevant works using weakly supervised contrastive loss, and so the contribution of this work is not clear. More specifically, there are other works that also use auxiliary information/weak-supervision with contrastive loss [A] (the proposed Eq (1) in this paper seems similar to the weakly supervised contrastive loss in Eq (4) of [A], both using additional weak supervision on contrastive loss (in contrast to fully supervised contrastive loss as in (Khosla et al., 2020))). Another set of works are other clustering-based contrastive loss methods [B] (On ImageNet100, looks like the contrastive loss proposed in [B] performs better than this method. Table 2 in [B] = 79.77, Table 5 here = 77.9). Comparing against these as existing baselines would be best, or at least there should be a discussion to contrast with such existing works.\n- The performance of this method seems limited relative to existing works - Table 2 shows that CMC is similar in performance to the proposed method. The authors explain that \"Cl-InfoNCE performs better with less informative auxiliary information\": it would make the paper stronger if the authors specify more concretely when their method should be used over CMC. Also [B] seems to perform better than this method on ImageNet100.\n- What if the auxiliary information is incorrect or noisy? Would this method perform worse than SimCLR, or will the method learn to ignore the incorrect label information?\n- Minor: The paper would be stronger if it discussed other works using auxiliary information or weak supervision on data, since using auxiliary information is a major part of this paper. Adding a paragraph discussing this would be great (ex: works like [C]) so that the contribution of this paper is more clear.\n- Minor: Minor issues with formatting - for example, the plot in Figure 5 looks squished. Also, the writing could be more clear in places (ex: \"For instance, if having continuous attributes as auxiliary information, binning or quantization cannot be avoided when constructing the clusters.\" my understanding is that the authors are saying, the proposed method cannot work with continuous attribute labels?). The Appendix also appears to be submitted with the main PDF (instead of being in supp materials).\n\n[A] Sun, Jennifer J., et al. \"Task programming: Learning data efficient behavior representations.\" CVPR 2021.\n\n[B] Zheng, Mingkai, et al. \"Weakly Supervised Contrastive Learning.\" ICCV 2021.\n\n[C] Tan, Reuben, et al. \"Learning similarity conditions without explicit supervision.\" CVPR 2019.", "summary_of_the_paper": "The paper proposes a weakly-supervised contrastive loss based on using existing auxiliary information from datasets (such as shoe attributes for classifying shoe categories (UT-zappos50K)). The auxiliary information is used in a contrastive loss formulation similar to the unsupervised contrastive loss of (Chen et al., 20) and the supervised contrastive loss formulation of (Khosla et al., 2020), but instead using clustering information from auxiliary information (Eq (1) of the paper). Experiments are across 4 image datasets (UT-zappos50K, Wider Attribute, CUB-200-2011, ImageNet-100) and shows that representations learned from the proposed weakly-supervised loss performs between unsupervised & fully supervised contrastive loss. There are additional experiments comparing to a few other clustering-based contrastive loss approaches.", "summary_of_the_review": "Overall, the experimental section of the paper was clear and included a good variety of image classification datasets. However, given the lack of discussion/comparison to relevant existing works and limited performance of the proposed method, the contribution of the proposed method is not clear to me at this point.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2hy5EghZcWP", "reviewer": "Reviewer_h3zd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths：\n- The idea is clear and simple, and the authors give theoretical definitions to support their assumptions and experiments.\n- The authors prove that Cl-InfoNCE maximization learns to include the clustering information. And thus can use I(Z; T) and H(Z|T) to characterize the goodness of the learned representations. The authors also present experiments to validate it.\n- The authors conduct experiments with different supervision levels and show that CL-InfoNCE can better bridge the gap with the supervised learned representations by using auxiliary information.\n- The proposed approach can also be applied without auxiliary information with K-means and EM optimization.\n\nWeaknesses:\n- In Figure 4, the authors observe that the best performing clusters happen at the intersection between I(Z;T) and negative H(Z|T), could you present proves on this?\n- Comparison with related work is required, such as [1]\n\n[1] Weakly Supervised Contrastive Learning. ICCV 2021.", "summary_of_the_paper": "This paper proposes a weakly-supervised contrastive representation by using the auxiliary clustering information. Data are clustered with auxiliary tags and the clustering InfoNCE loss is introduced to optimize the system. The authors prove that maximizing the Cl-InfoNCE learns to include the clustering information.", "summary_of_the_review": "Overall, this paper proposes an easy but effective method for weakly-supervised contrastive learning with auxiliary information. The authors provide clear mathematical proof and good results with adequate experiments. However, comparisons with related work should be included to make to final results more convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a weakly supervised contrastive learning, using auxiliary cluster information, for representation learning. Their method generates similar representations for the intra-cluster samples and dissimilar representations for inter-cluster samples via a clustering InfoNCE objective. Their approach is evaluated thoroughly on three image classification task.

The reviewers agree that the paper is well written, presenting interesting theoretical analysis (Reviewer h3zd,  a8kw) and solid experimetal results (Reviewer RhYi, 1ziy). The core idea of the paper is relatively simple and well motivated (Reviewer h3zd). While the focus is using the clustering with auxiliary labels, the method can be applied without auxiliary labels with K-means. 
There were some concerns from the reviewers:  the overlap with a concurrent work [1]. The authors have provided detailed discussions on conceptual (concurrent work focuses on unsupervised cases where this work focuses on weakly-supervised setting) and emprical comparisons. Accordingly, reviewer a8kw and 1ziy had some issues with the novelty of the paper, as it can be interpreted as slight modification from previously explored idea (vanilla InfoNCE loss). 

Despite some overlap with existing approaches, the paper presents an interesting and well conducted study of integrating clustering information for learning representation, so I vote for acceptance. 

[1] Weakly Supervised Contrastive Learning. ICCV 2021.
Decision: Accept (Poster)