Intriguing Properties of Contrastive Losses
Authors: Ting Chen
Calvin Luo
Lala Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
contrastive loss
feature suppression
self-supervised learning
computer vision

Abstract: We study three intriguing properties of contrastive learning. First, we generalize the standard contrastive loss to a broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layer non-linear projection head. Second, we study if instance-based contrastive learning (with a global image representation) can learn well on images with multiple objects present. We find that meaningful hierarchical local features can be learned despite the fact that these objectives operate on global instance-level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such as "color distribution" vs "object class". We construct datasets with explicit and controllable competing features, and show that, for contrastive learning, a few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques.

Intriguing Properties of Contrastive Losses
Ting Chen Google Research namtinnghee&@goglee.. com
Calvin Luo Google Research cal calvinluo@googeecom com
Lala Li Google Research
lala@google. com
Abstract
We study three intriguing properties of contrastive learning. First, generalize the standard contrastive loss to broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layed non-linear projection head. Second, we study instance- based contrastive learning (with global image representation) can learn well images with multiple objects present. We find that meaningful hierarchical local features be learned despite the fact that these objectives operate global instance- level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such "color distribution" vs "object class". We construct datasets with explicit and controllable competing features and show that, for contrastive learning, few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. scenarios where there are multiple objects an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques!
1 Introduction
Contrastive learning [1, 2, 3, 6, 8, 9, 10, 11, 12, 13, 14] has recently achieved great successes in learning visual representations without supervision. As shown in [13, 14], contrastive learning can learn representations that rival supervised learning, and significantly improve the state-of-the-art in semi-supervised learning ImageNet. One successful case of contrastive loss for self-supervised learning is to make augmented views of the same example agree [1, 2, 13]. A widely used contrastive loss encourage agreement is based on cross entropy [15, 3, 4, 13]. Given an augmented view of an example, the contrastive prediction task aims to classify a set of candidates into the positive example (i.e. the other augmented view of the same example) and negative ones via the cross entropy loss.
In this work, to understand the effectiveness and limitation of existing contrastive learning methods, study three intriguing aspects. First, propose generalization of the standard contrastive loss, and systematically study their performance differences. Second, we study if the instance-base contrastive learning, for which the contrastive loss operates on global representation of an input image, can learn well on images with multiple objects present, and whether or not it leads to meaningful local features. Finally, systematically study the feature suppression phenomenon in contrastive learning. The suppression effect occurs among competing features shared across augmented views. For example, with random cropping as the augmentation, "color distribution" and "object class" often competing features as they are likely shared between two augmented views. The suppression effect among competing features can significantly degenerate the representation quality, or even
"oode and visualization at
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

completely disable the learning of certain features, as shown in our experiments. Existing methods critically rely on hand-crafted data augmentation favor certain sets of competing features than others.
Our main findings and contributions are summarized below.
We propose a generalized contrastive loss, and show that differences between contrastive losses are small with a deep projection head.
We show that the instance- based objective widely used in existing contrastive learning methods can learn on images with multiple objects, and also learn meaningful local features despite operating on global image representation.
We construct three datasets with explicit and controllable competing features to systematically study the feature suppression effect in contrastive learning.
We show that few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in image, the dominant object would suppress the learning of smaller objects. This poses open challenges to existing contrastive learning.
Generalized contrastive loss and differences among its instantiations
The common contrastive loss used in most recent work is based on cross entropy [15, 3, 4J. Following the notation in [13], the contrastive loss can be defined between two augmented views (f. of the same example for mini-batch of size of n, and can be written as the following.
NT-Xoot
1
Σ K.EMM Lk11 [oyl exp(sim [z; zx)/T)
log
exp(sim( 2,; 2/7/r)
(1)
7
where 2;, 2, are hidden representations of two augmented views of the same example; sim (u, v) u² /"/o/[]/u]]] [[l]) is the cosine similarity between two vectors; is a temperature scalar and MB is a randomly sampled mini-batch consisting of augmented pairs of images. In [13], MLP projection head is introduced between intermediate layer h (e.g. output of ResNet encoder) and final output It is shown that the projection head is very beneficial and a much better feature representation than 2.
In this work, generalize the standard contrastive loss the following form.
C gegneralized contrastive
Calignment
Xedistiiution
Both terms are defined on hidden representations. Caligmment encourages representations of augmented views to be consistent, while Ldistriutiion encourages representations (or random subset of them) to match a prior distribution (of high entropy). is not difficult to see that the standard contrastive loss in Eq. 1 is a special case as it can be re-written as follows (scaled by constant r).
TC NT-Xxat
Σ sim sim 2;)
Σ log
(kee)) exp(sim(zz, #x)/T)
Laligmment
Edistibution
This form of factorization in Eq. 3 has been proposed in [16], where the second LogSumExp term is referred to uniformity since encourages representation to uniformly distributed in the hypersphere. Different from [16], here we generalize the hypersphere uniform distribution and study a wider set of prior distributions for their effectiveness in learning representations.
SWD for supporting diverse prior distributions. One issue of using more diverse set of priors we cannot rely LogSunExp for matching the distribution. To this end, we resort to the theory of optimal transport, via Sliced Wasserstein Distance (SWD) [17, 18, 19]. For two sets of equal- sized samples from two 1-D distributions, the optimal transport can be obtained by computing two

---

Algorithm Sliced Wasserstein Distance (SWD) loss.
input:
activation vectors H E TRbxd prior distribution (e.g. Gaussian) sampler S draw prior vectors E Rxdd using S
generate random orthogonal matrix W E "
make projections: H HW: P PW
initialize SWD loss & = 0
f E f1,2, .d') do
f= ||sort H.))) sort P... P2 end for return t/(dd")
Table Instantiations of the generalized contrastive loss, i.e. Lalignment disttiibutioons that we use in this work. ž denotes Ez-normalized = E Rd,, and only used for uniform hypersphere prior.
Calign
Prior distribution
Ldistribation
|| 81|1
Uniform hypersphere
Σ. log E, exp(E? 2//T)
nd
nd Eij
|
Uniform hypersphere
SWD(Z, WVD(Ziuurion Lurioo)
Ei,
|| l3|
Uniform hypercube
SWD(Z, (WD(2,Zprior)
nd Eis
||
Normal distribution
SWD(Z, ZAnoo
permutations that order the values of both sets of samples respectively. The 1-D Wasserstein distance can then be computed with & distance between the ordered values. For n-D distributions, we first project the samples to randomly-generater orthogonal I-D subspaces, and then compute the sum of 1-D Wasserstein distance across all 1-D subspaces. By adjusting the network weights to minimize the SWD, we are able to reduce the mismatch between the distribution of hidden vectors and a known prior distribution. The detailed algorithm can be found Algorithm 1. With SWD loss, we are able to use wider set of priors, and Table I summarizes instantiations of the generalized contrastive loss with different prior distributions and distribution matching loss.
Connection with mutual information. The connection between the standard contrastive loss and mutual information has been shown before [3, 20], where the contrastive loss (a.k.a. InfoNCE loss [3]) is shown to be lower bound of the mutual information. To connect the generalized contrastive loss mutual information, we start by the definition of mutual information between two latent variables U. V. which is T(U; V) = HUU) (V)).. Comparing this factorization of mutual information with generalized contrastive loss, not difficult to see that: 1) the alignment term Laligmment is directly related to (U) which aims to reduce uncertainty of the other views given one view of the example; and 2) the distribution matching term Laissritution can considered as proxy to (u) for maximizing the entropy in the representation. It is perhaps worth noting that different from mutual information, the generalized contrastive loss (Eq. 2) allows a tunable weight (A) between the alignment and distribution matching term. The weighting scalar À is (inversely) related to the temperature (details in Appendix A.2).
Comparing different instantiations of generalized contrastive loss. Here we ask: Is it essential to use uniform hypersphere prior for the effectiveness of contrastive loss? How much difference does it make when distinct generalized contrastive losses are used? To answer this question, we conduct experiments following SimCLR settings [13, 14], and use the linear evaluation protocol. Detailed experimental setup can be found in Appendix A.
Figure 1 shows linear evaluation results of models trained with different losses under different training epochs. On CIFAR-10, 10, we see little difference in terms of linear evaluation for variants of the generalized contrastive losses, especially when trained longer than 200 epochs. As for ImageNet, there are some discrepancies between different losses, but they disappear when a deeper 3-layer non-line non-linear projection head is used.
3

---

67.5
67.5
65.0
Top-1 (8)
60.0
60.0
NI-Xent
57.5
Xont
57.5
NT Xont
Decoupted NT-Xoom
Dccoupled Xont
Dccoupled NT Xont
SIND (unilomm (operashere)
SWD can form typerapherei
SWDD can mm toper anpershhrre
MAID (normal)
SAD mio)ma))
SVDD (oormal)
(unffrm
SAD (an form cubbr)
SADD Can Top cubbub
50.0
50.0
100
200
400
8:00
Epochs
Epochs
Epochs
(a) CIFAR-IO (2 layers)
(b) ImageNet (2 layers)
(c) ImageNet (3 layers)
Figure 1: Linear evaluation accuracy of ResNet-50 trained with different losses CIFAR-10 and ImageNet datasets. Numbers of projection head layers are in parentheses. Differences between variants of generalized contrastive loss small with deep projection head. Decoupled NT-Xent loss is introduced in A.2. Numerical results can be found in Appendix A.3.
Furthermore, we find that deep pro- jection head not only reduces the
Table Linear eval accuracy ResNet-50 on ImageNet.
differences among different gener- alized contrastive losses, but has a similar effect for batch size. With proper learning rate scaling across batch sizes (e.g. square root scal- ing with LARS optimizer [21]), the impact of batch size on representa- tion quality is small. Table 2 demon- strate this phenomenon for the stan- dard contrastive loss, and more re- sults on other losses can be found Appendix A.3.
Epoch
Projection head
Batch size
100
200
400
800
65.4
67.3
68.7
69.3
2 layers
1024
65.6
67.6
68.8
69.8
2048
65.3
67.6
69.0
70.1
512
66.6
68.4
70.0
71.0
3 layers
1024
66.8
68.9
70.1
70.9
2048
66.8
69.1
70.4
71.3
512
66.8
68.8
70.0
70.7
layers
1024
67.0
69.0
70.4
70.9
2048
67.0
69.3
70.4
71.3
Instance- based objective can learn on images with multiple objects and learn good local features
Most existing contrastive learning methods [13, 10, 22, define their objectives the instance level where each image is encoded into single vector representation (e.g. representations of two random crops the same image instance are treated a positive pair). In other words, the objective operates on global representation of its input rather than on some local regions (of its input). We pose two questions regarding mnstance-based global objective: 1) when there is only a single (dominant) object in the image, the objective seems reasonable as it encourages the model to learn features relevant to object class, but when there are multiple objects present in the image, can instance-based objective still learn well? 2) Since the instance-based objective uses a global summary of its input, can it still learn good local features (e.g. parts an object, or multiple objects the same scheme)? To answer these questions,
SimCLR representative for the mastdnce-bseed objective.
3.1 SimCLR can learn on images with multiple objects
Commonly used self-supervised learning datasets, such as MNIST, CIFAR-10 -00 ImageNet, are object centered, i.e. the image is mainly occupied by single (dominant) object. To experiment with multiple objects a controllable setting, we propose a new dataset setting by composing multiple digits as follows.
MultiDigits dataset. We place MNIST digits (28 28 size) on shared canvas (112 x 112 size). We vary the number of digits placed on the canvas. One factor that could interfere with learning of multiple digits overlapping digits, therefore we use two placement strategies: random vs in-grid (Figure 2). Random placement of digits incurs no constraint on where digits can be placed on the

---

3
A
0 2
7 6
4
6
74
(a) digits, random placement.
(b) 16 digits, random placement.
6
9
9
6
3
3
6
3
8
0
/ 7
9
(c) 4 digits, in-grid placement.
(d) 16 digits, in-grid placement.
Figure MultiDigit dataset. More digits lead to more overlapping in random placement.
canvas, whereas in in-grid placement puts each digit in one of the x grid cells the canvas is divided into, and no two digits can fall in the same cell. In-grid placement ensures no overlapping of digits.
We first pretrain ResNet-18 18 with SimCLR supervised learning with the same augmentation policy (random cropping and resize) on MultiDigits dataset. To access the representation quality, we then train linear classifiers for images with single digit of size 28 x 28 the canvas. Similarly during evaluation, we place only one digit of size 28 x 28 on the canvas.
As shown in Table 3, representations learned using supervised loss maintains its quality when up to 8 digits are placed the image. After that representation becomes worse as canvas gets more crowded. Notably, representations learned using SimCLR display similar phenomenon. Regardless of placement strategy, top-1 accuracy stays at the same level up to digits, demonstrating that SimCLR can Iearn from images with multiple objects. In addition, the increased performance gap between the two placement strategies with increased number of digits shows that object overlapping makes it harder for contrastive losses to learn from multiple objects.
Table Top-1 linear evaluation accuracy (%) for pretrained ResNet- 18 on the MultiDigits dataset. We vary the number of digits placed on the canvas during training from I to 16. During evaluation only digit is present. As baseline, network with random weights gives 18% top-1 accuracy.
Placing of digits
Number of digits (size 28 28)
Random In-grid
99.5
99.5
99.3
99.4
98.9
98.3
Supervised
99.5
99.6
99.5
99.3
98.6
Random In-grid
98.9
98.9
99.0
SimCLR
98.3
98.6
99.1
99.2
99.1
98.3
3.2 SimCLR learns local features that exhibit hierarchical properties
To understand the local features learned by SimCLR, we apply K-means intermediate features of the pretrained ResNet with SimCLR, and see how local regions of an image are grouped together. For good representations, expect that regions of similar objects object parts should be grouped together.
Specifically, we take pretrained Resnet-50 2x ImageNet, and run inference on images (from ImageNet validation set and COCO [23]) of size 448x 448. We run K-means with various numbers of clusters the 12-normalized hidden features from middle layers of the network (e.g. block group 2,3,4 of the ResNet). We also compare SimCLR learned features with supervised learned features, as well as the raw pixel (RGB) features extracted from each 14 x 14 patch.
Figure 3a shows that the number of clusters increases, the learned representations tend group image regions based on parts of the object (i.e. facial components of the dog). This phenomenon appears in both SimCLR and supervised learned features, but not with raw pixel features, indicating meaningful local features learned by SimCLR and supervised learning. In Figure 3b, we compare ResNet intermediate features at different layers, and suggests that earlier layers contain more edge-related features, while later layers contain more object/part features.
5

---

Sinc
Slock Croup
Itlc Gruto
Thas Tww
(a) Features from different methods.
(b) SimCLR features at different ResNet layers.
Figure 3: Visualizing features on a ImageNet validation image with K-means clustering. Each row denotes type of local features used, and each column denotes the number of K-means clusters. Later layers of SimCLR JimCCR/supervised ResNet tend to group by object parts. More visualization examples can be found in https:/coontratiie-llamii wwwnniinnnggttu.iiii. atthub.io/intriguing
Elastecs
Clucters
Clusters
SinC
Suce Hiee
Dion Neenk (com)
Hhan Thenk (ielll
(a)
Figure 4: Visualizing features on two images from COCO. Each row denotes a type of local features (SimCLR, Supervised, and raw pixels; both SimCLR and Supervised are trained on ImageNet), and each column denotes the number of K-means clusters. Region grouping by SimCLR/supervised features tend to overlap with object class.
Region grouping results on two COCO images for SimCLR and supervised learning (trained on ImageNet) are shown in Figure 4. Again, region grouping by local features tend overlap with object class, indicating good local features learned.
4 Feature suppression limits the potential of contrastive learning
Contrastive learning requires good design of data augmentation to work well. As shown in [13], without color augmentations that randomly shift color distribution (while maintaining information regarding object class), the quality of learned representations are significantly worse. In other words, the presence of "color distribution" features suppresses their competing feature of "object class", and is addressed by color augmentation. However, there may be scenarios where the known augmentations cannot fully address this feature suppression effect, and it can thus limit the potential of contrastive learning. Here we quantitatively study the feature suppression phenomenon by constructing datasets with explicit and controllable competing features, and see how well contrastive learning method could learn

---

(a) ImageNet images overlaid with MNIST digits. The left most column is original image, and others augmented views via random crop and color distortion. MNIST digits and ImageNet classes competing features. vary the number of unique MNIST digits to control the competing features.
7
6
4
7
3
7
(b) Two MNIST digits randomly placed shared canvas (of size 112 x 112). The two digits can have the same size (upper row) different sizes (lower row), and digits of different sizes be considered competing features. We size one digit and vary other.
Random
integer
Random
Concatemation
(c) Images (of RGB channels) are concatenated with additional channels of random integer sampled from range of [1, (n)). The integer, shared between two views, replicated for spatial dimension and represented binary channels. RGB channels and random bits competing features.
Figure 5: Probing datasets with explicit and controllable competing features.
4.1 Datasets with explicit and controllable competing features
To construct datasets with controllable competing features, leverage strategies: channel addition that adds different feature information in shared canvas, and channel concatenation that expand the RGB channels to include additional features. With these strategies, construct three datasets below.
DigitOn dataset. We overlay MNIST digits ImageNet images via channel addi- tion/summation (Figure 5a). For each ImageNet image, we assign unique MNIST digit and replicate it in nine fixed locations before the standard SimCLR augmentations [13] are applied to create augmented views. Therefore the original ImageNet images and added MNIST digits are competing features. Although is difficult quantify information in MNIST digits, the number of unique MNIST digits used. Ideally, we want the model to learn both set of features so that it could perform well for both MNIST digit and ImageNet object recognition.
manually control
MultiDigits dataset (varying the size of one digit). This dataset modified from MultiDigits introduced above. Here we only consider two digits, and vary the size of one of them (Figure 5b). In this work, we place two digits on canvas size 112 x 112. We fix the size of one of the digits to be 20 20 while varying the other from 20 x 20 to 80 x 80. Digits of different sizes can be considered as competing features. Ideally, we want the model to learn features for digits of all sizes appeared during training.
RandBit dataset. We concatenate a real image with an image random integer in the channel dimension (Figure 5c). The random integer is randomly sampled from range of [1, (n)) where replicated across spatial dimension (i.e. all pixel location shares the same valuc), and it is also represented binary bits/channels instead of an integer floating
a parameter to control.

---

number to make it easily learnable. Furthermore, unlike RGB channels, these additional channels of random bits will not be altered by augmentation, so they are identical for both augmented views of the same image. The RGB channels and the added channels of random bits are competing features, and this construction allows us to control the amount of information the added competing feature, which is bits. Also, we know that the mutual information between two views given this construction is at least (n).
4.2 Easy-to-learn features (MNIST digit) suppress the learning of other features (ImageNet object class)
Sup ervised
Temperature 0.05
Temperature 0.1
Temperature 0.2
100
ImageNet
80
60
Top (%)
60
60
60
40
40
40
40
20
20-
20
20
0
2'
2F
2 29 24k
0
0
20
22
2
2 200k
0
29
22
2
29 2 2-0kk
Mumb or unique MNIST digits
Number of unique MNVIST digits
Number of unique MNIST digits
Number of unique MNIST digits
(a) Supervised learning
(b) Unsupervised contrastive learning
Figure 6: (a) Supervised learning accuracy on ImageNet classification. (b) Linear evaluation of learned features for both MNIST classification and ImageNet classification on the DigiiOnlmageNet dataset. Batch size of 1024 and 2-layer projection head is used. Different batch sizes and projection head layers have negligible influence on the trade-off between ImageNet vs MNIST accuracy.
On DigitOnllmageNet datasets, we vary the number of unique MNIST digits used in the training set, and all MNIST digits are used in the validation/test set. As a baseline, we train supervised ResNet-50 on the created datasets with ImageNet labels, and the number of unique MNIST digits has little impact on the top-1 ImageNet classification accuracy (Figure 6a).
We then train SimCLR on the datasets with different temperatures. As shown in Figure 6b, when increase the number of unique MNIST digits, the linear evaluation performance of the learned features for MNIST classes increases accordingly, while the accuracy for ImageNet classes decreases dramatically. The trade-off between digit recognition ability and object recognition ability shows that simple features suppress the learning of difficult features, when both are shared between two augmented views. Different batch sizes and projection head depths have negligible influence to the outcome we observe here. Therefore, it is difficult to learn both of the competing features using existing contrastive losses (e.g. SimCLR).
4.3 The presence of dominant object suppresses the learning of features of smaller objects
On the MultiDigits dataset, as mentioned, we fix one digit to be size of 20 20 while varying the other from 20 20 80 80, on canvas 112 x 112. We first pretrain a ResNet-18 with SimCLR or supervised learning with the same augmentation policy (random cropping and resize) and batch size of 1024. To access the representation quality, we then train linear classifiers for each of the digit sizes that appeared during pretraining. For training of the linear classifier, only place single digit at a time on the canvas of the same size as during pretraining.
The results are summarized in Table 4. For supervised learning, the learned representations for the smaller digit do not change much as the other digit increases its size, and the model perform well for both small and large digits (accuracy > 99%). However, for SimCLR, the learned representations of the smaller digit degenerate significantly when the size of the other digit increases, almost to the level of a random untrained network. The dominant object can be learned very well (accuracy > 99%) while suppressing the learning of the smaller object. Although tuning temperature has some effects on reducing the feature suppression, the trend stays unchanged.
8

---

Table 4: Top-1 linear evaluation accuracy (%) for pretrained ResNet-18 on the MiltiDigits dataset. We fix the size of 1st digit while increasing the size of the 2nd digit. For SimCLR, results are presented for two temperatures. Accuracies suffered from significant drop when increasing 2nd digit size are red colored.
2nd digit size (Ist digit is kept the same size of
20)
20 x 20
30 x 30
40 x 40
50
50
60 x 60
70 x 70
80 x 80
1st digit
99.1
99.2
99.2
99.2
99.1
99.1
99.0
Supervised
2nd digit
99.1
99.5
99.5
99.6
99.5
99.5
99.6
SimCLR
1st digit
97.8
97.6
96.2
96.5
88.5
74.5
39.9
(r = 0.05)
2nd digit
97.8
97.9
97.8
98.3
98.2
97.7
98.2
SimCLR
1st digit
98.7
98.8
98.3
87.5
24.9
19.8
20.3
(r = 0.2)
2nd digit
98.7
99.2
99.2
99.0
99.1
98.9
99.4
Random net
1st digit 16.5
16.7
16.6
16.6
16.6
16.9
16.5
(untrained)
2nd digit
16.5
19.1
21.9
24.1
26.5
28.1
29.0
4.4 Extra channels with a few bits of easy-to-learn mutual information suppress the learning of all features in RGB channels
In the RandBit datasets, we add additional channels (identical across pixels) of random bits to MNIST and ImageNet. As mentioned above, SimCLR augmentation is only applied to RGB channels so extra added channels will be shared among two view.
NT-Xent (z=0.05)
MT-Xent (t= 0.1)
NT-Xent XT=Xent(re0..2
VAE
100
100
batch_siee
80
80
512
Top-1 (96)
60
1024
60
40
40
40
batch_sim
256
20
20
20
512
1024
0
20
10
20
30
Bits
Bits
Bits
Bits
(a) Contrastive loss.
(b) Generative loss.
Figure 7: Linear evaluation of learned features when few bits of competing features added (on MNIST), Adding few bits completely disables contrastive learning (across various batch size or losses). Interestingly, it has little effects on a generative model (VAE). The detrimental effects are just as strong for larger datasets such as CIFAR-10 and ImageNet (Appendix B.1).
Figure shows the linear evaluation accuracy of models trained on MNIST (with additional random bits added). We observe that the linear evaluation accuracy quickly drops with few bits of competing feature added. This detrimental effect on the representation quality persists on bigger datasets like CIFAR-10 and ImageNet as well, and cannot be avoided by using different contrastive losses, batch sizes, or memory mechanism based on momentum contrast (details in Appendix B.1). We believe the fact that just few bits of learning is related to the saturation of the distribution matching loss. As shown in Appendix B. 2, linear increase in bits requires an exponential increase in batch size, which not sustainable as the required batch size can quickly go beyond the size of the dataset size. In practice, we rely on using data augmentation to remove those uninformative easy-to-learn features so that contrastive learning can learn useful representations. Interestingly, the extra bits do not affect generative model, variational autoencoder [24, 25], nearly as much, despite other settings such as model size are held the same, prompting potential direction of addressing the issue.
features can completely disable the good representation
5 Related Work
Our work studies the contrastive loss based on cross entropy loss [15, 3, 13]. This loss is widely
used in recent successful contrastive learning methods [3, 4, 5, 6, 7, 8, 9, 10. 11, 12, 13, 14]. In

---

terms of the contrastive loss, our work is perhaps most related to [16], which shows that formulating contrastive loss as aligmment and uniformity the hypersphere gives similar performance as the standard contrastive loss. We further generalize this factorization, and show other distribution matching losses can used, and they could achieve similar results. Other than standard contrastive loss that directly utilize negative examples, BYOL, [22] demonstrates another way to maintain representation distribution/enttopyy without directly relying on distribution matching, and SWAV [26] shows clustering-basee method equipped with proper data augmentations could also achieve similar performance. We conduct preliminary experiments YOL on RandBit and found that also suffers from feature suppression generalized contrastive loss. It is expected that SWAV would exhibit similar behaviors on RandBit as those random bits could fuel representations for perfect clustering,
The connection between contrastive loss and mutual information has been studied before [3, 20J. We show that for the generalized contrastive loss, can also be related to mutual information. Despite the connection between contrastive loss and mutual information, it has been pointed out that mutual information estimation may suffer from certain limitations [27, 28]. Moreover, [29, 12] show that higher mutual information learned by the network does not warrant better representation quality, In our work, we find adding mutual information bits between two views which are irrelevant to downstream tasks can be harmful for the quality of learned representations. Data augmentation plays an important role at favoring certain bits mutual information than others.
There is growing number of recent work on the topic of understanding contrastive learning. both theoretically [30, 31, 32, 33, 34] and empirically [16, 12, 35, 36). However, little work has been done to study the phenomenon of feature suppression. To our knowledge, we are the first one to quantitatively and systematically study this problem. We believe this still very open question and could benefit from more future investigation. Finally, the feature suppression effect in unsupervised contrastive learning that we study in this work may also exist standard supervised learning ("contrastive loss" between examples and class labels), as suggested by [37, 38), though the specific form would be different.
Conclusion
In this work, study three intriguing properties of contrastive losses. In particular, results highlight that feature suppression is still an open challenge in contrastive learning. While there is plethora of work on improving contrastive learning, few of them directly aim address feature suppression. This limitation of contrastive learning becomes a bottleneck for scenarios where existing augmentation cannot fully address the feature suppression phenomenon, and learning would saturate at a level of dissatisfaction.
We would also like point out some limitations of our study. Firstly, we focus mostly contrastive learning with explicit negatives (e.g. SimCLR and MoCo). We believe other methods based on clustering and/or without negative pairs would exhibit similar phenomenon but we leave that as future work. Secondly, many of our proposed image datasets are not fully realistic despite being composed from some (challenging) natural image datasets such ImageNet. We admit it is very hard to explore competing features or multiple objects in controllable fashion on realistic large scale image datasets.
Acknowledgements
We specially thank Geoffrey Hinton for many inspiring discussions and helpful advice. We would also like to thank David Fleet, Simon Korbblith, Mohammad Norouzi, Kevin Swersky and Katherine Hermann for insightful discussions. In addition, we are thankful to William Chan and Sara Sabour for ideas on implementation of sorting on TPUs. We also thank the anonymous reviewers for their constructive feedback.
10

---

References
[1] Suzanna Becker and Geoffre E Hinton. Self-organizing neural network that discovers surfaces in random- dot stereograms. Nature, 355(6356):111-163 1992.
[2] Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox Discriminative unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems, pages 766-774, 2014.
[3] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
[4] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[5] R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1xxivv:808.... 06670, 2018.
[6] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15509-15511 2019.
[7] Olivier Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272. 2019.
[8] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 058499 2019.
[9] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext- pretext-invariant representations. arXiv preprint (512:099110099 2019.
[10] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised 2019.
visual representation learning. arXiv preprint
[11] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04900 2020.
[12] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for gond views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
[13] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, arXiv preprint arXiv:2002.05709, 2020.
[14] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton, Big self- supervised models are strong semi- aunissupeeeied learners. arXiv preprint arXiv:2006. 10029, 2020.
[15] Kihyuk Sohn. Improved deep metric learning with multi-class n- n-pair loss objective. Advances in neural information processing systems. pages 1857-1865, 2016.
[16] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprint arXiv:2005. 10242, 2020.
[17] Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its application texture mixing. International Conference on Scale Space and Variational Methods in Computer Vision, pages 435-446. Springer, 2011.
[18] Nicolas Bonneel, Julien Rabin, Gabriel Peyré. and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51()):22 45, 2015.
[19] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced wasserstein distances. In Advances in Neural Information Processing Systems, pages 261-272, 2019.
[20] Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint arXiv: wet.i 06922, 2019.
[21] Vang You, Igor Gitman, and Boris Ginshurg. Large batch training of convolutional networks. arXiv preprint Xrv:: 3:0x::700.000 03888, 2017.
[22] lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya. Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bital Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self- supervised learning. 2020.
[23] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnikk. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
11

---

[24] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint urXiv:1312.6114, 2013.
[25]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic hackpropagation and approxi- mate inference in deep generative models. arXiv preprint arXiv: 4082, 2014.
[26] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. arXiv preprint wVXv:2006.098822 2020.
[27] David McAllester and Karl Stratos. Formal limitations on measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pages 875- 875-884, 2020,
[28] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators, arXiv preprint 190.6221 2019.
[29] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning, arXiv preprint arXiv: 1at.i99977...., 2019.
[30] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 190..0999, 2019.
[31] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov. and Louis-Philippe Morency. Demystifying self-supervised learning: An infommation-thoortticc framework. arXiv preprint urXiv:2006.05570 2020.
[32] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy,
and linear models. arXiv preprint arXiv:2008. 10150, 2020.
[33] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self- self-suppr learning with dual deep networks. arXiv preprint arXiv:2010.005: 00578, 2020.
[34] Jason Lee. Qi Lei, Nikunj Saunshi. and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008.00 001644 2020.
[35] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? arXiv preprint arXiv: 06606, 2020.
[36] Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning Invariances, augmentations and dataset biases. arXiv preprint arXiv:2007. 13916, 2020.
[37] Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv preprint arXiv:2006. 12433, 2020.
[38] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint arXiv:2006.077100 2020.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
12


Reviews:
[{"review_id": "odBLoaU_zQe", "reviewer": "Reviewer_cf57", "summary": "This work presents three empirical analyses on contrastive learning algorithms. The first analysis shows that when using a multi-layer non-linear project head, the different ways of instantiating the generalized contrastive learning loss lead to final representations with similar downstream visual task performance. The second analysis finds that the instance-based contrastive learning algorithms can still achieve good performance on clustered images, even though these algorithms are based on global image representations. The third analysis is about the preference of contrastive learning algorithms on competing features that are easy-to-learn. In this analysis, the authors show that after manually adding diverse competing information into the images and linking them with image identity, the contrastive learning algorithms fail to capture the general statistics but chooses to only encode the competing information. These three empirical analyses seem to be unrelated and all have certain limitations making the analyses not general to all algorithms. Although the final analysis illustrates that the contrastive learning algorithms indeed have certain limitations, the authors have not proposed a better or modified algorithm to address that issue. In summary, I feel the analyses presented in this paper are useful for the community to be aware of, so I would weakly recommend acceptance. But the current paper can be much stronger if the authors can address the limitations in the analyses (see detailed review below) and even further leverage the analyses to propose even better algorithms resolving the presented issues in the analyses.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below, I will describe the limitations for all three analyses in my view, hoping the authors to address them in the rebuttal phase or to motivate the authors to develop stronger analyses. \n\n1. In the loss formalization analysis, the authors start from the generalized contrastive loss definition, which is the sum of the alignment loss and the uniformity loss. They then test several ways of defining the “alignment” and the “uniformity” including different prior distributions (hypercube or hypersphere) and different ways of encouraging uniformity (SWD or typical LogSum) to show whether the formalization of the generalized contrastive loss definition actually influences the results. After training a ResNet-50 on CIFAR-10 and ImageNet with two or three hidden layers in the projection head using a SimCLR setting, the authors find that when the projection is deeper (3 layers), there is no significant difference between these different formalizations. Although this result seems to indicate that the contrastive loss formalization does not matter with a deep projection head, the fact that only SimCLR setting is tested makes the analysis less useful. The innovation in most of the newer contrastive learning algorithms is not in how the prior distribution is defined nor in how the uniformity is encouraged, like BYOL has no uniformity loss but has a momentum-encoder and a prediction head. These architecture and algorithmic differences make the analysis possibly limited only to SimCLR-like settings. \n2. In the learning from “crowded” images analysis, the authors show that the even learning from a multi-digit dataset, SimCLR can still extract useful features that can be leveraged later to identify single digit. However, I feel that the digit recognition problem is really too easy and poses no challenge for the unsupervised learning algorithms. Specifically, recognizing the digits may just require the network to be able to extract line-features and it is not difficult to imagine that this type of features are indeed needed for doing the instance-level contrastive learning on multi-digit images as the models need to use the edge information to further separate the images. However, if the images contain multiple objects with overlapping, it is unclear whether the models can still learn meaning high-level information that is useful for single object identification, instead of just learning multi-object features like how they overlap with each other. Therefore, this analysis is limited by the easy-to-learn datasets the authors construct and may not have clear indications on real-world learning scenario. The later sub-analysis in the same section is about whether intermediate features of the SimCLR model trained on ImageNet cluster according to meaningful semantic similarity. Although I agree that the presented visualization indeed shows the cluster phenomenon in the intermediate layers, I feel this result is not surprising at all, as the SimCLR network (and other contrastive learning algorithms) have proven useful in COCO tasks, which would require this type of information. This sub-analysis can also benefit from more quantitative metrics instead of just visualizations.\n3. In the final analysis, the authors show that contrastive learning algorithms can be “misled” by the competing information in the images that is easier to learn. This analysis indeed looks interesting and inspiring to me. The basic result of this analysis is intuitive, as the contrastive learning algorithms seek to find cues to separate different images and the added information to the images is like the cues that can be easy to use to do the separation, especially as it varies more with respect to the image identity. The learning algorithms therefore has no further motivation to find the other features to separate the images as the digits are already enough. This feature can be used as a way to “attack” the contrastive learning algorithms through adding possibly “invisible” features to the images to make them mislead the learning algorithms. Therefore, clearly illustrating this issue is indeed important for finding a way to address this issue later. I think explicitly forcing the algorithm to seek for additional information that can separate the images might be needed and even beneficial for general learning.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "Y-3T5n5KMcU", "reviewer": "Reviewer_mgbf", "summary": "The paper introduces an empirical study on contrastive loss and its variants. In the paper, the contrastive loss is generalized as a linear combination of the alignment between positive representation pairs and the regularization between representation and prior distribution. Based on this formulation, then the authors present some interesting properties of contrastive loss: 1) meaningful local features can be learned when there exist multiple objects in the image; 2) competing features across augmentations could suppress other parts and limit the effectiveness of contrastive loss.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nIntroducing the intriguing properties of contrastive losses looks interesting and important to me. It also provides new insights to the community. Both the motivation and the technical details in the experiment designs seem sound. The evaluation shows the effectiveness of the proposed method on both synthetic datasets and standard datasets. \n\nWeakness:\nThe generalized contrastive loss seems heuristic and could be enhanced if the authors could elaborate the as done in [16]. The alignment and distribution regularization seems to be closely related to variational autoencoder, where the alignment can be achieved with reconstruction. Is there any connection?\n\nSome details are not clarified. For example, if we choose the normal distribution as prior, are the feature vectors normalized like on hypersphere?  \n\nSome open questions:\nAs the authors mention the suppression, if we construct the positive samples to resist the feature competing, e.g., we take channel-wise concatenation as a way to augment samples, will the model be enhanced to resist?  In the paper, the SWD is chosen for the distribution matching, how does it perform when using other metrics like JSD (discriminator-based method), MMD (IPM-based method)? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not discuss the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "Rxa4tf9_hn", "reviewer": "Reviewer_q7am", "summary": "This paper studies properties of the contrastive losses. The contributions are three-folds:\nFirst, the authors find that when using a deep multi-layer nonlinear projection head, the choice of objectives does not matter within a family of generalized losses, the same goes for the choice of batch size. The authors then focus on instance-based objectives, and find that they help learning meaning hierarchical local features despite the objectives being global.\n\nThe most interesting contribution is that the authors identify a phenomenon called \"feature suppression\", where a few dominating features can hinder the learning of other features, posing an open challenge to existing contrastive learning techniques.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Significance__: this paper provides interesting empirical observations on properties of the contrastive loss. I find \"feature suppression\" the most interesting. My concerns are 1) the robustness of this finding, e.g. whether feature suppression is a result of improper optimization; 2) it seems that similar observations have been made in previous work, but this paper does not provide deeper insights as in why this is happening; please see detailed comments below.\nDespite these concerns, I think the phenomena provided in this work are interesting and hence recommend an accept.\n\n__Questions__:\n* Results about Table 3:\n  * the numbers are reported without an error bar; are they from a single run or that the variance is small?\n  * for supervised learning: 1) how is it trained, i.e. which digit is selected as the class label? 2) why is in-grid worse than random placement? Why is there a sudden drop of performance for in-grid when going from 12 to 16 digits?\n  * why is learning multiple objects surprising, especially when the model has sufficient capability?\n\n* (Main concern) Feature suppression:\n  * Is this a result of the optimization algorithm? Can this be resolved by an initial large learning rate? e.g. Figure 3 in \"Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks\" by Li et. al.\n  * Citation 12 (Tian et. al.) made a similar observation in their \"Colorful-Moving-MNIST\" (which is similar to DigitOnImageNet dataset used in this work) section with \"Multiple factors shared\" that sharing the digits dominates learning compared to the position of the digit, which is a form of feature suppression. I agree that this paper is the first to explicitly identify this phenomenon and emphasizes it as a problem, but would appreciate more comments from the authors about the difference, implications, and deeper explanations.\n\n* Clarification on \"Number of unique digits\" (Fig 6): what does the x-axis mean? Is this the total number of distinct digits appearing in the dataset?\n\n* The feature clustering results seem to suggest that supervised & contrastive learning learn similar features, however the feature suppression experiments (especially MultiDigit) seem to say that contrastive learning is more susceptible to being suppressed by \"dominating features\" than supervised learning. Is this it a result of optimization or the objective?\n\n__Writing__: the paper is clearly written and easy to follow. I'd appreciate more details on the training procedure (e.g. the learning rates used, and the variances in different runs), but the message is clear. A minor typo: line 269 \"but we leave that as...\".\n\n====== \nPost-rebuttal update: I thank the authors for their responses and keep my score of recommending an accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss two limitations: first, this paper focuses only on methods with explicit negatives. This is not a problem for me since it is okay for an analysis paper to focus on one type of methods. The second limitation is that the datasets used in the experiments are not fully realistic. This again is not an issue for me, since 1) the datasets used in this paper are variants of ImageNet and MNIST which are both realistic, and 2) fully realistic datasets will make it hard to control multiple aspects of variation with precision.\n\nI agree with the authors' judgement that there is no immediate societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "6d4Biv_vLhp", "reviewer": "Reviewer_MCm8", "summary": "The paper empirically studies three properties of contrastive loss: (1) deep projection head, (2) learning under multiple objects, and (3) feature suppression. In particular, the studies on feature suppression uncover open challenges for contrastive representation learning. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper address an important problem that will be of interest to a wide audience who works on contrastive learning. The paper is well written, I can follow all the details for the paper. The founded three properties are non-trivial and could be influential to algorithm design. For instance, the feature suppression observation has motivate several follow-up works to address this issue [1,2]. It is also interesting to see that making the projection head deeper consistently improves the performance across several formulation. Figure 3 and 4 also provide a possible explanation to why contrastive learning also lead to decent performance beyond classification. The good local features make contrastive learning a promising tool for detection or segmentation. I really like the studies in this paper.\n\nHere are a few questions and suggestions:\n\n1. The paper discover these properties, but does not explain them. For instance, why can deep projection heads lead to such improvement? What if we keep increasing the depth beyond three layers? Will the performance drops at the end? By answering these question could make the study more complete. \n\n2. The feature suppression phenomenon has also been explored in supervised learning setting [3, 4]. As info-NCE loss can be interpreted as instance discrimination, the same analysis from [3, 4] might also be applicable here.\n\n3. It seems that VAE is robust to the features suppression problem as the reconstruction loss encourage all the information about images be preserved. As [5] shows, the uniformity (distribution) loss can be interpreted as an empirical approximation to the entropy of the feature variable Z. In particular, the entropy H(Z) can be interepreted how much input information is preserved in the features as I(X;Z) = H(Z) where X is the input variable. Therefore, an interesting experiment might be varying the weight for the distribution loss in equation (2) to see whether larger \\lambda increase the robustness agains feature suppression.\n\nIn conclusion, the paper conduct interesting experiments that could impact algorithm design. I did not observe obvious flaw in the paper.\n\n[1] Robinson et al., Can contrastive learning avoid shortcut solutions?, 2021\n\n[2] Li et al., Information-Preserving Contrastive Learning for Self-Supervised Representations, 2021\n\n[3] Shah et al., The Pitfalls of Simplicity Bias in Neural Networks, 2020\n\n[4] Hermann et al., What shapes feature representations? Exploring datasets, architectures, and training, 2020\n\n[5] Wang and Isola., Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies three properties of contrastive learning.  Reviewers found that although these properties were shown in experiments the paper did not explain why they happen, but thought they can guide better algorithm design in contrastive learning and of interest to a wide audience. Reviewers agreed that this is a good paper. Please include in the final manuscript the additional explanation provided during the rebuttal.  Accept 
Decision: Accept (Poster)