Intriguing Properties of Contrastive Losses

1 Introduction . Contrastive learning [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] has recently achieved great successes in learning visual representations without supervision . As shown in [ 13 , 14 ] , contrastive learning can learn representations that rival supervised learning , and significantly improve the state-of-the-art in semi-supervised learning on ImageNet . One successful use case of contrastive loss for self-supervised learning is to make augmented views of the same example agree [ 1 , 2 , 13 ] . A widely used contrastive loss to encourage agreement is based on cross entropy [ 15 , 3 , 4 , 13 ] . Given an augmented view of an example , the contrastive prediction task aims to classify a set of candidates into the positive example ( i.e . the other augmented view of the same example ) and negative ones via the cross entropy loss . In this work , to understand the effectiveness and limitation of existing contrastive learning methods , we study three intriguing aspects . First , we propose a generalization of the standard contrastive loss , and systematically study their performance differences . Second , we study if the instance-based contrastive learning , for which the contrastive loss operates on global representation of an input image , can learn well on images with multiple objects present , and whether or not it leads to meaningful local features . Finally , we systematically study the feature suppression phenomenon in contrastive learning . The suppression effect occurs among competing features shared across augmented views . For example , with random cropping as the augmentation , “ color distribution ” and “ object class ” are often competing features as they are likely shared between two augmented views . The suppression effect among competing features can significantly degenerate the representation quality , or even 1Code and visualization at https : //contrastive-learning.github.io/intriguing . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . completely disable the learning of certain features , as shown in our experiments . Existing methods critically rely on hand-crafted data augmentation to favor certain sets of competing features than others . Our main findings and contributions are summarized below . • We propose a generalized contrastive loss , and show that differences between contrastive losses are small with a deep projection head . • We show that the instance-based objective widely used in existing contrastive learning methods can learn on images with multiple objects , and also learn meaningful local features despite operating on global image representation . • We construct three datasets with explicit and controllable competing features to systematically study the feature suppression effect in contrastive learning . • We show that a few bits of easy-to-learn shared features can suppress , and even fully prevent , the learning of other sets of competing features . In scenarios where there are multiple objects in an image , the dominant object would suppress the learning of smaller objects . This poses open challenges to existing contrastive learning . 2 Generalized contrastive loss and differences among its instantiations . The common contrastive loss used in most recent work is based on cross entropy [ 15 , 3 , 4 ] . Following the notation in [ 13 ] , the contrastive loss can be defined between two augmented views ( i , j ) of the same example for a mini-batch of size of n , and can be written as the following . LNT-Xent = − 1 n ∑ i , j∈MB log exp ( sim ( zi , zj ) /τ ) ∑2n k=1 1 [ k 6=i ] exp ( sim ( zi , zk ) /τ ) ( 1 ) where zi , zj are hidden representations of two augmented views of the same example ; sim ( u , v ) = uTv/ ( ‖u‖‖v‖ ) is the cosine similarity between two vectors ; τ is a temperature scalar andMB is a randomly sampled mini-batch consisting of augmented pairs of images . In [ 13 ] , a MLP projection head is introduced between intermediate layer h ( e.g . output of ResNet encoder ) and final output z . It is shown that the projection head is very beneficial and h is a much better feature representation than z . In this work , we generalize the standard contrastive loss to the following form . Lgeneralized contrastive = Lalignment + λLdistribution ( 2 ) Both terms are defined on hidden representations . Lalignment encourages representations of augmented views to be consistent , while Ldistribution encourages representations ( or a random subset of them ) to match a prior distribution ( of high entropy ) . It is not difficult to see that the standard contrastive loss in Eq . 1 is a special case as it can be re-written as follows ( scaled by a constant τ ) . τLNT-Xent = − 1 n ∑ i , j sim ( zi , zj ) ︸ ︷︷ ︸ Lalignment + τ n ∑ i log 2n∑ k=1 1 [ k 6=i ] exp ( sim ( zi , zk ) /τ ) ︸ ︷︷ ︸ Ldistribution ( 3 ) This form of factorization in Eq . 3 has been proposed in [ 16 ] , where the second LogSumExp term is referred to as uniformity since it encourages representation to uniformly distributed in the hypersphere . Different from [ 16 ] , here we generalize the hypersphere uniform distribution and study a wider set of prior distributions for their effectiveness in learning representations . SWD for supporting diverse prior distributions . One issue of using more diverse set of priors is we can not rely on LogSumExp for matching the distribution . To this end , we resort to the theory of optimal transport , via Sliced Wasserstein Distance ( SWD ) [ 17 , 18 , 19 ] . For two sets of equalsized samples from two 1-D distributions , the optimal transport can be obtained by computing two Algorithm 1 Sliced Wasserstein Distance ( SWD ) loss . Lalign Prior distribution Ldistribution 1 nd ∑ i , j ‖z̃i − z̃j‖ 2 Uniform hypersphere 1 n ∑ i log ∑ j exp ( z̃i T z̃j/τ ) 1 nd ∑ i , j ‖z̃i − z̃j‖ 2 Uniform hypersphere SWD ( Z̃ , Zprior ) 1 nd ∑ i , j ‖zi − zj‖ 2 Uniform hypercube SWD ( Z , Zprior ) 1 nd ∑ i , j ‖zi − zj‖ 2 Normal distribution SWD ( Z , Zprior ) permutations that order the values of both sets of samples respectively . The 1-D Wasserstein distance can then be computed with ` 2 distance between the ordered values . For n-D distributions , we first project the samples to n randomly-generated orthogonal 1-D subspaces , and then compute the sum of 1-D Wasserstein distance across all 1-D subspaces . By adjusting the network weights to minimize the SWD , we are able to reduce the mismatch between the distribution of hidden vectors and a known prior distribution . The detailed algorithm can be found in Algorithm 1 . With SWD loss , we are able to use a wider set of priors , and Table 1 summarizes instantiations of the generalized contrastive loss with different prior distributions and distribution matching loss . Connection with mutual information . The connection between the standard contrastive loss and mutual information has been shown before [ 3 , 20 ] , where the contrastive loss ( a.k.a . InfoNCE loss [ 3 ] ) is shown to be a lower bound of the mutual information . To connect the generalized contrastive loss to mutual information , we start by the definition of mutual information between two latent variables U , V , which is I ( U ; V ) = H ( U ) −H ( U |V ) . Comparing this factorization of mutual information with generalized contrastive loss , it is not difficult to see that : 1 ) the alignment term Lalignment is directly related to H ( U |V ) which aims to reduce uncertainty of the other views given one view of the example ; and 2 ) the distribution matching term Ldistribution can be considered as a proxy to H ( u ) for maximizing the entropy in the representation . It is perhaps worth noting that different from mutual information , the generalized contrastive loss ( Eq . 2 ) allows a tunable weight ( λ ) between the alignment and distribution matching term . The weighting scalar λ is ( inversely ) related to the temperature τ ( details in Appendix A.2 ) . Comparing different instantiations of generalized contrastive loss . Here we ask : Is it essential to use a uniform hypersphere prior for the effectiveness of contrastive loss ? How much difference does it make when distinct generalized contrastive losses are used ? To answer this question , we conduct experiments following SimCLR settings [ 13 , 14 ] , and use the linear evaluation protocol . Detailed experimental setup can be found in Appendix A.1 . Figure 1 shows linear evaluation results of models trained with different losses under different training epochs . On CIFAR-10 , we see little difference in terms of linear evaluation for variants of the generalized contrastive losses , especially when trained longer than 200 epochs . As for ImageNet , there are some discrepancies between different losses , but they disappear when a deeper 3-layer non-linear projection head is used . Furthermore , we find that deep projection head not only reduces the differences among different generalized contrastive losses , but has a similar effect for batch size . With proper learning rate scaling across batch sizes ( e.g . square root scaling with LARS optimizer [ 21 ] ) , the impact of batch size on representation quality is small . Table 2 demonstrate this phenomenon for the standard contrastive loss , and more results on other losses can be found in Appendix A.3 . 3 Instance-based objective can learn on images with multiple objects and learn good local features . Most existing contrastive learning methods [ 13 , 10 , 22 , 4 ] define their objectives at the instance level where each image is encoded into a single vector representation ( e.g . representations of two random crops of the same image instance are treated as a positive pair ) . In other words , the objective operates on a global representation of its input rather than on some local regions ( of its input ) . We pose two questions regarding instance-based global objective : 1 ) when there is only a single ( dominant ) object in the image , the objective seems reasonable as it encourages the model to learn features relevant to object class , but when there are multiple objects present in the image , can instance-based objective still learn well ? 2 ) Since the instance-based objective uses a global summary of its input , can it still learn good local features ( e.g . parts of an object , or multiple objects in the same scheme ) ? To answer these questions , we use SimCLR as representative for the instance-based objective . 3.1 SimCLR can learn on images with multiple objects . Commonly used self-supervised learning datasets , such as MNIST , CIFAR-10 , ImageNet , are object centered , i.e . the image is mainly occupied by a single ( dominant ) object . To experiment with multiple objects in a controllable setting , we propose a new dataset setting by composing multiple digits as follows . MultiDigits dataset . We place MNIST digits ( 28 × 28 size ) on a shared canvas ( 112 × 112 size ) . We vary the number of digits placed on the canvas . One factor that could interfere with learning of multiple digits is overlapping digits , therefore we use two placement strategies : random vs in-grid ( Figure 2 ) . Random placement of digits incurs no constraint on where digits can be placed on the canvas , whereas in-grid placement puts each digit in one of the 4× 4 grid cells the canvas is divided into , and no two digits can fall in the same cell . In-grid placement ensures no overlapping of digits . We first pretrain a ResNet-18 with SimCLR or supervised learning with the same augmentation policy ( random cropping and resize ) on MultiDigits dataset . To access the representation quality , we then train linear classifiers for images with a single digit of size 28× 28 on the canvas . Similarly during evaluation , we place only one digit of size 28× 28 on the canvas . As shown in Table 3 , representations learned using supervised loss maintains its quality when up to 8 digits are placed in the image . After that the representation becomes worse as the canvas gets more crowded . Notably , representations learned using SimCLR display a similar phenomenon . Regardless of placement strategy , top-1 accuracy stays at the same level up to 8 digits , demonstrating that SimCLR can learn from images with multiple objects . In addition , the increased performance gap between the two placement strategies with increased number of digits shows that object overlapping makes it harder for contrastive losses to learn from multiple objects .

Summary: This work presents three empirical analyses on contrastive learning algorithms. The first analysis shows that when using a multi-layer non-linear project head, the different ways of instantiating the generalized contrastive learning loss lead to final representations with similar downstream visual task performance. The second analysis finds that the instance-based contrastive learning algorithms can still achieve good performance on clustered images, even though these algorithms are based on global image representations. The third analysis is about the preference of contrastive learning algorithms on competing features that are easy-to-learn. In this analysis, the authors show that after manually adding diverse competing information into the images and linking them with image identity, the contrastive learning algorithms fail to capture the general statistics but chooses to only encode the competing information. These three empirical analyses seem to be unrelated and all have certain limitations making the analyses not general to all algorithms. Although the final analysis illustrates that the contrastive learning algorithms indeed have certain limitations, the authors have not proposed a better or modified algorithm to address that issue. In summary, I feel the analyses presented in this paper are useful for the community to be aware of, so I would weakly recommend acceptance. But the current paper can be much stronger if the authors can address the limitations in the analyses (see detailed review below) and even further leverage the analyses to propose even better algorithms resolving the presented issues in the analyses.