Intriguing Properties of Contrastive Losses

1 Introduction . Contrastive learning [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] has recently achieved great successes in learning visual representations without supervision . As shown in [ 13 , 14 ] , contrastive learning can learn representations that rival supervised learning , and significantly improve the state-of-the-art in semi-supervised learning on ImageNet . One successful use case of contrastive loss for self-supervised learning is to make augmented views of the same example agree [ 1 , 2 , 13 ] . A widely used contrastive loss to encourage agreement is based on cross entropy [ 15 , 3 , 4 , 13 ] . Given an augmented view of an example , the contrastive prediction task aims to classify a set of candidates into the positive example ( i.e . the other augmented view of the same example ) and negative ones via the cross entropy loss . In this work , to understand the effectiveness and limitation of existing contrastive learning methods , we study three intriguing aspects . First , we propose a generalization of the standard contrastive loss , and systematically study their performance differences . Second , we study if the instance-based contrastive learning , for which the contrastive loss operates on global representation of an input image , can learn well on images with multiple objects present , and whether or not it leads to meaningful local features . Finally , we systematically study the feature suppression phenomenon in contrastive learning . The suppression effect occurs among competing features shared across augmented views . For example , with random cropping as the augmentation , “ color distribution ” and “ object class ” are often competing features as they are likely shared between two augmented views . The suppression effect among competing features can significantly degenerate the representation quality , or even 1Code and visualization at https : //contrastive-learning.github.io/intriguing . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . completely disable the learning of certain features , as shown in our experiments . Existing methods critically rely on hand-crafted data augmentation to favor certain sets of competing features than others . Our main findings and contributions are summarized below . • We propose a generalized contrastive loss , and show that differences between contrastive losses are small with a deep projection head . • We show that the instance-based objective widely used in existing contrastive learning methods can learn on images with multiple objects , and also learn meaningful local features despite operating on global image representation . • We construct three datasets with explicit and controllable competing features to systematically study the feature suppression effect in contrastive learning . • We show that a few bits of easy-to-learn shared features can suppress , and even fully prevent , the learning of other sets of competing features . In scenarios where there are multiple objects in an image , the dominant object would suppress the learning of smaller objects . This poses open challenges to existing contrastive learning . 2 Generalized contrastive loss and differences among its instantiations . The common contrastive loss used in most recent work is based on cross entropy [ 15 , 3 , 4 ] . Following the notation in [ 13 ] , the contrastive loss can be defined between two augmented views ( i , j ) of the same example for a mini-batch of size of n , and can be written as the following . LNT-Xent = − 1 n ∑ i , j∈MB log exp ( sim ( zi , zj ) /τ ) ∑2n k=1 1 [ k 6=i ] exp ( sim ( zi , zk ) /τ ) ( 1 ) where zi , zj are hidden representations of two augmented views of the same example ; sim ( u , v ) = uTv/ ( ‖u‖‖v‖ ) is the cosine similarity between two vectors ; τ is a temperature scalar andMB is a randomly sampled mini-batch consisting of augmented pairs of images . In [ 13 ] , a MLP projection head is introduced between intermediate layer h ( e.g . output of ResNet encoder ) and final output z . It is shown that the projection head is very beneficial and h is a much better feature representation than z . In this work , we generalize the standard contrastive loss to the following form . Lgeneralized contrastive = Lalignment + λLdistribution ( 2 ) Both terms are defined on hidden representations . Lalignment encourages representations of augmented views to be consistent , while Ldistribution encourages representations ( or a random subset of them ) to match a prior distribution ( of high entropy ) . It is not difficult to see that the standard contrastive loss in Eq . 1 is a special case as it can be re-written as follows ( scaled by a constant τ ) . τLNT-Xent = − 1 n ∑ i , j sim ( zi , zj ) ︸ ︷︷ ︸ Lalignment + τ n ∑ i log 2n∑ k=1 1 [ k 6=i ] exp ( sim ( zi , zk ) /τ ) ︸ ︷︷ ︸ Ldistribution ( 3 ) This form of factorization in Eq . 3 has been proposed in [ 16 ] , where the second LogSumExp term is referred to as uniformity since it encourages representation to uniformly distributed in the hypersphere . Different from [ 16 ] , here we generalize the hypersphere uniform distribution and study a wider set of prior distributions for their effectiveness in learning representations . SWD for supporting diverse prior distributions . One issue of using more diverse set of priors is we can not rely on LogSumExp for matching the distribution . To this end , we resort to the theory of optimal transport , via Sliced Wasserstein Distance ( SWD ) [ 17 , 18 , 19 ] . For two sets of equalsized samples from two 1-D distributions , the optimal transport can be obtained by computing two Algorithm 1 Sliced Wasserstein Distance ( SWD ) loss . Lalign Prior distribution Ldistribution 1 nd ∑ i , j ‖z̃i − z̃j‖ 2 Uniform hypersphere 1 n ∑ i log ∑ j exp ( z̃i T z̃j/τ ) 1 nd ∑ i , j ‖z̃i − z̃j‖ 2 Uniform hypersphere SWD ( Z̃ , Zprior ) 1 nd ∑ i , j ‖zi − zj‖ 2 Uniform hypercube SWD ( Z , Zprior ) 1 nd ∑ i , j ‖zi − zj‖ 2 Normal distribution SWD ( Z , Zprior ) permutations that order the values of both sets of samples respectively . The 1-D Wasserstein distance can then be computed with ` 2 distance between the ordered values . For n-D distributions , we first project the samples to n randomly-generated orthogonal 1-D subspaces , and then compute the sum of 1-D Wasserstein distance across all 1-D subspaces . By adjusting the network weights to minimize the SWD , we are able to reduce the mismatch between the distribution of hidden vectors and a known prior distribution . The detailed algorithm can be found in Algorithm 1 . With SWD loss , we are able to use a wider set of priors , and Table 1 summarizes instantiations of the generalized contrastive loss with different prior distributions and distribution matching loss . Connection with mutual information . The connection between the standard contrastive loss and mutual information has been shown before [ 3 , 20 ] , where the contrastive loss ( a.k.a . InfoNCE loss [ 3 ] ) is shown to be a lower bound of the mutual information . To connect the generalized contrastive loss to mutual information , we start by the definition of mutual information between two latent variables U , V , which is I ( U ; V ) = H ( U ) −H ( U |V ) . Comparing this factorization of mutual information with generalized contrastive loss , it is not difficult to see that : 1 ) the alignment term Lalignment is directly related to H ( U |V ) which aims to reduce uncertainty of the other views given one view of the example ; and 2 ) the distribution matching term Ldistribution can be considered as a proxy to H ( u ) for maximizing the entropy in the representation . It is perhaps worth noting that different from mutual information , the generalized contrastive loss ( Eq . 2 ) allows a tunable weight ( λ ) between the alignment and distribution matching term . The weighting scalar λ is ( inversely ) related to the temperature τ ( details in Appendix A.2 ) . Comparing different instantiations of generalized contrastive loss . Here we ask : Is it essential to use a uniform hypersphere prior for the effectiveness of contrastive loss ? How much difference does it make when distinct generalized contrastive losses are used ? To answer this question , we conduct experiments following SimCLR settings [ 13 , 14 ] , and use the linear evaluation protocol . Detailed experimental setup can be found in Appendix A.1 . Figure 1 shows linear evaluation results of models trained with different losses under different training epochs . On CIFAR-10 , we see little difference in terms of linear evaluation for variants of the generalized contrastive losses , especially when trained longer than 200 epochs . As for ImageNet , there are some discrepancies between different losses , but they disappear when a deeper 3-layer non-linear projection head is used . Furthermore , we find that deep projection head not only reduces the differences among different generalized contrastive losses , but has a similar effect for batch size . With proper learning rate scaling across batch sizes ( e.g . square root scaling with LARS optimizer [ 21 ] ) , the impact of batch size on representation quality is small . Table 2 demonstrate this phenomenon for the standard contrastive loss , and more results on other losses can be found in Appendix A.3 . 3 Instance-based objective can learn on images with multiple objects and learn good local features . Most existing contrastive learning methods [ 13 , 10 , 22 , 4 ] define their objectives at the instance level where each image is encoded into a single vector representation ( e.g . representations of two random crops of the same image instance are treated as a positive pair ) . In other words , the objective operates on a global representation of its input rather than on some local regions ( of its input ) . We pose two questions regarding instance-based global objective : 1 ) when there is only a single ( dominant ) object in the image , the objective seems reasonable as it encourages the model to learn features relevant to object class , but when there are multiple objects present in the image , can instance-based objective still learn well ? 2 ) Since the instance-based objective uses a global summary of its input , can it still learn good local features ( e.g . parts of an object , or multiple objects in the same scheme ) ? To answer these questions , we use SimCLR as representative for the instance-based objective . 3.1 SimCLR can learn on images with multiple objects . Commonly used self-supervised learning datasets , such as MNIST , CIFAR-10 , ImageNet , are object centered , i.e . the image is mainly occupied by a single ( dominant ) object . To experiment with multiple objects in a controllable setting , we propose a new dataset setting by composing multiple digits as follows . MultiDigits dataset . We place MNIST digits ( 28 × 28 size ) on a shared canvas ( 112 × 112 size ) . We vary the number of digits placed on the canvas . One factor that could interfere with learning of multiple digits is overlapping digits , therefore we use two placement strategies : random vs in-grid ( Figure 2 ) . Random placement of digits incurs no constraint on where digits can be placed on the canvas , whereas in-grid placement puts each digit in one of the 4× 4 grid cells the canvas is divided into , and no two digits can fall in the same cell . In-grid placement ensures no overlapping of digits . We first pretrain a ResNet-18 with SimCLR or supervised learning with the same augmentation policy ( random cropping and resize ) on MultiDigits dataset . To access the representation quality , we then train linear classifiers for images with a single digit of size 28× 28 on the canvas . Similarly during evaluation , we place only one digit of size 28× 28 on the canvas . As shown in Table 3 , representations learned using supervised loss maintains its quality when up to 8 digits are placed in the image . After that the representation becomes worse as the canvas gets more crowded . Notably , representations learned using SimCLR display a similar phenomenon . Regardless of placement strategy , top-1 accuracy stays at the same level up to 8 digits , demonstrating that SimCLR can learn from images with multiple objects . In addition , the increased performance gap between the two placement strategies with increased number of digits shows that object overlapping makes it harder for contrastive losses to learn from multiple objects .

Summary: This paper studies properties of the contrastive losses. The contributions are three-folds: First, the authors find that when using a deep multi-layer nonlinear projection head, the choice of objectives does not matter within a family of generalized losses, the same goes for the choice of batch size. The authors then focus on instance-based objectives, and find that they help learning meaning hierarchical local features despite the objectives being global.  The most interesting contribution is that the authors identify a phenomenon called "feature suppression", where a few dominating features can hinder the learning of other features, posing an open challenge to existing contrastive learning techniques.