Chaos is a Ladder: A New Understanding of Contrastive Learning

1 INTRODUCTION . Contrastive Learning ( CL ) emerges to be a promising paradigm for learning data representations without labeled data ( Oord et al. , 2018 ; Hjelm et al. , 2019 ) . Recently , it has achieved impressive results and gradually closed the gap between supervised and unsupervised learning , hopefully leading to a new era that resolves the hunger for labeled data in the deep learning field ( He et al. , 2020 ; Chen et al. , 2020b ) . However , despite its intriguing empirical success , a theoretical understanding of how contrastive learning actually works in practice is still under-explored . The general methodology of contrastive learning is quite simple , that is to maximize the similarity between augmented views of the same image ( a.k.a . positive samples ) , and minimize the similarity between that of two random images ( a.k.a . negative samples ) . Intuitively , it is an instance discrimination task ( differing each image from others ) instead of a classification task ( clustering images from the same class together and differing with other classes ) . Nevertheless , as shown in Figure 1 ( a ) , CL representations are also class-separated . Therefore , understanding how the pretraining ( CL ) task and the downstream ( classification ) task interact plays a central role in both theoretical understandings and practical designings of contrastive methods . Previously , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) have tried to establish guarantees on the classification performance for self-supervised representations . However , their analysis relies heavily on the assumption that the two positive samples , as augmented views of the same image , are ( nearly ) conditionally independent on class y . However , this is hardly practical as the augmented views are still strongly input-dependent ( see Figure 1 ( b ) ) . In fact , if the conditional independence is satisfied , the unsupervised task will become as informative as the supervised task , making this discussion almost unnecessary . This motivates us to find more practical and weaker assumptions to understand how contrastive learning actually works ( even without conditional independence ) . To achieve this , we need to reexamine the contrastive learning process . Previously , Wang & Isola ( 2020 ) show that CL objective involves two goals : alignment ( for positive samples ) and uniformity ( for negative samples ) . Nevertheless , we show that there exist bad cases where the features could still have poor performance even with perfect alignment and uniformity . Thus , contradictory to the common belief of contrastive learning as learning invariance , we note that invariance alone is inadequate to learning useful representations for downstream tasks . In this paper , we provide a novel understanding of contrastive learning that requires only practical and minimal assumptions , while also guarantee class-separated representations . Our core insight hinges on the observation that contrastive learning usually adopts much more aggressive data augmentations than that in supervised learning ( He et al. , 2020 ; Chen et al. , 2020a ) . As shown in Figure 1 ( b ) , we notice that aggressive random cropping of two images can generate views that are very much alike that we could even hardly tell them apart , e.g. , the wheels of two different cars . In other words , there will be connections between different intra-class images through aggressively augmented views of them . Thus , the alignment of positive samples will also cluster all the intraclass samples together , and lead to class-separated representations . From our perspective , the role of data augmentation is to create a certain degree of “ chaos ” between intra-class samples , and the role of contrastive loss is to “ climb the ladder of chaos ” , i.e. , the process that we gradually cluster intra-class samples by aligning positive samples . Following this intuition , we develop a new theory for understanding the effectiveness of contrastive learning . Specifically , we derive the upper and lower bounds for its downstream performance and show how the two will asymptotically converge with our assumptions on the support overlapping effect of data augmentations . Driven by this analysis , we further discuss how varying augmentation will affect the performance of contrastive learning from both synthetic and real-world datasets , and show that the results align well with our theory . In summary , • We characterize the failure of the previous analysis of contrastive learning , and develop a new understanding through the support overlapping effect of data augmentations . Compared to existing theories on contrastive learning , ours can provide guidance to the practical designing of contrastive methods and evaluation metrics . • We establish general guarantees ( both upper and lower bounds ) for the downstream performance without assumptions on conditional independence . And we further show how the two bounds could asymptotically converge under our less restrictive assumptions . • We provide a quantitative discussion on the effect of augmentation strength , which verifies our theory from both theoretical and empirical aspects . Motivated by our theory , we further propose a new unsupervised evaluation metric for contrastive learning named ARC and show that it aligns well with downstream performance on real-world datasets . 2 RELATED WORK . Contrastive Learning in Practice . Contrastive self-supervised learning originates from a mutual information perspective of representation learning ( Oord et al. , 2018 ; Hjelm et al. , 2019 ) , and soon becomes a general learning paradigm that contrasts between positive and negative pairs ( He et al. , 2020 ; Chen et al. , 2020a ) . It is rapidly closing the performance gap between unsupervised and supervised learning on large-scale dataset like ImageNet ( Chen et al. , 2021 ) , and outperforms supervised learning when combined with a few ( e.g. , 10 % ) labels ( Chen et al. , 2020b ) . Several recent works show that similar performance could be achieved without negative samples by adopting certain training techniques ( Grill et al. , 2020 ; Chen & He , 2020 ) . Understanding Contrastive Learning Objectives . Both the original InfoNCE loss ( Oord et al. , 2018 ) and its InfoMax variants ( Hjelm et al. , 2019 ; Poole et al. , 2019 ) are designed as variational estimates of the mutual information between inputs and representations , but these estimators are shown to have poor bias-variance trade-offs ( Song & Ermon , 2020 ) . Instead , Wang & Isola ( 2020 ) simply understand contrastive learning through the two terms in the InfoNCE loss : alignment of positive samples and uniformity of negative samples . However , as we show later , this perspective is also insufficient to explain the effectiveness of contrastive learning , and we should take the interplay between augmentation and alignment into consideration . Understanding Downstream Generalization . Saunshi et al . ( 2019 ) propose the first theoretical guarantees by bridging the contrastive and classification objectives . Lee et al . ( 2020 ) further link the reconstruction-based objective to the downstream objective . However , both Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) rely on the unrealistic assumption that the positive samples are ( nearly ) conditionally independent . Besides , there are several other perspectives for analyzing contrastive learning , e.g. , information-theoretical perspective ( Tian et al. , 2020 ; Tsai et al. , 2021 ; Tosh et al. , 2020 ; 2021 ) . Recently , similar to our analysis , HaoChen et al . ( 2021 ) also study the augmentation graph and establish guarantees in terms of graph connectivity . Our work differs to theirs mainly in three aspects : 1 ) our analysis is applicable for the widely adopted InfoNCE and CE losses , while theirs is developed for their own spectral loss ; 2 ) ours starts from the alignment and uniformity perspective while theirs starts from the matrix decomposition perspective ; 3 ) our theory is empirically verified and inspires a useful evaluation metric for data augmentation , while their analysis focusing on minimizing the decomposition error is farther from the practical designing of positive and negative samples . In a nutshell , compared to previous discussions , our theory has a closer connection to the actual contrastive learning process , and we verify the feasibility of each assumption with empirical evidence . 3 LIMITATIONS OF PREVIOUS UNDERSTANDINGS . 3.1 PROBLEM SETUP . We begin by introducing the basic notations and common practice of contrastive learning in the image classification task . In general , it has two stages , unsupervised pretraining , and supervised finetuning . In the first stage , with N unlabeled samples Du = { xi } Ni=1 , we pretrain an encoder mapping from the d-dimensonal input space to a unit hypersphere f ∈ F : Rd → Sm−1 in the mdimensional space . In the second stage , we evaluate the learned representations z with the labeled data Dl = { ( xi , yi ) } where labels yi ∈ { 1 , . . . , K } . Specifically , we fix the encoder and learn a linear classification head g : Rm → RK on top from D̃l = { ( z , y ) |z = f ( x ) ∈ Rm } . Contrastive Pretraining . Taking a training example x ∈ Du , we draw its positive sample x+ = t ( x ) by applying a random data augmentation t ∼ T , and draw M randomly augmented samples { x−i } Mi=1 from Du as its negative samples . Then , we can learn the encoder f with the widely used InfoNCE loss ( Oord et al. , 2018 ) LNCE ( f ) =Ep ( x , x+ ) E { p ( x−i ) } [ − log exp ( f ( x ) > f ( x+ ) ) ∑M i=1 exp ( f ( x ) > f ( x−i ) ) ] =Ep ( x ) [ −Ep ( x+|x ) f ( x ) > f ( x+ ) + E { p ( x−i ) } log M∑ i=1 exp ( f ( x ) > f ( x−i ) ) ] . ( 1 ) Let p ( x ) be the data distribution , p ( x , x+ ) be the joint distribution of positive pairs , and we simply assume p ( x , x+ ) = p ( x+ , x ) and p ( x ) = ∫ p ( x , x+ ) dx+ , ∀ x ∈ Rd following Wang & Isola ( 2020 ) . Linear Evaluation . To evaluate the learned representations by contrastive learning , we usually adopt the Cross Entropy ( CE ) loss ( Chen et al. , 2020a ) for a labeled pair ( x , y ) ∈ Dl LCE ( f , g ) = Ep ( x , y ) [ − log exp ( f ( x ) > wy ) ∑K i=1 exp ( f ( x ) > wi ) ] , ( 2 ) with a linear classifier g ( z ) = Wz where W = [ w1 , w2 , . . . , wK ] . 3.2 EXISTING THEORETICAL ASSUMPTIONS AND THEIR LIMITATIONS . As discussed above , there are some previous understandings on how contrastive learning yields good performance , and they mainly differ by their theoretical assumptions . First , Wang & Isola ( 2020 ) interpret the first and second terms of the InfoNCE loss ( Eq . 1 ) as they are aiming at the following two properties : 1 ) alignment ( the 1st term ) : positive samples x , x+ has similar features , i.e. , f ( x ) ≈ f ( x+ ) ; 2 ) uniformity ( the 2nd term ) : features are roughly uniformly distributed in the unit hypersphere Sm−1 . In particular , they show that InfoNCE can be minimized with 1 ) perfect alignment and 2 ) perfect uniformity . However , as we illustrate in Figure 2 , the features could still have very poor downstream performance in the finite sample scenario . Proposition 3.1 ( Class-uniform features also minimize the InfoNCE loss ) . For N training examples of K classes , consider the case when features { f ( xi ) } Ni=1 are randomly distributed in Sm−1 with maximal uniformity ( i.e. , , minimizing the 2nd term of Eq . 1 ) while also satisfying ∀xi , x+i ∼ p ( x , x+ ) , f ( xi ) = f ( x + i ) . Because we have these two properties , the InfoNCE loss achieves its minimum . However , the downstream classification accuracy is at most 1/K + ε and ε is nearly zero when N is large enough . This indicates that the instance discrimination task ( alignment + uniformity ) alone can not guarantee the learning of class-discriminative features as desired in the downsteam classification . Instead , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) both establish the relationship between pretraining and classification objectives and provide guarantees for the downstream performance . In fact , the two works both assume the conditional independence of the two positive samples , i.e. , p ( x , x+|y ) = p ( x|y ) p ( x+|y ) . However , this assumption is too strong as it is hardly practical . As shown in Figure 1 ( b ) , augmented views from the same class are not actually independent as views from the same sample are more alike than that from other samples .

Summary: This paper aims to provide theoretical understanding for contrastive learning where "similar pairs" of points $x$ and $x^+$ are encouraged to have similar representations through an InfoNCE inspired objective function. Some prior works show the benefit of learned representations for linearly classifying downstream classes, by making conditional independence like assumption on the similar pairs or positive samples, i.e. $x$ and $x^+$ are (approximately) conditionally independent given downstream label $y$. This work argues that these assumptions are quite strong for contrastive learning with data augmentations, and aims to show guarantees under the following weaker and more realistic assumption: support of augmentation distribution of different inputs from the same class overlap to form a "connected graph" of inputs within a class, whereas support of augmentations of inputs from different classes do not overlap. Lower and upper bounds using this and some other assumptions, connecting the downstream performance of representation function to the contrastive loss. Some simulation experiments are presented to support some aspects of the theoretical analysis.  Using the insights from the analysis, the paper proposes an "Average Confusion Ratio (ACR)" metric that can be used to predict the ranking of downstream performances of different augmentations **using only unlabeled data**. Experimental evidence is provided on CIFAR and STL datasets to verify the efficacy of this metric for some practical augmentations.    While there are some interesting aspects in the paper (especially the ACR metric), the theoretical analysis seems to have raised many questions and concerns that I have summarized below (details in main review).   - **Soundness of assumptions**: Assumption 4.6, which is crucial, seems questionable and may not be coherent or appropriate to make in this setting. More on this in point (W2) of main review  - **Deeper dive into theoretical results**: There is a lack of discussion about the (non-)vacuousness of the bounds in the main results Theorem 4.2 and 4.8, that puts the interpretation and significance of the result in question. More on this and related issues in point (W2) of main review.  - **Comparison to prior work**: The work of HaoChen et al. in particular is not adequately compared to, especially since some of the points being addressed here are covered through a different kind of analysis in that paper. More on this in point (W3) of main review.