Chaos is a Ladder: A New Understanding of Contrastive Learning

1 INTRODUCTION . Contrastive Learning ( CL ) emerges to be a promising paradigm for learning data representations without labeled data ( Oord et al. , 2018 ; Hjelm et al. , 2019 ) . Recently , it has achieved impressive results and gradually closed the gap between supervised and unsupervised learning , hopefully leading to a new era that resolves the hunger for labeled data in the deep learning field ( He et al. , 2020 ; Chen et al. , 2020b ) . However , despite its intriguing empirical success , a theoretical understanding of how contrastive learning actually works in practice is still under-explored . The general methodology of contrastive learning is quite simple , that is to maximize the similarity between augmented views of the same image ( a.k.a . positive samples ) , and minimize the similarity between that of two random images ( a.k.a . negative samples ) . Intuitively , it is an instance discrimination task ( differing each image from others ) instead of a classification task ( clustering images from the same class together and differing with other classes ) . Nevertheless , as shown in Figure 1 ( a ) , CL representations are also class-separated . Therefore , understanding how the pretraining ( CL ) task and the downstream ( classification ) task interact plays a central role in both theoretical understandings and practical designings of contrastive methods . Previously , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) have tried to establish guarantees on the classification performance for self-supervised representations . However , their analysis relies heavily on the assumption that the two positive samples , as augmented views of the same image , are ( nearly ) conditionally independent on class y . However , this is hardly practical as the augmented views are still strongly input-dependent ( see Figure 1 ( b ) ) . In fact , if the conditional independence is satisfied , the unsupervised task will become as informative as the supervised task , making this discussion almost unnecessary . This motivates us to find more practical and weaker assumptions to understand how contrastive learning actually works ( even without conditional independence ) . To achieve this , we need to reexamine the contrastive learning process . Previously , Wang & Isola ( 2020 ) show that CL objective involves two goals : alignment ( for positive samples ) and uniformity ( for negative samples ) . Nevertheless , we show that there exist bad cases where the features could still have poor performance even with perfect alignment and uniformity . Thus , contradictory to the common belief of contrastive learning as learning invariance , we note that invariance alone is inadequate to learning useful representations for downstream tasks . In this paper , we provide a novel understanding of contrastive learning that requires only practical and minimal assumptions , while also guarantee class-separated representations . Our core insight hinges on the observation that contrastive learning usually adopts much more aggressive data augmentations than that in supervised learning ( He et al. , 2020 ; Chen et al. , 2020a ) . As shown in Figure 1 ( b ) , we notice that aggressive random cropping of two images can generate views that are very much alike that we could even hardly tell them apart , e.g. , the wheels of two different cars . In other words , there will be connections between different intra-class images through aggressively augmented views of them . Thus , the alignment of positive samples will also cluster all the intraclass samples together , and lead to class-separated representations . From our perspective , the role of data augmentation is to create a certain degree of “ chaos ” between intra-class samples , and the role of contrastive loss is to “ climb the ladder of chaos ” , i.e. , the process that we gradually cluster intra-class samples by aligning positive samples . Following this intuition , we develop a new theory for understanding the effectiveness of contrastive learning . Specifically , we derive the upper and lower bounds for its downstream performance and show how the two will asymptotically converge with our assumptions on the support overlapping effect of data augmentations . Driven by this analysis , we further discuss how varying augmentation will affect the performance of contrastive learning from both synthetic and real-world datasets , and show that the results align well with our theory . In summary , • We characterize the failure of the previous analysis of contrastive learning , and develop a new understanding through the support overlapping effect of data augmentations . Compared to existing theories on contrastive learning , ours can provide guidance to the practical designing of contrastive methods and evaluation metrics . • We establish general guarantees ( both upper and lower bounds ) for the downstream performance without assumptions on conditional independence . And we further show how the two bounds could asymptotically converge under our less restrictive assumptions . • We provide a quantitative discussion on the effect of augmentation strength , which verifies our theory from both theoretical and empirical aspects . Motivated by our theory , we further propose a new unsupervised evaluation metric for contrastive learning named ARC and show that it aligns well with downstream performance on real-world datasets . 2 RELATED WORK . Contrastive Learning in Practice . Contrastive self-supervised learning originates from a mutual information perspective of representation learning ( Oord et al. , 2018 ; Hjelm et al. , 2019 ) , and soon becomes a general learning paradigm that contrasts between positive and negative pairs ( He et al. , 2020 ; Chen et al. , 2020a ) . It is rapidly closing the performance gap between unsupervised and supervised learning on large-scale dataset like ImageNet ( Chen et al. , 2021 ) , and outperforms supervised learning when combined with a few ( e.g. , 10 % ) labels ( Chen et al. , 2020b ) . Several recent works show that similar performance could be achieved without negative samples by adopting certain training techniques ( Grill et al. , 2020 ; Chen & He , 2020 ) . Understanding Contrastive Learning Objectives . Both the original InfoNCE loss ( Oord et al. , 2018 ) and its InfoMax variants ( Hjelm et al. , 2019 ; Poole et al. , 2019 ) are designed as variational estimates of the mutual information between inputs and representations , but these estimators are shown to have poor bias-variance trade-offs ( Song & Ermon , 2020 ) . Instead , Wang & Isola ( 2020 ) simply understand contrastive learning through the two terms in the InfoNCE loss : alignment of positive samples and uniformity of negative samples . However , as we show later , this perspective is also insufficient to explain the effectiveness of contrastive learning , and we should take the interplay between augmentation and alignment into consideration . Understanding Downstream Generalization . Saunshi et al . ( 2019 ) propose the first theoretical guarantees by bridging the contrastive and classification objectives . Lee et al . ( 2020 ) further link the reconstruction-based objective to the downstream objective . However , both Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) rely on the unrealistic assumption that the positive samples are ( nearly ) conditionally independent . Besides , there are several other perspectives for analyzing contrastive learning , e.g. , information-theoretical perspective ( Tian et al. , 2020 ; Tsai et al. , 2021 ; Tosh et al. , 2020 ; 2021 ) . Recently , similar to our analysis , HaoChen et al . ( 2021 ) also study the augmentation graph and establish guarantees in terms of graph connectivity . Our work differs to theirs mainly in three aspects : 1 ) our analysis is applicable for the widely adopted InfoNCE and CE losses , while theirs is developed for their own spectral loss ; 2 ) ours starts from the alignment and uniformity perspective while theirs starts from the matrix decomposition perspective ; 3 ) our theory is empirically verified and inspires a useful evaluation metric for data augmentation , while their analysis focusing on minimizing the decomposition error is farther from the practical designing of positive and negative samples . In a nutshell , compared to previous discussions , our theory has a closer connection to the actual contrastive learning process , and we verify the feasibility of each assumption with empirical evidence . 3 LIMITATIONS OF PREVIOUS UNDERSTANDINGS . 3.1 PROBLEM SETUP . We begin by introducing the basic notations and common practice of contrastive learning in the image classification task . In general , it has two stages , unsupervised pretraining , and supervised finetuning . In the first stage , with N unlabeled samples Du = { xi } Ni=1 , we pretrain an encoder mapping from the d-dimensonal input space to a unit hypersphere f ∈ F : Rd → Sm−1 in the mdimensional space . In the second stage , we evaluate the learned representations z with the labeled data Dl = { ( xi , yi ) } where labels yi ∈ { 1 , . . . , K } . Specifically , we fix the encoder and learn a linear classification head g : Rm → RK on top from D̃l = { ( z , y ) |z = f ( x ) ∈ Rm } . Contrastive Pretraining . Taking a training example x ∈ Du , we draw its positive sample x+ = t ( x ) by applying a random data augmentation t ∼ T , and draw M randomly augmented samples { x−i } Mi=1 from Du as its negative samples . Then , we can learn the encoder f with the widely used InfoNCE loss ( Oord et al. , 2018 ) LNCE ( f ) =Ep ( x , x+ ) E { p ( x−i ) } [ − log exp ( f ( x ) > f ( x+ ) ) ∑M i=1 exp ( f ( x ) > f ( x−i ) ) ] =Ep ( x ) [ −Ep ( x+|x ) f ( x ) > f ( x+ ) + E { p ( x−i ) } log M∑ i=1 exp ( f ( x ) > f ( x−i ) ) ] . ( 1 ) Let p ( x ) be the data distribution , p ( x , x+ ) be the joint distribution of positive pairs , and we simply assume p ( x , x+ ) = p ( x+ , x ) and p ( x ) = ∫ p ( x , x+ ) dx+ , ∀ x ∈ Rd following Wang & Isola ( 2020 ) . Linear Evaluation . To evaluate the learned representations by contrastive learning , we usually adopt the Cross Entropy ( CE ) loss ( Chen et al. , 2020a ) for a labeled pair ( x , y ) ∈ Dl LCE ( f , g ) = Ep ( x , y ) [ − log exp ( f ( x ) > wy ) ∑K i=1 exp ( f ( x ) > wi ) ] , ( 2 ) with a linear classifier g ( z ) = Wz where W = [ w1 , w2 , . . . , wK ] . 3.2 EXISTING THEORETICAL ASSUMPTIONS AND THEIR LIMITATIONS . As discussed above , there are some previous understandings on how contrastive learning yields good performance , and they mainly differ by their theoretical assumptions . First , Wang & Isola ( 2020 ) interpret the first and second terms of the InfoNCE loss ( Eq . 1 ) as they are aiming at the following two properties : 1 ) alignment ( the 1st term ) : positive samples x , x+ has similar features , i.e. , f ( x ) ≈ f ( x+ ) ; 2 ) uniformity ( the 2nd term ) : features are roughly uniformly distributed in the unit hypersphere Sm−1 . In particular , they show that InfoNCE can be minimized with 1 ) perfect alignment and 2 ) perfect uniformity . However , as we illustrate in Figure 2 , the features could still have very poor downstream performance in the finite sample scenario . Proposition 3.1 ( Class-uniform features also minimize the InfoNCE loss ) . For N training examples of K classes , consider the case when features { f ( xi ) } Ni=1 are randomly distributed in Sm−1 with maximal uniformity ( i.e. , , minimizing the 2nd term of Eq . 1 ) while also satisfying ∀xi , x+i ∼ p ( x , x+ ) , f ( xi ) = f ( x + i ) . Because we have these two properties , the InfoNCE loss achieves its minimum . However , the downstream classification accuracy is at most 1/K + ε and ε is nearly zero when N is large enough . This indicates that the instance discrimination task ( alignment + uniformity ) alone can not guarantee the learning of class-discriminative features as desired in the downsteam classification . Instead , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) both establish the relationship between pretraining and classification objectives and provide guarantees for the downstream performance . In fact , the two works both assume the conditional independence of the two positive samples , i.e. , p ( x , x+|y ) = p ( x|y ) p ( x+|y ) . However , this assumption is too strong as it is hardly practical . As shown in Figure 1 ( b ) , augmented views from the same class are not actually independent as views from the same sample are more alike than that from other samples .

Summary: The paper proposes a new theory for understanding contrastive representation learning. The novelty is the focus on the interplay between alignment and augmentation. Prior work has identified alignment as one of the factors of contrastive learning, but have not investigated how different types of augmentations may affect the learned embeddings. This work adds that missing piece. The results intuitively make sense, showing that proper amount of augmentation (that connects samples of the same class) has positive effect on downstream classification. Empirically, the authors verify that too weak or too strong augmentation harms performance. Based on observations, the authors define a metric on ratio of positive pairs among nearest (embedding) neighbors, and found the change of this metric throughout training positively correlate with performance.