Chaos is a Ladder: A New Understanding of Contrastive Learning

1 INTRODUCTION . Contrastive Learning ( CL ) emerges to be a promising paradigm for learning data representations without labeled data ( Oord et al. , 2018 ; Hjelm et al. , 2019 ) . Recently , it has achieved impressive results and gradually closed the gap between supervised and unsupervised learning , hopefully leading to a new era that resolves the hunger for labeled data in the deep learning field ( He et al. , 2020 ; Chen et al. , 2020b ) . However , despite its intriguing empirical success , a theoretical understanding of how contrastive learning actually works in practice is still under-explored . The general methodology of contrastive learning is quite simple , that is to maximize the similarity between augmented views of the same image ( a.k.a . positive samples ) , and minimize the similarity between that of two random images ( a.k.a . negative samples ) . Intuitively , it is an instance discrimination task ( differing each image from others ) instead of a classification task ( clustering images from the same class together and differing with other classes ) . Nevertheless , as shown in Figure 1 ( a ) , CL representations are also class-separated . Therefore , understanding how the pretraining ( CL ) task and the downstream ( classification ) task interact plays a central role in both theoretical understandings and practical designings of contrastive methods . Previously , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) have tried to establish guarantees on the classification performance for self-supervised representations . However , their analysis relies heavily on the assumption that the two positive samples , as augmented views of the same image , are ( nearly ) conditionally independent on class y . However , this is hardly practical as the augmented views are still strongly input-dependent ( see Figure 1 ( b ) ) . In fact , if the conditional independence is satisfied , the unsupervised task will become as informative as the supervised task , making this discussion almost unnecessary . This motivates us to find more practical and weaker assumptions to understand how contrastive learning actually works ( even without conditional independence ) . To achieve this , we need to reexamine the contrastive learning process . Previously , Wang & Isola ( 2020 ) show that CL objective involves two goals : alignment ( for positive samples ) and uniformity ( for negative samples ) . Nevertheless , we show that there exist bad cases where the features could still have poor performance even with perfect alignment and uniformity . Thus , contradictory to the common belief of contrastive learning as learning invariance , we note that invariance alone is inadequate to learning useful representations for downstream tasks . In this paper , we provide a novel understanding of contrastive learning that requires only practical and minimal assumptions , while also guarantee class-separated representations . Our core insight hinges on the observation that contrastive learning usually adopts much more aggressive data augmentations than that in supervised learning ( He et al. , 2020 ; Chen et al. , 2020a ) . As shown in Figure 1 ( b ) , we notice that aggressive random cropping of two images can generate views that are very much alike that we could even hardly tell them apart , e.g. , the wheels of two different cars . In other words , there will be connections between different intra-class images through aggressively augmented views of them . Thus , the alignment of positive samples will also cluster all the intraclass samples together , and lead to class-separated representations . From our perspective , the role of data augmentation is to create a certain degree of “ chaos ” between intra-class samples , and the role of contrastive loss is to “ climb the ladder of chaos ” , i.e. , the process that we gradually cluster intra-class samples by aligning positive samples . Following this intuition , we develop a new theory for understanding the effectiveness of contrastive learning . Specifically , we derive the upper and lower bounds for its downstream performance and show how the two will asymptotically converge with our assumptions on the support overlapping effect of data augmentations . Driven by this analysis , we further discuss how varying augmentation will affect the performance of contrastive learning from both synthetic and real-world datasets , and show that the results align well with our theory . In summary , • We characterize the failure of the previous analysis of contrastive learning , and develop a new understanding through the support overlapping effect of data augmentations . Compared to existing theories on contrastive learning , ours can provide guidance to the practical designing of contrastive methods and evaluation metrics . • We establish general guarantees ( both upper and lower bounds ) for the downstream performance without assumptions on conditional independence . And we further show how the two bounds could asymptotically converge under our less restrictive assumptions . • We provide a quantitative discussion on the effect of augmentation strength , which verifies our theory from both theoretical and empirical aspects . Motivated by our theory , we further propose a new unsupervised evaluation metric for contrastive learning named ARC and show that it aligns well with downstream performance on real-world datasets . 2 RELATED WORK . Contrastive Learning in Practice . Contrastive self-supervised learning originates from a mutual information perspective of representation learning ( Oord et al. , 2018 ; Hjelm et al. , 2019 ) , and soon becomes a general learning paradigm that contrasts between positive and negative pairs ( He et al. , 2020 ; Chen et al. , 2020a ) . It is rapidly closing the performance gap between unsupervised and supervised learning on large-scale dataset like ImageNet ( Chen et al. , 2021 ) , and outperforms supervised learning when combined with a few ( e.g. , 10 % ) labels ( Chen et al. , 2020b ) . Several recent works show that similar performance could be achieved without negative samples by adopting certain training techniques ( Grill et al. , 2020 ; Chen & He , 2020 ) . Understanding Contrastive Learning Objectives . Both the original InfoNCE loss ( Oord et al. , 2018 ) and its InfoMax variants ( Hjelm et al. , 2019 ; Poole et al. , 2019 ) are designed as variational estimates of the mutual information between inputs and representations , but these estimators are shown to have poor bias-variance trade-offs ( Song & Ermon , 2020 ) . Instead , Wang & Isola ( 2020 ) simply understand contrastive learning through the two terms in the InfoNCE loss : alignment of positive samples and uniformity of negative samples . However , as we show later , this perspective is also insufficient to explain the effectiveness of contrastive learning , and we should take the interplay between augmentation and alignment into consideration . Understanding Downstream Generalization . Saunshi et al . ( 2019 ) propose the first theoretical guarantees by bridging the contrastive and classification objectives . Lee et al . ( 2020 ) further link the reconstruction-based objective to the downstream objective . However , both Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) rely on the unrealistic assumption that the positive samples are ( nearly ) conditionally independent . Besides , there are several other perspectives for analyzing contrastive learning , e.g. , information-theoretical perspective ( Tian et al. , 2020 ; Tsai et al. , 2021 ; Tosh et al. , 2020 ; 2021 ) . Recently , similar to our analysis , HaoChen et al . ( 2021 ) also study the augmentation graph and establish guarantees in terms of graph connectivity . Our work differs to theirs mainly in three aspects : 1 ) our analysis is applicable for the widely adopted InfoNCE and CE losses , while theirs is developed for their own spectral loss ; 2 ) ours starts from the alignment and uniformity perspective while theirs starts from the matrix decomposition perspective ; 3 ) our theory is empirically verified and inspires a useful evaluation metric for data augmentation , while their analysis focusing on minimizing the decomposition error is farther from the practical designing of positive and negative samples . In a nutshell , compared to previous discussions , our theory has a closer connection to the actual contrastive learning process , and we verify the feasibility of each assumption with empirical evidence . 3 LIMITATIONS OF PREVIOUS UNDERSTANDINGS . 3.1 PROBLEM SETUP . We begin by introducing the basic notations and common practice of contrastive learning in the image classification task . In general , it has two stages , unsupervised pretraining , and supervised finetuning . In the first stage , with N unlabeled samples Du = { xi } Ni=1 , we pretrain an encoder mapping from the d-dimensonal input space to a unit hypersphere f ∈ F : Rd → Sm−1 in the mdimensional space . In the second stage , we evaluate the learned representations z with the labeled data Dl = { ( xi , yi ) } where labels yi ∈ { 1 , . . . , K } . Specifically , we fix the encoder and learn a linear classification head g : Rm → RK on top from D̃l = { ( z , y ) |z = f ( x ) ∈ Rm } . Contrastive Pretraining . Taking a training example x ∈ Du , we draw its positive sample x+ = t ( x ) by applying a random data augmentation t ∼ T , and draw M randomly augmented samples { x−i } Mi=1 from Du as its negative samples . Then , we can learn the encoder f with the widely used InfoNCE loss ( Oord et al. , 2018 ) LNCE ( f ) =Ep ( x , x+ ) E { p ( x−i ) } [ − log exp ( f ( x ) > f ( x+ ) ) ∑M i=1 exp ( f ( x ) > f ( x−i ) ) ] =Ep ( x ) [ −Ep ( x+|x ) f ( x ) > f ( x+ ) + E { p ( x−i ) } log M∑ i=1 exp ( f ( x ) > f ( x−i ) ) ] . ( 1 ) Let p ( x ) be the data distribution , p ( x , x+ ) be the joint distribution of positive pairs , and we simply assume p ( x , x+ ) = p ( x+ , x ) and p ( x ) = ∫ p ( x , x+ ) dx+ , ∀ x ∈ Rd following Wang & Isola ( 2020 ) . Linear Evaluation . To evaluate the learned representations by contrastive learning , we usually adopt the Cross Entropy ( CE ) loss ( Chen et al. , 2020a ) for a labeled pair ( x , y ) ∈ Dl LCE ( f , g ) = Ep ( x , y ) [ − log exp ( f ( x ) > wy ) ∑K i=1 exp ( f ( x ) > wi ) ] , ( 2 ) with a linear classifier g ( z ) = Wz where W = [ w1 , w2 , . . . , wK ] . 3.2 EXISTING THEORETICAL ASSUMPTIONS AND THEIR LIMITATIONS . As discussed above , there are some previous understandings on how contrastive learning yields good performance , and they mainly differ by their theoretical assumptions . First , Wang & Isola ( 2020 ) interpret the first and second terms of the InfoNCE loss ( Eq . 1 ) as they are aiming at the following two properties : 1 ) alignment ( the 1st term ) : positive samples x , x+ has similar features , i.e. , f ( x ) ≈ f ( x+ ) ; 2 ) uniformity ( the 2nd term ) : features are roughly uniformly distributed in the unit hypersphere Sm−1 . In particular , they show that InfoNCE can be minimized with 1 ) perfect alignment and 2 ) perfect uniformity . However , as we illustrate in Figure 2 , the features could still have very poor downstream performance in the finite sample scenario . Proposition 3.1 ( Class-uniform features also minimize the InfoNCE loss ) . For N training examples of K classes , consider the case when features { f ( xi ) } Ni=1 are randomly distributed in Sm−1 with maximal uniformity ( i.e. , , minimizing the 2nd term of Eq . 1 ) while also satisfying ∀xi , x+i ∼ p ( x , x+ ) , f ( xi ) = f ( x + i ) . Because we have these two properties , the InfoNCE loss achieves its minimum . However , the downstream classification accuracy is at most 1/K + ε and ε is nearly zero when N is large enough . This indicates that the instance discrimination task ( alignment + uniformity ) alone can not guarantee the learning of class-discriminative features as desired in the downsteam classification . Instead , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) both establish the relationship between pretraining and classification objectives and provide guarantees for the downstream performance . In fact , the two works both assume the conditional independence of the two positive samples , i.e. , p ( x , x+|y ) = p ( x|y ) p ( x+|y ) . However , this assumption is too strong as it is hardly practical . As shown in Figure 1 ( b ) , augmented views from the same class are not actually independent as views from the same sample are more alike than that from other samples .

Summary: The authors provided a new understanding of contrastive learning from the perspective of data augmentation for intra-class samples. In particular, the authors proposed to understand the role of data augmentation as to create certain ``chaos'' between intra-class samples so to encourage the clustering of intra-class samples and also the learning of class-separated representations. Additionally, a new metric ARC is proposed to evaluate the downstream performance. The conclusion is validated via both synthetic and real-world datasets.