The Power of Contrast for Feature Learning: A Theoretical Analysis

1 INTRODUCTION . Deep supervised learning has achieved great success in various applications , including computer vision ( Krizhevsky et al. , 2012 ) , natural language processing ( Devlin et al. , 2018 ) , and scientific computing ( Han et al. , 2018 ) . However , its dependence on manually assigned labels , which is usually difficult and costly , has motivated research into alternative approaches to exploit unlabeled data . Self-supervised learning is a promising approach that leverages the unlabeled data itself as supervision and learns representations that are beneficial to potential downstream tasks . At a high level , there are two common approaches for feature extraction in self-supervised learning : generative and contrastive ( Liu et al. , 2021 ) . Both approaches aim to learn latent representations of the original data , while the difference is that the generative approach focused on minimizing the reconstruction error from latent representations , and the contrastive approach targets to decrease the similarity between the representations of contrastive pairs . Recent works have shown the benefits of contrastive learning in practice ( Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ; c ) . However , why the contrastive approach outperforms the generative approach remains mysterious . Additionally , recent works aim to further improve contrastive learning by introducing the label information . Specifically , Khosla et al . ( 2020 ) proposed the supervised contrastive learning , where the contrasting procedures are performed across different classes rather than different instances . With the help of label information , their proposed method outperforms self-supervised contrastive learning and classical cross entropy based supervised learning . However , despite this improvement on in-domain downstream tasks , Islam et al . ( 2021 ) found that such improvement in transfer learning is limited and even negative for such supervised contrastive learning . This phenomenon motivates us to rethink the role of labeled data in the contrastive learning framework . In this paper , we first compare contrastive learning with a representative method in the generative approach – the autoencoders . Specifically , we initialize the investigation in the linear representation setting , which has been widely adopted in theory to shed light upon complex machine learning phenomena such as in Du et al . ( 2020 ) ; Tripuraneni et al . ( 2021 ) . We provide a theoretical analysis of their feature learning performances on the spiked covariance model ( Bai & Yao , 2012 ; Yao et al. , 2015 ; Zhang et al. , 2018 ) and theoretically justify why contrastive learning outperforms autoencoders—contrastive learning is able to remove more noises by constructing contrastive samples . Then we investigate the role of label information in the contrastive learning framework and provide a theoretical justification of why labeled data help to gain accuracy in same-domain classification while can hurt multi-task transfer learning . Related works The idea of contrastive learning was first proposed in Hadsell et al . ( 2006 ) as an effective method to perform dimension reduction . Following this line of research , Dosovitskiy et al . ( 2014 ) proposed to perform instance discrimination by creating surrogate classes for each instance and Wu et al . ( 2018 ) further proposed to preserve a memory bank as a dictionary of negative samples . Other extensions based on this memory bank approach include He et al . ( 2020 ) ; Misra & Maaten ( 2020 ) ; Tian et al . ( 2020 ) ; Chen et al . ( 2020c ) . Rather than keeping a costly memory bank , another line of works exploits the benefit of mini-batch training where different samples are treated as negative to each other Ye et al . ( 2019 ) ; Chen et al . ( 2020a ) . Moreover , Khosla et al . ( 2020 ) explores the supervised version of contrastive learning where pairs are generated based on label information . Despite its success in practice , theoretical understanding of contrastive learning is still limited . Previous works provide provable guarantees for contrastive learning under conditional independence assumption ( or its variants ) ( Arora et al. , 2019 ; Lee et al. , 2020 ; Tosh et al. , 2021 ; Tsai et al. , 2020 ) . Specifically , they assume the two contrastive views are independent conditioned on the label and show that contrastive learning can provably learn representations beneficial for downstream tasks . In addition to this line of research , Wang & Isola ( 2020 ) ; Graf et al . ( 2021 ) investigated the representation geometry of supervised contrastive loss , and HaoChen et al . ( 2021 ) provided analysis via a novel concept of augmentation graph with a new loss function that performs spectral decomposition on such graph . Moreover , Wen & Li ( 2021 ) considered the representation learning under the sparse coding model and studied the optimization properties on shallow ReLU neural networks . Different from all previous works , which aim to show that contrastive learning can learn useful representation , our paper aim to explain why contrastive learning outperforms other representation learning methods and also shed light on the role of labeled data in contrastive learning framework , which is under-explored in prior works . 2 PRELIMINARIES . Notations In this paper , we use O , Ω , Θ to hide universal constants and we write ak . bk for two sequences of positive numbers { ak } and { bk } if and only if there exists an universal constant C > 0 such that ak < Cbk for any k. We use ‖ · ‖ , ‖ · ‖2 , ‖ · ‖F to represent the ` 2 norm of vectors , spectral norm of matrices and Frobenius norm of matrices respectively . Let Od , r be a set of d × r orthogonal matrices . i.e. , Od , r , { U ∈ Rd×r : U > U = Ir } . We use |A| to denote the cardinality of a set A . For any n ∈ N+ , let [ n ] = { 1 , 2 , · · · , n } . We use ‖ sin Θ ( U1 , U2 ) ‖F to refer to the sine distance between two orthogonal matrices U1 , U2 ∈ Od , r , which is defined by : ‖sin Θ ( U1 , U2 ) ‖F , ∥∥U > 1⊥U2∥∥F . More properties of sine distance can be found in Section A.1 . We use { ei } di=1 to denote the canonical basis in d-dimensional Euclidean space Rd , that is , ei is the vector whose i-th coordinate is 1 and all the other coordinates are 0 . Let I { A } be an indicator function that takes 1 when A is true , otherwise takes 0 . We write a∨ b and a∧ b to denote max ( a , b ) and min ( a , b ) , respectively . 2.1 SETUP . Given an input x ∈ Rd , contrastive learning aims to learn a low dimensional representation h = f ( x ; θ ) ∈ Rr by contrasting different samples , i.e. , maximizing the agreement between positive pairs , and minimizing the agreement between negative pairs . Suppose we have n data points X = [ x1 , x2 , · · · , xn ] ∈ Rd×n from the population distribution D. The contrastive learning task can be formulated to be an optimization problem : min θ L ( θ ) = min θ 1 n n∑ i=1 ` ( xi , BPosi , B Neg i ; f ( · , θ ) ) + λR ( θ ) , ( 2.1 ) where ` ( · ) is a contrastive loss and λR ( θ ) is a regularization term ; BPosi , B Neg i are the sets of positive samples and negative samples corresponding to xi , which we will describe in detail below . Losses and Models . We then present the model setup considered in this paper . ( a ) . Linear representation and regularization term . We consider the linear representation function f ( x , W ) = Wx , where the parameter θ is a matrix W ∈ Rr×d . Since regularization techniques have been widely adapted in contrastive learning practice ( Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ) , we further consider penalizing the representation by a regularization term R ( W ) = ‖WW > ‖2F /2 to encourage the orthogonality of W and therefore promote the diversity of wi to learn different representations . ( b ) . Triplet Contrastive loss . The contrastive loss is set to be the average similarity between positive pairs minus that between negative pairs : ` ( x , BPos , BNeg , f ( · , θ ) ) = − ∑ xPos∈BPos 〈f ( x , θ ) , f ( xPos , θ ) 〉 |BPos| + ∑ xNeg∈BNeg 〈f ( x , θ ) , f ( xNeg , θ ) 〉 |BNeg| , ( 2.2 ) where BPos , Bneg are sets of positive samples and negative samples corresponding to x . This loss has been commonly used in constrastive learning ( Hadsell et al. , 2006 ) and metric learning ( Schroff et al. , 2015 ; He et al. , 2018 ) . In Khosla et al . ( 2020 ) , the authors show that it is an approximation of the NT-Xent contrastive loss , which has been highlighted in recent contrastive learning practice ( Sohn , 2016 ; Wu et al. , 2018 ; Oord et al. , 2018 ; Chen et al. , 2020a ) . ( c ) . Generation of positive and negative pairs . There are two common approaches to generate such pairs , depending on whether or not label information is available . When the label information is not available , the typical strategy is to generate different views of the original data via augmentation ( Hadsell et al. , 2006 ; Chen et al. , 2020a ) . Two views of the same data point serve as positive pair for each other while those of different data serve as negative pairs . Definition 2.1 ( Augmented pairs generation ) . Given two augmentation functions g1 , g2 : Rd → Rd and n training samples B = { xi } i∈ [ n ] , the augmented views are given by : { ( g1 ( xi ) , g2 ( xi ) ) } i∈ [ n ] . Then for each view gv ( xi ) , v = 1 , 2 , the corresponding positive samples and negative samples are defined by : BPosi , v = { gs ( xi ) : s ∈ [ 2 ] \ { v } } and B Neg i , v = { gs ( xj ) : s ∈ [ 2 ] , j ∈ [ n ] \ { i } } . The loss function of self-supervised contrastive learning problem can be written as : LSelfCon ( W ) = − 1 2n n∑ i=1 2∑ v=1 [ 〈Wgv ( xi ) , Wg [ 2 ] \ { v } ( xi ) 〉− ∑ j 6=i 2∑ s=1 〈Wgv ( xi ) , Wgs ( xj ) 〉 2n− 2 ] + λ 2 ‖WW > ‖2F . ( 2.3 ) In particular , we adopt the following augmentation in our analysis . Definition 2.2 ( Random masking augmentation ) . The two views of the original data are generated by randomly dividing its dimensions to two sets , that is , g1 ( xi ) = Axi , and g2 ( xi ) = ( I − A ) xi , where A = diag ( a1 , · · · , ad ) ∈ Rd×d is the diagonal masking matrix with { ai } di=1 being i.i.d . random variables sampled from a Bernoulli distribution with mean 1/2 . A similar augmentation was considered in Wen & Li ( 2021 ) . However , our primary interest lies in comparing the performance of contrastive learning against autoencoders and analyzing the role of labeled data , while their work focuses on understanding the training process of neural networks in contrastive learning . When the label information is available , Khosla et al . ( 2020 ) proposed the following approach to generate pairs . Definition 2.3 ( Supervised pairs generation ) . In a K-class classification problem , given nk samples for each class k ∈ [ K ] : { xki : i ∈ [ nk ] } Kk=1 and let n = ∑K k=1 nk , the corresponding positive samples and negative samples for xki are defined by BPosi , k = { xkj : j ∈ [ nk ] \ i } and B Neg i , k = { xsj : s ∈ [ K ] \ k , j ∈ [ ns ] } . That is , the positive samples are the remaining ones in the same class with xki and the negative samples are the samples from different classes . Correspondingly , the loss function of the supervised contrastive learning problem can be written as : LSupCon ( W ) = − 1 nK K∑ k=1 n∑ i=1 [ ∑ j 6=i 〈Wxki , Wxkj 〉 n− 1 − n∑ j=1 ∑ s6=k 〈Wxki , Wxsj〉 n ( K − 1 ) ] + λ 2 ‖WW > ‖2F . ( 2.4 ) ( d ) . Spiked Covariance Model . We consider the following spiked covariance model ( Bai & Yao , 2012 ; Yao et al. , 2015 ; Zhang et al. , 2018 ) to study the power of contrastive learning : x = U ? z + ξ , Cov ( z ) = ν2Ir , Cov ( ξ ) = Σ , ( 2.5 ) where z ∈ Rr and ξ ∈ Rd are both zero mean sub-Gaussian random variables . In particular , U ? ∈ Od , r and Σ = diag ( σ21 , · · · , σ2d ) . The first term U ? z represents the signal of interest residing in a low-dimensional subspace spanned by the columns of U ? . The second term ξ is the dense noise with heteroskedastic noise . Given that , the ideal low-dimensional representation is to compress the observed x into a low-dimensional representation spanned by the columns of U ? . In this paper , we aim to learn a good projection W ∈ Rr×d onto a lower-dimensional subspace from observation x . Since the information of W is invariant with the transformation W ← OW for any invertible matrix O ∈ Rr , r , the essential information of W is contained in the right eigenvector of W . Thus we quantify the goodness of the representation W by the sine distance ‖ sin Θ ( U , U ? ) ‖F , where U is the top-r right eigenspace of W .

Summary: This paper presents theoretical results to explain the effectiveness of contrastive learning (CL). For the self-supervised model, the authors prove new sin-distance-based error bounds for autoencoder (AE) and CL. For the fully-supervised model, the generalization error bound is derived. Numerical experiments validate the theoretical finds.