The Power of Contrast for Feature Learning: A Theoretical Analysis

1 INTRODUCTION . Deep supervised learning has achieved great success in various applications , including computer vision ( Krizhevsky et al. , 2012 ) , natural language processing ( Devlin et al. , 2018 ) , and scientific computing ( Han et al. , 2018 ) . However , its dependence on manually assigned labels , which is usually difficult and costly , has motivated research into alternative approaches to exploit unlabeled data . Self-supervised learning is a promising approach that leverages the unlabeled data itself as supervision and learns representations that are beneficial to potential downstream tasks . At a high level , there are two common approaches for feature extraction in self-supervised learning : generative and contrastive ( Liu et al. , 2021 ) . Both approaches aim to learn latent representations of the original data , while the difference is that the generative approach focused on minimizing the reconstruction error from latent representations , and the contrastive approach targets to decrease the similarity between the representations of contrastive pairs . Recent works have shown the benefits of contrastive learning in practice ( Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ; c ) . However , why the contrastive approach outperforms the generative approach remains mysterious . Additionally , recent works aim to further improve contrastive learning by introducing the label information . Specifically , Khosla et al . ( 2020 ) proposed the supervised contrastive learning , where the contrasting procedures are performed across different classes rather than different instances . With the help of label information , their proposed method outperforms self-supervised contrastive learning and classical cross entropy based supervised learning . However , despite this improvement on in-domain downstream tasks , Islam et al . ( 2021 ) found that such improvement in transfer learning is limited and even negative for such supervised contrastive learning . This phenomenon motivates us to rethink the role of labeled data in the contrastive learning framework . In this paper , we first compare contrastive learning with a representative method in the generative approach – the autoencoders . Specifically , we initialize the investigation in the linear representation setting , which has been widely adopted in theory to shed light upon complex machine learning phenomena such as in Du et al . ( 2020 ) ; Tripuraneni et al . ( 2021 ) . We provide a theoretical analysis of their feature learning performances on the spiked covariance model ( Bai & Yao , 2012 ; Yao et al. , 2015 ; Zhang et al. , 2018 ) and theoretically justify why contrastive learning outperforms autoencoders—contrastive learning is able to remove more noises by constructing contrastive samples . Then we investigate the role of label information in the contrastive learning framework and provide a theoretical justification of why labeled data help to gain accuracy in same-domain classification while can hurt multi-task transfer learning . Related works The idea of contrastive learning was first proposed in Hadsell et al . ( 2006 ) as an effective method to perform dimension reduction . Following this line of research , Dosovitskiy et al . ( 2014 ) proposed to perform instance discrimination by creating surrogate classes for each instance and Wu et al . ( 2018 ) further proposed to preserve a memory bank as a dictionary of negative samples . Other extensions based on this memory bank approach include He et al . ( 2020 ) ; Misra & Maaten ( 2020 ) ; Tian et al . ( 2020 ) ; Chen et al . ( 2020c ) . Rather than keeping a costly memory bank , another line of works exploits the benefit of mini-batch training where different samples are treated as negative to each other Ye et al . ( 2019 ) ; Chen et al . ( 2020a ) . Moreover , Khosla et al . ( 2020 ) explores the supervised version of contrastive learning where pairs are generated based on label information . Despite its success in practice , theoretical understanding of contrastive learning is still limited . Previous works provide provable guarantees for contrastive learning under conditional independence assumption ( or its variants ) ( Arora et al. , 2019 ; Lee et al. , 2020 ; Tosh et al. , 2021 ; Tsai et al. , 2020 ) . Specifically , they assume the two contrastive views are independent conditioned on the label and show that contrastive learning can provably learn representations beneficial for downstream tasks . In addition to this line of research , Wang & Isola ( 2020 ) ; Graf et al . ( 2021 ) investigated the representation geometry of supervised contrastive loss , and HaoChen et al . ( 2021 ) provided analysis via a novel concept of augmentation graph with a new loss function that performs spectral decomposition on such graph . Moreover , Wen & Li ( 2021 ) considered the representation learning under the sparse coding model and studied the optimization properties on shallow ReLU neural networks . Different from all previous works , which aim to show that contrastive learning can learn useful representation , our paper aim to explain why contrastive learning outperforms other representation learning methods and also shed light on the role of labeled data in contrastive learning framework , which is under-explored in prior works . 2 PRELIMINARIES . Notations In this paper , we use O , Ω , Θ to hide universal constants and we write ak . bk for two sequences of positive numbers { ak } and { bk } if and only if there exists an universal constant C > 0 such that ak < Cbk for any k. We use ‖ · ‖ , ‖ · ‖2 , ‖ · ‖F to represent the ` 2 norm of vectors , spectral norm of matrices and Frobenius norm of matrices respectively . Let Od , r be a set of d × r orthogonal matrices . i.e. , Od , r , { U ∈ Rd×r : U > U = Ir } . We use |A| to denote the cardinality of a set A . For any n ∈ N+ , let [ n ] = { 1 , 2 , · · · , n } . We use ‖ sin Θ ( U1 , U2 ) ‖F to refer to the sine distance between two orthogonal matrices U1 , U2 ∈ Od , r , which is defined by : ‖sin Θ ( U1 , U2 ) ‖F , ∥∥U > 1⊥U2∥∥F . More properties of sine distance can be found in Section A.1 . We use { ei } di=1 to denote the canonical basis in d-dimensional Euclidean space Rd , that is , ei is the vector whose i-th coordinate is 1 and all the other coordinates are 0 . Let I { A } be an indicator function that takes 1 when A is true , otherwise takes 0 . We write a∨ b and a∧ b to denote max ( a , b ) and min ( a , b ) , respectively . 2.1 SETUP . Given an input x ∈ Rd , contrastive learning aims to learn a low dimensional representation h = f ( x ; θ ) ∈ Rr by contrasting different samples , i.e. , maximizing the agreement between positive pairs , and minimizing the agreement between negative pairs . Suppose we have n data points X = [ x1 , x2 , · · · , xn ] ∈ Rd×n from the population distribution D. The contrastive learning task can be formulated to be an optimization problem : min θ L ( θ ) = min θ 1 n n∑ i=1 ` ( xi , BPosi , B Neg i ; f ( · , θ ) ) + λR ( θ ) , ( 2.1 ) where ` ( · ) is a contrastive loss and λR ( θ ) is a regularization term ; BPosi , B Neg i are the sets of positive samples and negative samples corresponding to xi , which we will describe in detail below . Losses and Models . We then present the model setup considered in this paper . ( a ) . Linear representation and regularization term . We consider the linear representation function f ( x , W ) = Wx , where the parameter θ is a matrix W ∈ Rr×d . Since regularization techniques have been widely adapted in contrastive learning practice ( Chen et al. , 2020a ; He et al. , 2020 ; Grill et al. , 2020 ) , we further consider penalizing the representation by a regularization term R ( W ) = ‖WW > ‖2F /2 to encourage the orthogonality of W and therefore promote the diversity of wi to learn different representations . ( b ) . Triplet Contrastive loss . The contrastive loss is set to be the average similarity between positive pairs minus that between negative pairs : ` ( x , BPos , BNeg , f ( · , θ ) ) = − ∑ xPos∈BPos 〈f ( x , θ ) , f ( xPos , θ ) 〉 |BPos| + ∑ xNeg∈BNeg 〈f ( x , θ ) , f ( xNeg , θ ) 〉 |BNeg| , ( 2.2 ) where BPos , Bneg are sets of positive samples and negative samples corresponding to x . This loss has been commonly used in constrastive learning ( Hadsell et al. , 2006 ) and metric learning ( Schroff et al. , 2015 ; He et al. , 2018 ) . In Khosla et al . ( 2020 ) , the authors show that it is an approximation of the NT-Xent contrastive loss , which has been highlighted in recent contrastive learning practice ( Sohn , 2016 ; Wu et al. , 2018 ; Oord et al. , 2018 ; Chen et al. , 2020a ) . ( c ) . Generation of positive and negative pairs . There are two common approaches to generate such pairs , depending on whether or not label information is available . When the label information is not available , the typical strategy is to generate different views of the original data via augmentation ( Hadsell et al. , 2006 ; Chen et al. , 2020a ) . Two views of the same data point serve as positive pair for each other while those of different data serve as negative pairs . Definition 2.1 ( Augmented pairs generation ) . Given two augmentation functions g1 , g2 : Rd → Rd and n training samples B = { xi } i∈ [ n ] , the augmented views are given by : { ( g1 ( xi ) , g2 ( xi ) ) } i∈ [ n ] . Then for each view gv ( xi ) , v = 1 , 2 , the corresponding positive samples and negative samples are defined by : BPosi , v = { gs ( xi ) : s ∈ [ 2 ] \ { v } } and B Neg i , v = { gs ( xj ) : s ∈ [ 2 ] , j ∈ [ n ] \ { i } } . The loss function of self-supervised contrastive learning problem can be written as : LSelfCon ( W ) = − 1 2n n∑ i=1 2∑ v=1 [ 〈Wgv ( xi ) , Wg [ 2 ] \ { v } ( xi ) 〉− ∑ j 6=i 2∑ s=1 〈Wgv ( xi ) , Wgs ( xj ) 〉 2n− 2 ] + λ 2 ‖WW > ‖2F . ( 2.3 ) In particular , we adopt the following augmentation in our analysis . Definition 2.2 ( Random masking augmentation ) . The two views of the original data are generated by randomly dividing its dimensions to two sets , that is , g1 ( xi ) = Axi , and g2 ( xi ) = ( I − A ) xi , where A = diag ( a1 , · · · , ad ) ∈ Rd×d is the diagonal masking matrix with { ai } di=1 being i.i.d . random variables sampled from a Bernoulli distribution with mean 1/2 . A similar augmentation was considered in Wen & Li ( 2021 ) . However , our primary interest lies in comparing the performance of contrastive learning against autoencoders and analyzing the role of labeled data , while their work focuses on understanding the training process of neural networks in contrastive learning . When the label information is available , Khosla et al . ( 2020 ) proposed the following approach to generate pairs . Definition 2.3 ( Supervised pairs generation ) . In a K-class classification problem , given nk samples for each class k ∈ [ K ] : { xki : i ∈ [ nk ] } Kk=1 and let n = ∑K k=1 nk , the corresponding positive samples and negative samples for xki are defined by BPosi , k = { xkj : j ∈ [ nk ] \ i } and B Neg i , k = { xsj : s ∈ [ K ] \ k , j ∈ [ ns ] } . That is , the positive samples are the remaining ones in the same class with xki and the negative samples are the samples from different classes . Correspondingly , the loss function of the supervised contrastive learning problem can be written as : LSupCon ( W ) = − 1 nK K∑ k=1 n∑ i=1 [ ∑ j 6=i 〈Wxki , Wxkj 〉 n− 1 − n∑ j=1 ∑ s6=k 〈Wxki , Wxsj〉 n ( K − 1 ) ] + λ 2 ‖WW > ‖2F . ( 2.4 ) ( d ) . Spiked Covariance Model . We consider the following spiked covariance model ( Bai & Yao , 2012 ; Yao et al. , 2015 ; Zhang et al. , 2018 ) to study the power of contrastive learning : x = U ? z + ξ , Cov ( z ) = ν2Ir , Cov ( ξ ) = Σ , ( 2.5 ) where z ∈ Rr and ξ ∈ Rd are both zero mean sub-Gaussian random variables . In particular , U ? ∈ Od , r and Σ = diag ( σ21 , · · · , σ2d ) . The first term U ? z represents the signal of interest residing in a low-dimensional subspace spanned by the columns of U ? . The second term ξ is the dense noise with heteroskedastic noise . Given that , the ideal low-dimensional representation is to compress the observed x into a low-dimensional representation spanned by the columns of U ? . In this paper , we aim to learn a good projection W ∈ Rr×d onto a lower-dimensional subspace from observation x . Since the information of W is invariant with the transformation W ← OW for any invertible matrix O ∈ Rr , r , the essential information of W is contained in the right eigenvector of W . Thus we quantify the goodness of the representation W by the sine distance ‖ sin Θ ( U , U ? ) ‖F , where U is the top-r right eigenspace of W .

Summary: This paper performs a theoretical study of various empirical phenomena regarding contrastive learning in a simplified setting, with a linear representation function, spiked covariance data model and linearized version of contrastive loss. In such a setting, the following results are shown theoretically (a) contrastive learning with a particular data augmentation can learn much better feature than an autoencoder by virtue of learning the underlying low rank signal as features, (b) supervised contrastive learning can do better than contrastive learning by getting rid some bias that data augmentation introduces, (c) for transfer learning, a combination of unlabeled contrastive loss and supervised learning can do better than either of them individually. Simulation experiments are used to verify many of these findings.