Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning

1 INTRODUCTION . One of the current main bottlenecks in deep network training is the dependence on heavy annotated training data , and this motivates the recent surge of interests in unsupervised ( Donahue & Simonyan , 2019 ) and self-supervised ( Chen & He , 2021 ; Chen et al. , 2020 ) methods . Specifically , in self-supervised representation learning ( SSL ) , a network is pre-trained without any form of manual annotation , thus providing a means to extract information from unlabeled data sources ( e.g. , text corpora , videos , images from the Internet , etc. ) . In self-supervision , label-based information is replaced by a prediction problem using a certain context or using a pretext task . Pretext task in SSL can mainly be divided into three categories : 1 ) Generative based approaches ( Donahue & Simonyan , 2019 ) learn to generate or otherwise model pixels in the input space . However , pixel-level generation is computationally expensive and may not be necessary for representation learning . 2 ) Contextual based methods ( Vincent et al. , 2008 ; Pathak et al. , 2016 ; Ye et al. , 2019 ) design pretext tasks ( denoising auto-encoders ( Vincent et al. , 2008 ) , context auto encoders ( Zhang et al. , 2016 ; 2017 ) , etc ) . 3 ) Contrastive based methods ( Chen et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Asano et al. , 2019 ) take augmented views of the same image as positive pair and others as negative pairs . Generally , one positive sample corresponds to lots of negative samples . In recent works , contrastive based methods have shown great promise , achieving state-of-the-art results in image classification ( Chen et al. , 2020 ) , video classification ( Han et al. , 2020 ) and other downstream tasks ( Chen & He , 2021 ) . However , trivial constant solutions ( different samples get the same representation ) is easily happening without the proper design of architecture and objective function . The well-known solutions to avoid this problem can be summarized into two parts : asymmetric model architecture and proper objective function . 1 ) Model architecture : MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) update encoders separately and stopping gradient operation is adopted to avoid such problem . Then , BYOL and SimSiam ( Chen & He , 2021 ) introduce a predictor module to avoid collapse , which is composed by MLP ( Goodfellow et al. , 2016 ) . Current mainstream interpretation is using the predic- tor to construct an asymmetric structure , which is useful to alleviate trivial solutions . 2 ) Objective function : SimCLR uses symmetric framework in contrastive learning . They prevent trivial solutions by using negative pairs and InfoNCE , where InfoNCE can be divided into an alignment term and a uniformity term ( Arora et al. , 2019 ) . The uniformity term pulls different samples to a hyper-sphere uniformly , forcing obtaining different representations and avoiding trivial solutions . Recently , Barlow Twins ( Zbontar et al. , 2021 ) designs a new objective function from the information redundancy perspective , which also has two terms ( an invariance term and a redundancy reduction term ) . The invariance term maximizes the correlation of the same feature across different views , and the redundancy term reduces information redundancy . However , for the symmetric framework , both SimCLR and Barlow Twins require square order complexity in objective functions , and the main complexity comes from the uniformity and redundancy term . In this paper , we propose two methods , named Zero-ICL and Zero-FCL , where Zero-ICL discards the uniformity term and only requires O ( N ) complexity by instance-wise whitening . Correspondingly , Zero-FCL discards the redundancy term by feature-wise whitening and only requires O ( d ) complexity . Our contributions are : 1 ) We propose a new contrastive learning framework to prevent trivial solutions , Zero-CL , which includes two parts , i.e. , Zero-ICL ( instance-wise ) and Zero-FCL ( feature-wise ) , either of which can work independently and only requires linear order complexity ( objective function ) . 2 ) To our best knowledge , Zero-ICL is the first attempt of instance-wise whitening , which is conceptually comprehensible for preventing collapses in contrastive learning . Note that most previous methods ( including other domains beyond vision ) e.g . ( Eldar & Oppenheim , 2003 ; Kessy et al. , 2018 ) only use whitening transformation to reduce the information redundancy on feature-wise . 3 ) We give empirical analysis on the relationship between previous methods and our Zero-CL , where previous negative sample consuming methods ( Zbontar et al. , 2021 ; Chen et al. , 2020 ) can be regarded as our method with Lagrangian transformation . Then , we theoretically introduce ZCA-based whitening from the maximal correlation ( Kessy et al. , 2018 ) perspective . 4 ) Experimental results on standard image benchmarks ( CIFAR-10/100 and ImageNet-100/1k ) show our method achieves new state-of-the-art results for symmetric contrastive learning compared with ( Chen et al. , 2020 ; Zbontar et al. , 2021 ) , especially for small hidden dimension and batch size . 2 RELATED WORK . We review recent contrastive learning works . We mainly divide the previous contrastive methods into feature-wise and instance-wise according to different contrastive dimensions . We particularly discuss the similarity and differences of some most related methods in Section 5 in detail . instance-wise contrastive learning . instance-wise contrastive learning aims to attract positive pairs and repulse negative pairs ( Chen et al. , 2020 ; He et al. , 2020 ) , where each pair is composed of two instance views . One of the widely used objective functions in instance-wise learning is InfoNCE , and extensive experiments in ( Oord et al. , 2018 ) show that InfoNCE requires negative pairs to avoid trivial solutions . SimCLR ( Chen et al. , 2020 ) regards views augmented from different images in a mini-batch as negative pairs . However , SimCLR requires a large batch size to improve the accuracy , which is GPU intensive . To address this issue , MoCo ( He et al. , 2020 ) proposes storing negative samples in the memory bank and updates the bank by the first-in-first-out principle . MoCo further proposes two operations to prevent trivial solutions , i.e. , momentum update key encoder and stop gradient . In recent works ( Chen et al. , 2020 ; Grill et al. , 2020 ) , the mainstream explanation of the two operations is they construct an asymmetric framework , which can avoid trivial solutions . Inspired by this , BYOL ( Grill et al. , 2020 ) adopts stop gradient operation and EMA algorithm to update target encoder . Further , BYOL introduces a predictor module to build a more asymmetric framework . Then , SimSiam ( Chen & He , 2021 ) empirically shows that stop gradient and the predictor is the key component to avoid trivial solutions . Another approach to preventing such solutions is introduced in W-MSE ( Ermolov et al. , 2021 ) . Different from previous works ( Chen et al. , 2020 ; Chen & He , 2021 ; He et al. , 2020 ) , W-MSE requires multiple augmented views to improve the accuracy . Besides , WMSE simply borrows Cholesky-based whitening transformation in SSL and the multi-view objective requires O ( NM2 ) complexity , where M is the number of views . feature-wise contrastive learning . Different from instance-wise learning , feature-wise methods contrast on feature dimension . Barlow Twins ( Zbontar et al. , 2021 ) is the first feature-wise method , which pushes the cross-correlation matrix to identity matrix I . Its objective includes both invariance term and redundancy reduction term , where the latter term can avoid trivial solutions . Inspired by Barlow Twins , VICReg ( Bardes et al. , 2021 ) proposes a new regularization term named invariancecovariance on feature dimension , and achieves similar results to Barlow Twins . However , both Barlow Twins and VICReg require large contrastive dimensions and the covariance regularization requires O ( d2 ) complexity , which could be also regarded as requiring negative pairs on feature dimension ( each negative pair is composed of features in different dimensions across images ) . Whitening transformation . Whitening , or sphering , is a common pre-processing step to transform random variables to orthogonality . However , due to rotational freedom , there are infinitely many possible whitening procedures . There is a diverse range of sphering methods , e.g . principal component analysis ( PCA ) ( Jégou & Chum , 2012 ) , Cholesky matrix decomposition ( Higham , 1990 ) and zero-phase component analysis ( ZCA ) ( Bell & Sejnowski , 1997 ) . Compared with other whitening methods , ZCA remains maximal correlation of original data , which we will clarify in Sec . 3.6 . 3 THE PROPOSED ZERO-CL . In this section , we present the framework of Zero-CL in detail , starting with the whitening transformation procedure , followed by the model ’ s framework as well as the objective function . At last , we provide theoretical analysis on whitening from maximal correlation perspective . 3.1 WHITENING TRANSFORMATION . Given a set of images { x } Ni=1 , we extract an embedding in contrastive space zi = g ( f ( xi , θ ) , γ ) using an encoder network ( He et al. , 2016 ) f ( · , θ ) and a MLP module g ( · , γ ) , where zi ∈ R1×d . Denote the set of embeddings as Z ∈ RN×d , whitening transformation can be formulated as : H = ( h1 , h2 , · · · , hN ) = WZ > ( 1 ) where H ∈ Rd×N is the whitened embedding matrix . The square matrix W ∈ Rd×d is the socalled whitening matrix . Since var ( H ) = I , it follows that WΣW = I and thus W ( ΣW > W ) = W , which is fulfilled if W satisfies W > W = Σ−1 . However , the constraint does not uniquely determine the whitening matrix W. Specifically , for any Wrot = RW with orthogonal matrix , R will also be a whitening transformation matrix . Here we present ZCA whitening matrix in detail . ZCA-based whitening takes E ( stacks together eigenvectors of the covariance matrix ) as the orthogonal matrix R. Then ZCA whitening matrix can be written as : WZCA = EΛ−1/2E > ( 2 ) By rotation matrix E , the whitened data will be as close as possible to original data ( see Theorem 1 ) . Note that in low-rank conditions , i.e. , some eigenvalues equal to 0 , a common approach is replacing Λ−1/2 with ( Λ + λI ) −1/2 to prevent zero division , where λ = 1e− 4 as default . 3.2 SYMMETRIC OBJECTIVE FUNCTION BACKGROUND . For instance-wise contrastive learning , to avoid trivial solutions in the symmetric framework , the objective function usually can be divided into alignment and uniformity terms ( Arora et al. , 2019 ) . We now review the loss formulated in SimCLR ( Chen et al. , 2020 ) as : Linfo = E ( x , x+ ) ∼ppos [ −f ( x ) > f ( x+ ) /τ ] ︸ ︷︷ ︸ alignment +E ( x , x+ ) ∼ppos { x− i } N i=1 ∼pdata [ log ( ef ( x ) > f ( x + ) /τ + ∑ i ef ( x ) > f ( x−i ) /τ ) ] ︸ ︷︷ ︸ uniformity ( 3 ) where τ is the temperature hyper-parameter . x+ and x− are positive and negative samples , respectively . The first term is the so-called alignment , which maximizes the similarity between positive pairs . The second term is the key to prevent trivial solution , pushing all the data points on a unit hyper-sphere uniformly . Then , we rewrite the feature-wise objective of Barlow Twins as : LBT = ∑ i ( 1−Cii ) 2︸ ︷︷ ︸ invariance +λ ∑ i ∑ j 6=i C2ij︸ ︷︷ ︸ redundancy reduction ( 4 ) where C ∈ Rd×d is the covariance matrix . The first term encourages correlation of the same feature across different views , while the second term minimizes correlation of different features . We can observe that the objective of both instance-wise and feature-wise methods requires square order calculation complexity , where the first term ( alignment , invariance ) requires linear order complexity and the second term ( uniformity , redundancy reduction ) requires square order complexity . We wonder if we can discard the second term , i.e. , completely discard negative pairs .

Summary: This paper presents a new self-supervised learning objective that aims to further adapt the previous decorrelation-based [1] contrastive learning method [Bardes, 2021], which largely alleviates the trivial solutions in SSL. Compared with prior arts, the proposed Zero-ICL/FCL are constructed from mainly the aspects of instance and feature-wise. In particular, Zero-CL requires no negative samples; feature-wise FCL discards the redundancy term by feature-wise whitening, and the proposed ICL prevents the collapse of contrastive learning effectively. Quantitative evaluations validate that Zero-CL leads to on-par performances with previous state-of-the-art results.