Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning

1 INTRODUCTION . One of the current main bottlenecks in deep network training is the dependence on heavy annotated training data , and this motivates the recent surge of interests in unsupervised ( Donahue & Simonyan , 2019 ) and self-supervised ( Chen & He , 2021 ; Chen et al. , 2020 ) methods . Specifically , in self-supervised representation learning ( SSL ) , a network is pre-trained without any form of manual annotation , thus providing a means to extract information from unlabeled data sources ( e.g. , text corpora , videos , images from the Internet , etc. ) . In self-supervision , label-based information is replaced by a prediction problem using a certain context or using a pretext task . Pretext task in SSL can mainly be divided into three categories : 1 ) Generative based approaches ( Donahue & Simonyan , 2019 ) learn to generate or otherwise model pixels in the input space . However , pixel-level generation is computationally expensive and may not be necessary for representation learning . 2 ) Contextual based methods ( Vincent et al. , 2008 ; Pathak et al. , 2016 ; Ye et al. , 2019 ) design pretext tasks ( denoising auto-encoders ( Vincent et al. , 2008 ) , context auto encoders ( Zhang et al. , 2016 ; 2017 ) , etc ) . 3 ) Contrastive based methods ( Chen et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Asano et al. , 2019 ) take augmented views of the same image as positive pair and others as negative pairs . Generally , one positive sample corresponds to lots of negative samples . In recent works , contrastive based methods have shown great promise , achieving state-of-the-art results in image classification ( Chen et al. , 2020 ) , video classification ( Han et al. , 2020 ) and other downstream tasks ( Chen & He , 2021 ) . However , trivial constant solutions ( different samples get the same representation ) is easily happening without the proper design of architecture and objective function . The well-known solutions to avoid this problem can be summarized into two parts : asymmetric model architecture and proper objective function . 1 ) Model architecture : MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) update encoders separately and stopping gradient operation is adopted to avoid such problem . Then , BYOL and SimSiam ( Chen & He , 2021 ) introduce a predictor module to avoid collapse , which is composed by MLP ( Goodfellow et al. , 2016 ) . Current mainstream interpretation is using the predic- tor to construct an asymmetric structure , which is useful to alleviate trivial solutions . 2 ) Objective function : SimCLR uses symmetric framework in contrastive learning . They prevent trivial solutions by using negative pairs and InfoNCE , where InfoNCE can be divided into an alignment term and a uniformity term ( Arora et al. , 2019 ) . The uniformity term pulls different samples to a hyper-sphere uniformly , forcing obtaining different representations and avoiding trivial solutions . Recently , Barlow Twins ( Zbontar et al. , 2021 ) designs a new objective function from the information redundancy perspective , which also has two terms ( an invariance term and a redundancy reduction term ) . The invariance term maximizes the correlation of the same feature across different views , and the redundancy term reduces information redundancy . However , for the symmetric framework , both SimCLR and Barlow Twins require square order complexity in objective functions , and the main complexity comes from the uniformity and redundancy term . In this paper , we propose two methods , named Zero-ICL and Zero-FCL , where Zero-ICL discards the uniformity term and only requires O ( N ) complexity by instance-wise whitening . Correspondingly , Zero-FCL discards the redundancy term by feature-wise whitening and only requires O ( d ) complexity . Our contributions are : 1 ) We propose a new contrastive learning framework to prevent trivial solutions , Zero-CL , which includes two parts , i.e. , Zero-ICL ( instance-wise ) and Zero-FCL ( feature-wise ) , either of which can work independently and only requires linear order complexity ( objective function ) . 2 ) To our best knowledge , Zero-ICL is the first attempt of instance-wise whitening , which is conceptually comprehensible for preventing collapses in contrastive learning . Note that most previous methods ( including other domains beyond vision ) e.g . ( Eldar & Oppenheim , 2003 ; Kessy et al. , 2018 ) only use whitening transformation to reduce the information redundancy on feature-wise . 3 ) We give empirical analysis on the relationship between previous methods and our Zero-CL , where previous negative sample consuming methods ( Zbontar et al. , 2021 ; Chen et al. , 2020 ) can be regarded as our method with Lagrangian transformation . Then , we theoretically introduce ZCA-based whitening from the maximal correlation ( Kessy et al. , 2018 ) perspective . 4 ) Experimental results on standard image benchmarks ( CIFAR-10/100 and ImageNet-100/1k ) show our method achieves new state-of-the-art results for symmetric contrastive learning compared with ( Chen et al. , 2020 ; Zbontar et al. , 2021 ) , especially for small hidden dimension and batch size . 2 RELATED WORK . We review recent contrastive learning works . We mainly divide the previous contrastive methods into feature-wise and instance-wise according to different contrastive dimensions . We particularly discuss the similarity and differences of some most related methods in Section 5 in detail . instance-wise contrastive learning . instance-wise contrastive learning aims to attract positive pairs and repulse negative pairs ( Chen et al. , 2020 ; He et al. , 2020 ) , where each pair is composed of two instance views . One of the widely used objective functions in instance-wise learning is InfoNCE , and extensive experiments in ( Oord et al. , 2018 ) show that InfoNCE requires negative pairs to avoid trivial solutions . SimCLR ( Chen et al. , 2020 ) regards views augmented from different images in a mini-batch as negative pairs . However , SimCLR requires a large batch size to improve the accuracy , which is GPU intensive . To address this issue , MoCo ( He et al. , 2020 ) proposes storing negative samples in the memory bank and updates the bank by the first-in-first-out principle . MoCo further proposes two operations to prevent trivial solutions , i.e. , momentum update key encoder and stop gradient . In recent works ( Chen et al. , 2020 ; Grill et al. , 2020 ) , the mainstream explanation of the two operations is they construct an asymmetric framework , which can avoid trivial solutions . Inspired by this , BYOL ( Grill et al. , 2020 ) adopts stop gradient operation and EMA algorithm to update target encoder . Further , BYOL introduces a predictor module to build a more asymmetric framework . Then , SimSiam ( Chen & He , 2021 ) empirically shows that stop gradient and the predictor is the key component to avoid trivial solutions . Another approach to preventing such solutions is introduced in W-MSE ( Ermolov et al. , 2021 ) . Different from previous works ( Chen et al. , 2020 ; Chen & He , 2021 ; He et al. , 2020 ) , W-MSE requires multiple augmented views to improve the accuracy . Besides , WMSE simply borrows Cholesky-based whitening transformation in SSL and the multi-view objective requires O ( NM2 ) complexity , where M is the number of views . feature-wise contrastive learning . Different from instance-wise learning , feature-wise methods contrast on feature dimension . Barlow Twins ( Zbontar et al. , 2021 ) is the first feature-wise method , which pushes the cross-correlation matrix to identity matrix I . Its objective includes both invariance term and redundancy reduction term , where the latter term can avoid trivial solutions . Inspired by Barlow Twins , VICReg ( Bardes et al. , 2021 ) proposes a new regularization term named invariancecovariance on feature dimension , and achieves similar results to Barlow Twins . However , both Barlow Twins and VICReg require large contrastive dimensions and the covariance regularization requires O ( d2 ) complexity , which could be also regarded as requiring negative pairs on feature dimension ( each negative pair is composed of features in different dimensions across images ) . Whitening transformation . Whitening , or sphering , is a common pre-processing step to transform random variables to orthogonality . However , due to rotational freedom , there are infinitely many possible whitening procedures . There is a diverse range of sphering methods , e.g . principal component analysis ( PCA ) ( Jégou & Chum , 2012 ) , Cholesky matrix decomposition ( Higham , 1990 ) and zero-phase component analysis ( ZCA ) ( Bell & Sejnowski , 1997 ) . Compared with other whitening methods , ZCA remains maximal correlation of original data , which we will clarify in Sec . 3.6 . 3 THE PROPOSED ZERO-CL . In this section , we present the framework of Zero-CL in detail , starting with the whitening transformation procedure , followed by the model ’ s framework as well as the objective function . At last , we provide theoretical analysis on whitening from maximal correlation perspective . 3.1 WHITENING TRANSFORMATION . Given a set of images { x } Ni=1 , we extract an embedding in contrastive space zi = g ( f ( xi , θ ) , γ ) using an encoder network ( He et al. , 2016 ) f ( · , θ ) and a MLP module g ( · , γ ) , where zi ∈ R1×d . Denote the set of embeddings as Z ∈ RN×d , whitening transformation can be formulated as : H = ( h1 , h2 , · · · , hN ) = WZ > ( 1 ) where H ∈ Rd×N is the whitened embedding matrix . The square matrix W ∈ Rd×d is the socalled whitening matrix . Since var ( H ) = I , it follows that WΣW = I and thus W ( ΣW > W ) = W , which is fulfilled if W satisfies W > W = Σ−1 . However , the constraint does not uniquely determine the whitening matrix W. Specifically , for any Wrot = RW with orthogonal matrix , R will also be a whitening transformation matrix . Here we present ZCA whitening matrix in detail . ZCA-based whitening takes E ( stacks together eigenvectors of the covariance matrix ) as the orthogonal matrix R. Then ZCA whitening matrix can be written as : WZCA = EΛ−1/2E > ( 2 ) By rotation matrix E , the whitened data will be as close as possible to original data ( see Theorem 1 ) . Note that in low-rank conditions , i.e. , some eigenvalues equal to 0 , a common approach is replacing Λ−1/2 with ( Λ + λI ) −1/2 to prevent zero division , where λ = 1e− 4 as default . 3.2 SYMMETRIC OBJECTIVE FUNCTION BACKGROUND . For instance-wise contrastive learning , to avoid trivial solutions in the symmetric framework , the objective function usually can be divided into alignment and uniformity terms ( Arora et al. , 2019 ) . We now review the loss formulated in SimCLR ( Chen et al. , 2020 ) as : Linfo = E ( x , x+ ) ∼ppos [ −f ( x ) > f ( x+ ) /τ ] ︸ ︷︷ ︸ alignment +E ( x , x+ ) ∼ppos { x− i } N i=1 ∼pdata [ log ( ef ( x ) > f ( x + ) /τ + ∑ i ef ( x ) > f ( x−i ) /τ ) ] ︸ ︷︷ ︸ uniformity ( 3 ) where τ is the temperature hyper-parameter . x+ and x− are positive and negative samples , respectively . The first term is the so-called alignment , which maximizes the similarity between positive pairs . The second term is the key to prevent trivial solution , pushing all the data points on a unit hyper-sphere uniformly . Then , we rewrite the feature-wise objective of Barlow Twins as : LBT = ∑ i ( 1−Cii ) 2︸ ︷︷ ︸ invariance +λ ∑ i ∑ j 6=i C2ij︸ ︷︷ ︸ redundancy reduction ( 4 ) where C ∈ Rd×d is the covariance matrix . The first term encourages correlation of the same feature across different views , while the second term minimizes correlation of different features . We can observe that the objective of both instance-wise and feature-wise methods requires square order calculation complexity , where the first term ( alignment , invariance ) requires linear order complexity and the second term ( uniformity , redundancy reduction ) requires square order complexity . We wonder if we can discard the second term , i.e. , completely discard negative pairs .

Summary: The paper is concerned with preventing the model collapse in the self-supervised learning scenario using contrastive losses. The main addition is an adaptation to the existing formulations that performs instance or feature whitening, avoiding the use of negative examples than can be expensive to store and to compute the similarity. Experiments on the selected datasets show that the method generally matches the current top-performing methods.