Contrastive Attraction and Contrastive Repulsion for Representation Learning

1 INTRODUCTION . The conventional Contrastive Learning ( CL ) loss ( Oord et al. , 2018 ; Poole et al. , 2018 ) has achieved remarkable success in representation learning , benefiting downstream tasks in a variety of areas ( Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020a ; Fang & Xie , 2020 ; Giorgi et al. , 2020 ) . This loss typically appears in a one-vs-many softmax form to make the encoder distinguish the positive sample within multiple negative samples . In image representation learning , this scheme is widely used to encourage the encoder to learn representations that are invariant to unnecessary details in the representation space , for which the unit hypersphere is the most common assumption ( Wang et al. , 2017 ; Davidson et al. , 2018 ; Hjelm et al. , 2018 ; Tian et al. , 2019 ; Bachman et al. , 2019 ) . Meanwhile , the contrast with negative samples is demystified as avoiding the collapse issue , where the encoder outputs a trivial constant , and uniformly distributing samples on the hypersphere ( Wang & Isola , 2020 ) . To improve the quality of the contrast , various methods , such as large negative memory bank ( Chen et al. , 2020c ) , hard negative mining ( Chuang et al. , 2020 ; Kalantidis et al. , 2020 ) , and using strong or multi-view augmentations ( Chen et al. , 2020a ; Tian et al. , 2019 ; Caron et al. , 2020 ) , are proposed and succeed in learning powerful representations . Since the conventional CL loss achieves the one-vs-many contrast with a softmax cross-entropy loss , a notable concern is that the contrast could be sensitive to the sampled positive and negative pairs ( Saunshi et al. , 2019 ; Chuang et al. , 2020 ) . Given a sampled query , conventional CL methods usually randomly take one positive sample and multiple negative samples , and equally treat them in a softmax cross-entropy form , regardless of how informative they are to the query . The sampled positive pair could make the contrast either easy or difficult , while trivially selecting hard negative pairs could make the pretraining inefficient . Thus in practice , a large negative sample size is desired to ensure diverse samples are incorporated to make the contrast more effective ( Wu et al. , 2021 ) , and the pretraining becomes less effective in large-scale real-world data , where the label could be noisy and rarely distributed in a balanced manner ( Li et al. , 2020b ; 2021 ) . Considering the CL loss aims to train the encoder to distinguish the positive sample from multiple negative samples , an alternative intuition is that the positive samples need to be pulled close , while negative samples need to be pushed far away from the given query in the representation space . This motivates the construction of Contrastive Attraction and Contrastive Repulsion ( CACR ) , a doubly CL framework where the positive and negative samples are first contrasted within themselves before getting pulled and pushed from the query , respectively . As shown in Figure 1 , unlike conventional CL , which equally treats samples and pulls/pushes them in the softmax cross-entropy contrast , CACR not only considers moving positive/negative samples close/away , but also models two conditional distributions to guide the movement of different samples . The conditional distributions apply a doubly-contrastive strategy to compare the positive samples and the negative ones within themselves separately . As an interpretation , if a selected positive sample is far from the query , it indicates the encoder does not sufficiently capture some information . CACR will then assign a higher probability for the query to pull this positive sample . Conversely , if a selected negative sample is too close to the query , it indicates the encoder has difficulty distinguishing them , and CACR will assign a higher probability for the query to push away this negative sample . This double-contrast method determines if a positive/negative sample is easy or hard to move , making the optimization more effective . Our main contributions include : i ) We propose CACR , which achieves contrastive learning and produces useful representations by attracting the positive samples towards the query and repelling the negative samples away from the query , guided by two conditional distributions . ii ) Our theoretical analysis shows that CACR generalizes the conventional CL loss . The conditional distributions help treat the samples differently by modeling the intra-relation of positive/negative samples , which is proved to be important when the datasets are less curated . iii ) Our experiments show that CACR consistently outperforms existing CL methods in a variety of settings , achieving state-of-the-art results on various benchmark datasets . Moreover , in the case where the dataset has an imbalanced label distribution , CACR has better robustness and provides consistent better pretraining results than conventional CL methods . 2 RELATED WORK . Plenty of unsupervised representation learning ( Bengio et al. , 2013 ) methods have been developed to learn good data representations , e.g. , PCA ( Tipping & Bishop , 1999 ) , RBM ( Hinton & Salakhutdinov , 2006 ) , VAE ( Kingma & Welling , 2014 ) . Among them , CL ( Oord et al. , 2018 ) is investigated as a lower bound of mutual information in early-stage ( Gutmann & Hyvärinen , 2010 ; Hjelm et al. , 2018 ) . Recently , many studies reveal that the effectiveness of CL is not just attributed to the maximization of mutual information ( Tschannen et al. , 2019 ; Tian et al. , 2020a ) , motivating various works to demystify the contrastive learning scheme . Contrastive representation learning . In vision tasks , SimCLR ( Chen et al. , 2020a ; b ) studies extensive augmentations for positive and negative samples and intra-batch-based negative sampling . A memory bank that caches representations ( Wu et al. , 2018 ) and a momentum update strategy are introduced to enable the use of an enormous number of negative samples ( He et al. , 2020 ; Chen et al. , 2020c ) . Tian et al . ( 2019 ; 2020b ) consider the image views in different modalities and minimize the irrelevant mutual information between them . Empirical researches observe the merits of using “ hard ” negative samples in CL , motivating additional techniques , such as Mixup and adversarial noise ( Bose et al. , 2018 ; Cherian & Aeron , 2020 ; Li et al. , 2020a ) . In a view that not all negative pairs are “ true ” negatives ( Saunshi et al. , 2019 ) , Chuang et al . ( 2020 ) propose a decomposition of the data distribution to approximate the true negative distribution . RingCL ( Wu et al. , 2021 ) proposes to use “ neither too hard nor too easy ” negative samples by predefined percentiles , and HN-CL ( Robinson et al. , 2021 ) applies Monte-Carlo sampling for selecting hard negative samples . CL has also been developed in learning representations for text ( Logeswaran & Lee , 2018 ) , sequential data ( Oord et al. , 2018 ; Hénaff et al. , 2019 ) , structural data like graphs ( Sun et al. , 2020a ; Li et al. , 2019 ; Hassani & Khasahmadi , 2020 ; Velickovic et al. , 2019 ) , reinforcement learning ( Srinivas et al. , 2020 ) , and few-shot scenarios ( Khosla et al. , 2020 ; Sylvain et al. , 2020 ) . Along with the success of these designs , works like Wang & Isola ( 2020 ) reveal the contrastive scheme is optimizing the alignment of positive samples and the uniformity of negative pairs in the limit of an infinite number of negative samples . Besides vision tasks , CL methods are widely applied to benefit in a variety of areas such as NLP , graph learning , and cross-modality learning ( Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020d ; Fang & Xie , 2020 ; Giorgi et al. , 2020 ; Gao et al. , 2021 ; Korbar et al. , 2018 ; Jiao et al. , 2020 ; Monfort et al. , 2021 ) . Deep representation learning . Besides contrastive methods for representation learning , a variety of relevant methods share the same purpose to learn powerful representation without label supervision . Triplet loss and its extensions consider the pair-wise similarity between the query and all other samples in the given mini-batch , where the positive/negative similarities need to be maximized/minimized ( Hoffer & Ailon , 2015 ; Sohn , 2016 ; Wang et al. , 2019 ; Sun et al. , 2020b ) . Instead of using both positive and negative pairs , some recent self-supervised methods require only positive pairs , deploying the spirit of Siamese networks and avoiding the collapse issue with stop gradient or a momentum updating strategy ( Chen & He , 2021 ; Grill et al. , 2020 ; Caron et al. , 2021 ) . In addition , Zbontar et al . ( 2021 ) propose to train the encoder to make positive feature pairs have higher correlation and decrease the cross-correlation in different feature dimensions to avoid the collapse . Another category is based on clustering , Caron et al . ( 2020 ) and Li et al . ( 2020c ) introduce the prototypes as a proxy and train the encoder by learning to predict the cluster assignment . CACR is closely related to the previous methods , where the conditional probability can be regarded as a re-weighting mechanism in triplet loss . The CA part could also be naturally combined with the recent self-supervised methods without negative pairs and replace the CR part with stop-gradient or momentum updating to avoid collapse issues . 3 THE PROPOSED APPROACH . In CL , for observations x0 : M ∼ pdata ( x ) , we commonly assume that each xi can be transformed in certain ways , with the samples transformed from the same and different data regarded as positive and negative samples , respectively . Specifically , we denote T ( xi , i ) as a random transformation of xi , where i ∼ p ( ) represents the randomness injected into the transformation . In computer vision , i often represents a composition of random cropping , color jitter , Gaussian blurring , etc . For each x0 , with query x = T ( x0 , 0 ) , we sample a positive pair ( x , x+ ) , where x+ = T ( x0 , + ) , and M negative pairs { ( x , x−i ) } 1 : M , where x − i = T ( xi , − i ) . Denote τ ∈ R+ , where R+ : = { x : x > 0 } , as a temperature parameter . With encoder fθ : Rn → Sd−1 , where we follow the convention to restrict the learned d-dimensional features with a unit norm , we desire to have similar and distinct representations for positive and negative pairs , respectively , via the contrastive loss as E ( x , x+ , x−1 : M ) [ − ln e fθ ( x ) > fθ ( x + ) /τ efθ ( x ) > fθ ( x+ ) /τ+ ∑ ie fθ ( x − i ) > fθ ( x ) /τ ] . ( 1 ) Note by construction , the positive sample x+ is independent of x given x0 and the negative samples x−i are independent of x . Intuitively , this 1-vs-M softmax cross-entropy encourages the encoder to not only pull the representation of a randomly selected positive sample closer to that of the query , but also push the representations of M randomly selected negative samples away from that of the query .

Summary: This paper proposes a new contrastive attraction and contrastive repulsion (CACR) loss for self-supervised representation learning. By formulating two conditional distributions, CACR considers different importance of positive/negative samples according to their distances to the query sample. Experimental results show superior performance of CACR against traditional contrastive loss and state-of-the-art methods.