Contrastive Attraction and Contrastive Repulsion for Representation Learning

1 INTRODUCTION . The conventional Contrastive Learning ( CL ) loss ( Oord et al. , 2018 ; Poole et al. , 2018 ) has achieved remarkable success in representation learning , benefiting downstream tasks in a variety of areas ( Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020a ; Fang & Xie , 2020 ; Giorgi et al. , 2020 ) . This loss typically appears in a one-vs-many softmax form to make the encoder distinguish the positive sample within multiple negative samples . In image representation learning , this scheme is widely used to encourage the encoder to learn representations that are invariant to unnecessary details in the representation space , for which the unit hypersphere is the most common assumption ( Wang et al. , 2017 ; Davidson et al. , 2018 ; Hjelm et al. , 2018 ; Tian et al. , 2019 ; Bachman et al. , 2019 ) . Meanwhile , the contrast with negative samples is demystified as avoiding the collapse issue , where the encoder outputs a trivial constant , and uniformly distributing samples on the hypersphere ( Wang & Isola , 2020 ) . To improve the quality of the contrast , various methods , such as large negative memory bank ( Chen et al. , 2020c ) , hard negative mining ( Chuang et al. , 2020 ; Kalantidis et al. , 2020 ) , and using strong or multi-view augmentations ( Chen et al. , 2020a ; Tian et al. , 2019 ; Caron et al. , 2020 ) , are proposed and succeed in learning powerful representations . Since the conventional CL loss achieves the one-vs-many contrast with a softmax cross-entropy loss , a notable concern is that the contrast could be sensitive to the sampled positive and negative pairs ( Saunshi et al. , 2019 ; Chuang et al. , 2020 ) . Given a sampled query , conventional CL methods usually randomly take one positive sample and multiple negative samples , and equally treat them in a softmax cross-entropy form , regardless of how informative they are to the query . The sampled positive pair could make the contrast either easy or difficult , while trivially selecting hard negative pairs could make the pretraining inefficient . Thus in practice , a large negative sample size is desired to ensure diverse samples are incorporated to make the contrast more effective ( Wu et al. , 2021 ) , and the pretraining becomes less effective in large-scale real-world data , where the label could be noisy and rarely distributed in a balanced manner ( Li et al. , 2020b ; 2021 ) . Considering the CL loss aims to train the encoder to distinguish the positive sample from multiple negative samples , an alternative intuition is that the positive samples need to be pulled close , while negative samples need to be pushed far away from the given query in the representation space . This motivates the construction of Contrastive Attraction and Contrastive Repulsion ( CACR ) , a doubly CL framework where the positive and negative samples are first contrasted within themselves before getting pulled and pushed from the query , respectively . As shown in Figure 1 , unlike conventional CL , which equally treats samples and pulls/pushes them in the softmax cross-entropy contrast , CACR not only considers moving positive/negative samples close/away , but also models two conditional distributions to guide the movement of different samples . The conditional distributions apply a doubly-contrastive strategy to compare the positive samples and the negative ones within themselves separately . As an interpretation , if a selected positive sample is far from the query , it indicates the encoder does not sufficiently capture some information . CACR will then assign a higher probability for the query to pull this positive sample . Conversely , if a selected negative sample is too close to the query , it indicates the encoder has difficulty distinguishing them , and CACR will assign a higher probability for the query to push away this negative sample . This double-contrast method determines if a positive/negative sample is easy or hard to move , making the optimization more effective . Our main contributions include : i ) We propose CACR , which achieves contrastive learning and produces useful representations by attracting the positive samples towards the query and repelling the negative samples away from the query , guided by two conditional distributions . ii ) Our theoretical analysis shows that CACR generalizes the conventional CL loss . The conditional distributions help treat the samples differently by modeling the intra-relation of positive/negative samples , which is proved to be important when the datasets are less curated . iii ) Our experiments show that CACR consistently outperforms existing CL methods in a variety of settings , achieving state-of-the-art results on various benchmark datasets . Moreover , in the case where the dataset has an imbalanced label distribution , CACR has better robustness and provides consistent better pretraining results than conventional CL methods . 2 RELATED WORK . Plenty of unsupervised representation learning ( Bengio et al. , 2013 ) methods have been developed to learn good data representations , e.g. , PCA ( Tipping & Bishop , 1999 ) , RBM ( Hinton & Salakhutdinov , 2006 ) , VAE ( Kingma & Welling , 2014 ) . Among them , CL ( Oord et al. , 2018 ) is investigated as a lower bound of mutual information in early-stage ( Gutmann & Hyvärinen , 2010 ; Hjelm et al. , 2018 ) . Recently , many studies reveal that the effectiveness of CL is not just attributed to the maximization of mutual information ( Tschannen et al. , 2019 ; Tian et al. , 2020a ) , motivating various works to demystify the contrastive learning scheme . Contrastive representation learning . In vision tasks , SimCLR ( Chen et al. , 2020a ; b ) studies extensive augmentations for positive and negative samples and intra-batch-based negative sampling . A memory bank that caches representations ( Wu et al. , 2018 ) and a momentum update strategy are introduced to enable the use of an enormous number of negative samples ( He et al. , 2020 ; Chen et al. , 2020c ) . Tian et al . ( 2019 ; 2020b ) consider the image views in different modalities and minimize the irrelevant mutual information between them . Empirical researches observe the merits of using “ hard ” negative samples in CL , motivating additional techniques , such as Mixup and adversarial noise ( Bose et al. , 2018 ; Cherian & Aeron , 2020 ; Li et al. , 2020a ) . In a view that not all negative pairs are “ true ” negatives ( Saunshi et al. , 2019 ) , Chuang et al . ( 2020 ) propose a decomposition of the data distribution to approximate the true negative distribution . RingCL ( Wu et al. , 2021 ) proposes to use “ neither too hard nor too easy ” negative samples by predefined percentiles , and HN-CL ( Robinson et al. , 2021 ) applies Monte-Carlo sampling for selecting hard negative samples . CL has also been developed in learning representations for text ( Logeswaran & Lee , 2018 ) , sequential data ( Oord et al. , 2018 ; Hénaff et al. , 2019 ) , structural data like graphs ( Sun et al. , 2020a ; Li et al. , 2019 ; Hassani & Khasahmadi , 2020 ; Velickovic et al. , 2019 ) , reinforcement learning ( Srinivas et al. , 2020 ) , and few-shot scenarios ( Khosla et al. , 2020 ; Sylvain et al. , 2020 ) . Along with the success of these designs , works like Wang & Isola ( 2020 ) reveal the contrastive scheme is optimizing the alignment of positive samples and the uniformity of negative pairs in the limit of an infinite number of negative samples . Besides vision tasks , CL methods are widely applied to benefit in a variety of areas such as NLP , graph learning , and cross-modality learning ( Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020d ; Fang & Xie , 2020 ; Giorgi et al. , 2020 ; Gao et al. , 2021 ; Korbar et al. , 2018 ; Jiao et al. , 2020 ; Monfort et al. , 2021 ) . Deep representation learning . Besides contrastive methods for representation learning , a variety of relevant methods share the same purpose to learn powerful representation without label supervision . Triplet loss and its extensions consider the pair-wise similarity between the query and all other samples in the given mini-batch , where the positive/negative similarities need to be maximized/minimized ( Hoffer & Ailon , 2015 ; Sohn , 2016 ; Wang et al. , 2019 ; Sun et al. , 2020b ) . Instead of using both positive and negative pairs , some recent self-supervised methods require only positive pairs , deploying the spirit of Siamese networks and avoiding the collapse issue with stop gradient or a momentum updating strategy ( Chen & He , 2021 ; Grill et al. , 2020 ; Caron et al. , 2021 ) . In addition , Zbontar et al . ( 2021 ) propose to train the encoder to make positive feature pairs have higher correlation and decrease the cross-correlation in different feature dimensions to avoid the collapse . Another category is based on clustering , Caron et al . ( 2020 ) and Li et al . ( 2020c ) introduce the prototypes as a proxy and train the encoder by learning to predict the cluster assignment . CACR is closely related to the previous methods , where the conditional probability can be regarded as a re-weighting mechanism in triplet loss . The CA part could also be naturally combined with the recent self-supervised methods without negative pairs and replace the CR part with stop-gradient or momentum updating to avoid collapse issues . 3 THE PROPOSED APPROACH . In CL , for observations x0 : M ∼ pdata ( x ) , we commonly assume that each xi can be transformed in certain ways , with the samples transformed from the same and different data regarded as positive and negative samples , respectively . Specifically , we denote T ( xi , i ) as a random transformation of xi , where i ∼ p ( ) represents the randomness injected into the transformation . In computer vision , i often represents a composition of random cropping , color jitter , Gaussian blurring , etc . For each x0 , with query x = T ( x0 , 0 ) , we sample a positive pair ( x , x+ ) , where x+ = T ( x0 , + ) , and M negative pairs { ( x , x−i ) } 1 : M , where x − i = T ( xi , − i ) . Denote τ ∈ R+ , where R+ : = { x : x > 0 } , as a temperature parameter . With encoder fθ : Rn → Sd−1 , where we follow the convention to restrict the learned d-dimensional features with a unit norm , we desire to have similar and distinct representations for positive and negative pairs , respectively , via the contrastive loss as E ( x , x+ , x−1 : M ) [ − ln e fθ ( x ) > fθ ( x + ) /τ efθ ( x ) > fθ ( x+ ) /τ+ ∑ ie fθ ( x − i ) > fθ ( x ) /τ ] . ( 1 ) Note by construction , the positive sample x+ is independent of x given x0 and the negative samples x−i are independent of x . Intuitively , this 1-vs-M softmax cross-entropy encourages the encoder to not only pull the representation of a randomly selected positive sample closer to that of the query , but also push the representations of M randomly selected negative samples away from that of the query .

Summary: This paper introduces Contrastive Attraction and Contrastive Repulsion (CACR) loss for self-supervised learning with a doubly contrastive strategy. It constructs two conditional distributions to model the importance of positive samples and negative samples according to their distances to the query. The importance, which is reflected by weights, guides the query to not only more strongly pull more distant positive samples, but also more strongly push away closer negative samples. Theoretical analysis and empirical results show that CACR generalizes the conventional CL loss and is more robust in more general cases. Superior performances are achieved on both curated and uncurated datasets, and on various downstream tasks.