Year: 2021

Abstract:
Contrastive learning (CL) methods effectively learn data representations without label supervision, where the encoder contrasts each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on ImageNet, a well-curated dataset with balanced image classes. However, they tend to yield worse performance when pretrained on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness on uncurated datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL’s behavior by positive attraction and negative repulsion. It further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets in representation learning, but also shows better robustness when generalized to pretrain on imbalanced image datasets.

heading: 1 INTRODUCTION
text: The conventional Contrastive Learning (CL) loss (Oord et al., 2018; Poole et al., 2018) has achieved remarkable success in representation learning, benefiting downstream tasks in a variety of areas (Misra & Maaten, 2020; He et al., 2020; Chen et al., 2020a; Fang & Xie, 2020; Giorgi et al., 2020). This loss typically appears in a one-vs-many softmax form to make the encoder distinguish the positive sample within multiple negative samples. In image representation learning, this scheme is widely used to encourage the encoder to learn representations that are invariant to unnecessary details in the representation space, for which the unit hypersphere is the most common assumption (Wang et al., 2017; Davidson et al., 2018; Hjelm et al., 2018; Tian et al., 2019; Bachman et al., 2019). Meanwhile, the contrast with negative samples is demystified as avoiding the collapse issue, where the encoder outputs a trivial constant, and uniformly distributing samples on the hypersphere (Wang & Isola, 2020). To improve the quality of the contrast, various methods, such as large negative memory bank (Chen et al., 2020c), hard negative mining (Chuang et al., 2020; Kalantidis et al., 2020), and using strong or multi-view augmentations (Chen et al., 2020a; Tian et al., 2019; Caron et al., 2020), are proposed and succeed in learning powerful representations. Since the conventional CL loss achieves the one-vs-many contrast with a softmax cross-entropy loss, a notable concern is that the contrast could be sensitive to the sampled positive and negative pairs (Saunshi et al., 2019; Chuang et al., 2020). Given a sampled query, conventional CL methods usually randomly take one positive sample and multiple negative samples, and equally treat them in a softmax cross-entropy form, regardless of how informative they are to the query. The sampled positive pair could make the contrast either easy or difficult, while trivially selecting hard negative pairs could make the pretraining inefficient. Thus in practice, a large negative sample size is desired to ensure diverse samples are incorporated to make the contrast more effective (Wu et al., 2021), and the pretraining becomes less effective in large-scale real-world data, where the label could be noisy and rarely distributed in a balanced manner (Li et al., 2020b; 2021).
Considering the CL loss aims to train the encoder to distinguish the positive sample from multiple negative samples, an alternative intuition is that the positive samples need to be pulled close, while negative samples need to be pushed far away from the given query in the representation space. This motivates the construction of Contrastive Attraction and Contrastive Repulsion (CACR), a doubly CL framework where the positive and negative samples are first contrasted within themselves before getting pulled and pushed from the query, respectively. As shown in Figure 1, unlike conventional CL, which equally treats samples and pulls/pushes them in the softmax cross-entropy contrast, CACR not only considers moving positive/negative samples close/away, but also models two conditional distributions to guide the movement of different samples. The conditional distributions apply a doubly-contrastive strategy to compare the positive samples and the negative ones within themselves separately. As an interpretation, if a selected positive sample is far from the query, it indicates the encoder does not sufficiently capture some information. CACR will then assign a higher probability for the query to pull this positive sample. Conversely, if a selected negative sample is too close to the query, it indicates the encoder has difficulty distinguishing them, and CACR will assign a higher probability for the query to push away this negative sample. This double-contrast method determines if a positive/negative sample is easy or hard to move, making the optimization more effective. Our main contributions include:
i) We propose CACR, which achieves contrastive learning and produces useful representations by attracting the positive samples towards the query and repelling the negative samples away from the query, guided by two conditional distributions.
ii) Our theoretical analysis shows that CACR generalizes the conventional CL loss. The conditional distributions help treat the samples differently by modeling the intra-relation of positive/negative samples, which is proved to be important when the datasets are less curated.
iii) Our experiments show that CACR consistently outperforms existing CL methods in a variety of settings, achieving state-of-the-art results on various benchmark datasets. Moreover, in the case where the dataset has an imbalanced label distribution, CACR has better robustness and provides consistent better pretraining results than conventional CL methods.

heading: 2 RELATED WORK
text: Plenty of unsupervised representation learning (Bengio et al., 2013) methods have been developed to learn good data representations, e.g., PCA (Tipping & Bishop, 1999), RBM (Hinton & Salakhutdinov, 2006), VAE (Kingma & Welling, 2014). Among them, CL (Oord et al., 2018) is investigated as a lower bound of mutual information in early-stage (Gutmann & Hyvärinen, 2010; Hjelm et al., 2018). Recently, many studies reveal that the effectiveness of CL is not just attributed to the maximization of mutual information (Tschannen et al., 2019; Tian et al., 2020a), motivating various works to demystify the contrastive learning scheme.
Contrastive representation learning. In vision tasks, SimCLR (Chen et al., 2020a;b) studies extensive augmentations for positive and negative samples and intra-batch-based negative sampling. A memory bank that caches representations (Wu et al., 2018) and a momentum update strategy are introduced to enable the use of an enormous number of negative samples (He et al., 2020; Chen et al.,
2020c). Tian et al. (2019; 2020b) consider the image views in different modalities and minimize the irrelevant mutual information between them. Empirical researches observe the merits of using “hard” negative samples in CL, motivating additional techniques, such as Mixup and adversarial noise (Bose et al., 2018; Cherian & Aeron, 2020; Li et al., 2020a). In a view that not all negative pairs are “true” negatives (Saunshi et al., 2019), Chuang et al. (2020) propose a decomposition of the data distribution to approximate the true negative distribution. RingCL (Wu et al., 2021) proposes to use “neither too hard nor too easy” negative samples by predefined percentiles, and HN-CL (Robinson et al., 2021) applies Monte-Carlo sampling for selecting hard negative samples. CL has also been developed in learning representations for text (Logeswaran & Lee, 2018), sequential data (Oord et al., 2018; Hénaff et al., 2019), structural data like graphs (Sun et al., 2020a; Li et al., 2019; Hassani & Khasahmadi, 2020; Velickovic et al., 2019), reinforcement learning (Srinivas et al., 2020), and few-shot scenarios (Khosla et al., 2020; Sylvain et al., 2020). Along with the success of these designs, works like Wang & Isola (2020) reveal the contrastive scheme is optimizing the alignment of positive samples and the uniformity of negative pairs in the limit of an infinite number of negative samples. Besides vision tasks, CL methods are widely applied to benefit in a variety of areas such as NLP, graph learning, and cross-modality learning (Misra & Maaten, 2020; He et al., 2020; Chen et al., 2020d; Fang & Xie, 2020; Giorgi et al., 2020; Gao et al., 2021; Korbar et al., 2018; Jiao et al., 2020; Monfort et al., 2021).
Deep representation learning. Besides contrastive methods for representation learning, a variety of relevant methods share the same purpose to learn powerful representation without label supervision. Triplet loss and its extensions consider the pair-wise similarity between the query and all other samples in the given mini-batch, where the positive/negative similarities need to be maximized/minimized (Hoffer & Ailon, 2015; Sohn, 2016; Wang et al., 2019; Sun et al., 2020b). Instead of using both positive and negative pairs, some recent self-supervised methods require only positive pairs, deploying the spirit of Siamese networks and avoiding the collapse issue with stop gradient or a momentum updating strategy (Chen & He, 2021; Grill et al., 2020; Caron et al., 2021). In addition, Zbontar et al. (2021) propose to train the encoder to make positive feature pairs have higher correlation and decrease the cross-correlation in different feature dimensions to avoid the collapse. Another category is based on clustering, Caron et al. (2020) and Li et al. (2020c) introduce the prototypes as a proxy and train the encoder by learning to predict the cluster assignment. CACR is closely related to the previous methods, where the conditional probability can be regarded as a re-weighting mechanism in triplet loss. The CA part could also be naturally combined with the recent self-supervised methods without negative pairs and replace the CR part with stop-gradient or momentum updating to avoid collapse issues.

heading: 3 THE PROPOSED APPROACH
text: In CL, for observations x0:M ∼ pdata(x), we commonly assume that each xi can be transformed in certain ways, with the samples transformed from the same and different data regarded as positive and negative samples, respectively. Specifically, we denote T (xi, i) as a random transformation of xi, where i ∼ p( ) represents the randomness injected into the transformation. In computer vision, i often represents a composition of random cropping, color jitter, Gaussian blurring, etc. For each x0, with query x = T (x0, 0), we sample a positive pair (x,x+), where x+ = T (x0, +), and M negative pairs {(x,x−i )}1:M , where x − i = T (xi, − i ). Denote τ ∈ R+, where R+ := {x : x > 0}, as a temperature parameter. With encoder fθ : Rn → Sd−1, where we follow the convention to restrict the learned d-dimensional features with a unit norm, we desire to have similar and distinct representations for positive and negative pairs, respectively, via the contrastive loss as
E (x,x+,x−1:M )
[ − ln e fθ(x) >fθ(x +)/τ
efθ(x) >fθ(x+)/τ+ ∑ ie fθ(x − i )>fθ(x)/τ
] . (1)
Note by construction, the positive sample x+ is independent of x given x0 and the negative samples x−i are independent of x. Intuitively, this 1-vs-M softmax cross-entropy encourages the encoder to not only pull the representation of a randomly selected positive sample closer to that of the query, but also push the representations of M randomly selected negative samples away from that of the query.

heading: 3.1 CONTRASTIVE ATTRACTION AND CONTRASTIVE REPULSION
text: In the same spirit of letting the query attract positive samples and repel negative samples, Contrastive Attraction and Contrastive Repulsion (CACR) directly models the cost of moving from the query to positive/negative samples with a doubly contrastive strategy:
LCACR := Ex∼p(x)Ex+∼π+θ (· |x,x0) [ c(fθ(x), fθ(x +)) ]︸ ︷︷ ︸
Contrastive Attraction
+Ex∼p(x)Ex−∼π−θ (· |x) [ −c(fθ(x), fθ(x−)) ]︸ ︷︷ ︸ Contrastive Repulsion ,
:= LCA + LCR, (2) where we denote π+ and π− as the conditional distributions of intra-positive contrasts and intranegative contrasts, respectively, and c(z1, z2) as the point-to-point cost of moving between two vectors z1 and z2, e.g., the squared Euclidean distance ‖z1 − z2‖2 or the negative inner product −zT1 z2. In the following we explain the doubly contrastive components with more details. Contrastive attraction: The intra-positive contrasts is defined in a form of the conditional probability, where the positive samples compete to gain a larger probability to be moved from the query. Here we adapt to CACR a Bayesian strategy in Zheng & Zhou (2020), which exploits the combination of an energy-based likelihood term and a prior distribution, to quantify the difference between two implicit probability distributions given their empirical samples. Specifically, denoting dt+(·, ·) as a distance metric with temperature t+ ∈ R+, e.g., dt+(z1, z2) = t+‖z1 − z2‖2, given a query x = T (x0, 0), we define the conditional probability for moving it to positive sample x+ = T (x0, +) as
π+θ (x + |x,x0) := e
d t+
(fθ(x),fθ(x +))
p(x+ |x0) Q+(x |x0) , Q +(x |x0) =: ∫ edt+ (fθ(x),fθ(x +))p(x+ |x0)dx+, (3)
where fθ(·) is an encoder parameterized by θ and Q+(x) is a normalization term. This construction makes it more likely to pull x towards a positive sample that is more distant in their latent representation space. With (3), the contrastive attraction loss LCA measures the expected cost of moving a query to its positive samples, as defined in (2), which more heavily weighs c(fθ(x), fθ(x+)) if fθ(x) and fθ(x+) are further away from each other.
Contrastive repulsion: On the contrary of the contrastive attraction shown in (3), we define the conditional probability for moving query x to a negative sample as
π−θ (x − |x) := e
−d t− (fθ(x),fθ(x −)) p(x−)
Q−(x) , Q −(x) :=
∫ e−dt− (fθ(x),fθ(x −))p(x−)dx−, (4)
where t− ∈ R+ is the temperature. This construction makes it more likely to move query x to a negative sample that is closer from it in their representation space. With (4), the contrastive repulsion loss LCR measures the expected cost to repel negative samples from the query shown in (2), which more heavily weighs c(fθ(x), fθ(x−)) if fθ(x) and fθ(x−) are closer to each other.
Choice of c(·, ·), dt+(·, ·) and dt−(·, ·). There could be various choices for the point-to-point cost function c(·, ·), distance metric dt+(·, ·) in (3), and dt−(·, ·) in (4). Considering the encoder fθ outputs normalized vectors on the surface of a hypersphere, maximizing the inner product is equivalent to minimizing squared Euclidean distance. Without loss of generality, we define them as
c(z1, z2) = ‖z1 − z2‖22, dt+(z1, z2) = t+‖z1 − z2‖22, dt−(z1, z2) = t−‖z1 − z2‖22, where t+, t−∈ R+. There are other choices for c(·, ·) and we show the ablation study in Appendix B.5.

heading: 3.2 MINI-BATCH BASED STOCHASTIC OPTIMIZATION
text: Under the CACR loss as in (2), to make the learning of fθ(·) amenable to mini-batch stochastic gradient descent (SGD) based optimization, we draw (xdatai , i) ∼ pdata(x)p( ) for i = 1, . . . ,M and then approximate the distribution of the query using an empirical distribution of M samples as
p̂(x) = 1M ∑M i=1 δ(x− xi); xi = T (xdatai , i).
where the δ(·) denotes the Dirac delta function. With query xi and 1:K iid∼ p( ), we approximate
p(x−i ) for Eqn. (4) and p(x + i |xdatai ) for Eqn. (3) as
p̂(x−i ) = 1 M−1 ∑ j 6=i δxj , p̂(x + i |xdatai ) = 1K ∑K k=1 δx+ik , x+ik = T (xdatai , k). (5)
Note we may improve the accuracy of p̂(x−i ) in (5) by adding previous queries into the support of this empirical distribution. Other more sophisticated ways to construct negative samples (Oord et al., 2018; He et al., 2020; Khosla et al., 2020) could also be adopted to define p̂(x−i ). We will elaborate these points when describing experiments.
Plugging (5) into (3) and (4), we approximate the conditional distributions with discrete distributions and obtain a mini-batch based CACR loss as L̂CACR = L̂CA + L̂CR, where
L̂CA := 1M ∑M i=1 ∑K k=1
e d t+ (fθ(xi),fθ(x + ik
))∑K k′=1 e d t+ (fθ(xi),fθ(x + ik′ )) ×c(fθ(xi), fθ(x+ik)),
L̂CR := − 1M ∑M i=1 ∑ j 6=i e −d t− (fθ(xi),fθ(xj))∑
j′ 6=i e −d t− (fθ(xi),fθ(xj′ ))
× c(fθ(xi), fθ(xj)).
We optimize θ via SGD using∇θL̂CACR, with the framework instantiated as in Figure 2.
Relation with typical CL loss: As shown in (2), with both the contrastive attraction component and contrastive repulsion component, CACR loss shares the same intuition of conventional CL in pulling positive samples closer to and pushing negative samples away from the query in their representation space. How-
ever, CACR realizes this intuition by introducing the double-contrast strategy on the point-to-point moving cost, where the contrasts appear in the intra-comparison within positive and negative samples, respectively. The use of the double-contrast strategy clearly differs the CACR loss in (2) from the conventional CL loss in (1), which typically relies on a softmax-based contrast formed with a single positive sample and multiple equally-weighted independent negative samples. A summary of the comparison between some representative CL losses and CACR is shown in Table 1.

heading: 4 PROPERTY ANALYSIS OF CACR

heading: 4.1 ON CONTRASTIVE ATTRACTION
text: We first analyze the effects w.r.t. the positive samples. With contrastive attraction, the property below suggests that the optimal encoder produces representations invariant to the noisy details. Property 1. The contrastive attraction loss LCA is optimized if and only if all positive samples of a query share the same representation as that query. More specifically, for query x that is transformed from x0 ∼ pdata(x), its positive samples share the same representation with it, which means
fθ(x +) = fθ(x) for any x+ ∼ π(x+ |x,x0). (6)
This property coincides with the characteristic (learning invariant representation) of the CL loss in Wang & Isola (2020) when achieving the optima. However, the optimization dynamic in contrastive attraction evolves in the context of x+ ∼ πθ(x+ |x,x0), which is different from that in the CL.
Lemma 1. Let us instantiate c(fθ(x), fθ(x+)) = −fθ(x)>fθ(x+). Then, the contrastive attraction loss LCA in (2) can be re-written as
Ex0Ex,x+∼p(· |x0) [ −fθ(x)>fθ(x+) π+θ (x + |x,x0)
p(x+ |x0)
] ,
which could further reduce to the alignment loss Ex0∼pdata(x)Ex,x+∼p(· |x0) [ −fθ(x)>fθ(x+)) ] in Wang & Isola (2020), iff π+θ (x + |x,x0) = p(x+ |x0).
Property 1 and Lemma 1 jointly show contrastive attraction in CACR and the alignment loss in CL reach the same optima, while working in different sampling mechanism. In practice x+ and x are usually independently sampled augmentations in a mini-batch, as shown in Section 3.2, which raises a gap between the empirical distribution and the true distribution. Our method makes the alignment more efficient by considering the intra-relation of these positive samples to the query.

heading: 4.2 ON CONTRASTIVE REPULSION
text: Next we analyze the effects w.r.t. the contribution of negative samples. Wang & Isola (2020) reveal that a perfect encoder will uniformly distribute samples on a hypersphere under an uniform isometric assumption, i.e., for any uniformly sampled x,x− iid∼ p(x), their latent representations z = fθ(x) and z− = fθ(x−) also satisfy p(z) = p(z−). We follow their assumption to analyze contrastive repulsion via the following lemma. Lemma 2. Without loss of generality, we define the moving cost and metric in the conditional distribution as c(z1, z2) = d(z1, z2) = ‖z1 − z2‖22. When we are with an uniform prior, namely p(x) = p(x−) for any x,x− iid∼ p(x) and p(z) = p(z−) given their latent representations z = fθ(x) and z− = fθ(x−), then optimizing θ with LCR in (2) is the same as optimizing θ to minimize the mutual information between x and x−:
I(X;X−) = Ex∼p(x) Ex−∼π−θ (· |x) [ ln π−θ (x − |x) p(x−) ] , (7)
and is also the same as optimizing θ to maximize the conditional differential entropy of x− given x: H(X− |X) = Ex∼p(x)Ex−∼π−θ (· |x)[− lnπ − θ (x
− |x)]. (8) Here the minimizer θ? of LCR is also that of I(X;X−), whose global minimum zero is attained iff X and X− are independent, and the equivalent maximum ofH(X− |X) indicates the optimization of LCR is essentially aimed towards the uniformity of representation about negative samples.
We notice that one way to reach the optimum suggested in the above lemma is optimizing θ by contrastive repulsion until that for any x ∼ p(x), d(fθ(x), fθ(x−)) is equal for all x− ∼ π−θ (· |x). This means for any sampled negative samples, their representations are also uniformly distributed after contrastive repulsion. Interestingly, this is consistent with the uniformity property achieved by CL (Wang & Isola, 2020), which connects contrastive repulsion with CL in the perspective of negative sample effects.
Note that, although the above analysis builds upon the uniform isometric assumption, our method actually does not rely on it. Here, we formalize a more general relation between the contrastive repulsion and the contribution of negative samples in CL without this assumption as follows. Lemma 3. As the number of negative samples M goes to infinity, the contribution of the negative samples to the CL loss become the Uniformity Loss in AU-CL (Wang & Isola, 2020), termed as Luniform for simplicity. It can be expressed as an upper bound of LCR by adding the mutual information I(X;X−) in (7):
Ex∼p(x) [ lnEx−∼p(x−)efθ(x −)>fθ(x)/τ ]
︸ ︷︷ ︸ Luniform +I(X;X−) > LCR,
As shown in Lemma 3, the mutual information I(X;X−) helps quantify the difference between Luniform and LCR. The difference between drawing x− ∼ π−θ (x− |x) (in CR) and drawing x− independently in a mini-batch (in CL) is non-trivial as long as I(X;X−) is non-zero. In practice, this is true almost everywhere since we have to handle the skewed data distribution in real-world applications, e.g., the label-shift scenarios (Garg et al., 2020). In this view, CR does not require the representation space to be uniform like CL does, and is more robust to the complex cases through considering the intra-contrastive relation within negative samples.

heading: 5 EXPERIMENTS AND EMPIRICAL ANALYSIS
text: In this section, we first study the CACR behaviors with small-scale experiments, where we use CIFAR-10, CIFAR-100 (Hinton, 2007) and create two class-imbalanced CIFAR datasets as empirical verification of our theoretical analysis. We mainly compare with representative CL methods, divided into two different categories according to their positive sampling size: K = 1 and K = 4. For methods with a single positive sample (K = 1), the baseline methods include the conventional CL loss (Oord et al., 2018), AlignUniform CL loss (AU-CL) (Wang & Isola, 2020), and the CL loss with hard negative sampling (HN-CL) (Robinson et al., 2021). In the case of K = 4, we take contrastive multi-view coding (CMC) loss (Tian et al., 2019) (align with our augmentation settings and use augmentation views instead of channels) as the comparison baseline. For a fair comparison, we keep for all methods with the same experiment setting including learning-rate, training epochs, etc., but use their best temperature parameters; the mini-batch size for K = 4 is divided by 4 from those when K = 1 to make sure the encoder leverages same samples in each iteration.
For large-scale datasets, we use ImageNet-1K (Deng et al., 2009) and compare with the state-of-theart frameworks (He et al., 2020; Zbontar et al., 2021; Chen et al., 2020a; Caron et al., 2020; Grill et al., 2020; Huynh et al., 2020) on linear probing, where we report the Top-1 validation accuracy on ImageNet-1K data. We also report the results of object detection/segmentaion following the transfer learning protocol. To further justify our analysis, we also leverage two large-scale but label-imbalanced datasets (Webvision v1 and ImageNet-22K) for linear probing pretraining. The reported numbers for baselines are from the original papers if available, otherwise we report the best ones reproduced with the settings according to their corresponding papers. Please refer to Appendix C for detailed experiment setups.

heading: 5.1 STUDIES AND ANALYSIS ON SMALL-SCALE DATASETS
text: Classification accuracy: To facilitate the analysis, we apply all methods with an AlexNet-based encoder following the setting in Wang & Isola (2020), trained in 200 epochs. We pretrained the encoder on regular CIFAR-10/100 data and create class-imbalanced cases by randomly sampling a certain number of samples from each class with a “linear” or “exponentional” rule by following the setting in Kim et al. (2020). Specifically, given a dataset with C classes, for class l ∈ {1, 2, ..., C}, we randomly take samples with proportion b lC c for “linear” rule and proportion exp(b l C c) for “exponential” rule. For evaluation we keep the standard validation/testing datasets. Thus there is a label-shift between the training and testing data distributions.
Summarized in Table 2 are the results on both regular and class-imbalanced datasets. The first two columns show the results pretrained with curated data, where we can observe that in the case of K = 1, where the intra-positive contrast of CACR degenerates, CACR slightly outperforms all CL methods. When K = 4, it is interesting to observe an obvious boost in performance, where CMC improves CL by around 2-3% while CACR improves CL by around 3-4%, which supports our analysis that CA is helpful when the intra-positive contrast is not degenerated. The right four columns present the linear probing results pretrained with class-imbalanced data, which show all the methods have a performance drop. It is clear that CACR has the least performance decline in most cases. Especially, when K = 4, CACR shows better performance robustness due to the characteristic of doubly contrastive within positive and negative samples. For example, in the “exponentional” setting of CIFAR-100, CL and HN-CL drop 12.57% and 10.73%, respectively, while CACR (K = 4) drops 9.24%. It is also interesting to observe HN-CL is relatively better among the baseline methods. According to Robinson et al. (2021), in HN-CL the negative samples are sampled according to the “hardness” w.r.t. the query samples with an intra-negative contrast. Its loss could converge to CACR (K = 1) with infinite negative samples. This performance gap indicates that directly optimizing the CACR loss could be superior when we have a limited number of samples. With this class-imbalanced datasets, we provide the empirical support to our analysis: When the condition in Lemma 2 is violated, CACR shows a clearer difference than CL and a better robustness with its unique doubly contrastive strategy within positive and negative samples.
On the effect of CA and CR: To further study the contrasts within positive and negative samples, in each epoch, we calculate the conditional entropy with (8) on every mini-batch of the validation data and take the average across mini-batches. Then, we illustrate in Figure 3 the evolution of conditional entropyH(X−|X) w.r.t. the training epoch on regular CIFAR-10 and class-imbalanced CIFAR-10.
As shown,H(X−|X) is getting maximized as the encoder is getting optimized, indicating the encoder learns to distinguish the negative samples from given query. It is also interesting to observe that in the case with multiple positive samples, this process is much more efficient, where the conditional entropy reaches the possible biggest value rapidly. This implies the CA module can further boost the repulsion of negative samples. From the gap between CACR and CMC, we can learn although CMC uses multiple positive in CL loss, the lack of intra-positive contrast shows the gap of repulsion efficiency. In the right panel of Figure 3, the difference between CACR and baseline methods are more obvious, where we can find the conditional entropy of baselines is slightly lower than pretrained with regular CIFAR-10 data. Especially for vanilla CL loss, we can observe the conditional entropy has a slight decreasing tendency, indicating the encoder hardly learns to distinguish negative samples in this case. Conversely, CACR still shows to remain the conditional entropy at a higher level, which explains the robustness shown in Table 2, and indicating a superior learning efficiency of CACR. See Appendix B.2 for similar observations on CIFAR-100 and exponential label-imbalanced cases. In that part, we provide more quantitative and qualitative studies on the effects of conditional distributions.

heading: 5.2 COMPARISON WITH SOTAS ON LARGE-SCALE DATASETS
text: For large-scale experiments, following the convention, we adapt CACR loss into the MoCo-v2 framework and pretrain a ResNet50 encoder with mini-batch size 256 on ImageNet-1K.
Linear probing: Table 3 summarizes the results of linear classification, where a linear classifier is trained on ImageNet-1K on top of fixed representations of the pretrained ResNet50 encoder. Similar to the case on small-scale datasets, CACR consistently shows better performance than the baselines using contrastive loss, improving SimCLR and MoCov2 by 2.7% and 2.2% respectively. Compared with other self-supervised SOTAs, CACR also shows on par performance.
Label-imbalanced case: To strengthen our analysis on small-scale label-imbalanced data, we specially deploy two real-world, but less curated datasets Webvision v1 and ImageNet-22K that have long-tail label distributions for encoder pretraining and evaluate the linear classification accuracy on ImageNet-1K. We pretrain encoder with 100/20 epochs on Webvision v1/ImageNet-22K and compare with the encoder pretrained with 200 epochs on ImageNet-1K to make sure similar samples
have been seen in the pretraining. The results are shown in Table 4, where we can see CACR still outperforms the MoCov2 baseline and shows better robustness when generalized to wild image data.
Object detection and segmentation: Besides the linear classification evaluation, following the protocols in previous works (Tian et al., 2019; He et al., 2020; Chen et al., 2020c; Wang & Isola, 2020), we use the pretrained ResNet50 on ImageNet-1K for object detection and segmentation task on Pascal VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) by using detectron2 (Wu et al., 2019). The experimental setting details are shown in Appendix C.2 and kept the same as He et al. (2020) and Chen et al. (2020c). The test AP, AP50, and AP75 of bounding boxes in object detection and test AP, AP50, and AP75 of masks in segmentation are reported in Table 5. We can observe that the performances of CACR is consistently better than baselines using contrastive objectives, and even better than recent self-supervised learning SOTAs.

heading: 6 CONCLUSION
text: In this paper, we rethink the limitation of conventional contrastive learning (CL) methods that use the contrastive loss but merely consider the intra-relation between samples. We introduce a novel Contrastive Attraction and Contrastive Repulsion (CACR) loss with a doubly contrastive strategy, which constructs for two conditional distributions to respectively model the importance of a positive sample and that of a negative sample to the query according to their distances to the query. Our theoretical analysis and empirical results show that optimizing the CACR loss can effectively attract positive samples and repel negative ones from the query as CL intends to do, but is more robust in more general cases. Extensive experiments on small, large-scale, and imbalanced datasets consistently demonstrate the superiority and robustness of CACR over the state-of-the-art methods in contrastive representation learning and related downstream tasks.

heading: ETHICS STATEMENT
text: Contrastive learning (CL) is effective in learning data representations without label supervision and has led to notable recent progresses in a variety of research areas, such as computer vision. Recently proposed advanced CL methods often require a huge amount of data and thus cost large computational energy. Especially in the case where one needs to use multiple positive pairs in the contrast. Instead of contrasting each positive pair over multiple negative pairs with the classic softmax cross-entropy, our work discovers that the contrastive attraction within positives and contrastive repulsion within negatives bring new insight in self-supervised representation learning. CACR, which naturally takes multiple positive samples in the contrast without making the contrast complexity become combinatorial in the number of positive pairs, has demonstrated clear improvements over existing CL methods. However, the same as existing CL methods, our method is not designed to resist the potential biases existing in the dataset, e.g. the false negatives in data. At the current stage, CACR relies on the positive contrast to implicitly alleviate this issue: if a false negative sample is repelled too far away from the query, in the positive attraction, it will be assigned with larger probability to be pulled back. This raises the risk of the quality of learned representations. In the future work, we aim and also encourage other researchers to consider the resistance of these potential risks to make the learned representations more robust and powerful.

heading: REPRODUCIBILITY STATEMENT
text: Our paper provides both theoretical analysis and empirical studies on the proposed algorithm. For theoretical results, to make sure the properties hold and the analysis are correct, we provide stepby-step proof in Appendix A. For empirical results on both small-scale and large-scale datasets, we provide detailed experiment settings and Pytorch-style pseudo-code in Appendix C. We also include an example code with instruction as supplementary material to ensure the reproducibility.

heading: Contrastive Attraction and Contrastive Repulsion for Representation Learning: Appendix

heading: A PROOFS AND DETAILED DERIVATION
text: Proof of Property 1. By definition, the point-to-point cost c(z1, z2) is always non-negative. Without loss of generality, we define it with the Euclidean distance. When (6) is true, the expected cost of moving between a pair of positive samples, as defined as LCA in (2), will reach its minimum at 0. When (6) is not true, by definition we will have LCA > 0, i.e., LCA = 0 is possible only if (6) is true.
Proof of Lemma 1. By changing the reference distribution of the expectation from π+θ (· |x,x0) to p(· |x0), we can directly re-write the CA loss as:
LCA = Ex∼p(x)Ex+∼π+θ (· |x,x0) [ c(fθ(x), fθ(x +)) ]
= Ex0Ex,x+∼p(· |x0) [ −fθ(x)>fθ(x+) π+θ (x + |x,x0)
p(x+ |x0)
] ,
which complete the proof.
Proof of Lemma 2. Denoting
Z(x) = ∫ e−d(fθ(x),fθ(x −))p(x−)dx−,
we have lnπ−θ (x
− |x) = −d(fθ(x), fθ(x−)) + ln p(x−)− lnZ(x). Thus we have
LCR = Ex∼p(x)Ex−∼π−θ (· |x)[lnπ − θ (x − |x)− ln p(x−) + lnZ(x)]
= C1 + C2 −H(X− |X) (9)
where C1 = Ex∼p(x)Ex−∼π−θ (· |x)[ln p(x −)] and C2 = −Ex∼p(x) lnZ(x). Under the assumption of a uniform prior on p(x), C1 becomes a term that is not related to θ. Under the assumption of a uniform prior on p(z), where z = fθ(x), we have
Z(x) = Ex−∼p(x)[e−d(fθ(x),fθ(x −))]
= Ez−∼p(z)[e−(z −−z)T (z−−z)] ∝ ∫ e−(z −−z)T (z−−z)dz−
= √ π, (10)
which is also not related to θ. Therefore, under the uniform prior assumption on both p(x) and p(z), minimizing LCR is the same as maximizing H(X− |X), as well as the same as minimizing I(X,X−).
Proof of Lemma 3. The CL loss can be decomposed as an expected dissimilarity term and a logsum-exp term:
LCL := E (x,x+,x−1:M )
[ − ln e fθ(x) >fθ(x +)/τ
efθ(x)>fθ(x+)/τ + ∑ ie fθ(x − i ) >fθ(x)/τ
]
= E(x,x+) [ −1 τ fθ(x) >fθ(x +) ] + E
(x,x+,x−1:M )
[ ln ( efθ(x) >fθ(x +)/τ +
M∑ i=1 efθ(x − i ) >fθ(x)/τ
)] ,
where the positive sample x+ is independent of x given x0 and the negative samples x−i are independent of x. As the number of negative samples goes to infinity, following Wang & Isola (2020), the normalized CL loss is decomposed into the sum of the align loss, which describes the contribution of the positive samples, and the uniform loss, which describes the contribution of the negative samples:
lim M→∞ LCL − lnM = E(x,x+) [ −1 τ fθ(x) >fθ(x +) ] ︸ ︷︷ ︸
contribution of positive samples
+ E x∼p(x) [ ln E x−∼p(x−) efθ(x −)>fθ(x)/τ ] ︸ ︷︷ ︸
contribution of negative samples
With importance sampling, the second term in the RHS of the above equation can be further derived into:
E x∼p(x) [ ln E x−∼p(x−) efθ(x −)>fθ(x)/τ ] = E x∼p(x) [ ln E x−∼πθ(x−|x) [ efθ(x −)>fθ(x)/τ p(x−) πθ(x−|x)
]] Apply the Jensen inequality, the second term is decomposed into the negative cost plus a log density ratio:
E x∼p(x) [ ln E x−∼p(x−) efθ(x −)>fθ(x)/τ ] > E x∼p(x) [ E x−∼πθ(x−|x) [ fθ(x −)>fθ(x)/τ ]] + E x∼p(x) [ E x−∼πθ(x−|x) [ ln p(x−) πθ(x− |x)
]] = E x∼p(x) [ E x−∼πθ(x− |x) [ fθ(x −)>fθ(x)/τ ]] − I(X;X−)
Defining the point-to-point cost function between two unit-norm vectors as c(z1, z2) = −z>1 z2 (same as the Euclidean cost since ‖z1 − z2‖22/2 = 1− z>1 z2 ), we have
E x∼p(x) [ ln E x−∼p(x) efθ(x −)>fθ(x)/τ ] + I(X;X−)
> E x∼p(x)
[ E
x−∼πθ(x− |x)
[ fθ(x −)>fθ(x)/τ ]]
= − E x∼p(x)
[ E
x−∼πθ(x− |x)
[ c(fθ(x −), fθ(x))/τ ]]
= LCR.
This concludes the relation between the contribution of the negative samples in CL and that in CACR.

heading: B ADDITIONAL EXPERIMENTAL RESULTS
text: In this section, we provide additional results in our experiments, including ablation studies, and corresponding qualitative results.

heading: B.1 ADDITIONAL RESULTS WITH ALEXNET AND RESNET50 ENCODER ON SMALL-SCALE DATASETS
text: Following benchmark works in contrative learning, we add STL-10 dataset to evaluate CACR in small-scale experiments. As an additional results on small-scale datasets, we test the performance of CACR two different encoder backbones. Here we strictly follow the same setting of Wang & Isola (2020) and Robinson et al. (2021), and the results are shown in Table 6 and 7. We can observe with ResNet50 encoder backbone, CACR with single positive or multiple positive pairs consistently outperform the baselines. Compared with the results in Table 6, the CACR shows a more clear improvement over the CL baselines.

heading: B.2 ON THE EFFECTS OF CONDITIONAL DISTRIBUTION
text: Supplementary studies of CA and CR: As a continuous ablation study shown in Figure 3, we also conduct similar experiments on CIFAR-100, where we study the evolution of conditional entropy H(X−|X) w.r.t. the training epoch. The results are shown in Figure 4, and the results of exponential label-imbalanced data are shown in Figure 5. Similar to the observation on CIFAR-10, shown in Figure 3, we can observeH(X−|X) is getting maximized as the encoder is getting optimized with these methods, as suggested in Lemma 2. In the right panel, We can observe baseline methods have lower conditional entropy, which indicates the encoder is less effective in distinguish the nagative samples from query, while CACR consistently provides better performance than the other methods indicating the better robustness of CACR.
As a qualitative verification, we randomly take a query from a mini-batch, and illustrate its positive and negative samples and their conditional probabilities in Figure 6. As shown, given this query of a dog image, the positive sample with the largest weight contains partial dog information, indicating the encoder to focus on texture information; the negatives with larger weights are more related to the dog category, which encourages the encoder to focus on distinguishing these “hard” negative samples. In total, the weights learned by CACR enjoy the interpretability compared to the conventional CL.
We study different definition of the conditional distribution. From Table 8, we can observe that the results are not sensitive to the distance space. In addition, as we change π+ to assign larger probability to closer samples, the results are similar to those using single positive pair (K=1). Moreover, the performance drops if we change π− to assign larger probability to more distant negative samples.
Uniform Attraction and Uniform Repulsion: A degenerated version of CACR
To reinforce the necessity of the contrasts within positives and negatives before the attraction and repulsion, we introduce a degenerated version of CACR here, where the conditional distributions are forced to be uniform. Remind c(z1, z2) as the point-to-point cost of moving between two vectors z1 and z2, e.g., the squared Euclidean distance ‖z1 − z2‖2 or the negative inner product −zT1 z2. In the same spirit of (1), we have considered a uniform attraction and uniform repulsion (UAUR) without doubly contrasts within positive and negative samples, whose objective is
min θ
{ Ex0∼pdata(x)E 0, +∼p( ) [ c(fθ(x), fθ(x +)) ] − Ex,x−∼p(x) [ c(fθ(x), fθ(x −)) ]} . (11)
The intuition of UAUR is to minimize/maximize the expected cost of moving the representations of positive/negative samples to that of the query, with the costs of all sample pairs being uniformly weighted. While (1) has been proven to be effective for representation learning, our experimental results do not find (11) to perform well, suggesting that the success of representation learning is not guaranteed by uniformly pulling positive samples towards and pushing negative samples away from the query.
Distinction between CACR and UAUR: Compared to UAUR in (11) that uniformly weighs different pairs, CACR is distinct in considering the dependency between samples: as the latent-space distance between the query and its positive sample becomes larger, the conditional probability becomes higher, encouraging the encoder to focus more on the alignment of this pair. In the opposite, as the distance between the query and its negative sample becomes smaller, the conditional probability becomes higher, encouraging the encoder to push them away from each other.
In order to further explore the effects of the conditional distribution, we conduct an ablation study to compare the performance of different variants of CACR with/without conditional distributions. Here, we compare 4 configurations of CACR (K = 4): (i) CACR with both positive and negative conditional distribution; (ii) CACR without the positive conditional distribution; (iii) CACR without the negative conditional distribution; (iv) CACR without both positive and negative conditional distributions, which refers to UAUR model (see Equation 11). As shown in Table 9, when discarding the positive conditional distribution, the linear classification accuracy slightly drops. As the negative
conditional distribution is discarded, there is a large performance drop compared to the full CACR objective. With the modeling of neither positive nor negative conditional distribution, the UAUR shows a continuous performance drop, suggesting that the success of representation learning is not guaranteed by uniformly pulling positive samples closer and pushing negative samples away. The comparison between these CACR variants shows the necessity of the conditional distribution.
As qualitative illustrations, we randomly fix one mini-batch, and randomly select one sample as the query. Then we extract the features with the encoder trained with CL loss and CACR (K = 1) loss at epochs 1, 20, and 200, and visualize the (four) positives and negatives in the embedding space with t-SNE (van der Maaten & Hinton, 2008). For more clear illustration, we center the query in the middle of the plot and only show samples appearing in the range of [−10, 10] on both x and y axis. The results are shown in Figure 7(c), from which we can find that as the the encoder is getting trained, the positive samples are aligned closer and the negative samples are pushed away for both methods. Compared to the encoder trained with CL, we can observe CACR shows better performance in achieving this goal. Moreover, we can observe the distance between any two data points in the plot is more uniform, which confirms that CACR shows better results in the maximization of the conditional entropyH(X−|X).

heading: B.3 ABLATION STUDY
text: Does CACR(K > 2) outperform by seeing more samples? To address this concern, in our main paper, we intentionally decrease the mini-batch size as M = 128. Thus the total number of samples
used per iteration is not greater than those used when K = 1. To further justify if the performance boost comes from seeing more samples when using multiple positive pairs, we also let the methods allowing single positive pair train with double epochs. As shown in Table 10, we can observe even trained with 400 epochs, the performance of methods using single positive pair still have a gap from those using multiple positive pairs.
On the effects of negative sampling size: We investigate the model performance and robustness with different sampling size by varying the mini-batch size used in the training. On all the small-scale datasets, the mini-batches are applied with size 64, 128, 256, 512, 768 and the corresponding linear classification results are shown in Figure 8. From this figure, we can see that CACR (K = 4) consistently achieves better performance than other objectives. For example, when mini-batch size is 256, CACR (K = 4) outperforms CMC by about 0.4%-1.2%. CACR (K = 1) shows better performance in most of the cases, while slightly underperforms than the baselines with mini-batch size 64. A possible explanation could be the estimation of the conditional distribution needs more samples to provide good guidance for the encoder.
On the effects of positive sampling size: We conduct experiments to investigate the model performance with different positive sampling size by using different K values in the pretraining: K ∈ {1, 2, 4, 6, 8, 10} on CIFAR-10/100 and K ∈ {1, 2, 3, 4} on ImageNet-1K. Similar to our experiment setting, in 200 epochs, we apply AlexNet encoder on CIFAR-10 and CIFAR-100 and apply ResNet50 encoder on ImageNet-1K. Shown in Figure 9, we can observe as K increases, the linear classification accuracy increases accordingly. Correspondingly, we also evaluate the GPU time of CACR loss with different choices of K, as shown in Table 11 and Table 16.
On the effects of hyper-parameter t+, t−: Remind in the definition of positive and negative conditional distribution, two hyper-parameters t+ and t− are involved as following:
π+θ (x + |x,x0) := e
t+‖fθ(x)−fθ(x +)‖2p(x+ |x0)∫
et +‖fθ(x)−fθ(x+)‖2p(x+ |x0)dx+
; π−θ (x − |x) := e
−t−‖fθ(x)−fθ(x −)‖2p(x−)∫
e−t −‖fθ(x)−fθ(x−)‖2p(x−)dx−
.
In this part, we investigate the effects of t+ and t− on representation learning performance on small-scale datasets, with mini-batch size 768 (K = 1) and 128 (K = 4) as an ablation study. We search in a range {0.5, 0.7, 0.9, 1.0, 2.0, 3.0}. The results are shown in Table 12 and Table 13. As shown in these two tables, from Table 12, we observe the CACR shows better performance with smaller values for t+. Especially when t+ increases to 3.0, the performance drops up to about
1.9% on CIFAR-100. For analysis, since we have K = 4 positive samples for the computation of positive conditional distribution, using a large value for t+ could result in an over-sparse conditional distribution, where the conditional probability is dominant by one or two positive samples. This also explains why the performance when t+ = 3.0 is close to the classification accuracy of CACR (K = 1).
Similarly, from Table 13, we can see that a small value for t− will lead to the degenerated performance. Here, since we are using mini-batches of size 768 (K = 1) and 128 (K = 4), a small value for t− will flatten the weights of the negative pairs and make the conditional distribution closer to a uniform distribution, which explains why the performance when t− = 0.5 is close to those without modeling π−θ . Based on these results, the values of t
+ ∈ [0.5, 1.0] and t− ∈ [0.9, 2.0] could be good empirical choices according to our experiment settings on these datasets.

heading: B.4 ADDITIONAL COMPARISONS
text: In this part we provide more comparisons with baseline methods. For small-scale experiments, we still compare with contrastive learning methods, conventional CL loss, align-uniform loss, and hard negative sampling CL loss. For large-scale experiments, we continue to compare with contrastive learning loss on ImageNet-100 and ImagNet-1K with MoCov2 framework, and provide comparisons with SOTAs pretrained with different epochs.
Training efficiency on small-scale datasets: On CIFAR-10, CIFAR-100 and STL-10, we pretrained AlexNet encoder in 200 epochs and save linear classification results with learned representations every 10 epochs. Shown in Figure 10, CACR consistently outperforms the other methods in linear classification with the learned representations at the same epoch, indicating a superior learning efficiency of CACR.
Comparison with contrastive learning methods on ImageNet: For large-scale experiments, following the convention, we adapt all methods into the MoCo-v2 framework and pre-train a ResNet50 encoder in 200 epochs with mini-batch size 128/256 on ImageNet-100/ImageNet-1k. Table 14 summarizes the results of linear classification on these two large-scale datasets. Similar to the case on small-scale datasets, CACR consistently shows better performance, improving the baselines at least by 1.74% on ImageNet-100 and 0.71% on ImageNet-1K. In MoCo-v2, with multiple positive samples, CACR improves the baseline methods by 2.92% on ImageNet-100 and 2.75% on ImageNet-1K. It is worth highlighting that the improvement of CACR is more significant on these large-scale datasets, where the data distribution could be much more diverse compared to these small-scale ones. This is not surprising, as according to our theoretical analysis, CACR’s double-contrast within samples enhances the effectiveness of the encoder’s optimization. Moreover, we can see CACR (K = 1)
shows a clear improvement over HN-CL. A possible explanation is that although both increasing the negative sample size and selecting hard negatives are proposed to improve the CL loss, the effectiveness of hard negatives is limited when the sampling size is increased over a certain limit. As CACR targets to repel the negative samples away, the conditional distribution still efficiently guides the repulsion when the sampling size becomes large.
Comparison with other SOTAs: Besides the methods using contrastive loss, we continue to compare with the self-supervised learning methods like BYOL, SWaV, SimSiam, etc. that do not involve the contrasts with negative samples. Table 15 provides more detailed comparison with all state-of-the-arts in different epochs and could better support the effectiveness of CACR: We can observe CACR achieves competitive results and generally outperforms most of SOTAs at the same epoch in linear classification tasks. We also compare the computation complexity. Table 16 reports computation complexity to provide quantitative results in terms of positive number K, where we can observe the computation cost of CACR slightly increases as K increase, but does not increase as that when using multi-positives in CL loss.

heading: B.5 CONNECTION TO OTHER REPRESENTATION LEARNING METHODS
text: Results of different cost metrics
Recall that the definition of the point-to-point cost metric is usually set as the quadratic Euclidean distance:
c(fθ(x), fθ(y)) = ||fθ(x)− fθ(y)||22. (12) In practice, the cost metric defined in our method is flexible to be any valid metrics. Here, we also investigate the performance when using the Radial Basis Function (RBF) cost metrics:
cRBF(fθ(x), fθ(y)) = −e−t||fθ(x)−fθ(y)|| 2 2 , (13)
where t ∈ R+ is the precision of the Gaussian kernel. With this definition of the cost metric, our method is closely related to the baseline method AU-CL (Wang & Isola, 2020), where the authors calculate pair-wise RBF cost for the loss w.r.t. negative samples. Following Wang & Isola (2020), we replace the cost metric when calculate the negative repulsion cost with the RBF cost and modify L̂CR as:
L̂CR−RBF := ln [ 1 M M∑ i=1 ∑ j 6=i e −d t− (fθ(xi),fθ(xj))∑ j′ 6=i e −d t− (fθ(xi),fθ(xj′ )) × cRBF(fθ(xi), fθ(xj)) ] (14)
= ln [ 1 M M∑ i=1 ∑ j 6=i e −d t− (fθ(xi),fθ(xj))∑ j′ 6=i e −d t− (fθ(xi),fθ(xj′ )) × e−t‖fθ(xi)−fθ(xj)‖ 2] .
Here the negative cost is in log scale for numerical stability. When using the RBF cost metric, we use the same setting in the previous experiments and evaluate the linear classification on all small-scale datasets. The results of using Euclidean and RBF cost metrics are shown in Table 17. From this table, we see that both metrics achieve comparable performance, suggesting the RBF cost is also valid in our framework. In CACR, the cost metric measures the cost of different sample pairs and is not limited on specific formulations. More favorable cost metrics can be explored in the future.
Discussion: Relation to triplet loss CACR is also related to the widely used triplet loss (Schroff et al., 2015; Sun et al., 2020b). A degenerated version of CACR where the conditional distributions are all uniform can be viewed as triplet loss, while underperform the proposed CACR, as discussed in Section B.2. In the view of triplet loss, CACR is dealing with the margin between expected positive pair similarity and negative similarity:
LCACR = [Eπt+ (x+|x)[c(x, x +)]− Eπt− (x−|x)[c(x, x −)] +m]+
which degenerates to the generic triplet loss if the conditional distribution degenerates to a uniform distribution:
LUAUR = [Ep(x+)[c(x, x+)]− Ep(x−)[c(x, x−)] +m]+ = [c(x, x+)− c(x, x−) +m]+ This degeneration also highlights the importance of the Bayesian derivation of the conditional distribution. The experimental results of the comparison between CACR and the degenerated uniform version (equivalent to generic triplet loss) are presented in Table 9.
Moreover, CACR loss can degenerate to a triplet loss with hard example mining if πt+(x+|x) and πt+(x
+|x) are sufficiently concentrated, where the density shows a very sharp peak: LCACR = [max(c(x, x+))−min(c(x, x−)) +m]+
which corresponds to the loss shown in Schroff et al. (2015). As shown in Table 12 and 13, when varying t+ and t− to sharpen/flatten the conditional distributions. Based on our observations, when
t+ = 3 and t− = 3, the conditional distributions are dominated by 1-2 samples, where CACR can be regarded as the above-mentioned triplet loss, and this triplet loss with hard mining slightly underperforms CACR. From these views, CACR provides a more general form to connect the triplet loss. Meanwhile, it is interesting to notice CACR explains how triplet loss is deployed in the self-supervised learning scenario.
Relation to CT. The CT framework of Zheng & Zhou (2020) is primarily focused on measuring the difference between two different distributions, which are referred to as the source and target distributions, respectively. It defines the expected CT cost from the source to target distributions as the forward CT, and that from the target to source as the backward CT. Minimizing the combined backward and forward CT cost, the primary goal is to optimize the target distribution to approximate the source distribution. By contrast, in CACR, since the marginal distributions of x and x+ are the same and these of x and x− are also the same, there is no need to differentiate the transporting directions. In addition, the primary goal of CACR is not to regenerate the data but to learn fθ(·) that can provide good latent representations for downstream tasks.

heading: C EXPERIMENT DETAILS
text: On small-scale datasets, all experiments are conducted on a single GPU, including NVIDIA 1080 Ti and RTX 3090; on large-scale datasets, all experiments are done on 8 Tesla-V100-32G GPUs.

heading: C.1 SMALL-SCALE DATASETS: CIFAR-10, CIFAR-100, AND STL-10
text: For experiments on CIFAR-10, CIFAR-100, and STL-10, we use the following configurations:
• Data Augmentation: We strictly follow the standard data augmentations to construct positive and negative samples introduced in prior works in contrastive learning (Wu et al., 2018; Tian et al., 2019; Hjelm et al., 2018; Bachman et al., 2019; Chuang et al., 2020; He et al., 2020; Wang & Isola, 2020). The augmentations include image resizing, random cropping, flipping, color jittering, and gray-scale conversion. We provide a Pytorch-style augmentation code in Algorithm 1, which is exactly the same as the one used in Wang & Isola (2020).
Algorithm 1 PyTorch-like Augmentation Code on CIFAR-10, CIFAR-100 and STL-10
import torchvision.transforms as transforms
# CIFAR-10 Transformation def transform_cifar10():
return transforms.Compose([ transforms.RandomResizedCrop(32, scale=(0.2, 1)), transforms.RandomHorizontalFlip(),# by default p=0.5 transforms.ColorJitter(0.4, 0.4, 0.4, 0.4), transforms.RandomGrayscale(p=0.2), transforms.ToTensor(), # normalize to value in [0,1] transforms.Normalize(
(0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010),
) ])
# CIFAR-100 Transformation def transform_cifar100():
return transforms.Compose([ transforms.RandomResizedCrop(32, scale=(0.2, 1)), transforms.RandomHorizontalFlip(),# by default p=0.5 transforms.ColorJitter(0.4, 0.4, 0.4, 0.4), transforms.RandomGrayscale(p=0.2), transforms.ToTensor(), # normalize to value in [0,1] transforms.Normalize(
(0.5071, 0.4867, 0.4408), (0.2675, 0.2565, 0.2761),
) ])
# STL-10 Transformation def transform_stl10():
return transforms.Compose([ transforms.RandomResizedCrop(64, scale=(0.08, 1)), transforms.RandomHorizontalFlip(),# by default p=0.5 transforms.ColorJitter(0.4, 0.4, 0.4, 0.4), transforms.RandomGrayscale(p=0.2), transforms.ToTensor(), # normalize to value in [0,1] transforms.Normalize(
(0.4409, 0.4279, 0.3868), (0.2683, 0.2610, 0.2687),
) ])
• Feature Encoder: Following the experiments in Wang & Isola (2020), we use an AlexNetbased encoder as the feature encoder for these three datasets, where encoder architectures are the same as those used in the corresponding experiments in Tian et al. (2019) and Wang

heading: C.2 LARGE-SCALE DATASETS: IMAGENET-100 AND IMAGENET-1K
text: For large-scale datasets, the Imagenet-1K is the standard ImageNet dataset that has about 1.28 million images of 1000 classes. The ImageNet-100 contains randomly selected 100 classes from the standard ImageNet-1K dataset, and the classes used here are the same with Tian et al. (2019) and Wang & Isola (2020), listed in Table 18. For pretraining with less curated data4, considering Webvision v1 and ImageNet-22k respectively have 2.4 and 14.2 million images, we decrease the training epoch to 1/2 and 1/10 when pretraining on these two datasets as done in Li et al. (2021). We follow the standard settings in these works and describe the experiment configurations as follows:
• Data Augmentation: Following Tian et al. (2019; 2020b); Wang & Isola (2020); Chuang et al. (2020); He et al. (2020), and Chen et al. (2020c), the data augmentations are the
1https://github.com/SsnL/align uniform 2https://github.com/joshr17/HCL 3Please refer to the original paper for the specific meanings of the hyper-parameter in baselines. 4https://data.vision.ee.ethz.ch/cvl/webvision/dataset2017.html
Algorithm 2 PyTorch-like Augmentation Code on ImageNet-100 and ImageNet-1K
import torchvision.transforms as transforms # ImageNet-100 and ImageNet-1K Transformation # MoCo v2’s aug: similar to SimCLR https://arxiv.org/abs
/2002.05709 def transform_imagenet():
return transforms.Compose([ transforms.RandomResizedCrop(224, scale=(0.2, 1.)), transforms.RandomApply([transforms.ColorJitter(0.4, 0.4,
0.4, 0.1) ], p=0.8),
transforms.RandomGrayscale(p=0.2), transforms.RandomApply([moco.loader.GaussianBlur([.1, 2.])], p=0.5), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]) ])
same as the standard protocol, including resizing, 1x image cropping, horizontal flipping, color jittering, and gray-scale conversion with specific probability. The full augmentation combination is shown in Algorithm 2.
• Feature Encoder: On these two datasets, we apply the MoCo-v2 framework (Chen et al., 2020c), where the ResNet50 (He et al., 2016) is a commonly chosen feature encoder architecture. The output dimension of the encoder is set as 128.
• Model Optimization: Following the standard setting in He et al. (2020); Chen et al. (2020c); Wang & Isola (2020), the training mini-batch size is set as 128 on ImageNet-100 and 256 on ImageNet-1K. We use a mini-batch stochastic gradient descent (SGD) optimizer with 0.9 momentum and 1e-4 weight decay. The total number of training epochs is set as 200. The learning rate is initialized as 0.03, decayed by a cosine scheduler for MoCo-V2 at epoch 120 and epoch 160. In all experiments, the momentum of updating the offline encoder is 0.999.
• Parameter Setup: On ImageNet-100 and ImageNet-1K, for all methods, the queue size for negative sampling is 65,536. The training batch size is 128 on ImageNet-100 and 256 on ImageNet. For CACR, we train with two positive sampling sizes K = 1 and K = 4 and the parameters in the conditional weight metric are respectively set as t+ = 1.0, t− = 2.0. For baselines, according to their papers and Github pages (Tian et al., 2019; Wang & Isola, 2020; Robinson et al., 2021), the temperature parameter of CL is τ = 0.2, the hyper-parameters of AU-CL are t = 3.0, τ = 0.2, and the hyper-parameters of HN-CL are τ = 0.5, β = 1.0. Note that CMC (K = 1) reported in the main paper is trained with 240 epochs and with its own augmentation methods (Tian et al., 2019). For CMC (K = 4), the temperature is set τ = 0.07 according to the setting in Tian et al. (2019) and the loss is calculated with Equation (8) in the paper, which requires more GPU resources than 8 Tesla-V100-32G GPUs with the setting on ImageNet-1K.
• Linear Classification Evaluation: Following the standard linear classification evaluation (He et al., 2020; Chen et al., 2020c; Wang & Isola, 2020), the pre-trained feature encoders are fixed, and a linear classifier added on top is trained on the train split and test on the validation split. The linear classifier is trained with SGD over 100 epochs, with the momentum as 0.9, the mini-batch size as 256, and the learning rate as 30.0, decayed by a factor of 0.1 at epoch 60 and epoch 80.
• Feature Transferring Evaluation (Detection and Segmentation): The pre-trained models are transferred to various tasks including PASCAL VOC5 and COCO6 datsets. Strictly following the same setting in He et al. (2020), for the detection on Pascal VOC, a Faster R-CNN (Ren et al., 2016) with an R50-C4 backbone is first fine-tuned end-to-end on the VOC 07+12 trainval set and then evaluated on the VOC 07 test set with the COCO suite of
5http://host.robots.ox.ac.uk/pascal/VOC/index.html 6https://cocodataset.org/#download
metrics (Lin et al., 2014). The image scale is [480, 800] pixels during training and 800 at inference. For the detection and segmentation on COCO dataset, a Mask R-CNN (He et al., 2017) with C4 backbone (1x schedule) is applied for the end-to-end fine-tuning. The model is tuned on the train2017 set and evaluate on val2017 set, where the image scale is in [640, 800] pixels during training and is 800 in the inference.
• Estimation of π̂−θ with MoCo-v2: Following the strategy in Wang & Isola (2020), we estimate π̂−θ with not only the cost between queries and keys, but also with the cost between queries. Specifically, at each iteration, let M be the mini-batch size, N be the queue size, {fqi}Mi=1 be the query features, and {fkj}Nj=1 be the key features. The conditional distribution is calculated as:
π̂−θ (·|fqi) = e −d t− (·,fqi )∑N
j=1 e −d t− (fkj ,fqi )+ ∑ j 6=i e −d t− (fqj ,fqi )
To be clear, the Pytorch-like pseudo-code is provided in Algorithm 3. In MoCo-v2 framework, as the keys are produced from the momentum encoder, this estimation could help the main encoder get involved with the gradient from the conditional distribution, which is consistent with the formulation in Section 3.2.
Algorithm 3 PyTorch-like style pseudo-code of CACR with MoCo-v2 at each iteration.
######################## Inputs ###################### # t_pos, t_neg: hyper-parameters in CACR # m: momentum # im_list=[B0, B2, ..., BK] list of mini-batches of length (K+1) # B: mini-batches (Mx3x224x224), M denotes batch_size # encoder_q: main encoder; encoder_k: momentum encoder # queue: dictionary as a queue of N features of keys (dxN); d denotes the feature dimension
######## compute the embeddings of all samples ################# q_list = [encoder_q(im) for im in im_list] # a list of (K+1) queries q: M x d k_list = [encoder_k(im) for im in im_list] stacked_k = torch.stack(k_list, dim=0) # keys k: (K+1) x M x d
################### compute the loss #################### CACR_loss_pos, CACR_loss_neg = 0.0, 0.0 for k in range(len(im_list)): #load a mini-batch with M samples as queries
q = q_list[k] mask = list(range(len(im_list))) mask.pop(k) # the rest mini-batches are used as positive and negative samples
##################### compute the positive cost ##################### # calculate the cost of moving positive samples: M x K cost_for_pos = (q - stacked_k[mask]).norm(p=2, dim=-1).pow(2).transpose(1, 0) # point-to-
point cost with torch.no_grad(): # the calculation involves momentum encoder, so with no grad here.
# calculate the conditional distribution: M x K weights_for_pos = torch.softmax(cost_for_pos.mul(t_pos), dim=1) # calculate the
positive conditional distribution # calculate the positive cost with the empirical mean CACR_loss_pos += (cost_for_pos*weights_for_pos).sum(1).mean()
#################### compute the loss of negative samples ###################### # calculate the cost and weights of negative samples from the queue: M x K sq_dists_for_cost = (2 - 2 * mm(q, queue)) sq_dists_for_weights = sq_dists_for_cost
if with_intra_batch: # compute the distance of negative samples in the mini-batch: Mx(M -1)
intra_batch_sq_dists = torch.norm(q[:,None] - q, dim=-1).pow(2).masked_select(˜torch. eye(q.shape[0], dtype=bool).cuda()).view(q.shape[0], q.shape[0] - 1) # combine the distance of negative samples from the queue and intra-batch: Mx(K+M-1) sq_dists_for_cost = torch.cat([sq_dists_for_cost, intra_batch_sq_dists], dim=1) sq_dists_for_weights = torch.cat([sq_dists_for_weights, intra_batch_sq_dists], dim=1)
# calculate the negative conditional distribution: if with_intra_batch==True Mx(K+M-1), else MxK weights_for_neg = torch.softmax(sq_dists_for_weights.mul(-t_neg), dim=1) # calculate the negative cost with the empirical mean CACR_loss_neg += (sq_dists_for_cost.mul(-1.0)*weights_for_neg).sum(1).mean()
# combine the loss of positive cost and negative cost and update main encoder CACR_loss = CACR_loss_pos/len(im_list)+CACR_loss_neg/len(im_list) CACR_loss.backward() update(encoder_q.params) # SGD update: main encoder encoder_k.params = m*encoder_k.params+(1-m)*encoder_q.params #momentum update: key encoder
# update the dictionary, dequeue and enqueue enqueue(queue, k_list[-1]) # enqueue the current minibatch dequeue(queue) # dequeue the earlist minibatch
pow: power function; mm: matrix multiplication; cat: concatenation.

Review summaries:
This paper proposes a new contrastive attraction and contrastive repulsion (CACR) loss for self-supervised representation learning. By formulating two conditional distributions, CACR considers different importance of positive/negative samples according to their distances to the query sample. Experimental results show superior performance of CACR against traditional contrastive loss and state-of-the-art methods.

This paper aims to improve the conventional contrastive learning (CL) by introducing a CACR loss, which takes the distance-based weights into the consideration for CL loss computation. Theoretical analysis is provided on the robustness property of the representation learned by CACR. Experiments are also presented to evaluate the performance of CACR compared with MoCo V2, SimCLR and other recent CL methods. 

In this paper, the authors present a new contrastive representation learning method. Specifically, a doubly contrastive learning (CL) approach is proposed, in which it contrasts positive and negative samples within themselves separately. The proposed method is termed "contrastive attraction and contrastive repulsion (CACR)". Extensive experimental, as well as theoretical analyses, show the effectiveness and robustness of the proposed method. The main contributions of this work are the proposed CACR strategy and the corresponding experimental analysis.

This paper introduces Contrastive Attraction and Contrastive Repulsion (CACR) loss for self-supervised learning with a doubly contrastive strategy. It constructs two conditional distributions to model the importance of positive samples and negative samples according to their distances to the query. The importance, which is reflected by weights, guides the query to not only more strongly pull more distant positive samples, but also more strongly push away closer negative samples. Theoretical analysis and empirical results show that CACR generalizes the conventional CL loss and is more robust in more general cases. Superior performances are achieved on both curated and uncurated datasets, and on various downstream tasks.