Contrastive Attraction and Contrastive Repulsion for Representation Learning
Authors: Huangjie Zheng
Xu Chen
Jiangchao Yao
Hongxia Yang
Chunyuan Li
Ya Zhang
Hao Zhang
Ivor Tsang
Jingren Zhou
Mingyuan Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: contrastive learning
Bayesian methods
conditional distribution
label imbalance
doubly contrastive

Abstract: Contrastive learning (CL) methods effectively learn data representations without label supervision, where the encoder needs to contrast each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on ImageNet, a well-curated dataset with balanced image classes. However, they tend to yield worse performance when pretrained on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness on uncurated datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion. It further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets in representation learning, but also shows better robustness when generalized to pretrain on wild large image datasets.

Under review as a conference paper at ICLR 2022
CONTRASTIVE ATTRACTION AND CONTRASTIVE RE- PULSION FOR REPRESENTATION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Contrastive learning (CL) methods effectively learn data representations without label supervision, where the encoder contrasts each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising ImageNet, a well-curated dataset with balanced image
results when pretrained classes. However, they tend to yield worse performance when pretrained on images in the wild. In this paper, further improve the performance of CL and enhance its robustness on uncurated datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force attract more distant positive samples but also do so to repel closer negative samples. Thcoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion. further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets in representation learning, but also shows better robustness when generalized to pretrain on imbalanced image datasets.
INTRODUCTION
The conventional Contrastive Learning (CL) loss (Oord et al. 2018 Poole et al. 2018) has achieved remarkable success representation learning, benefiting downstream tasks in a variety of areas (Misra
& Maaten 2020 He et al. 2020 Chen et al. 2020a Fang & Xie 2020 Giorgi et al. 2020 This manyy softmax form make the encoder distinguish the positive sample within multiple negative samples. In image representation learning, this scheme is widely used to encourage the encoder to learn representations that are invariant to unnecessary details in the representation space, for which the unit hypersphere is the most common assumption (Wang cet al., 2017: Davidson et al.) 2018 Hjelm et al., 2018 Tian et al., 2019: Bachman et al. 2019). Meanwhile, the contrast with negative samples is demystified as avoiding the collapse issuc, where the encoder outputs a trivial constant, and uniformly distributing samples on the hypersphere (Wang & Isola 2020). To improve the quality of the contrast, various methods, such large negative
loss typically appears
memory bank (Chen et al.) 2020c), hard negative mining (Chuang et al.) 2020 Kalantidis et al.
2020), and using strong multi-view augmentations (Chen et al. 2020a Tian al. 2019; Caron
ct al. 2020), are proposed and succeed in learning powerful representations. Since the conventional
CL loss achieves the -s-mmnyy contrast with softmax cross-entropy loss, a notable concern
that the contrast could be sensitive to the sampled positive and negative pairs (Saunshi et al. 2019
Chuang al. 2020). Given a sampled query, conventional CL methods usually randomly take one positive sample and multiple negative samples, and equally treat them in softmax cross-entropy form, regardless of how informative they are to the query. The sampled positive pair could make the contrast either easy or difficult, while trivially selecting hard negative pairs could make the pretraining inefficient. Thus in practice, a large negative sample size desired to ensure diverse samples are incorporated to make the contrast more effective (Wu et al. 2021). , and the pretraining becomes less effective in large-scale real-world data, where the label could be noisy and rarely distributed in a balanced manner (Li et al. 2020b 2021).

---

Under review as a conference paper at ICLR 2022
Pootiie
Regative
Pesitive (updated)
Negativn (opdhted)
Pesitvel
Posihhs
Posifie
Hi
12
Negathee
Nogaline
Nogutive
Heguine 4
Conventional Dastrastive Learning
Contrcative Attruciion & Contrastive Repulsion
Figure 1: Comparison of conventional contrastive learning (CL) and the proposed Contrastive Attraction and Contrastive Repulsion (CACR) framework. For conventional CL, given query, the model randomly takes positive sample to form a positive pair and compares against multiple negative pairs, with all samples equally treated. For CACR, using multiple positive and negative pairs, the weight of sample (indicated by point scale) is contrastively computed to allow the query to not only more strongly pull more distant positive samples, but also more strongly push away closer negative samples.
Considering the CL loss aims to train the encoder to distinguish the positive sample from multiple negative samples, an alternative intuition is that the positive samples need to be pulled close, while negative samples need be pushed far away from the given query in the representation space. This motivates the construction of Contrastive Attraction and Contrastive Repulsion (CACR), a doubly CL framework where the positive and negative samples are first contrasted within themselves before getting pulled and pushed from the query, respectively. As shown in Figure I unlike conventional CL. which equally treats samples and pulls/pushes them in the softmax cross-entropy contrast, CACR not only considers moving positive/negative samples close/away, but also models two conditional distributions to guide the movement of different samples. The conditional distributions apply doubly- contt-ccont strategy to compare the positive samples and the negative ones within themselves separately. As an interpretation, if a selected positive sample is far from the query, it indicates the encoder does not sufficiently capture some information. CACR will then assign a higher probability for the query to pull this positive sample. Conversely, if a selected negative sample is too close to the query, it indicates the encoder has difficulty distinguishing them, and CACR will assign higher probability for the query to push away this negative sample. This double-contrast method determines a positive/negative sample easy or hard move, making the optimization more effective. Our main contributions include:
We propose CACR, which achieves contrastive learning and produces useful representations by attracting the positive samples towards the query and repelling the negative samples away from the query, guided by two conditional distributions.
ii) Our theoretical analysis shows that CACR generalizes the conventional CL loss. The conditional distributions help treat the samples differently by modeling the intra-relation of positive/negative samples, which is proved to be important when the datasets are less curated.
iii) Our experiments show that CACR consistently outperforms existing CL methods in a variety of settings, achieving fate-of-the--ee ahe results on various benchmark datasets. Moreover, in the case where the dataset has an imbalanced label distribution, CACR has better robustness and provides consistent better pretraining results than conventional CL. methods.
RELATED WORK
Plenty of unsupervised representation learning (Bengio et al. 2013) methods have been developed to
learn good data representations, e.g., PCA (Tipping & Bishop 1999). RBM (Hinton & Salakhutdinov
2006), VAE (Kingma & Welling 2014). Among them, CL. (Oord et al., 2018) is investigated as lower bound mutual information early-stage (Gutmann & Hyvärinen 2010 Hjelm et al. 2018). Recently, many studies reveal that the effectiveness of CL is not just attributed to the maximization of mutual information (Tschannen et al.) 2019 Tian et al., 2020a), motivating various works to demystify the contrastive learning scheme.
Contrastive representation learning. In vision tasks, SimCLR (Chen et al., 2020a b) studies
extensive augmentations for positive and negative samples and intra- intra-batch- batch- negative sampling. memory bank that caches representations (Wu et al.) 2018) and a momentum update strategy are introduced to enable the use of enormous number of negative samples (He et al. 2020 Chen et al.
2

---

Under review as a conference paper at ICLR 2022
2020c). Tian et al. (2019: 2020b consider the image views in different modalities and minimize the irrelevant mutual information between them. Empirical researches observe the merits of using "hard" negative samples in CL, motivating additional techniques, such as Mixup and adversarial 2020a). In a view that not all negative pairs "true" negatives (Saunshi al.] 2019). Chuang et al.
noise (Bose et al., 2018: Cherian & Aeron 2020; Li et al.
2020p propose decomposition of the data distribution to approximate the true negative distribution. RingCL (Wu et al.) (021) 1) proposes to use "neither too hard nor too easy" negative samples by predefined percentiles, and HN-CL (Robinson) et al. 2021) applies Monte-Carlo sampling for selecting hard negative samples. CL has also been developed in learning representations for text (Logeswaran & Lee 2018), sequential data (Oord et al., 2018, Hénaff et al. 2019), structural data like graphs (Sun et al. [2020a] Li et al. 2019 Hassani Khasahmadi |020 Velickovic et al., (019)), reinforcement learning (Srinivas et al. (020), and few-shot scenarios (Khosla et al. 2020 SSylviin et al.] 2020). Along with the success of these designs, works like Wang & Isola (2020) reveal the contrastive scheme is optimizing the alignment of positive samples and the uniformity of negative pairs in the limit of an infinite number of negative samples. Besides vision tasks, CL methods are widely applied to benefit in a variety of areas such as NLP. graph learning, and cross-modality learning (Misra & Maaten 2020 He et al. 2020 Chen al.
2020d Fang & Xie 2020 Giorgi et al. 2020
Gao et al. 202
Korbar et al. 2018 Jiao et al. 2020
Monfort et al. 2021
Deep representation learning. Besides contrastive methods for representation learning, a variety of relevant methods share the same purpose to learn powerful representation without label supervision. Triplet loss and its extensions consider the wiie similarity between the query and all other samples in the given mini-batch, where the positive/negative similarities need to be maximized/minimizee
(Hoffer& & Ailon 2015; Sohn 2016: Wang et al.) 2019; Sun et al. (0220).. Instead of using both positive and negative pairs, some recent self self-supervised methods require only positive pairs, deploying the spirit of Siamese networks and avoiding the collapse issue with stop gradient or a momentum updating strategy (Chen & He| 2021: Grill et al.] 2020 Caron et al., 2021). In addition. [Zbontar et al. (2021) propose to train the encoder to make positive feature pairs have higher correlation and decrease the corss-correlation in different feature dimensions to avoid the collapse. Another category is based on clustering, Caron et al. (020)) and et al. (2020c) introduce the prototypes proxy and train the encoder by learning to predict the cluster assignment. CACR is closely related to the previous methods, where the conditional probability triplet loss. The CA part could also be naturally combined with the recent self-supervised methods without negative pairs and replace the CR part with stop-gradient or momentum updating to avoid collapse issues.
be regarded re-weighting mechanism in
3 THE PROPOSED APPROACH
In CL, for observations To:M ~ Plaat(()), we commonly assume that each æ; can be transformed in certain ways, with the samples transformed from the same and different data regarded as positive and negative samples, respectively. Specifically, we denote T(x,, 6,) as a random transformation of 2; where 6; ~ p(c) represents the randomness injected into the transformation. In computer vision, G often represents a composition of random cropping, color jitter, Gaussian blurring, etc. For each Vo with query T (x0,,E) Eo), we sample positive pair (xx, x ))), where x + T( e e+), and M negative pairs {xx, x; )h:,, where x, T(x C, ). Denote T E Rt, where R+ == fx as temperature parameter. With encoder fe R + sd-1, where we follow the convention to restrict the learned d-dimensional features with a unit norm, we desire to have similar and distinct representations for positive and negative pairs, respectively, via the contrastive loss
0}.
clo(e) T Tole t/ (oo(w) 10 1/1 + Jolo, 10(w)/ 10
E (ar, M
In
(1)
Note by construction, the positive sample x is independent of x given Dy and the negative samples 2; are independent of æ. Intuitively, this 1-vs-M softmax cross-entropy encourages the encoder to not only pull the representation of a randomly selected positive sample closer to that of the query, but also push the representations of M randomly selected negative samples away from that of the query.
3

---

Under review as a conference paper at ICLR 2022
lu pootice condiiss wrighs
wwg.ineerdiieelllloo.ooeooo vilglo
postie palr
regeits par
sumple
fo
aodedding bypreephom
Janustttor shuring
mohrs
panaeettrr sharing
fo
Rg
Figure 2: Illustration the CACR framework. The encoder extracts features from samples and the conditional distributions help weigh the samples differently given the query, according to the distance of a query x and its contrastive samples x: x & means element-wist multiplication between costs and conditional weights.
3.1 CONTRASTIVE ATTRACTION AND CONTRASTIVE REPULSION
In the same spirit of letting the query attract positive samples and repel negative samples, Contrastive Attraction and Contrastive Repulsion (CACR) directly models the cost of moving from the query to positive/negative samples with a doubly contrastive strategy:
LCACR E tt (-1 (0,))) [<(Fo(=). Ex-n((n) Eg - (-1w)) c(Jo(a)). fo(x ))},
Contrastive Attraction
Contrastive Repulsion
LCA LeR: where we denote ग and π as the conditional distributions of intra-positive contrasts and intra- negative contrasts, respectively, and c(#1.=2) as the point-to-point cost of moving between two vectors 21 and ø2. e.g., the squared Euclidean distance || - z2|1² or the negative inner product -2 ø2. the following we explain the doubly contrastive components with more details.
The intra- vntra-positive contrasts is defined in form of the conditional probability, where the positive samples compete to gain a larger probability to be moved from the query. Here we adapt to CACR Bayesian strategy in Zheng & Zhou (2200), which exploits the combination of an energy-based based likelihood term and a prior distribution, to quantify the difference between two implicit probability distributions given their empirical samples. Specifically, denoting d++ (::-) as distance metric with temperature t+ E we define the conditional probability for moving positive sample x T(xo, et)
Contrastive attraction:
e.g., ..g. (=1,
++ - z212². given a query æ = T T(xo, Eo),
Tel (ar (.) = Aet (at) p(=t p((=t
mz))
Q! [m|)))> =: e fo(a ") p(ax
1o)dr
(3)
@= m0]
where f-(-) is an encoder parameterized by and Q+ is a normalization term. This construction makes it more likely to pull æ towards positive sample that is more distant in their latent represen- tation space. With (3). the contrastive attraction loss LCA measures the expected cost of moving query its positive samples, defined in (2), which more heavily weighs c(fo(x), fo( if fo (x) and (xxx) are further away from each other.
Contrastive repulsion: On the contrary of the contrastive attraction shown in define the conditional probability for moving query x to a negative sample as
(fa()). p(=T)
To
(x:
|m))
(=(x)
Q (x) = fe (o(c)) 1) "a(a")dd""
(4)
where 1- E IR+ the temperature. This construction makes it more likely move query
negative sample that is closer from it in their representation space. With (4), the contrastive repulsion loss LoR measures the expected cost repel negative samples from the query shown in (2). which
more heavily weighs c(fe(z) (z), fe( )) if fe( and fe are closer to each other.
Choice of -(.,,,, det (---) and de (..-). There could be various choices for the point-to-point cost function c(-,-,, distance metric d,+ (+--)in in (3) and de- (---) in (4). Considering the encoder fo outputs normalized vectors on the surface of hypersphere, maximizing the inner product is equivalent to
minimizing squared Euclidean distance. Without loss of generality, we define them as
cx x2) ) = || 22 where 1t, E RR+ There are other choices for c(., :) and we show the ablation study in Appendix B.5
det (z1, = t+ || - 11,
de (z1, x2) t- || - 22 1
4

---

Under review as a conference paper at ICLR 2022
3.2 MINI-BATCH BASED STOCHASTIC OPTIMIZATION
Under the CACR loss as in (2) to make the learning of fa(-) amenable to mini-batch stochastic gradient descent (SGD) based optimization, draw (aqiata, <4) Pdota (x))p(e) for -I.... M and then approximate the distribution of the query using an empirical distribution of M samples as
(xx) M
M Ei
0(æ æ;);
æt Ta ex).
where the 5()) denotes the Dirac delta function. With query æs and c1:K iid p(e). we approximate p(x;)) ) for Eqn. 44) and p(æ:"" aggataa for Eqn. 3 as
f( M Ejft Smy, p(ar a mdaat))
K Ek=1 m a = T(aaaaa cx).
(5)
Note we may improve the accuracy of p(z; (5) by adding previous queries into the support of this empirical distribution. Other more sophisticated ways to construct negative samples (Oord et al.)
2018: He et al. 2020; Khosla et al. 2020) could also be adopted to define p(x; ). We will elaborate these points when describing experiments.
Plugging 5 into 3 and 4 we approximate the conditional distributions with discrete distributions and obtain mini-batch based CACR loss as LCACR LCs + LCR, where
Lox= ==
1 M7
EM EK
(olo Aet (o
x (x.)) fa fo(2;))
LCR: - MI EiS M
Ejtt Eyece 2 Uo (()
4.- (o(e))))
We optimize via SGD using VoLCACR. with the framework instantiated as in Figure
Relation with typical CL loss: As shown in (2). with both the contrastive attraction component and contrastive repulsion com- ponent, CACR loss shares the same intuition of conventional CL in pulling positive samples closer to and pushing negative samples away from the query in their representation space. How- ever, CACR realizes this intuition by introducing the double-contrast strategy on the point-to-poo point moving cost, where the contrasts appear in the intra-comparison within positive and negative samples, respectively. The use of the double-contrast strategy clearly differs the CACR loss in (2) from the conventional CL loss in ([), which typically relies a softmax- based contrast formed with single positive sample and multiple equally-weighe weighted independent negative samples. summary of the comparison between some representative CL losses and CACR shown in Table [
Table I: Comparison with representative CL methods. K and M
denotes the number of positive and negative samples, respectively.
positive Intra Intr-negative
Method
Contrast Loss
conrmstt
contrist
CL. Oard et al
2018
L-vs-M cross-entropy
AU-CL Wang & Sola 2020
I-v9-M cross-emiropy
x
HN-CI.
Rohinson
1-ys-M cross-entrogy
et
(5) x (1-vs-M cross entropy)
X
X
X positive Intra- M-negative
PROPERTY ANALYSIS OF CACR
4.1 ON CONTRASTIVE ATTRACTION
We first analyze the effects w.r.l the positive samples. With contrastive attraction, the property below suggests that the optimal encoder produces representations invariant to the noisy details.
The contrastive altraction loss Lex is optimized if and only if all positive samples of query share the same representation as that query. More specifically, for query x that is transformed from To ~ pdata its positive samples share the same representation with it, which means
Property 1.
fo(x ) So (x: for any x. ~ [xx x. vo)-
(6)
This property coincides with the characteristic (learning invariant representation) of the CL loss in Wang & Isola (2020) when achieving the optima. However, the optimization dynamic in contrastive attraction evolves in the context of at To( | a,,, which is different from that in the CL..
5

---

Under review as a conference paper at ICLR 2022
Lemma 1. Let instantiate c( (x(xx)) -fe(z)) fx) Then, the contrastive attraction loss LCa in 2 can be re-written as
Ea E.g xpp- mb) [- (oo())) (o(
Te (w 10,
plat mo)
which could further reduce the alignment loss Breoreencaa (a) E m, ~p(- mo) 1oppl-oool (- fe( in Wang Isola 2020), iff TT (at
x0) = p(** 20).
Property T and Lemma Jjointly show contrastive attraction CACR and the alignment loss in CL reach the same optima, while working in different sampling mechanism. In practice x and usually independently sampled augmentations in a mini mini-batch, as shown in Section 3.2 which raises gap between the empirical distribution and the true distribution. Our method makes the alignment more efficient by considering the intra-relation of these positive samples to the query.
4.2 ON CONTRASTIVE REPULSSION
Next analyze the effects w.E.t the contribution of negative samples. Wang & Isola (2020) reveal that a perfect encoder will uniformly distribute samples on a hypersphere under an uniform isometric assumption, i.e., for any uniformly sampled æ, x sid p(z). their latent representations and = fo also satisfy p(z) = p(=7) We follow their assumption to analyze contrastive repulsion via the following lemma.
fx)
Lemma 2. Without loss of generality, we define the moving cost and metric in the conditional distribution as c(Z1, 22) d(z1, 22) ||= 22ll2. When we are with an uniform prior, namely p(æ)) = =p(#=) for any x, itd p(x) and p(z) = p(z)) given their lutent representations x = fe(x) and z fe(x ). then optimizing 0 with LCR in (2) is the same as optimizing 0 to minimize the mutual information between and x
x(x;-) Ba=p(=) (. (.1m)
(7)
pl
and is also the same as optimizing O to muximize the conditional differential entropy of x given a:: H(X-1X) = Ex-g(ax) -a- ( (-1=)|= In (x x)].
(8)
Here the minimizer O' of LCR, also that of I-x; whose global minimun zero is atlained iff X and X- are independent, and the equivalent maximum of H[[ |X) indicates the optimization of LeR is essentially aimed towards the uniformity of representation about negative samples.
We notice that one way to reach the optimum suggested in the above lemma is optimizing @ by contrastive repulsion until that for any p(æ), d(fo(v)) fe (o(=") )) is equal for all x ~ no ( x)). This means for any sampled negative samples, their representations are also uniformly distributed after contrastive repulsion. Interestingly, this is consistent with the uniformity property achieved by CL (Wang& & Isola 2020). which connects contrastive repulsion with CL in the perspective of negative sample effects.
Note that, although the above analysis builds upon the uniform isometric assumption, our method actually does not rely it. Here, formalize a more general relation between the contrastive repulsion and the contribution of negative samples in CL without this assumption as follows.
Lemma 3. As the number of negative samples M goes to infinity, the contribution of the negative samples to the CL loss become the Uniformity Loss AU-CL (Wang & Isola (020), termed as Luniform for simplicity. It can be expressed an upper bound of LCR. by adding the mutual information I(X;X-) in
In Ex ppla -pla--e/olze lelle so(w)/T] +I(X;X-) > LCR-
Em=-(())
Contoorm
As shown in Lemma 3 the mutual information I(X;"") helps quantify the difference between Cuniform and LCR. The difference between drawing x ~ To (x x) (in CR) and drawing æ independently in a mini-batch (in CL) is non-trivial long [(X; X-) is non-zero. In practice, this is true almost everywhere since we have to handle the skewed data distribution applications, e.g., the label-shift scenarios (Garg et al. 2020). In this view, CR does not require the representation space to be uniform like CL does, and is more robust to the complex cases through considering the intra-contrastive relation within negative samples.
real-world

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS AND EMPIRICAL ANALYSIS
In this section, we first study the CACR behaviors with small-scale experiments, where we use CIFAR-I0, CIFAR-I00 (Hinton 2007) and create two iaaa-iimbllnceed CIFAR datasets as empirical verification of our theoretical analysis. We mainly compare with representative CL methods, divided into two different categories according to their positive sampling size: K = and K = 4. For methods with single positive sample (K = 1), the baseline methods include the conventional CL loss (Oord et al. 2018), AlignUniform CL loss (AU-CL) (Wang & Isola 2020), and the CL loss with hard negative sampling (HN-CL) (Robinson et al.] 202Ib. In the case of K = 4, we take contrastive multi-view coding (CMC) loss (Tian et al. 2019) (align with our augmentation settings and use augmentation views instead of channels) as the comparison baseline. For fair comparison, we keep for all methods with the same experiment setting including learningrrate, training epochs, etc., but use their best temperature parameters; the mini-batch size for K is divided by from those when K = to make sure the encoder leverages same samples in each iteration.
For large-scale datasets, we use ImageNet- IK (Deng et al. 2009) and compare with the state-of- the- art frameworks (He et al. 2020 Zbontar ct al.) [021 Chen et al., 2020aa] Caron et al.) [020 Grill ct al., 2020; Huynh et al. 2020) on linear probing, where we report the Top-1 validation accuracy on ImageNet-11 KK data. We also report the results of object detectiio/ssemmentiin following the transfer learning protocol. To further justify our analysis, we also leverage two large-scale but label-imbalanced datasets (Webvision vI and ImageNet-22K) for linear probing pretraining, The reported numbers for baselines are from the original papers if available, otherwise we report the best ones reproduced with the settings according their corresponding papers. Please refer to Appendix C for detailed experiment setups.
5.1 STUDIES AND ANALYSIS ON SMALL-SCALE DATASETS
Classification accuracy: To facilitate the analysis, we apply all methods with an AlexNet-based encoder following the setting in Wang & Isola (020). trained in 200 epochs. We pretrained the encoder on regular CIFAR-10/100 data and create inss-imbalanced cases by randomly sampling a certain number of samples from each class with "linear" or "exponentional" rule by following the setting in Kim et al. (2020). Specifically, given a dataset with C' classes, for class E {1,2,... C}. we randomly take samples with proportion & for "linear" rule and proportion exp(L&] for "exponential" rule. For evaluation we keep the standard validation/testing datasets. Thus there is a label-shift between the training and testing data distributions.
Summarized in Table are the results on both regular and -lass-imbalanced datasets. The first two columns show the results pretrained with curated data, where we can observe that in the case of K = 1, where the intra-positive contrast of CACR degenerates, CACR slightly outperforms all CL methods. When K 4, it is interesting to observe obvious boost in performance, where CMC improves CL by around 2-3% while CACR improves CL by around 3-4%, which supports our analysis that CA is helpful when the intra- stra-positive contrast is not degenerated. The right four columns present the linear probing results pretrained with class- inaallieadedceed data, which show all the methods have a performance drop. It is clear that CACR has the least performance decline in most cases. Especially, when K = 4, CACR shows hetter performance robustness due to the characteristic of doubly contrastive within positive and negative samples. For example, in the "exponentional" setting of CIFAR-100, CL and HN-CL drop 12.57% and 10.73%, respectively, while CACR (K = 4) drops 9.24%. It is also interesting observe HN-CL relatively better among the baseline methods. According to Robinson et al. (202I). in HN-CL the negative samples are sampled according to the "hardness" w.E.f. the query samples with an intra-negative contrast. Its loss could converge to CACR (K = 1) with infinite negative samples. This performance gap indicates that directly optimizing the CACR loss could be superior when we have limited number of samples. With this class-imbalanced datasets, we provide the empirical support to our analysis: When the condition in Lemma 2 violated, CACR shows a clearer difference than CL and a better robustness with its unique doubly contrastive strategy within positive and negative samples.
On the effect of CA and CR: To further study the contrasts within positive and negative samples, in each epoch, we calculate the conditional entropy with (8) on every mini-batch of the validation data and take the average across mini- batchs. Then, we illustrate in Figure 3 the evolution of conditional entropy H(--IX w.r.t. the training epoch on regular CIFAR-10 10 and class-imbalanced 10
7

---

Under review as a conference paper at ICLR 2022
Table 2:
The linear classification accuracy (%) of different contrastive objectives on small-scale datasets, pretrained on regular and label-imbalanced CIFAR 10/100 with AlexNet backbone. "Linear" and "Exponentional" indicate the number of samples in cach class are chosen by following linear rule or an exponential rulc, respectively. The performance drops compared with the performance in regular CIFAR data are shown next to each result.
Label imbalance
Regular
Linear
Exponential
Dataset
CIFAR- IO
CIFAR- 100
CIFAR-I0 10
CIFAR- 100
CIFAR- 10
CIFAR-IO0
SimCLR (CL)
83.47
55.41
79.88a.30 so.
52.299.57L
71.74
23.2912.774
AU-CL
83.49
55.31
80.255.14.
52.742.57
71.6211.701
44.3810.03)
HN-CL
83.67
55.87
a0.51
52.72a.14
72. 72.7410.94
45. 13..317773
CACR (K = 1)
83.73
56.52
80.46g.27.
54.122.00
73.02
46.59,.83
CMC (K 4)
85.54
58.64
82.20 340
55.38g.261
48.87g.774 14.771..77
CACR (K~
86.54
59,41
83.62g.92 922
26.912.00
75.8910.651
50.179.243
5.3
5.3
Nats
Nats
Mdvin
MCCC 411
Minit
MCCACR (X)
CK 441)
2.5
2.5
Nats
munif)
IHHN CLI
Munii
28th
MCLL)
HOCACR 1K 11)
HitL3
MEDACI (X- 11)
WAUL-LI
H(AU-CD
5.5
25
100
125
175
200
5.5
50
300
150
175
200
Epoch
Epach
Figure Conditional entropy H(X w.r.f. epoch on CIFAR-I0 (left) and linearly habel-ibbllaneed
CIFAR-10 (right). The maximal possible conditional entropy is indicated by a dotted line.
As shown, H(X-IX) is getting maximized as the encoder is getting optimized, indicating the encoder learns to distinguish the negative samples from given query. is also interesting to observe that in the case with multiple positive samples, this process is much more efficient, where the conditional entropy reaches the possible biggest value rapidly. This implies the CA module can further boost the repulsion of negative samples. From the gap between CACR and CMC, can learn although CMC uses multiple positive in CL loss, the lack of intra-positive contrast shows the gap of repulsion efficiency. In the right panel of Figure 3 the difference between CACR and baseline methods more obvious, where we can find the conditional entropy of baselines is slightly lower than pretrained with regular CIFAR-10 10 data. Especially for vanilla CL loss, we can observe the conditional entropy has a slight decreasing tendency. indicating the encoder hardly learns to distinguish negative samples in this case. Conversely, CACR still shows to remain the conditional entropy at higher level, which explains the robustness shown in Table and indicating superior learning efficiency of CACR. See Appendix B.2 for similar observations on CIFAR- 100 and exponential label-imbalanced cases. that part, we provide more quantitative and qualitative studies on the effects of conditional distributions.
5.2 COMPARISON WITH SOTAS ON LARGE-SCALE DATASETS
For large-scale experiments, following the convention, we adapt CACR loss into the MoCo-v2 framework and pretrain a ResNet50 encoder with mini-batch size 256 on ImageNet-
Linear probing: Table 3 summarizes the results of linear classification, where a linear classifier is trained on ImageNet-11 K on top of fixed representations of the pretrained ResNet50 encoder. Similar to the case on small-scale datasets, CACR consistently shows better performance than the baselines using contrastive loss, improving SimCLR and MoCov2 by 2.7% and 2.2% respectively. Compared with other self-super SOTAS, CACR also shows on par performance.
Label-imbalanced case: To strengthen our analysis on small-scale label-imbalanced data, we specially deploy two real-world, but less curated datasets Webvision v1 and ImageNet-22K that have long-ta tail label distributions for encoder pretraining and evaluate the linear classification accuracy on ImageNet- IK. We pretrain encoder with 100/20 epochs on Webvision v v1/ImageNet-22 and compare with the encoder pretrained with 200 epochs on ImageNet- IK to make sure similar samples
8

---

Under review as a conference paper at ICLR 2022
Table 3: Top-1 classification accuracy (%) compari- son with SOTAs, pretrained with ResNet50 encoder on ImageNet- IK dataset. We mark Top-3 best results in bold and highlight CL methods.
Table 4: Top-1 classification accuracy (%) on ImageNet-IK, with the pre-trained ResNet50 on large- scale regular (200 epochs) and label-imbalanced (100/20 epochs) datasets. The performance drops are shown next to each result.
Methods
Batch-size
Accuracy
BarlowTwie
1024
73.2
Pretrained data Methods
Accuracy
Simsiam
256
71.3
MoCov2
67.5
SWAV (wo/w multi-crop)
4096
71.8 75.3
ImageNet-1K IK
CACR (K=1)
69.5
BYOL
4096
74.3
CACR (K=4)
70.4
SimCLR
4096
71.7
MoCov2
62.35.21
MoCov2
256
72.2
Webvision vl
CACR (K=1)
64.55.01
FNC (w multi-crop)
4096
74.4
CACR (K=4)
66.14.31
MoCov2
59.97.61
CACR (K=l)
256
73.7
ImageNet-22K
CACR (K=1)
61.97.61
CACR (K=4)
256
74.4
CACR (K=4)
64.55.91
Table 5: Results of transferring features to object detection and segmentation task on Pascal VOC, with the pre-trained ResNet50 on ImageNct-1kk kk. Contrastive learning methods are highlighted.
VOC 07+ 12 detection
COCO detection
COCO instance seg.
Data
A0
AP
AP7s
APso
AP
AP7s
AP50
AP
AP7s
scratch
60.2
33.8
33.1
44.0
26.4
27.8
46.9
29.3
30.8
supervised
81.3
53.5
58.8
58.2
38.2
41.2
54.7
33.3
35.2
BYOL
81.4
55.3
61.1
57.8
37.9
40.9
54.3
33.2
35.0
SwAV
81.5
55.4
61.4
57.6
37.6
40.3
54.2
33.1
35.1
SimSiam
82.4
57.0
63.7
59.3
39.2
42.1
56.0
34.4
36.7
Barlow Twins
82.6
56.8
63.4
59.0
39.2
42.5
56.0
34.3
36.5
SimCLR
81.8
55.5
61.4
57.7
37.9
40.9
54.6
33.3
35.3
MoCov2
82.3
57.0
63.3
58.8
39.2
42.5
55.5
34.3
36.6
AU-CL
82.5
57.2
63.8
58.4
39.1
42.2
55.7
34.1
36.3
CACR(K=I)
82.8
57.8
64.2
58.9
39.3
42.5
55.6
34.4
36.7
CACR(K=4)
82.8
57.9
64.9
59.8
40.0
42.7
55.8
35.0
37.0
have been seen in the pretraining. The results shown in Table where outperforms the MoCov2 baseline and shows better robustness when generalized to wild image data.
CACR still
Object detection and segmentation: Besides the linear classification evaluation, following the protocols in previous works (Tian et al. 2019 He et al. 2020 Chen et al., 2020c Wang & Isola 2020),
the pretrained ResNet50 on ImageNet- 1K for object detection and segmentation task on Pascal VOC (Everingham et al.) 2010) and COCO (Lin et al.) 2014) by using detectron2 (Wu et al., 2019). The experimental setting details are shown in Appendix C.2 and kept the same He al. and/Chen et al. (2020c). The test AP, AP50. and AP7s of bounding boxes in object detection and test AP, AP50, and AP75 masks in segmentation are reported in Table We can observe that the performances of CACR is consistently better than baselines using contrastive objectives, and even better than recent sulf-supe learning SOTAs.
(2020) and Chen
6 CONCLUSION
In this paper, we rethink the limitation of conventional contrastive learning (CL) methods that use the contrastive loss but merely consider the intra-relation between samples. We introduce novel Contrastive Attraction and Contrastive Repulsion (CACR) loss with a doubly contrastive strategy, which constructs for two conditional distributions to respectively model the importance of a positive sample and that of a negative sample to the query according to their distances to the query. Our theoretical analysis and empirical results show that optimizing the CACR loss can effectively attract positive samples and repel negative ones from the query as CL intends to do, but more robust in more general cases. Extensive experiments small, large-scale, and imbalanced datasets consistently demonstrate the superiority and robustness of CACR over the state-of the methods in contrastive representation learning and related downstream tasks.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Contrastive learning (CL) is effective in learning data representations without label supervision and has led to notable recent progresses in variety of research areas, such as computer vision. Recently proposed advanced CL methods often require huge amount of data and thus cost large computational energy. Especially in the case where one needs to use multiple positive pairs in the contrast. Instead of contrasting each positive pair over multiple negative pairs with the classic softmax cross-entropy, our work discovers that the contrastive attraction within positives and contrastive repulsion within negatives bring new insight in self-supe representation learning. CACR, which naturally takes multiple positive samples in the contrast without making the contrast complexity become combinatorial in the number of positive pairs, has demonstrated clear improvements existing CL methods. However, the same as existing CL methods, our method is not designed to resist the potential biases existing in the dataset, egg. the false negatives in data. At the current stage, CACR relies on the positive contrast to implicitly alleviate this issue: a false negative sample is repelled far away from the query, in the positive attraction, it will be assigned with larger probability be pulled back. This raises the risk of the quality of learned representations. In the future work, we aim and also encourage other researchers learned representations more robust and powerful.
consider the resistance of these potential risks make the
REPRODUCIBILITY STATEMENT
Our paper provides both theoretical analysis and empirical studies on the proposed algorithm. For theoretical results, to make sure the properties hold and the analysis are correct, we provide step- by-step proof in Appendix [A] For empirical results on both scall-scale and large-scale datasets, we provide detailed experiment settings and Pytorch-style pseudo-code in Appendix C We also include an example code with instruction as supplementary material to ensure the reproducibility.
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances Neural Information Processing Systems, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-182 2013.
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. arXiv preprint 1005::1805.03642, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint (xXiv:2002.05000 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0420 2020c.
10

---

Under review as a conference paper at ICLR 2022
Xu Chen, Ya Zhang, Ivor Tsang, and Yuangang Pan. Learning robust node representations graphs. arXiv preprint 0rXiv:2008.114:0 2020d.
Anoop Cherian and Shuchin Aeron. Representation learning via adversarialll--ootrastiee optimal transport. arXiv preprint arXiv:2007.05840, 2020.
Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical variational auto-encoders. arXiv preprint arXiv: 1oXivv/8044. 000//80440000 00891, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. Icce, 2009.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Hongchao Fang and Pengtao Xie. CERT: Contrastive self-supervised learning for language under- standing, arXiv preprint arXiv:2005. 12766, 2020.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence 2021.
embeddings. arXiv preprint
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary C Lipton. unified view of label shift estimation. arXiv preprint arXiv:2003.07554 2020.
John M Giorgi, Osvald Nitski. Gary D. Bader, and Bo Wang. DeCLUTR: Deep contrastive learning for unsupervised textual representations. ArXiv, abs/2006.03657 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenct in hour. arXiv preprint arXiv:1706.022
2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 33, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Staristics. pp. 297-304, 2010.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning graphs. In International Conference on Machine Learning, pp. 3451-3461, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1 10.1109/CPP..016..999
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint 195.0:190. 2019.
Geoffrey Hinton. Learning multiple layers of representation. Trends in cognitive sciences, 11(10): 428 434, 2007.
11

---

Under review as a conference paper at ICLR 2022
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504- 507, 2006.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Moe -arrhi-Ma Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
Elad Hoffer and Nir Ailon. Deep metrie learning using triplet network. In International workshop on similarity-bal -bssee pattern recognition, pp. 84 94-92. Springer. 2015.
Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting con- trastive self-supervised learning with false negative cancellation. arXiv preprint 1.iv:2011.111000 2020.
Jianbo Jiao, Yifan Cai, Mohammad Alsharid, Lior Drukker, Aris T Papageorghiou, and J Alison Noble. Self-supervised contrastive video-speech representation learning for ultrasound. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 534-534455 543.
Springer, 2020.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprini arXiv:2010.01028. 2(6), 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Yechan Kim, Younkwan Lee, and Moongu Jeon. Imbalanced image classification with complement cross entropy. arXiv preprint arXiv:2009.02/89, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. International Conference Learning Representations, 2014.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronizatinn. arXiv preprint arXiv:1807.00233 2018.
Chunyuan Li, Xiujum Li, Lei Zhang, Baolin Peng, Mingyuan Zhou, and Jianfeng Gao. Self-supervised pre-training with hard examples improves visual representations. arXiv preprint vrXiv:2012.13493 2020a.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arXiv preprint arXiv:2106.09785, 2021.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009. 007995, 2020b.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint 2rXiv:2005.04966, 2020c.
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. In International Conference Machine Learning, pp. 3835 3835-3845, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In International Conference on Learning Representations, 2018.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 17, 2020.
12

---

Under review as a conference paper at ICLR 2022
Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visuee representations from video descriptions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
[4871-14881] 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint drXiv:1807.03748, 2018.
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A Alemi, and George Tucker. On variational lower bounds of mutual information. In NewrIPS Workshop on Bayesian Deep Learning. 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE transactions pattern analysis and machine intelligence, 39(6):1 137-1149, 2016.
Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Confer- ence on Machine Learning, pp. 5628-5637. PMLR, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information processing systems, pp. 1857-1865, 2016.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In International Conference on Machine Learning, pp. 10360-10371, 2020.
Fan-Yur Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. InfoGraph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2020a.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6398-6407, 2020b.
Tristan Sylvain, Linda Petrini, and Devon Hjelm. Locality and compositionallty in zero-shot learning.
In International Conference on Learning Representations, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint /006:0900.04444 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- tional Conference on Learning Representations, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In Advances in Neural Information Processing Systems, 2020b.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61): 1-622, 1999.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint 1rXiv:1907.13625 2019.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, (86))2579-2605 2008.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep Graph Infomax. In International Conference Learning Representations, 2019.
13

---

Under review as a conference paper at ICLR 2022
Feng Wang, Xiang Xiang. Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1041-1049, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Xun Wang. Xintong Han, Weilin Huang, Dengke Dong, and Matthew Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5022-5030, 2019.
Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. In International Conference on Learning Representations, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo. and Ross Girshick. Detectron2. 2019.
Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint
2021.
Huangjie Zheng and Mingyuan Zhou. Exploiting chain rule and bayes' theorem compare probability distributions. arXiv preprint arXiv:2012.14100, 2020.
14

---

Under review as a conference paper at ICLR 2022
Contrastive Attraction and Contrastive Repulsion for Representation Learning: Appendix
A PROOFS AND DETAILED DERIVATION
Proof of Property| /
By definition, the point-to-point cost c(21, 22) is always non-negative. Without
loss of generality, define it with the Euclidean distance. When (6) true, the expected cost of
moving between pair of positive samples, as defined as Lca in (2). will reach its minimum at O.
When not true, by definition we will have LCA > O, i.c., LCA is possible only if (6 is true.
By changing the reference distribution of the expectation from 7T( æ; 370 p(. (o), we can directly re-write the CA loss as:
Proof of Lemma /
LCs Ex F. wa 2..00)
fo(x"))) [
Ean En, pep mn)
(fo(2z) fo(x:)) f((
"6 (ar" mmmo) plat m0)
which complete the proof.
Proof of Lemma
2
Denoting
Z(x)= f e dlloolo))sol f Dpla ydx
we have
Iu (x (x) -d([[((x)) Sa(o Inp(x - In Z(xr).
Thus we have
LCR = Exnp(e) m- (+1=)) (1)=)|1 (x xx) - Imp(x + In (x²)
= C1 H(X XX)
where C1 = Em- B - (.) [mpp( ) and C2 (m=> In (xx). Under the assumption
of uniform prior p(x), C1 becomes term that is not related to 0. Under the assumption of uniform prior on p(z), where z = fo(x), we have
Z(x) = E -p==>((=) d(fo(a).fole (( 1)
Eg <p(z) -0=
x / - -z)
-z)
(s (x= -x)=)==) -x)
VT
(10)
which is also not related to 0. Therefore, under the uniform prior assumption both p(x) and p(z), minimizing LCR is the same as maximizing H(X [X), as well as the same as minimizing x-x
Proof of Lemma 3 sum-exp term:
The CL loss can be decomposed an expected dissimilarity term and a log-
so(() T fo(=+)//
LCI.
efo(z) /o(xt)// E, T
M In cfola) fole + eelo fo(a)/T
fola))) ]
=E(e,at)
[ [o(x))
fo(2)
E
(n te.
15

---

Under review as a conference paper at ICLR 2022
where the positive sample x independent of given ao and the negative samples x; independent of æ. As the number of negative samples goes to infinity, following Wang & Isola (2020), the normalized CL loss is decomposed into the sum of the align loss, which describes the contribution of the positive samples, and the uniform loss, which describes the contribution of the negative samples:
lim LCL- In M (a
So(r)) so(art)
In
Jo (o(wa) (z))
am-pla))
~op(
contribution of positive samples
contribation of nogative samples
With importance sampling, the second term in the RHS of the above equation be further derived into:
fa(m)//
E
In
E
(m)p(())
x
In roo
efol ) fa(m//
p(æ To x (x)
x>p((
Apply the Jensen inequality, the second term decomposed into the negative cost plus a log density ratio:
ef((m))
)o(aa)))o(0)//1 fo(or)/
xovp(#)
ppp(
E
E
fo(x)/T] [fo(x f((x)))
+ E
IE a0 Lonnraa
In
p x
x>>p(x))
20 mao (w
(x)
aomp(())
(e)
(x:
x:)
[fo(x=] fe (z))//
-(X;X-)
Aeme (
m)
Defining the point-to-poing cost function between two unit-norm vectors as a(21,22) = - -27 22 (same as the Euclidean cost since || 22113/2 2, 22 ) we have
eola fo(m))/ I(x;x)
E @mpp(a))
E pp(e)
E m=p((r)
E
(
x ~ To
[x)
(llo(z)))oo(z)))
LCR-
This concludes the relation between the contribution of the negative samples in CL and that in CACR.
16

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide additional results in our experiments, including ablation studies, and corresponding qualitative results.
B.1 ADDITIONAL RESULTS WITH ALEXNET AND RESNET5O ENCODER ON SMALL-SCALL DATASETS
Following benchmark works in contrative learning, we add STL-10 dataset to evaluate CACR in small-scale experiments. As an additional results on small-scale datasets, we test the performance of CACR different encoder backbones. Here we strictly follow the same setting of Wang & Isola (020) and Robinson et al. (202I), and the results are shown in Table[6and 7 We can observe with ResNet50 encoder backbone, CACR with single positive or multiple positive pairs consistently
outperform the baselines. Compared with the results in Table 6 the CACR shows more clear improvement over the CL baselines.
Table 6: The top-1 classification accuracy (%) of different contrastive objectives with SimCLR framework on scall-scale datasets. All methods follow SimCLR setting and apply AlexNet encoder and trained with 200 epochs.
Dataset CIFAR-I0
CL
AU-CL
HN-CL
CACR(K=1)
CMC(K=4) CACR(K=4)
83,47
83.39
83.67
83.73
85.54
86.54
CIFAR-100
55.41
55.31
55.87
56.52
58.64
59.41
STL-10
83.89
84.43
83.27
84.51
84.50
85.59
Table 7: The top-1 classification accuracy (%) of different contrastive objectives with SimCLR framework on small-scale datasets. All methods follow SimCLR setting and apply ResNet50 encoder and trained with 400 epochs.
Dataset
CL
AU-CL HN-CL CACR(K=1)
CMC(K=4) CACR(K=4)
CIFAR-10 10
88.70
88.63
89.02
90.97
90.05
92.89
CIFAR-100
62.00
62.57
62.96
62.98
65.19
66.52
STL-10
84.60
83.81
84.29
88.42
91.40
93.04
B.2 ON THE EFFECTS OF CONDITIONAL DISTRIBUTION
Supplementary studies of CA and CR: As continuous ablation study shown in Figure also conduct similar experiments on CIFAR-100 where we study the evolution of conditional entropy H(X-IX) w.r.f the training epoch. The results are shown in Figure and the results of exponential label-imbalanced data are shown in Figure Similar to the observation on CIFAR-10, shown in Figure 3 we can observe -XX-1X IX) is getting maximized as the encoder is getting optimized with these methods, as suggested in Lemma 2 In the right panel, We can observe baseline methods have lower conditional entropy, which indicates the encoder is less effective in distinguish the nagative samples from query. while CACR consistently provides better performance than the other methods indicating the better robustness of CACR.
As qualitative verification, we randomly take query from mini-batch, and illustrate its positive and negative samples and their conditional probabilities Figure ( As shown, given this query of a dog image, the positive sample with the largest weight contains partial dog information, indicating the encoder to focus on texture information; the negatives with larger weights are more related to the dog category, which encourages the encoder to focus distinguishing these "hard" negative samples. In total, the weights Iearned by CACR enjoy the interpretability compared to the conventional CL.
We study different definition of the conditional distribution. From Table results are not sensitive to the distance space. In addition, as we change प to assign larger probability to closer samples, the results similar to those using single positive pair (K=1). Moreover, the performance drops we change т_ to assign larger probability to more distant negative samples.
can observe that the
Uniform Attraction and Uniform Repulsion: degenerated version of CACR
17

---

Under review as a conference paper at ICLR 2022
5.3-
Nets
3.99
3.9
Munit
MMMC (X 43)
Munnti
NMMCC (X 43)
ICCARR
PNCACR
2.5
6.3
N85
Muni)
5.9
TOF NICACA IK 1M
NGCLI
NICACR IK 13)
NCC
HAU CL
MAU CL
5.5
25
50
75
100
125
150
175
260
5.5
25
50
75
100
125
150
175
200
Epoch
Epech
Figure 4: (Supplementary to Figure 3 Conditional entropy H(X w.E.t training epoch on
CIFAR-100 (left) and linear label-imbalanced CIFAR-100 (right). The maximal possible conditional entropy is indicated by dotted line.
5.3
5.3
Nets
Munit
MCCAR
MYCAR IK 4]1
2.5
2.5
6.3
6.3
Nets
3.9
Munn))
NHH CL)
5.9
Munn
TIFN CL NICACR
NICACR IK
NICU
25
100
150
260
5.5
125
175
Epoch
Epoch
Figure 5: (Supplementary Figure Conditional entropy H(X (X) W.E. training epoch on
exponential label-imbalanced CIFAR-10 (left) and CIFAR-100 (right). The maximal possible conditional entropy is indicated by dotted line.
To reinforce the necessity of the contrasts within positives and negatives before the attraction and repulsion, we introduce a degenerated version of CACR here, where the conditional distributions are forced to be uniform. Remind c(=1 z2) as the point-to-point cost of moving between two vectors 21 and Z2, e.g., the squared Euclidean distance =1 Z2 2212 or the negative inner product - 2 z2. In the same spirit of (I). we have considered a uniform attraction and uniform repulsion (UAUR) without doubly contrasts within positive and negative samples, whose objective is
min [ee (2) Beo.et [c([o(2)) fo(x+))] (g.a-~<(a)) [c(Jo(z), fo(a ))]}-
(1I)
The intuition of UAUR is to minimize/maxiimize the expected cost of moving the representations of positive/negative samples to that of the query, with the costs of all sample pairs being uniformly weighted. While (1) has been proven to be effective for representation learning, our experimental results do not find I to perform well, suggesting that the success of representation learning is guaranteed by uniformly pulling positive samples towards and pushing negative samples away from the query.
Compared to UAUR in [II that uniformly weighs different pairs, CACR distinct in considering the dependency between samples: as the latent-space distance between the query and its positive sample becomes larger, the conditional probability becomes higher, encouraging the encoder to focus more on the alignment of this pair. In the opposite, as the distance between the query and its negative sample becomes smaller, the conditional probability becomes higher, encouraging the encoder to push them away from each other.
Distinction between CACR and UAUR:
In order to further explore the effects of the conditional distribution, we conduct an ablation study to compare the performance of different variants of CACR with/without conditional distributions. Here, we compare 4 configurations of CACR (K 4): (i) CACR with both positive and negative conditional distribution; (ii) CACR without the positive conditional distribution; (iii) CACR without the negative conditional distribution; (iv) CACR without both positive and negative conditional distributions, which refers to UAUR model (see [qquation[T]. As shown in Table 9 when discarding
the positive conditional distribution, the linear classification accuracy slightly drops. As the negative
18

---

Under review as a conference paper at ICLR 2022
Table Linear classification performance (%) of different variants of conditional probability. This experiment is done on CIFAR-10, with K and mini-batch size M 128.
+ (ola)) plar an)
(ole p(ar lao)de
(a(e)
plar
4-- (o "aala""pppee "
86.48
83.91
4,- Maishhlll Ide
8 64, Uplit))00 p(ar
79.46
74.91
Je (f6 p'(("
Data
Top 4 Negative samples
Bottom 4 Negative samples
Query
Positive samples
Figure 6: Illustration of positive/negative samples and their corresponding weights. (Left) For query augmented from original dog image, positive samples are shown, with their weights visualized as the blue distribution. (Right) The sampling weights for negative samples are visualized as the red distribution; we visualize negative samples with the highest and 4 with the lowest weights, with their original images shown below.
conditional distribution is discarded, there is a large performance drop compared the full CACR objective. With the modeling of neither positive nor negative conditional distribution, the UAUR shows a continuous performance drop, suggesting that the success of representation learning not guaranteed by uniformly pulling positive samples closer and pushing negative samples away. The comparison between these CACR variants shows the necessity of the conditional distribution.
Table 9: Linear classification performance (%) of different variants of our method. "CACR" represents the normal CACR configuration, "w/o T means without the positive conditional distribution, "w/o
To means without the negative conditional distribution. "UAUR" indicates the uniform cost (see the model we discussed in Equation TT.. i.e. without both positive and negative conditional distribution.
This experiment is done all small-scale datasets, with K = 4 and mini-batch size M 128.
Methods CIFAR-10 CIFAR- 100
STL-10
CACR
85.94
59.51
85.59
w/o T
85.22
58.74
85.06
w/o Tg
78.49
47.88
72.94
UAUR 77.17
44.24
71.88
As qualitative illustrations, we randomly fix one mini-batch, and randomly select one sample as the query. Then extract the features with the encoder trained with CL loss and CACR (K 1) loss at epochs 1, 20, and 200, and visualize the (four) positives and negatives the embedding space with t-SNE (van der Maaten & Hinton| 2008). For
clear illustration, the middle of the plot and only show samples appearing in the range of [-10, 10) on both æ and axis. The results shown Figure 7cc) from which we can find that as the the encoder getting trained, the positive samples are aligned closer and the negative samples are pushed away for both methods. Compared to the encoder trained with CL, we can observe CACR shows better performance in achieving this goal. Moreover, we can observe the distance between any two data points in the plot more uniform, which confirms that CACR shows better results the maximization of the conditional entropy H(X-IX).
center the query in
B.3 ABLATION STUDY
Does CACR(K > 2) outperform by seeing more samples? To address this concern, in our main paper, we intentionally decrease the mini-batch size M 128. Thus the total number of samples
19

---

Under review as a conference paper at ICLR 2022
540
-23
178
Outry
15
21
(a) Epoch
(b) Epoch 20
(c) Epoch 200
Figure 7: The t-SNE visualization of the latent space at different training epochs, learned by CL loss (top) and CACR loss (bottom). The picked query is marked in green, with its positive samples marked in blue and its negative samples marked in red. The circle with radius t shown as the black dashed line. As the encoder gets trained, we can observe the positive samples are aligned closer the query (Property D. and the conditional differential entropy H(X- X) is progressively maximized, driving the distances d( d() fo( ") towards uniform (Lemma
Table 10: The top-1 classification accuracy (%) of different contrastive objectives with different training epochs on small-scale datasets, following SimCLR setting and applying the AlexNet-based encoder.
Trained with 400 epochs
Trained with 200 epochs
Dataset
CL
AU-CL HN-CI. CACR(KEI)
CMC(K=4)
CACR(K=4)
CIFAR-10
83.61
83.57
83.72
83.86
85.54
86.54
CIFAR-100
55.41
56.07
55.80
56.41
58.64
59.41
STL-10
83,49
83.43
82.41
84.56
84.50
85.59
used per iteration is not greater than those used when == 1. To further justify if the performance boost comes from seeing more samples when using multiple positive pairs, we also let the methods allowing single positive pair train with double epochs. As shown in Table TO we can observe even trained with 400 epochs, the performance of methods using single positive pair still have a gap from those using multiple positive pairs.
On the effects of negative sampling size: We investigate the model performance and robustness with different sampling size by varying the mini-batch size used in the training. On all the small-scale datasets, the classification results are shown in Figure g From this figure, we can see that CACR (K = 4) consistently achieves better performance than other objectives. For example, when mini-batch size 256, CACR (K = 4) outperforms CMC by about 0.4%-1.25 1.2%. CACR (K = 1) shows better performance in most of the cases, while slightly underperforms than the baselines with mini-batch size 64. A possible explanation could be the estimation of the conditional distribution needs more samples to provide good guidance for the encoder.
are applied with size 64, 128, 256, 512, 768 and the corresponding linear
20

---

Under review as a conference paper at ICLR 2022
666
503
300
8030 Ango
135
825
OROM
350
AUCL
223
120
124
CIFAR-10
(b) CIFAR-100
(c) STL-10
Figure 8: The linear classification results of training with different sampling size on small-scale datasets. The training batch size is proportional to the negative sampling size.
On the effects of positive sampling size: We conduct experiments to investigate the model per- formance with different positive sampling size by using different K values the pretraining: K E 2,, 4, 6, 8, 10) on CIFAR-10/100 and K 21, 4} on ImageNet-1KK IK. Similar to experiment setting, in 200 epochs, we apply AlexNet encoder on CIFAR-10 and CIFAR-100 and apply ResNet50 encoder ImageNet-1K. Shown in Figure 9 linear classification accuracy increases accordingly. Correspondingly, we also evaluate the GPU time of CACR loss with different choices of K. shown in Table and Table I6
observe K increases, the
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet
Figure 9: The linear classification results of training with different positive sampling size on CIFAR- 10, CIFAR- 100 and ImageNet- IK. An AlexNet encoder applied on CIFAR-10 and CIFAR- 100; ResNet50 encoder is applied on ImageNet.
Table 11: GPU time (s) per iteration of CACR w.r.t different K CIFAR-10 with AlexNet framework (mini-batch size is 128), tested on Tesla-v 100 GPU.
K
2
6
8
10
GPU time /itrrtttio 0.0021 0.0026 0.0035 0.0045 0.0054 0.0064
On the effects of hyper-parameter /+ IT: Remind in the definition of positive and negative conditional distribution, two hyper-parameters and t- are involved as following:
*t -f p(a= mo) I 110 "1" p(x (
- (f((m) fo1= p(u
T (x xx xx)))) =
To (x
|x)
116 101m "1" p(u da
In this part, investigate the effects of ++ and t" representation learning performance small-scale datasets, with mini-batch size 768 (K = 1) and 128 (K = 4) as an ablation study. We search in range (0.5, 0.7, 0.9, 2.0, 3.0}. The results shown in Table 12 and Table 13
As shown in these two tables, from Table 12, we observe the CACR shows better performance with smaller values for + Especially when / increases to 3.0, the performance drops up to about
21

---

Under review as a conference paper at ICLR 2022
Table 12: The classification accuracy(%) of CACR (K parameters t on small-scale datasets.
4, M
128) with different hyper-
Method
Dataset
0.5
0.7
0.9
1.0
2.0
3.0
CIFAR-10
86.07
85.78
85.90
86.54
84.85
84.76
CACR (K = 4)
CIFAR-100
59.47
59.61
59.41
59.41
57.82
57.55
STL-10
85.90
85.91
85.81
85.59
85.65
85.14
1.9% CIFAR-100. For analysis, since have K positive samples for the computation of positive conditional distribution, using a large value for t could result in an over-sparse conditional distribution, where the conditional probability dominant by one or two positive samples. This also explains why the performance when ++ = 3.0 is close to the classification accuracy of CACR (K' -K=1) 1).
Similarly. from Table 13
we can see that small value for t will lead to the degenerated performance.
Here, since we are using mini-batches of size 768 (K 1) and 128 (K = 4), a small value for t" will flatten the weights of the negative pairs and make the conditional distribution closer to uniform 0.5 is close to those without modeling Ta Based these results, the values of t+ [0.5, 1.0] and t e (0.9,2.0) could be good empirical choices according to our experiment settings on these datasets.
distribution, which explains why the performance when
B.4 ADDITIONAL COMPARISONS
In this part we provide more comparisons with baseline methods. For small-scale experiments, we still compare with contrastive learning methods, conventional CL loss, align-uniform loss, and hard negative sampling CL loss. For large-scale experiments, we continue to compare with contrastive learning loss on ImageNet- 100 and ImagNet-IN 1K with MoCov2 framework, and provide comparisons with SÓTAs pretrained with different epochs.
Training efficiency on small-scale datasets:
On CIFAR-10, CIFAR-100 and STL-10,, we pretrained AlexNet encoder in 200 epochs and save linear classification results with learned representations every 10 epochs. Shown in Figure CACR consistently outperforms the other methods in linear classification with the learned representations at the same epoch, indicating a superior learning efficiency of CACR.
Comparison with contrastive learning methods on ImageNet: For large-scale experiments, fol- lowing the convention, adapt all methods into the MoCo-v2 framework and pre-train train ResNet50 encoder in 200 epochs with mini- mani-bacch size 128/256 on ImageNet- ImageNet-100//magee 100/0000eeeeee----- lk. Table [4] summarizes the results of linear classification on these two large-scale datasets. Similar the small-scale datasets, CACR consistently shows better performance, improving the baselines at least by 1.74% on ImageNet-100 and 0.71% ImageNet- IK. In MoCo-v2, with multiple positive samples, CACR improves the baseline methods by 2.92% on ImageNet- 100 and 2.75% on ImageNet-IKK worth highlighting that the improvement of CACR is more significant on these large-scale datasets, where the data distribution could be much more diverse compared to these small-scale ones. This surprising. as according our theoretical analysis, CACR's s double-contrast within samples enhances the effectiveness of the encoder's optimization. Moreover, we can see CACR (K 1)
is
Table 13: The classification accuracy(%) of CACR (K = 1, M 768) and CACR (K 4, M 128) with different hyper-parameters t small-scale datasets.
Methods
Dataset
0.5
0.7
0.9
1.0
2.0
3.0
CIFAR-10
81.66
82.40
83.07
82.74
83.73
83.11
CACR (K =
CIFAR-100
51.42
52.81
53.36
54.20
56.21
56.52
STL-10
80.37
81.47
84.46
82.16
84.21
84.51
CIFAR-10 10
85.67
86.19
86.54
86.41
85.94
85,69
CACR (K = 4)
CIFAR- 100
58.17
58.63
59.37
59.35
59.41
59.31
STL-10
83.81
84.42
84.71
85.25
85.59
85.41
22

---

Under review as a conference paper at ICLR 2022
375
4GCL
AUCL
D1C55-11
AUCL
CIFAR-10
(b) CIFAR-100
(c) STL-10
Comparison of training efficientcy: Linear classification with learned representations w.z. training epoch on CIFAR-10, CIFAR- 100 and STL-10.
Figure 10:
shows clear improvement over HN-CL. A possible explanation is that although both increasing the negative sample size and selecting hard negatives are proposed to improve the CL loss, the effectiveness of hard negatives is limited when the sampling size increased over a certain limit. As CACR targets to repel the negative samples away, the conditional distribution still efficiently guides the repulsion when the sampling size becomes large.
Table 14: Comparison with contrastive learn- ing methods: Top-1 classification accuracy (%) of different contrastive learning objectives MoCo-v2 framework and ResNet50 encoder, pretrained ImageNet- IK dataset with 200
Table 15: Comparison with state-of-the-an on linear probe classification accuracy, pretrained with different epochs, using ResNet50 encoder backbone on ImageNet- lk.
Epochs
100 200 400 800
1000
epochs. The results from paper or Github page
BYOL
66.5 70.6 73.2
74.3
marked by
Barlow Twins
73.2
Methods
ImageNet- 100
ImageNet- 1K
SWAV
66.5 69.1
70.7
71.8
MoCov2 (CL)
77.54*
67.50"
Simsiam
68.1 70.0
70.8 71.3
AU-CL
77.66*
67.69*
HN-CL
76.34
67.41
SimCLR
66.5 68.3
69.8
70.4 71.7
CACR 1)
79.40
68.40
MoCov2
67.4 69.9
71.0 72.2
CMC (CL, K 4)
78.84
69.45
FNC (multi-crop)
70.4
74.4
CACR (K 4)
80.46
70.35
CACR
68.3 70.4 73.8 74.0 74.4
Comparison with other SOTAs: Besides the methods using contrastive loss, continue to compare with the seff-supervised learning methods like BYOL, SWaV, SimSiam, etc. that do not involve the contrasts with negative samples. T5 provides more detailed comparison with all sate-offtthearde in different epochs and could better support the effectiveness of CACR: We can observe CACR achieves competitive results and generally outperforms most of SOTAS at the same epoch in linear classification tasks. We also compare the computation complexity. Table T6 reports computation complexity to provide quantitative results in terms of positive number K. where we can observe the computation cost of CACR slightly increases as K increase, but does not increase as that when using multi-positives CI. loss.
Table 16: GPU time (s) per iteration of different loss on McCov2 framework, tested on 32G-V100 GPU
Methods
CL
AU-CL IIN-CL CACR(K=I) CL (K=4) CACR(K=2) CACR(K33) CACR(K=4)
Batch size M
256
256
256
256
64
128
64
64
samples (KxM) / iteration
256
256
256
256
256
256
192
256
GPU time (s) iteration
0.837
0.840
0.889
0.871
3.550
0.996
1.017
1.342
B.5 CONNECTION TO OTHER REPRESENTATION LEARNING METHODS
Results of different cost metries
23

---

Under review as a conference paper at ICLR 2022
Recall that the definition of the point-to to-point metric is usually set the quadratic Euclidean distance:
(a So())) = -
(12)
In practice, the cost metric defined our method flexible be any valid metrics, Here, we also investigate the performance when using the Radial Basis Function (RBF) cost metrics:
CRBF (fo()) fo - -( 16
(13)
where / E R+ is the precision of the Gaussian kernel. With this definition of the cost metric, our method is closely related to the baseline method AU-CL (Wang & Isola 2020), where the authors calculate pair-wise RBF cost for the loss w.E.. negative samples. Following Wang & Isola (020), replace the cost metric when calculate the negative repulsion cost with the RBF cost and modify Len as:
ÉCR- NBF In M
ΣΣ
Eyte - (oltoaata (nar(
4 (otte))lolee))
(14)
M Eytgi Uplewhoaee)) -01100000)0110000)01
d,- (olee)
Here the negative cost is in log scale for numerical stability. When using the RBF cost metric, we use the same setting in the previous experiments and evaluate the linear classification all small-scale datasets. The results of using Euclidean and RBF cost metrics are shown in Table 17 From this table, we see that both metrics achieve comparable performance, suggesting the RBF cost is also valid in our framework. In CACR, the cost metric measures the cost of different sample pairs and is not limited on specific formulations. More favorable cost metrics can be explored in the future.
Table 17:
The classification accuracy (%) CACR (K = 1) and CACR (K 4) with different cost metrics on CIFAR-10, CIFAR-100 and STL-10. Euclidean indicates the cost defined in 12 and RBF indicates the cost metrics defined in [3
Methods
Cost Metric CIFAR-10 CIFAR-100 STL-10
Fuclidean
83.73
56.21
83.55
CACR(K- 1)
RBF
83.08
55.90
84.20
Euclidean
85.94
59.41
85.59
CACR(K = 4)
RBF 86.20
58.81
85.80
Discussion:
Relation to triplet loss CACR is also related to the widely used triplet loss (Schroff et al. 2015 Sun et al.) 2020b). A degenerated version of CACR where the conditional distributions all uniform can be viewed as triplet loss, while underperform the proposed CACR, discussed in Section B.2 In the view of triplet loss, CACR is dealing with the margin between expected positive pair similarity and negative similarity:
LCACr = [Ex, (at [c), x+]] - Ex, (=-1w][o(2, x ]] m]+
which degenerates to the generic triplet loss if the conditional distribution degenerates uniform distribution:
LUAUR = [Epeatt)eee (c(x,, x +]] Ep(- (apaa//((a,. [
m]+ = [c(x, c(x, c(2, m]+
This degeneration also highlights the importance of the Bayesian derivation of the conditional distribution. The experimental results of the comparison between CACR and the degenerated uniform version (cquivalent to generic triplct loss) are presented in Table
Moreover, CACR loss can degenerate to triplet loss with hard example mining if m. (x+ (x)) and Tel (x+12) are sufficiently concentrated, where the density shows very sharp peak:
LCAcr [max(c(z,
min(c(æ, x =)) +m]++
which corresponds to the loss shown in Schroff et al. 2015). As shown in Table [2 and [3 when varying 1+ and t- to sharpen/flatten the conditional distributions. Based on observations, when
24

---

Under review as a conference paper at ICLR 2022
* 3 and t- 3, the conditional distributions are dominated by 1-2 samples, where CACR can be regarded as the above-mentions triplet loss, and this triplet loss with hard mining slightly underperforms CACR. From these views, CACR provides a more general form to connect the triplet interesting to notice CACR explains how triplet loss is deployed in the
loss. Meanwhile, self-supervised learning scenario.
Relation to CT. The CT framework Zfeng & Zhou 2020) is primarily focused on measuring the difference between two different distributions, which are referred as the source and target distributions, respectively. It defines the expected CT cost from the source to target distributions as the forward CT. and that from the target to source as the backward CT. Minimizing the combined backward and forward CT cost, the primary goal is to optimize the target distribution to approximate the source distribution. By contrast, in CACR, since the marginal distributions of and a are the same and these of æ and x: are also the same, there is no need to differentiate the transporting directions. In addition, the primary goal of CACR is not to regenerate the data but learn fe that can provide good latent representations for downstream tasks.
25

---

Under review as a conference paper at ICLR 2022
C
EXPERIMENT DETAILS
On small-scale datasets, all experiments are conducted on single GPU, including NVIDIA 1080 Ti and RTX 3090; large-scale datasets, all experiments are done 8
GPUs.
C.1 SMALL-SCALE DATASETS: CIFAR-10, CIFAR- 100, AND STL--10
For experiments on CIFAR-10, 10, CIFAR-100, and STL-10, we use the following configurations:
Data Augmentation: We strictly follow the standard data augmentations to construct positive and negative samples introduced in prior works in contrastive learning (Wu et al. 2018; Tian et al. 2019 Hjelm et al. 2018 Bachman et al. 2019, Chuang et al. 2020 He et al. 2020; Wang & Isola 2020). The augmentations include image resizing, random cropping. flippin,, color jittering, and gray-scale conversion. We provide a Pytorch-style augmentation code in Algorithm which is exactly the same the used in Wang &
Isola 2020).
Algorithm 1 PyTorch-like Augmentation Code CIFAR-10, CIFAR-100 and STL-10
Import torchvision tra.ssffrrmrrrmommmmm as transforms
# CIFAR-10 Transformation
def hat.:/rmmm..nndmoonnssieeeeeooo (32, scale- 1)), htt.://mw....aaoooooooooottaaaaa (), by default p=0.5 0.4, 0.4, 0.4),
cifarl0():
return transforms Composoo
trans forms htw.:/mmwe..noooooooaoaaaaaeeeee normalize to value in 10,11
trans frrms wtw.seooms......ror....... (oTensor trans Nww.aalii..
(9.4914, 0.4822, 0.4465), (C.2023, 0.1994, 0.2010),
1)
i CIFAR-100 Transformation
transform transfor__cifar10001 ci
return transforms.
trans iandommessizssCroppps (32, scale- (0.2, 1)), transforms trarsfoorm P by default p=0.5 transforms oot.://mmm....ootttttotr....... 0.4, 0.4, 0,4),
traasfformm.iandmmrrrsyyyalllll (p-0.)))
Tontssoorrm normalize to value in 10,11
(0.5071, 0.4867, 0.4408), (C.2675, 0.2565, 0.2761),
1)
i STL-10 formatormation
def ciinssfor_sstl0111
return sranstorms.comppoo
transforms hannmmm.nnnnnneeiiaadrooooo (64, scale= 1)), by default p=0.5 transsforme..oooorJtttttttttttte 0.4, 0.4, 0,4),
trans forms ht normalize to value in [0,1]
cransformss ToTensorr) trans www.ormww...mmaiiee..m
(G. 4409, 0.4279, 0.3868), (6.2683, 0.2610, 0.2687),
1)
Feature Encoder: Following the experiments in Wang & Isola (2020), AlexNet- based encoder as the feature encoder for these three datasets, where encoder architectures are the same those used in the corresponding experiments in Tian al. (2019) and Wang
26

---

Under review as a conference paper at ICLR 2022
Table 18: The 100 randomly selected classes from ImageNet forms the ImageNet- 100 dataset. These classes are the same as (Wang & Isola 2020 Tian et al. 2011).
ImageNet- 100 Classes
nO2869837 #01749339
nb2158291
nD210-112
n13031106 m02051331 40151
a01589830 n03062215 701773797
no1735189 207821146
n57753275
n03083c13
no4485082 m02105505 A51333431
nC2788148
no3530642 204415853
n02086910 =02859413
a13040303
n03594734
n02565620 =02059919
401558933
a84433381
n02109047 m04111531
n02g77765 E04429376
n02009229
no197a455 m02106550 m01820546
a51692333
a67714571
n02974303 202114455
nO3785016 n033647736
#0377554E
n02087045
n07836838 h04099369
#015927<1
n83831251
n02701002 m03379051
n02259212 207715133
n53947888
no4026417
n02326432 =03E2731E
a01930_66
aC2113799
no2086240 2038C3388
nO2183362 =04127249
=02089973
n03017168 n02093428 m02804411
a02396427
a$4418357
n02172162 m01729322
n02113978 m03187032
n02089887
mG2113022 n03777754 204218763
a52231437
aC3032252 n0213844_ m021C4029
n03837869 =033194278
#04136333
n03794056 n03492542 m02618207
404067472
a03930630
n03584829 402123045
n04229916 n02100583
n53642808
nD4336792 n03259280 F0211673E
n02108089
nU3424325 n01855672 n02050822
& Isola (020).
Moreover, we also follow the setups in Robinson et al. 2022 and test the performance of CACR with ResNet50 encoder (results shown in Table 7
Model Optimization: We apply the mini-batch SGD with 0.9 momentum and le-4 weight decay. The learning rate is linearly scaled as 0.12 per 256 batch size (Goyal al.) 2017). The optimization is done over 200 epochs, and the learning rate is decayed by factor of 0. at epoch 155, 170, and 185.
Parameter Sctup: On CIFAR-10, CIFAR-100, and STL-10,, we follow Wang & Isola (2020) to set the training batch size as M = 768 for baselines. The hyper-parameters of CL, AU-CL', and HN-CL are set according
the original paper or online codes. Specifically, the temperature parameter of CL is T = 0.19, the hyper-parameters of AU-CL are 2.0, 7 0.19, and the hyper-parameter of HN-CL are 0.5, = 1.0", which shows the best performance according to our tuning. For CMC and CACR with multiple positives, the positive sampling size is K 4. To make sure the performance is not improved by using more samples, the training batch size is set as M = 128. For CACR. in both single and multi-positive sample settings, we set 1 = 1.0 for all small-scale datasets. As for for CACR (K 1), t is 3.0. and 3.0 CIFAR- 10, CIFAIIARIIU 100, and STL-10, respectively. For CACR (K = 4), t is 0.9, 2.0, and 2.0 on CIFAR-10, CIFAR 100, and STI.-10, respectively. For further ablation studies, we test e+ and t- with the search in the range of 0..5 0.7, 0.9, 1.0, 2.0, 3.0), and we test all the methods with several mini-batch sizes M {64, 128, 256, 512, 768).
The feature encoder is trained with the default built-in training set of the datasets. In the evaluation, the feature encoder is frozen, and a linear classifier is trained and tested on the default training set and validation set of each dataset, respectively. Follow- ing Wang & Isola 2020), we train the linear classifier with Adam optimizer over 100 epochs, with 31 = 0.5, 322 0.999,
Evaluation:
10 -8 and 128 the batch size. The initial learning rate is 0.001 and decayed by a factor of 0.2 at epoch 60 and epoch 80. Extracted features from "fc7" employed for the evaluation. For the ResNet50 setting in Robinson al. (2021).
the extracted features are from the encoder backbone with dimension 2048.
C.2 LARGE-SCALE DATASETS: IMAGENET-I06 AND IMAGENET-IK
For large-scale datasets, the Imagenet- 1K the standard ImageNet dataset that has about 1.28 million images of 1000 classes. The ImageNet-100 100 contains randomly selected 100 classes from the standard ImageNet-I1 K dataset, and the classes used here are the same with Tian et al. (2019) and [Wan & Isola (2020), listed in Table For pretraining with less curated data", considering Webvision vl and ImageNet-222
respectively have 2.4 and 14.2 million images, we decrease the training epoch to 1/2 and 1/10 when pretraining these two datasets as done in Li et al. (222)) We follow the standard settings in these works and describe the experiment configurations as follows:
Data Augmentation: Following Tian et al. 2020b 2019
Wang & Isola (2020); Chuang
et al. 2000);; He et al. (2000), and Chen et al. (202c)) the data augmentations are the
http://llluuub.oo/SSaLLLLs
uniform
"https://gthhu..co. http:///ithubb.oo//ssh//// 17/HCL
Plcasse refer to the original paper for the specific meanings of the hyper-parameter in baselines.
https://dataa vision. wwtp:/chccel/llcciiiiiiitttttiii
27

---

Under review as a conference paper at ICLR 2022
Algorithm 2 PyTorch-like Augmentation Code on ImageNet-100 and ImageNet-1K
Import torchvision, wtt.:riisio..ttanor.....m
trans forms
4 ImageNet=1 and ImageNet-1K franstoo
+ MoCo v2' s aug: similar to SimCLR https://arxdvv.rr//aet
/2002.0570 05709
def caaasfrrmiimaenee
return transforms. Compose
transforms .ww.orrms..annnnnnno. RandomResizedCrgg (224, scale= 1.1), transforms Rraasforre..Rnndmngg...... RandonApply (ttaaapoomttttttatoommmooomoo col.rrttterooooooroooooooooo (0.4, 0.4,
0.4, 0.1)
p=0.8),
transforms. htt.:m/..aaccoooaaa/ooooooooeeee
GaussianBlur (I.1, 2.11],
p=0.5), transforms. t wwn.ommoorizonttlllll.....
transforms Transforrss.ooeeseeee ToTensor
transforms Nor.N (mean
(mean= (0. 0.456, 0. 0.406),
std=10.222 0.224, 0.2251)
same as the standard protocol, including resizing, 1x image cropping, horizontal flipping, color jittering, and gray-scale conversion with specific probability. The full augmentation combination is shown in Algorithm 2
Feature Encoder: On these two datasets, we apply the MoCo-v2 framework (Chen et al. 2020c), where the ResNet50 (He et al., 2016) is a commonly chosen feature encoder architecture. The output dimension of the encoder is set as 128.
Following the standard setting in He al. (2020): Chen et al. 2020c): Wang & Isola (2020), the training mini-batch size set as 128 on ImageNet- 100 and 256 on ImageNet- IK. We use mini-batch stochastic gradient descent (SGD) optimizer with 0.9 momentum and le-4 weight decay. The total number of training epochs is set as 200. The learning rate is initialized as 0.03, decayed by cosine scheduler for MoCo-V2 epoch 120 and epoch 160. In all experiments, the momentum of updating the offline encoder is 0.999.
Model Optimization:
Parameter Setup: On ImageNet-100 100 and ImageNet-IK, IK, for all methods, the queue size for negative sampling 65,536. The training batch size is 128 on ImageNet- 100 and 256 on ImageNet. For CACR, we train with two positive sampling sizes K = and K = 4 and the parameters in the conditional weight metric are respectively set ++ 1.0,
2.0. For baselines, according to their papers and Github pages (Tian et al. 2019 Wang & Isola 2020 Robinson et al. 2021), D the temperature parameter of CL is r = 0.2, the hyper-parameters of 0.2, and the sypee--prameter of HN-CL are = 0.5, 3 = 1.0. Note that CMC (K = 1) reported the main paper is trained with 240 epochs and with its augmentation methods (Tian et al.) 2019). For CMC (K 4), the temperature is set T = 0.07 according to the setting in Tian et al. (2019) and the loss is calculated with Equation (8) in the paper, which requires more GPU resources than Tesla-V100-33
AU-CL are ( 3.0,
GPUs with the setting on ImageNet-1K.
Linear Classification Evaluation: Following the standard linear classification evalua- tion (He al. 2020, Chen al. 2020c Wang & Isola 2020), the pre-trained feature encoders are fixed, and linear classifier added on top is trained on the train split and test on the validation split. The linear classifier trained with SGD over 100 epochs, with the momentum as 0.9, the mini-batch size as 256, and the learning rate as 30.0, decayed by factor of 0.1 at epoch 60 and epoch 80.
Feature Transferring Evaluation (Detection and Segmentation): The pre-trained models are transferred to various tasks including PASCAL VOC and COCO datsets. Strictly following the same setting in He et al. (2020), for the detection on Pascal VOC, a Faster R-CNN (Ren al. 2016) with an R50-C4 backbone is first fine-tuned end-to-end the VOC 07+12 trainval set and then evaluated on the VOC 07 test set with the COCO suite of
28

---

Under review as a conference paper at ICLR 2022
2014). The image scale is [480, 800] pixels during training and 800 inference. For the detection and segmentation on COCO dataset, a Mask R-CNN (He et al.) 2017) with C4 backbone (Ix schedule) is applied for the end-to-end fine-tuning. The model is tuned on the train2017 set and evaluate on va12017 set, where the image scale is in [640, 800] pixels during training and is 800 in the inference.
metrics (Lin et al.
Estimation of ft with MoCo-v2: Following the strategy in Wang & Isola (2020) estimate #g with not only the cost between queries and keys, but also with the cost between queries. Specifically, at each iteration, let M be the mini-batch size, N be the queue size,
1 Ji= be the query features, and (Jk, / be the key features. The conditional distribution is calculated as:
# (.15)))
د5 Uky F9z) +Digeee " sfg))
To be clear, the sytorch-like pseudo-code is provided in Algorithm In MoCo-v2 frame- work, as the keys are produced from the momentum encoder, this estimation could help the main encoder get involved with the gradient from the conditional distribution, which is consistent with the formulation in Section 3.2
29

---

Under review as a conference paper at ICLR 2022
Algorithm 3 PyTorch-like style pseudo-code of CACR with MoCo-v2 at each iteration.
АдАенАннНнА Inpuu S
4 t_pos, t_neg: hyger parameters in CACR
momensum
4 im_ [R0,t=RO, R2,
BR) Hat of mini-oatcches of length (K+1)
4 mini batches (8x3x224x224), M denotes batch_aige
encoder_g: main encoder; encoder_ks comentum
queua: diccionary as a queua of features of koya (dxN); donotes the feature dimension
esbeddings 1ist lencoder_ for in in im_liet] 4 1ist of (KI1) queries ez X
**44*44 compute
samples #*4**4ессеен4ся
k_isst [encader_k (im) for im in im list]
stacked_k -crchhssacck (k_ dim=0) keys (K+1)
*сснеtсetttt compute the less НАААНААААА
CACR_ cACR_lass_pos, EACR_loss_nae #Load mini-batch with samples as queries for k in range (in [k]
0,0, 0.0
mask list (range flen (im_list))) mask.pop(k) the rest mint-batches are used positive and negative samples
compute the positive cast **Асннин
calovlate the cast of moving positive samp es: cost cost_for__et pas_ (q stacked_k(mask) normm(p=2 dimm-l) (an=-1).pox22 transpose 0) $ point-ta- encoder,
point cost
torch.no_gzadd
celculation involves
grad
calculate the conditional distribution: M x K.K beighta_tor__ne
torch.sottmax (cost_tor, foost__oo___r__po ttr___oos___ooooe (t_pos), dim=l) ca calouuate the positive conditional distributior.
caleulaze the positive cast with the empirical mean
CACR_ oss_pos
(1) me)n)))())))))
несснн compute the loss of negative samples 4*еес calculate the cast and woights of nogative samples from the quouc: X x s__dista_tor_oon (2 mi queue) ag_distt__oo__weihht sg_dists_for_cost
if with compute the distance of negative samples in the mini-batch: Xx (M
-11
Satra_batch_sse si_tiats = norch.norm//l: Kone] 4, dim- (an--1)..pe poo (2))(())aaee)))) .ww.ked_ seleett ("toreh.
eye (q.shape [0], dsype-boel), cuda()) (q.ppe q.shape [0] combine the distance of segative samples
queue int.ra-barchs xx (X+M-1)
sq co_disss__tor_oo sg_disss_for_weigite
torea.cat [[aq_diata__ ccs_ttst, intra_batch_see sistss,, din=l)
torch. cat [[sg_dists_for__eeigg
intre__atchh_sg_die
dim-1)
ca catovlate the negative conditional. distribution: it else MxK
Mx (K+M-1),
weights__oo for_neg
for wow.wwoighss..iii. (-t_neg), dimml)
calou calotlate the negativg ung cost with the empirica ri mean
CAcR_ioas_neg
laq wog_ddatt_tor_oott.ooo
11.0) meeigats_for__eeg) sum ()ean
combine the loss of positive cost and negative cost and update main encoder cacR_lass = CACR loa___oos__po/ee (in_)ist))CkKKRloa los.loss.bboo backwaaak
(im list)
update (encoder p.aaaamm) SED updace: main encoder enceder_k.pate
= seencoder_k.parane [1-m] vencoder_g.pareee imomentum update: key encoder
update the dietionary, dequeue and enqueue engueue (queue, k k_list[-11 eaqueue the current minibatch. dequoua (queue) doqucue the earlist miribatch
pow: function; mn: matrix multiplication; cat: concatenation.
30


Reviews:
[{"review_id": "s1Mv9WtdGO-", "reviewer": "Reviewer_edUG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n\\+ The authors provide a theoretical analysis of the proposed CACR.\n\n\\+ Extensive experimental analysis validate the effectiveness of the proposed method.\n\n\\+ The proposed CACR is shown to be more robust compared to other existing methods, when evaluated on label-imbalanced data.\n\n\\+ The writing of the paper is generally good.\n\n### Weaknesses\n\n\\- The main contribution of this paper is the \"double-contrast\" strategy. Compare to the traditional contrastive loss, the novelty is a bit limited. Besides, it is unclear why such a \"double-contrast\" is necessary, and why have to use the multiplication weight as a selection approach instead of others.\n\n\\- The proposed method is motivated by the conventional CL's robustness on uncurated datasets, as claimed by the authors. But the experimental analysis does not validate this well. The \"wild\" uncurated data is not only data-imbalance, it is also related to the quality and noise level of the data. As a result, suggest the authors either tone down the claim or provide more convincing justifications, either empirical or theoretical.\n\n\\- Although the authors demonstrate the effectiveness of the proposed method in a controlled experiment (Sec. 5.1), when compared to SOTAs (Sec. 5.2), the performance is not significant, just comparable/incremental to prior works.\n\n\\- It would be better if the authors could present an analysis of the add on computational complexity / number of parameters, by using the proposed CACR strategy.\n\n\\- The proposed method is also related to (though not exactly the same scenario) the hard-negative and hard-positive sampling used in video-audio representation learning (e.g. [*1, *2, *3]) and it would be better to discuss the relationships.\n\n[*1] \"Cooperative learning of audio and video models from self-supervised synchronization\", NeurIPS 2018\n\n[*2] \"Self-supervised contrastive video-speech representation learning for ultrasound\", MICCAI 2020\n\n[*3] \"Spoken moments: Learning joint audio-visual representations from video descriptions\", CVPR 2021.", "summary_of_the_paper": "In this paper, the authors present a new contrastive representation learning method. Specifically, a doubly contrastive learning (CL) approach is proposed, in which it contrasts positive and negative samples within themselves separately. The proposed method is termed \"contrastive attraction and contrastive repulsion (CACR)\". Extensive experimental, as well as theoretical analyses, show the effectiveness and robustness of the proposed method. The main contributions of this work are the proposed CACR strategy and the corresponding experimental analysis.", "summary_of_the_review": "Although the proposed method seems to be an incremental improvement to the conventional contrastive learning, the \"doubly contrastive strategy\" is shown to be effective by performing extensive experimental evaluations. This might be of interest to a group of audiences in ICLR 2022 and contribute to the community. On the other hand, there are some unclear statements I hope the authors could address in rebuttal. As a result, I recommend a marginal acceptance at the current stage and may change my rating after rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "daEEqQ9RJMv", "reviewer": "Reviewer_WK7k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is well organized and easy to understand.\n2. The idea of CACR for self-supervised representation learning is reasonable and shows promising performance.\n3. The experiments and analysis are sufficient.\n\nWeaknesses:\n1. It’s not clear why the conditional distributions should be like Eq. 3&4. Are there any theories? For example, why the probability of “moving it to a positive sample” is high when query is far from its positive pair? More explanations are needed.\n2. It shows that increasing number of positive samples (number of augmentations, controlled by hyperparameter “K”) benefits the performance. It would be better to know the performance curve in a large range of K. Currently only K=1,4 are evaluated.\n", "summary_of_the_paper": "This paper proposes a new contrastive attraction and contrastive repulsion (CACR) loss for self-supervised representation learning. By formulating two conditional distributions, CACR considers different importance of positive/negative samples according to their distances to the query sample. Experimental results show superior performance of CACR against traditional contrastive loss and state-of-the-art methods.", "summary_of_the_review": "This paper is generally well motivated and well written. The experiments are sufficient and the proposed CACR shows impressive performance. My minor concern is better explanations of the conditional distributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "axuv7-0pFA", "reviewer": "Reviewer_qbmC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. CACR models the importance of positive/negative samples, making the optimization more effective.\t\n2. CACR is more robust than conventional contrastive loss, according to the performance on label-imbalanced datasets.\n3. Pretraining representations of CARC consistently perform better than conventional CL methods when finetuned on downstream tasks.\n4. Theoretical analysis, empirical results and ablation studies are sufficient to support its effectiveness.\n\nWeakness:\nThere are some minor problems.\n1.\tI can’t find the definition of \\delta in eq. (5).\n2.\tIn the last sentence on page 1, why both easy and hard negative samples are desired for contrast? What’s the reference?\n", "summary_of_the_paper": "This paper introduces Contrastive Attraction and Contrastive Repulsion (CACR) loss for self-supervised learning with a doubly contrastive strategy. It constructs two conditional distributions to model the importance of positive samples and negative samples according to their distances to the query. The importance, which is reflected by weights, guides the query to not only more strongly pull more distant positive samples, but also more strongly push away closer negative samples. Theoretical analysis and empirical results show that CACR generalizes the conventional CL loss and is more robust in more general cases. Superior performances are achieved on both curated and uncurated datasets, and on various downstream tasks.", "summary_of_the_review": "Though the proposed framework is closely related to previous CL methods on the ideas of multi-crop and hard negatives, it still shows novelty on modeling the intra-relation among positives/negatives. The paper provides detailed theoretical analysis for understanding the method. Experiments and ablation studies are also sufficient to demonstrate its effectiveness. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AbDZFz06E_2", "reviewer": "Reviewer_MXU1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1. The proposed method is intuitively correct. \n\n2. The experiments demonstrate the proposed method works better than SimCLR and MoCo V2. \n\nWeakness:\n\n1. The novelty is limited. \n\n2. The paper is not very easy to follow. \n\n3. From the experiments, the performance advantage of the proposed method over existing CL methods (like BYOL) is not very significant. The compared baseline is not always consistent. The proposed method is compared against SimCLR in some experiments and MoCo in other experiments. ", "summary_of_the_paper": "This paper aims to improve the conventional contrastive learning (CL) by introducing a CACR loss, which takes the distance-based weights into the consideration for CL loss computation. Theoretical analysis is provided on the robustness property of the representation learned by CACR. Experiments are also presented to evaluate the performance of CACR compared with MoCo V2, SimCLR and other recent CL methods. ", "summary_of_the_review": "Overall, I feel this paper makes the presentation of the proposed method over-complicated and the resulted approach has limited novelty. The experiment is not very solid and convincing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission receives mixed ratings initially. Reviewer WK7k stays positive, edUG and qbmC are borderline, and MXU1 is negative. They raise several issues including limited technical contribution, insufficient sota experimental comparison, and detailed theoretical proof. The authors have responded to these issues in the rebuttal. The final ratings of these reviewers do not alter. 

After checking the revised manuscript, all the reviews and responses, the AC feels the review from MXU1 lacks sufficient details and the claim shall be better supported with evidence. On the other hand, the issue of the technical contribution raised by edUG still exists. The modification upon original CL is marginal. Also, the experimental validation, although tested in several configurations, only includes ResNet 50 as the encoder backbone. The sufficient validation upon different scales of CNNs are common strategies used in sota SSL methods (e.g., MoCo, BYOL, SimCLR). Without a thorough evaluation, the effectiveness of CACR on different scales of CNN backbones is in doubt. Meanwhile, the experimental comparison upon sota methods (e.g., BYOL, SwAV+multi crop) does not clearly show the performance advantages of CACR. The small-scale dataset utilized for ablation study (i.e., CIFAR) is not as convincing as ImageNet. 

Overall, the AC feels that the marginal technical contribution is acceptable, but it shall be equipped with thorough experimental validation and thus serves as a fundamental baseline (i.e., potentially benefits the SSL community). Based on the current form, the authors are suggested to further improve the current submission and are welcome to submit for the next venue.
Decision: Reject