Contrastive Attraction and Contrastive Repulsion for Representation Learning
Authors: Huangjie Zheng
Xu Chen
Jiangchao Yao
Hongxia Yang
Chunyuan Li
Ya Zhang
Hao Zhang
Ivor Tsang
Jingren Zhou
Mingyuan Zhou
Venue: Accepted by TMLR ()

Abstract: Contrastive learning (CL) methods effectively learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples via a one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. However, most of them consider the augmented views from the same instance are positive pairs, while views from other instances are negative ones. Such binary partition insufficiently considers the relation between samples and tends to yield worse performance when generalized on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness on various datasets, we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion. It further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. Extensive large-scale experiments on standard vision tasks show that CACR not only consistently outperforms existing CL methods on benchmark datasets, but also shows better robustness when generalized on imbalanced image datasets.

Published in Transactions on Machine Learning Research (07/2023)
Contrastive Attraction and Contrastive Repulsion for Representation Learning
Huangjie Zheng'
Tuanggee.henng@uueee..dl
Deportment of Statistics and Data Science
The University of Teaas at Austin
Xu Chen'
muchen20110@sshuucdd..
Shanghai Jiao Tong University Alibabo. Group
Jiangchao Yao
sunaker@sjtu.edu.co
Cooperative Medianet Innovaion Center, Shanghai Jiao Tong University Shanghas AI Laboratory
Hongxia Yang
hongria.yyng @gmail com
Shonghai Institute for Advanced Study of Zhejiong University (SIAS)
Chunyuan Li
deunguan.k@@nicrrooff.ccm
Microsoft Research, Redmond
Ya Zhang
ya So__ahhn@@@jiuueu....
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University Shanghai Al Labovatory
Hao Zhang
zhanghao01 @éridian. edu.cc
Xidian University
Ivor Tsang STAR Centre for Frontier Al Research (CFAR)
ioo feo__ssnn@efar.coo aettffarrassta....m c-sttaredd..g.. edu.sg
Jingren Zhou
Albbaba Group
Mingyuan Zhou
McCombs School of Business The University of Teaas at Austin
Reviewed on OpenReview: https: openreview. net/ forum? id= f39UIDkuwe
Abstract
Contrastive learning (CL) methods effectively learn data representations in self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples via. one-vs-many softmax cross-entropy loss. By leveraging large amounts of unlabeled image data, recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. However, most of them consider the augmented views from the same instance are positive pairs, while views from other instances are negative ones. Such binary partition insufficiently considers the relation between samples and tends to yield worse performance when generalized on images in the wild. In this paper, to further improve the performance of CL and enhance its robustness various datascts, propose
*The first authors share equal contribution

---

Published in Transactions on Machine Learning Research (07/2023)
doubly CL strategy that separately compares positive and negative samples within their own groups, and then proceeds with a contrast between positive and negative groups. We realize this strategy with contrastive attraction and contrastive repulsion (CACR), which makes the query not only exert a greater force to attract more distant positive samples but also do so to repel closer negative samples. Theoretical analysis reveals that CACR generalizes CL's behavior by positive attraction and negative repulsion, and it further considers the intra-contrastive relation within the positive and negative pairs to narrow the gap between the sampled and true distribution, which is important when datasets are less curated. With our extensive experiments, CACR not only demonstrates good performance on CL benchmarks, but also shows better robustness when generalized on imbalanced image datasets. Code and pre-trained checkpoints are available at https: htopa:///ithub. diggttuub...t com/ Comm/JJehheng/CACRR
1 Introduction
The conventional Contrastive Learning (CL) loss (Oord et al. 2018 Poole et al. 2018 has achieved remarkable success in representation learning, benefiting downstream tasks in a variety areas (Misra & Maaten 2020 He et al.) 2020: Chen et al., 2020a Fang & Xie, 2020 Giorgi et al. 2020). This loss typically appears in tne-vs-man softmax form to make the encoder distinguish the positive sample within multiple negative samples. In image representation learning, this scheme is widely used to encourage the encoder to learn representations that are invariant to unnecessary details in the representation space, for which the unit hypersphere is the most common assumption (Wang et al.) 2017] Davidson et al.) 2018 Hjelm et al.) 2018 Tian et al., 2019; Bachman et al., 2019). Mcanwhile, the contrast with negative samples is demystified as avoiding the collapse issue, where the encoder outputs a trivial constant, and uniformly distributing samples the hypersphere (Wang & Tsola| 2020). Beyond the usage of negative samples, several non-contrastive
methods in parallel considers using momentum encoders, stop gradient operation (Caron et al. 2020
Chen &
He 2021
Chen et al. 2020a Caron et al. 2021), etc.
To improve the quality of the contrast, varions hods such as large negative memory bank (Chen et al., Kalantidis et al. 2020), and using strong or multi-view Caron et al. 2020). are proposed and succoed in learning powerful representations. Since the conventional CL loss achieves the one-vs-many contrast with a softmax cross-entropy loss, notable concern is that the contrast could be sensitive to the sampled positive and negative pairs (Saunshi et al.) 2019 Chuang et al. 2020). Given sampled query, conventional CL methods usually randomly take one positive sample and multiple negative samples, and equally treat them in softmax cross-entropy form, regardless of how informative they are to the query. The sampled positive pair could make the contrast either easy or difficult, while trivially selecting hard negative pairs could make the protraining inefficient, making the pretraining become less effective when generalized to real-world data, where the labels are rarely distributed in a balanced manner (Li et al.) 2020b 2021). In recent studies, a large of negative sample manipulation is proposed to make the contrast more effective, such as ring anncaling (Wu et al.
2020e1, hard negative mining (Chuang et al.) 2020
augmentations (Chen al. 2020a Tian et al. 2019
2021
maximizing margin within negatives (hahh et al. 2022), hard/soft nearest neighbor selection (Dwibedi
et al.
2021 GE et al. 2023)
Considering the CL loss aims to train the encoder to distinguish the positive sample from multiple negative samples, an alternative intuition is that the positive samples need to be pulled close, while negative samples need to be pushed far away from the given query in the representation space. In addition to such a push-pull diagram, the intra-relation within positive and negative samples should also be considered. This motivates us to investigate the CL in view of transport and propose Contrastive Attraction and Contrastive Repulsion (CACR), doubly CL framework where the positive and negative samples are first contrasted within themselves before getting pulled and pushed from the query, respectively. As shown in Figure I unlike conventional CL, which equally treats samples and pulls/pushin pulles/ them in the softmax cross-entropy contrast, CACR not only considers moving positive/negative samples close/ /away, but also models two conditional distributions to guide the movement of different samples. The conditional distributions apply a doubly-contrasti strategy to compare the positive samples and the negative ones within themselves separately. As an interpretation, if a selected positive sample is far from the query, it indicates the encoder does not sufficiently capture
2

---

Published in Transactions on Machine Learning Research (07/2023)
Query
Positive
Negative
Positive (updated)
Negative (updated)
P1
Pl
Query
Positive
Positive
Positive
Negative
Negative
Negative
Negative 4
N4
N2
N4
N5
Conventional Contrastive Learning
Contrastive Attraction Contrastive Repuision
Comparison of conventional contrastive learning (CL) and the proposed Contrastive Attraction and Contrastive Repulsion (CACR) framework. For conventional CL, given a query, the model randomly takes one positive sample to form a positive pair and compares against multiple negative pairs, with all samples equally treated. For CACR, using multiple positive and negative pairs, the weight of sample (indicated by point scale) is contrastively computed to allow the query to not only more strongly pull more distant positive samples, also more strongly push away closer negative samples.
Figure 1:
some information. CACR will then assign higher probability for the query to pull this positive sample. Conversely, if a selected negative sample is too close to the query, it indicates the encoder has difficulty distinguishing them, and CACR will assign a higher probability for the query to push away this negative sample. This double-contrast method contrast positive samples from negative samples, in a context of the relation within positives and negatives. We theoretically analyze CACR is universal under general situations conditions, without the need for modification, and empirically demonstrate the learned representation are more effective and robust in various tasks. Our main contributions include:
i) We propose CACR, which achieves contrastive learning and produces useful representations by attracting the positive samples towards the query and repelling the negative samples away from the query, guided by two conditional distributions.
ii) Our theoretical analysis shows that CACR generalizes the conventional CL loss. The conditional samples,
distributions help treat the samples differently by modeling the intra-relation of which is proved to be important when the datasets are less curated.
iii) Our experiments demonstrate the effectiveness of CACR in a variely of standard CL settings, with both convolutional and mansformer-basee architectures on various benchmark datasets. Moreover, in the case where the dataset has an imbalancod label distribution, CACR has better robustness and provides consistent better pretraining results than conventional CL.
2 Related work
Plenty of unsupervised representation learning (Bengio et al.
2013
methods have been developed learn
good data representations, e.g., PCA (Tipping & Bishop 1999
RBM (Hinton & Salakhutdinov 2006);
VAE (Kingma & Welling 2014). Among them, CL (Oord et al. 2018 investigated as lower bound
of mutual information in early-stage (Gutmann & Hyvärinen 2010 Hjelm et al. 2018 Recently, many studies reveal that the effectiveness of CL is not just attributed to the maximization of mutual information
Tschannen et al. 2019 Tian et al. 2020a). In vision tasks, SimCLR (Chen et al. 2020a b) studies extensive augmentations for positive and negative samples and intra-batch-based negative sampling. memory bank that caches representations (Wu et al., 2018) and a momentum update strategy are introduced to enable the use of enormous number of negative samples (He al. 2020 Chen et al.) 2020c). Tian et al. (2019 2020b
consider the image views in different modalities and minimize the irrelevant mutual information between them. Empirical rescarches obscrve the merits of using "hard" negative samples in CL, motivating additional
techniques, such as Mixup and adversarial noise (Bose et al. 2018 Cherian & Aeron
2020
Li el. al.) 2020a).
CL has also been developed learning representations for text (Logeswaran & Lee
2018
sequential data
(Ordd et al. 2018 Hénaff al. (209)), structural data like graphs (Sun al. [2020a]
al. 2019 Hassani
& Khasahmadi 2020 Velickovic et al. 2019). reinforcement learning (Srinivas al.
2020 downstream
fine-tuning scenarios [Khosla et al. 2020 Sylvain al.] 2020
Cui et. al.
2021
Besides vision tasks, CT.
3

---

Published in Transactions on Machine Learning Research (07/2023)
methods are widely applied to benefit in variety of areas such NLP, graph learning. and cross- modaliy
learning MMsra & Maaten 2020 He et al. 2020 Chen et al. 2020d
Fang & Xie 2020 Giorgi et al. 2020
Gao et al. 2021 Korbar et al. 2018 Jiao et al. 2020; Li & Zhao 2021
Monfort et al. 2021).
In a view that not all negative pairs are "true" negatives (Saunshi et al., 2019). Chuang et al. 2020) propose a decomposition of the data distribution to approximate the true negative distribution. RingCI. (Wu et al. 2021) proposes to use "neither too hard nor too easy" negative samples by predefined percentiles, and HN-CL (Robinson et al., (021]) applies Monte-Carlo sampling for selecting hard negative samples. [hhag et al. selects the top-k important samples based feature similarity. [Shah et al. (2022) selects negatives as the sparse support vectors and optimize in a max-margin manner. Besides, hard /soft nearest neighbor selection are also consider as an effective way to select useful negative samples (Dwibedi et al. 2021 GE et al. 2023) Following works like Wang & Isola 2020), which reveal the contrastive scheme optimizing the alignment of positive samples and keeping the uniformity of negative pairs, instead of abusively using negative pairs, recent self-supervised methods do not necessarily require negative pairs, avoiding the collapse issue with
stop gradient or moment.um updaling stralegy (Chen & He 2021 Grill et. al. 2020 Caron et. al. 2021 In addition, Zbontar& et al. (2021) propose to train the encoder to make positive feature pairs have higher correlation and decrease the cross-correlation in different fcature dimensions to avoid the collapse. Another category is based on clustering, Caron et al.] (2020), Feng & Patras (2022) and Li et al. (2020c) introduce the prototypes as a proxy and train the encoder by learning to predict the cluster assignment. CACR is closely related to the previous methods, and additionally consider the relation within positive samples and negative samples. In our work, we leverage two conditional distribution to describe the relation between both positives and negatives with respect to the query samples.
3 The proposed approach
In CT., for observations Fo.M ~ Pdata (a:), commonly assume that each æ; be transformed in certain ways, with the samples transformed from the same and different data regarded as positive and negative samples, respectively. Specifically, we denote T(x; <) as random transformation of 2,, where 6, ~ p(e) represents the randomness injected into the transformation. In computer vision, 6 often represents a composition of random cropping, color jitter, Gaussian blurring, etc. For cach øo, with quory x = ( co), we sample a positive pair (2r, x+),, where x+ = TT (ean,e*) ++), and M negative pairs {(x, )))); where 2; = T(x,,, Denote R where 1 == fx: x 0}, as a temperature parameter. With encoder fo E + S where follow the convention to restrict the learned d-dimensional features with unit norm, desire to have similar and distinct represenlations for positive and negative pairs, respectively, via the contrastive loss as
(a(e) sol=t)/r
E
In elo(e)T fotut)/ Σ, Sole, So(e)/T
(1)
(00,0 2 .MM
Note by construction, the positive sample at is independent of given 2gg and the negative samples a; are independent of æ. Intuitively, this 1-vs-M softmax cross-entropy encourages the encoder not only pull the representation of randomly selected positive sample closer to that of the query, but also push the representations of M randomly selected negative samples away from that of the query.
3.1 Contrastive attraction and contrastive repulsion
In the same spirit of letting the query attract positive samples and repel negative sam- ples, Contrastive Attraction and Contrastive Repulsion (CACR) directly models the cost of moving from the query to positive/negative samples with a doubly contrastive strategy:
LCACh Benp m.).))) (c(fo(z).
Contrastive Attraction
Empoltar Eng Nxp (11m) [-c(f(z),
Contrastive Repulsion
== LCA + LCR.
(2)
4

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Taas jositine conditional weights fud,, negative conditional
puolive coel
aegulive
queyyposiieeegggi embedking
sample construction
conaliiional absinduuiin l1
contrraniiv
fo
dislance
embbesding lyperrpheere
porameter shoring
oncodor
fa aurri
parameter shartng
distoncc
mettric
Conaiitional abstribution flg
Figure Illustration of the CACR framework. The encoder extracts features from samples and the conditional distributions help weigh the samples differently given the query, according to the distance of a query and its contrastive samples æt, x7. means element-wise multiplication between costs and conditional weights.
where we denote πt and π as the conditional distributions of intra- antra-positive contrasts and intra-negative contrasts, respectively, and c(21, #2) as the point-to-point cost of moving between two vectors 2 and #2, e.g., the squared Euclidean distance || - =22l2 or the negative inner product -z z2. In the following we explain the doubly contrastive components with more details.
Contrastive attraction: The intra-positive contrasts is defined in form of the conditional probabiliiy where the positive samples compete to gain larger probability to be moved from the query. Here we adapt to CACR a Bayesian strategy in Zheng & Zhou (2021), which exploits the combination of an energy-basod likelihood term and a prior distribution, to quantify the difference between two implicit probability distributions given their empirical samples. Specifically, denoting de+ (::)) as a distance metric with temperature /+ E IR+,, e.g., de+ (2), ++1 +||| =2| given query x = =T(xo EQ), we define the conditional probability for moving it to positive sample x= = T(zu, c+)
" (x: ar 20)) = eat (oolo))aaa plp (ta)
@+(a|an ma)
Q+ (x (0)) =: / edit (oo(z) p(æ wo)dx
(3)
where [-()) is encoder parameterized by 0 and Q+ (ar) is normalization term. This construction makes it more likely to pull æ towards positive sample that is more distant in their latent representation space. With equation 3 the contrastive attraction loss LCA measures the cost of moving query to its positive samples, as defined in equation 2 which more heavily weighs cáfe(m), f((((t) ) if fe(x) and fe(æ+) are further away from cach other, providing flexible distributions in hard-positive selection [Wang et al. 2020).
Contrastive repulsion: On the contrary of the contrastive attraction shown in equation we define the conditional probability for moving query æ to a negative sample as
To (x (x) = - (ol( pp(x=)
Q-
(x) = / e-d, (ool Dp( - ddx
(4)
where 8- + is the lemperature. This construction makes more likely to move query æ to a negative sample that is closer from it in their representation space. With equation 4 the contrastive repulsion loss LeR measures the expected cost to repel negative samples from the query shown in equation 2 which more heavily weighs () fo( ) if fe( and fe(x-) are closer to each other. To this sense, the distribution To (ar æ)) also assigns larger weights hard-negatives (Robinson et al. 2021
Choice of e( ) det (+ and d.(..). There could be various choices for the point-to-poins point cost function c(.,)), distance metric det (-..) in equation 3 and d--(-.) in equation 4 Considering the encoder fo outputs normalized vectors the surface of hypersphere, maximizing the inner product is equivalent to minimizing
5

---

Published in Transactions on Machine Learning Research (07/2023)
Table 1: Comparison with representative CL methods. K and M denotes the number of positive and negative
samples, respectively.
Intra- straa-positive Intra negative
Method
Contrast Loss
contrast
contrast
CL
Chen et al.
2020a
1-vs-M cross-entropy
AU-CL
Wang & Isola
2020
1-vs-M cross-entropy
X
HN-CL
Robinson et al.
2021
1-vs-M cross-entropy
CMC
Tian et al.
2019
x (1-vs-M cross-entropy)
X
X
CACR (ours)
Intra- K-positive vs Intra-M-negative
squared Euclidean distance. Without loss of generality, we define them as
c(z, y) = | -yl12
de (x, y) ++1 *|a y1l
1 E RR+
d,- (ax, y) = 1Ia y E R4.
where It, R There are other choices for c(.,)) and we show the ablation study in Appendix B.5
3.2
Mini-batch based stochastic optimization
Under the CACR loss as in equation 2 to make the learning of fe(-) amenable to mini-batch stochastic gradient descent (SGD) based optimization, we draw (adaat, ..) ~ Paata (x)p(<) for i = 1....,M M and then approximate the distribution of the query using an empirical distribution
samples as
p(x) =
1 M En 8(Φ x,): x; = T(x:fata e.).
where the 8()) denotes the Dirac delta function, and we note dR, as the Dirac function centered at æ; iee. 8z. 8(x Zz). With query æ; and 61:K equation with at Tamglasa (x):
sid p(e), we approximate p(x,)) ) for equation and p(z,"' zatltay for
p(x,) = WHT Ejft Bay, p(x adata)
1 K
Ek 6 0zt
(5)
Note we may improve the accuracy of p(x;)) in equation by adding previous queries into the support of this empirical distribution. Other more sophisticated ways to construct negative samples (Oord et al.) 2018 He
et al. 2020 Khosla et al. 2020) could also be adopted to define p(x;") We will elaborate these points when describing experiments.
Plugging equation into equation and equation we approximate the conditional distributions with discrete distributions and obtain mini-batch based CACR loss as LCACr Lea + L where
LCs = 1 M K All Aolessse M Ei=l k=
det Uolerroott xc(fola;;) fo
ΣK_
ECR: = 1J EA ijfi E/er Uoleswotejo x (([ fo(#;)).
d (ates))
We optimize 0 via SGD using VoLCACR, with the framework instantiated as Figure 2
3.3 Relation with typical CL loss
As shown in equation 2 with both the contrastive attraction component and contrastive repulsion component, CACR loss shares the same intuition of conventional CL (Oord al.) 2018 Chen et al., 2020a) in pulling positive samples closer and pushing negative samples away from the query in their representation space. However, CACR realizes this intuition by introducing the double-contrast. strategy the point-to-point.

---

Published in Transactions on Machine Learning Research (07/2023)
moving cost, where the contrasts appear in the intra-comparison within positive and negative samples, respectively. The use of the double-contrast strategy clearly differs the CACR loss in equation 2 from the conventional CL loss in equation | which typically relies on beltmax--aseed contrast formed with a single positive sample and multiple equally--eightted independent negative samples. The conditional distributions in CA and CR loss also provide a more flexible way to deal with hard-positive/mmee /negativ samples (Robinson) et al., 2021 Wang et al., 2020 2019 Tabassum et al. 2022] Xu al. 2022) and does not require heavy labor in tuning the syper-parameters for the model. A summary of the comparison between some representative
CL losses and CACR is shown in Table !
4 Property analysis of CACR
4.1 On contrastive attraction
We first analyze the effects w.r.l the positive samples. With contrastive attraction, the property below suggests that the optimal encoder produces representations invariant to the noisy details.
Properly 1. The cont rastive attraction loss LcA is optimized if and only if all positive samples query share the same representation as that query. More specifically, for query æ that is transformed from To ~ Pdata (z), its positive samples share the same representation with it, which means
fo(x*) )- = fo(x) for any xt ~ + x1,
(6)
This property coincides with the characteristic (learning invariant representation) of the CL loss in Wang & Isola (2020) when achieving the optima. However, the optimization dynamic in contrastive attraction evolves in the context of æ ~ Tx( x. v0); which is different from that in the CL.
Lemma 4.1. Let instantiate ((f((z), fo fxxxx)) fo(x)) fo(x Then, the contrastive attraction loss LCa in equation 2 can be re-urillen as
(fo(()) fo(x ) (== am
Exg E. -p(-1 mo)
p(ac 1o0
which could further reduce to the alignment loss Rxarrppaea (2) Be 2 xp 220) (o fo(x [=(((t]][((x{]]] in Wang & Isola 2020), iff क (a+ xx, 20) = p(x* [x)). (xo).
Property and Lemma 4.1 jointly show contrastive altraction in CACR and the alignment loss in CL reach the same optima, while working in different sampling mechanism. In practice at and æ are usually independently sampled augmentations in a mini-batch, as shown in Section 3.2 which raises a gap between the empirical distribution and the true distribution. Our method makes the alignment more efficient by considering the intra-relation of these positive samples to the query.
4.2
On contrastive repulsion
Next we analyze the effects w.r.t. the contribution of negative samples. Wang & Isola (2020) reveal that perfect encoder will uniformly distribute samples on a hypersphere under an uniform isometric assumption, i.e., for any uniformly sampled x, x ~ p(æ), their latent representations z = fe()) and =- = fo(* also satisfy p(z) = p(=-). We follow their assumption to analyze contrastive repulsion via the following lemma. Lemma 4.2. Without loss of generality, we define the moving cost and metric in the conditional distribution as o( k2) = d(z1, 22) = || - 22113. When we are with an uniform prior, namely p(x) = =p(x)) for any x, x sid p(x) and p(z) p(2 given their latent representations z = fo(x)) and =- = fo then optimizing 0 with LeR equation the same as optimizing O minimize the mutual information between and x=
sid
I(X:X-) Em=p(()) E.g B
Naj x (.1x)) In " ( plæ
x))
(7)
7

---

Published in Transactions on Machine Learning Research (07 (07/2023)
and is also the same as optimizing 0 to marimize the conditional differreniaal entropy of x given
H(X (x) = Beeplat) B ~ag ( mi In mo (a
[x)]
(8)
Here the minimizer 0" of LeR is also that of T(X; whose global mindmum zero is attained iff X and X are independent, and the equivalent masimum of H(X- (X) indicates the oplimization of LCR is essentially aimed lowards the uniformily of representation. aboul negative samples.
We notice that one way reach the optimum suggested in the above lemma is optimizing 0 by contrastive repulsion until that for any æ p(z), fo(x is equal for all x= ~ Tg (- |z). This means for any sampled negative samples, their representations are also uniformly distributed after contrastive repulsion. Interestingly, this is consistent with the uniformity property achieved by CL (Wang & Isola 2020), which connects contrastive repulsion with CL in the perspective of negative sample effects.
Note that, although the above analysis builds upon the uniform isometric assumption, our method actually does not rely on it. Here, we formalize a more general relation between the contrastive repulsion and the contribution of negative samples in CL without this assumption follows.
Lemma 4.3. As the number of negalive samples M goes to infinity, the contribution of the negalive samples to the CL loss becomes the Uniformity Loss in AU-CL [Wang & Isola 2020), termed as Luntform for simplicity. It can be expressed as an upper bound of LCR by adding the mutual information I(X:X-) in equation 7
Ew-p(a) In Stee~pla---pe peloo-)elol so(a)/TT +I(X; > LCR,
Lusitorm
As shown in Temma 4.3 the mutual information T(X; X-) helps quantify the difference between Luniform and CCR. The difference between drawing x ~ Tg (a x)) (in CR) and drawing x independently in a mini- batch (in CL) is non-trivial as long I(X;X-) is non-zero. In practice, this is true almost everywhere since we have to handle the skewed data distribution in real-world applications, e.g., the label-shift scenarios (Garg ltt al. 2020). In this view, CR does not require the representation space to be uniform like CL does, and is more robust to the complex cases through considering the intra-contrastive relation within negative samples.
5 Experiments and empirical analysis
In this section, we first study the CACR behaviors with small-scale experiments, where we use CIFAR-10, CIFAR-100 (Hinton) 2007) and create two class-imi imsalancelnn CIFAR datasets as empirical verification of our theoretical analysis. We mainly compare with representative CL methods, divided into two different. categories according to their positive sampling size: K and K 4. For methods with a single positive sample (K = 1), the bascline methods include the conventional CL loss (Oord et al. 2018), AlignUniform CL loss (AU-CL) (Wang& & Isola, 2020), and the CL loss with hard negative sampling (HN-CL) (Robinson et al. 2021). In the case of K = 4, we take contrastive multi-view coding (CMC) loss (Tian et al. 2019) (align with our augmentation settings and use augmentation views instead of channels) as the comparison baseline. For a fair comparison, we keep for all methods with the same experiment setting including learning-rate training epochs, efc., but use their best temperature parameters; the mini-batch size for = is divided by from those when K to make sure the encoder leverages same samples in cach itcration.
For large-scale datasets, we use ImageNet- 1K (Deng al. 2009 and compare with the state-of-the-art frameworks (He et al. 2020 Zbontar et al. 2021 Chen et al. 2020a Caron et al., 2020 Grill et al. 2020 Huynh et al. (0200) on linear probing, where we report the Top-1 validation accuracy on ImageNet- data. We also report the results of object
seretiiinn/emmenaaiion following the transfer learning protocol. To further justify our analysis, we also leverage two large-scale but
datasets (Webvision vI and ImageNet-22K) for lincar probing pretraining. The reported numbers for basclines are from the original papers available, otherwise we report the best ones reproduced with the settings according to their corresponding papers. Please refer to Appendix C for detailed experiment setups.
8

---

Published in Transactions on Machine Learning Research (07/202) 2023)
5.3
5.3
Nats
Nats
3.9
3.9
Alunn))
MCCMC (K 431
Alunif)
NLCMC (K 41)
MCCACR 4))
MYCACR (X= 4))
2.5
2.5
6.3
Nats
Nats
5.9
Munii)
XHH--CL MYCACR
5.9
Muuinn
XHHN-C NICACR
NICL)
11)
HICL)
NIAU-CL)
7HAU-CC
5.5
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Epoch
Epoch
Figure 3: Conditional entropy IX) w.r.t. epoch CIFAR-10 (left) and linearly label-imbalanced CIFAR-10
(right). The maximal possible conditional entropy is marked by a dotted line.
5.1 Studies and analysis on small-scale datasets
Classification accuracy: To facilitate the analysis, we apply all methods with an AlexNet-based encoder following the setting in Wang & Isola (2020), trained 200 epochs. We pretrained the encoder on regular CIFAR-10/100 data and create class-imbalandee
cases by randomly sampling certain number of samples from each class with a "lincar" or "exponentional" rule by following the sctting in Kim et al.) (2020). Specifically, given dataset with C classes, for class 1 E 21, C}2 we randomly take samples with proportion Lt] for "lincar" rule and proportion exp(1 [) for "cxponential" rule. For evaluation we keep the standard validation/testtig datascts. Thus there is a label-shift between the training and testing data distributions.
Summarized in Table 2 are the results on both regular and class- iaaasimmallaceed datasets. The first two columns show the results pretrained with curated data, where can observe that in the case of K = 1, where the intra-positive contrast of CACR degenerates, CACR slightly outperforms all CL methods. When K = 4, it is interesting to observe an obvious boost in performance, where CMC improves CT. by around 2-3% while CACR improves CL by around 3-4%, which supports our analysis that CA is helpful when the intra-positive contrast is not degenerated. The right four columns present the linear probing results pretrained with clas--mmalaanee data, which show all the methods have performance drop. It is clear that CACR has the least performance decline in most cases. Especially, when K K=4 4, CACR shows better performance robustness due to the characteristic of doubly contrastive within positive and negative samples. For example, in the "exponentional" setting of CIFAR-100 CL and HN-CL drop 12.57% and 10.73%, respectively, while CACR (K = 4) drops 9.24%. It is also interesting to observe HN-CL is relatively better among the baseline methods. According to Robinson et al. (2021). in HN-CL the negative samples are sampled according to the "hardness" w.r.l. the query samples with an int nngratiee contrast. Its loss could converge to CACR (K = 1) with infinite negative samples. This performance gap indicates that directly optimizing the CACR loss could be superior when we have limited number of samples. With this class-imbalan iassssaballnnedd datascts, we provide the empirical support to our analysis: When the condition in Lemma 4.2 is violated, CACR shows clearer difference than CL and better robustness with its unique doubly contrastive strategy within positive and negative samples.
On the effect of CA and CR: To further study the contrasts within positive and negative samples, in each epoch, we calculate the conditional entropy with equation g on every mini-batch of the validation data and take the average across mini-batches. Then, we illustrate Figure 3 the evolution of conditional entropy - w.r.t. the training epoch regular CIFAR-10 and dass-imbalanned CIFAR-10. As shown, H (X) is getting maximized the encoder is getting optimized, indicating the encoder learns to distinguish the negative samples from given query. It is also interesting to observe that in the case with multiple positive samples, this process is much more efficient, where the conditional entropy reaches the possible biggest value rapidly. This implies the CA module can further boost the repulsion of negative samples. From the gap between CACR and CMC, we can learn although CMC uses multiple positive in CL loss, the lack of instavpositive contrast shows the gap of attraction efficiency. In the right panel of Figure 3 the difference between CACR and baseline methods more obvious, where we can find the conditional entropy
9

---

Published in Transactions on Machine Learning Research (07/2023)
Table 2: The linear classification accuracy (%) of different contrastive objectives on small-scale datasets, pretrained regular and label-imbalanced CIFAR10/ 100 with AlexNet backbone. "Linear" and "Exponentional" indicate the number samples in each class are chosen by following a linear rule or an exponential rule, respectively. The performance drops compared with the performance in regular CIFAR data are shown next to each result.
Label inbalance
Regular
Linear
Exponential
Dataset
CIFAR-10 CIFAR-
CIFAR-10
CIFAR-100
CIFAR-10
CIF
SimCLR (CL)
83.47
55.41
79.883.593
52.293.574
71.7411.734
43.2912.574
AU-CL HN-CL CACR (K = 1)
83.49
55.31
80.253.14. 14.
52.742.574
71.6211.70)
14.3810.934
83.67
55.87
80.513.154
52.723.141
72.7410.931
45. 45.1310.731
83.73
56.52
80.463.27J
54.122.40;
73.0210.714
46.59,.90 9.8
CMC (K- = 4)
85.54
58.64
82.20,.34J
55.383.261
10.77
98.87
CACR (K =) 4)
86.54
59.41
83.622.921
56.912.501
15.89
50.179.211
Table The top-1 classification accuracy (%) different contrastive objectives with different training epochs on small-scale datasets, following SimCLR setting and applying the AlexNet-based encoder.
Trained with 400 epochs
Trained with 200 epochs
Dataset
CL
AU-CL HN-CL CACR(K-1)
CMC(K-4) CACR(K-4)
CIFAR-10
83.61
83.57
83.72
83.86
85.54
86.54
CIFAR-100
55.41
56.07
55.80
56.41
58.64
59.41
STL-10
83.49
83.43
82.41
84.56
84.50
85.59
of baselines is slightly lower than pretrained with regular CIFAR-10 data. Especially for vanilla CL loss, we can observe the conditional entropy has a slight decreasing tendency, indicating the encoder hardly learns to distinguish negative samples in this case. Conversely, CACR still shows to remain the conditional entropy at a higher level, which explains the robustness shown in Table 2 and indicating a superior learning efficiency of CACR. See Appendix [B22 for similar observations on CIFAR-100 and exponential label-imbalanced cases. In that part, we provide more quantitative and qualitative studies the effects of conditional distributions.
Does CACR(K > 2) outperform by seeing more samples? To address this concern, in our main paper, we intentionally decrease the mini-batch size as M- = 128. Thus the total number of samples used per iteration is not greater than those used when K ==1, To further justify if the performance boost comes from seeing more samples when using multiple positive pairs, we also let the methods allowing single positive pair train with double epochs. As shown in Table we can observe even trained with 400 epochs, the performance of methods using single positive pair still have a gap from those using multiple positive pairs.
5.2 Experiments on large-scale datasets
For large-scale experiments, we follow the self-supervised evaluation pipeline to examine the performance of CACR: we first leverage MoCov2 Chen et al. (2020c) design to pre-train a ResNet-50 with CACR loss, and then evaluate the capacity of the pre-trained model in variety of tasks, including linear probing, downstream few/full-shot image classification, and doteetioon/semmenaatiine Besides these tasks, additional ablation studies are provided in Appendix B
4 summarizes the results of linear classification, where a linear classifier is trained on ImageNet-1K on top of fixed representations of the pretrained ResNet50 encoder. Similar to the case on small-scale datasets, CACR consistently shows better performance than the baselines using contrastive loss, improving SimCLR and MoCov2 by 2.7% and 2.2% respectively. Compared with other non-contrastive self-supervised SOTAs, CACR also shows on par performance.
Linear probing: Table
analysis small-scale label-imbalanced data, we specially deploy two rcal-world, but less curated datasets Webvision v1 and ImageNet-221K that have long-tail label distributions for encoder pretraining and evaluate the linear classification accuracy on ImageNet- 1K. We pretrain encoder with 100/20 epochs on Webvision v1 >1/ImageNet-221 and compare with the encoder pretrained with 200 epochs on ImageNet-1K to make sure similar samples have been seen in the pretraining. The results are shown in Table 3 where we can see CACR still outperforms the MoCov2 bascline and shows better robustness when generalized to wild image data.
Label-imbalanced case: To strengthen
10

---

Published in Transactions on Machine Learning Research (07 2203))
Table 4: Top-1 classification accuracy (%) comparison with SOTAs including non-contrastive and contrastive methods, pretrained with ResNet50 encoder on ImageNet-1K dataset. We mark Top-3 best results in bold and highlight CL methods.
Table 5: Top-1 classification accuracy (%) on ImageNet-IK, with the pre-trained ResNet50 on large-scale regular (200 epochs) and label- imbalanced (100/20 epochs) datasets. The
Methods
Batch-size
Accuracy
performance drops are shown next to each result.
Barlow
1024
73.2
Con-Contrastive
Simsiam
256
71.3
Pretrained data Methods
Accuracy
SWAV (wo/w multi-crop)
4096
71.8 / 75.3
MoCov2
67.5
(wo. Negalives)
BYOL
4096
74.3
ImageNet- 1K
CACR (K-1)
69.5
SimCLR
4096
71.7
CACR (K=4)
70.4
MoCov2
256
72.2
MoCov2
62.3s.21
Contrastive
ASCL
256
71.5
Webvision vl
CACR (K-1)
64..5.01
FNC (w multi-crop)
1096
74.4
CACR (K-4)
66.14.31
ADACLR
1096
72.3
MaCov2
59.97.61
(w. Negatives)
CACR (K=1)
256
73.7
ImageNet-227
CACR (K=1)
61.97.01
CACR (K=4)
256
74.7
CACR (K=4)
64.5%.9)
Datovet
Coltoch101
CIPARIO
CIFAR100
Commtyy211
Dear Doori Texture
EureSAT
PER2013
FGVC Aireran
Food101
GTSRD
HatetulMemes
KITTI
MNIST
Oxford Plawers
Oxford Pata
PatchC
Reatlered 58'12
R1838C445
Stanford Cans
VOC2007
Mean Aec.
Wins
SFT
MoCovv
73.7
70.3
17.4
45.6
00.0
13.5
7.2
27.6
16.5
50.8
43.5
18.1
65.7
77.1
50.9
50.7
58.2
11.2
25.7
39.3
4
CACR
84.8
67.6
24.3
2.5
51.2
73,6
23.0
21.4
17.0
23.7
51.8
45.4
44.0
81.1
79.4
56.4
51.2
49.1
10.8
69.0
46.5
16
Galas
+11.1
2.7
+6.9
+0.2
45.66
+-3.6
+9.5
+14.2
10.6
47.2
+1.0
+1.9
+25.9
+15.4
+2.3
47.5
+0.4
+-43.3
+7.2
G-LP
MoCava
78.5
57.1
77.1
20.5
31.4
48.7
64.1
79.5
76.2
54.7
50.0
61.1
47.9
50.2
CACR
79.3
57.1
76.1
18.3
40.9
32.9
50.3
69.2
84.6
81.2
56.9
51.8
61.7
21.1
74.4
54.1
Gains
+0.09
4-2.4
40.0
-1.0
-2.2
4-0.8
4-4.8
4-0.2
4-8.6
4-5.1
4-5.1
4.5.0
4-2.2
4-1.8
4-0.6
4-7.7
4-26.5
4-8.9
FulkFr
MoClvv3
96.1
66.7
71.3
97.3
68.3
98.8
80.5
90.6
90.9
52.5
86.6
77.6
77.7
3
CACR
58.1
72.0
97.7
68.3
99.1
80.6
98.1
89.3
91.6
56.6
79.1
76.4
78.8
15
Gaims
+0.0
+00.0
+1.2
+1.2
40.77
40.4
4-0.0
+-4.4
+1.1
+-0.
4-0.3
4.0.1
-0.5
4-2.2
4.0.7
-3.3
+-4.1
4-0.2
+11.2
-2.2
+1.11
Full-P LP
MoCova
92.1
96.9
85.3
13.7
73.1
95.9
60.1
78.7
53.7
68.8
98.4
91.4
86.7
57.1
86.3
81.7
74.9
CACR
96.4
77.9
98.6
56.5
63.4
75.3
Gains
+0088
+0.0
0.2
0.4
+1.0
+0.5
0.3
0.2
+0.6
0.8
+0.8
0.7 +0.2
43.4
+1.2
1.5
0.6
+0.4
41.1
+1.7
40.4
Table 6: Comparison of CACR and MoCov3 pre-trained ViT-B/16 encoder on ELEVATER benchmark (Li et al. 2022) We conduct fine-tuning (FT) and linear-probing (LP) in both 5-shot (top 2 rows) and full-show (bottom 2 rows) on 20 datasets. We calculate the gains marked in green for positive results. The mean score and number of wins are reported in the last two columns.
Downstream image classification: To measure the efficiency in adapting the pre- pre-trained model to a wide range of downstream data-sets (Kornblith et al. 2021), we employ the recently developed ELEVATER benchmark (Li et al., 2022) to consider both 5-shot and full-shot transfer lcarning setting: the pre-trained ViT-B/16 evaluated with fine-tuning and linear probing on 20 public image classification data sels, where for each data set 5 training samples are randomly selected in 5-shot setting, otherwise all data are used to train the model for 50 epochs before the test score is reported, and 3 random seeds are considered for each data set. We deploy the automatic hyper-parameter tuning pipeline implemented in ELEVATER that searches for the best parameter for each model to make a fair fine-tuning and linear probing comparison of pre-trained models. The original metrics of each dataset are used with more details provided in Li et al. (2022) and Appendix C To measure the overall performance, we consider the average scores over 20 datasets, and # Wins' indicates the number of data sets on which the current. model outperforms its counterpart. As shown in Table G we obscrve in 5-shot scenarios lincar probing tends to outperform fine-tuning, likely due to the model being heavily adapted to the pre-training data, thus making it less flexible for new tasks. Despite such challenge, we can still observe CACR preserves better transferability with significant gain. As the amount of task-specific data increases, we observe that fine-tuning starts to outperform lincar probing, and CACR still outperforms MoCov3, even though the gap between these two methods become smaller. Overall, in both settings, CACR outperforms MoCov3 in 75% of the rownstteam datasets, indicating the representation efficiency of transferring in downstream applications.
Object detection and segmentation: Besides the linear classification evaluation, following the protocols
in previous works Tian et al. 2019 He et al. 2020 Chen et al. 2020c Wang & Isola, 2020), we use the
11

---

Published in Transactions on Machine Learning Research (07 (203))
Table Results of transferring features to object detection and segmentation task on Pascal VOC, with the pre-trained ResNet50 on ImageNet-lk, lk. Contrastive learning methods are highlighted.
VOC 07+12 detection
COCO detection
COCO instance seg.
Method
A0
AP
AP
AP50
AP
AP7s
APso
AP
AP75
scratch
60.2
33.8
33.1
44.0
26.4
27.8
46.9
29.3
30.8
supervised
81.3
53.5
58.8
58.2
38.2
41.2
54.7
33.3
35.2
BYOL
81.4
55.3
61.1
57.8
37.9
40.9
54.3
33.2
35.0
Non-Contrastive
SwAV
81.5
55.4
61.4
57.6
37.6
40.3
54.2
33.1
35.1
SimSiam
82.4
57.0
63.7
59.3
39.2
42.1
56.0
34.4
36.7
(wo. Negatives)
Barlow Twins
82.6
56.8
63.4
59.0
39.2
42.5
56.0
34.3
36.5
SimCLR
81.8
55.5
61.4
57.7
37.9
40.9
54.6
33.3
35.3
Contrastive
MoCov2
82.3
57.0
63.3
58.8
39.2
42.5
55.5
34.3
36.6
AU-CL
82.5
57.2
63.8
58.4
39.1
42.2
55.7
34.1
36.3
(w. Negatives)
CA
82.8
57.8
64.2
58.9
39.3
42.5
55.6
34.4
36.7
CAC)(K=4)
82.8
57.9
64.9
59.8
40.0
42.7
55.8
35.0
37.0
pretrained ResNet50 ImageNet-11k 1K for object detection and segmentation task on Pascal VOC (Everingham
ct al. 2010) and COCO (Lin ct al. 2014) by using detectron2 (Wu ct al. 2019). The experimental sctting details are shown in Appendix C.2 and kept. the same as He et al. (2020) and Chen et al. (2020ch. The test AP, A550, and AP7s of bounding boxes in object detection and test AP, AP50, and AP of masks in
We can observe that the performances of CACR are consistently better than baselines using contrastive objectives, and better than non-contrastive ceff-supervied learning SOTAs.
segmentation are reported in Table 7
6 Conclusion
In this paper, we rethink the limitation of conventional contrastive learning (CL) methods that use the contrastive loss but merely consider the intra-relation between samples. In the spirit of a distributional transport between positive and negative samples, we introduce Contrastive Attraction and Contrastive Repulsion (CACR) loss with a doubly contrastive strategy, which constructs for two conditional distributions to respectively model the importance of a positive sample and that of a negative sample to the query according to their distances to the query. Our theoretical analysis and empirical results show that the CACR loss can effectively attract positive samples and repel negative ones from the query as CT. intends to do, but is more robust in more general cases. Extensive experiments on small, large-scale, and imbalanced datasets consistently demonstrate the superiority and robustness of CACR over the stato-of-the-art mothods in contrastive representation learning and related rownstream tasks.
Broader Impact Statement
Contrastive learning (CL) is effective in learning data representations without label supervision and has led to notable recent. progresses in variety of research areas, such as computer vision. Recently proposed advanced CL methods often require a huge amount of data and thus cost large computational energy. Especially in the case where one needs to use multiple positive pairs in the contrast. Instead of contrasting each positive pair over multiple negative pairs with the classic softmax cross-entropy, our work discovers that the contrastive attraction within positives and contrastive repulsion within negatives bring new insight in self-supervised representation learning. CACR, which naturally takes multiple positive samples in the contrast without making the contrast complexity become combinatorial in the number of positive pairs, has demonstrated method is not clear improvements over existing CL methods. However, the same existing CL methods, designed to resist the potential biases existing in the datasct, e.g. the false negatives in data. At the current stage, CACR relies on the positive contrast to implicitly alleviate this issue: a false negative sample is repelled too far away from the query, in the positive attraction, will be assigned with larger probability to be pulled back. This raises the risk of the quality of learned representations. In the future work, we aim and also encourage other researchers to consider the resistance of these potential risks to make the learned representations more robust. and powerful.
12

---

Published in Transactions on Machine Learning Research (07 2022)
Acknowledgments
H. Zheng and M. Zhou acknowledge the support of NSF-IIS 2212418 and TACC.
References
Philip Bachmaan, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pottern analysis and machine intelligence, 35(8):1798-1828, 2013.
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. arXiv preprint arXiv:1505.03643 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances in Newral Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. arXiv preprint arXiv:2002.0577 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. orXiv preprint orXiv:2006.1002: 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese reprosentation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04.20 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. Tn Proceedings of the IEFE/CV International Conference on Computer Vision, pp. 9640-994000 16009969, 2021.
Xu Chen, Ya Zhang, Tvor Tsang, and Yuangang Pan. Learning robust node representations on graphs. arXio preprint 2rxi::2008.114 11416, 2020d.
Anoop Cherian and Shuchin Aeron. Representation learning via adversariaally-contrssti optimal transport. arXiv preprint arXix:2007.055 05840, 2020.
Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715 715-724, 2021.
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.008: 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference computer vision and pottern recognition, pp. 248- 488-255. Lece, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. Xiv preprint ar 201::2000. 1929,, 2020.
13

---

Published in Transactions on Machine Learning Research (07/2023)
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588 9597, 2021.
Mark Everingham, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International journal of computer vision, 88(2):303-385 -3888,33,- 2010.
Hongchao Fang and Pengtao Xie. CERT: Contrastive self-supervised learning for language understanding. arXiv preprint vrXiv:2005.27666 2020.
Chen Feng and Ioannis Patras. Adaptive soft contrastive learning. In 2022 26th International Conference on Pollern Recognition (ICPR), pp. 2721 2727. IEEE, 2022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence emboddings. arXiv preprinl ar xrx:22204 088211 2021.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary C Lipton. A unified view of label shift. estimation. XrXiv preprint arXio:2003.075:: 2020.
Chongjian GE, Jiangliu Wang, Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo. Soft neighbors are positive supporters in contrastive visual reprosentation lcarning. In The Eleventh International Conference on Learning Representations, 2023. URL
http:/iwwwwweeeeeeeeeeee seet/forun?id=19vhh PaUkz
John M Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang. DeCLUTR: Deep contrastive learning for unsupervised textual representations. ArXiv, Abs/2006.03659, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv: 100. 1706002677 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020.
Michael Gutmann and Aapo Hyvärinen. roise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Kavch Hassani and Amir Hoscin Khasahmadi. Contrastive multi-view reprosentation lcarning on graphs. International Conference on Machine Learning, pp. 3451 3461, 2020.
K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pp. 770-778, 2016. doi: 10. 1009//999P..0000
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IFFE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2020.
Olivier J Hénaff, Ali Razavi, Carl Docrsch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.099 099272, 2019.
Geoffrey E Hinton. Learning multiple layers of representation. Trends in cognitive sciences, 11(10):428 434, 2007.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(678)):504-55 507, 2006.
14

---

Published in Transactions on Machine Learning Research (07/2023)
R. Devon Hjelm, Alex Fedorov, Samuel Mavoie-Marchilddos Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. arXiv preprint arXit:2011.1176; 2020.
Jianbo Jiao, Yifan Cai, Mohammad Alsharid, Lior Drukker, Aris T Papagoorghion, and J Alison Noble. Self-supervised contrastive video-speech representation learning for ultrasound. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 534-543. Springer, 2020.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.0100 2(6), 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Yechan Kim, Younkwan Lee, and Moongu Jeon. Imbalanced image classification with complement cross entropy. arXiv preprint arXiv:2009.00 02189, 2020.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. International Conference Learning Representations, 2014.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. arXiv preprint Xrx 00230, 2018
Simon Korublith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead less transferable features? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Curran
Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. Associates, Inc., 2021.
Chunyuan Li, Xiujun Li, Lei Zhang, Baolin Peng, Mingyuan Zhou, and Jianfeng Gao. Self-supervised pre-training with hard examples improves visual representations. arXiv preprint ar Xiv:22012.13493, 2020a.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mci Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arxie preprint ur urXiv:2100 0et.:1856.9755, 2021.
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoli Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating segguagg-augmenter visual models. arXiv preprint Xii::2204008900 2022.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. XrXiv preprint arXim:2009.0790 2020b.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXim:2005.04p0 2020c.
Yante Li and Guoying Zhao. Intra- and inter-contrastive learning for micro-expression action unit detection. In Proceedings of the 2021 International Conference on Multimodal Interaction, ICMI '21, pp. 702-706, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384810.
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. In International Conference on Machine Learning, pp. 2019.
Tsung-Y: Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer mision, pp. 740-755. Springer, 2014.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. In International Conference on Learning Representalions, 2018.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14871 14881, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.
arxiv preprinl arxiv: 807.03748, 2018.
Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A Alemi, and George Tucker. On variational lower bounds of mutual information. In NewrIPS Workshop on Bayesian Deep Learning, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748 8763.77.. PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6): 1137-1149, 2016.
Joshua David Robinson, Ching-Yau Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning, In International Conference Muchine Learning, pp. 5628 0628-5667. PMLR, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 815 823, 2015.
2022. Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-margin contrastive learning. In Proceedings of the AAAT Conference Arlificial Intelligence, volume 36, pp.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 10360-10371, 2020.
Fan- Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. InfoGraph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2020a.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 6398-6407, 2020b.
Tristan Sylvain, Linda Petrini, and Devon Hjelm. Locality and compositionality in zero-shot learning. In International Conference on Learning Representations, 2020.
Afrina Tabassum, Muntasir Wahed, Hoda Eldardiry, and Ismini Lourentzou. Hard negative sampling strategies for contrastive representation learning. arXiv preprint arXiv:2206.01 01197, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 15:6:990005844 2019.
Contrastive multiview coding.
arXiv preprint
16

---

Published in Transactions on Machine Learning Research (07/2023)
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Regresentations, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In Advances in Newral Information Processing Systems, 2020b.
Michael E. Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society; Series B (Statistical Methodology), 61(3):611 622, 1999.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint #rXiv:107.13666 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, (186))2579-2606 2008
Petar Velickovic, William Fedus, William L Hamilton, Pietro Lio, Yoshua Bengio, and Devon Hjelm. Deep Graph Infomax. In International Conference on Learning Represenlations, 2019.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Mullimedia, pp- 1041-1049, 2017.
Feng Wang, Huaping Liu, Di Guo, and Sun Fuchum. Unsupervised representation learning by invariance propagation. Advances in Neural Information Processing Systems, 33:3510-3520, 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pp. 9929-9939. PMLR, 2020.
Xun Wang, Xinlong Han, Weilin Huang, Dengke Dong, and Matthew Scolt. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5022-5022000 20300000.440 2019.
Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. In International Conference on Learning Representations, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. 2019.
https:
//ithub.ccom http.////ccccbooreeeerr//atttttt
Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non- non-paraettric instance discrimination. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2018.
Lanling Xu, Jianxun Lian, Wayne Xin Zhao, Ming Gong, Linjun Shou, Daxin Jiang, Xing Xie, and Ji-Rong Wen. Negative sampling for contrastive represcntation learning: A review. orXiv preprint arXi::22060002 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint 2rXic:2108.03230 2021.
Shaofeng Zhang, Junchi Yan, and Xiaokang Yang. Adaptive contrastive learning of representation by nearest. positive expansion.
Huangjie Zheng and Mingyuan Zhou. Exploiting chain rule and bayes' theorem to compare probability distributions. Advances in Neural Information Processing Systems, 34, 2021.
17

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Appendix
A Proofs and detailed derivation
Proof of Property 7 By definition, the point-to-point cost c(21, 22) is always non-negative. Without loss of generalily, define it with the Euclidean distance. When equation true, the expected cost of moving between pair of positive samples, as defined CCa in equation 2 will reach its minimum at 0. When
equation
B is not true, by definition we will have LCA > 0, i.e., LCa 0 is possible only if equation
is
true.
Proof of Lemma 4.1 By changing the reference distribution of the expectation from TØ x. (.) to p(. Øo), we can directly re-wrile the CA loss as:
LCA = Eeexp(a)) F Bottonn (m,)))) [c(So(ar), So fo(xx))))
Egg Eg, (-)(1 ( (xxxt) vo (wo a.00)
p(at m0)
which complete the proof.
Proof of Lemma
4.
Denoting
Z(x)= = f e -dlloll))oo ppp ddx
have
In ng (x (x)) -(So(a)), So(x Inp(a - In 2(z))
Thus we have
LCR = Ex-p( B
Ba- --N Na, (-1z) (11 (x xx) - InI((xx))) + In Z(z)
= + + C2 - H(x X)
(9)
where C1 = F ~ng ( (-1=) [In p(æ )] and C2 E Ex)p(a) In 7(x). Under the assumption of a uniform prior on p(z), Ci becomes term that is not related to O. Under the assumption of uniform prior on p(z),
where z = fx()), we have
Z(x) Eu- ~p=x) le d(fo(m).fo(= ")] Ex- xp(z)(e - -2) -2)]
/ (x =)= -=-- -x)
VTT
(10)
which is also not related to 0. Therefore, under the uniform prior assumption on both p(æ) and p(z), minimizing LCR is the same as maximizing H(X- X), as well as the same as minimizing
Proof of Lemma 4.3
The CL loss can be decomposed as an expected dissimilarity term and a log-sum-exp
term:
Sola) fo(w 3/7
LOL:=
E
In
edo(ai) fo(xt)/7 E,sula T folw)/T
In efola) falwo)// + Σ etola \ fo(w)/T )
M
[-f0x)) fo(x)) sooTTt)((ttt))
1
= E (2,000
+
E (m,ar (a
18

---

Published in Transactions on Machine Learning Research (07/2023)
where the positive sample x* is independent of given zu and the negative samples a; are independent of a. As the number of negative samples goes to infinity, following Wang & Isola (2020), the normalized CL loss
is decomposed into the sum of the align loss, which describes the contribution of the positive samples, and the uniform loss, which describes the contribution of the negative samples:
lim LCr. In M B (I -0 So((rt))o)))))))) M-
So((r) fo(a)/7
E
E (e-mp(x=))
anppl())
contribution of positive samples
contribution negative samples
With importance sampling, the second term in the RHS of the above equation can be further derived into:
In
efol
fa(m//
mmpp(m)
~ppa
E
In
E
fo(m)/T
p(æ To x (x0
o mola
(e)
Apply the Jensen inequality, the second term is decomposed into the negative cost plus log density ratio:
Jo( ) fo(0))/
E
In
~pp(
In p(x -
[Jo(2] [o(x))/T]
E
x>pp(a))
1 mo ( (2))
xxpp(())
(a= (00)
Te
x))
E
[fo(x fo(x)//]] I(X:X-)
>>p((>)
m))
Defining the point-to-poing cost function between two unit-norm vectors as c(z1, x2) 27 22 (same as the Euelidean cost since ||=]
22|1 zI 22 ), have
Sala >T fo(0)//T + T(X;8-
In x>)((())
pp)))
[fo(x]] fo(x))/]]
E 2²)
(p)p(())
) (c(fo(x"),fo(x)))
x<)(())
LCR-
This concludes the relation between the contribution of the negative samples in CL and that in CACR.
19

---

Published in Transactions on Machine Learning Research (07 (07/2023)
B
Additional experimental results
In this scction, we provide additional results in our experiments, including ablation studies, and corresponding qualitative results.
B.1 Additional results with AlexNet and ResNet50 encoder on small-scale datasets
Following benchmark works in contrative learning, we add STL-10 dataset to evaluate CACR in small-scale experiments. As an additional results on small-scale datasets, we test. the performance of CACR two different. encoder backbones. Here we strictly follow the same sctting of Wang & Isola (2020) and Robinson et al. (2021); and the results are shown in Table and We can observe with ResNet50 encoder backbone, CACR with single positive or multiple positive pairs consistently outperform the baselines. Compared with the results in Table the CACR shows more clear improvement over the CL baselines.
Table The top-1 classification accuracy (%) of different contrastive objectives with SimCLR framework on small-scale datasets. All methods follow SimCLR setting and apply AlexNet encoder and trained with 200 epochs.
Dataset
CL
AU-CL
HN-CL CACR(K=1) CMC(K=4) CACR(K=4)
CIFAR-10
83.47
83.39
83.67
83.73
85.54
86.54
CIFAR-100
55.41
55.31
55.87
56.52
58.64
59.41
STL-10
83.89
84.43
83.27
84.51
84.50
85.59
Table 9: The top-1 classification accuracy (%) of different contrastive objectives with SimCLR framework small-scale datasets. All methods follow SimCLR setting and apply ResNet50 encoder and trained with 400 epochs.
Datasct
CL
AU-CL
HN-CL CACR(K=1)
CMC(K=4) CACR(K=4)
CIFAR-10
88.70
88.63
89.02
90.97
90.05
92.89
CIFAR-100
62.00
62.57
62.96
62.98
65.19
66.52
STL-10
84.60
83.81
84.29
88.42
91.40
93.04
B.2 On the effects of conditional distribution
Supplementary studies of CA and CR: As a continuous ablation study shown in Figure 3 we also conduct similar experiments on CIFAR-100, where we study the evolution of conditional entropy H(X- w.r.t the training epoch. The results are shown in Figure 4 and the results of exponential label-imbalanced data are shown in Figure 5 Similar to the observation on CIFAR-10, shown in Figure 3 we can observe H(X (X) is getting maximized as the encoder is getting optimized with these methods, as suggested in Lemma. 4.2 In the right pancl, We can observe bascline methods have lower conditional entropy, which indicates the encoder is less effective in distinguish the nagative samples from query, while CACR. consistently provides better performance than the other methods indicating the better robustness of CACR.
As a qualitative verification, we randomly take a query from a mini-batch, and illustrate its positive and negative samples and their conditional probabilities in Figure G As shown, given this query of dog image, the positive sample with the largest weight contains partial dog information, indicating the encoder focus on texture information; the negatives with larger weights are more related to the dog category, which encourages the encoder to focus distinguishing these "hard" negative samples. In total, the weights learned by CACR enjoy the interpretability compared to the conventional CL.
We study different definition of the conditional distribution. From Table 10 we can observe that the results are not sensitive to the distance space. In addition, as we change т. to assign larger probability to closer samples, the results are similar to those using single positive pair (K=1). Moreover, the performance drops if we change x_ assign larger probability to more distant negative samples.
Uniform Attraction and Uniform Repulsion: A degenerated version of CACR
20

---

Published in Transactions on Machine Learning Research (07 (07/202) /023)
Nets
Nats
3.9
3.9
Mluiif)
MTCMC IK 4]]
Hlunif
HICMC (K 4))
(K=
2.5
2.5
6.3
6.3
Nats
Nets
5.9
Hluiif)
MIHN-CL M/CNCR 1))
5.9
AIHTCC H(CACR IK 1/1
H(CL)
MCL)
HIAU-CL)
MAU-CU
5.5
25
50
75
100
125
150
175
200
5.5
25
50
160 Epoch
125
150
175
200
Epoch
Figure 4: (Supplementory to Figure Conditional entropy H(X w.r.t. training epoch on CIFAR-100 (left) and linear label-imbalanced CIFAR-100 (right). The maximal possible conditional entropy indicated by a dotted line.
5.3
5.3
Nats
Munii)
MCCC IK
Mlunif)
HICMC
N(CACR (K 4)1
MCCACR = 4])
2.5
2.5
6.3
6.3
Nats
Nats
5.9
5.9
HICL)
HICACR (K = 1))
MCLL)
HICACR IK 13)
HLAU-CL)
MAU-CL)
5.5
100
175
5.5
160
175
Epoch
Epoch
Figure 5: (Supplementary to Figure St Conditional entropy H(X- (X) w.r.t. training epoch on exponential label-imbalanced CIFAR-10 (left.) and CIFAR-100 (right). The maximal possible conditional entropy is indicated by a dotted line.
To reinforce the necessity of the contrasts within positives and negatives before the attraction and repulsion, we introduce degenerated version of CACR here, where the conditional distributions are forced to be uniform. Remind z()1,22) as the point-to-point cost of moving between two vectors and 22, e.g., the squared Euclidean distance ||=| - =2 or the negalive inner product -2f #2. In the same spirit of equation 1 we have considered a uniform attraction and uniform repulsion (UAUR) without doubly contrasts within positive and negative samples, whose objective
min (urorpacto Ben Beo.et [c(fe(z) E,,, <p(=) [c(fo()), fo [)]
(11)
The intuition of UAUR to minimize minimize/maximine maximmzee the expected cost of moving the representations of posi- tive/ nvegetgtivve samples to that the query, with the costs of all sample pairs being uniformly weighted. While equation has been proven to be effective for representation learning, our experimental results do not find equation perform well, suggesting that the success of representation learning is not guaranteed by uniformly pulling positive samples towards and pushing negative samples away from the query.
Distinction between CACR and UAUR: Compared to UAUR in equation 11 that uniformly weighs different pairs, CACR is distinct in considering the dependency between samples: as the latent-space distance between the query and its positive sample becomes larger, the conditional probability becomes higher, encouraging the encoder focus more on the alignment of this pair. In the opposite, as the distance between the query and its negative sample becomes smaller, the conditional probability bocomes higher, encouraging the encoder to push them away from each other.
21

---

Published in Transactions on Machine Learning Research (07 (07/202) /023)
Data
Top 4 Negative samples
Bottom 4 Negative samples
Query
Positive samples
Figure 6: Illustration of ausitive//neaative samples and their corresponding weights. (Left) For a query augmented from the original dog image, 4 positive samples shown, with their weights visualized the blue distribution. (Right) The sampling weights for negative samples are visualized as the red distribution; we visualize 4 negative samples with the highest and with the lowest weights, with their original images shown below.
Table 10:
Linear classification performance (%) of different variants of conditional probability. This experiment is done on CIFAR-10, with 4 and mini-bateh size M
128.
14, (ol 1 "p(ant (o))
det Uo(e)) "p(an"
[ ( fot "(axt wo)dæt
(aam)) p(a+ / ddat
- (f)))) p(
86.48
83.91
π
e --- (fla p(ce da
eta, dole p(20
79.46
74.91
fold, (Dalw))doee pp(p )dæ
In order to further explore the effects of the conditional distribution, we conduct an ablation study to compare the performance of different variants of CACR with/withh without conditional distributions. Here, compare 4 configurations of CACR (K = 4): (1) CACR with both positive and negative conditional distribution; (wi) CACR without the posilive conditional distribution; (wi) CACR without the negative conditional distribution; (fe) CACR without both positive and negative conditional distributions, which refers to UAUR model (sce Equation 11}. As shown in Table 11. when discarding the positive conditional distribution, the linear classification accuracy slightly drops. As the negative conditional distribution is discarded, there is large performance drop compared to the full CACR objective. With the modeling of neither positive nor negative conditional distribution, the UAUR shows a continuous performance drop, suggesting that the success of representation lcarning not guarantced by uniformly pulling positive samples closer and pushing negative samples away. The comparison between these CACR variants shows the necessity of the conditional distribution.
Table 11: Linear classification performance (%) of different variants of our method. "CACR" represents the normal CACR configuration, "w/o T means without the positive conditional distribution, "w/o ف means without the negative conditional distribution. "UAUR" indicates the uniform cost (see the model we discussed in Equation 11], f.e. without both positive and negative conditional distribution. This experiment is done all small-scale datasets, with K and mini-batch size M 128.
Methods CIFAR-10 CIFAR- 100
STL-10
CACR
85.94
59.51
85.59
w/o m
85.22
58.74
85.06
w w/o Te
78.49
47.88
72.94
UAUR
77.17
44.24
71.88
22

---

Published in Transactions on Machine Learning Research (07 (07/2023)
As qualitative illustrations, randomly fix one mini-batch, and randomly select one sample as the query. Then we extract the features with the encoder trained with CL. loss and CACR (K= loss at epochs 1. 20, and 200, and visualize the (four) positives and negatives in the embedding space with t-SNE (van der Maaten & Hinton, 2008). For more clear illustration, we center the query in the middle of the plot and only show samples appearing the range of [-10, 10] on both and axis. The results are shown Figure 7(c) from which we can find that as the the encoder getting trained, the positive samples are aligned closer and the negative samples are pushed away for both methods. Compared to the encoder trained with CL, we observe CACR shows better performance in achieving this goal. Moreover, between any two data points in the plot is more uniform, which confirms that CACR shows better results in the maximization of the conditional entropy -
observe the distance
182
Query Postlive
Negative
235
-3
330
-35
-350
-123
Query
Podtivv
Negelivw
235
25
123
150
31
(a) Epoch
(b) Epoch 20
(c) Epoch 200
Figure 7: The t-SNE visualization of the latent space at different training epochs, learned by CL loss (top) and CACR loss (bottom). The picked query is marked in green, with its positive samples marked in blue and its negative samples marked in red. The circle with radius t- is shown as the black dashed line. As the encoder gets trained, we can observe the positive samples are aligned closer to the query (Property [ and the conditional differontial entropy H(X X)) is progressively maximized, driving the distances difo f fo(x towards uniform (Lemmna 4.2
B.3
Ablation study
On the effects of negative sampling size: We investigate the model performance and robustness with different sampling size by varying the mini-batch size used in the training, On all the small-scale datasets, the maii-batches are applied with size 64, 128, 256, 512, 768 and the corresponding linear classification results are shown Figure & From this figure, we can see that CACR (K 4) consistently achieves better performance than other objectives. For example, when mini-batch size is 256, CACR (K 4) outperforms CMC by about 0.4%-1. 1.2%. CACR (K 1) shows better performance in most of the cases, while slightly underperforms
23

---

Published in Transactions on Machine Learning Research (07 (07/2023)
than the baselines with mini-batch size 64. A possible explanation could be the estimation of the conditional distribution needs more samples to provide good guidance for the encoder.
66.0
16.0
89.3
15.5
85.0
15.0
O4
CACRI
58.0
CYOC-N 4
CAONK wf
84.5
10 CHIC+
CRONIC
14.0
56.5
5550
13.0
+0- 0
HNC.
18.C1
120
256
112
708
120
255
61
255
512
761
Batch Size
Batch Sice
Batch Size
(a) 10
(b) CIFAR-100
(c) STL-10
Figure 8: The linear classification results of training with different sampling size on small-scale datasets. The training batch size is proportional to the negative sampling size.
We conduct experiments to investigate the model performance with different positive sampling size by using different values in the pretraining: E (1,2, 6, 10) on Similar to our experiment sctting, in 200 epochs, we apply Nee encoder CIFAR-10 and CIFAR-100 and apply ResNet50 encoder ImageNet-1K. Shown we observe as K increases, the linear classification accuracy increases accordingly.
On the effects of positive sampling size:
CIFAR-10/ and K E f1,2, 2, 3, 4] on ImageNet-
in Figure
#15
665
860
58.
850
645
55.
666
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet
Figure The linear classification results of training with different positive sampling size on CIFAR- 10, CIFAR-100 and ImageNet-1K. An AlexNet encoder is applied on CIFAR-10 and CIFAR-100; ResNet50 encoder is applied on ImageNet.
On the effects of hyper-parameter tt, t Remind in the definition of positive and negative conditional distribution, two dyper-paramotonn t+ and t- are involved following:
Tg ( xe, x0) = ct (1gtm)))000 p( [a To (x
eet (o(()) 21 p(at mo)
(x) = (
111
p(a
(0(m)
fpim
p(
dde
In this part, we investigate the effects of +t and t- representation learning performance on small-scale datasets, with mini-batch size 768 (K = 1) and 128 (K =4) as an ablation study. We search in a range (0.5, 0.7, 1.0, 2.0,0033.0 3.0). The results are shown in Table 12 and Table 13
As shown these two tables, from Table 12 we observe the CACR shows better performance with smaller values for t+. Especially when + increases to 3.0, the performance drops up about 1.9% on CIFAR-100. For analysis, since have K 4 positive samples for the computation of positive conditional distribution,
24

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table 12: The classification accuracy(%) of CACR (K = 4, M = 128) with different hyper-parameters t on
small-scale datasets.
Method
Dataset
0.5
0.7
0.9
1.0
2.0
3.0
CIFAR-10
86.07
85.78
85.90
86.54
84.85
84.76
CACR (K
CIFAR-100
59.47
59.61
59.41
59.41
57.82
57.55
STL-10
85.90
85.91
85.81
85.59
85.65
85.14
using large value for ++ could result in over-sparse conditional distribution, where the conditional probability is dominant by one or two positive samples. This also explains why the performance when tt = 3.0 is close to the classification accuracy of CACR (K= 1).
Similarly, from Table 13 we can see that a small value for t- will lead to the dogenerated performance. Here, since we are using mini-batches of size 768 (K 1) and 128 (K = =), a small value for t will flatten the weights of the negative pairs and make the conditional distribution closer to a uniform distribution, which explains why the performance when t 0.5 is close to those without modeling "g- Based these results, the values of ++ E [0.5, 1.0] and t E [0.9, 2.0] could be good empirical choices according to our experiment settings on these datasets.
B.4 Additional comparisons
In this part we provide more comparisons with baseline methods. For small-scale experiments, we still compare with contrastive learning methods, conventional CL loss, align-uniform loss, and hard negative sampling CL loss. For large-scale experiments, continue to compare with contrastive learning loss on ImageNet-1000 and ImagNet-1K with MoCov2 framework, and provide comparisons with SOTAs pretrained with different epochs.
Training efficiency on small-scale datasets: On CIFAR-10, CIFAR-100 and STL-10, we pretrained AlexNet encoder in 200 epochs and save linear classification results with learned representations every epochs. Shown in Figure I0 CACR consistently outperforms the other methods in linear classification with the learned representations at the same epoch, indicating superior learning efficiency of CACR. Correspondingly,
we also evaluate the GPU time of CACR loss with different choices of K, as shown in Table 14
Comparison with contrastive learning methods on ImageNet: For large-scale experiments, following the convention, we adapt all methods into the MoCo-v2 framework and pre-train a ResNet50 encoder in 200 epochs with mini-batch size 128/256 on ImageNet-100/ 100/mmmgeeee--lk. Table 15 summarizes the results of lincar classification on these two large-scale datascts. Similar the case on small-scale datascts, CACR consistently shows belter performance, improving the baselines at least by 1.74% on ImageNet-1000 and 0.71% ImageNet- 1KK In MoCo-v2, with multiple positive samples, CACR improves the baseline methods by 2.92% on ImageNet-100 and 2.75% on ImageNet-IK. It is worth highlighting that the improvement of CACR is more significant on these large-scale datasets, where the data distribution could be much more diverse compared to these small-scale ones. This is not surprising, as according to our theoretical analysis, CACR's
Table 13: The classification accuracy (%) of CACR 1, M = 7688 and CACR (K = =,, -128) with different paremeparameters t on scall-scale datasets.
Methods
Dataset
0.5
0.7
1.0
2.0
3.0
CIFAR-10
81.66
82.40
83.07
82.74
83.73
83.11
CACR (K =
CIFAR-100
51.42
52.81
53.36
54.20
56.21
56.52
STL-10
80.37
81.47
84.46
82.16
84.21
84.51
CIFAR- 10
85.67
86.19
86.54
86.41
85.94
85.69
CACR (K =
CIFAR-100
58.17
58.63
59.37
59.35
59.41
59.31
STL-10
83.81
84.42
84.71
85.25
85.59
85.41
25

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Aoc.
O4XX CARRX 4)
173
CHCIC
Axcumoo
15
38
HNC
C
19-CL CACRIC-
AUCL
Epoch
Epach
Epoch
(a) CIFAR-10
(b) CIEAR-100
STL-110
Figure 10: Comparison of training efficientcy: Linear classification with learned representations w.r.t. training epoch on CIFAR-10, CTFAR-100 and STT-10.
Table 14: GPU time (s) per iteration of CACR w.r.l. different K on CIFAR-10 with AlexNet framework (mini-batch size 128), tested on Tesla-v100 GPU.
K
2
10
GPU time (8) iteration 0.0021 0.0026 0.0035 0.0045 0.0054 0.0064
double-contrast within samples enhances the effectiveness of the encoder's optimization. Moreover, we can see CACR (K = shows clear improvement over HN-CL. A possible explanation that although both proposed to improve the CL loss, the
increasing the negative sample size and selecting hard negatives effectiveness of hard negatives is limited when the sampling size is increased over certain limit. As CACR targets to repel the negative samples away, the conditional distribution still efficiently guides the repulsion when the sampling size becomes large.
Table 15:
Comparison with contrastive learning methods: Top-1 classification accuracy (%) of differ- ent contrastive learning objectives on MoCo-v2 frame- work and ResNet50 encoder, pretrained on ImageNet- IK dataset with 200 epochs. The results from paper Github page marked by
Table 16: Comparison with state-of-the-arts on linear probe classification accuracy, pretrained with differ- ent epochs, using ResNet50 encoder backbone on ImageNet-lk.
Epochs
100
200
400
800
1000
BYOL
66.5
70.6
73.2
74.3
Methods
ImageNet-1000 ImageNet-II
BarlowTwins
73.2
MoCov2 (CL)
77.54*
67.50*
SWAV
66.5
69.1
70.7
71.8
AU-CL
77.66*
67.69*
Simsiam
68.1
70.0
70.8
71.3
HN-CL
76.34
67.41
SimCLR
66.5
68.3
69.8
70.4
71.7
CACR (K = 1)
79.40
68.40
MoCov2
67.4
69.9
71.0
72.2
CMC (CL, K 4)
78.84
69.45
FNC (multi-crop)
70.4
74.4
CACR (K
80.46
70.35
CACR
68.3
70.4
73.8
74.0
74.4
Comparison with other SOTAs: Besides the methods using contrastive loss, continue to compare with the self-supervised learning methods like BYOL, SWaV, SimSiam, etc. that do not involve the contrasts with negative samples. Table 16 provides more detailed comparison with all stato-of-tho-arts in different epochs and could better support the effectiveness of CACR: We can observe CACR achieves competitive results and generally outperforms most of SOTAs at the same epoch in linear classification tasks. We also compare the computation complexity. Table 17 reports computation complexity to provide quantitative results in terms of positive number K, where we can observe the computation cost of CACR slightly increases K increase, but does not increase that when using multi-positives in CT. loss.
26

---

Published in Transactions on Machine Learning Research (07 (273))
Table 17: GPU time (s) per iteration of different loss on MoCov2 framework, tested on 32G-V100 GPU
Methods
CL
AU-CL IIN-CL CACR(K=1) CL (K- 4) CACR(K-22 2) CACR(K-3) CACR(K-))
Batch size M # samples (KxM) / iteration
256
256
256
256
64
128
64
64
256
256
256
256
256
256
192
256
GPU time (s) / iteration
0.837
0.840
0.889
0.871
3.550
0.996
1.017
1.342
ResNet50
VIT B/16
Method
FT
Liu-cls
FT
Lin-cls
SimCLRv22
77.2
71.7
83.1
73.9
MoCov3
77.0
73.8
83.2
76.5
CACR.
78.1
74.7
83.4
76.8
SWAVT
77.8
75.3
82.8
71.6
CACR
78.4
75.3
83.4
77.1
Table 18: Comparison with state-of-the-arts. fine-tuning and linear probing classification accuracy (%), pre-trained using ResNet50 and ViT-Base/16 encoder backbone on ImageNet-1k. indicatess using SWAV multi-crops.
Comparison with advanced architectures: Beyond the conventional evaluation on linear probing, recent self-supervised learning methods use advanced encoder architccture such as Vision Transformers (ViT) (Dosovitskiy et al., 2020), and are evaluated with end-to-end fine-tuning. We incorporate these perspectives with CACR for a complete comparison. Table provides comparison with the state-of-the-arts using ResNet50 and ViT-BBse/IK 16 backbone, where follow their experiment settings and preitrain ResNet50 with 800 epochs and ViT-B/16 with 300 epochs. We can observe CACR generally outperforms these methods in both fine-tuning and linear probing classification tasks.
CLIP Radford et al.
2021
CLIP reeroreprdd
CACR
19.8
19.2
22.7
Table 19: Top-1 zero-shot classification accuracy (%) on ImageNNet1K, pre-trained using ResNe150 on CC3M dataset.
Multi-modal contrastive learning: Besides self-supervised learning on vision tasks, we follow CLIP Rad ford al. (2021) evaluate CACR on multi-modaa representation learning. We compare CACR's performance with CLIP, with our reproduced result and the results reported in Li et al. (2022) in Table 19 All methods
are pre-trained CC3M dataset with ResNet50 backbone for 32 epochs. We can observe CACR surpasses CLIP by 2.9% in terms of zero-shot accuracy on ImageNet.
B.5 Connection to other representation learning methods
Results of different cost metrics
Recall that the definition of the point-to-poing cost metric is usually set as the quadratic Euclidean distance:
(( = [[)((=) - fo(v))1
(12)
In practice, the cost metric defined in our method ficxible to be any valid metrics. Here, we also investigate the performance when using the Radial Basis Function (RBF) cost metrics:
(eae(fe))) = -e -f]]f((=)) fo(y))
(13)
where E R is the precision of the Gaussian kernel. With this definition of the cost metric, our method is closely related to the baseline method AU-CL (Wang Isola] 2020). where the authors calculate pair-wise RBF cost for the loss w.r.t. negative samples. Following Wang & Isola (2020), replace the cost metric
27

---

Published in Transactions on Machine Learning Research (07/2023)
when calculate the negative repulsion cost with the RBF cost and modify LCR
LCR
In
[M
Voleiaaaaa canr((o(xs))
(14)
#/ "
= M
(ol Upleod X fo=1)
Here the negative cost is log scale for numerical stability. When using the RBF cost metric, we use the same setting in the previous experiments and evaluate the linear classification on all small-scale datasets. The results of using Euclidean and RBF cost metrics are shown in Table 20 From this table, see that both metrics achieve comparable performance, suggesting the RBF cost is also valid in our framework. In CACR, the cost metric measures the cost of different sample pairs and is not limited on specific formulations. More favorable cost metrics can be explored in the fulure.
Table 20: The classification accuracy (%) of CACR (K = 1) and CACR (K= = 4) with different cost metrics on CIFAR-10, CIFAR-100 and STL-10. Euclidean indicates the cost defined in 12 and RBF indicates the cost metrics defined in 13
Methods
Cost Metric CIFAR-10 CIFAR-100 STL-10
Euclidean
83.73
56.21
83.55
CACR(K 1)
RBF
83.08
55.90
84.20
Euclidean
85.94
59.41
85.59
CACR(K
RBF
86.20
58.81
85.80
Discussion: Relation to triplet loss CACR is also related to the widely used triplet loss (Schroff et al. 2015; Sun et al. 2020b). A degenerated version of CACR where the conditional distributions are all uniform can be viewed as triplet loss, while underperform the proposed CACR, as discussed in Section B.2. In the view triplet loss, CACR is dealing with the margin between expected positive pair similarity and negative similarity:
LCACR = [Bx, (wt [e) x - -E, (a [=)[c(<, )] + m]+
which degenerates to the generic triplet loss if the conditional distribution degenerates uniform distribution:
LUAUR = EBp [e(2,, +1] Epp flo(x, x: [] m]+ [c(2, x+) c(2, m]+
This degeneration also highlights the importance of the Bayesian derivation of the conditional distribution. The experimental results of the comparison between CACR and the degenerated uniform version (equivalent to generic triplet loss) are presented in Table I1
Moreover, CACR loss can degenerate to triplet loss with hard example mining if Te (x+(x) and ne (a+ are sufficiently concentrated, where the density shows very sharp peak:
CCACR [max(c(æ, (c)) mnincc(w,a + m]+
which corresponds to the loss shown in Schrolf et al. (2015). As shown in Table 12 and 13 when varying tt and t- to sharpen/ sarppnn/llttenn /flatten the conditional distributions. Based our observations, when + 3 and t- = 3, the conditional distributions are dominated by 1-2 samples, where CACR can be regarded as the above-mentioned triplet loss, and this triplet loss with hard mining slightly underperforms CACR. From these views, CACR provides a more general form to connect the triplet. loss. Meanwhile, it is interesting to notice CACR explains how triplet loss is deployed in the self-supervised learning scenario.
Relation to CT. The CT framework of Zheng Zhou (202)) is primarily focused on measuring the difference between two different distributions, which referred to the source and target distributions, respectively.
28

---

Published in Transactions on Machine Learning Research (07/2023)
It defines the expected CT cost from the source to target distributions as the forward CT, and that from the target to source as the backward CT. Minimizing the combined backward and forward CT cost, the primary goal is to optimize the target distribution to approximate the source distribution with both mode-covering and modo-sccking properties. In CACR, we did not find any performance boost by modeling the reverse conaltional transport, since the marginal distributions of and x+ are the same and these of æ and x are also the same, there is no need to differentiate the transporting directions. In addition, the primary goal of CACR is not to regenerate the data but to learn fo(-) that can provide good latent representations for downstream tasks.
29

---

Published in Transactions on Machine Learning Research (07 2022))
C Experiment details
On small-scale datasots, all experiments are conducted on single GPU, including NVIDIA 1080 Ti and RTX 3090; on large-scale datasets, all experiments are done on Tesla-V100-32G GPUs.
C.1 Small-scale datasets: CIFAR-10, CIFAR-100, and STL-10
For experiments on CIFAR-10, CIFAR-100, and STL-10, use the following configurations:
Data Augmentation: We strictly follow the standard data augmentations construct posilive and
negative samples introduced in prior works in contrastive learning (Wu et al 2018 Tian al. 2019
Hjelm et al. 2018 Bachman et al. 2019: Chuang et al. 2020 He et al. 2020 Wang & Isola 2020
The augmentations include image resizing, random cropping, flipping, color jittering, and gray-scale conversion. We provide a Pytorch-style augmentation code in Algorithm which is exactly the same
as the one used Wang Isola (2020).
Algorithm 1 PyTorch-like Augmentation Code on CIFAR-10, CIFAR-100 and STL-100
import torchvision.c transforms transforms
CIFAR-10 Transformation
def transform, bramsform_cifartte cifar10():
return transforms. Compose
transforms RandomResizeddrrppcce (32, scale=(0.2, 1)), transforms RandomllorizonttalPPip] , by default p=0.5 transforms ColorJitter Prensffors..Coloooitttttoo---... 0.4, 0.4, 0.4),
transforms Randonnrrayscaleeppoo.
transforms. ToTensor transforms Normalize( (0. 4914, 0.4822, 0.4465), (0. 2023, 0. 0994 0.2010),
normalize to value in [0,1]
1)
CIFAR-100 Transformation def transform, cifar100();
return transforms Compose
transforms RandonResizeeCCogn(33 (32, scale=(0.2, 1)), # by default p=0.5 transforms Cow.ee/oooooOoOoOOO.O... 0.4, 0.4, 0.4),
transforms Randon@raaysaaee(pp))
transforms. ToTensor O), normalize to value in [0,1]
transforms Normalizee 5071, 0.4867, 0.4408), (0. 2675, 0. 2565, 0. 0.2761),
1)
# STL-10 Transformation
def transform
sarefoommsst1():== stl10():
return transforms. Cranssorrns.Copose((
transforms RandomResizzedCrop (64, scale=(0.08, 1)), transforms wandonnoorizontalll........ , by default p=0.5 branafornee.CloorIittterrr ColorJitter(O.:: 0.4, 0.4, 0.4),
tranaforms.
transforms Ran.don@raascall(ppoo....
transforms. transforms..TTeesor(() transforms, Normalize( M (0. 4409, 0.4279, 0.6868)
normalize to value in [0,
(0. 2683, 0. 2610, 0.2687)
)
))
Feature Encoder: Following the experiments in Wang & Isola (2020), we use an AlexNet-based
encoder as the feature encoder for these three datasets, where encoder architectures are the
30

---

Published in Transactions on Machine Learning Research (07/ (07/2023)
Table 21: The 100 randomly selected classes from ImageNett forms the ImageNet-100 dataset. These classes
are the same [Wang & Isola 2020
Tian et al.
2019
TeugeNNt. 100
=02669837
b01749939
n02406291
n02107142
9037406
a02091631
=04517623
=04689090
m03062245
n01773797
501736189
147831148
n07763275
n03085013
m04485082
002106506
=01983481
n02788148
mO3630642
n0443:663
E02036910
602859443
a13040303
H03594734
m02085620
a02099849
201558993
604493381
m92109047
m04111551
#02677765
nO44293366
n02009229
H01978455
#02106550
a01820546
601692333
n07714571
n02974003
n02114855
203735016
m05764736
m05775546
i02087046
a07836638
a04099969
=04592741
#03691251
m02701002
n03379051
n022259212
n07715103
n03347888
n04026417
n02326432
n03637318
601980166
n.21 13799
n02086220
n03903868
E32483362
n04127249
n02088973
n03017168
m02093428
a02804414
=02396427
604418367
n02172182
n01729322
8021
n03787032
n02085867
n02119022
703777754
n04238763
n02231487
n03032262
n02138941
n02104029
#03837869
p03494278
n04136353
n03790056
m03492542 a02018207
=04087472
#03930630
mQ3584829
m02123045
E04229816
Io2100583
μ03542806
n04336792
a03259280 a02116738
202108089
633424325
p01855672
p02090622
as those used in the corresponding experiments in Tian et al. 2019 and Wang & Isola 2020
Moreover, also follow the setups in Robinson et al. (2021) and test the performance of CACR
with ResNet50 encoder (results are shown in Table 9
Model Optimization: We apply the mini- batch SGD with 0.9 momentum and le-4 weight decay. The learning rate is linearly scaled as 0.12 per 256 batch size (Goyal et al. 2017). The optimization is done over 200 epochs, and the learning rate is decayed by a factor of 0.1 at epoch 155, 170, and 185.
Parameter Setup: On CIFAR-10, CIFAR-100, and STL-10, we follow Wang Isola 2020 to set the training batch size as M 768 for baselines. The are set according to the original paper or online codes. Specifically, the temperature parameter of CL is 0.19, the hyper-parameters of AU-CL are t 2.0, 0.19, and the hyper-paramcter of HN-CL are = 0.5, B 1.0", which shows the best performance according to our luning. For CMC and CACR with multiple positives, the positive sampling size is K = 4. To make sure the performance is not improved by using more samples, the training batch size is set as M = 128. For CACR, in both single and multi-positive sample settings, we set t+ = 1.0 for all small-scale datasets. As for 1-, for CACR (K = 1), t 2.0, 3.0, and 3.0 on CI CITARR00, and STI-10, respectively. For CACR (K = 4), t- 0.9, 2.0, and 2.0 on CIFAR-10, CIFAR100, and STL-10, respectively. For further ablation studies, test + and t- with the search in the range of [0.5, 0.7, 0.9, 1.0, 2.0, and test all the methods with several mini-batch sizes M {64, 128, 256, 512, 562.
of CL, AU-CL", and HN-CL*
The feature encoder is trained with the default built-in training set of the datasets. In the evaluation, the feature encoder is frozen, and a lincar classifier is trained and tested on the default training set and validation set of each dataset, respectively. Following Wang & Isola (002)), we train the linear classifier with Adam optimizer over 100 epochs, with 31 = 0.5, 32 = 0.999, e = 10 and 128 as the batch size. The initial learning rate is 0.001 and decayed by a factor of 0.2 at epoch and epoch 80. Extractod features from "fc7" are employed for the evaluation. For the ResNet50 setting in Robinson et. al. 2021)) the extracted features are from the encoder backbone with dimension 2048.
Evaluation:
C.2 Large-scale datasets
For large-scale datasets, the Imagenet- MK the standard ImageNet dataset that has about 1.28 million images of 1000 classes. The ImageNet- 100 contains randomly selected 100 classes from the standard ImageNet- -K dataset, and the classes used here are the same with Tian et al. (2019) and Wang Isola (2020), listed in Table 21] For pretraining with less curated data*, considering Webvision v1 and ImageNet-229 respectively have 2.4 and 14.2 million images, we decrease the training epoch to 1/2 and 1/10 when pretraining on these two datasets as done in Li et al. (2021). We follow the standard scttings in these works and describe the experiment configurations as follows:
* http://gttuub..oo///ss//////
uniform
* https://github.com cotpuu/oooo/oos////icc
* Please refer the original paper for the specific meanings hyper-parameter in baselines.
"https://dar haappss//datttae http://aaaaaaiiinaeeee/ www..ch//cxlwwwwissiooooo
31

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm 2 PyTorch-like Augmentation Code on ImageNet-100 and ImageNet-II
import torchvision. transforms transforms
# Imageliet-100 and ImageNet-11 Transformation
# MoCo v2's aug: similar to SimCLR def transform, (itooormmmmgee()))
return transforms crampoorrms.
transforms RandomResizzedrroppppooooooooo (224, scale=(0.2, 1.)), transforms RandomApply [transforms ColorJitter 0.4, 0.4, 0.1)
1, p=0.8),
transforms. bransforme..RndddGrrryy. Raaddommrraaaccaeeppp0..))
transforms www.omAppnyyyyccccmmmmma Lranssoonss.BBnddoodppppyyy [moco. Loader GaussianBlur ([.1, 2.])], p=0.5),
transforms. trraasforrnBBaadddootoooooaaaaa Raadomtfor11onntaallip..
transforms. (oTesorr
Gransforms.None Normalooeeeeeeee (mean= [0. 485, 0.456, 0.406],
std= 229, 0. 022 0. 022]]
})
Data Augmentation: Following Tian et al. 2019 2020b:; Wang & Isola (2020): Chuang al.
(2020); He et al. ),000), and Chen et al. (2020c), the data augmentations are the same the standard protocol, including resizing, 1x image cropping, horizontal flipping, color jittering, and gray-scale conversion with specific probability. The full augmentation combination is shown in Algorithm 2
Feature Encoder: For the main experiments, we apply MoCo-v3 setting, where the framework consists of a teacher and a student encoder. The teacher encoder follows a EMA updating stralegy and all experiment settings follows the MoCo-v3 paper (Chen et al. 2021). We apply the MoCo-v2 framework (Chen et al. 2020c) for further justification that CACR is also applicable with framework including a large queue, where the ResNet50 THe et al., 2016) is commonly chosen feature encoder
architecture. The output dimension of the encoder is set 128.
Model Optimization: Following the standard setting in He et al. 2020 Chen al. (2020c); Wang & Isola (2020). the training mini-batch size is set 128 ImageNet-100 and 256 ImageNet- IK. We use mini-batch stochastic gradient descent (SGD) optimizer with 0.9 momentum and lc-4 weight decay. The total number of training epochs is set as 200. The learning rate is initialized as 0.03, decayed by a cosine scheduler for MoCo-V2 at epoch 120 and epoch 160. In all experiments, the momentum of updating the offline encoder is 0.999.
Parameter Setup: On ImageNet-100 and ImageNet- for all methods, the queue size for negative sampling 65,536. The training batch size is 128 on ImageNet-100 and 256 on ImageNet. For CACR, we train with two positive sampling sizes K and K 4 and the parameters in the conditional weight metric respectively set /+ 1.0, t- 2.0. For baselines, according to their papers and Github pages (Tian et al. 2019 Wang & Isola, 2020 Robinson et al. 2021 the temperature parameter of CL is r 0.2, the typer-parameters of AU-CL are 3:0.0, 3.0,
0.2, and
the syper-parameter of HN-CL are 7 = 0.5, 3 = 1.0. Note that CMC (K reported in the main paper is trained with 240 epochs and with its own augmentation methods (Tian et al. 2019 For CMC (K = 4), the temperature set T = 0.07 according to the setting in Tian et al. 2019 and the loss calculated with Equation (8) in the paper, which requires more GPU resources than 8 Tesla-V V100-32G GPUs with the setting on ImageNet-1K.
Linear Classification Evaluation: Following the standard linear classification evaluation (He et al. 2020 Chen al. 2020c Wang & Isola 2020), the pre-trained feature encoders are fixed, and linear classifier added on top is trained the train split and test the validation split. The linear classifier is trained with SGD over 100 epochs, with the momentum as 0.9, the mini-batch size as 256, and the learning rate as 30.0, decayed by a factor of 0.1 at epoch 60 and epoch 80.
Feature Transferring Evaluation (Detection and Segmentation): The pre-trained models are transferred to various tasks including PASCAL VOC* and COCO datsets. Strictly following the
"http///nast.roattsoocaaa. pasppaaa///////eee..mmm
hhtpp///rocooiaaasett.oo//// #owwnoadd
32

---

Published in Transactions on Machine Learning Research (07 (07/2023)
same setting in He al. 2020). for the detection Pascal VOC, Faster R-CNN Ren et al. 2016 with R50-C4 backbone is first fine-tuned end-to-end on the VOC 12-12 trainval set and then evaluated on the VOC 07 test set with the COCO suite of metrics (Lin et al. 2014 The image scale is [480, 800] pixels during training and 800 at inference. For the detection and segmentation on COCO dataset, a Mask R-CNN (He al. 2017) with C4 backbone (1x schedule) is applied for the end-to-end fine-tuning. The model is tuned on the train2017 set and evaluate on val2017 set, where the image scale is in [640, 800] pixels during training and is 800 in the inference.
Imago-Toxe Text representation lcarning: Following the CLIP sctting (Radford et al. 2021 1. we adopt ResNet50 as the vision encoder backbone. All methods are pre-trained for 32 epochs, with the mini-batch size as 2048, and the learning rate as 5e-4, with cosine annealing scheduler.
Estimation of fgg with MoCo-v2: Following the strategy in Wang & Isola 2020), we estimate fgg with not only the cost between queries and keys, but also with the cost between queries. Specifically, at each iteration, let be the mini-batch size, N be the queue size, ( JM44 the query features, and (5, be the key features. The conditional distribution is calculated as:
t (-1a))
Σ Db J9L) En e (5g
To be clear, the Pytorch-like pseudo-code is provided in Algorithm In MoCo-v2 framework, as the keys are produced from the momentum encoder, this estimation could help the main encoder get involved with the gradient from the conditional distribution, which is consistent with the formulation in Section
33

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Algorithm Py'oorch-like style pseudo-code CACR with MoCo-v2 at each iteration.
MAGNSSHARRRRSE Inputs RUGSSHRGGUQPSSS
t. t_pos, t_neg: hyper-paranaterss in CACR nomentun
in_1iat=[B0, B2, .... BK] 1ist nini-batchea length (K+1) mini-batchee (Mx3:224:224), N denotes batch_eize encoder q: main encoder; encoder k: nomentum encoder
queae: dictionary
queue
features keys (dxl);; denotes the feature dimension
ллваноо the enbeddings of all samples чнннлллванеенн [encodar_q for k_iiat (encoder_KKin) for in in_list]
in_list] a list of (K+1) queries q: N
stacked_k stec din=0) keys x: (K+1) x M d
şасннtettta:: compute the loss **лннеееTА:: CACR_1oss_ 1os,__ps, CACR_lose_neg 0.0, 0.0
for rango(Len(in, #load mini-batch with y sanples queries
q q.liet
mask list maax.pop(k) the rest mini-batches used as positive and negative samples
*uннaaaaaa compute the positive cost #аtereeussss
# calculate the cost of noving positive samples: x
cost_for_poo (q
(eacked_[(nakk)) (pr dim--1) (in=-1).poo(2).... (2) transpose point-to-point
with torch.no_ gradd)) # the calculation involves mementun encoder, so with no grad here.
calculate the conditional distribation: H
veighte_for. for_po
toftner (co_p_for_____p..______ (ot_po..mu((tpoo)))) dim-1) calculate the positive conditionsl distribution
calculate the positive cest with the empirical mean
CACR, loss + (cost_ Beact_for__oatte www.opeigtts_torrpppppp. wtt.toooossss((((( aum(()).maa()))
compute the loss of negative samples ИААЛИИААИИ # caloulate the cost and reights of negative samples from the queue: H x K sq_ distt_ffr__coeeee (2 2 mn(q, queue))
aq_ dists_ forats_tor_weightt sq_diata_foo for_coat-
sith_intra, batch: compute the distance negative samples in the mini-bated Kx(H-1)
intra batch, sq_ dists torch.norm(q[: [onne] Mona] 9, (oun-1)-ppotte masked select(-torch, (-ttorcheyy(l..hhaoo(()) dtyperbool).
cuda()), (oda()).veevqg.... viev [6], q.shape 1)
combine the distance of negative samples from the batch_ sq_ditta)) din-1)
and intra-batch: (X+()-)1)
sq feg_dists_for_cooo fos_ oq_dista_for_wet for__righr_eee torch. torch.cat
(rrch.cat((sgddiii fist_ cost. intra_bater fiiata_for_eeghtaa intra_ [at__attchh___dttaaa din=i)
calculate nogative
batch--True (x(X)-))), din=1)
we weights for nag torch.softmax softmax dists for Beights.mml(tttte.
# calculate the negative cost with the empirical mean beg_dista_for_coee (=1.)) (eeel))aeiaaaaaaorrr.... .ang).sur (1) nean()
cACR_loss_neg
combine the loss of positive cast and negative cost and update main encoder CACR
LACC__loa__po/lenttcc (in, HCAC) wat.CCCRoo___a_eeeeeeeeee aaa__neg/een(in_Ila LACR_lose backward()
update (ancoder, q.paaans) # SCD update: main encoder ancoder papanass n*encoder, paparan (1-m)+ancoder 9-parans #nomentun update: key ancoder
update the dictionary, dequeue and enqueue enquena (queue, k_liat(-1] anqueue the ninibatch dequoue (queus) dequeue the earlist ninibatch
pos: power function;
multipliration; concatomation.
34


Reviews:
[{"review_id": "YhearZqZ1j", "reviewer": "Reviewer_co9r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is easy to implement and sensible.\n- Experimental evaluation is largely extensive enough (see caveat in weaknesses).\n- It is encouraging to see that the method can effectively leverage multiple positive samples (K > 1).\n- The paper is well-written, and the narrative is clear and easily understandable for readers.\n\nWeaknesses:\n- Concerns over novelty as discussed above. \n- Experimental results are all of the flavor “our method beats the baselines on these benchmarks”. The paper would benefit from a closer study of the effect CACR has on the learned model (i..e, not benchmark evaluations, but experiments that probe the properties of the method). \n- Why include the 5-shot fine-tune results in table 6? The networks have all overfitted and get worse performance than linear probe - why not just show linear probe\n\nConclusion:\n\nI am loathe to reject papers due to lack of novelty, and shall not do so in this case. But I would like to register my dissatisfaction at the way the paper is written. It presents itself in a form in which it purports to present a “powerful new contrastive leanring method”, but the method is very similar to prior work, and the experimental results, whilst solid, are not enough to overcome this (e.g., 1.5% improvement when CACR is added to the MoCo-v2 method). My recommendation to the authors is to position it more as a study of the role of hard positive and negative samples, and position the paper more as an exploration of contrastive learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "First, thank you to the authors. I enjoyed reading their work, and believe it has a number of merits.\n\nSummary:\n\nThis paper presents a simple and effective contrastive leanring method for assigning increased importance weighting to positive samples that are far from the query and negative samples that are close to the query. In other words, the proposed method samples hard negative and hard positive sampling in contrastive learning.\n\nBackground:\n\nThe paper's contribution lies in proposing a unified approach to both hard positive and negative sampling. The method demonstrates promising results, and the message of the paper is simple and clear to understand.\n\nHowever, I have concerns about the novelty of the paper. The negative sampling strategy (“contrastive repulsion”) is exactly the same probability distribution as [1] and should be cited accordingly in Sec 3.1. The main methodological contribution is the suggestion of simultaneously applying the same approach (with inverted weighting) to sample hard positives. Whilst this is reasonable, the conceptual contribution is marginal as other works also consider similar methods (see e.g., [2]). \n\nThe theory section is reasonable and correct, but not a significant contribution: the results are all standard. \n\nReferences:\n\n[1] Contrastive learning with hard negative samples, Robinson et al ICLR 2021.\n[2] Unsupervised Representation Learning by Invariance Propagation, Wang et al. NeurIPS 2020.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Add discussion of related hard negative and positive sampling approaches and cite in appropriate places (e.g., \"contrastive repulsion\" section 3.1). ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FehhKJYoZ2", "reviewer": "Reviewer_p4Wu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The targeting problem \"contrastive representation learning\" is an essential topic in the machine learning community, and the proposed algorithm seems reasonable. Moreover, the writing is easy to follow.\n\nHowever, there are also some critical defects in this work.\n\n1. The survey along the contrastive representation learning is insufficient, and some advanced/sota methods are not considered/discussed in both related work and experiments, such as [1,2,3,4].\n2. The theoretical analysis in Sec.4 is limited and needs further discussion. Specifically, a) In Lem.4.1, it is non-trivial to make sure $\\pi_{\\theta}^{+}=[(x^+|x_0)$. b) It seems that we have to let $c(z_1,z_2)=d(z_1, z_2)$ to derive Lem.4.2 and Lem.4.3, which means that both $t^+$ and $t^-$ should be $1.0$ in Eqn.(3) and Eqn.(4). Yet CACR cannot achieve promising performance in this case as shown in Tab.12 and Tab.13. c) In Sec.4.1, $x$ and $x^+$ are not strictly independent because they are obtained by transforming from the same sample $x_0$ as described in Sec.3.\n3. The experiments are unconvincing because some of the sota methods are not included, such as [1,2,3,4].\n4. Some of the notions are not introduced/explained. For example, in Eqn.(5), $\\delta_{x_j}$ and $\\delta_{x_{ik}^+}$. In Lem.4.3, is the M same as that one in Eqn. (5)?\n\n\nRef:\n1. ADAPTIVE CONTRASTIVE LEARNING OF REPRESENTATION BY NEAREST POSITIVE EXPANSION.\n2. Adaptive Soft Contrastive Learning.\n3. Intra- and Inter-Contrastive Learning for Micro-expression Action Unit Detection.\n4. Parametric Contrastive Learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies contrastive representation learning and points out that the current literature merely considered the intra-relation between samples, leading to unsatisfactory performance. To address this, this paper introduces a doubly contrastive strategy called CACR that can attract positive samples and repel negative samples separately. Finally, experiments are conducted to validate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please consider my concerns first in Strengths & Weaknesses.\n\n1. According to Wang&Isola (2020), it is better to conduct the Alignment and Uniformity analysis to show the effectiveness of the proposed approach. \n2. In Sec.3.2, the number of M seems essential to the approximate distribution performance. Therefore, it is necessary to make an ablation study about this. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VxVXHNkBHU", "reviewer": "Reviewer_3T1n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=======\n* The proposed method is sound\n* Experiments show that the proposed method is more robust to data imbalances and noise than MocoV2.\n* The authors provide ablations, details to reproduce their results, and a theoretical analysis.\n* The authors provide experiments on a large number of benchmarks.\n* In general the paper is well-written\n* The authors acknowledge that their method could fail in the presence of false negatives.\n\nWeaknesses\n=========\n* Although CACR improves robustness, results still show a big drop in performance on imbalanced settings, close to methods not using CACR.\n* Hard-negative reweighing is a known technique in the metric learning literature [A], and, as the authors acknowledge in the related work, hard negative mining has been explored in SSL [B, C].\n\n[A] Wang, Xun, et al. \"Multi-similarity loss with general pair weighting for deep metric learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n[B] Tabassum, Afrina, et al. \"Hard negative sampling strategies for contrastive representation learning.\" arXiv preprint arXiv:2206.01197 (2022).\n[C] Xu, Lanling, et al. \"Negative Sampling for Contrastive Representation Learning: A Review.\" arXiv preprint arXiv:2206.00212 (2022).\n\nMinor issues\n==========\n* I find this sentence a bit complicated to read: \"we propose a doubly CL strategy that contrasts positive samples and negative ones within themselves separately.\"\n* \"Eqn. equation 4 and p(x+ | xdata ) for Eqn. equation 3 with x+ = T (xdata , ε )\" -> Repeated Eqn. equation\n* Lemma 4.3: \"become the Uniformity\" -> becomes\n* \"the lack of intra-positive contrast shows the gap of repulsion efficiency\" -> attraction efficiency?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a contrastive learning approach that decomposes the loss into an attraction and repulsion term and re-weight hard positives and hard negatives. An advantage of such procedure is that it puts more weight on hard negatives, which makes it more robust on imbalanced distributions. The authors provide a theoretical analysis based on Wang and Isola's work on uniformity and alignment and empirical results comparing their method to other robust contrastive learning approaches such as Huynh et al. 2020 and AU-CL. The authors provide an extensive appendix with proofs, additional results and analysis, an ablation, GPU times, experiment details, and algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Fix minor issues\n", "broader_impact_concerns": "The current broader impact is enough.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qOpxSbUeWa", "reviewer": "Reviewer_SXhF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe algorithm is well motivated, as it is intuitive to think that the single positive example typically used by CL algorithms should not be optimal. And indeed, slightly earlier proposed algorithm SwAV already showed that adding multiple small-crops as positive examples during training helps boost the performance. So this work is another positive support to this idea, with some minor difference in how the positive examples are selected and used.\n2.\tThe experiment results are good. CACR outperforms most of the existing algorithms and surpasses the MoCo-v3 by large margins on the ELEVATER benchmark.\n3.\tThe paper is in general clearly written. It also provides a lot of details in the appendix.\n\t\nWeaknesses:\n1.\tAlthough the paper is mostly written clearly, the algorithm itself is not clearly explained. Especially, some key details are not consistent between the formulas and (some of) the code. For example, the equation for loss of CA at the bottom of page 5 seems to indicate that there is a “central” representation for “x_i”, which I don’t think is actually happening as most recent contrastive learning algorithms have used the augmentation examples as positive examples without a central representation (like what was stored in memory bank in Wu et al. 2018). It is also unclear to me how the augmentations of the other examples within the same batch are used to compute the loss of CR, if the algorithm uses SimCLR style loss, they would also be treated as the negative examples, but that’s inconsistent to what was described in the equation for CR. This becomes especially confusing when the section of the large-scale datasets starts. The authors say that “following the convention, we adapt CACR loss to the MoCo-v2 framework”. What convention is followed? What does it mean by “adapting CACR loss to the MoCo-v2 framework”? \n2.\tAlthough the performance gain on the ELEVATER dataset is very significant, it feels a little bit too good to make people feel that the MoCo-v3 baseline may not be carefully done. This worry is really amplified by the relatively small performance gains on the other datasets. I feel the authors need to justify more about whether the MoCo-v3 baseline is carefully done, or to provide more model results, or to explain why these gains are so large in more convincing way (like how the gains will be built up by estabilishing the special mechanisms proposed by the authors).\n3.\tThe citation for the convention CL algorithm (Oord et al. 2018) seems to be inaccurate. The Oord et al. 2018 is a contrastive predictive coding method, which is not a contrastive learning method. I would not think this method can achieve so good performance on the reported datasets. Did the author mean the SimCLR paper?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work extends the general contrastive learning (CL) algorithms by sampling more positive examples during training. The authors first use theoretic analysis to motivate the idea of separating repulsion and aggregation in the loss formulation. They then use empirical studies showing that the proposed CACR algorithm consistently outperforms current SOTA algorithms and yield significant improvement on few-shot image classification task on dataset ELEVATER.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness.", "broader_impact_concerns": "No clear negative society impacts from this. The authors have addressed this in their paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is reviewed by four experts on this topic, and all the reviewers contributed very good comments and suggestions. We finally collect three final recommendations from the reviewers. All these three reviewers suggested the acceptance, considering that the paper provided enough evidence that support their claims through theory and experiments. In general, this work is interesting enough to be published at TMLR. So the AE suggested the acceptance of this paper.
Decision: Accept (poster)