EqCo:   Equivalent Rules for Self-supervised Contrastive Learning
Authors: Benjin Zhu
Junqiang Huang
Zeming Li
Xiangyu Zhang
Jian Sun
Venue:  (2021)

Abstract: In this paper, we propose a method, named EqCo (Equivalent Rules for Contrastive Learning), to make self-supervised learning irrelevant to the number of negative samples in the contrastive learning framework. Inspired by the InfoMax principle, we point that the margin term in contrastive loss needs to be adaptively scaled according to the number of negative pairs in order to keep steady mutual information bound and gradient magnitude. EqCo bridges the performance gap among a wide range of negative sample sizes, so that for the first time, we can use only a few negative pairs (e.g. 16 per query) to perform self-supervised contrastive training on large-scale vision datasets like ImageNet, while with almost no accuracy drop. This is quite a contrast to the widely used large batch training or memory bank mechanism in current practices. Equipped with EqCo, our simplified MoCo (SiMo) achieves comparable accuracy with MoCo v2 on ImageNet (linear evaluation protocol) while only involves 16 negative pairs per query instead of 65536, suggesting that large quantities of negative samples might not be a critical factor in contrastive learning frameworks.

Under review as a conference paper at ICLR 2021
EQCo: EQUIVALENT RULES FOR SELF-SUPERVISED CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose a method, named EqCo (Equivalent Rules for Contrastive Learning), make self-supervised learning irrelevant the num- ber of negative samples in the contrastive learning framework. Inspired by the InfoMax principle, we point that the margin adaptively scaled according to the number of negative pairs in order to keep steady mutual information bound and gradient magnitude. EqCo bridges the performance gap among a wide range of negative sample sizes, so that we can use only few negative pairs (e.g. 16 per query) to perform self-supervised contrastive train- ing on large-scale vision datasets like ImageNet, while with almost no accuracy drop. This is quite contrast to the widely used large batch training or memory bank mechanism in current practices. Equipped with EqCo, our simplified MoCo (SiMo) achieves comparable accuracy with MoCo v2 on ImageNet (linear evalu- ation protocol) while only involves 16 negative pairs per query instead of 65536, suggesting that large quantities negative samples might not be a critical factor in contrastive learning frameworks.
in contrastive loss needs be
INTRODUCTION AND BACKGROUND
Self-supervised learning has recently received much attention the field of visual representation
learning (Hadsell al. 2006); Dosovitskiy et al. (2014); 4): Oord et al. (2018) Bachman et al. (2019);
Hénaff et al. (2019): Wu et al. (2018): Tian et al. (2019); He et al. (2020); Misra & Maaten 2020
Grill al. 20200; Cao et al. (2020); Tian al. (2020)) as its potential learn universal represen-
tations from unlabeled data. Among various se suppevvie methods, one of the most promising
research paths is contrastive learning (Oord et al. (2018), which has been demonstrated achieve
comparable or even better performances than supervised training for many downstream tasks such as image classification, object detection, and semantic segmentation (Chen et al. 2020c He al.
2020 Chen et al. 2020a b).
The core idea of contrastive learning briefly summarized as follows: first, extracting a pair of embedding vectors (q(Z). k())) (named query and key respectively) from the two augmented views of each instance T; then, learning to maximize the similarity of each positive pair (q(T), k()) while pushing the negative pairs (q() (), k(T') (i.e., query and key extracted from different instances accordingly) away from each other. To learn the representation, an InfoNCE loss (Oord et al. (2018); Wu et al. (2018)) is conventionally employed in the following formulation (slightly modified with an additional margin term):
ela km-/
LNCE
log qr M. kD) elq m)/= + EK eq k
where q and k; (i = O..., K) stand for the query and keys sampled from the two (augmented) data distributions D and D' respectively. Specifically, ko is associated to the same instance as q's while other k,s not; hence we name Kn and k; (i > 0) positive sample and negative samples respectively in the remaining text, in which K is the number of negative samples (or pairs) for each query. The temperature T and the margin m are hyper-parameters. parameters. most previous works, m is trivially set to zero (e.g. Oord al. (2018); He et al. 2000): Chen et al. (2020a): Tian et al. (2020)) some

---

Under review as a conference paper at ICLR 2021
handcraft values (e.g. Xie et al. (2020). ). In the following text, we mainly study contrastive learning frameworks with InfoNCE loss as in Eq. unless otherwise specified.
In contrastive learning research, it has been widely believed that enlarging the number of negative samples K boosts the performance (Hénaff et al. (2019); Tian et al. (2019) Bachman et al. (201)) ).
For example, in MoCo (He et al.] (2020) the ImageNet accuracy rises from 54.7% to 60.6% under linear classification protocol when grows from 256 to 65536. Such observation further drives a line of studies how to effectively optimize under a number of negative pairs, such memory bank methods (Wu et al. (2018); He et al. (2020)) and large batch training (Chen et al. (2020a)), either of which empirically reports superior performances when K becomes large. Analogously, in the field of supervised metric learning (Deng et al. (2019); Wang et al. (0:18)((Sum) et al. (2020); Wang et al. (2020)) loss in the similar form as lq. D is often applied on a lot of negative pairs for hard negative mining. Besides, there are also a few theoretical studies supporting the viewpoint. For instance, Oord et al. (2018) points out that the mutual information between the positive pair tends to increase with the number of negative pairs K; Waan & Isola| (2020) find that the negative pairs encourage features' uniformity on the hypersphere: CCuang et al. (2020) suggests that large K leads to more precise estimation of the debiased contrastive loss; etc.
Despite the above empirical or theoretical evidence, however, we point out that the reason for using many negative pairs is still less convincing. First, unlike the metric learning mentioned above, in self-supervised learning, the negative terms k; in Eq. [ incllude both "true negative" (whose underly- ing class label is different from the query's, s, similarly hereinafter) and "false negative" samples, since the actual ground truth label is not available. So, intuitively large K should not always be beneficial because the risk of false negative samples also increases (known as class collision problem). AAooa et al. (2019) thus theoretically concludes that a large number of negative samples could not neces- sarily help. Second, some recent works have proven that by introducing new architectures (e.g., a predictor network in YOL GGril et al. 20200), or designing new loss functions (e.g.. Caron et al. (2020a): Ermolov et al. (0200), state-of -ftee-rrtth-rrttt performance can still be obtained even without any explicit negative pairs. In conclusion, it is still an open question whether large quantities of negative samples are essential to contrastive learning.
After referring to the above two aspects, we rise a question: is a large K really essential in the contrastive learning framework? We propose to rethink the question from a different view: note that in Eq. T there are three typer-parameters: the number of negative samples K, temperature т, and margin m. In most of previous empirical studies (He et al. 22000; Chen et al. (2020a),, only K is changed while r and m are usually kept constant. Do the optimal hyper-parameters of T and m varies with K? If so, the performance gains observed from larger Kss may be a wrong interpretation choices for small Ks, rather than much of an
merelyy brought by suboptimal essential.
In the paper, we investigate the relationship among three hyper-paramettrs and suggest an equivalent rule:
a
m- =loo K"
where is a constant. We find that if the margin is adaptively adjusted based on the above rule, the performance of contrastive learning is irrelevant to the size of K, in a very large range (e.g. K 2 16). For example, in MoCo framework, by introducing EqCo the performance gap between K = 256 and K = 65536 (the best configuration reported in He et al. (2020)) almost disappears (from 6.1% %% decrease 0.2%). We call this method "Equivalent Rules for Contrastive leamnin"' (EqCo). For completeness, as the other part of EqCo we point that adjusting the learning rate according to the conventional linear scaling rule satisfies the equivalence for different number of queries per batch.
Theoretically, following the InfoMax principle (Linsker (9888) and the derivation in CPC (Ord et al.] (2018)), we prove that in Eq the lower bound of the mutual information keeps steady under various numbers of negative samples K. Moreover, from the back-propagation perspective, we further prove that in such configuration the upper bound of the gradient norm is also free of
Recently some self- learning algorithums achieve new state-of of the results using different frameworks instead of conventional InfoNCE loss in Fq. [ e.g. mean teacher (in BYOI, Grill et al. (2020) and online clustering (in SWAV Caron al. (2020b)). We will investigate them in the future.
2

---

Under review as a conference paper at ICLR 2021
K's scale. The proposed equivalent rule implies that, by assigning Ko, it can "mimic" the optimization behavior under Ko negative samples even if the physical number negatives o Ku-
The "equivalent" methodology of EqCo follows the well-known linear scaling rule (Krizhevsky (014): Goyal et al. (217)), which suggests scaling the learning rate proportional to the batch size if the loss satisfies with the linear averaged form: L N Σn= O') However, linear scaling rule cannot be directly applied on InfoNCE loss (Eq. D.. which partially because InfoNCE loss includes two batch sizes (number of queries and keys respectively) while linear scaling rule only involves one, in addition the nonlinearity of the keys in InfoNCE loss. In the experiments of SimCLR (Chen et al. (2020a)), learning rates under different batch sizes are adjusted with linear scaling rule, but the accuracy gap is still very large (57.5% batchh2256 vs. 64+%@ Bete@aaaathh9999 100 epochs training).
EqCo challenges the belief that self-supervised contrastive learning requires large quantities of neg- ative pairs to obtain competitive performance, making it possible design simpler algorithms. We thus present SiMo, a simplified contrastive learning framework based on MoCo v2 (Chen et al. (020c)). SiMo is elegant, efficient, free of large batch training and memory bank; moreover, it can achieve superior performances over state-of-the-an even if the number of negative pairs is extremely small (e.g. 16), without bells and whistles.
The contributions our paper are summarized as follows:
We challenge the widely accepted belief that on large-scale vision datasets like ImageNet, large size of negative samples is critical for contrastive learning. We interpret it from different view: it may be because the hyper-parameters are not set to the optimum.
We propose EqCo, an equivalent rule adaptively set hyyee--arametted between small and large numbers negative samples, which proves to bridge the performance gap.
We present SiMo, simpler but stronger baseline for contrastive learning.
EqCo: EQUIVALENT RULES FOR CONTRASTIVE LEARNING
In this section we introduce EqCo. We mainly consider the circumstance of optimizing the InfoNCE loss (Eq. D with SGD. For each batch of training, there two meanings of the concept "batch size", i.c. the size of negative samples/pairs K per query, and the number of queries (or positive pairs) N per batch. Hence equivalent rules accordingly consist of two parts, which will be introduced in the next subsections.
2.1 THE CASE OF NEGATIVE PAIRS
Our derivation is mainly inspired by the model of Contrastive Predictive Coding (CPC) (Oord et al. (018)), in which InfoNCE loss interpreted as a mutual information estimator. We further ex- tend the method so that it is applicable to InfoNCE loss with margin term (Eq. I. which is not
considered in Oord et al. (2018).
Following the concept in Oord et al. (2018). given a query embedding q (namely the context in Oord et al. (2018)) and suppose + random key embeddings x = {xx (x.)i=o.....K where there exists exactly one entry (e.g., x,) sampled from the conditional distribution P(xilq) while others (e.g., xj) sampled from the "proposal" distribution P(x;) independently. According to candidate distributions for x (denoted by {H.).o....kk. where the probability density of x under H; is P (x) P(x//q) IIzz, P(xy). So, given the observed data X (kn...., kx} of x, the prob- ability where x sampled from Ho rather than other candidates is thus derived with Bayes theorem:
which entry corresponds
the conditional distribution, we therefore defines
PTPH. (X)
Pr(x Holq. P+P Ho (X) + E PH. (X)
pl P(ko
P(kg)
P Pika gg)
+ >K p(k.
P(ko)
P(k.)
3

---

Under review as a conference paper at ICLR 2021
where we denote PT and P the prior probabilities of Hu and H 0) respectively. We point that Eq.22nntroduce. a generalized form to that in Oord et al. (018)) by taking the priors into account. Referring to the notations in Eq. [ we suppose that Ho is the ground truth distribution of (since ko the only positive sample). By modeling the density ratio P(k; q) /P(k)) x ea k./T O,.... K) and letting Pt P* =
e-m/T the negative log-likelihood Lopt - Eg - log Pr(w Holq. X] can
be regarded the optimal value of LNCE-
Similar to the methodology offorr al. (0188), we explore the lower bound of Copti
P(ko)
P(k:lq)
Lopt q-D.ko
log
1 1 1 em/
(q).k-
P(kal q))
P(k,)
P(ko)
P(k:
(kk
~
E
log
1 m/T
E
P(kalq)
kinDD
P(k;)
(3)
P(kn)
E q DrDD.kaDD'' (q)
log
1+ K/m
P(ko
2 log(1 + Kee//r= - I(ko: q):
where - means mutual information. The approximation in the second row is guaranteed by Law of Large Numbers as well as the fact P(ki]q) = P(k,) since k, > 0) and q are "almost" independent. The inequality in the last row is resulted from P(kolq) P(kg) ko and q extracted from the same instance. Therefore the lower bound of the mutual information (noted as Shound (m, K)) between the positive pair (ko: Q) is:
I(ko, q) 2 f K) - log(1 Kem/T)= Copt
E log 1 K/n P(ko)
P(kulq)) )
(4)
logg1 KenhT)-
U-D.kg-D'ig
So, minimizing LNCE (Eq. towards Copt implies maximizing the lower bound of the mutual information, which is also satisfied when m of In the case of O, the result consistent with that in (ordd et al. (2018). Oord et al. (2018) further points out the bound increases with K, which indicates larger encourages to learn more mutual information thus could help to improve the performance.
Nevertheless, different from Oord al. (2018) our model does not require m to zero, so the lower bound Eq. is also a function of "n/ Thus we have the following theorem:
Theorem 1. (Main, EqCo for negative pairs) The mutual information lower bound of InfoNCE loss Eq. is irrelevant to the number of negative pairs K,
m 7log a K
(5)
where a is constant coefficient. And in the cireumstances the bound is given by:
P(ko)
Sbound (rlog x. K) x log(1 + +
E
log 1 P(kolq)
~ Sround (0, a))
(6)
which can be immediately obtained by substituting Eq. into Eq. We name Eq. as "equivalent condition"..
Theorem 1 suggests property of equivalency: under the condition of Eq no matter what the number of physical negative pairs K is, the optimal solution of LNCE (Eq. "equivalent" in the sense of the same mutual information lower bound. The bound controlled by hyper-parameter rather than K. Eq. further implies that the lower bound also correlates the configuration of K = a without margin, which suggests we can "mimic" the InfoNCE loss's behavior of K = Ko under different physical negative sample size K1, just by applying Eq. [ith us to simplify the existing state-of-the-ee frameworks (e.g. MoCo (He et al.] (2020)) with fewer negative samples but accurate the original configurations, which will be introduced next.
Kg. It inspires
We empirically validate Theorem as follows. Notice that fround is difficult to calculate di- rectly because Copt is not known. Instead, we plot the empirical mutual information lower bound
4

---

Under review as a conference paper at ICLR 2021
Toand
foord
18
4086
14
3 4068
00000
epochs
epochs
(a) MoCo v2 without EqCo
(b) MoCo v2 with EqCo
Figure 1: Evolution of the empirical mutual information lower bound foound during training. We use 65536 for EqCo. Results are evaluated with MoCo v2 on ImageNet. Refer Theorem for details. Best viewed in color.
fround (m, K) A log(1 +
So, we have fround fboundi when LNCE converges
the optimum Copt. fbound is an approximation of foound. In Fig. 1 we plot the evolution of fround during the training of MoCo v2 under different configurations. Obviously, when it converges, with- out EqCo foound keeps increasing with the number of negative pairs K; in contrast, after applying the equivalent condition (Eq. 5 focund converges almost the same value under different Ks. The empirical results are thus consistent with Theorem
Remarks 1. The equivalent condition in Eq. S suggests the margin is inversely correlated with K. is intuitive, because the larger K is, the more risks of class collision (Arora et al.] (2019) suffers from, used; in contrast, if is very small, we use larger m to exploit more "hard" negative samples.
need avoid over-penalty for negative samples near the query, thus smaller m
Besides, recall that the margin term m// is defined as the ratio the prior probabilities P /p* in Eq. 2 If the equivalent condition Eq. satisfies, i.e., P /P+ a K, we have p+ 1/(1 + a) (notice that KP P 1). suggesting that the prior probability of the ground truth distribution Ho is supposed to be constant ignoring the number of negative samples K. While in previous works (usually without the margin term, or 0) have p+ 1/(K + 1). It is hard to distinguish which prior is more reasonable. However at least, we intuitively suppose keeping constant prior for the ground truth distribution may help keep the optimal choices hyper-parameters steady under different Kss, which is also consistent with our empirical observations.
In Theorem 1, worth noting that K refers the number of negative samples per query. In the conventional batched training scheme, negative samples for different queries could be either (fully or partially) shared or isolated, i.e., the total number distinguishing negatives samples per batch could be different, which is not ruled by Theorem 1. However, we empirically find the differences in implementation do not result in much of the performance variation.
Remarks 2.
The following theorem further supports the equivalent rule (Theorem 1) from back propagation view:
Theorem 2. Given the equivalent condition (Eq. and query embedding q as well as the cor- responding positive sample kn, for LNCE in Eq. the expectation of the gradient norm w.r.t. is bounded by
dLNCE dq
2
exp(q ko/T)
I
1
exp(q ko/T) + al eepp(q k:/T)]
Some works (e.g., He al. 2020)) only use HLNcE//q for optimization. In contrast, other works (Chen (020at) also involve dLNce/dk, O,..., K), which we will investigate in the future.
5

---

Under review as a conference paper at ICLR 2021
Please refer the Appendix A.1 for the detailed proof. Note that assume the embedding vectors are normalized, i.e., |k,] = I = 0,.., which is also a convention in recent contrastive
learning works.
Theorem indicates that, equipped with the equivalent rule (Eg. 5p. the upper bound of the gradient norm is irrelevant the number of negative samples K. Fig. 4 (see the Appendix (A.2) further val- idates our theory: the gradient norm becomes much more steady after using EqCo under different Ks. Since the size of K affects little on the gradient magnitude, gradient scaling techniques, e-g. linear scaling rule, are not required specifically for different Ks.. Eq. alaso implies that the lemper- ature T significantly affects the gradient norm even EqCo is applied is why we only recommend to modify m for equivalence (Eq. 5). though the mutual information lower bound is determined by em/r as a whole.
2.2 THE CASE OF POSITIVE PAIRS
In practice the InfoNCE loss (Eq. D is usually optimized with batched SGD, which can be repre- sented as empirical risk minimization:
N Σ N
1
chatch NCE
=
N k,,0,,
(8)
where N is the number of queries (or positive pairs) per batch; (q, kj,) (D, (D;)) is the
j-th positive pair, and c(ح) CNCE(95 kj.0) is the corresponding loss. For different j. c()) is (almost)
independent of each other, because qj is sampled independently. Hence, | satisfies the form of linear scaling rule (Krizhevsky (2014); Goyal et al. (2017)), suggesting that the learning rate should be adjusted proportional to the number of queries N per batch.
Remarks 3.
Previous work like SimCLR (Chen et al. (2020a) also proposes to apply linear scaling rule. The difference is, in SimCLR it does not clarify the concept of "batch size" refers to the number of queries or the number of keys. However in our paper, we explicitly point that the linear scaling rule needs to be applied corresponding to the number of queries per batch (N) rather than K.
2.3 EMPIRICAL EVALUATION
In this subsection we conduct experiments on the three state-of-the-art self-supervised contrastive learning frameworks- - MoCo (He et al. (2020)), MoCo v2 (Chen et al. (2020c)) and SimCLR (Chen) cct al. (2020a)) to verify our theory in Sec. 2.1 and Sec. [2.2] We propose to alter K and N separately to examine the correctness of our equivalent rules.
Implementation details. We follow most of the training and evaluation settings recommended in the original papers respectively. The only difference is, for SimCLR, we adopt SGD with momentum rather than LARS (You et al. (2017)) the optimizer. We use ResNer-50 (He et al. (2016)) as the default network architecture. 128-d features are employed for query and key embeddings. Unless specially mentioned, all models trained ImageNer (Deng et (2009) for 200 epochs without using the ground truth labels. We report the top-1 accuracy under the conventional linear evaluation protocol according to the original paper respectively. The number of queries per batch (N) is set to 256 by default. All models are trained with 8 GPUs.
It is worth noting the way alter the number of negative samples K independent of N during train- ing. For MoCo and MoCo v2, we simply need to set the size of the memory bank to K. Specially, if K N, in the current batch the memory bank is actually composed of K random keys sampled from the previous batch. While for SimCLR, if K < N we random sample K negative keys for
*In SimCLR, the authors find that square-root learning rate scaling more desirable with LARS optimizer You et al, 2017)), rather than linear scaling rule. Also, their experiments suggest that the performance gap between large and small batch sizes become smaller under that configuration. We point that the direction is orthogonal to our equivalent rule. Besides, SimCLR does not explore the case of very small Ks (c.g. K <= 128).

---

Under review as a conference paper at ICLR 2021
70
70
66.6
51.5
37.6
01.5
07.5
65.3
Tap- Accuaay (%)
60
68.7
29.7
00.5
60.7
61.
00.5
Top Accuacy (%)
60
58.0
80.8
51.5
61.8
60.6
62.0
62.0
58.9
57.5
55
66
33.2
52.1
00
49.9
68.0
MoCo
14.4
MoCo v1 w/ EqCo
SimCLR
40
MoCo v2
40
SimCLR w/ EqCo
MoCo wy FqCo
SinCI w/ EqCojo 4093)
37.3
206
10224
4196
16384
66636
(a) MoCo and MoCo v2
(b) SimCLR
Figure 2: Comparisons with/without EgCo under different number of negative samples (noted by
K). Results are evaluated with ImageNet top-1 accuracy using linear evaluation protocol. In EqCo, set 65536 for MoCo and MoCo v2, and 256 for SimCLR (except for one data point with a = 4096, as noted in the legend). Best viewed in color.
each query independently. We do not study the case that K > N for SimCLR. We mainly consider the ease of implementation in designing the strategies; as mentioned in Remarks (Sec. 2.1 it does not affect the empirical conclusion.
Quantitative results. Fig. illustrates the effect of our equivalent rule under different K'.. Our experiments start with the best configurations (i.e. K 63536 for MoCo and MoCo v2, and K = 256 for SimCLR"). then we gradually reduce K and benchmark the performance. Results in Fig. 2 indicates that, without EqCo the accuracy significantly drops if K becomes very small (e.g. K < 64). While with EqCo, by setting to "mimic" the optimal K, the performance surprisingly keeps steady under a wide range of Ks. Fig. ( further shows that SimCLR, by setting a to number larger than the physical batch size (c.g. 4096 vs. 256), the accuracy significantly improves from 62.0% to 65.3%, suggesting the benefit of EqCo especially when the memory limited. The comparison fully demonstrates EqCo is essential especially when the number of negative pairs is small.
Besides, Table compares the results of MoCo v2 under different number of queries N, while K 65536 fixed. It clear that, with linear scaling rule (Krizhevsyy (2014); Goyal et al. (017 ) the final performance almost unchanged under different N, suggesting the effectiveness of our equivalent rule for N.
N(K 65536)
256
512
1024
Top-1 accuracy (%)
67.5
67.5
67.4
Table ImageNet accuracy (MoCo v2) vs. the number of queries per batch (N). The learning rates during training are adjusted with linear scaling rule.
SIMO: A SIMPLER BUT STRONGER BASELINE
EqCo inspires to rethink the design of contrastive learning frameworks. The previous state-of- the-arts like MoCo and SimCLR heavily rely on large quantities of negative pairs to obtain high
'In the original paper of SimCLR (Chen al. 2020a)). the best number of negative pairs is around 4096. use in experiment is 256 due GPU memory limit.
However, the largest K
'Our "mimicking" result (65.3%, a 4096, K 256) is slightly lower than the counterpart score reported in the original SimCLR paper (66.6%, with a physical batch size of K = 4096). which we think may be resulted from the extra benefits of SyncBN along with LARS optimizer used in SimCI.R, especially when the physical batch size large.

---

Under review as a conference paper at ICLR 2021
performances, hence implementation tricks such memory bank and large batch training intro- duced, which makes the system complex and tends to be costly. Thanks to EqCo, we are able to design simpler contrastive learning framework with fewer negative pairs.
Method
Epochs
Top-I (%)
CPC v2 (Henaff al.
2019
200
63.8
CMC TTin cf at. 2019
240
66.2
SimCLK Chen et a..
Z020a
200
66.6
MoCo v2 [Chen et a.
2020c
200
67.5
20.7
21.
Top- Aruniy (8)
InfoMin Alg.. [1an cf al. 20200
200
70.1
06
SiMo (K' 16, 256)
200
68.1
314
SiMo (K 256. o 256)
200
68.0
SiMo (K 256, 65536)
200
68.5
33.5
PIRL (Misra & Maaten 2020
800
63.6
SimCLK Chen et al. 20208
1000
69.3
05
5IM (0 2060
MoCo v2 [Chen ct aij 20200
800
71.1
SIMo w/la Exjo
InfoMin Aug. Tian ct 2020
800
73,0
SiMo (K 256, 256)
800
71.8
138
8
SiMo (K 256, 65536)
800
72.1
Figure 3: SiMo with/without EqCo
Table 2: State-of-the-art InfoNCE-based frameworks
We propose SiMo, simplified variant of MoCo v2 (Chen et al.
2020c equipped with EqCo. We follow most of the design in Chen et al. (2020c), where the key difterences are as follows:
Memory bank. MoCo, MoCo v2 and SimCLR v2 o CCen al. (2020b)) employ memory bank to maintain large number of negative embeddings k, in which there is side effect: every positive embedding kg always extracted from a "newer" network than the negatives' in the same batch, which could harm the performance. In SiMo, we thus cancel the memory bank as we only rely few negative samples per batch. Instead, we use the momentum encoder to extract both positive and negative key embeddings from the current batch.
Shuffling BN vs. Syne BN. In MoCo v1/v2, shuffling BN (He et al. (2020) is proposed to remove the obvious dissimilarities of the BN (Ioffe & Szegedy (0155) statistics between the positive (from current mini-batch) and the negatives (from memory bank), so that the model can make predictions based the semantic information of images rather than the BN statistics. In contrast, since the positive and negatives are from the same batch in SiMo, therefore, we use sync BN (Peng et al. 2018) for simplicity and more stable statistics. Sync BN is also used in SimCLR (Chen et al. 2020al) and SimCLR v2 (Chen et al. (2020b)).
There are few other differences, including 1) we use BN attached each of the fully-connected layers; 2) we introduce a warm-up stage at the beginning of the training, which follows the method- ology in SimCLR (Chen et al. (2020a)). Apart from all the differences mentioned above, the ar- chitecture and the training (including data augmentations) details in SiMo are exactly the same as MoCo v2's. In the following text, the number of queries per batch (N) set to 256, and the backbone network is ResNet-50 by default.
Quantitative results. First, we empirically demonstrate the necessity of EqCo in SiMo frame- work. We choose the number of negative samples K 256 as the baseline, then reduce K to evaluate the performance. Fig. F shows the result on ImageNet using linear evaluation protocol. Without EqCo, the accuracy significantly drops when K is very small. In contrast, using EqCo to "mimic" the case of large K (by setting to 256), the accuracy almost keeps steady even under very small Ks.
Table 2 further compares our SiMo with state-of-the self-supervised contrastive learning methods on ImageNet. _ Using only 16 negative samples per query. SiMo outperforms MoCo v2 (68.1% vs. 67.5%) If we increase a to 65536 to "simulate" the case under huge number of negative pairs, the accuracy further increases to 68.5%. Moreover, when we extend the training epochs to 800, we get the accuracy of 72.1 %,, surpassing the baseline MoCo v2 by 1.0%. The only entry that surpasses
SSiCCLR v2 compares the settings with/without memory bank. However, they suggest employing memory bank as the hest configuration.
We mainly compare the methods with InfoNCE loss (Eq. 1 here, though recently BYOI. ( et al. 2020 and SWAV achieve better results using different loss functions.
8

---

Under review as a conference paper at ICLR 2021
results is InfoMin Aug. (Tian et al. (2020)), which is mainly focuses data generation and orthogonal to ours. The experiments indicate that SiMo is simpler but more powerful baseline for self-supervised contrastive learning. Readers can refer to the Appendix B for more experimental results of SiMo.
4 LIMITATIONS AND FUTURE WORK
Theorem suggests that given the equivalent condition (Eq. 5 InfoNCE losses under various Ks "equivalent" in the sense of the
mutual information lower bound, which is also backed up with the experiments in Fig. [ However, Fig. 2 (a) shows that if K is smaller than a certain value (e.g. K S 16), some frameworks like MoCo start degrade significantly even with EqCo; while for other frameworks like SiMo (Fig. 3). the accuracy almost keeps steady for very small Ks. Tschannen al. (2019) also point that the principle of InfoMax cannot explain all the phenomena in contrastive learning. We will investigate the problem in the future, e.g. from other viewpoints such as gradient noise brought by small Kss (Fig. in Appendix (..nnddxxAA..80000 gives some insights).
Though the formulation of Eq. Dis very common in the field of supervised metric learning, which is usually named margin softmax cross-entropy loss (Deng al. 2019) Wang et al.) 2018 [Sun al.] 2020). Nevertheless, unfortunately, our equivalent rule seems invalid to be generalized to those problems (e.g. face recognition). The major issue lies the approximation negative samples k; to be independent of the query q. which is not satisfied in supervised tasks.
Eq. 3 we need
According to Fig. 2 and Fig. 3 the benefits of EqCo become significant if is sufficiently small (e.g. K 64). But in practice, for modern computing devices (c.g. GPUs) is not that difficult to use ~ 256 negative pairs per query. Applying EqCo to "simulate" more negative pairs via adjusting further boost the performance, however, whose accuracy gains become relatively marginal. For example, in Table 2
under 200 epochs training, SiMo with a = 65536 outperforms that of 256 by only 0.5%. It could be fundamental limitation of InfoNCE loss. We will investigate the problem in the future.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp- 15535-15545, 2019.
Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classi- fication for unsupervised visual feature learning. arXiv preprint arXiv: 14618, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020a.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint trxiv:2003.04297, 2020c.
Ching Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. arXiv preprint arXiv:2007.00224 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 248-255- Ieee, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1690-4699, 2019.
Alexey Dosovitskiy, Jost Tobias Springenherg, Martin Riedmiller, and Thomas Brox. Discrimina- tive unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems, pp. 766-774, 2014.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. arXiv preprini arXiv:2007.063-66 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.02671 776.02667, 2017.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 07733, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- fion (CVPR '66), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv: 2014.
Tsung- Yi Lin, Piotr Dollár. Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
2017.
Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):105-117, 1988.
Ishan Mísra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748. 2018
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181-6189 2018.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint 1rXiv:1905.06990 2019.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng. Zhongdao Wang, and Yichen Wei. Circle loss: unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6398-6407, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint
arXiv: 19Xiv:1906.000 05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole. Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint 0rXiv:2005.0024. 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 13625, 2019.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5265-5274, 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprini urXiv:2005. /0242, 2020.
Xun Wang. Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embed- ding learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6388 0888-637,777 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametrie instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Delving into inter- image invariance for unsupervised visual representations, arXiv preprint arXiv:2008. 1.0008117702. 1/022, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint a Xiv: 7708. 03888, 2017.
11

---

Under review as a conference paper at ICLR 2021
A DETAILS ABOUT THEOREM 2
A. PROOF OF EQ.
Given the equivalent condition (Eq. and a query embedding q as well the corresponding posi- tive sample ko, for LNCE in Eq. the expectation of the gradient norm w.r.t. is bounded by:
dLNCE
2 7
exp(q"ko/T)
E
(9)
D
dq
explq ko/r) aLE,, jexp(q k:/T)]
Proof. For simplicity, we denote the term exp(q T k;/T) as s; O,..., K). Then LNCE can be rewritten as:
8D
LNCE = -log
(10)
$u + K E
The gradient of LNCE with respect to q is easily to derived:
dLNCE dq T 80
80 K
80
ko
k
(11)
ri
TK
+
K
Σ
Owing the Triangle Inequality and the fact that ki O...., , K) is normalized, the norm of gradient is bounded by:
dLNCE dq
VI
80
|
Σ
K
a
8;
|g:
80 + 2i=1
K
8:
1=1
TK 80
+
K
K >
8i
8u
8i
+
(12)
80
K
0 Σ
1=1
80
+
a K
Σ
80
80
K
2i=1
Since the cosine similarity between and k; (i = 11=11.... 1,..., K)
is bounded in [-1,1], we know the expectation Ex,-D [s,] exists. According to Incquality and Jensen' Inequality, we have:
2
50
2
80 +
1
1
K
+
1i=
84
k,DD
2i=1
84
(13)
2
80
So + a EkINDr [sq]
Replacing 8; by Tasp(/g.///0 the proof of Theorem is completed.
12

---

Under review as a conference paper at ICLR 2021
A.2
EMPIRICAL EVALUATION ON THE MAGNITUDE OF GRADIENTS
EqCo
McCo EqCo
IdEncz/ogl
2
16
64
256
1024
4096
16384 65536
4
256 1024
4096
16384 65536
(a) @50 epochs
(b) @100 epochs
MoCo
McCo
MoCo EqCo
McCo EqCo
3
K
(c) @150 epochs
(d) @200 epochs
Figure 4: The means (solid line) and variances (ribbon, =t) of IdCnce/dgl under different Ks. We train normal MoCo v2 for 200 epochs and show the statistics at different epochs.
13

---

Under review as a conference paper at ICLR 2021
B MORE EXPERIMENTS ON SIMO
For the following experiments of this section, we report the top-1 accuracy of SiMo on ImageNet
[Deng et al. 2009) under the linear evaluation protocol. The backbone of SiMo is ResNet-50 (He al. 2016) and we train SiMo for 200 epochs unless noted otherwise.
B.1 ABLATION ON MOMENTUM UPDATE
In MoCo (He et al. 2020) and MoCo v2 (Chen et al. 2020c), the key encoder is updated by the following rule:
0₂ = 30 + (1 8)
where 9g and 0g stand for the weights of query encoder and key encoder respectively, and is the momentum coefficient. For SiMo, we also adopt the momentum update and use the key encoder to compute the features of positive sample and negative samples.
In Table 3]
we report the results of SiMo with different momentum coefficients. The number of training epochs is set to be 100, the top-1 accuracy of baseline (3 0.999) drops to 64.4%. Compared to the baseline, SiMo without momentum update (3 0) is inferior, showing the advan- tage of momentum update.
B
0.999
Accuracy (%)
62.1
64.4
Table 3: Ablation on momentum update.
B.2 ABLATION ON BN
Ssows the performance of SiMo equipped with shuffling BN or Sync BN. Likewise, we train SiMo for 100 epochs. It is easy check out that SiMo with shuffling BN struggles to perform well. Besides, compared to MoCo v2, SiMo with shuffling BN degrades significantly, and we conjecture that it is because the MLP structure of SiMo is more suitable for Syne BN, rather than shuffling BN.
Shufiling BN
Sync BN
Accuracy (%)
58.8
64.4
Table 4: Sync BN vs. shuffling BN.
B.3 SIMO WITH DIFFERENT &
As shown in 2.c a is related the lower bound of mutual information. Table reveals how accuracy of SiMo varies with the choice of a. As increase to 65536, the accuracy tends to improve, in accordance with the Eq However, when a is too large (e.g.. 262 1 44), the performance slightly drops by 0.2%.
256
1024
4096
16384
65536
262144
Accuracy (%)
68.0
68.1
68.1
68.4
68.5
68.3
Table 5: SiMo with different a.
Similar results can be found in MoCo v2. We increase K to 262144 in MoCo v2, the accuracy also descends (in Table 6)
14

---

Under review as a conference paper at ICLR 2021
K
256
1024
4096
16384
65536
262144
Accuracy (%)
67.0
67.1
67.6
67.3
67.5
67.4
Table 6: MoCo v2 with different K.
B.4 SIMO WITH WIDER MODELS
Results using wider models are presented in Table 7 For SiMo, the performance is further boosted with wider models (more channels). For instance, SiMo with ResNet-50 (2x) and ResNet-50 (4x) outperforms the baseline (68.5%) by 2% and 3,8% respectively.
Architecture
Param. (M)
a
Top-1 (%)
ResNet-50 (2x)
94
256
70.2
ResNet-50 (2x)
94
65536
70.5
ResNet-50 (4x)
375
256
71.9
ResNet-50 (4x)
375
65536
72.3
Table SiMo with wider models, All models are trained with 200 epochs.
B.5 TRANSFER TO OBJECT DETECTION
Setup We utilize FPN (Lin et al. 2017) with stack of 4 convolution layers in R-CNN head to validate the effectiveness of SiMo. Following the MoCo training protocol, we fine-tune with synchronized aatch-normalizatioon (Peng al., 2018 across GPUs. The additional initialized layers are also equipped with BN for stable training. To effectively validate the transferability of features, the training schedule is set to 12 epochs (known as 1x), which learning rate initialized 0.2 and decreased at 7 and 11 epochs with factor of 0.1. The image scales are random sampled of [640, (00) pixels during training and fixed with 800 at inference.
Results Table summarizes the fine-tuning results on COCO val2017 of different pre-training methods. Random initialization indicates training COCO from scratch, and supervised represents conventional pre-training with ImageNet labels. Compared with MoCo, SiMo achieves competi- tive performance without large quantities of negative pairs. It is also par with the supervised counterpart and significantly outperforms random initialized one.
pre-train
AP
AP50
AP75
75
AP.
AP m
API
random init
31.4
49.4
34.0
17.9
32.3
41.6
supervised
39.0
59.1
42.6
22.4
42.2
50.6
MoCo v2
39.1
59.2
42.5
23.3
42.1
50.8
SiMo
39.0
59.2
42.3
22.9
41.8
50.5
Table 8: Object detection fine-tuned on COCO.
C Toy EVALUATION OF EQCO
To evaluate the effectivvness of EqCo as mutual information (MI) estimator, following the con- figuration of Poole et al. 2019), estimate the MI lower bound of between two simple random vectors.
Specifically, given that (X, Y) are drawn from the known correlated Gaussian distribution, cal- culate the lower bound of MI between and based on their embedding. X is a 20-dimensional random variables drawn from standard Gaussian distribution. And sampled Y with the follow- ing rule:
Y px + V 1
(14)
15

---

Under review as a conference paper at ICLR 2021
where P is the given correlation coefficient and is a random variable sampled from standard Gaussian distribution and independent from X. With a known p. the ground truth MI between X and Y is easy to compute:
d I (X,Y))= - log p²)
(15)
Here, is the dimension of X and Y, and mentioned above we set d 20.
To embed X and Y, we adopt two MLPs respectively. and each MLP has hidden layer of 256 units, followed by ReLU activation function. We use Adam optimizer with learning rate of 0.0005 to optimize InfoNCE or EqCo for 5000 steps. For each training iteration, K pairs of (X, Y) are independently sampled, which means there are K- negative samples for each query. After training, the weights of MLPs frozen and we repeat estimating the lower bound of MI for 1000 times to reduce the estimating variance. For experiments with EqCo, we set a 512.
As shown in Table 9 INCE varies with K, while TEgCo remains steady. Especially, when the ground truth MI is relatively large (c.g., 8, 10), significant differences between EgCo and InfoNCE can be observed. The experiment further validates the effectiveness of EgCo.
K
64
128
256
512
Mutual Information 2.0
INCE
1.7
1.8
1.9
1.9
LEgCo
1.9
1.9
1.9
1.9
Mutual Information = 4.0
INCE
2.9
3.2
3.4
3.6
IEqCo
3.8
3.7
3.6
3.6
Mutual Information 6.0
INCE
3.6
4.1
4.5
4.9
LEqCo
5.1
5.0
4.9
4.9
Mutual Information = 8.0
INCE
3.9
4.6
5.1
5.6
IEqCo
5.8
5.7
5.7
5.6
Mutual Information 10.0
INCE
4.1
4.7
5.4
6.0
IEgCo
6.1
6.0
6.0
6.0
Table 9: Estimating mutual information by InfoNCE and EqCo with different batch size and various ground truth mutual information.
16


Reviews:
[{"review_id": "g6Hdn0vdAss", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a margin term for positive pair in contrastive loss, and discovered an equivalent rule between margin m, temperature \\tau, and number of negatives K. The authors demonstrate that once they set the three values following the equivalent rule, then relatively smaller number of negatives can also work well for contrastive learning.\n\nPros:\n- It is interesting to introduce a margin m, and properly set a relation between it and temperature and number of negatives.\n- Theoretically, the authors show that once you set the term contains K in the lower bound of MI to be constant, then the lower bound will be irrelevant to K. The equivalent rule is very interesting, but the theorem and Figure 1 is a bit trivial to me(see cons).\n- The experimental results with MoCo, MoCo v2, and SimCLR shows that the equivalent rule improves the stability over different number of negatives for contrastive learning.\n\nCons:\n- The reason why the lower bound now doesn't depends on K, is because you set the term that contains K as constant now. Now K is not an variable for the lower bound anymore, and the only variable in the lower bound is the density ratio, which is almost stable once the training converges. Also, theoretically the converged density ratio should be invariant to K. Therefore, you don't even need to plot Figure 1(b), but you will guess it's constant. To extreme, you can set I >= 0 = f_bound, but it tells you nothing. Thus,\n(1) The real question is that, how this bound can sensitively and faithfully reflect the change of ground truth mutual information? To this end, I would recommend authors to follow the experiments in [A1] to generate variables that share different amount of mutual information, and check how your bounds track that. E.g., Figure 2 in A1.\n(2) you can do such bound estimation on held-out data to check.\n- I found myself not convinced by some statements in this paper:\n(1) In abstract, \"for the first time, we can perform self-supervised contrastive training using only a few negative pairs\", some prior work [A2] has already reduce the number of negatives. So the tone here is a bit arrogant to me. Besides, BYOL paper has already shown that you can completely remove the denominator term in contrastive loss, and only do alignment.\n(2) The first bullet of contribution section that says \"large size of negative samples is critical\" is a wrong interpretation. I kind of agree that small negative also works well, at the ImageNet scale. But what will happen for uncurated dataset whose size is far beyond ImageNet, is still unknown. So the claim here should be more restricted.\n- The SIMO design choices you proposed here is exactly the same as SimCLR v2 [A3].\n(1) No memory bank, key and negative from the same batch of momentum encoder. Yes, check SimCLR v2.\n(2) SyncBN. Yes, it's in SimCLR and SimCLR v2.\n- Missing related work:\n(1) Big Self-Supervised Models are Strong Semi-Supervised Learners. NeurIPS 2020.\n(2) Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. NeurIPS 2020.\n(3) What makes for good views for contrastive learning. NeurIPS. NeurIPS 2020.\n(4) Whitening for Self-Supervised Representation Learning. arXiv\n\n[A1] On Variational Bounds of Mutual Information. ICML 2019\n[A2] Whitening for Self-Supervised Representation Learning.\n[A3] Big Self-Supervised Models are Strong Semi-Supervised Learners. NeurIPS 2020.\n\nOverall, I like the equivalent rule for stabilizing the training across different number of negatives. Meanwhile, I hope the authors could reconsider some statements. Also, it would be of great interest to see the experiments on estimating mutual information. I would consider changing my rating if my concerns get solved properly.\n\n\n==== update ====\n\nI found the bound the author provides is problematic (see my response for details). In short, the bound is problematic in that the bound will be more accurate when only one sample from the product of marginal (or negative) is used, i.e., $K=1$, while more negatives leads to less accurate bound. This definitely counters the intuition in theory where more samples should let you estimate the KL divergence between $p(x,y)$ and $p(x)p(y)$ better.\n\nI believe there is a mathematical issue of directly setting up $m=\\tau \\log \\frac{\\alpha}{K}$ as in Eq(5). Though the empirical results look good, I recommend **NOT** accept papers with theory issues.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VqaVeBKG1AV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*****************************\nSummary: \n\nThis paper focuses on self-supervised contrastive learning. Previous contrastive learning methods heavily rely on a large number of negative samples. This paper proposed a novel method with an additional margin term, and mathematically investigate the relationship among the margin term, the temperature, and the number of negative samples. The number of negative samples can be significantly reduced by tuning the margin term, while the performance remains more stable compared to previous contrastive learning methods. Furthermore, this paper proposed a MoCo-based strong baseline that can achieve comparable results with an extremely small number of negative samples.\n\n*****************************\nReasons for Score: \n\nOverall, I vote for acceptance. This paper proposed equivalent rules which successfully reduce the number of negative samples for contrastive representation learning while maintaining the performance. This motivation is very novel and interesting, and I believe the findings in this paper significantly contributes to the community of contrastive representation learning. My main concern is the assumption that the prior probabilities $P^+$ and $P^-$ satisfy $P^+/P^-=e^{-m/\\tau}$ (see Sec. 2.1). \n\n*****************************\n \nStrengths: \n\n\\+ This paper focused on the problem of using a large number of negative samples in self-supervised contrastive learning, which is very important but not fully investigated before.\n\n\\+ This paper proposed a simple but strong baseline that can achieve comparable results with an extremely small number of negative samples. To the best of my knowledge, this is the first time to succeed with a small number of negative samples in contrastive learning. It may also inspire the applications in other self-supervised learning problems where a large number of negative samples are hard to be maintained.\n\n\\+ This paper provides detailed mathematical analysis and theorems, which makes the paper theoretically strong.\n\n\\+ The proposed strategy achieves significant improvement compared to the most popular self-supervised contrastive learning methods, MoCo and SimCLR, with a small number of negative samples. \n\n*****************************\n \nConcerns & Questions: \n\n\\- The Eq. (2) includes $P^+$ and $P^-$ as prior probabilities. It is not clear to me how the priors are taken into account and whether it is theoretically correct. In addition, it is not clear the assumption $P^+/P^−=e^{−m/\\tau}$ is correct. Could more mathematical details be introduced? Why is including $P^+$ and $P^-$ called a generalized form?\n\n\\* In this paper, the temperature $\\tau$ is fixed when tuning $K$ and selecting $m$. It would be an interesting issue if Eq. (5) is rewritten as $\\tau=\\frac{m}{\\log\\alpha-\\log K}$. I wonder whether it is possible to select $\\tau$ by fixing $m$ and tuning $K$. If not, is there any insight behind this?\n\n\\* In the proposed SiMo, all the negative examples are obtained from the current batch via the momentum encoder rather than the dictionary. Table 3 in the appendix shows that momentum update is important. Note that the momentum encoder in MoCo is used for the consistent dictionary of negative samples. Since the dictionary is not used in SiMo, the momentum update should have a different role in contrastive learning, perhaps like Mean Teacher. It would be great if more analysis or explanation on this can be provided.\n\n\\* SiMo uses Sync BN rather than Shuffling BN. I wonder whether Sync BN causes extra computational cost or slowers the training speed.\n\n\n\n\n\n============================================\n\n\nAfter rebuttal:\n\nAccording to the reviewers' feedback, I would keep my score to 8 and still vote for acceptance. However, there are still two details that are expected to be fixed in the future version. First, the reason why momentum update in SiMo is important is not convincing to me. It is not clear why letting $\\theta_k=\\theta_q$ cannot ensure the loss becoming smaller. More theoretical and experimental analyses are expected to address this issue. I still encourage the authors to rethink this detail. Second, the computational cost is provided in the feedback. I encourage the authors to include the numbers in the paper. 35\\% additional cost cannot be ignored.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7LdQ9PcWV5d", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose Eqco which can get rid of the effect of the large negative sample size in contrastive learning. \n\nPros:\n1. The authors tackle an important question whether the large negative sample size is important in contrastive learning\n2. The authors propose Eqco which introduces an margin term to the InfoNCE loss. \n3. The authors empirically shows that the result algorithm Simo is less sensitive to negative sample size.\n4. The authors prove that large negative sample size does not affect the lower bound and the bound of the gradient.\n5. The authors improve Moco-v2's result using Eqco.\n\nCons:\n1. In figure 2, even though the accuracy is less sensitive to K. But increasing K still helps the performance. This result is against the theorem 1 and 2. Is there any other explanation?\n2. Gradient and lower bound may not be the only thing that affect the performance. What about generalization? Will large negative sample size help generalization? This needs a discussion\n3. Does representation learning through minimizing the mutual information help improve the performance of the downstream classifier (linear or MLP)? Will large negative sample size help because of the downstream classifier is linear/MLP? This needs a discussion.\n\nSignificance:\nThe problem is significant and the authors made some progress on it.\n\nClarity:\nThe paper is clearly written in general.\n\nOriginality:\nThe paper is novel.", "extra_scores": {}, "extra_text": {}}, {"review_id": "40GIBKn1DYm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the performance of contrastive learning under a varying number of negative examples. The authors propose a small modification to the InfoNCE loss and show tuning some hyper-parameter could significantly improve the results with a small number of negatives. I found this topic very interesting, and the proposed solution to be quite simple. It challenges the conventional belief that a large number of negatives are required for contrastive learning (though this has been challenged before, recently by BYOL). The results show that when the number of negatives (K) is small, by using the proposed method, it could greatly improve the performance.\n\nHowever, there are some questions/issues that I came across:\n\nThe theoretical analysis shows by setting m using eq 5, the mutual information lower bound could be made completely irrelevant to K; instead it’s controlled by a constant alpha. This is clearly problematic: 1) it suggests one could simply adjust alpha to improve the lower bound to an arbitrarily large number, which is impossible given that mutual information is a fixed number given a dataset. 2) experimental results show there is still a significant gap when using small K (Figure 2). This suggests alpha is not some number one could arbitrary set to improve the bound, and it may actually be related to K, which makes the current theoretical analysis potentially flawed or incomplete.\n\nThe experimental results are pretty nice when K is small (compared to baselines), however, these are somewhat inconsequent since a small K is obtained by dropping out some negative examples in the current or past mini-batch, not actual small batch size (one still cannot use a small batch size due to batch norm and actual training time). The improvement on the best of MoCov2 result looks somewhat marginal and may not be related to the number of negatives (which contradicts the claim on why SiMo helps), as the performance of MoCov2 has saturated with increased K (Table 6). If the author’s claim were true, it should be possible to improve SimCLR's performance on small batch size (to reach the same performance of large batch size training), but this was not shown in the paper (Figure 2b). What's more, Figure 2b looks worse than expected. According to SimCLR paper (Table B.1), the gap between small and big batch sizes can be significantly reduced by using a proper learning rate, so it would be more convincing to use a better learning rate (or compare performances with different learning rates).\n\nOther minor issues: 1) for Table 7, the number of training epochs is not specified. 3) the methods in Table 2 looks a bit arbitrary, several existing similarly/better performing methods (e.g. BYOL, SWAV) are not included.\n\nGiven these issues, I will give an initial rating of 5, which is contingent on the responses from the authors.\n\n--after rebuttal--\n\nI appreciate the response, and it addressed my initial concern on the mutual information bound. However, the major concern remains which is the toy-ish setup and its practical value. The main result authors showed is that dropping some negatives doesn't hurt the performance if one adjusts m using EqCo rule. While this is interesting, the current setting of few negatives (by throwing out available negatives) is toy-ish, and may not reflect the real situation when a mini-batch is small thus only few negatives are available. I'd really like to see how this could be used to deal with the situation where you indeed only has few negatives. In MoCo, one could always easily buffer negatives with EMA the network, so there's no need to use a smaller set of negatives. In SimCLR, there seems to be a bigger potential of improvement from EqCo with actual smaller batch size, but the authors should also tune the learning rate (for smaller batch sizes) to make the baseline convincing, and compare to the results where large batch size is used. While the authors added a new point of alpha=4096, the current results of the paper are still incomplete, so I would keep my score. I'd encourage the authors to update/complete these results regardless the paper is immediately accepted by ICLR.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The equivalence rules for the margin are quite interesting, but I have two main concerns with the current paper (1) the theory does not seem to justify why increasing the number of negative examples helps in contrastive learning -- in fact Table 9 shows that the bound gets smaller as K increases. (2) The experiments use a large batch size (N) but use a smaller number of negative examples (K), which does not reduce the computation cost by much. The theoretical issue (1) can be a matter of improving the writing to put less emphasis on the theory. I invite the authors to address these issues and resubmit to other ML venues. 

Detailed feedback: I believe you are missing a negative sign in your definition of optimal "loss" $\mathcal{L}_{opt}$ in Eq. (3), which resulted in AnonReviewer4's final comment.
Decision: Reject