Implicit Contrastive Representation Learning with Guided Stop-gradient
Authors: Byeongchan Lee
Sehyun Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
self-supervised learning
contrastive learning
TL;DR: We propose a self-supervised representation learning method to improve performance by implicitly applying the idea of contrastive learning using an asymmetric network architecture.

Abstract: In self-supervised representation learning, Siamese networks are a natural architecture for learning transformation-invariance by bringing representations of positive pairs closer together. But it is prone to collapse into a degenerate solution. To address the issue, in contrastive learning, a contrastive loss is used to prevent collapse by moving representations of negative pairs away from each other. But it is known that algorithms with negative sampling are not robust to a reduction in the number of negative samples. So, on the other hand, there are algorithms that do not use negative pairs. Many positive-only algorithms adopt asymmetric network architecture consisting of source and target encoders as a key factor in coping with collapse. By exploiting the asymmetric architecture, we introduce a methodology to implicitly incorporate the idea of contrastive learning. As its implementation, we present a novel method guided stop-gradient. We apply our method to benchmark algorithms SimSiam and BYOL and show that our method stabilizes training and boosts performance. We also show that the algorithms with our method work well with small batch sizes and do not collapse even when there is no predictor. The code is available in the supplementary material.

Implicit Contrastive Representation Learning with Guided Stop-gradient
Byeongchan Lee"
Sehyun Lee" KAIST Daejeon, Korea sehyun. www.:m/eteeaaaaaccccoooo
Gauss Labs
Seoul, Korea byeongchan. Leeggassslgssaabbbbbooooo
Abstract
In self-supervised representation learning, Siamese networks are a natural architec- ture for learning transformation -ivaatiiinn-vviinneeeeeee by bringing representations of positive pairs closer together. But it is prone to collapse into degenerate solution. To address the issue, in contrastive learning, contrastive loss is used prevent collapse by moving representations negative pairs away from each other. But it is known that algorithms with negative sampling the number of negative samples. So, on the other hand, there are algorithms that do not negative pairs. Many positive-only algorithms adopt asymmetric network architecture consisting of source and target encoders as key factor in coping with collapse. By exploiting the asymmetric architecture, we introduce methodology to implicitly incorporate the idea of contrastive learning. As its implementation, we present novel method guided stop-gradient. We apply our method benchmark algorithms SimSiam and BOL and show that our method stabilizes training and boosts performance. We also show that the algorithms with our method work well with small batch sizes and do not collapse even when there is no predictor. The code is available supplementary material.
robust reduction in
Introduction
Representation learning has been critical topic in machine learning. In visual representation learning, image representations containing high-level semantic information (e.g., visual concept) are learned for efficient training downstream tasks. Because human annotation labor-intensive and imperfect, learning representations without labels is getting more attention. In many cases, unsupervised or self-supervised learning (SSL) has surpassed its supervised counterpart.
In SSL [Jaiswal et al. 2020 Jing and Tian 2020 Liu et al., 2021], there are roughly two branches of algorithms. One branch is set of algorithms trained pretext tasks with pseudo labels. Examples of the pretext tasks are predicting relative positions [Doersch et al. 2015). solving jigsaw puzzles Noroozi and Favaro 2016), colorization [Zhang al. 2016]. and identifying different rotations Gidaris et al. 20188. However, relying on a specific pretext task can restrict the generality of the learned representations. The other branch is set of algorithms trained by maximizing the agreement of representations of randomly augmented views from the same image. Many algorithms in this branch adopt Siamese networks with parallel encoders as their architecture. Siamese networks natural architecture with a minimal inductive bias to learn transformationinnaan However, naive use of Siamese networks can result in collapse, i.e., constant representation for all images.
representations.
To tackle the collapse problem, there have been four strategies. The first strategy is contrastive learning [Bachman et al.) 2019 Ye et al.) 2019 Chen et al. 2020a He et al.) 2020 Misra and Maaten 2020 Chen al. 2021 Contrastive learning positive pairs (views of the same
'These authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

similarity
target
Pi
predictor (stop-gradient)
stop-gradient (prodictor)
source
source encoder fi (target encooder)
target encoder fs (source encoder)
source
target
(a) Implicit contrastive representation learning,
(b) Symmetrized use of asymmetric networks.
Figure I: (a) Dots of the same color are representations positive pair. Without contrastive loss, it aims for repelling effect by carefully determining which to make the source representation and which to make the target representation. (b) In SimSiam and B YOL, a given image x is randomly transformed into two views 21 and 22-- The views are processed by encoders fi and f2 to have projections 21 and 22. A predictor is applied on one side, and -rpdgrraieettt is applied on the other. Then, the similarity between the outputs from both sides is maximized. By using the predictor and stop-gradient alternately, a symmetric loss is constructed.
image) and negative pairs (views from different images). Minimizing contrastive loss encourages representations of negative pairs to push each other while representations of positive pairs pull each other. The second strategy is to use clustering [Caron al. 2018 Asano al.] 2019 Caron al.) 2019 2020 Li et al., 2020]. It clusters the representations and then predicts the cluster assignment. Repeating this process keeps the cluster assignments consistent for positive pairs. The third strategy is to decorrelate between other variables of representations while maintaining the variance of each variable so that does not get small [Bardes et al. [021 Zbontar al. [021]. The fourth strategy is to break the symmetry of Siamese networks [Caron et al. [2200] Grill et al. 2020 [He et al. 2020 Caron et al., 202T Chen and He 202T]. Many algorithms using this strategy make the representation from one encoder (called source encoder) follow the representation from the other encoder (called target encoder M
In this paper, we introduce a methodology to do contrastive learning implicitly by leveraging asymmetric relation between the source and target encoders. For given positive pair, one of its representations becomes the source representation (from the source encoder), and the other becomes the target representation (from the target encoder). The target representation attracts the source representation. By investigating how representations are located in the embedding space, we carefully determine which representation will be the source (or target) representation so that representations of negative pairs repel each other (Figure Ta). There is no explicit contrastive part in our loss, thus the name implicit contrastive representation learning. The main idea of the methodology can be expressed as follows: Repel in the service of attracting. We also present our guided stop-gradient method, an instance of the methodology. Our method can be applied existing algorithms SimSiam [Chen and He| 2021] and BYOL GGril et al., (2020]. We show that by applying our method, the performance of the original algorithms is boosted on various tasks and datasets.
The technical contributions of our work can be summarized as follows:
We introduce a new methodology called implicit contrastive representation learning that exploits the asymmetry of network architecture for contrastive learning (Section and 4p.
We present algorithms by applying method to benchmark algorithms SimSiam and BOL and show performance improvements in various tasks and datasets (Section 5)
We demonstrate through empirical studies that our idea can be used to improve training stability to help prevent collapse, which is fundamental problem of SSL (Section 6).
"The source encoder is also called online, query, or student encoder, and the target encoder is also called key or teacher encoder in the literature depending on the context [Wang et al. 2022 Tarvainen and Valpola 2017
Grill et al. 2020 He et al. 2020 Caron et al. 20211.

---

2 Related work
Siamese networks Siamese networks are symmetric in many respects. There are encoders of the same structure both sides, and they share weights. The inputs the two encoders have the same distribution, and the outputs are induced to be similar by the loss. On the one hand, this symmetric structure helps to learn centffrmmationnnnviranet collapsing to trivial solution. There have been several approaches to solving this collapse problem, and the approaches related to our work are contrastive learning and asymmetric learning. So, we introduce them in more detail below.
well, but the other hand, all representations risk
Contrastive learning Contrastive learning [Hddslll et al. 2006 Wang and Isola 2020 Wu et al.
2018 Hjelm et al. 2018 Tian et al. 2020] can be characterized by its contrastive loss [Le- he-Khac et al. 2020 Chopra al.] 2005) J. The basic idea is design the loss that representations of positive pairs pull together and representations of negative pairs push away. Through the loss, representations are formed where equilibrium is achieved between the pulling and pushing forces. Contrastive losses that have been used so far include noise contrastive estimation (NCE) loss Guummann and Hyvärinen 2010). triplet loss (Schroff et al. 2015 lifted structured loss (Oh Song 2016). multi-class N-pair loss [Sohn] 2016], InfoNCE loss (Oord et al. 2018). soft-nearest neighbors loss [Frosst et al.
cross-entropy loss [Chen et al.) 2020a J. All the losses mentioned above contain explicit terms in their formula that cause the negative pairs to repel each other. Under self-supervised learning scenarios, since we don't know the labels, a negative pair simply consists of views from different images. Then, in practice, there is risk that the negative pair consists of views from images with the same label, and performance degradation occurs due to this sampling bias [Chuang et al.) [2020]. In addition, contrastive learning algorithms are sensitive to changes in the number of negative samples, so performance deteriorates when the number of negative samples is small.
2019). and normalizee--eeppratuu
Asymmetric learning Another approach to avoid collapse is to break the symmetry of Siamese networks and introduce asymmetry [Wang et al. 2022). Asymmetry can be imposed on many aspects, including data, network architectures, weights, loss, and training methods. In MoCo [He et al.! 2020). the encoders do not simply share weights, and the weights of one encoder (key encoder) are moving average of the weights of the other encoder (query encoder). This technique of slowly updating an encoder is called a momentum encoder. SwAV [Caron et al. [020]] and DINO [Caron et al. 202T) apply a multi-crop strategy when performing data augmentation to make the distribution of inputs into encoders different. Also, the loss applied to the outputs from the encoders is asymmetric. Furthermore, DINO, stop-gradient applied to the output of one encoder (leacher encoder) and not the other (student encoder). In SimSiam [Chen and He 202T and BYO (Grill et al.) 2020]. an additional module called predictor is stacked on one encoder (source encoder), and stop-gradien is applied to the output from the other encoder (target encoder). Compared to SimSiam, in BYOL, the target encoder is a momentum encoder.
3
Main idea
In this section, we explain our motivation and intuition behind our method using an example.
Asymmetric architecture In SimSiam and BYOL, for a given image x, two views Tı and ag are generated by random transformations. The views are fed to encoders fi and fa to yield projections 21 and 22. An encoder is a backbone plus a projecto Then, a predictor h is applied to one encoder (source encoder) to have a prediction, and -f is applied to the other encoder (target encoder). The algorithms maximize the similarity between the resulting prediction and projection. The difference between SimSiam and BYOL is in the weights of the encoders. In SimSiam, the source and target encoder share the weights On the other hand, in BOL, the target encoder is a momentum encoder. That is, the weights of the target encoder are an exponential moving average of the weights of the source encoder. Due to the existence of the predictor and stop-gradient (also momentum encoder in the case of B BYOL), the algorithms have asymmetric architecture (Figure Tbp.
Ultimately, we representations from the backbone, but it is common practice to compose the loss with projections from the projector, i.e., the encoder [Chen al. 2020a].
Applying stop- oooggradient to means treating as a constant. In actual implementation. is detached from the computational graph prevent the propagation of the gradient.
3

---

Symmetric loss Despite the asymmetry of the architecture, the losses in SimSiam and BYLL are symmetrized. After alternately applying a predictor and stop- cop-gradient to the two encoders, the following loss is obtained by adding the resulting loss terms:
-= - 2
1 (pl(p,ss(2))) + D(p2. sg(21)),
(1)
2
where -): denotes negative cosine similarity, i.c., D(p, 2) = -pp/llplla) (z/[|z][2), and sg(+)
denotes the stop-gradien operator. The range of possible values for the loss is [-1, Minimizing
the first term brings p1 (closely related to 21) closer to 22, and minimizing the second term brings p2
(closely related to 22) closer to 21- By minimizing the loss, we want to move 21 in the direction of 22 and 2i in the direction of 21.
Guided stop-gradient The main idea of our guided stop-gradient method is to design an asymmetric loss to leverage the asymmetry of the architecture. In other words, we select one of the two loss terms in Equation (T) to help with training. However, it is known that randomly constructing an asymmetric loss is not beneficial [Chen and He 2021]. To do this systematically, consider two different images 21 and 22 si- multaneously and let each other be a reference. When trying to bring the representations of one image closer, we give a directional guide on which representation to apply stop-gradient to by utilizing the geometric relationship with the representations of the other reference image.
202
221
(a) Loss terms for two imgs. (b) Selected loss terms.
Figure 2: An example for two images. The dots represent four projections of the two images. The arrows represent the expected effect of the loss terms. We want dots of the same color to come close to each other. We select loss terms so that two closest dots with different colors will fall apart.
Specifically, let ž11 (P11) and 212 (P12) be projections (predictions) from the image 21, and #21 (P21) and z22 (p22) be projections (predictions) from the image T2g Then, the symmetrized loss for two images will be as follows (Figure 2ap:
1 p(pu. L=
+ P(112 sg(zu))) + + P(pai,s(2ze)) sg(222)) 1 4
4 P(pe1.
+ ((p222.g(221))
(2)
4
Now, we select one term from the first two terms (terms to maximize agreement between 211 and 212) and another term from the remaining two terms (terms to maximize agreement between ž21 and 222). One view of the image Tj and one view of the image 22 form a negative pair. So we want the projections of the image 1 and the projections of the image 22 not to be close to each other. In the example of Figure 2 since ž11 and 221 are closest, we apply a predicto to 211 and 221 and apply stop-gradient to 212 and 222 try to separate the projections. Then, the resulting loss will be as follows (Figure 2
(p(p))))(((2)) 1 - sg((12)) + 2 D(221 sg(222)).
C C=
(3)
Selecting loss terms is equivalent determining which of the two projections of each image apply stop-gradient. Since we do this in a guided way by observing how the projections are located, we call the method Guided Moo-Grrdieent (GSG). In this way, by continuously moving toward representations that are not close together, representations of negative pairs are induced to spread well in the long run.
Implicit contrastive representation learning In our loss, there is no explicit part where the projec- tions of a negative pair repulse each other. However, the loss we designed implicitly does it. We aim for contrastive effect by making good use of the fact that the source projections go after the target projections in an asymmetric network architecture. Therefore, SimSiam and BYO with GSG can also be viewed mixture of contrastive learning and asymmetric learning.
'In the case of SimSiam and BYOL, the presence of predictor can make the interpretation tricky because when we move two close points 2 and z'. we move them indirectly by moving p = h(x) and p' h(2²) through the predictor h. We assume that the predictor /t has a good regularity as a function, that is, if || z 2|a is small, [l/(z)) 1a(²)) 2|2 is also small, So, by trying to separate and p', we separate and 2', Note that in practice. the predictor h is usually MLP with few layers.

---

similarity
similarit
Pi
determine where to apply slop gradieni
prodictor
X SG
predicter A
SK
investigate how the are loeated
Ria
enooder f
encader fz
encoder f
encoder fa
d12,21
211
d11,31
d12,28
#21
212
221
da1,22
Figure 3: Overview of our guided stop-gradient method. (1) The encoders process two images 21, D2 that are reference to each other. (2) Investigate the distances d11,21. d11,22, d12,21, and d12,22 between the projections of negative pairs. (3) Determine which side to apply hop-gradient and which to apply predictor.
Method
In this section, we explain the specific application of GSG based on the main idea described above.
For given batch images for training, a batch of pairs of images created by matching the images of the batch with the images of the shuffled batch one by one. Then, pair of images in the resulting batch consists of two images, I1 and T2. Note that since the images are paired within given batch, the batch size is the same as the existing algorithm. By applying random augmentation, we generate views æ11 and 212 from the image æ and views 221 and #22 from the image Z2. The views are processed by encoder f to yield projections and 212 of the image and projections 221 and 722 of the image Z-2-
Let dij,kd denote the Euclidean distance between projections 2i and 2kt. i.c., dij,M = || =ij- - 2sa||2:
(4)
where |1-1|2 is norm. We investigate distances d11,21, d11.22. d12,21, and d12,22. Note that there four cases total since we are looking at the distance between one of the two projections of # and of the projections of x2. Now, we find the minimum of the distances. That is,
m min 21, d11,22, d12,21. d12,22).
(5)
We apply a predictor to the two projections corresponding to the smallest of the four distances and stop-gradient to the remaining two projections. Depending on which distance is the smallest, the loss as follows:
D(p22, sg(221)), 1o4 D D(p11, + D(22,, sg(222)) ),
if m d11,21 if m d11,20
c ((111))(2112))
(6)
D(P12 sg(an) DD(21, sg(222)),
if m d12,21
(((n)) ( DD(22,, Sg(221)), if m = d12.22-
For a better understanding, refer to SegureBand Appendix A. For simplicity, we present the overview and pseudocode for SimSiam with GSG, but they analogous to BYOL with GSG.
5 Comparison
In this section, we compare SimSiam and BYOL with GSG the original SimSiam and BYOL on various datasets and tasks. For a fair comparison, we use the same experimental setup for all four algorithms on each dataset and task. For example, all algorithms perform the same number of gradient updates and exploit the same number of images in each update. We implement the algorithms with Pytorch [Paszke et al. 2019) and run all the experiments on 8 NVIDIA A A100 GPUs. Our algorithms take about two days for ImageNet pre-training and 12 hours for linear evaluation.
5

---

Table I: Comparison of representation quality under standard evaluation protocols.
Algorithm
ImageNet
CIFAR-10 10
k-NN acc. (%)
Linear acc. (%)
k-NN acc. (%)
Linear acc. (%)
SimSiam
51.7:10.11
67.9:10.09
77.0:E0.67
82.7:10.26
SimSiam w/ GSG
58.4:10.17
69.4
82.2:10.48
1.24
BYOL
56.5++0.16 62.2
69.9+-0.02
85.4-10.24
88.0+-0.09
BYOL w/ GSG
71.1++0.12
89.4+-0.21
90.3
Table 2: Comparison in transfer learning for image recognition.
Algorithm
CIFAR-10
Aireraft
Caltech
Cars
DTD
Flowers
Food
Pets
SUN397
VOC2007
SimSiam
90.0
39.7
86.5
31.8
70.9
88.8
61.6
81.4
57.8
80.5
SimSiam w/ GSG
92.3
47.0
89.6
41.2
73.0
89.5
64.3
85.1
59.8
82.6
BYOL
91.0
42.5
88.9
39.3
71.7
89.1
64.0
85.3
60.4
82.2
BYOL w/ GSG
93.6
47.1
89.9
46.9
72.6
89.5
67.1
89.1
61.6
82.7
5.1 Pre-training
We first pre-train networks an unsupervised manner. The trained networks will later used in downstream tasks. We use ImageNet (Deng al., 2009] and CIFAR-10 10 (Krizhevsky et al. 2009 as benchmark datasets. Refer to Appendix B for data augmentation details.
For ImageNet, use the ResNet-50 backbone (He et al. 2016), three-layered MLP projector, and a two-layered MLP predictor. We use batch size of 512 and train the network for 100 epochs. We the SGD optimizer with momentum of 0.9. learning rate of 0.1, and weight decay rate of 0.0001. We use a cosine decay schedule [Chen et al. 2020a Loshchilov and Hutter 2016) for the learning rate.
For CIFAR-10. 10, we use CIFAR variant of the ResNet- backbone, a two-layered MLP projector, and two-layered MLP predictor. We use a batch size of 512 and train the network for 200 epochs. use the SGD optimizer with momentum of 0.9, learning rate of 0.06, and weight decay rate of 0.0005. We do not use learning rate schedule since no scaling shows better training stability. Refer to Appendix for other implementation details.
After pre-training, we obtain representations of the images with the trained backbone and then evaluate their quality. We both k-nearest neighbors [Wu et 2018 and linear evaluation, which are standard evaluation protocols.
k-nearest neighbors For given test set image, find its representation and obtain k training set images with the closest representation. Then, we determine the predicted label the image by majority voting of the labels of the k images. We set i 200 for ImageNet and & 1 for CIFAR- 10.
Linear evaluation We freeze the trained backbone, attach a linear classifier to the backbone, fit the classifier on the training set in a supervised manner for 90 epochs, and test the classifier the test set. For ImageNet, we use a batch size of 4096 and the LARS optimizer YYuu et al. 2017]. which can work well with a large batch size. For CIFAR- 10, with momentum of 0.9, Iearning rate of 30, and cosine decay schedule. We report top-1 accuracy for all cases.
a batch size of 256 and the SGD optimizer
Results Table [ shows that applying GSG consistently increases the performance. We report error bars (mean + standard deviation) by running each algorithm three times independently and show that our method improves the performance reliably. The performance gain is obtained by keeping other experimental setups the same and changing only the hoop-gradient application method. This suggests that there is room for improvement the symmetrized use of asymmetric networks in the existing algorithms.

---

Table 3: Comparison in transfer learning for object detection and semantic segmentation.
Algorithm
VOC detection
COCO detection
Semantic segmentation
50
AP
AP75
AP
AP
AP75
Mean loU (%)
Pixel acc. (%)
SimSiam
77.0
48.8
52.2
50.7
31.2
32.8
0.2626
65.63
SimSiam w/ GSG
79.8
51.2
55.1
53.2
33.4
35.6
0.3345
76.57
BYOL
79.2
50.3
54.5
52.0
32.5
34.5
0.2615
62.79
B BYOL w/ GSG
80.5
52.0
56.4
53.7
33.8
36.1
0.2938
74.78
Table 4: Comparison between explicit and implicit contrastive learning algorithms.
Methodology
Algorithm
b 1024
b 512
= 256
Explicit contrastive
End-to-end He et al.
2020
57.3
56.3
54.9
InstDisc Wu al. 2018
54.1
52.0
50.0
MoCo He et al. [2020
57.5
56.4
54.7
SimCLK [Chen al. 2020a]
62.8
60.7
57.5
Implicit contrastive
SimSiam w/ GSG
70.1
69.4
69.9
BYOL w/ GSG
71.9
71.0
71.6
5.2 Transfer learning
An essential goal of representation learning is to obtain pre-trained backbone that can be transferred
to various downstream tasks. To evaluate whether our pre-trained backbones are transferable, we consider image recognition, object detection, and semantic segmentation tasks. We use the ResNet-50 backbones pre-trained on ImageNet. We follow the experimental setup in EEicsson et al. 2021 More implementation details can be found in Appendix D.
Image recognition We carry out image recognition tasks on different datasets. For datasets, we
adopt widely used benchmark datasets in transfer learning such as CIFAR- 10, Aireraft (Maji et al.)
2013). Caltech (Fei-Fei et al.] [004], Cars [Krause et al. [2013]. DTD (Cimpoi et al.
2014], Flowers
Nilsback and Zisserman 2008), Food [Bossard al. [014]) Pets (Parkhi al. 2012]. SUN397 (iaoo
et al. 2010). and VOC2007 [Everingham et al. 2010). These datasets vary in terms of the amount of data or the number of classes. We report the average precision AP 11 recall levels {0, 0.1,..,..
on VOC2007, mean per-class accuracy on Aircraft, Pets, Caltech, and Flowers, and top-1 accuracy on the rest of the datasets. For the evaluation protocol, we perform the linear evaluation.
Object detection We perform object detection tasks on Pascal-VOC [Eeernnghm et al. 2010 and MS-COCO [Lin et al. 2014]. We use VOC2007 trainval the training set and VOC2007 test the test set. We report AP AP, and AP7s- APso and AP7s are average precision with intersection over union (IoU) threshold 0.5 and 0.75, respectively. We freeze the pre-trained backbone except for the last residual block. We use a Feature Pyramid Network [Lin et al. [017)] to extract representations, predict. We do the experiments the Detectron2 platform
and Faster R-CNN (Ren al. 2015
[W et al. 2019].
Semantic segmentation We conduct semantic segmentation tasks on MIT ADE20K Zhou et al. 2019]. We use ResNet-50 as the encoder and use UPerNet [Xiao et al. 2018 (the implementation in the decoder. It is based on Feature Pyramid Network and Pyramid Pooling Module Zhao et al. 2017). We train for 30 epochs and test on the validation set. We report mean IoU and pixel accuracy. For the predicted results of each algorithm, refer to Appendix E.
the CSAIL semantic segmentation framework [Zhou al. 2018 2017
Results Table 2 and 3 show that applying GSG increases the performance consistently. We can conclude that pre-trained backbones are transferable to different tasks, and our method helps to get better quality representations.
7

---

aidaanee Dubled Senismet Rowetes St
4.74
ImageNet, SimSiam
ImageNet, BYOL
on Aeviiu 50
(c) CIFAR-IO, SimSiam
(d) CIFAR-1O, BYOL
Figure 4: Importance of guiding. Depending on how stop-gradient is used, performance is significantly different. It shows the best performance when used along with GSG.
5.3 The number of negative samples
known that the performance of contrastive learning algorithms vulnerable to reducing the number of negative samples TTan al. 2020 Wu et al. 20188 He al. 2020 Chen al. [020a]] In this respect, we compare our algorithms to benchmark contrastive learning algorithms (all with the ResNer-50 backbone). Table 4 reports linear evaluation accuracy on ImageNet (the performance of the benchmark algorithms is from He et al. 2020 and [Chen et al. [2020a]). ). End-to-end, SimCLR, and our algorithms use samples in the batch negative samples, so for these algorithms, in the
table denotes the batch size. InstDisc and MoCo maintain a separate memory bank or dictionary, so for these algorithms, b in the table denotes the number of negative samples from the memory bank or dictionary. The table shows that our algorithms work well with small batch sizes.
Empirical study
In this section, we broaden our understanding of stop-gradient, predictor, and momentum encoder, which are components of SimSiam and BYOL by comparing variants of the algorithms.
6.1 Importance of guiding
We demonstrate that is crucial to apply stop-gree coo-ggradeent a guided way by changing how stop gro-ggradeent used. We compare GSG with what random stop-gradient, stop- Koop-gradient is randomly applied to one of the two encoders, and a predictor is applied the other. That is, randomly select one of the four equations in Equation (6). On the other hand, in reverse stop-gradee gradient, stop-gradient is applied opposite to the intuition in GSG. In other words, we select the remaining loss terms other than the selected loss terms when GSG is used. In Equation (2), two loss terms will be selected to form the loss as follows:
random stop-gradion
and reverse stop- gradient. In
L = D(12, sg(zn))) + (P22, sg(22)))
Therefore, the random stop- noopggradient a baseline where prodggradient naïvely applied, and the reverse doo-ggradeent is the worst-case scenario according to our intuition.

---

(a) ImageNet, SimSiam
(b) ImageNet, BYOL
(c) CIFAR-10, 10, SimSiam
(d) CIFAR--RR 10, BYOL
Figure 5:
Preventing collapse. Unlike existing algorithms, algorithms to which GSG is applied do not
collapse even when the predictor is removed.
F shows the results of applying GSG, random stop-gradient, and reverse stop-gradient along with the existing algorithm for SimSiam and BYOL. We observe the k-NN accuracy at each epoch while training ImageNet and CIFAR-10. First of all, in all cases, it can be seen that the algorithm applying our GSG outperforms algorithms using other methods. In addition, it can be seen that the performance of the existing algorithm and the algorithm to which random stop-gradient is applied are similar. When applying random stop- gradient, the number of loss terms is doubled, and half of them are randomly selected, so there is expected to be significant difference from the existing algorithm. In the case of reverse stop-gradion gradddinnnt,, the performance drops significantly. This highlights the importance of using stop-gradin in guided manner.
If we look at the case of CIFAR- (Figure 4c and Figure 4dp, which relatively much smaller data and is more challenging to train stably, reverse stop-gradient is applied to SimSiam, it collapses, and the accuracy converges to 10%, which is the chance-level accuracy of CIFAR-10. However, this was not the case for BYOL. This implies that the momentum encoder can help prevent collapse. Note that SimSiam and BYOL are identical in our experimental setup, except that BYL has momentum encoder. In addition, in the case of the existing SimSiam and SimSiam with random stop-gree nop-gradient, the fluctuation the accuracy at the beginning of training is severe. However, it is relatively less in the case of SimSiam with GSG. This suggests that GSG can help the stability of training.
obtain some more interesting results. First, when
6.2 Preventing collapse
One of the expected effects of GSG prevent collapse by implicitly repelling each other away from negative pairs. It is known that SimSiam and BYOL collapse without a predictor [Chen and He 2021 Grill al. 2020]. Then, natural question is whether SimSiam and BYO will collapse even the predictor is removed when GSG is applied. Figure repports the performance when the predictor removed. In the case of CIFAR-10, 10, we run up 400 epochs to confirm complete collapse.
First, the existing algorithms collapse as expected, and the accuracy converges to the chance-level accuracy (0.1% for ImageNet and 10% for CIFAR-10). Interestingly, however, our algorithms do not collapse. This shows that GSG contributes to training stability, which is our method's intrinsic advantage. Nevertheless, the final accuracy is higher when there a predictor. This indicates that the predictor in SimSiam and B BOL contributes to performance improvement.
7 Conclusion
We have proposed implicit contrastive representation learning for visual SSL., In this methodology, while representations of positive pairs attract each other, representations of negative pairs are promoted to repel each other. It exploits the asymmetry of network architectures with source and target encoders without contrastive loss. We have instantiated the methodology and presented our guided stop-gradient method, which can be applied to existing SSL algorithms such as SimSiam and B BYOL. We have shown that our algorithms consistently perform better than the benchmark asymmetric learning algorithms for various tasks and datasets. We have also shown that our algorithms are more robust to reducing the number of negative samples than the benchmark contrastive learning algorithms. Our empirical study has shown that our method contributes to training stability. We hope our work leads the community to better leverage the asymmetry between source and target encoders and enjoy the ad advantages of both contrastive learning and asymmetric learning.

---

References
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning, arXiv preprint arXiv:1911.05371, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Yarianne-ivvaraanee-cvvaaa regularization
for self-supervised learning. arXiv preprint TrXiv:2/05.04900 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-minn 101-mmn-mii discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446 461. Springer, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
Mathilde Caron, Piotr Bojanowski. Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2959- 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9922,, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607.- PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2tt:::20000049999 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
pages
Sumit Chopra. Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OS) volume 1, pages 539-546. IEEE, 2005.
Ching- Cao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8777 8775, 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 lece, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
10

---

Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self supe-viid models transfer? In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5414-5423, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voe) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. pages 178-178 178-178.77 IEEE, 2004.
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In International conference on machine learning, pages 2012-2020. PMLR, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0788, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings. 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (06), volume 2, pages 1745- IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Maroie-Marccildon. Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808.06670, 2018.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(1 ((((1))4437444 4058, 2020.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8: 1:193907-193934, 2020.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXtv:2005.04966, 2020.

---

Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
Xiao Liu, Fanjin Zhang. Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 100.0386,, 2016.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rXiv:1306.517 5151, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 17, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 19-84. Springer, 2016.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4004-4012, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini urXiv:1807.05:: 2018
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, beghhperformanc deep learning library. Advances in neural information processing systems, 32, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer, 2020.
12

---

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929- 9939. PMLR, 2020.
Xiao Wang. Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16570-16579, 2022.
Yuxin Wu, Alexander Kirillov,, Francisco Massa, Wan- Lo, and Ross Girshick. Detectron2. 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-37422 2018
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 0210-6219, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: <ttv:/008 033888 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310-1332.. PMLR, 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision, 2018
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 27:302-3221,011
13


Reviews:
[{"review_id": "zgJQjtsP11", "reviewer": "Reviewer_cNZ6", "summary": "The paper introduces the Guided Stop-Gradient (GSG) method that can be applied to SSL algorithms that adopt asymmetric dual encoders such as BYOL and SimSiam in order to boost their performance and stabilize their training. The idea of the GSG is to augment the loss function to attract different views of two different images instead of just one as it is done in BYOL and SimSiam. However, the method does not explicitly repel the representations of the views of the different images but carefully selects which positive views to attract with a stop-gradient operation. Experiments with pretraining are performed on BYOL and SImSiam models using ImageNet and CIFAR10 datasets and their trained model is applied on several downstream tasks. \n", "questions": "I have the following questions:\n\n1. I find that choices of k in Section 5.1. unintuitive. In the case of CIFAR10, wouldn’t a large k give more insights into how well the latent space is separated? On the other hand, using k=200 for ImageNet seems very large. Could you report the variation in labels for the retrieved neighbors, i.e., is the majority vote significant? This would be interesting to analyze also in case of CIFAR10 for larger k.\n\n2. In linear probing, you train the classification layer for the same amount of epochs as the backbone and on a much larger batch size, which sounds a lot. Is there a reason for increasing the batch size to 4096? Have you tried with just a few iterations and smaller batch size?\n\n3. I was wondering if you could add some discussion and analysis that would help better understand why GSG works well. It would be particularly interesting to understand why GSG isn’t affected by changes in the batch size. My hypothesis is that this is because the model isn’t explicitly penalized for misplacing (potentially many) negative examples. This is partially addressed in Appendix E, however, I find tSNE visualization subjective. Do you have any numbers supporting the separation of clusters, for example, std of representations of each class? Another experiment could be to add more images to your loss (x3, x4, …), would that hurt or boost the performance? \n\n4. I am wondering what is the dimension of the representation space? Just to be clear, in my understanding you use z in all your experiments?\n\nMinor:\n5. In Section 5.2. it is not clear whether you finetune the backbone or use the frozen model.\n\n6. When applying random stop-gradient in Section 6.1, do you randomly choose two terms out of 4 or do you randomly choose one term for each of the images? i.e, randomly choosing either first or second term and either third or fourth in Equation 2? \n\n7. I believe that the claim in line 284 that the fluctuation of the accuracy in the beginning of the training of SimSiam is less severe when the model is trained with GSG is too strong without supporting numbers. Could you add some variance analysis?\n\n8. Section 3 is helpful for understanding the idea of the model but it would be great if you could make it explicit that x1 and x2 are different images.\n", "limitations": "Code is provided but the paper checklist is not. Societal impacts nor limitations are not discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is simple and practical to implement.\n- Results are consistent and demonstrate increased performance on pretraining and downstream tasks.\n- Using this method, models can be trained with lower batch size which is a plus.\n", "weaknesses": "I believe that the paper in the current form lacks analysis (and intuition) that would help understanding why the method works better than the baselines. For example, I would like to see more analysis on the structure of the learned representation space, and the effect of the choice of the batch size and number of “negative” images added to the loss. Given that the experiments are performed on smaller datasets, this is feasible to add. See also my questions below.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1] analysis on the learned representation space\n\nPlease see [G1] in the global response above.\n\n---\n\n[Q1] choices of $k$ for $k$-NN\n\nWe tried to make the experimental settings identical to previous studies for easy and fair comparison. So for the value $k$, we used the default value in [1] for the CIFAR-10 experiments and the value used in [2, 3] for the ImageNet experiments. It is hard to report the label variation for the retrieved neighbors since it varies per example. Table 11 of [3] also shows the results for $k=20$ in ImageNet experiments. So we also report the results for $k = 20$ below. Performance was still better when GSG was applied.\n\n|Algorithm|$k$-NN acc. (%)|\n|-|-|\n|SimSiam|55.6|\n|SimSiam w/ GSG|**62.8**|\n|BYOL|60.8|\n|BYOL w/ GSG|**66.2**|\n\n---\n\n[Q2] number of iterations and batch size for linear evaluation\n\nFor linear evaluation, we used the same number of iterations and batch size as in SimSiam's paper (please refer to Section A in [4]). In linear evaluation, even after a few iterations, the accuracy comes close to the final accuracy. For example, in the case of SimSiam, the accuracy after 90 epochs is about 67.9% and the accuracy after 10 epochs is about 64.3%. We also tried batch size 256, and it gave a slightly lower accuracy (~1%).\n\n---\n\n[Q3] analysis and discussion on why GSG works well\n\nPlease see [G1] and [G2] in the global response above. Regarding adding more images to the loss, it is hard to scale naturally, given the nature of our method of comparing distances between projections. If there are N examples, we need to compare $4{N \\choose 2}$ distances. Also, when considering one example, the results obtained by comparing with the remaining N-1 negative examples may not be consistent.\n\n---\n\n[Q4] dimension of the representation space\n\nIn many algorithms (SimSiam, BYOL, SimCLR, etc.) including ours, the encoder is a backbone plus a projector, which is a shallow MLP head. In actual evaluation, we use the representations (from the backbone), which are closely related to the projections (from the projector). Attaching a projector after the backbone is a common practice to improve performance. We refer the reviewer to Line 108 and Footnote 2. So the dimension of the representation space is same to the input dimension of the projector. For CIFAR-10, it is 512, and for ImageNet, it is 2048 (please refer to Lines 42 and 49 in Appendix).\n\n---\n\n[Q5] transfer learning evaluation mode\n\nWe used the frozen model (please refer to Line 232).\n\n---\n\n[Q6] random stop-gradient\n\nWe randomly chose one of the four equations in Equation (6).\n\n---\n\n[Q7] supporting numbers for the fluctuation of the accuracy\n\nTo measure the fluctuating degree of the accuracy $x_t$ at the beginning of training, we take the first difference $y_t=x_t-x\\_{t-1}$ and find the standard deviation of $\\\\{y_t\\\\}_{t=1}^{50}$. The following shows that SimSiam w/ GSG has a much smaller standard deviation. Thus, its accuracy fluctuates less than other algorithms.\n\n|Algorithm|Std|\n|-|-|\n|SimSiam|11.306|\n|SimSiam w/ Random SG|9.755|\n|SimSiam w/ Guided SG|**5.527**|\n\n---\n\n[Q8] $x_1$ and $x_2$ are different images.\n\nThank you for your suggestion. We will add this in the revised version.\n\n---\n\n[L1] paper checklist, societal impacts, limitations\n\nIn this NeurIPS, the paper checklist was not attached to the back of the paper, but was to be filled in on the OpenReview system. You can see it at the top of this page. Societal impacts and limitations are discussed in Section G.\n\n---\n\n[1] Susmelj, Igor et al., Lightly, 2020.\n\n[2] Wu, Zhirong et al., Unsupervised feature learning via non-parametric instance discrimination, 2018, CVPR.\n\n[3] Caron, Mathilde et al., Deep clustering for unsupervised learning of visual features, 2018, ECCV.\n\n[4] Chen, Xinlei and He, Kaiming, Exploring simple siamese representation learning, 2021, CVPR."}, {"review_id": "Snqyh8Mw9r", "reviewer": "Reviewer_Wcuk", "summary": "This paper proposes a novel SSL technique that can be applied on top of SimSiam or BYOL to select where to apply asymmetric predictor. This method first computes embeddings (before the predictor) and compute relevant distances. Then based on this distance, it chooses where to apply the predictor. Experiment results show improvement of this method over SimSiam/BYOL baseline.\n", "questions": "1. Does this method effectively double the batch size? How exactly each `pairs` of images are sampled? What is memory/compute overhead?\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is well motivated, based on the dynamics of SimSiam/BYOL representation space. \n2. Experiment shows the effectiveness of the proposed method. I appreciate full evaluation including linear probe, kNN, transfer learning, and detection.\n3. Ablation of several different strategies for choosing predictors is very insightful.\n", "weaknesses": "1. Overall the innovation is limited. It is an interesting trick over SimSiam/BYOL, but not transferrable to general SSL architecture. \n2. Experiment is questionable. Using batch size 512 is not considered optimal with ImageNet pre-training. In fact, the imagenet linear probe numbers reported for the BYOL baseline are lower than the original paper, with this batch size (it is supposed to be only -0.5% lower than using 4k batch size). I understand that it is only trained on 8 NVIDIA A100 GPUs. But it is possible that hyperparameters are no longer optimal.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1] generalizability\n\nAsymmetry is an important topic in recent self-supervised representation learning [1]. Algorithms that utilize asymmetry such as SimSiam, BYOL, SwAV, and DINO are continuously emerging. In this paper, we showed that asymmetry, which was previously introduced to prevent collapse, can help improve performance if exploited more actively. Since this is the first paper in this direction, we first applied our ideas to the relatively simple SimSiam and BYOL. However, we believe that the idea of implicitly performing contrastive learning using asymmetry can be applied to other algorithms in a different way or provide a clue to the emergence of new algorithms.\n\n---\n\n[W2] performance in the original paper of BYOL\n\nWe guess the performance of BYOL in the original paper you mentioned is from Table 1 in [2]. The performance in the table is the result of a different setting including longer training (please refer to Section 3.3 in [2]). Our paper focuses on the relative performance gap between the algorithms. So the set of hyperparameters of the algorithms are all set to that of the original SimSiam paper for apple-to-apple comparison.\n\n---\n\n[Q1] sampling pairs of images\n\nWe put the given batch and a shuffled batch side by side and pair the images one by one (please refer to Appendix A). The batch size remains the same because negative samples are paired within a given batch.\n\n---\n\n[1] Wang, Xiao et al., On the importance of asymmetry for siamese representation learning, 2022, CVPR.\n\n[2] Grill, Jean-Bastien et al., Bootstrap your own latent-a new approach to self-supervised learning, 2020, NeurIPS."}, {"review_id": "fhEKDQTIx8", "reviewer": "Reviewer_imWm", "summary": "This article presents a way to improve the learning of non-contrastive self-supervised learning methods such as BYOL and SimSiam.\nIt uses incorporates implicitly contrastive notions, by removing elements of the loss that may lead to close representations collapsing together. This modification is proven to improve the representations for BYOL and SimSiam with different experiments. In particular, variations of the algorithm with other Stop Gradients are shown to be detrimental to learning, and the utility of the algorithm to prevent collapse at low batch sizes or without projectors is shown.", "questions": "Computing distances between representations is quite costly. I would expect a slowdown due to the computation for the method. Is it present, or is it compensated by the reduced number of computations needed by the Stop Gradients?\n\nHave other alternatives, such as reweighting the \"attracting\" part of the loss rather than simply removing them been considered?\n\nDo the authors have some hypotheses about the links between this work and explicit contrastive learning methods?", "limitations": "The authors have adequately addressed the limitations of the article, albeit in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presented method is simple but provides consistent improvement for both BYOL and the SimSiam learning methods. \n\nIn particular, the method improves low batch sizes as well as when the predictor is removed. \n\nThe method is ablated to consider different Stop Gradients variations.\n\nThe article is clear and well written.", "weaknesses": "I find the article a bit lacking in ablations and inquiries about the method. For instance:\n- I would have liked a variation of the algorithm with more than only 2 examples. Do using N examples for the implicit contrast improves further the method? \n- And the method prevents collapse without a predictor, but the improvement is more unclear in the general case. Some measure of dimensionality for instance, or something similar, would have helped demonstrate the improvement of the method is due to helping a potential collapse of the representations. (I do not find the t-SNE visualisations to be convincing)\n\nI am surprised by the accuracy results presented for the benchmark of the method. Exploring Simple Siamese Representation Learning, Chen et. al, 2021 provide in Table 4 accuracies for SimCLR and Moco which are much closer to the SimSiam accuracies than the one presented in this article. In particular, the End-to-End accuracies seem very low. For these reasons, I am a bit unconvinced by the benchmark.\n\nI found the 3 Figures redundant for explaining the method. Similarly, I found the method to be a bit overexplained.\n\nThe paper in its present state is a bit light, which is why I am only considering a weak accept for now.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1-1] using $N$ examples\n\nTo use $N$ examples at once when constructing the loss, we need to create decision criteria like Equation (6) that considers $4{N \\choose 2}$ distances together. We think it is hard to be done by a straightforward extension of our idea since when deciding which side to apply stop-gradient for each example, $N-1$ results from comparison with other examples may not be consistent.\n\nIn our idea, we drive representations away from each other by pairing each example with another. So it is natural to consider two examples at a time when constructing the loss. Since we calculate the cost by aggregating the losses over the batch, we after all use as many examples as the batch size in each gradient update.\n\n---\n\n[W1-2] measure of dimensionality\n\nPlease see [G1] in the global response above.\n\n---\n\n[W2] accuracy for the benchmark\n\nAs for the performance of the benchmark algorithms in Table 4, we used those reported in SimCLR and MoCo's original papers (please refer to Table B.1 in [1] and Figure 3 in [2]). The SimCLR and MoCo accuracies in Table 4 in [3] you mentioned are obtained under different settings than ours. \n\nFor SimCLR, the higher accuracy is that of a much larger batch size 4096. This again shows that the performance is greatly affected by the batch size in the case of explicit contrastive learning. \n\nFor MoCo, the performance the table is reporting is for MoCo v2 [4], which is an improved version of MoCo. Also, in the MoCo framework, negative samples are from a dictionary decoupled from the batch, but the dictionary size is not written in the paper.\n\n---\n\n[Q1] distance computation cost\n\nThere is additional computation due to computing distance. However, it is a vectorized computation that is performed once for each iteration. Passing through a ResNet is a more dominant cost.\n\n---\n\n[Q2] reweighting the loss terms\n\nThe experiment in Section 6.1 can be seen as the reweighting. When we construct the loss, we need to choose one from $L_1=\\\\{\\frac{1}{2}\\mathcal{D}(p_{11}, \\text{sg}(z_{12})), \\frac{1}{2}\\mathcal{D}(p_{12}, \\text{sg}(z_{11}))\\\\}$ and one from $L_2=\\\\{\\frac{1}{2}\\mathcal{D}(p_{21}, \\text{sg}(z_{22})), \\frac{1}{2}\\mathcal{D}(p_{22}, \\text{sg}(z_{21}))\\\\}$ (refer to Figure 2a). \n\nThis can be relaxed to put two-point distribution over each set. For each set, if we give a probability of 1 to the term chosen with the idea of GSG and 0 to the other term, it becomes GSG. Conversely, if probabilities of 0 and 1 are given, it becomes Reverse SG, and if probabilities of 0.5 and 0.5 are given, it becomes Random SG.\n\n---\n\n[Q3] link between implicit and explicit contrastive learning\n\nPlease see [G2] in the global response above.\n\n---\n\n[1] Chen, Ting et al., A simple framework for contrastive learning of visual representations, 2020, ICML.\n\n[2] He, Kaiming et al., Momentum contrast for unsupervised visual representation learning, 2020, CVPR.\n\n[3] Chen, Xinlei and He, Kaiming, Exploring simple siamese representation learning, 2021, CVPR.\n\n[4] Chen, Xinlei et al., Improved baselines with momentum contrastive learning, 2020, arXiv preprint arXiv:2003.04297."}, {"review_id": "4MqEulfKrC", "reviewer": "Reviewer_CW7J", "summary": "The paper proposes the implicit contrastive learning algorithm, which uses the guided stop-gradient to push away negative samples without the uniformity term in the contrastive loss. By applying the method to the non-contrastive methods including SimSiam and BYOL, the paradigm combines the advantages of contrastive and non-contrastive algorithms and boosts the downstream performance in various downstream tasks.", "questions": "See Weaknesses", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The improvements brought by GSG are significant and the experiments are solid.\n2. The main idea and insights of GSG are easy to follow. \n3. The combination between the asymmetric architectures and contrastive loss is interesting.", "weaknesses": "1. Besides the guided stop-gradient, simply combing the asymmetric architectures and the contrastive loss (e.g., InfoNCE) seems would have the same effect. So what is the advantage of implicit contrastive loss? Is it possible that the advantages of GSG with small batch sizes are brought by the asymmetric architecture? More comparisons between the implicit contrastive loss and the explicit contrastive loss would make this paper more solid.\n2. In Section 6.2, the authors show that GSG contributes to training stability by removing the predictors. However, the final accuracy is still far from the results with the predictor. Is it possible to show that the GSG stabilizes the training process of SimSiam with the predictor? It would be better to show more differences between SimSiam and SimSiam with GSG during the training process.\n3. When selecting where to apply stop-gradient, the paper uses the distance in the projection layer. However, the loss is calculated in the prediction layer. Besides the brief explanations on page 4, it would be better to provide more theoretical or empirical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1] asymmetric architecture + contrastive loss\n\nMoCo is an algorithm that combines asymmetric architecture (stop-gradient, momentum encoder) and contrastive loss (InfoNCE). We refer the reviewer to Figure 2c in [1]. As shown in Table 4, even in the case of MoCo, the performance is not good when the batch size is small. The advantage of implicit contrastive loss is that, unlike contrastive loss, it works well with small batch sizes. From this example, we can see that this advantage comes from the implicit application of contrastive learning rather than from asymmetry.\n\n---\n\n[W2] training stability\n\nAs described in Lines 282-285 and Figure 4c, SimSiam w/ GSG shows a more stable learning curve in the training process. To quantify this, we measure how much the accuracy $x_t$ fluctuates at the beginning of the training. We take the first difference $y_t=x_t-x\\_{t-1}$ and find the standard deviation of $\\\\{y_t\\\\}_{t=1}^{50}$. As can be seen in the following, SimSiam w/ GSG has the smaller standard deviation and is therefore more stable. Please also note that in the case of SimSiam w/ Reverse SG, it collapsed.\n\n|Algorithm|Std|\n|-|-|\n|SimSiam|11.306|\n|SimSiam w/ Random SG|9.755|\n|SimSiam w/ Guided SG|**5.527**|\n\n---\n\n[W3]\n\nSince the predictor $h$ is a two-layer MLP head, it can be expressed as $h = f_2 \\circ \\phi \\circ f_1$, where $f_1$ and $f_2$ are affine functions ($x \\mapsto Ax+b$) and $\\phi$ is the ReLU activation function ($x \\mapsto \\max(0, x)$). It is known that affine functions and ReLU are Lipschitz continuous. Note that a function $f: \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ is Lipschitz continuous if there exists a constant $L$ such that $\\lVert f(x) - f(y) \\rVert_2 \\leq L \\lVert x- y \\rVert_2$ for all $x, y \\in \\mathbb{R}^n$. Since the composition of Lipschitz continuous functions is also Lipschitz continuous, $h$ is Lipschitz continuous. So where $\\lVert z - z' \\rVert_2$ is small, $\\lVert h(z) - h(z') \\rVert_2$ is also small.\n\n---\n\n[1] He, Kaiming et al., Momentum contrast for unsupervised visual representation learning, 2020, CVPR."}]

Author rebuttal:
Dear reviewers,

We thank you for your careful reading and constructive feedback. Your comments will help us improve the quality of the paper. We have detailed responses to each reviewer individually below. We also write responses to some common questions here. We write [Gx], [Wx], [Qx], and [Lx] for a global response, weakness, question, and limitation reference, respectively. If additional explanations are needed, we will be happy to provide them.

---

[G1] analysis of why our method works well

In addition to qualitative analysis, t-SNE in Section E, we performed the following quantitative analysis to support this. We first defined between-class, within-class, and relative variance and investigated whether our method would improve representation quality in terms of the relative variance.

Let $\mathcal{X}$ be the set of all representations, and $N = \vert \mathcal{X} \vert$. For each label $i$ ($1 \leq i \leq K$), let $\mathcal{C_i}$ be the set of all representations with label $i$, and $N_i = \vert \mathcal{C_i} \vert$. Then, the total mean $\mathcal{X}$ and the class mean $\mathcal{C_i}$ are written as

$\bar{x} = \frac{1}{N} \sum_{x \in \mathcal{X}} x, \quad \bar{x_i} = \frac{1}{N_i} \sum_{x \in \mathcal{C_i}} x.$

We define the between-class variance $v_b$ and the within-class variance $v_w$ as follows.

$v_b = \frac{1}{K} \sum_{i \in [K]} d(\bar{x_i}, \bar{x}), \quad v_w = \frac{1}{K} \sum_{i \in [K]} \left( \frac{1}{N_i} \sum_{x \in \mathcal{C_i}} d(x, \bar{x}_i) \right)$,

where $[K]=\\{1,2,\cdots,K\\}$, and $d(\cdot, \cdot)$ is the Euclidean distance. So $v_b$ is the average distance between a class mean and the total mean, and $v_w$ is the average distance between a representation and its class mean. Then, the relative variance $v_r$ is written as
$v_r = v_b / v_w.$

The following is the relative variance obtained for the algorithms.

|Algorithm|Relative Variance|
|-|-|
|SimSiam|2,323|
|SimSiam w/ GSG|**2.821**|
|BYOL|2.458|
|BYOL w/ GSG|**3.049**|

The result shows our method increases between-class variance relative to within-class variance. Note also that this is similar to the goal of linear discriminant analysis (LDA) [1].

---

[G2] discussion on why our method works well

Self-supervised learning (SSL) can basically be seen as harnessing two forces. One is attracting force between positive pairs, and the other is repelling force between negative pairs (if any). Where these two forces are in balance, representations are formed. 
- If the attracting force is too large, all representations come together (collapse). 
- If the repelling force is too large, it repels even the positive pairs (sampling bias from not knowing the labels).

First, contrastive learning gained momentum in SSL research, but it had drawbacks such as performance degradation due to sampling bias and the need for a large batch size to secure many negative samples.

Later, non-contrastive learning using only positive pairs, such as SimSiam or BYOL, gained more and more attention, but these algorithms raise the question of whether there might be room for improvement by using a contrastive effect.

Our implicit contrastive learning uses negative sampling like contrastive learning, but on the surface, it uses only the attracting force like non-contrastive learning. So it can be seen as an attempt to find a sweet spot between these two domains. To this end, we leverage asymmetry, which many non-contrastive learning algorithms introduce to prevent collapse initially.

As shown in Section 5.3, our algorithms performed more robustly than the contrastive learning algorithms at small batch sizes. In many explicit contrastive learning algorithms, N-1 negative examples are explicitly repelled from one example. However, in our method, one negative example is paired with one example in an implicit way. This may be why batch size reduction does not affect our algorithms much.

Also, as shown in Sections 5.1-2, it performed better than the non-contrastive learning algorithms. This may be because our method contributes to better separation of clusters, as shown in t-SNE and [G1]. As such, we confirmed the untapped potential of the use of asymmetry through this study.

---

[1] Härdle et al., Applied multivariate statistical analysis, 2019, Springer Nature.

Meta-review: Post rebuttal, reviewers have achieved a consensus to accept the work. The AC checked all the materials and concurs that the paper has done a reasonable exploration in the direction of non-contrastive self-supervised representation learning, by implicitly doing contrastive learning with guided stop-gradient. The proposed method has been validated by a series of experimental results, including the additional ones during the rebuttal period. Please incorporate necessary changes in the final version.
Decision: Accept (poster)