How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning
Authors: Chaoning Zhang
Kang Zhang
Chenshuang Zhang
Trung X. Pham
Chang D. Yoo
In So Kweon
Venue: ICLR 2022 Poster (2022)
Keywords: SimSiam
Negative samples
SSL
Collapse
Covariance

Abstract: To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work~\citep{chen2021exploring} has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the $l_2$-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL. 

Published as a conference paper at ICLR 2022
How DOES SIMSIAM AVOID COLLAPSE WITHOUT NEGATIVE SAMPLES? A UNIFIED UNDERSTANDING WITH SELF-SUPERVISED CONTRASTIVE LEARNING
Chaoning Zhang & Kang Zhang"
Chenshuang Zhang & Trung X. Pham
Chang D. Yoo & In So Kweon
Korea Advanced Institute of Science and Technology (KAIST), South Korea chaoningzhang @990@ mon0llmmal.cmmmm & zhangkang@ kaist.ac.kr
ABSTRACT
To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires large number negative samples. Without negative samples yet achieving competitive performance, a recent work (Chen & He 202T) has attracted significant attention for providing a minimalist simple Siamese (Sim- Siam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the /z-normalized representation vector. This yields a uni- fied perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.
INTRODUCTION
Radford al. 2019 Devlin et al. 2019. Su et al. Nie et al. 22020) supervide learning (SSL) has also shown its potential in the field of vision tasks (Li et al.
Beyond the success of NLP (Lan et al.,
2020
2020
El-Nouby et al. 2021). Without the ground-truth label, the core of most SSL methods lies in leamning an encoder with augmentation-invaria repre- sentation (Bachman et al. 2019 He et al. 2020) Chen et al., 2020a; Caron et al.] 2020) Grill et al.]
2021 Chen et al. 2021
2020). Specifically, they often minimize the representation distance between two positive samples, T.e. two augmented views of the same image, based on a Siamese network architecture (Bromley let al. 1993). It is widely known that for such Siamese networks there exists degenerate solution, Le. all outputs "collapsing" to an undesired constant (Chen et al., 2020a) Chen & He (021]. Early works have attributed the collapse lacking repulsive component in the optimization goal and adopted contrastive learning (CL) with negative samples, i.e. views of different samples, to alleviate this problem. Introducing momentum into the target encoder, BYOL shows that Siamese architec- tures can be trained with only positive pairs. More recently. Simmiam (Chen & He (021] has caught great attention by further simplifying BYOL by removing the momentum encoder, which has been seen as a major milestone achievement in SSL. for providing minimalist method for achieving competitive performance. However, more investigation is required for the following question:
How does SimSiam avoid collapse without negative samples?
Our investigation starts with revisiting the explanatory claims in the original SimSiam paper (Chen) & He 2021) }. Notably, two components, i.e. stop gradient and predictor, are essential for the success of SimSiam Chen & He 2021). The reason has been mainly attributed to the stop gradient (Chen & He 2 by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set. [Chen & He argue that the predictor is helpful in SimSiam because h fills the gap to approximate expectation over augmentations (EOA).
Unfortunately, the above explanatory claims are found to be flawed due to reversing the two paths with and without gradient (see Sec. 2.2).. This motivates to find alternative explanation, for which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL.
"qual contribution

---

Published as a conference paper at ICLR 2022
Specifically, we propose to decompose representation vector into center and residual components. This decomposition facilitates understanding which gradient component is beneficial for avoiding collapse. Under this framework, show that a basic Siamese architecture cannot prevent collapse, for which an extra gradient component needs to be introduced. With SimSiam interpreted as pro- cessing the optimization target with inverse predictor, the analysis of its extra gradient shows that (a) its center vector helps prevent collapse via the de-centering effect; (b) its residual vector achieves dimensional de-correlation which also alleviates collapse.
Moreover, under the same gradient decomposition, we find that the extra gradient caused by neg- ative samples in InfoNCE (He et al. 2019, Chen al., 2020b aj Tian et al.) 2019; Khosla et al., 2020) also achieves de-centering and de-correlation in the same manner. It contributes to a unified understanding on various frameworks in SSL,, which also inspires the investigation of hardness- awareness Wang & Liu (202T) from the inter-anchor perspective /Zhang et al. (2022) for further bridging the gap between CL. and non-CL. frameworks in SSL., Finally, simplifying the predictor for more explainable SimSiam, we show that single bias layer is sufficient for preventing collapse.
The basic experimental settings for our analysis are detailed Appendix A. with more specific setup discussed in the context. Overall, our work is the first attempt for performing comprehensive study on how SimSiam avoids collapse without negative samples. Several works, however, have attempted demystify the success of BYOL (Grill et al., 2020). close variant of SimSiam. A technical report (Fetterman & Albrecht, 2020) has suggested the importance of batch normalization (BN) in BYOL for its success, however, a recent work (Richemond al. (020) refutes their claim by showing BYO works without BN, which is discussed in Appendix
2 REVISITING SIMSIAM AND ITS EXPLANATORY CLAIMS
Iz-normalized vector and optimization goal. SSL trains an encoder f for learning discriminative where is a certain
representation and we denote such representation vector 2, i.e. f(x) input. For the augmentation-innarian representation, a straightforward goal is to minimize the dis- tance between the representations of two positive samples, i.e. augmented views of the same image, for which mean squared error (MSH) is a default choice. To avoid scale ambiguity, the vectors are often l2 normaallee,, i.e. Z = 2 2/||= (Chen & He 2021). before calculating the MSE:
LMS ZZ 3
- Zee Zb Leosine:
which shows the equivalence of a normalized MSE loss to the cosine loss (Grill et al. 2020).
Collapse in SSL and solution of SimSiam. Based a Siamese architecture, the loss in Eq causes the collapse, i.e. I always outputs a constant regardless of the input variance. We refer to this Siamese architecture with loss Eq [ as Naive Siamese in the remainder of paper. Contrastive loss with negative samples is a widely used solution (Chen et al. (020aa). Without using negative samples, SimSiam solves the collapse problem via predictor and stop gradient, based on which the encoder is optimized with symmetric loss:
LsimStam -Pa sg((z) + P. (g(Z)))
(2)
where sg(.) is stop gradient and P is the output of predictor h, i.e. p h(z)) and P == =pp/lpp)
2.1 REVISING EXPLANATORY CLAIMS IN SIMSIAM
Interpreting stop gradient as AO. Chen & He hypothesize that the stop gradient in Eq 2 is an implementation of Alternating between the Optimization of two sub-probles -roblross,ss, which is denoted as AO. Specifically, with the loss considered as L(0,n) = Fx..T [ ((o((()))))))))))
n= the op- timization objective ming., v L(0, v)) can be solved by alternating v/' + arg min, L(0', 7) and gt-1 + arg ming C(O, ), It is acknowledged that this hypothesis does not fully explain why the collapse is prevented (Chen & Hel 2021). Nonetheless, they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difficult to approach a constant Vz.
Interpreting predictor as EOA. The AO problem (Chen & He 2021 is formulated independent of predictor h, for which they believe that the usage of predictor is related to approximating EOA for filling the gap of ignoring [. in sub-problem of AO. The approximation of TE is summarized
2

---

Published as a conference paper at ICLR 2022
Chen & support their interpretation by proof-of-concept experiments. Specif- ically, they show that updating The with a moving-averran average 7/2 < m * n72 + +1 - m) For (T)( can help prevent collapse without predictor (see Fig. (b)). Given that the training completely fails when the predictor and moving average are both removed, at first sight, their reasoning seems valid.
in Appendix A.2
2.2 DOES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA?
Reasoning flaw. Considering the stop gradient. we divide the framework into two sub-moded with different paths and term them Gradient Path (GP) and Stop Gradient Path (SGP). For SimSiam, only the sub-model with GP includes the predictor (see Fig. (a)). We point out that their reasoning flaw of predictor analysis lies in the reverse of GP and SGP. By default, the moving-averagg sub-model as shown in Fig. 1 (b), is on the same side as SGP. Note that Fig. (b) is conceptually similar to [[ (a). It is worth mentioning that the Mirror SimSiam in Fig. (c) is what stop gradient in the original SimSiam avoids. Therefore, problematic to perceive as EOA.
Fig. (c) instead of Fig.
GP
SGP
GP
GP
similarity
similarity
similarity
P
P.
predicter
predictor
average
2
2,
3
SGP
3,
SGP
encoder
encoder f
encoder
encoder
encoder
encoder
image x
image
antgge
(a) SimSiam
(b) Moving Average
(c) Mirror SimSiam
Figure I:
Reasoning Flaw in SimSiam. (a) Standard SimSiam architecture. (b) Moving-Average Model proposed in the sooo-oof-conceye experiment (Chen & He 202I). (c) Mirror SimSiam, which has the same model architecture as SimSiam but with the reverse of GP and SGP.
GP
SGP
GP
SGP
GP
similarit
similarity
similarity
inverse peedeter h
SGP
P
P.
predictor
prodictor h
pruficter
7
image x
image
image
(6) Naive Stamess
(b) SimSlam with Symmetric Predictor
SimSiam with Inverse Predictor
Figure 2: Different architectures of Siamese model. When it is trained experimentally, the inverse predictor in (c) has the same architecture as predictor h.
Method
aug
Collapse
Std
Top-1 (%)
Moving average
2
0.0108
46.57
Same batch
10
0
1
Same batch
25
Table 1: Influence of Explicit EOA. Detailed setup is reported in Appendix A.3
Explicit EOA does not prevent collapse. (Chen & He 2021 points out that "in practice, it would be unrealistic to actually compute the expectation ET But it may be possible for neural net- work (e.g., the preditor h)) to learn to predict the expectation, while the sampling of T is implicitly distributed across multiple epochs." " implicitly sampling across multiple epochs is beneficial, ex- plicitly sampling sufficient large N augmentations in a batch with the latest model would be more beneficial to approximate I.]. However, Table I shows that the collapse still occurs and suggests that the equivalence between predictor and EOA does not hold.
3

---

Published as a conference paper at ICLR 2022
2.3 ASYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN SIMSIAM
Symmetric Predictor does not prevent collapse. The difference between Naive Siamese and Sim- siam lies in whether the gradient in backward propagation flows through a predictor, however, we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path. With h being trained the same as Eq 2 we optimize the encoder f through replacing the Z Eq with P. The results in Table. show that it still leads collapse. Actually, this is well expected by perceiving h to be part of the new encoder F, i.e. p F(z) = h(f(z)). In other words, the symmetric architectures with and without predictor h both lead collapse.
Predictor with stop gradient is asymmetric. Clearly, how SimSiam avoids collapse lies in its asymmetric architecture, iee. one path with h and the other without h. Under this asymmetric archi- tecture, the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target, not vice versa. In other words, the SimSiam avoids collapse by exclud- ing Mirror SimSiam (Fig. (c)) which has loss (mirror-like Eq 2 as MMirer
(Pa Zo + P P.- Za), where stop gradient is put on the input of h, iee Pa = h(sg zal) and p. = (sg(ze)).
Predictor inverse predictor. We interpret h function mapping from & to p. and introduce a con- ceptual inverse mapping h h''iee
Method
Collapse
Top-1 (%) 66.62
Simsiam Mirror SimSiam
h (p)) Here,
as shown in Table 2 SimSiam with symmetric predic- tor (Fig. (b)) leads collapse, while SimSiam (Fig. (a)) avoids collapse. With the conceptual h-1, we in- terpret Fig. (a) the same as Fig. [ (c) which differs from Fig. (b) via changing the optimization target h -p)) This interpretation
Naive Siamese
Symmetric Predictor
Table 2: Results of various Siamese archi- tectures. Detailed trend and setup are re- ported in Appendix A.4
from Pa to 2b, i.e. 2₁
suggests that the collapse can be avoided by processing the optimization target with h-1 By con- trast, Fig. D (c) and Fig. 2 (a) both lead to collapse, suggesting that processing the optimization target with h is not beneficial for preventing collapse. Overall, asymmetry alone does not guarantee collapse avoidance, which requires the optimization target to be processed by h not h.
Trainable inverse predictor and its implication EOA. In the above, we propose a conceptual inverse predictor h- in Fig. 2 (c), however, it remains yet unknown whether such an inverse predictor is experi- mentally trainable. A detailed setup for this investiga- tion is reported in Appendix A.5 The results in Fig. 3
70
60
50
40
Top-1
30
20
show that learnable h leads to slightly inferior performance, which is expected because h-1 cannot make the trainable inverse predictor output 2 com- pletely the same as 2g. Note that it would be equiva- lent SimSiam if 2 = z,. Despite slight perfor- mance drop, the results confirm that 1-1 is trainable. The fact that -1 is trainable provides additional ev- idence that the role h plays in SimSiam is not EOA because theoretically
(1 SimSiam
10
(10 Inverse Predictor
200
400
600
800
1000
Epoch
Figure 3: Comparison of original SimSiam and SimSiam with Inverse Predictor.
cannot restore a random augmentation T from an expectation p. where
p h(z) = ET Fo (T()))
3 VECTOR DECOMPOSITION FOR UNDERSTANDING COLLAPSE
By default, InfoNCE (Chen et al. 2020a) and SimSiam (Chen & He, 2021 both adopt 42- normalization in their loss for avoiding scale ambiguity. We treat the Ia-normalized vector, i.e. Z. as the encoder output, which significantly simplifies gradient derivation and the following analysis.
Vector decomposition. For the purpose of analysis, we propose to decompose Z into two parts, Z = o + r, where o, r denote center vector and residual vector respectively. Specifically, the center vector o is defined as an average of Z over the whole representation space o, = E However, we approximate it with all vectors in current mini- batch, i.e. o = J >MM Zm, where M is the mini-batch size. We define the residual vector the residual part of Z. Le. Z Oz.
4

---

Published as a conference paper at ICLR 2022
3.1 COLLAPSE FROM THE VECTOR PERSPECTIVE
Collapse: from result to cause. A Naive Siamese is well expected to collapse since the loss is designed minimize the distance between positive samples, for which constant constitutes an optimal solution to minimize such loss. When the collapse occurs, Vi, Z; M _m=1 Zm = 0. where i denotes a random sample index, which shows the constant vector is O₂ in this case. This interpretation only suggests a possibility that a dominant o can be one of the viable solutions, while the optimization, such as SimSiam, might still lead to non-collapse solution. This merely describes o as the consequence of the collapse, and our work investigates the cause of such collapse through analyzing the influence of individual gradient components, i.e. and during training.
1
Competition between O and т. Complementary to the Standard Deviation (Std) (Chen & He, 2021), for indicating collapse, we introduce the ratio of in z, i.e. m. |o||/||=||, where * is the L2 norm. Similarly, the ratio of r in z is defined as m, = ||>|//||=|| When collapse happens, iee all vectors Z are close the center vector o, m, approaches and m, approaches O, which is not desirable for SSL. A desirable case would be a relatively small m, and a relatively large My, suggesting a relatively small (large) contribution of (r) in each Z. We interpret the cause of collapse as a competition between and r where o dominates over r, i.e. mo >> my. For Eq I the derived negative gradient on Za (ignoring Z₁ for simplicity due to symmetry) shown as:
ALMSE Zi Ze oZ.
iccosine
Geosine
Zb,
(3)
OZa
where the gradient component Zo is dummy term because the loss having zero gradient on the encoder f.
-Za Za -1 is a constant
Conjecturel. With Z. og + Ta- conjecture that the gradient component of o, is expected to up- date the encoder to boost the center vector thus in- crease m., while the gradient component of ra is ex- pected to behave in the opposite direction to increase my. A random gradient component is expected to have a relatively small influence.
m.
mo
02
random
To verify the above conjecture, we revisit the dummy gradient term Z We design loss -z, sg( and Za sg( Za - o,) to show the influence of gradi- ent component and ra respectively. The results in Fig. show that the gradient component o₂ has the effect of increasing m, while decreasing my. On the contrary, Ta helps increase m, while decreasing m..
1000 2000 3000 4000
5000 6000
Iterationn Step
Figure 4: Influence of various gradient com- ponents m, and m.-
Overall, the results verify Conjecturel.
3.2 EXTRA GR COMPONENT FOR ALLEVIATING COLLAPSE
Revisit collapse in a symmetric architecture. Based on Conjecture 1, here, we provide an intuitive interpretation on why symmetric Siamese architecture, such as Fig. (a) and (b), cannot be trained without collapse. Take ( as example, the gradient in Eq 3 can be interpreted as two equivalent forms, from which we choose Z,, Z. (o, + +rb) (o. ra) Tb- - Ta. Since r' comes from the same positive sample as Ta. it is expected that ra also increases myr however, this effect is expected be smaller than that of Ta. thus causing collapse.
Basic gradient and Extra gradient components. The negative gradient on Z, in Fig. 2 (a) is derived as Zr while that on P. in Fig. (b) is derived as P.. We perceive Z, and P, these basic Siamese architectures as the Basic Gradient. Our above interpretation shows that such basic components cannot prevent collapse, for which an Extra Gradient component, denoted as Ge, needs to be introduced to break the symmetry. As the term suggests, G. is defined as a gradient term that is relative to the basic gradient in a basic Siamese architecture. For example, negative samples can be introduced to Naive Siamese (Fig. 2 (a)) for preventing collapse, where the extra gradient caused by negative samples can thus be perceived as Ge with ZB as the basic gradient. Similarly, we can also disentangle the negative gradient on P. in SimSiam (Fig. (a)), i.e. Zz. into basic gradient (which is P,) and G, which is derived as Z, P, (note that Z, P, + G.). We analyze how G. prevents collapse via studying the independent roles of its center vector o, and residual vector T.-
5

---

Published as a conference paper at ICLR 2022
3.3 A TOY EXAMPLE EXPERIMENT WITH NEGATIVE SAMPLE
Which repulsive component helps avoid collapse? Existing works often attribute the collapse in Naive Siamese to lacking a repulsive part during the optimization. This explanation has motivated previous works to adopt contrastive learning, i.e. attracting the positive samples while repulsing the negative samples. We experiment with a simple triplet los - -sg indicates the representation of Negative sample. The derived negative gradient on Zo is Zi Zn. where Z the basic gradient component and thus G. Z, in this setup. For a sample represen- tation, what determines it as a positive sample for attracting or negative sample for repulsing is the residual component, thus might be tempting to interpret that Te is the key component of repulsive part that avoids the collapse. However, the results in Table show that the component beneficial for preventing collapse inside Ge is O, instead of re- Specifically, to explore the individual influence of oe and Те in the G., design two experiments by removing one component while keeping the other one. In the first experiment, we remove the т. in G2 while keeping the o.. By contrast, the Og is removed while keeping the re in the second experiment. In contrast to what existing explana- tions may expect, we find that the residual component O₂ prevents collapses. With Conjecture a gradient component alleviates collapse if has negative center vector. In this setup, Oe- -o, thus O, has the de-centering role for preventing collapse. On the contrary, Te does not prevent collapse and keeping re even decreases the performance (36.21% 47.41%). Since the negative sample randomly chosen, just behaves like a random noise on the optimization to decrease performance.
=
Zn). where Zn
Method
Ceriplet
Std
m,
m,
Collapse
Top-1 (%)
Baseline
Z. sg( Zb + Ge)
0.020
0.026
0.99
x
36.21
Removing Te
Za sg( Zb Oe)
0.02005
0.026
0.99
x
47.41
Removing On
Za sg( Zb, Te)
0
/
1
Table 3: Gradient component analysis with a random negative sample.
3.4
DECOMPOSED GRADIENT ANALYSIS IN SIMSIAM
challenging to derive the gradient on the encoder output in SimSiam due to a nonlinear MLP module in h. The negative gradient on Pa for LSimSiam in Eq 2 can be derived as
GSimSiam dL SimStam aP.
Zo + (Zo Pb) P. Ge,
(4)
where G. indicates the aforementioned extra gradient component. To investigate the influence of oe and r. on the collapse, similar to the analysis with the toy exam- ple experiment in Sec. B.3] design the experiment by removing one component while keeping the other. The results are reported in Table 4 As expected, the model collapses when both components in Ge are removed and the best performance achieved when both components are kept. Interestingly, the model does not collapse when either o or Te is kept. To start, we analyze how Oe affects the collapse based on Conjecturel.
O₂
re
Collapse
Top- (%)
x
66.62
v
X
x
48.08
V
66.15
X
X
V
Table 4: Gradient component analysis for SimSiam.
How alleviates collapse in SimSiam. Here, Op is used to denote the center vector of P to differentiate from the above introduced o, for denoting that of Z. In this setup G. Zi, P.- thus the residual gradient component is derived to be expected that oe helps prevent collapse of contains negative Op since the analyzed vector is P.. To determine the amount of component of o, existing in O., we measure the cosine similarity between O, Op and Op for a wide range of Vpp- The results in Fig. 5 (a) show that their cosine similarity is zero when Πp is around -0.5, suggesting Oe has - -.55pp With Conjecture 1, this negative 1pp explains why SimSiam avoids collapse from the perspective of de-centering.
o Op- With Conjecturel, it is well
How O causes collapse in Mirror SimSiam. As mentioned above. the collapse occurs in Mirror SimSiam, which can also be explained by analyzing its o.. Here, O, Op Oz, for which we evaluate the amount of component o existing in o via reporting the similarity between o 7=0:
Note that the triplet loss here does not have clipping form in Schroff al. 2015 for simplicity.

---

Published as a conference paper at ICLR 2022
and Oz. The results in Fig. 5 (a) show that their cosine similarity is when 7, is set to around 0.2. This positive 7 explains why Fig. (c) causes collapse from the perspective of de-centering.
Overall, find that processing the optimization target with h- in Fig, 2 (c), alleviates collapse (7p ~ -0.5), while processing it with h, as in Fig. [c), actually strengthens the collapse (7/ ~ 0.2). In other words, via the analysis of Oe. results help explain how SimSiam avoids collapse well as how Mirror SimSiam causes collapse from a straightforward de-centering perspective.
Cosine similanty
0.5
Ap
0.75
1.00
Cosine similariit rityy
0.5
0.25
0.0
1.00
0.75
0.5
E
0.2
0.25
0.00
-1.0
-0.5
0.0
0.5
1.0
1000
2000
3000
0.2
0.8 1.0
lteration Step
Tempture 2
(a)
(b)
(C)
Figure 5: (a) Investigating the amount of Op existing in o, Op and the amount of o, existing in
Op- O,. (b) Normally train the model as SimSiam for 5 epochs, then using collapsing loss for epoch to reduce my followed by correlation regularization loss. (c) Cosine similarity between r. (0x)) and gradient on Z, induced by a correlation regularization loss.
Relation to prior works. Motivated from preventing the collapse to constant, multiple prior works, such as W-MSE (Ermolov et al.) 2021). Barlow-twins (Zbontar et al. (021), DINO (Caron al. 202T), explicitly adopt de-centering prevent collapse. Despite various motivations, we find that they all implicitly introduce an Oe that contains a negative center vector. The success of their approaches aligns well with our Conjecturel as well as our above empirical results. Based on our findings, we argue that the effect of de-centering can be perceived as o, having a negative center vector. With this interpretation, we are the first to demonstrate that how SimSiam with predictor and stop gradient avoids collapse can be explained from the perspective of de-centering.
Beyond de-centering for avoiding collapse. In the toy example experiment Sec. 3.3 re is found to be not beneficial for preventing collapse and keeping т. even decreases the performance. Interestingly, as shown in Table 4 we find that тe alone is sufficient for preventing collapse and achieves comparable performance as Ge. This can be explained from the perspective of dimensional de-correlation, which will be discussed in Sec. 3.5
3.5
DIMENSIONAL DE-CORRELATION HELPS PREVENT COLLAPSE
Conjecture2 and motivation. We conjecture that dimensional de de-correlation increases m, for pre- venting collapse. The motivation is straightforward as follows. The dimensional correlation would be minimum if only a single dimension has very high value for every individual class and the dimension changes for different classes. In another extreme case, when all the dimensions have the same values, equivalent to having single dimension, which already collapses by itself in the sense of losing representation capacity. Conceptually, Te has no direct influence on the center vector, thus we interpret that re prevents collapse through increasing m,.
To verify the above conjecture, we train SimSiam normally with the loss in Eq 2 and train for several epochs with the loss [q[ for intentionally decreasing the m, to close zero. Then, we train the loss with only correlation regularization term, which is detailed in Appendix A.6 The results in Fig. 5 (b) show that this regularization term increases m, at a very fast rate.
Dimensional de-correlation in SimSiam. Assuming h only has single FC layer to exclude the influence of o.. the weights in FC are expected to learn the correlation between different dimensions for the encoder output. This interpretation echos well with the finding that the eigenspace of h weight aligns well with that of correlation matrix (Tian et al. 2021). In essence, the h is trained to minimize the cosine similarity between h(zz)) and [xx),, where I is identity mapping. Thus, h that learns the correlation is optimized close to I, which is conceptually equivalent to optimizing with the goal of de-correlation for Z. As shown in Table 4 for SimSiam, r. alone also prevents collapse, which
7

---

Published as a conference paper at ICLR 2022
0.8
SimSiam InfoNCE
0.0
covarlance
0.001
Top-1 (%)
40-
0.0.6
0.003
0.4
removal
0.4
0.202
0.7
0.2
0.0
100
150
0.0
100
150
200
50
150
200
30
100
150
200
Epoch
Epoch
Epoch
Epoch
Figure 6: Influence of various gradient components on m, and mo-
is attributed to the dorrollllttion effect since Te has no de-centering effect. We observe from Fig.
6 that except in the first few epochs, SimSiam decreases the covariance during the whole training. Fig. oaals reports the results for InfoNCE which will be discussed in Sec.
4 TOWARDS A UNIFIED UNDERSTANDING OF RECENT PROGRESS IN SSL
De-centering and de-correlation in InfoNCE. InfoNCE loss is a default choice in multiple seminal contrastive learning frameworks (Sohn 2016 Wu al. 2018 Oord al. 2018 Wang & Liu 2021).
The derived negative gradient of InfoNCE on Z. is proportional to Zb +
X Zi. where for the detailed
exp(Zz Z./T) A; 2xpoexp22 2./77)
and Zo Zg for notation simplicity. See Appendix A.7 4.75
derivation. The extra gradient component G, - Σi - = -0; Ei Ar for which
O₂ = -0; and T. = Ei Arr. Clearly, Oe contains negative o= as de-centering for avoiding collapse, which is equivalent to the toy example Sec. [33 when the Te is removed. Regarding Te, the main difference between Ltri in the toy example and InfoNCE is that the latter exploits batch of negative samples instead of a random one. X is proportional to exp Ze Z,), indicating that large weight is put on the negative sample when it is more similar the anchor Za. for which, intuitively, its dimensional values tend to have a high correlation with Z.. Thus, Te containing such negative representation with high weight tends to decrease dimensional correlation. To verify this intuition, we measure the cosine similarity between Te and the gradient on Zn induced by a correlation regularization loss. The results Fig. (c) show that their gradient similarity is high for a wide range of temperature values, especially when T is around 0.1 or 0.2, suggesting Te achieves similar role as an explicit regularization loss for performing de-correlation. Replacing r. with o, leads to a low cosine similarity, which is expected because O₂ has no de-correlation effect.
The results of InfoNCE in Fig. 6 resembles that of SimSiam in terms of the overall trend. For ex- ample, InfoNCE also decreases the covariance value during training. Moreover, we also report the results of InfoNCE where re removed for excluding the de-correlation effect. Removing re from the InfoNCE loss leads to high covariance value during the whole training. Removing leads to a significant performance drop, which echos with the finding in (Bardes et al. (021] that dimensional de-correlation is essential for competitive performance. Regarding how Te InfoNCE achieves de-correlation, formally, we hypothesize that the de-correlation effect in InfoNCE arises from the biased weights (X,) on negative samples. This hypothesis is corroborated by the tempera- ture analysis in Fig. 7 We find that a higher temperature makes the weight distribution of X; more balanced indicated higher entropy of A; which echos with the finding in (Wang & Liu 2021). Moreover, we observe that higher temperature also tends to increase the covariance value. Over- all, with temperature as the control variable, we find that more balanced weights among negative samples decrease the de-correlation effect, which constitutes an evidence for our hypothesis.
also
Unifying SimSiam and InfoNCE. At first sight, there is no conceptual similarity between SimSiam and InfoNCE, and this is why the community is intrigued by the success of SimSiam without nega- tive samples. Through decomposing the Ge into oe and re, we find that for both, their o, plays the role of de-centering and their r. behaves like de-correlation. In this sense, we bring two seemingly irrelevant frameworks into a unified perspective with disentangled de-centering and de-correlation.
Beyond SimSiam and InfoNCE. In SSL, there is trend of performing explicit manipulation of de-centering and de-correlation, for which W-MSE (Ermolov et al. 2021), Barlow-twins (Zbontar et al. 1 mance comparable to those with InfoNCE or SimSiam. Towards a unified understanding of recent progress in SSL, our work is most similar to concurrent work (Bardes et al., 2021). Their work is mainly inspired by Barlow-twing (Zbontar et al. (202I) but decomposes its loss into three explicit components. By contrast, our work is motivated to answer the question of how SimSiam prevents
DINO (Caron et al. 2021) are three representative works. They often achieve perfor-
8

---

Published as a conference paper at ICLR 2022
50
1.5
Entropy
8 40
30
T.0.4
T=0.1
Coverianc
1.0
20
0.5
10
1=1.0
100 Epoch
150
200
Temperature
Epoch
(a)
(h)
(c)
Figure 7: Influence of temperature. (a) Entropy of X with regard to temperature; (b) Top- accuracy trend with various temperature; (c) Covariance trend with various temperature.
collapse without negative samples. Their work claims that variance component (equivalent to de- centering) an indispensable component for preventing collapse, while we find that de-correlation itself alleviates collapse. Overall, our work helps understand various frameworks in SSL from an unified perspective, which also inspires an investigation of inter-anchor hurdness-awarenee Zhang et al. (2022) for further bridging the gap between CL and non-CI CL frameworks in SSL.
5 TOWARDS SIMPLIFYING THE PREDICTOR IN SIMSIAM
Based understanding of how SimSiam prevents collapse, we demonstrate that simple com- ponents (instead a non-linear MI.P in SimSiam) in the predictor are sufficient for preventing collapse. For example, to achieve dimensional de-correlation, single FC layer might be sufficient because a single FC layer can realize the interaction among various dimensions. On the other hand, achieve de-centering, single bias layer might be sufficient because a bias vector can represent the center vector. Attaching an Ig-normalization layer at the end of the encoder, i.e. before the predictor, found to critical for achieving the above goal.
Pridictor with FC layers. To learn the dimensional correlation, an FC layer sufficient theoretically but can be difficult train in practice. Inspired by the property that Multiple FC layers make the training more stable even though they can be mathematically equiva- lent to a single FC layer (BilleKligler et al. 2019), we adopt two consecutive FC layers which equivalent to removing the BN and ReLU in the original predictor.
Method SimSiam
Predictor Mon-linear MLP
Top-1 I (%)
66.9
Two FC
FC+FC+Bias
66.7
One FC
Tanh(FC)
64.82
One bias
Bias
49.82
Table 5: Linear evaluation on CIFAR
The training can be made more stable if a Tanh layer is applied the adopted single FC after every iteration. T that they achieve performance comparable to that with non-linear MLP.
Predictor with a bias layer. A predictor with a single bias layer can be utilized for preventing collapse (see Table 5) and the trained bias vector found to have a cosine similarity of 0.99 with the center vector (see Table 6). A bias the MLP predictor also has a high cosine similarity of 0.89, suggesting that it is not co- incidence. theoretical derivation for justifying such high similarity well as how this single bias layer prevents collapse discussed in Appendix A.8
Bias Similarity
(l) single bias
(2) bias in MLP
0.99
0.89
Table 6: Similarity between center vector and (1) single bias layer (bp), (2) the last bias layer of MLP in the predictor.
CONCLUSION
We point out hidden flaw in prior works for explaining the success of SimSiam and propose to decompose the representation vector and analyze the decomposed components of extra gradient. We find that its center vector gradient helps prevent collapse via the de-centering effect and its residual gradient achieves de-correlation which also alleviates collapse. Our further analysis reveals that InfoNCE achieve the two effects in similar manner, which bridges the gap between SimSiam and InfoNCE and contributes to unified understanding of recent progress in SSL. Towards simplifying the predictor have also found that single bias layer is sufficient for preventing collapse.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was partly supported by Institute for Information & communications Technology Plan- ning & Evaluation (ITTP) grant funded by the Korea government (MSIT) under grant No.2019-0- 01396 (Development of framework for analyzing, detecting, mitigating of bias Al model and training data), No.2021-0-0138 (Development of Causal Al through Video Understanding and Re- inforcement Learning, and Its Applications to Real Environments) and 10-12200-0008 (Artificial Intelligence Innovation Hub). During the rebuttal, multiple anonymous reviewers provide valuable advice to significantly improve the quality of this work. Thank you all.
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.009/0 2019.
Adrien Bardes, Ponce, and Yann LeCun. Vicreg: Varinne-ivvrrane--ovviice reg.aaaaiiiiiiiggglagiiiiiiooooi for self-super learning. arXiv preprint arXiv:2 04906, 2021.
Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan NeurIPS, 2019.
Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 1993.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0900 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2/04. /4294, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint trXiv:2003.04297 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.
Victor G. Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. Solo-learn: A library of self-supervised methods for visual representation learning, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume I (Long and Short Pupers), 2019.
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross- covariance image transformers. arXiv preprint arXiv:2106.0968) 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In ICMI. PMLR, 2021.
Abe Fetterman and Josh Albrecht. Understanding self-supervised and contrastive learning with
"bootstrap your own latent" (byol), 2020.
pttp:://untiiiedaiiiiiiiit. /
iuddesstanddinn-sse-----ovvvvvss
Cortrrastive-learning
10

---

Published as a conference paper at ICLR 2022
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernaroo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to supervieer learning. Advances in Neural Information Processing Systems, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 13362, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arXiv preprint arXiv:2106.097855 2021.
Ping Nie. Yuyu Zhang, Xiubo Geng. Arun Ramamurthy, Le Song, and Daxin Jiang. De-bert: De- coupling question and document for efficient contextual encoding. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXiv:1807.03745. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl blog, 2019.
Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:201O. 10244 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS, 2016.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. (VL.)-(bert): Pre-training of generic visual-linguistic representations. In ICLR, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 1eXivv/906.088 2019.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. arXiv preprint arXiv:2102.06810, 2021.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. ICML, 2021.
Chaoning Zhang, Kang Zhang, Trung X. Pham, Changdong Yoo, and In-So Kweon. Towards un- derstanding and simplifying moco: Dual temperature helps contrastive learning without many negative samples. In CVPR, 2022.
11

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 EXPERIMENTAL SETTINGS
Self-supervised encoder training: Below are the settings for self-supervised encoder training. For simplicity, we mainly use the default settings in a popular open library termed solo-learn (da Costa et al. 202T
Data augmentation and normalization: We use series of transformations including Ran- domResizedCrop with scale [0.2, 1.0), bicubic interpolation. ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4). hue (0.1)) is randomly applied with the probability of 0.8. Random gray scale RandonGrayscale is applied with p = 0.2 Horizontal flip is applied with 0.5 The images normalized with the mean (0.4914, 0.4822, 0.4465) and Std (0.247, 0.243, 0.261).
Network architecture and initialization: The backbone architecture is ResNet- 18. The projection head contains three fully-connected (FC) layers followed by Batch Norm (BN) and ReLU, for which ReLU in the final FC layer is removed, i.e. FC1+ BN + ReLU FC2 + BN + + BN. All projection FC layers have 2048 neurons for input, output as well as the hidden dimensions. The predictor head includes two FC layers as follows: FC1 + BN + ReLU + FC2. Input and output of the predictor both have the dimension of 2048, while the hidden dimension is 512. All layers of the network are by default initialized in Pytorch.
Optimizer: SGD optimizer is used for the encoder training. The batch size M is 256 and the learning rate is linearly scaled by the formula Ir M/256 with the base learning rate ir set to 0.5. The schedule for learning rate adopts the cosine decay as SimSiam. Momentum 0.9 and weight decay 1.0 10-5 are used for SGD, We use one GPU for each pre-training experiment. Following the practice of SimSiam, the learning rate of the predictor is fixed during the training. We use warmup training for the first epochs. If not specified, by default we train the model for 1000 epochs.
Online linear evaluation: For the online linear revaluation, we also follow the practice in the solo-learn library (da Costa et al.] [00T]. The frozen features (2048 dimensions) from the training set are extracted (from the self-supervieed pre-trained model) to feed into linear classifier (1 FC layer with the input 2048 and output of 100). The test is performed on the validation set. The learning rate for the linear classifier is 0.1. Overall, we report Top-1 accuracy with the online linear evaluation in this work.
A.2 TWO SUB-PROBLEMS IN AO OF SIMSIAM
In the sub-problem 1f + arg min, L(0', n), nf indicating latent representation of images at step t actually obtained through v/a < ET Fgr (T(x)) where they in practice ignore [r[[] and sample only one augmentation T', iee nha + Fo (T'())) Conceptually, Chen & He equate the role of predictor to EOA.
A.3 EXPERIMENTAL DETAILS FOR EXPLICIT EOA IN TABLE
In the Moving average experiment, we follow the setting in SimSiam (Chen & He 2021 without predictor. In the Same batch experiment, multiple augmentations, 10 augmentations for instance, are applied on the same image. With multi augmentations, we get the corresponding encoded rep- resentation, iee. 2,, i E 10). We minimize the cosine distance between the first representation and the average of the remaining vectors, i.e. 1/ Iee 2, The gradient stop is put the aver- aged vector. We also experimented with letting the gradient backward through more augmentations, however, they consistently led to collapse.
12

---

Published as a conference paper at ICLR 2022
A.4 EXPERIMENTAL SETUP AND RESULT TREND FOR TABLE
Mirror SimSiam. Here we provide the pseudocode for Mirror SimSiam. In the Mirror SimSiam ex- periment which relates to Fig. (c). Without taking symmetric loss into account, the pseudocode shown in Algorithm Taking symmetric loss into account, the pseudocode is shown in Algorithm 2
Algorithm Pytorch-liie like Pseudocode: Mirror SimSiam
encoder (backbone projector)
# h: predictor
for x in loader: + load a minibatch x with samples x_a, x_h aug (x), aug it augmentation
2_8, z_b [(x_a), [x]bb projections
P_b (Vz_b.detach()) detach z_b still allowing gradient P_b
D_cosine P_b) $ loss
I. b.backward () back-propagate
uf,ate SGD update
def D_cesine (z, dim-i) 12-normalize
negative cosine similarity
normalize (2,
vuzmalize (p, dimml) 1) 22-normalize
return (22p)) (din-1) .mean.))
Algorithm Pytorch-like Pseudocode: Mirror SimSiam
E: encuder (backbone prejectuz)
4 hi predicter
for x in loader: + load a minibatch x with aamples X_a, x_b aug (x), aug (x) augmentation
2_a, z_b (X_a), f [X_b) # profections
P.b hta_b.detach (1) detach b_b but still allowing gradient pub
P_a h(z_a.detach() (1) datach z_a but still all allowing gradient PLa
D_cosine p_ [_ p2a)/2 & loss
L.backwerd back-propagate updace h) SGD update
def D_cosine (2, pl: negative cosine similarity noromlliee dim-1) 12-normalia normal (P, dim=l) 12-norma ze return ((zpp) sum (dim=1) .mean
Symmetric Predictor.
To implement the SimSiam with Symmetric Predictor as in Fig. 2 (b), we can just perceive the predictor part of the new encoder, for which the pseudocode is provided in Algorithm 3 Alternatively, we can additionally train the predictor similarly as that in SimSiam, for which the training involves two losses, one for training the predictor and another for training the new encoder (the corresponding pseudocode is provided in Algorithm 4 Moreover, for the second implementation, the new encoder and then train the predictor alternatingly. All of them lead to collapse with a similar trend as long as the symmetric predictor used for training the encoder. For avoiding redundancy, in Fig. we only report the result of the second implementation.
also experiment with another variant that fixes the predictor while optimizing
Result trend. The result trend of SimSiam, Naive Siamese, Mirror SimSiam, Symmetric Predictor are shown in Fig. X We observe that all architectures lead to collapse except for SimSiam. Mirroe SimSiam was stopped in the middle because a NaN value was returned from the loss.
A.5 EXPERIMENTAL DETAILS FOR INVERSE PREDICTOR.
In the inverse predictor experiment which relates to Fig. 2 (c), introduce predictor which has the same structure as that the original predictor. The training loss consists 3 parts: predictor training loss, inverse predictor training and new encoder (old encoder4predictor) training. The new
13

---

Published as a conference paper at ICLR 2022
Algorithm 3 Pytorch-like Pseudocode: Symmetric Predictor
f: encoder (backbone projecsor)
predicter
for in loador: t load minibatch with samples
X_å, x_b aug (x), aug (x) augmentation
E_a, s_b [X_a),, (X projections
PLä, R_b h(z_a), (2]b) predictions
D P_ Dip_b, P_a) loas
L.backwerd () back-propagate updacalf, SGD update
Dip, 2): negative cosine similarity - 2.detach () 4 stop gradient normalize dim=1) 12-normalize normalize (2, dim-1) 12-normalize return (p=z) aum (dim-1) (dim-1).mman .mean
Algorithm Pytorch-like Pseudocode: Symmetric Predictor (with additional training on predictor)
f: encoder (backbone projector)
f h: predicter
for x in loader: t load a minibatch x with sanples
X_8, x_b sug (x), aug augmentation
2_a, 2_b [(x_a), [( profections
P_b h(2_a), (2_b) predictions
d_p_a, d_P_b (11 d(t_b.)etach))) $ detached predictor output
predicter training loss
brreed = (d_p_a, /_ + b(d___, /2
encuder training lass (ppaa, /22_b//2 bip_b, d_2_a)/2
pred enee
b.backward () back-propagate updateli,
update
def ((p, negative cosine simi larity wi detach
2z.detach( stop gradient
normalise (p, dim-1) 12-normalize
nuzmalize (2, dim-1) 12-normalize
return - (pp=))..... summsummiee (dim=l) .me.
60
50
Top-1 (%)
SimSiam
40
Symmetric Predictor
30
Mirror SimSiam
Naive Siamese
20
10
200
400
800
1000
Epoch
Figure 8: Result trend of Naive Siamese, Mirror SimSiam. Symmetric Predictor.
encoder F consists of the old encoder predictor The practice of gradient stop needs to be considered in the implementation. We provide the pseudocode in Algorithm 5
14

---

Published as a conference paper at ICLR 2022
Algorithm 5 Pytorch-like Pseudocode: Trainable Inverse Predictor
fi encader (backbone projector) predicter
inv: inverse predictor
for x in loader: t load a minibatch x with samples
X_a, x_b aug aug 4 augmentation
R_ä, z_b f(X_a), (( profections
PLä, p_b h(2_a), b(2b) predictions
d_P_a, d_P_b h1s (1)_a, h(x_b.dstach()) f detached predictor output predicter training loss lupred [d_a,_, 7_b) D(d_p_,, z_a) train
_nv_p_a, inv_p_b hLinw(p a ()) hp b.detcct train h_vnv lnverse predictor training loss
i_iav_pred Diinv_ z_a)/2 (inv__p_,, x_b1/2
encoder training loss [aenc (PLa, h_inw(p_b)) D(p_,, h_inv(p_a))
pred pred L_ene
I.bbckwarr back-propagste update hy i_inv) SGD update
def (ip, z): negative cosine aimilarity with detach on z
2.detach stop gradient normalize (p, dim-1) 12-normalize
= norma 1 zi (7, dim=l) ) 122nnormall lize
return -(p=)) sum (din=1) .mean
A.6 REGULARIZATION LOSS
Following Zbontar al. (2021) compute covariance regularization loss of encoder output along
the mini-batch. The pseudocode for de-correlation loss calculation is put in Algorithm
Algorithm Pytorch-like Pscudocode: De-correlation loss
2 representation vector
4 N: basch size the number dimension representation vectur
z_a (dim-0)
cov 2_a.T a_a (N-1) diaq torch.eve eye (D)
cuv [ "diig.bboll buol (11 ppo__ .cm
A.7 GRADIENT DERIVATION AND TEMPERATURE ANALYSIS FOR INFONCE
With indicating the cosine similarity between vectors, the InfoNCE loss be expressed
exp( Z. Zs/T)
LInfoNCE log
exp Z.. Zn/T) Din exp( Z,/T)
(5)
exp( Z. Zt/T)
= -olo
Li=o exp(2x Za. Zi/T)*
where N indicates the number of negative samples and Zo Zn for simplifying the notation. By
treating Z, Z; as the logit normal CE loss, we have the corresponding probability for each exp(2, 2./T)
negative sample as X; Ex expl -./T)' where i = 0, 1, 2, ..., N and we have Σi X;
15

---

Published as a conference paper at ICLR 2022
The negative gradient of the InfoNCE the representation Za is shown as
@LInfoNCE 8Za
1
Ao) (1-0x)
1
Σ
i=1
N Σ X.Zi))
-2
1=0
N
= (2
1
Σ
Ae +r)))
(6)
i=0
N
(2) + -
Xprr)
i=0
N
x +
Σ Arti)
where + can be adjusted through learning rate and omitted for simple discussion. With Z3 as the basic gradient, Ge -0 Ei= ArT for which o, -0 and Te
- EizoAor
When the temperature is set to a large value, X; expl Z:/r)
- indicated Σ Z. Z,/T)' approaches N+I.
simple contrastive loss, i.e. by high entropy value (see Fig. 7.. InfoNCE will degenerate
Caimple = - Zo + N Σi= Za Z; which repulses every negative sample with an equal force. In contrast, relative smaller temperature will give more relative weight, i.e. larger X, to negative samples that are more similar to the anchor (Z.).
The influence of temperature on covariance and accuracy is shown in Fig. 7 (b) and (c). We observe that higher temperature tends to decrease the effect of de-correlation, indicated by a higher covariance value, which also leads to a performance drop. This verifies our hypothesis regarding how re in InfoNCE achieves de-correlation because large temperature causes more balanced weights A; which is found to alleviate the effect de-correlation. For the setup, we note that the encoder is trained for 200 epochs with the default setting in Solo-learn for the SimCLR framework.
A.8 THEORETICAL DERIVATION FOR SINGLE BIAS LAYER
With the cosine similarity loss defined as Eq 7 Eq g
cossim(a, b)
(7)
Va²
for which the derived gradient on the vector is shown
i Da
cossim (a,
b
cossim(a,
01 Ja
(8)
la|. [b]
The above equation is used as prior for our following derivations. As indicated in the main manuscript, the encoder output 2, is lz-normalized before feeding into the predictor, thus Pa Z, + bp. bp denotes the bias layer in the predictor. The cosine similarity loss (ignoring the symme- try for simplicity) is shown
Leosine - P. Zb
Pa 2b
(9)
Pa.l
1 |
16

---

Published as a conference paper at ICLR 2022
The gradient Pa is derived
Bcosine
2b
cossimm Za. Zz)
Pa |
Pp
Pal
| cossim Zo: ZT) P.
Pa ||
Zu + by
Zo cossim/ Zoo Zτ)
||p.|l
|paal
(10)
(o3 + Tb) cossim Za Zb)
(0z + bp)
[pa/|
Pa.l
((o: + rb) - mm m-(o= + raa bp))
||aal
((1 m) mb, + Tb ra),
Pa
where coasimi 2s)
||Pull
Given that Pa Za + bp. the negative gradient on by is the same as that on Pa as
OCoosine
HCessine
Öbp
Öpa
(11)
1 Pa
((L ((1-m) 10mm, -mb + Tb m .r.a....
We assume that the training is stable and the bias layer converges certain value when Ocossim Zs) 0. Thus, the converged by satisfies the following constraint:
aby
PPal ((1 m)o, mbb + rb -mr))) =0
(12)
1 - m
1 m
bp
Tb Ta-
With batch of samples, the average of and ra is expected to be close to 0 by the definition of residual vector. Thus, the bias layer vector is expected converge to:
-
O
(13)
17
Rational behind the high similarity between b, and O,, The above theoretical derivation shows that the parameters in the bias layer are excepted to converge to vector 1-m o,. This theoretical derivation justifies why the empirically observed cosine similarity between b, and o, is as high as 0.99. Ideally, it should 1, however, such a small deviation is expected with the training dynamics taken into account.
Rational behind how a single bias layer prevents collapse. Given that Pa = Zo + bp. the negative gradient Z, is shown as
OLcosine
dLcosine
aZ.
Op.
Za + bp |aall
Z, cossimm Zn, Zb)
(14)
|aall
cossim Za, Zz)
| Z cossim Za. Zo. bp.
Zi
Paal
Paa 1
Here, we highlight that since the loss Z. 2, - is a constant having zero gradients on the en-
coder, cossim Z, 2s) Z be seen dummy term. Considering Eq 1 and cossim Z, Z.) [..
D2
17

---

Published as a conference paper at ICLR 2022
lIp have b cossimm Ze 2z,
1)0g.
The above equation is equivalent to
① Leassine
1 Zb
cossimb Za. Zb,
bp
aZa
Pa
||Pal2
1
cossim Za,
Zb
Pall
Zb
1)0₂
|pa
Pa Cossim
Za,
Zo.
(15)
1
cossim Zai Zbb)
Za
)02
Pa
P|Pall
PPal
cossim
Zo. Zp)
oc Z, - (1
||Pal 00.
With Z as the basic gradient, the extra gradient component Ge = cossim(2.. 2z) )o.. Given I|Pu that Zu + b, and Z., = 1, thus |P: < 1 only when Zu is negatively correlated with by. In practice, however, Zo and by are often positively correlated some extent due to their shared center vector component. In other words, ||Pal| > 1. Moreover, cossim| Za, Zτ) is smaller than 1, thus suggesting G. consists of negative o with the effect of de-centerization. This above derivation justifies the rationale why a single bias layer can help alleviate collapse.
-(1 cossim| Z., Zu,)
B DISCUSSION: DOES BN HELP AVOID COLLAPSE?
MSE
covariance
0,00100
(90) 1oo 1
40
3.6
E
0.00050
0.4
0.4
0.00025
20
0.2
02
0.0
50
100
150
200
0.0
50
100
150
200
50
150
200
50
100 Epoch
130
200
Epoch
Epoch
Epoch
Figure 9: BN with MSE helps prevent collapse without predictor or stop gradient. Its performance, however, is inferior the cosine loss-based SimSiam (with predictor and stop gradient).
To our knowledge, our work is the first to revisit and refute the explanatory claims (Chen & He Several works, however, have attempted demystify the success of BYOL (Grill et al, 2020 a close variant of SimSiam. The success has been ascribed to BN in (Fetterman & Albrecht however, (Richemond al. (2020) refutes their claim. Since the role of intermediate BNs is ascribed to stabilize training (Richemond et al. 2020: Chen & He 2021) we only discuss the final BN in the SimSiam encoder. Note that with our Conjecture the final BN that removes the mean of representation vector is supposed to have de-centering effect. BY default SimSiam has such BN at the end of its encoder, however, it still collapses with the predictor and stop gradient. Why would such BN not prevent collapse in this case? Interestingly, alleviate collapse with a simple MSE loss (see Fig. 9. however, its performance is is inferior to the cosine bass- SimSiam (with predictor and stop gradient) due the lack of the de-correlation effect in SimSiam. Note that the cosine loss is essence equivalent to a MSE loss on the l2- normalized vectors. This phenomenon can be interpreted that the 2 normalization causes another mean after the BN removes it. Thus, with such Ig-normalization in the MSE loss, i.e. adopting the default cosine loss, it is important remove the o, from the optimization target. The results with the loss of - Z. sg( Zi + oe) in Table show that this indeed prevents collapse and verifies the above interpretation.
2021
2020
observe that such BN can help
18


Reviews:
[{"review_id": "XXayT7d0c05", "reviewer": "Reviewer_d9ds", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ It's interesting to see a framework to unified understand SSLs such as SimSiam, MoCo, SimCLR, etc.\n+ The hidden flaw in AO of SimSiam seems to be correct, which is interesting.\n--------------------\n- The paper lacks experimental results/details to demonstrate their statements. \n\na. In subsec. \"Symmetric Predictor does not prevent collapse\", the authors state \"The results in Fig. 3 (b) show that it still leads to collapse\" which is related to symmetric predictors in SimSiam. However, Fig. 3 (b) is actually about the basic SimSiam and SimSiam + Inverse predictor. \n\nb. in subsec. \"Predictor with stop gradient is asymmetric\", the authors \"the SimSiam avoids collapse by excluding Mirror SimSiam (Fig1 (c)) which has a loss (mirror-like Eq 2) as shown as eq. 2\". There is no experimental evidences to show if the mirror SimSiam will lead to collapse. If experimentally mirror SimSiam works, the statement does not hold.\n\nc. How did you design the inverse predictor? Can we just see it as the new predictor while the previous predictor is included in the projector part?\n\nd. in subsec. \"Predictor vs. inverse predictor\", \"we interpret Fig2 (b) differs from Fig1 (a) as changing the optimized target from p to z, i.e. h\n−1(p), suggesting processingthe optimized target with h−1 helps prevent collapse.\" Again, no experimental evidence.\n\ne. in subsec. \"Trainable h−1 and its implication on EOA\", \"we optimize h−1 by optimizing the pa approaching z∗b while simultaneously optimizing z∗b to zb via cosine loss, where z∗b is the h−1 output. The results proves that the model with h−1 (Fig2 (c)) is equivalent to\nSimSiam since it achieves comparable performance as the original SimSiam that directly optimizes pa approaching zb.\". Where are the \"results\"?\n\n- The paper might have flaws in their proposed math and sometimes it's hard to follow 3.1, 3.3, and 3.4 \n\nIn subsec \"Competition between o and r.\", why \"mo and mr is expected to be opposite of each other\". The denominators of the two terms are both ||z||. So only ||o_z|| and ||r_a|| define the values. However, r_a = Z_a − o_z. Thus r_a and o_z can have the same norms but with different directions? Why they need to be on the opposite directions?\n\n\n", "summary_of_the_paper": "This paper proposes a framework to understand why SimSiam avoid collapse without negative samples? It provides a hidden flaw of the Alternating Optimization for explain why SimSiam works. And the authors claim that  the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect.", "summary_of_the_review": "I think the paper needs to improve descriptions about their framework and provide more experimental evidences.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TiIht-bNskA", "reviewer": "Reviewer_S8KB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is generally well-written and well-structured. It is important and inspiring to explore the reason why SimSiam can avoid collapse. The authors provide a convincing explanation and reach a unified conclusion for the recent progress in SSL, which is very insightful. The theoretical analysis and experimental results are solid. ", "summary_of_the_paper": "This method aims to explore why a minimalist simple Siamese (SimSiam) method can avoid collapse. After refuting their claims, the authors introduce vector decomposition for analyzing the collapse based on the gradient analysis of l2 normalized vector, yielding a unified perspective on how negative samples and SimSiam predictor alleviate collapse.", "summary_of_the_review": "Based on the abovementioned strength of this paper and the interesting conclusion, I recommend to accept this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OZaeDWpAOZ0", "reviewer": "Reviewer_q4XL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The paper investigates the effects of center vectors and residual vectors in detail for both SimSiam and Info-NCE, which provides a unified explanation.\n\n2. The results of SimSiam++ which shows a simple bias as a predictor can also avoid collapse without negative samples are interesting.\n\n\nCons:\n 0. The writing of the paper needs to be improved. Many concepts are not explained or defined very clearly.\n\n 1. The original SimSiam paper only claimed that \"The usage of h may fill this gap (of missing EOA).\" I think it's clear that the predictor does not learn to approximate the EOA. So I don't think the paper's interpretation of SimSiam's explanation is correct.\n\n 2. In section 2.2, the paper claims that explicit EOA does not prevent collapse. But the experimental details are not explained very clearly here. I'm wondering whether the paper still uses one or two augmentations as the predictor's outputs or all the augmentations are used without stop-gradient.\n\n 3. In section 2.3, the paper mentions \"The results in Fig. 3(b)\" show that it still leads to collapse. But I cannot find the collapsed results in Fig. 3(b). Besides, to prove Mirror SimSiam does not work (Fig 1. (c)), the authors should not apply stop-gradient to the predictor, because it's clear in the original SimSiam paper that fixed init does not work for the predictor. One possible way is to apply the gradient on z_a and p_b, and apply stop-gradient on z_b.\n\n 4. In section 3.1, the paper mentions \"Note that Z is l2-normalized, thus the trend of mo and mr is expected to be opposite of each other.\" This does not always hold.\n\n\nPossible typos:\n\nSection 2.3: Fig1 (c) and Fig2 (a) both lead to success ==> Fig1 (c) and Fig2 (a) both lead to failure\n\nSection 3.1: loss - Z_a * sg(o_z) and loss - Z_a * sg(Z_a - o_z) ==> loss - Z_a \\cdot sg(o_z) and loss - Z_a \\cdot sg(Z_a - o_z)\nFigure 3: - Z_a \\cdot sg(Z_b - E(Z_b) ==> - Z_a \\cdot sg(Z_b - E(Z_b))\n\nSection 3.2: Z_n and r_n are used without definitions, which I guess means the representation for the negative examples.\n", "summary_of_the_paper": "The paper proposes another explanation for why SimSiam can avoid collapse without negative samples. Specifically, the paper decomposes the gradient of learned representation as center vector and residual vector and finds that the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect. Such an explanation can also be applied to Info-NCE, which unifies the theory of self-supervised learning with and without negative samples.", "summary_of_the_review": "Overall, I think the idea of analyzing the extra gradient component is novel. The writing quality of the paper is not very good.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2aKUtN3koko", "reviewer": "Reviewer_TeJe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The topic of the research is of significance, as it is important to understand why designs like SimSiam/BYOL does not collapse. The paper attempts to get more insights of it with empirical evidence, and also shows the potential of connecting to contrastive methods (via InfoNCE loss). This is a great step toward better understanding of existing methods.\n+ I like the style of providing conjectures and show results that empirically verifying them. The experimental designs are quite solid and insightful.\n\n- Paper writing: I am a bit lost when reading the second part of the paper about the center vector and residual ones. I think it is good to have a clear definition for X in \"centering w.r.t. X\". Is X the entire image set and all possible augmentations? Is X just for the current image? Or is X for the current batch? Also I think this center vector must be approximate (e.g., taking multiple crops to get the average), not fully exact. So it would be good to have some notational clarifications on this. Particularly when introducing the analysis on negative examples and predictor, the notation becomes quite messy and hard to follow.\n- About Fig 1 (c): if the stop gradient is applied this way, then the predictor is NOT going to be trained at all and would naturally leads to collapse to me. I think the better way is to have the stop gradient between the predictor and the encoder (so at the place of z_b). At least this way all the parameters in the framework are being trained. I do think even this placement will lead to collapse.\n- For the part that revisits SimSiam's explanation: while it is great to have this part, to me it is not the essential part of the paper (the second part is). So I think it would be good to make the second part more self-contained, with richer set of experiments, and move the SimSiam explanation revisit part to appendix.\n- Minor for the text that leads to Eq (2): I think symmetric loss is not the essential reason why SimSiam does not collapse, so it would be good to just focus on the asymmetric architecture (like SimSiam's teaser figure) and loss for the current analysis. It would also reduce the confusion.\n- Experimentation wise, it is good that the current paper provides a good explanation, but performance wise the theoretical insights have not led to better results. It is less important for a paper like this, but calling the last section \"SimSiam++\" but without numbers/speeds that actually outperform SimSiam is strange to me. ", "summary_of_the_paper": "As the title suggests, the paper does a detailed investigation of how SimSiam avoids collapse without negative training examples. The key idea is to decompose the original vector into a center vector component and a residual vector component. The center vector cannot be too large (otherwise it is indicating collapse). The high-level idea is that the designs in SimSiam (and contrastive frameworks that have InfoNCE loss)  are mainly to prevent the center vector from getting too large. There are conjectures (verified empirically) about the relationship between the gradient w.r.t. the center vector ands the gradient w.r.t. the residual vector, and with these the paper finds that for SimSiam, the predictor is important for preventing collapse, particularly by doing de-correlation among features.", "summary_of_the_review": "Overall I think this is a paper worthy of publication, with 1) its additional experiments to find flaws in the original hypothesis of SimSiam; and 2) explanations by decomposing the vector into two parts. However, I do have some concerns about the writing of the paper, so it would be good if the paper could have a higher quality (with proof-reading) and better organization so that it presents a more clear, unified message.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-miO4WCobZv", "reviewer": "Reviewer_D1d5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths:\n+ Recent successful approaches to SSL indicate that stable learning can be achieved without explicit incorporation of repulsive, negative image relations. Indeed, the underlying reasons are theoretically poorly understood and typically only an inuitive understanding presented. Hence, given the importance of contrastive learnig today, research in this direction is important.\n+ The analysis in section 2 seems sound and raises questions about the original insights of the SimSiam work regarding representation collapse. \n\n#### Weaknesses:\n- The presentation and outline of arguments, as well as empirical discussions especially in section 3.3, 3.4 and 3.5 are sometimes hard to follow.\n- The need of ‘de-centeralization’ (i.e. pushing samples apart in the embedding space) in gradient signals of SSL for stable learning seems straight-forward and inuitively trivial and does not seem novel.\n- The paper only empirically shows that some kind of repulsion (i.e. de-centralization) is implicitly happening in the SimSiam framework. However, no theoretical and clear explanation of why and how is presented.  The presented conjectures are insufficiently back-uped and proven beyond the intuition of implicit repulsion. The paper e.g. states “Since the original predictor involves a nonlinear MLP, it is hard to understand how the predictor actually achieves it” (Sec. 4). Thus, no real, novel insights about the learning mechanisms of SSL without negatives are provided which are required for a more complete understanding of the addressed problem.\n- The conceptual explanation of de-correlation in Info-NCE is hard is fuzzy and does not sound convincing. A more solid derivation and formulation of the hypothesis is needed.  \n\n#### Questions:\n- In the paragraph ‘Predictor with stop-gradient is asymmetric’ the paper states that the model setup shown in Fig. 2 (a) acutally results in successful, stable learning (“Similarly, […] Fig2 (a) […] leads to success, [...]”) It seems no prediction head is used in this case. Does the stop-gradient operation itself already prevent representation collapse?", "summary_of_the_paper": "The paper analyzes how the self-supervised learning (SSL) approach SimSiam avoids collapsed representations without explicit formulation of repulsive sample relations. To this end, first flaws in the original reasoning of the SimSiam paper are revealed. Next, based on center-residual vector decomposition, the role of the prediction head for preventing representation collapse in SimSiam is analyzed. Results indicate the importance of de-centralization and de-correlation as driving concepts for stable SSL.", "summary_of_the_review": "While the addressed phenomenon of preventing collapsing representations without the usage of negative samples is interesting, the presented analyis provides insufficient novel insights about how collapse is prevented. Although some novel framework based on vector decomposition is presented, the reasoning is often fuzzy, not convincing and lacks backup by clear theoretical derivations. Thus, I vote to reject this submission in its present form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provide an explanation why contrastive learning methods like SimSiam avoid collapse without negative samples. As the authors claimed, this is indeed a timely work for understanding the recent success in self-supervised learning (SSL). The key idea in this submission is to decomposes the gradient into a center vector and residual vector which respectively correspond to de-centering and de-correlation. Such an explanation is interesting and novel. The empirical results are solid and convincing. During the rebuttal stage, the concerns from the reviewers are well resolved, and the writing of the new version is significantly better than the original one.
Decision: Accept (Poster)