HOW DOES SIMSIAM AVOID COLLAPSE WITHOUT NEGATIVE SAMPLES? TOWARDS
Year: 2021

Abstract:
To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work (Chen & He, 2021) has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the l2-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.

heading: 1 INTRODUCTION
text: Beyond the success of NLP (Lan et al., 2020; Radford et al., 2019; Devlin et al., 2019; Su et al., 2020; Nie et al., 2020), self-supervised learning (SSL) has also shown its potential in the field of vision tasks (Li et al., 2021; Chen et al., 2021; El-Nouby et al., 2021). Without the ground-truth label, the core of most SSL methods lies in learning an encoder with augmentation-invariant representation (Bachman et al., 2019; He et al., 2020; Chen et al., 2020a; Caron et al., 2020; Grill et al., 2020). Specifically, they often minimize the representation distance between two positive samples, i.e. two augmented views of the same image, based on a Siamese network architecture (Bromley et al., 1993). It is widely known that for such Siamese networks there exists a degenerate solution, i.e. all outputs “collapsing” to an undesired constant (Chen et al., 2020a; Chen & He, 2021). Early works have attributed the collapse to lacking a repulsive component in the optimization goal and adopted negative samples, i.e. views of different samples, to alleviate this problem. Introducing momentum into the target encoder, BYOL shows that Siamese architectures can be trained with only positive pairs. More recently, SimSiam (Chen & He, 2021) has caught great attention by further simplifying BYOL by removing the momentum encoder, which has been seen as a major milestone achievement in SSL for providing a minimalist method for achieving competitive performance. However, the reason for such success remains unclear with a core question summarized as:
How does SimSiam avoid collapse without negative samples?
Our investigation starts with revisiting the explanatory claims in the original SimSiam paper (Chen & He, 2021). Notably, two components, i.e. stop gradient and predictor, are essential for the success of SimSiam (Chen & He, 2021). The reason has been mainly attributed to the stop gradient (Chen & He, 2021) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set. Chen & He argue that the predictor h is helpful in SimSiam because h fills the gap to approximate expectation over augmentations (EOA).
Unfortunately, the above explanatory claims are found to be flawed due to reversing the two paths with and without gradient (see Sec. 2.2). This motivates us to find an alternative explanation, for which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL. Specifically, we propose to decompose a representation vector into center and residual components. This decomposition facilitates understanding which gradient component is beneficial for avoiding
collapse. Under this framework, we show that a basic Siamese architecture cannot prevent collapse, for which an extra gradient component needs to be introduced. With SimSiam interpreted as processing the optimization target with an inverse predictor, the analysis of its extra gradient shows that (a) its center vector helps prevent collapse via the de-centering effect; (b) its residual vector achieves dimensional de-correlation which also alleviates collapse.
Moreover, under the same gradient decomposition, we find that the extra gradient caused by negative samples in InfoNCE (He et al., 2019; Chen et al., 2020b;a; Tian et al., 2019; Khosla et al., 2020) also achieves de-centering and de-correlation in the same manner. It bridges the gap between SimSiam and InfoNCE and contributes to a unified understanding of the recent progress of various frameworks in SSL. Finally, towards simplifying the predictor for more explainable SimSiam, we show that a single bias layer is sufficient enough for preventing collapse.
The basic experimental settings for our analysis are detailed in Appendix A.1 with a more specific setup discussed in the context. Overall, our work is the first attempt for performing a comprehensive study on how SimSiam avoids collapse without negative samples. Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam. A technical report (Fetterman & Albrecht, 2020) has suggested the importance of batch normalization (BN) in BYOL for its success, however, a recent work (Richemond et al., 2020) refutes their claim by showing BYOL works without BN, which is discussed in Appendix B.

heading: 2 REVISITING SIMSIAM AND ITS EXPLANATORY CLAIMS
text: l2-normalized vector and optimization goal. SSL trains an encoder f for learning discriminative representation and we denote such representation as a vector z, i.e. f(x) = z where x is a certain input. For the augmentation-invariant representation, a straightforward goal is to minimize the distance between the representations of two positive samples, i.e. augmented views of the same image, for which mean squared error (MSE) is a default choice. To avoid scale ambiguity, the vectors are often l2-normalized, i.e. Z = z/||z|| (Chen & He, 2021), before calculating the MSE:
LMSE = (Za −Zb)2/2− 1 = −Za ·Zb = Lcosine, (1) which shows the equivalence of a normalized MSE loss to the cosine loss (Grill et al., 2020).
Collapse in SSL and solution of SimSiam. Based on a Siamese architecture, the loss in Eq 1 causes the collapse, i.e. f always outputs a constant regardless of the input variance. We refer to this Siamese architecture with loss Eq 1 as Naive Siamese in the remainder of paper. Contrastive loss with negative samples is a widely used solution (Chen et al., 2020a). Without using negative samples, SimSiam solves the collapse problem via predictor and stop gradient, based on which the encoder is optimized with a symmetric loss:
LSimSiam = −(Pa · sg(Zb) + Pb · sg(Za)), (2) where sg(·) is stop gradient and P is the output of predictor h, i.e. p = h(z) and P = p/||p||.

heading: 2.1 REVISING EXPLANATORY CLAIMS IN SIMSIAM
text: Interpreting stop gradient as AO. Chen & He hypothesize that the stop gradient in Eq 2 is an implementation of Alternating between the Optimization of two sub-problems, which is denoted as AO. Specifically, with the loss considered as L(θ, η) = Ex,T [∥∥Fθ(T (x)) − ηx∥∥2], the optimization objective minθ,η L(θ, η) can be solved by alternating ηt ← arg minη L(θt, η) and θt+1 ← arg minθ L(θ, ηt). It is acknowledged that this hypothesis does not fully explain why the collapse is prevented (Chen & He, 2021). Nonetheless, they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difficult to approach a constant ∀x. Interpreting predictor as EOA. The AO problem (Chen & He, 2021) is formulated independent of predictor h, for which they believe that the usage of predictor h is related to approximating EOA for filling the gap of ignoring ET [·] in a sub-problem of AO. The approximation of ET [·] is summarized in Appendix A.2. Chen & He support their interpretation by proof-of-concept experiments. Specifically, they show that updating ηx with a moving-average ηtx ← m ∗ ηtx + (1 −m) ∗ Fθt(T ′(x)) can help prevent collapse without predictor (see Fig. 1 (b)). Given that the training completely fails when the predictor and moving average are both removed, at first sight, their reasoning seems valid.

heading: 2.2 DOES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA?
text: Reasoning flaw. Considering the stop gradient, we divide the framework into two sub-models with different paths and term them Gradient Path (GP) and Stop Gradient Path (SGP). For SimSiam, only the sub-model with GP includes the predictor (see Fig. 1 (a)). We point out that their reasoning flaw of predictor analysis lies in the reverse of GP and SGP. By default, the moving-average sub-model, as shown in Fig. 1 (b), is on the same side as SGP. Note that Fig. 1 (b) is conceptually similar to Fig. 1 (c) instead of Fig. 1 (a). It is worth mentioning that the Mirror SimSiam in Fig. 1 (c) is what stop gradient in the original SimSiam avoids. Therefore, it is problematic to perceive h as EOA.
New Figure 1
Explicit EOA does not prevent collapse. (Chen & He, 2021) points out that “in practice, it would be unrealistic to actually compute the expectation ET [·]. But it may be possible for a neural network (e.g., the preditor h) to learn to predict the expectation, while the sampling of T is implicitly distributed across multiple epochs.” If implicitly sampling across multiple epochs is beneficial, explicitly sampling sufficient large N augmentations in a batch with the latest model would be more beneficial to approximate ET [·]. However, Table 1 shows that the collapse still occurs and suggests that the equivalence between predictor and EOA does not hold.

heading: 2.3 ASYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN SIMSIAM
text: Symmetric Predictor does not prevent collapse. The difference between Naive Siamese and Simsiam lies in whether the gradient in backward propagation flows through a predictor, however, we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path. With h being trained the same as Eq 2, we optimize the encoder f through replacing the Z in Eq 2 with P . The results in Table. 2 show that it still leads to collapse. Actually, this is well expected by perceiving h to be part of the new encoder F , i.e. p = F (x) = h(f(x)). In other words, the symmetric architectures with and without predictor h both lead to collapse.
Predictor with stop gradient is asymmetric. Clearly, how SimSiam avoids collapse lies in its asymmetric architecture, i.e. one path with h and the other without h. Under this asymmetric architecture, the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target, not vice versa. In other words, the SimSiam avoids collapse by excluding Mirror SimSiam (Fig. 1 (c)) which has a loss (mirror-like Eq 2) asLMirror = −(Pa ·Zb+Pb ·Za), where stop gradient is put on the input of h, i.e. pa = h(sg[za]) and pb = h(sg[zb]).
Predictor vs. inverse predictor. We interpret h as a function mapping from z to p, and introduce a conceptual inverse mapping h−1, i.e. z = h−1(p). Here, as shown in Table 2, SimSiam with symmetric predictor (Fig. 2 (b)) leads to collapse, while SimSiam (Fig. 1 (a)) avoids collapse. With the conceptual h−1, we interpret Fig. 1 (a) the same as Fig. 2 (c) which differs from Fig. 2 (b) via changing the optimization target from pb to zb, i.e. zb = h−1(pb). This interpretation
suggests that the collapse can be avoided by processing the optimization target with h−1. By contrast, Fig. 1 (c) and Fig. 2 (a) both lead to collapse, suggesting that processing the optimization target with h is not beneficial for preventing collapse. Overall, asymmetry alone does not guarantee collapse avoidance, which requires the optimization target to be processed by h−1 not h.
Trainable inverse predictor and its implication on EOA. In the above, we propose a conceptual inverse predictor h−1 in Fig. 2 (c), however, it remains yet unknown whether such an inverse predictor is experimentally trainable. A detailed setup for this investigation is reported in Appendix A.5. The results in Fig. 3 show that a learnable h−1 leads to slightly inferior performance, which is expected because h−1 cannot make the trainable inverse predictor output z∗b completely the same as zb. Note that it would be equivalent to SimSiam if z∗b = zb. Despite a slight performance drop, the results confirm that h−1 is trainable. The fact that h−1 is trainable provides additional evidence that the role h plays in SimSiam is not EOA
because theoretically h−1 cannot restore a random augmentation T ′ from an expectation p, where p = h(z) = ET [ Fθt(T (x)) ] .

heading: 3 VECTOR DECOMPOSITION FOR UNDERSTANDING COLLAPSE
text: By default, InfoNCE (Chen et al., 2020a) and SimSiam (Chen & He, 2021) both adopt l2normalization in their loss for avoiding scale ambiguity. We treat the l2-normalized vector, i.e. Z, as the encoder output, which significantly simplifies gradient derivation and the following analysis.
Vector decomposition. For the purpose of analysis, we propose to decompose Z into two parts, Z = o + r, where o, r denote center vector and residual vector respectively. Specifically, the center vector o is defined as an average of Z over the whole representation space oz = E[Z]. However, we approximate it with all vectors in current mini-batch, i.e. oz = 1M ∑M m=1 Zm, where M is the mini-batch size. We define the residual vector r as the residual part of Z, i.e. r = Z − oz .

heading: 3.1 COLLAPSE FROM THE VECTOR PERSPECTIVE
text: Collapse: from result to cause. A Naive Siamese is well expected to collapse since the loss is designed to minimize the distance between positive samples, for which a constant constitutes an optimal solution to minimize such loss. When the collapse occurs, ∀i,Zi = 1M ∑M m=1 Zm = oz , where i denotes a random sample index, which shows the constant vector is oz in this case. This interpretation only suggests a possibility that a dominant o can be one of the viable solutions, while the optimization, such as SimSiam, might still lead to a non-collapse solution. This merely describes o as the consequence of the collapse, and our work investigates the cause of such collapse through analyzing the influence of individual gradient components, i.e. o and r during training.
Competition between o and r. Complementary to the Standard Deviation (Std) (Chen & He, 2021), for indicating collapse, we introduce the ratio of o in z, i.e. mo = ||o||/||z||, where || ∗ || is the L2 norm. Similarly, the ratio of r in z is defined as mr = ||r||/||z||. When collapse happens, i.e. all vectors Z are close to the center vector o, mo approaches 1 and mr approaches 0, which is not desirable for SSL. A desirable case would be a relatively small mo and a relatively large mr, suggesting a relatively small (large) contribution of o (r) in each Z. We interpret the cause of collapse as a competition between o and r where o dominates over r, i.e. mo mr. For Eq 1, the derived negative gradient on Za (ignoring Zb for simplicity due to symmetry) is shown as:
Gcosine = − ∂LMSE ∂Za = Zb −Za ⇐⇒ − ∂Lcosine ∂Za = Zb, (3)
where the gradient component Za is a dummy term because the loss −Za · Za = −1 is a constant having zero gradient on the encoder f .
Conjecture1. With Za = oz + ra, we conjecture that the gradient component of oz is expected to update the encoder to boost the center vector thus increasemo, while the gradient component of ra is expected to behave in the opposite direction to increase mr. A random gradient component is expected to have a relatively small influence.
To verify the above conjecture, we revisit the dummy gradient term Za. We design loss −Za · sg(oz) and −Za · sg(Za − oz) to show the influence of gradient component o and ra respectively. The results in Fig. 4 show that the gradient component oz has the effect of increasingmo while decreasingmr. On the
contrary, ra helps increase mr while decreasing mo. Overall, the results verify Conjecture1.

heading: 3.2 EXTRA GRADIENT COMPONENT FOR ALLEVIATING COLLAPSE
text: Revisit collapse in a symmetric architecture. Based on Conjecture1, here, we provide an intuitive interpretation on why a symmetric Siamese architecture, such as Fig. 2 (a) and (b), cannot be trained without collapse. Take Fig. 2 (a) as example, the gradient in Eq 3 can be interpreted as two equivalent forms, from which we choose Zb−Za = (oz+rb)−(oz+ra) = rb−ra. Since rb comes from the same positive sample as ra, it is expected that rb also increases mr, however, this effect is expected to be smaller than that of ra, thus causing collapse.
Basic gradient and Extra gradient components. The negative gradient on Za in Fig. 2 (a) is derived as Zb, while that on Pa in Fig. 2 (b) is derived as Pb. We perceive Zb and Pb in these basic Siamese architectures as the Basic Gradient. Our above interpretation shows that such basic components cannot prevent collapse, for which an Extra Gradient component, denoted as Ge, needs to be introduced to break the symmetry. As the term suggests, Ge is defined as a gradient term that is relative to the basic gradient in a basic Siamese architecture. For example, negative samples can be introduced to Naive Siamese (Fig. 2 (a)) for preventing collapse, where the extra gradient caused by negative samples can thus be perceived as Ge with Zb as the basic gradient. Similarly, we can also disentangle the negative gradient on Pa in SimSiam (Fig. 1 (a)), i.e. Zb, into a basic gradient (which is Pb) and Ge which is derived as Zb −Pb (note that Zb = Pb + Ge). We analyze how Ge prevents collapse via studying the independent roles of its center vector oe and residual vector re.

heading: 3.3 A TOY EXAMPLE EXPERIMENT WITH NEGATIVE SAMPLE
text: Which repulsive component helps avoid collapse? Existing works often attribute the collapse in Naive Siamese to lacking a repulsive part during the optimization. This explanation has motivated previous works to adopt contrastive learning, i.e. attracting the positive samples while repulsing the negative samples. We experiment with a simple triplet loss1, Ltri = −Za·sg(Zb −Zn), where Zn indicates the representation of a Negative sample. The derived negative gradient on Za is Zb −Zn, where Zb is the basic gradient component and thus Ge = −Zn in this setup. For a sample representation, what determines it as a positive sample for attracting or a negative sample for repulsing is the residual component, thus it might be tempting to interpret that re is the key component of repulsive part that avoids the collapse. However, the results in Table 3 show that the component beneficial for preventing collapse inside Ge is oe instead of re. Specifically, to explore the individual influence of oe and re in the Ge, we design two experiments by removing one component while keeping the other one. In the first experiment, we remove the re in Ge while keeping the oe. By contrast, the oe is removed while keeping the re in the second experiment. In contrast to what existing explanations may expect, we find that the residual component oe prevents collapses. With Conjecture1, a gradient component alleviates collapse if it has negative center vector. In this setup, oe = −oz , thus oe has the de-centering role for preventing collapse. On the contrary, re does not prevent collapse and keeping re even decreases the performance (36.21% < 47.41%). Since the negative sample is randomly chosen, re just behaves like a random noise on the optimization to decrease performance.

heading: 3.4 DECOMPOSED GRADIENT ANALYSIS IN SIMSIAM
text: It is challenging to derive the gradient on the encoder output in SimSiam due to a nonlinear MLP module in h. The negative gradient on Pa for LSimSiam in Eq 2 can be derived as
GSimSiam = − ∂LSimSiam
∂Pa = Zb = Pb + (Zb − Pb) = Pb + Ge, (4)
oe re Collapse Top-1 (%) X X × 66.62 X × × 48.08 × X × 66.15 × × X 1
Table 4: Gradient component analysis for SimSiam.
where Ge indicates the aforementioned extra gradient component. To investigate the influence of oe and re on the collapse, similar to the analysis with the toy example experiment in Sec. 3.3, we design the experiment by removing one component while keeping the other. The results are reported in Table 4. As expected, the model collapses when both components in Ge are removed and the best performance is achieved when both components are kept. Interestingly, the model does not collapse when
either oe or re is kept. To start, we analyze how oe affects the collapse based on Conjecture1.
How oe alleviates collapse in SimSiam. Here, op is used to denote the center vector of P to differentiate from the above introduced oz for denoting that of Z. In this setup Ge = Zb − Pb, thus the residual gradient component is derived to be oe = oz − op. With Conjecture1, it is well expected that oe helps prevent collapse if oe contains negative op since the analyzed vector is Pa. To determine the amount of component of op existing in oe, we measure the cosine similarity between oe − ηpop and op for a wide range of ηp. The results in Fig. 5 (a) show that their cosine similarity is zero when ηp is around −0.5, suggesting oe has ≈ −0.5op. With Conjecture1, this negative ηp explains why SimSiam avoids collapse from the perspective of de-centering.
How oe causes collapse in Mirror SimSiam. As mentioned above, the collapse occurs in Mirror SimSiam, which can also be explained by analyzing its oe. Here, oe = op − oz , for which we evaluate the amount of component oz existing in oe via reporting the similarity between oe − ηzoz
1Note that the triplet loss here does not have clipping form as in Schroff et al. (2015) for simplicity.
and oz . The results in Fig. 5 (a) show that their cosine similarity is zero when ηz is set to around 0.2. This positive ηz explains why Fig. 1(c) causes collapse from the perspective of de-centering.
Overall, we find that processing the optimization target with h−1, as in Fig. 2 (c), alleviates collapse (ηp ≈ −0.5), while processing it with h, as in Fig. 1(c), actually strengthens the collapse (ηz ≈ 0.2). In other words, via the analysis of oe, our results help explain how SimSiam avoids collapse as well as how Mirror SimSiam causes collapse from a straightforward de-centering perspective.
Relation to prior works. Motivated from preventing the collapse to a constant, multiple prior works, such as W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021), explicitly adopt de-centering to prevent collapse. Despite various motivations, we find that they all implicitly introduce an oe that contains a negative center vector. The success of their approaches aligns well with our Conjecture1 as well as our above empirical results. Based on our findings, we argue that the effect of de-centering can be perceived as oe having a negative center vector. With this interpretation, we are the first to demonstrate that how SimSiam with predictor and stop gradient avoids collapse can be explained from the perspective of de-centering.
Beyond de-centering for avoiding collapse. In the toy example experiment in Sec. 3.3, re is found to be not beneficial for preventing collapse and keeping re even decreases the performance. Interestingly, as shown in Table 4, we find that re alone is sufficient for preventing collapse and achieves comparable performance as Ge. This can be explained from the perspective of dimensional de-correlation, which will be discussed in Sec. 3.5.

heading: 3.5 DIMENSIONAL DE-CORRELATION HELPS PREVENT COLLAPSE
text: Conjecture2 and motivation. We conjecture that dimensional de-correlation increases mr for preventing collapse. The motivation is straightforward as follows. The dimensional correlation would be minimum if only a single dimension has a very high value for every individual class and the dimension changes for different classes. In another extreme case, when all the dimensions have the same values, equivalent to having a single dimension, which already collapses by itself in the sense of losing representation capacity. Conceptually, re has no direct influence on the center vector, thus we interpret that re prevents collapse through increasing mr.
To verify the above conjecture, we train SimSiam normally with the loss in Eq 2 and train for several epochs with the loss in Eq 1 for intentionally decreasing the mr to close to zero. Then, we train the loss with only a correlation regularization term, which is detailed in Appendix A.6. The results in Fig. 5 (b) show that this regularization term increases mr at a very fast rate.
Dimensional de-correlation in SimSiam. Assuming h only has a single FC layer to exclude the influence of oe, the weights in FC are expected to learn the correlation between different dimensions for the encoder output. This interpretation echos well with the finding that the eigenspace of hweight aligns well with that of correlation matrix (Tian et al., 2021). In essence, the h is trained to minimize the cosine similarity between h(za) and I(zb), where I is identity mapping. Thus, h that learns the correlation is optimized close to I , which is conceptually equivalent to optimizing with the goal of de-correlation for Z. As shown in Table 4, for SimSiam, re alone also prevents collapse, which
is attributed to the de-correlation effect since re has no de-centering effect. We observe from Fig. 6 that except in the first few epochs, SimSiam decreases the covariance during the whole training. Fig. 6 also reports the results for InfoNCE which will be discussed in Sec. 4.

heading: 4 TOWARDS A UNIFIED UNDERSTANDING OF RECENT PROGRESS IN SSL
text: De-centering and de-correlation in InfoNCE. InfoNCE loss is a default choice in multiple seminal contrastive learning frameworks (Sohn, 2016; Wu et al., 2018; Oord et al., 2018; Wang & Liu, 2021). The derived negative gradient of InfoNCE on Za is proportional to Zb + ∑N i=0−λiZi, where λi = exp(Za·Zi/τ)∑N i=0 exp(Za·Zi/τ) , and Z0 = Zb for notation simplicity. See Appendix A.7 for the detailed
derivation. The extra gradient component Ge = ∑N i=0−λiZi = −oz − ∑N i=0 λiri, for which
oe = −oz and re = − ∑N i=0 λiri. Clearly, oe contains negative oz as de-centering for avoiding collapse, which is equivalent to the toy example in Sec. 3.3 when the re is removed. Regarding re, the main difference between Ltri in the toy example and InfoNCE is that the latter exploits a batch of negative samples instead of a random one. λi is proportional to exp(Za·Zi), indicating that a large weight is put on the negative sample when it is more similar to the anchor Za, for which, intuitively, its dimensional values tend to have a high correlation with Za. Thus, re containing such negative representation with a high weight tends to decrease dimensional correlation. To verify this intuition, we measure the cosine similarity between re and the gradient on Za induced by a correlation regularization loss. The results in Fig. 5 (c) show that their gradient similarity is high for a wide range of temperature values, especially when τ is around 0.1 or 0.2, suggesting re achieves similar role as an explicit regularization loss for performing de-correlation. Replacing re with oe leads to a low cosine similarity, which is expected because oe has no de-correlation effect.
The results of InfoNCE in Fig. 6 resembles that of SimSiam in terms of the overall trend. For example, InfoNCE also decreases the covariance value during training. Moreover, we also report the results of InfoNCE where re is removed for excluding the de-correlation effect. Removing re from the InfoNCE loss leads to a high covariance value during the whole training. Removing re also leads to a significant performance drop, which echos with the finding in (Bardes et al., 2021) that dimensional de-correlation is essential for competitive performance. Regarding how re in InfoNCE achieves de-correlation, formally, we hypothesize that the de-correlation effect in InfoNCE arises from the biased weights (λi) on negative samples. This hypothesis is corroborated by the temperature analysis in Fig. 7. We find that a higher temperature makes the weight distribution of λi more balanced indicated a higher entropy of λi, which echos with the finding in (Wang & Liu, 2021). Moreover, we observe that a higher temperature also tends to increase the covariance value. Overall, with temperature as the control variable, we find that more balanced weights among negative samples decrease the de-correlation effect, which constitutes an evidence for our hypothesis.
Unifying SimSiam and InfoNCE. At first sight, there is no conceptual similarity between SimSiam and InfoNCE, and this is why the community is intrigued by the success of SimSiam without negative samples. Through decomposing the Ge into oe and re, we find that for both, their oe plays the role of de-centering and their re behaves like de-correlation. In this sense, we bring two seemingly irrelevant frameworks into a unified perspective with disentangled de-centering and de-correlation.
Beyond SimSiam and InfoNCE. In SSL, there is a trend of performing explicit manipulation of de-centering and de-correlation, for which W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron et al., 2021) are three representative works. They often achieve performance comparable to those with InfoNCE or SimSiam. Towards a unified understanding of recent progress in SSL, our work is most similar to a concurrent work (Bardes et al., 2021). Their work
is mainly inspired by Barlow-twins (Zbontar et al., 2021) but decomposes its loss into three explicit components. By contrast, our work is motivated to answer the question of how SimSiam prevents collapse without negative samples. Their work claims that variance component (equivalent to decentering) is an indispensable component for preventing collapse, while we find that de-correlation itself alleviates collapse. Overall, demystifying SimSiam and InfoNCE through de-centering and de-correlation helps understand various frameworks in SSL from a more unified perspective.

heading: 5 TOWARDS SIMPLIFYING THE PREDICTOR IN SIMSIAM
text: Towards simplifying the predictor. Based on our understanding of how SimSiam prevents collapse, we demonstrate that simple components (instead of a non-linear MLP in SimSiam) in the predictor are sufficient for preventing collapse. For example, to achieve dimensional de-correlation, a single FC layer might be sufficient because a single FC layer can realize the interaction among various dimensions. On the other hand, to achieve de-centering, a single bias layer might be sufficient because a bias vector can represent the center vector. Attaching an l2-normalization layer at the end of the encoder, i.e. before the predictor, is found to be critical for achieving the above goal.
Pridictor with FC layers. To learn the dimensional correlation, an FC layer is sufficient theoretically but can be difficult to train in practice. Inspired by the property that Multiple FC layers make the training more stable even though they can be mathematically equivalent to a single FC layer (Bell-Kligler et al., 2019), we adopt two consecutive FC layers which are equivalent to removing the BN and ReLU in the original predictor.
The training can be made more stable if a Tanh layer is applied on the adopted single FC after every iteration. Table 5 shows that they achieve performance comparable to that with a non-linear MLP.
Predictor with a bias layer. A predictor with a single bias layer can be utilized for preventing collapse (see Table 5) and the trained bias vector is found to have a cosine similarity of 0.99 with the center vector (see Table 6). A bias in the MLP predictor also has a high cosine similarity of 0.89, suggesting that it is not a coincidence. A theoretical derivation for justifying such a
high similarity as well as how this single bias layer prevents collapse are discussed in Appendix A.8.

heading: 6 CONCLUSION
text: We point out a hidden flaw in prior works for explaining the success of SimSiam and propose to decompose the representation vector and analyze the decomposed components of extra gradient. We find that its center vector gradient helps prevent collapse via the de-centering effect and its residual gradient achieves de-correlation which also alleviates collapse. Our further analysis reveals that InfoNCE achieve the two effects in a similar manner, which bridges the gap between SimSiam and InfoNCE and contributes to a unified understanding of recent progress in SSL. Towards simplifying the predictor we have also found that a single bias layer is sufficient for preventing collapse.

heading: A APPENDIX

heading: A.1 EXPERIMENTAL SETTINGS
text: Self-supervised encoder training: Below are the settings for self-supervised encoder training. For simplicity, we mainly use the default settings in a popular open library termed solo-learn (da Costa et al., 2021).
Data augmentation and normalization: We use a series of transformations including RandomResizedCrop with scale [0.2, 1.0], bicubic interpolation. ColorJitter (brightness (0.4), contrast (0.4), saturation (0.4), hue (0.1)) is randomly applied with the probability of 0.8. Random gray scale RandomGrayscale is applied with p = 0.2 Horizontal flip is applied with p = 0.5 The images are normalized with the mean (0.4914, 0.4822, 0.4465) and Std (0.247, 0.243, 0.261).
Network architecture and initialization: The backbone architecture is ResNet-18. The projection head contains three fully-connected (FC) layers followed by Batch Norm (BN) and ReLU, for which ReLU in the final FC layer is removed, i.e. FC1+BN+ReLU+FC2+BN+ReLU+FC3+BN . All projection FC layers have 2048 neurons for input, output as well as the hidden dimensions. The predictor head includes two FC layers as follows: FC1 + BN + ReLU + FC2. Input and output of the predictor both have the dimension of 2048, while the hidden dimension is 512. All layers of the network are by default initialized in Pytorch.
Optimizer: SGD optimizer is used for the encoder training. The batch size M is 256 and the learning rate is linearly scaled by the formula lr × M/256 with the base learning rate lr set to 0.5. The schedule for learning rate adopts the cosine decay as SimSiam. Momentum 0.9 and weight decay 1.0 × 10−5 are used for SGD. We use one GPU for each pre-training experiment. Following the practice of SimSiam, the learning rate of the predictor is fixed during the training. We use warmup training for the first 10 epochs. If not specified, by default we train the model for 1000 epochs.
Online linear evaluation: For the online linear revaluation, we also follow the practice in the solo-learn library (da Costa et al., 2021). The frozen features (2048 dimensions) from the training set are extracted (from the self-supervised pre-trained model) to feed into a linear classifier (1 FC layer with the input 2048 and output of 100). The test is performed on the validation set. The learning rate for the linear classifier is 0.1. Overall, we report Top-1 accuracy with the online linear evaluation in this work.

heading: A.2 TWO SUB-PROBLEMS IN AO OF SIMSIAM
text: In the sub-problem ηt ← arg minη L(θt, η), ηt indicating latent representation of images at step t is actually obtained through ηtx ← ET [ Fθt(T (x)) ] , where they in practice ignore ET [·] and sample only one augmentation T ′, i.e. ηtx ← Fθt(T ′(x)). Conceptually, Chen & He equate the role of predictor to EOA.

heading: A.3 EXPERIMENTAL DETAILS FOR EXPLICIT EOA IN TABLE 1
text: In the Moving average experiment, we follow the setting in SimSiam (Chen & He, 2021) without predictor. In the Same batch experiment, multiple augmentations, 10 augmentations for instance, are applied on the same image. With multi augmentations, we get the corresponding encoded representation, i.e. zi, i ∈ [1, 10]. We minimize the cosine distance between the first representation z1 and the average of the remaining vectors, i.e. z̄ = 19 ∑10 i=2 zi. The gradient stop is put on the averaged vector. We also experimented with letting the gradient backward through more augmentations, however, they consistently led to collapse.

heading: A.4 EXPERIMENTAL SETUP AND RESULT TREND FOR TABLE 2.
text: Mirror SimSiam. Here we provide the pseudocode for Mirror SimSiam. In the Mirror SimSiam experiment which relates to Fig. 1 (c). Without taking symmetric loss into account, the pseudocode is shown in Algorithm 1. Taking symmetric loss into account, the pseudocode is shown in Algorithm 2.
Algorithm 1 Pytorch-like Pseudocode: Mirror SimSiam
# f: encoder (backbone + projector) # h: predictor
for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b) # projections
p_b = h(z_b.detach()) # detach z_b but still allowing gradient p_b
L = D_cosine(z_a, p_b) # loss
L.backward() # back-propagate update(f, h) # SGD update
def D_cosine(z, p): # negative cosine similarity z = normalize(z, dim=1) # l2-normalize p = normalize(p, dim=1) # l2-normalize return -(z*p).sum(dim=1).mean()
Algorithm 2 Pytorch-like Pseudocode: Mirror SimSiam
# f: encoder (backbone + projector) # h: predictor
for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b) # projections
p_b = h(z_b.detach()) # detach z_b but still allowing gradient p_b p_a = h(z_a.detach()) # detach z_a but still allowing gradient p_a
L = D_cosine(z_a, p_b)/2 + D_cosine(z_b, p_a)/2 # loss
L.backward() # back-propagate update(f, h) # SGD update
def D_cosine(z, p): # negative cosine similarity z = normalize(z, dim=1) # l2-normalize p = normalize(p, dim=1) # l2-normalize return -(z*p).sum(dim=1).mean()
Symmetric Predictor. To implement the SimSiam with Symmetric Predictor as in Fig. 2 (b), we can just perceive the predictor as part of the new encoder, for which the pseudocode is provided in Algorithm 3. Alternatively, we can additionally train the predictor similarly as that in SimSiam, for which the training involves two losses, one for training the predictor and another for training the new encoder (the corresponding pseudocode is provided in Algorithm 4). Moreover, for the second implementation, we also experiment with another variant that fixes the predictor while optimizing the new encoder and then train the predictor alternatingly. All of them lead to collapse with a similar trend as long as the symmetric predictor is used for training the encoder. For avoiding redundancy, in Fig. 8 we only report the result of the second implementation.
Result trend. The result trend of SimSiam, Naive Siamese, Mirror SimSiam, Symmetric Predictor are shown in Fig. 8. We observe that all architectures lead to collapse except for SimSiam. Mirroe SimSiam was stopped in the middle because a NaN value was returned from the loss.

heading: A.5 EXPERIMENTAL DETAILS FOR INVERSE PREDICTOR.
text: In the inverse predictor experiment which relates to Fig. 2 (c), we introduce a new predictor which has the same structure as that of the original predictor. The training loss consists of 3 parts: predictor training loss, inverse predictor training and new encoder (old encoder+predictor) training. The new
Algorithm 3 Pytorch-like Pseudocode: Symmetric Predictor
# f: encoder (backbone + projector) # h: predictor
for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b) # projections p_a, p_b = h(z_a), h(z_b) # predictions
L = D(p_a, p_b)/2 + D(p_b, p_a)/2 # loss
L.backward() # back-propagate update(f, h) # SGD update
def D(p, z): # negative cosine similarity z = z.detach() # stop gradient p = normalize(p, dim=1) # l2-normalize z = normalize(z, dim=1) # l2-normalize return -(p*z).sum(dim=1).mean()
Algorithm 4 Pytorch-like Pseudocode: Symmetric Predictor (with additional training on predictor)
# f: encoder (backbone + projector) # h: predictor
for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b) # projections p_a, p_b = h(z_a), h(z_b) # predictions
d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach()) # detached predictor output
# predictor training loss L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2
# encoder training loss L_enc = D(p_a, d_p_b)/2 + D(p_b, d_p_a)/2
L = L_pred + L_enc
L.backward() # back-propagate update(f, h) # SGD update
def D(p, z): # negative cosine similarity with detach on z z = z.detach() # stop gradient p = normalize(p, dim=1) # l2-normalize z = normalize(z, dim=1) # l2-normalize return -(p*z).sum(dim=1).mean()
encoder F consists of the old encoder f + predictor h. The practice of gradient stop needs to be considered in the implementation. We provide the pseudocode in Algorithm 5.
Algorithm 5 Pytorch-like Pseudocode: Trainable Inverse Predictor
# f: encoder (backbone + projector) # h: predictor # h_inv: inverse predictor
for x in loader: # load a minibatch x with n samples x_a, x_b = aug(x), aug(x) # augmentation z_a, z_b = f(x_a), f(x_b) # projections p_a, p_b = h(z_a), h(z_b) # predictions
d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach()) # detached predictor output # predictor training loss L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2 # to train h
inv_p_a, inv_p_b = h_inv(p_a.detach()), h_inv(p_b.detach()) # to train h_inv # inverse predictor training loss L_inv_pred = D(inv_p_a, z_a)/2 + D(inv_p_b, z_b)/2
# encoder training loss L_enc = D(p_a, h_inv(p_b))/2 + D(p_b, h_inv(p_a))
L = L_pred + L_inv_pred + L_enc
L.backward() # back-propagate update(f, h, h_inv) # SGD update
def D(p, z): # negative cosine similarity with detach on z z = z.detach() # stop gradient p = normalize(p, dim=1) # l2-normalize z = normalize(z, dim=1) # l2-normalize return -(p*z).sum(dim=1).mean()

heading: A.6 REGULARIZATION LOSS
text: Following Zbontar et al. (2021), we compute covariance regularization loss of encoder output along the mini-batch. The pseudocode for de-correlation loss calculation is put in Algorithm 6.
Algorithm 6 Pytorch-like Pseudocode: De-correlation loss
# Z_a: representation vector # N: batch size # D: the number of dimension for representation vector
Z_a = Z_a - Z_a.mean(dim=0)
cov = Z_a.T @ Z_a / (N-1) diag = torch.eye(D)
loss = cov[˜diag.bool()].pow_(2).sum() / D

heading: A.7 GRADIENT DERIVATION AND TEMPERATURE ANALYSIS FOR INFONCE
text: With · indicating the cosine similarity between vectors, the InfoNCE loss can be expressed as
LInfoNCE = − log exp(Za ·Zb/τ) exp(Za ·Zb/τ) + ∑N i=1 exp(Za ·Zi/τ)
= − log exp(Za ·Zb/τ)∑N i=0 exp(Za ·Zi/τ) ,
(5)
where N indicates the number of negative samples and Z0 = Zb for simplifying the notation. By treating Za · Zi as the logit in a normal CE loss, we have the corresponding probability for each negative sample as λi =
exp(Za·Zi/τ)∑N i=0 exp(Za·Zi/τ) , where i = 0, 1, 2, ..., N and we have ∑N i=0 λi = 1.
The negative gradient of the InfoNCE on the representation Za is shown as
−∂LInfoNCE ∂Za = 1 τ (1− λ0)Zb − 1 τ N∑ i=1 λiZi
= 1
τ (Zb − N∑ i=0 λiZi)
= 1
τ (Zb − N∑ i=0 λi(oz + ri))
= 1
τ (Zb + (−oz − N∑ i=0 λiri)
∝ Zb + (−oz − N∑ i=0 λiri)
(6)
where 1τ can be adjusted through learning rate and is omitted for simple discussion. With Zb as the basic gradient, Ge = −oz − ∑N i=0 λiri, for which oe = −oz and re = − ∑N i=0 λiri. When the temperature is set to a large value, λi = exp(Za·Zi/τ)∑N i=0 exp(Za·Zi/τ)
, approaches 1N+1 , indicated by a high entropy value (see Fig. 7). InfoNCE will degenerate to a simple contrastive loss, i.e. Lsimple = −Za · Zb + 1N+1 ∑N i=0 Za · Zi , which repulses every negative sample with an equal force. In contrast, a relative smaller temperature will give more relative weight, i.e. larger λ, to negative samples that are more similar to the anchor (Za).
The influence of the temperature on the covariance and accuracy is shown in Fig. 7 (b) and (c). We observe that a higher temperature tends to decrease the effect of de-correlation, indicated by a higher covariance value, which also leads to a performance drop. This verifies our hypothesis regarding on how re in InfoNCE achieves de-correlation because a large temperature causes more balanced weights λi, which is found to alleviate the effect of de-correlation. For the setup, we note that the encoder is trained for 200 epochs with the default setting in Solo-learn for the SimCLR framework.

heading: A.8 THEORETICAL DERIVATION FOR A SINGLE BIAS LAYER
text: With the cosine similarity loss defined as Eq 7 Eq 8:
cossim(a, b) = a · b√ a2 · b2 , (7)
for which the derived gradient on the vector a is shown as
∂
∂a cossim(a, b) = b1 |a| · |b| − cossim(a, b) · a1 |a|2 . (8)
The above equation is used as a prior for our following derivations. As indicated in the main manuscript, the encoder output za is l2-normalized before feeding into the predictor, thus pa = Za + bp, bp denotes the bias layer in the predictor. The cosine similarity loss (ignoring the symmetry for simplicity) is shown as
Lcosine = −Pa ·Zb = − pa ||pa|| · zb ‖zb‖
(9)
The gradient on pa is derived as
−∂Lcosine ∂pa = zb ‖zb‖ · ‖pa‖ − cossim(Za,Zb) · pa ||pa||2
= 1
‖pa‖ ( zb ‖zb‖ − cossim(Za,Zb) · Pa )
= 1
‖pa‖
( Zb − cossim(Za,Zb) ·
Za + bp ‖pa‖ ) = 1
‖pa‖
( (oz + rb)− cossim(Za,Zb)
‖pa‖ · (oz + ra + bp) ) = 1
‖pa‖ ((oz + rb)−m · (oz + ra + bp))
= 1
‖pa‖ ((1−m)oz −mbp + rb −m · ra) ,
(10)
where m = cossim(Za,Zb)‖pa‖ .
Given that pa = Za + bp, the negative gradient on bp is the same as that on pa as
−∂Lcosine ∂bp = −∂Lcosine ∂pa
= 1
‖pa‖ ((1−m)oz −mbp + rb −m · ra) .
(11)
We assume that the training is stable and the bias layer converges to a certain value when −∂cossim(Za,Zb)∂bp = 0. Thus, the converged bp satisfies the following constraint:
1
‖pa‖ ((1−m)oz −mbp + rb −mra)) = 0
bp = 1−m m oz + 1 m rb − ra.
(12)
With a batch of samples, the average of 1mrb and ra is expected to be close to 0 by the definition of residual vector. Thus, the bias layer vector is expected to converge to:
bp = 1−m m oz. (13)
Rational behind the high similarity between bp and oz . The above theoretical derivation shows that the parameters in the bias layer are excepted to converge to a vector 1−mm oz . This theoretical derivation justifies why the empirically observed cosine similarity between bp and oz is as high as 0.99. Ideally, it should be 1, however, such a small deviation is expected with the training dynamics taken into account.
Rational behind how a single bias layer prevents collapse. Given that pa = Za+bp, the negative gradient on Za is shown as
−∂Lcosine ∂Za = −∂Lcosine ∂pa
= 1
‖pa‖
( Zb − cossim(Za,Zb) ·
Za + bp ‖pa‖ ) = 1
‖pa‖ Zb −
cossim(Za,Zb)
‖pa‖2 Za −
cossim(Za,Zb)
‖pa‖2 bp.
(14)
Here, we highlight that since the loss −Za ·Za = −1 is a constant having zero gradients on the encoder,− cossim(Za,Zb)‖pa‖2 Za can be seen as a dummy term. Considering Eq 13 andm = cossim(Za,Zb) ‖pa‖ ,
we have b = ( ‖pa‖cossim(Za,Zb) − 1)oz . The above equation is equivalent to
−∂Lcosine ∂Za = 1 ‖pa‖ Zb − cossim(Za,Zb) ‖pa‖2 bp
= 1
‖pa‖ Zb −
cossim(Za,Zb)
‖pa‖2 ( ‖pa‖ cossim(Za,Zb) − 1)oz
= 1
‖pa‖ Zb −
1 ‖pa‖ (1− cossim(Za,Zb) ‖pa‖ )oz
∝ Zb − (1− cossim(Za,Zb)
‖pa‖ )oz.
(15)
With Zb as the basic gradient, the extra gradient component Ge = −(1− cossim(Za,Zb)‖pa‖ )oz . Given that pa = Za +bp and ‖Za‖ = 1, thus ‖pa‖ < 1 only when Za is negatively correlated with bp. In practice, however, Za and bp are often positively correlated to some extent due to their shared center vector component. In other words, ‖pa‖ > 1. Moreover, cossim(Za,Zb) is smaller than 1, thus −(1− cossim(Za,Zb)‖pa‖ ) < 0, suggesting Ge consists of negative oz with the effect of de-centerization. This above derivation justifies the rationale why a single bias layer can help alleviate collapse.

heading: B DISCUSSION: DOES BN HELP AVOID COLLAPSE?
text: To our knowledge, our work is the first to revisit and refute the explanatory claims in (Chen & He, 2021). Several works, however, have attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam. The success has been ascribed to BN in (Fetterman & Albrecht, 2020), however, (Richemond et al., 2020) refutes their claim. Since the role of intermediate BNs is ascribed to stabilize training (Richemond et al., 2020; Chen & He, 2021), we only discuss the final BN in the SimSiam encoder. Note that with our Conjecture1, the final BN that removes the mean of representation vector is supposed to have de-centering effect. BY default SimSiam has such a BN at the end of its encoder, however, it still collapses with the predictor and stop gradient. Why would such a BN not prevent collapse in this case? Interestingly, we observe that such BN can help alleviate collapse with a simple MSE loss (see Fig. 9), however, its performance is is inferior to the cosine loss-based SimSiam (with predictor and stop gradient) due to the lack of the de-correlation effect in SimSiam. Note that the cosine loss is in essence equivalent to a MSE loss on the l2normalized vectors. This phenomenon can be interpreted as that the l2-normalization causes another mean after the BN removes it. Thus, with such l2-normalization in the MSE loss, i.e. adopting the default cosine loss, it is important to remove the oe from the optimization target. The results with the loss of −Za · sg(Zb + oe) in Table 3 show that this indeed prevents collapse and verifies the above interpretation.

Review summaries:
The paper proposes another explanation for why SimSiam can avoid collapse without negative samples. Specifically, the paper decomposes the gradient of learned representation as center vector and residual vector and finds that the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect. Such an explanation can also be applied to Info-NCE, which unifies the theory of self-supervised learning with and without negative samples.

This paper proposes a framework to understand why SimSiam avoid collapse without negative samples? It provides a hidden flaw of the Alternating Optimization for explain why SimSiam works. And the authors claim that  the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect.

The paper analyzes how the self-supervised learning (SSL) approach SimSiam avoids collapsed representations without explicit formulation of repulsive sample relations. To this end, first flaws in the original reasoning of the SimSiam paper are revealed. Next, based on center-residual vector decomposition, the role of the prediction head for preventing representation collapse in SimSiam is analyzed. Results indicate the importance of de-centralization and de-correlation as driving concepts for stable SSL.

As the title suggests, the paper does a detailed investigation of how SimSiam avoids collapse without negative training examples. The key idea is to decompose the original vector into a center vector component and a residual vector component. The center vector cannot be too large (otherwise it is indicating collapse). The high-level idea is that the designs in SimSiam (and contrastive frameworks that have InfoNCE loss)  are mainly to prevent the center vector from getting too large. There are conjectures (verified empirically) about the relationship between the gradient w.r.t. the center vector ands the gradient w.r.t. the residual vector, and with these the paper finds that for SimSiam, the predictor is important for preventing collapse, particularly by doing de-correlation among features.