How Does SimSiam Avoid Collapse Without Negative Samples? Towards a Unified Understanding of Progress in SSL

1 INTRODUCTION . Beyond the success of NLP ( Lan et al. , 2020 ; Radford et al. , 2019 ; Devlin et al. , 2019 ; Su et al. , 2020 ; Nie et al. , 2020 ) , self-supervised learning ( SSL ) has also shown its potential in the field of vision tasks ( Li et al. , 2021 ; Chen et al. , 2021 ; El-Nouby et al. , 2021 ) . Without the ground-truth label , the core of most SSL methods lies in learning an encoder with augmentation-invariant representation ( Bachman et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020a ; Caron et al. , 2020 ; Grill et al. , 2020 ) . Specifically , they often minimize the representation distance between two positive samples , i.e . two augmented views of the same image , based on a Siamese network architecture ( Bromley et al. , 1993 ) . It is widely known that for such Siamese networks there exists a degenerate solution , i.e . all outputs “ collapsing ” to an undesired constant ( Chen et al. , 2020a ; Chen & He , 2021 ) . Early works have attributed the collapse to lacking a repulsive component in the optimization goal and adopted negative samples , i.e . views of different samples , to alleviate this problem . Introducing momentum into the target encoder , BYOL shows that Siamese architectures can be trained with only positive pairs . More recently , SimSiam ( Chen & He , 2021 ) has caught great attention by further simplifying BYOL by removing the momentum encoder , which has been seen as a major milestone achievement in SSL for providing a minimalist method for achieving competitive performance . However , the reason for such success remains unclear with a core question summarized as : How does SimSiam avoid collapse without negative samples ? Our investigation starts with revisiting the explanatory claims in the original SimSiam paper ( Chen & He , 2021 ) . Notably , two components , i.e . stop gradient and predictor , are essential for the success of SimSiam ( Chen & He , 2021 ) . The reason has been mainly attributed to the stop gradient ( Chen & He , 2021 ) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set . Chen & He argue that the predictor h is helpful in SimSiam because h fills the gap to approximate expectation over augmentations ( EOA ) . Unfortunately , the above explanatory claims are found to be flawed due to reversing the two paths with and without gradient ( see Sec . 2.2 ) . This motivates us to find an alternative explanation , for which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL . Specifically , we propose to decompose a representation vector into center and residual components . This decomposition facilitates understanding which gradient component is beneficial for avoiding collapse . Under this framework , we show that a basic Siamese architecture can not prevent collapse , for which an extra gradient component needs to be introduced . With SimSiam interpreted as processing the optimization target with an inverse predictor , the analysis of its extra gradient shows that ( a ) its center vector helps prevent collapse via the de-centering effect ; ( b ) its residual vector achieves dimensional de-correlation which also alleviates collapse . Moreover , under the same gradient decomposition , we find that the extra gradient caused by negative samples in InfoNCE ( He et al. , 2019 ; Chen et al. , 2020b ; a ; Tian et al. , 2019 ; Khosla et al. , 2020 ) also achieves de-centering and de-correlation in the same manner . It bridges the gap between SimSiam and InfoNCE and contributes to a unified understanding of the recent progress of various frameworks in SSL . Finally , towards simplifying the predictor for more explainable SimSiam , we show that a single bias layer is sufficient enough for preventing collapse . The basic experimental settings for our analysis are detailed in Appendix A.1 with a more specific setup discussed in the context . Overall , our work is the first attempt for performing a comprehensive study on how SimSiam avoids collapse without negative samples . Several works , however , have attempted to demystify the success of BYOL ( Grill et al. , 2020 ) , a close variant of SimSiam . A technical report ( Fetterman & Albrecht , 2020 ) has suggested the importance of batch normalization ( BN ) in BYOL for its success , however , a recent work ( Richemond et al. , 2020 ) refutes their claim by showing BYOL works without BN , which is discussed in Appendix B . 2 REVISITING SIMSIAM AND ITS EXPLANATORY CLAIMS . l2-normalized vector and optimization goal . SSL trains an encoder f for learning discriminative representation and we denote such representation as a vector z , i.e . f ( x ) = z where x is a certain input . For the augmentation-invariant representation , a straightforward goal is to minimize the distance between the representations of two positive samples , i.e . augmented views of the same image , for which mean squared error ( MSE ) is a default choice . To avoid scale ambiguity , the vectors are often l2-normalized , i.e . Z = z/||z|| ( Chen & He , 2021 ) , before calculating the MSE : LMSE = ( Za −Zb ) 2/2− 1 = −Za ·Zb = Lcosine , ( 1 ) which shows the equivalence of a normalized MSE loss to the cosine loss ( Grill et al. , 2020 ) . Collapse in SSL and solution of SimSiam . Based on a Siamese architecture , the loss in Eq 1 causes the collapse , i.e . f always outputs a constant regardless of the input variance . We refer to this Siamese architecture with loss Eq 1 as Naive Siamese in the remainder of paper . Contrastive loss with negative samples is a widely used solution ( Chen et al. , 2020a ) . Without using negative samples , SimSiam solves the collapse problem via predictor and stop gradient , based on which the encoder is optimized with a symmetric loss : LSimSiam = − ( Pa · sg ( Zb ) + Pb · sg ( Za ) ) , ( 2 ) where sg ( · ) is stop gradient and P is the output of predictor h , i.e . p = h ( z ) and P = p/||p|| . 2.1 REVISING EXPLANATORY CLAIMS IN SIMSIAM . Interpreting stop gradient as AO . Chen & He hypothesize that the stop gradient in Eq 2 is an implementation of Alternating between the Optimization of two sub-problems , which is denoted as AO . Specifically , with the loss considered as L ( θ , η ) = Ex , T [ ∥∥Fθ ( T ( x ) ) − ηx∥∥2 ] , the optimization objective minθ , η L ( θ , η ) can be solved by alternating ηt ← arg minη L ( θt , η ) and θt+1 ← arg minθ L ( θ , ηt ) . It is acknowledged that this hypothesis does not fully explain why the collapse is prevented ( Chen & He , 2021 ) . Nonetheless , they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difficult to approach a constant ∀x . Interpreting predictor as EOA . The AO problem ( Chen & He , 2021 ) is formulated independent of predictor h , for which they believe that the usage of predictor h is related to approximating EOA for filling the gap of ignoring ET [ · ] in a sub-problem of AO . The approximation of ET [ · ] is summarized in Appendix A.2 . Chen & He support their interpretation by proof-of-concept experiments . Specifically , they show that updating ηx with a moving-average ηtx ← m ∗ ηtx + ( 1 −m ) ∗ Fθt ( T ′ ( x ) ) can help prevent collapse without predictor ( see Fig . 1 ( b ) ) . Given that the training completely fails when the predictor and moving average are both removed , at first sight , their reasoning seems valid . 2.2 DOES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA ? . Reasoning flaw . Considering the stop gradient , we divide the framework into two sub-models with different paths and term them Gradient Path ( GP ) and Stop Gradient Path ( SGP ) . For SimSiam , only the sub-model with GP includes the predictor ( see Fig . 1 ( a ) ) . We point out that their reasoning flaw of predictor analysis lies in the reverse of GP and SGP . By default , the moving-average sub-model , as shown in Fig . 1 ( b ) , is on the same side as SGP . Note that Fig . 1 ( b ) is conceptually similar to Fig . 1 ( c ) instead of Fig . 1 ( a ) . It is worth mentioning that the Mirror SimSiam in Fig . 1 ( c ) is what stop gradient in the original SimSiam avoids . Therefore , it is problematic to perceive h as EOA . New Figure 1 Explicit EOA does not prevent collapse . ( Chen & He , 2021 ) points out that “ in practice , it would be unrealistic to actually compute the expectation ET [ · ] . But it may be possible for a neural network ( e.g. , the preditor h ) to learn to predict the expectation , while the sampling of T is implicitly distributed across multiple epochs. ” If implicitly sampling across multiple epochs is beneficial , explicitly sampling sufficient large N augmentations in a batch with the latest model would be more beneficial to approximate ET [ · ] . However , Table 1 shows that the collapse still occurs and suggests that the equivalence between predictor and EOA does not hold . 2.3 ASYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN SIMSIAM . Symmetric Predictor does not prevent collapse . The difference between Naive Siamese and Simsiam lies in whether the gradient in backward propagation flows through a predictor , however , we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path . With h being trained the same as Eq 2 , we optimize the encoder f through replacing the Z in Eq 2 with P . The results in Table . 2 show that it still leads to collapse . Actually , this is well expected by perceiving h to be part of the new encoder F , i.e . p = F ( x ) = h ( f ( x ) ) . In other words , the symmetric architectures with and without predictor h both lead to collapse . Predictor with stop gradient is asymmetric . Clearly , how SimSiam avoids collapse lies in its asymmetric architecture , i.e . one path with h and the other without h. Under this asymmetric architecture , the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target , not vice versa . In other words , the SimSiam avoids collapse by excluding Mirror SimSiam ( Fig . 1 ( c ) ) which has a loss ( mirror-like Eq 2 ) asLMirror = − ( Pa ·Zb+Pb ·Za ) , where stop gradient is put on the input of h , i.e . pa = h ( sg [ za ] ) and pb = h ( sg [ zb ] ) . Predictor vs. inverse predictor . We interpret h as a function mapping from z to p , and introduce a conceptual inverse mapping h−1 , i.e . z = h−1 ( p ) . Here , as shown in Table 2 , SimSiam with symmetric predictor ( Fig . 2 ( b ) ) leads to collapse , while SimSiam ( Fig . 1 ( a ) ) avoids collapse . With the conceptual h−1 , we interpret Fig . 1 ( a ) the same as Fig . 2 ( c ) which differs from Fig . 2 ( b ) via changing the optimization target from pb to zb , i.e . zb = h−1 ( pb ) . This interpretation suggests that the collapse can be avoided by processing the optimization target with h−1 . By contrast , Fig . 1 ( c ) and Fig . 2 ( a ) both lead to collapse , suggesting that processing the optimization target with h is not beneficial for preventing collapse . Overall , asymmetry alone does not guarantee collapse avoidance , which requires the optimization target to be processed by h−1 not h. Trainable inverse predictor and its implication on EOA . In the above , we propose a conceptual inverse predictor h−1 in Fig . 2 ( c ) , however , it remains yet unknown whether such an inverse predictor is experimentally trainable . A detailed setup for this investigation is reported in Appendix A.5 . The results in Fig . 3 show that a learnable h−1 leads to slightly inferior performance , which is expected because h−1 can not make the trainable inverse predictor output z∗b completely the same as zb . Note that it would be equivalent to SimSiam if z∗b = zb . Despite a slight performance drop , the results confirm that h−1 is trainable . The fact that h−1 is trainable provides additional evidence that the role h plays in SimSiam is not EOA because theoretically h−1 can not restore a random augmentation T ′ from an expectation p , where p = h ( z ) = ET [ Fθt ( T ( x ) ) ] .

Summary: The paper analyzes how the self-supervised learning (SSL) approach SimSiam avoids collapsed representations without explicit formulation of repulsive sample relations. To this end, first flaws in the original reasoning of the SimSiam paper are revealed. Next, based on center-residual vector decomposition, the role of the prediction head for preventing representation collapse in SimSiam is analyzed. Results indicate the importance of de-centralization and de-correlation as driving concepts for stable SSL.