How Does SimSiam Avoid Collapse Without Negative Samples? Towards a Unified Understanding of Progress in SSL

1 INTRODUCTION . Beyond the success of NLP ( Lan et al. , 2020 ; Radford et al. , 2019 ; Devlin et al. , 2019 ; Su et al. , 2020 ; Nie et al. , 2020 ) , self-supervised learning ( SSL ) has also shown its potential in the field of vision tasks ( Li et al. , 2021 ; Chen et al. , 2021 ; El-Nouby et al. , 2021 ) . Without the ground-truth label , the core of most SSL methods lies in learning an encoder with augmentation-invariant representation ( Bachman et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020a ; Caron et al. , 2020 ; Grill et al. , 2020 ) . Specifically , they often minimize the representation distance between two positive samples , i.e . two augmented views of the same image , based on a Siamese network architecture ( Bromley et al. , 1993 ) . It is widely known that for such Siamese networks there exists a degenerate solution , i.e . all outputs “ collapsing ” to an undesired constant ( Chen et al. , 2020a ; Chen & He , 2021 ) . Early works have attributed the collapse to lacking a repulsive component in the optimization goal and adopted negative samples , i.e . views of different samples , to alleviate this problem . Introducing momentum into the target encoder , BYOL shows that Siamese architectures can be trained with only positive pairs . More recently , SimSiam ( Chen & He , 2021 ) has caught great attention by further simplifying BYOL by removing the momentum encoder , which has been seen as a major milestone achievement in SSL for providing a minimalist method for achieving competitive performance . However , the reason for such success remains unclear with a core question summarized as : How does SimSiam avoid collapse without negative samples ? Our investigation starts with revisiting the explanatory claims in the original SimSiam paper ( Chen & He , 2021 ) . Notably , two components , i.e . stop gradient and predictor , are essential for the success of SimSiam ( Chen & He , 2021 ) . The reason has been mainly attributed to the stop gradient ( Chen & He , 2021 ) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves like alternating between optimizing each set . Chen & He argue that the predictor h is helpful in SimSiam because h fills the gap to approximate expectation over augmentations ( EOA ) . Unfortunately , the above explanatory claims are found to be flawed due to reversing the two paths with and without gradient ( see Sec . 2.2 ) . This motivates us to find an alternative explanation , for which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL . Specifically , we propose to decompose a representation vector into center and residual components . This decomposition facilitates understanding which gradient component is beneficial for avoiding collapse . Under this framework , we show that a basic Siamese architecture can not prevent collapse , for which an extra gradient component needs to be introduced . With SimSiam interpreted as processing the optimization target with an inverse predictor , the analysis of its extra gradient shows that ( a ) its center vector helps prevent collapse via the de-centering effect ; ( b ) its residual vector achieves dimensional de-correlation which also alleviates collapse . Moreover , under the same gradient decomposition , we find that the extra gradient caused by negative samples in InfoNCE ( He et al. , 2019 ; Chen et al. , 2020b ; a ; Tian et al. , 2019 ; Khosla et al. , 2020 ) also achieves de-centering and de-correlation in the same manner . It bridges the gap between SimSiam and InfoNCE and contributes to a unified understanding of the recent progress of various frameworks in SSL . Finally , towards simplifying the predictor for more explainable SimSiam , we show that a single bias layer is sufficient enough for preventing collapse . The basic experimental settings for our analysis are detailed in Appendix A.1 with a more specific setup discussed in the context . Overall , our work is the first attempt for performing a comprehensive study on how SimSiam avoids collapse without negative samples . Several works , however , have attempted to demystify the success of BYOL ( Grill et al. , 2020 ) , a close variant of SimSiam . A technical report ( Fetterman & Albrecht , 2020 ) has suggested the importance of batch normalization ( BN ) in BYOL for its success , however , a recent work ( Richemond et al. , 2020 ) refutes their claim by showing BYOL works without BN , which is discussed in Appendix B . 2 REVISITING SIMSIAM AND ITS EXPLANATORY CLAIMS . l2-normalized vector and optimization goal . SSL trains an encoder f for learning discriminative representation and we denote such representation as a vector z , i.e . f ( x ) = z where x is a certain input . For the augmentation-invariant representation , a straightforward goal is to minimize the distance between the representations of two positive samples , i.e . augmented views of the same image , for which mean squared error ( MSE ) is a default choice . To avoid scale ambiguity , the vectors are often l2-normalized , i.e . Z = z/||z|| ( Chen & He , 2021 ) , before calculating the MSE : LMSE = ( Za −Zb ) 2/2− 1 = −Za ·Zb = Lcosine , ( 1 ) which shows the equivalence of a normalized MSE loss to the cosine loss ( Grill et al. , 2020 ) . Collapse in SSL and solution of SimSiam . Based on a Siamese architecture , the loss in Eq 1 causes the collapse , i.e . f always outputs a constant regardless of the input variance . We refer to this Siamese architecture with loss Eq 1 as Naive Siamese in the remainder of paper . Contrastive loss with negative samples is a widely used solution ( Chen et al. , 2020a ) . Without using negative samples , SimSiam solves the collapse problem via predictor and stop gradient , based on which the encoder is optimized with a symmetric loss : LSimSiam = − ( Pa · sg ( Zb ) + Pb · sg ( Za ) ) , ( 2 ) where sg ( · ) is stop gradient and P is the output of predictor h , i.e . p = h ( z ) and P = p/||p|| . 2.1 REVISING EXPLANATORY CLAIMS IN SIMSIAM . Interpreting stop gradient as AO . Chen & He hypothesize that the stop gradient in Eq 2 is an implementation of Alternating between the Optimization of two sub-problems , which is denoted as AO . Specifically , with the loss considered as L ( θ , η ) = Ex , T [ ∥∥Fθ ( T ( x ) ) − ηx∥∥2 ] , the optimization objective minθ , η L ( θ , η ) can be solved by alternating ηt ← arg minη L ( θt , η ) and θt+1 ← arg minθ L ( θ , ηt ) . It is acknowledged that this hypothesis does not fully explain why the collapse is prevented ( Chen & He , 2021 ) . Nonetheless , they mainly attribute SimSiam success to the stop gradient with the interpretation that AO might make it difficult to approach a constant ∀x . Interpreting predictor as EOA . The AO problem ( Chen & He , 2021 ) is formulated independent of predictor h , for which they believe that the usage of predictor h is related to approximating EOA for filling the gap of ignoring ET [ · ] in a sub-problem of AO . The approximation of ET [ · ] is summarized in Appendix A.2 . Chen & He support their interpretation by proof-of-concept experiments . Specifically , they show that updating ηx with a moving-average ηtx ← m ∗ ηtx + ( 1 −m ) ∗ Fθt ( T ′ ( x ) ) can help prevent collapse without predictor ( see Fig . 1 ( b ) ) . Given that the training completely fails when the predictor and moving average are both removed , at first sight , their reasoning seems valid . 2.2 DOES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA ? . Reasoning flaw . Considering the stop gradient , we divide the framework into two sub-models with different paths and term them Gradient Path ( GP ) and Stop Gradient Path ( SGP ) . For SimSiam , only the sub-model with GP includes the predictor ( see Fig . 1 ( a ) ) . We point out that their reasoning flaw of predictor analysis lies in the reverse of GP and SGP . By default , the moving-average sub-model , as shown in Fig . 1 ( b ) , is on the same side as SGP . Note that Fig . 1 ( b ) is conceptually similar to Fig . 1 ( c ) instead of Fig . 1 ( a ) . It is worth mentioning that the Mirror SimSiam in Fig . 1 ( c ) is what stop gradient in the original SimSiam avoids . Therefore , it is problematic to perceive h as EOA . New Figure 1 Explicit EOA does not prevent collapse . ( Chen & He , 2021 ) points out that “ in practice , it would be unrealistic to actually compute the expectation ET [ · ] . But it may be possible for a neural network ( e.g. , the preditor h ) to learn to predict the expectation , while the sampling of T is implicitly distributed across multiple epochs. ” If implicitly sampling across multiple epochs is beneficial , explicitly sampling sufficient large N augmentations in a batch with the latest model would be more beneficial to approximate ET [ · ] . However , Table 1 shows that the collapse still occurs and suggests that the equivalence between predictor and EOA does not hold . 2.3 ASYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN SIMSIAM . Symmetric Predictor does not prevent collapse . The difference between Naive Siamese and Simsiam lies in whether the gradient in backward propagation flows through a predictor , however , we show that this propagation helps avoid collapse only when the predictor is not included in the SGP path . With h being trained the same as Eq 2 , we optimize the encoder f through replacing the Z in Eq 2 with P . The results in Table . 2 show that it still leads to collapse . Actually , this is well expected by perceiving h to be part of the new encoder F , i.e . p = F ( x ) = h ( f ( x ) ) . In other words , the symmetric architectures with and without predictor h both lead to collapse . Predictor with stop gradient is asymmetric . Clearly , how SimSiam avoids collapse lies in its asymmetric architecture , i.e . one path with h and the other without h. Under this asymmetric architecture , the role of stop gradient is to only allow the path with predictor to be optimized with the encoder output as the target , not vice versa . In other words , the SimSiam avoids collapse by excluding Mirror SimSiam ( Fig . 1 ( c ) ) which has a loss ( mirror-like Eq 2 ) asLMirror = − ( Pa ·Zb+Pb ·Za ) , where stop gradient is put on the input of h , i.e . pa = h ( sg [ za ] ) and pb = h ( sg [ zb ] ) . Predictor vs. inverse predictor . We interpret h as a function mapping from z to p , and introduce a conceptual inverse mapping h−1 , i.e . z = h−1 ( p ) . Here , as shown in Table 2 , SimSiam with symmetric predictor ( Fig . 2 ( b ) ) leads to collapse , while SimSiam ( Fig . 1 ( a ) ) avoids collapse . With the conceptual h−1 , we interpret Fig . 1 ( a ) the same as Fig . 2 ( c ) which differs from Fig . 2 ( b ) via changing the optimization target from pb to zb , i.e . zb = h−1 ( pb ) . This interpretation suggests that the collapse can be avoided by processing the optimization target with h−1 . By contrast , Fig . 1 ( c ) and Fig . 2 ( a ) both lead to collapse , suggesting that processing the optimization target with h is not beneficial for preventing collapse . Overall , asymmetry alone does not guarantee collapse avoidance , which requires the optimization target to be processed by h−1 not h. Trainable inverse predictor and its implication on EOA . In the above , we propose a conceptual inverse predictor h−1 in Fig . 2 ( c ) , however , it remains yet unknown whether such an inverse predictor is experimentally trainable . A detailed setup for this investigation is reported in Appendix A.5 . The results in Fig . 3 show that a learnable h−1 leads to slightly inferior performance , which is expected because h−1 can not make the trainable inverse predictor output z∗b completely the same as zb . Note that it would be equivalent to SimSiam if z∗b = zb . Despite a slight performance drop , the results confirm that h−1 is trainable . The fact that h−1 is trainable provides additional evidence that the role h plays in SimSiam is not EOA because theoretically h−1 can not restore a random augmentation T ′ from an expectation p , where p = h ( z ) = ET [ Fθt ( T ( x ) ) ] .

Summary: This paper proposes a framework to understand why SimSiam avoid collapse without negative samples? It provides a hidden flaw of the Alternating Optimization for explain why SimSiam works. And the authors claim that  the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect.