Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: As the title suggests, the paper does a detailed investigation of how SimSiam avoids collapse without negative training examples. The key idea is to decompose the original vector into a center vector component and a residual vector component. The center vector cannot be too large (otherwise it is indicating collapse). The high-level idea is that the designs in SimSiam (and contrastive frameworks that have InfoNCE loss)  are mainly to prevent the center vector from getting too large. There are conjectures (verified empirically) about the relationship between the gradient w.r.t. the center vector ands the gradient w.r.t. the residual vector, and with these the paper finds that for SimSiam, the predictor is important for preventing collapse, particularly by doing de-correlation among features.
main_review: + The topic of the research is of significance, as it is important to understand why designs like SimSiam/BYOL does not collapse. The paper attempts to get more insights of it with empirical evidence, and also shows the potential of connecting to contrastive methods (via InfoNCE loss). This is a great step toward better understanding of existing methods.
+ I like the style of providing conjectures and show results that empirically verifying them. The experimental designs are quite solid and insightful.

- Paper writing: I am a bit lost when reading the second part of the paper about the center vector and residual ones. I think it is good to have a clear definition for X in "centering w.r.t. X". Is X the entire image set and all possible augmentations? Is X just for the current image? Or is X for the current batch? Also I think this center vector must be approximate (e.g., taking multiple crops to get the average), not fully exact. So it would be good to have some notational clarifications on this. Particularly when introducing the analysis on negative examples and predictor, the notation becomes quite messy and hard to follow.
- About Fig 1 (c): if the stop gradient is applied this way, then the predictor is NOT going to be trained at all and would naturally leads to collapse to me. I think the better way is to have the stop gradient between the predictor and the encoder (so at the place of z_b). At least this way all the parameters in the framework are being trained. I do think even this placement will lead to collapse.
- For the part that revisits SimSiam's explanation: while it is great to have this part, to me it is not the essential part of the paper (the second part is). So I think it would be good to make the second part more self-contained, with richer set of experiments, and move the SimSiam explanation revisit part to appendix.
- Minor for the text that leads to Eq (2): I think symmetric loss is not the essential reason why SimSiam does not collapse, so it would be good to just focus on the asymmetric architecture (like SimSiam's teaser figure) and loss for the current analysis. It would also reduce the confusion.
- Experimentation wise, it is good that the current paper provides a good explanation, but performance wise the theoretical insights have not led to better results. It is less important for a paper like this, but calling the last section "SimSiam++" but without numbers/speeds that actually outperform SimSiam is strange to me. 
summary_of_the_review: Overall I think this is a paper worthy of publication, with 1) its additional experiments to find flaws in the original hypothesis of SimSiam; and 2) explanations by decomposing the vector into two parts. However, I do have some concerns about the writing of the paper, so it would be good if the paper could have a higher quality (with proof-reading) and better organization so that it presents a more clear, unified message.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes another explanation for why SimSiam can avoid collapse without negative samples. Specifically, the paper decomposes the gradient of learned representation as center vector and residual vector and finds that the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect. Such an explanation can also be applied to Info-NCE, which unifies the theory of self-supervised learning with and without negative samples.
main_review: Pros:

1. The paper investigates the effects of center vectors and residual vectors in detail for both SimSiam and Info-NCE, which provides a unified explanation.

2. The results of SimSiam++ which shows a simple bias as a predictor can also avoid collapse without negative samples are interesting.


Cons:
 0. The writing of the paper needs to be improved. Many concepts are not explained or defined very clearly.

 1. The original SimSiam paper only claimed that "The usage of h may fill this gap (of missing EOA)." I think it's clear that the predictor does not learn to approximate the EOA. So I don't think the paper's interpretation of SimSiam's explanation is correct.

 2. In section 2.2, the paper claims that explicit EOA does not prevent collapse. But the experimental details are not explained very clearly here. I'm wondering whether the paper still uses one or two augmentations as the predictor's outputs or all the augmentations are used without stop-gradient.

 3. In section 2.3, the paper mentions "The results in Fig. 3(b)" show that it still leads to collapse. But I cannot find the collapsed results in Fig. 3(b). Besides, to prove Mirror SimSiam does not work (Fig 1. (c)), the authors should not apply stop-gradient to the predictor, because it's clear in the original SimSiam paper that fixed init does not work for the predictor. One possible way is to apply the gradient on z_a and p_b, and apply stop-gradient on z_b.

 4. In section 3.1, the paper mentions "Note that Z is l2-normalized, thus the trend of mo and mr is expected to be opposite of each other." This does not always hold.


Possible typos:

Section 2.3: Fig1 (c) and Fig2 (a) both lead to success ==> Fig1 (c) and Fig2 (a) both lead to failure

Section 3.1: loss - Z_a * sg(o_z) and loss - Z_a * sg(Z_a - o_z) ==> loss - Z_a \cdot sg(o_z) and loss - Z_a \cdot sg(Z_a - o_z)
Figure 3: - Z_a \cdot sg(Z_b - E(Z_b) ==> - Z_a \cdot sg(Z_b - E(Z_b))

Section 3.2: Z_n and r_n are used without definitions, which I guess means the representation for the negative examples.

summary_of_the_review: Overall, I think the idea of analyzing the extra gradient component is novel. The writing quality of the paper is not very good.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework to understand why SimSiam avoid collapse without negative samples? It provides a hidden flaw of the Alternating Optimization for explain why SimSiam works. And the authors claim that  the center vector gradient has the de-centering effect and the residual gradient vector has the de-correlation effect.
main_review: + It's interesting to see a framework to unified understand SSLs such as SimSiam, MoCo, SimCLR, etc.
+ The hidden flaw in AO of SimSiam seems to be correct, which is interesting.
--------------------
- The paper lacks experimental results/details to demonstrate their statements. 

a. In subsec. "Symmetric Predictor does not prevent collapse", the authors state "The results in Fig. 3 (b) show that it still leads to collapse" which is related to symmetric predictors in SimSiam. However, Fig. 3 (b) is actually about the basic SimSiam and SimSiam + Inverse predictor. 

b. in subsec. "Predictor with stop gradient is asymmetric", the authors "the SimSiam avoids collapse by excluding Mirror SimSiam (Fig1 (c)) which has a loss (mirror-like Eq 2) as shown as eq. 2". There is no experimental evidences to show if the mirror SimSiam will lead to collapse. If experimentally mirror SimSiam works, the statement does not hold.

c. How did you design the inverse predictor? Can we just see it as the new predictor while the previous predictor is included in the projector part?

d. in subsec. "Predictor vs. inverse predictor", "we interpret Fig2 (b) differs from Fig1 (a) as changing the optimized target from p to z, i.e. h
−1(p), suggesting processingthe optimized target with h−1 helps prevent collapse." Again, no experimental evidence.

e. in subsec. "Trainable h−1 and its implication on EOA", "we optimize h−1 by optimizing the pa approaching z∗b while simultaneously optimizing z∗b to zb via cosine loss, where z∗b is the h−1 output. The results proves that the model with h−1 (Fig2 (c)) is equivalent to
SimSiam since it achieves comparable performance as the original SimSiam that directly optimizes pa approaching zb.". Where are the "results"?

- The paper might have flaws in their proposed math and sometimes it's hard to follow 3.1, 3.3, and 3.4 

In subsec "Competition between o and r.", why "mo and mr is expected to be opposite of each other". The denominators of the two terms are both ||z||. So only ||o_z|| and ||r_a|| define the values. However, r_a = Z_a − o_z. Thus r_a and o_z can have the same norms but with different directions? Why they need to be on the opposite directions?



summary_of_the_review: I think the paper needs to improve descriptions about their framework and provide more experimental evidences.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes how the self-supervised learning (SSL) approach SimSiam avoids collapsed representations without explicit formulation of repulsive sample relations. To this end, first flaws in the original reasoning of the SimSiam paper are revealed. Next, based on center-residual vector decomposition, the role of the prediction head for preventing representation collapse in SimSiam is analyzed. Results indicate the importance of de-centralization and de-correlation as driving concepts for stable SSL.
main_review: #### Strengths:
+ Recent successful approaches to SSL indicate that stable learning can be achieved without explicit incorporation of repulsive, negative image relations. Indeed, the underlying reasons are theoretically poorly understood and typically only an inuitive understanding presented. Hence, given the importance of contrastive learnig today, research in this direction is important.
+ The analysis in section 2 seems sound and raises questions about the original insights of the SimSiam work regarding representation collapse. 

#### Weaknesses:
- The presentation and outline of arguments, as well as empirical discussions especially in section 3.3, 3.4 and 3.5 are sometimes hard to follow.
- The need of ‘de-centeralization’ (i.e. pushing samples apart in the embedding space) in gradient signals of SSL for stable learning seems straight-forward and inuitively trivial and does not seem novel.
- The paper only empirically shows that some kind of repulsion (i.e. de-centralization) is implicitly happening in the SimSiam framework. However, no theoretical and clear explanation of why and how is presented.  The presented conjectures are insufficiently back-uped and proven beyond the intuition of implicit repulsion. The paper e.g. states “Since the original predictor involves a nonlinear MLP, it is hard to understand how the predictor actually achieves it” (Sec. 4). Thus, no real, novel insights about the learning mechanisms of SSL without negatives are provided which are required for a more complete understanding of the addressed problem.
- The conceptual explanation of de-correlation in Info-NCE is hard is fuzzy and does not sound convincing. A more solid derivation and formulation of the hypothesis is needed.  

#### Questions:
- In the paragraph ‘Predictor with stop-gradient is asymmetric’ the paper states that the model setup shown in Fig. 2 (a) acutally results in successful, stable learning (“Similarly, […] Fig2 (a) […] leads to success, [...]”) It seems no prediction head is used in this case. Does the stop-gradient operation itself already prevent representation collapse?
summary_of_the_review: While the addressed phenomenon of preventing collapsing representations without the usage of negative samples is interesting, the presented analyis provides insufficient novel insights about how collapse is prevented. Although some novel framework based on vector decomposition is presented, the reasoning is often fuzzy, not convincing and lacks backup by clear theoretical derivations. Thus, I vote to reject this submission in its present form.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This method aims to explore why a minimalist simple Siamese (SimSiam) method can avoid collapse. After refuting their claims, the authors introduce vector decomposition for analyzing the collapse based on the gradient analysis of l2 normalized vector, yielding a unified perspective on how negative samples and SimSiam predictor alleviate collapse.
main_review: This paper is generally well-written and well-structured. It is important and inspiring to explore the reason why SimSiam can avoid collapse. The authors provide a convincing explanation and reach a unified conclusion for the recent progress in SSL, which is very insightful. The theoretical analysis and experimental results are solid. 
summary_of_the_review: Based on the abovementioned strength of this paper and the interesting conclusion, I recommend to accept this paper.
review_title: null

Meta-review: This paper provide an explanation why contrastive learning methods like SimSiam avoid collapse without negative samples. As the authors claimed, this is indeed a timely work for understanding the recent success in self-supervised learning (SSL). The key idea in this submission is to decomposes the gradient into a center vector and residual vector which respectively correspond to de-centering and de-correlation. Such an explanation is interesting and novel. The empirical results are solid and convincing. During the rebuttal stage, the concerns from the reviewers are well resolved, and the writing of the new version is significantly better than the original one.
Decision: Accept (Poster)