Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a new non-contrastive learning algorithm, with a additional variance preserving term in the objective, to get rid of the stop-gradient operation commonly used in non-contrastive learning. They provides some theoretical intuition for their method and empirical comparisons between different ways of preventing representation collapses.
strengths_and_weaknesses: Strengths:
The empirical comparison is thorough and the ablations are very informative of the effects of each components in the method. They have a (although flawed) theory to back their choice of the new regularizer.

Weaknesses:
1. The major weakness is that the theoretical derivations are not well explained, and do not provide any insights into their proposed method. The results in section 3.2, especially the later comments in equation (14) and line 132-133 only says that with iteration $t$ tends to infinity, the matrix $W_pW_p^{\top}$ will approach $1/n\sum_{i}f_if_i^{\top} $, which can still be a collapsed solution (for example when all $f_i$ are the same and $W_p$ is rank-1). The technical derivations in section 3.2 do not answer why this dynamics (the convergence of $W_pW_p^{\top}$) is needed for the network to avoid collapse, and why it can be utilized to help designing a new covariance regularizer.
2. The transition between equation (16) and (17) is unclear and not well-explained. The transition between (16) and (17) is muddled through with unjustified analogy. This weakens the validity of the explanation in section 3.2 on their final loss form (17) and (19).
questions: None
limitations: The major limitation of this paper is in its flawed explanation of the proposed method, which loosed its validity because (1) the unanswered reason why the theorem in section 3.2 is important for the non-contrastive learning process (with stop-gradient), and (2) why the objective in equation (16) is equivalent to that in (17), or can be motivated by the same theory behind (16).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper follows the theoretical framework proposed by Tian, Y. et al in [14], and further shown that the asymmetry in contrastive learning would lead to a feature decorrelation. As a demonstration (and an empirical proposal), the author demonstrated that the stop-gradient in contrastive learning (BYOL/ SimSiam) can be eliminated by explicitly imposing the derived constraint with additional feature standardization. The empirical results majorly supports the claim and theory of the paper. Eventually a better performance is reported with the proposed method on ImageNet (contrastive learning with linear prob).
strengths_and_weaknesses: The paper pinpointed that the a small set of asymmetry tricks (stop gradient, etc) employed by BYOL/SimSiam have an additional implicit effect of feature decorrelation. More importantly, one of the major trick of stop gradient can be removed and replaced with the constraint as an explicit regularizer. This finding provides a better understanding of the working mechanism of the asymmetry tricks.

The formula derivation is sound.

The ablation in the paper is compact and clear. 

The paper is in generally fairly elaborated in smooth writing.

The paper reports some meaningful datapoints to supported the claim. In table 3, the ablation prove that the stop grad can be eliminated and replaced with INV + STD (or with I-PRED) to keep the network train without collapse. Consistent datapoints supports the claim that COV helps the network converges better, outperform the counterpart without COV constraint. It is also very convincing to see empirical experiments (with no-augmented image) demonstrated the theory of the correlation constraint.




questions: Could the author share additional data points on the magnitude comparison between the two losses.

Could the author elaborate what are the bn (or bn without learnable parameters) introduced in the network? Do we have additional bn before the standardization. Do we perform sync for the standardization.





limitations: The standardization is not introduced/ discussed in the derivative derivation process, which might lead to different conclusion on the decorrelation part. The standardization is only introduced as an empirical trick, which weak the soundness of the paper and make the theory and finding less convincing.

The behavior of partition is completely not explained in the paper. The theory part does not introduce anything would lead to partition, which further weak the soundness of the paper. 

The partition behavior (2048 / 32 = 64 per batch) was originally introduced in the paper of "On the Importance of Asymmetry for Siamese Representation Learning", the author did not discuss or cite the paper.

Stop grad or mmt encoder are not the only "asymmetric tricks", there are other asymmetry tricks used in contrastive learning field (for example, multi crop, scalemix, Asym Aug, Mean Encoder and Sync BN, etc. It would make the paper less precise and overclaim if the terminology of "asymmetric tricks" were not well/ better defined.


Minor issues

In the Pseudo-code, the function of standardization was named as bn which can be misleading.

The Pseudo-code is over complicated, it is not necessary to introduce a hyper parameter of GPU. More importantly the num of partition are not necessarily num of gpu.

line 171, the writing is not unfinished.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the role of asymmetric “tricks” (like predictor layer and stop-gradient) to prevent feature collapse in *non-contrastive methods* like BYOL/SimSiam. Extending prior theoretical results on a linear representation model [Tian et al., 2021], the paper argues that these tricks implicitly enforce “decorrelation” of features of different inputs, similar to other non-contrastive methods like Barlow-Twins and VICRig, thus establishing a connection between these two paradigms. Based on this analysis, the paper also proposes a non-contrastive method that uses a linear prediction head, explicitly enforces the decorrelation-like constraint that is derived, and does not use stop-gradient. Experiments suggest that the proposed method performs similarly (or better) than all of the aforementioned non-contrastive method on ImageNet linear probe evaluation.
strengths_and_weaknesses: **Strengths**

(S1) The paper is well motivated and studies a relevant question about why non-contrastive methods perform well and avoid trivial solutions that collapsing the representations

(S2) Aims to bridge two sets of ideas to avoid collapse: asymmetric tricks and feature decorrelation

(S3) Paper is clearly written and easy to follow for the most part

(S4) Extensive experiments are conducted and useful ablation studies are reported. The rough conclusion is that the proposed regularization term (along with the standardization) is competitive with previous non-contrastive methods and is a better alternative to stop-gradient.




**Weaknesses**

My main concern with the paper is about the connetion to feature decorrelation in section 3.3 and the modification to the proposed regularization in eq 17, both of which are crucial points.

(W1) Section 3.3 is the main (probably only) part that supports the paper’s claim about connection between asymmetric tricks and feature decorrelation. However I found this section to be “hand-wavy” and imprecise. To elaborate, the theoretical result suggests that feature correlation for stop-gradient dynamics will converge to $WW^{\top}$ and if the predictor $W$ converges to $I$, then feature correlation will also converge to $I$ as desired. While the paper makes an interesting empirical observation that $W$ converges to $I$ in practice for many methods, the theoretical justification seems weak as currently presented. It suggests that explanation for $W \rightarrow I$ follows from Tian et al.’s results, but it is not clear how exactly it follows.
- L139: “each eigenvalue of $W_p$ will be either 0 or a  positive value $\lambda^*$ at its stable convergence/equilibrium state”. Why is this true? Which result of Tian et al. suggests this?
- L143: “If some of its eigenvalues are 0, $W_p$ is close to a diagonal matrix”. This is not necessarily true. If some eigenvalues are 0 then $W_p$ will be low rank and how close to diagonal it is will depend on the span of the non-zero eigenvectors of $W_p$. 

Overall I find the connection to feature decorrelation unconvincing until more explanation is provided. What helps the discussion is the empirical finding that $W$ converges to identity.




(W2)  Modification to regularization: The modification from $|W^{\top} W - F F^{\top}|^2$ to $|W^{\top} W - W F F^{\top} W^{\top}|^2$ seems benign, but it is not clear how benign it is. Although the objective does not change a lot with this when the first and second terms are sufficiently small, since non-contrastive is all about the dynamics and not the objective, this could potentially play a big role. Furthermore if $W$ is assumed to be symmetric, then $|W^{\top} W - W F F^{\top} W|^2 = |W (I - F F^{\top}) W|^2$ which almost directly enforces feature decorrelation for a full rank $W$. It would be good to get a clarification about this point from the authors and it might also help to report experiment results when using the original regularization $|W^{\top} W - F F^{\top}|^2$.



Given these issues, I am currently leaning towards a **weak reject**. Some other minor issues and questions are listed below.
questions: Other minor points

- Missing references: [1, 2, 3] either theoretically or empirically study non-contrastive learning (although [3] is very recent). [4] theoretically studies non-contrastive method (SimSiam) with feature normalization and connects it to non-linear CCA, providing a justification for feature decorrelation beyond an information theoretic one.
- Theorem 3.1 looks similar to Theorem 1 from reference [14] from the paper. It would help to compare and discuss the differences from it. The proof of Theorem 3.1, especially the jumps from eq. 11-13 can be written more clearly. Also useful to statement the assumption that F is linear in the input in the theorem and lemma statements.
- In the setting from Theorem 3.1, what happens in the dynamics if there is no stop-gradient? Will $W$ not converge to $I$ and there will be no feature correlation?
- L170-171: Editing issue


References

[1] X. Wang, X. Chen, S. Du, Y.Tian, Towards Demystifying Representation Learning with Non-contrastive Self-supervision. 2021

[2] X. Wang, H. Fan, Y. Tian, D. Kihara, X. Chen. On the Importance of Asymmetry for Siamese Representation Learning. CVPR 2022

[3] Z. Wen, Y. Li. The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. 2022

[4] J. Lee, Q. Lei, N. Saunshi, J. Zhou. Predicting What You Already Know Helps: Provable Self-Supervised Learning. NeurIPS 2021
limitations: Some limitations are discussed. No societal impact discussed; I do not forsee any immediate negative impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the dynamics behind non-contrastive self-supervised learning methods. It shows that the stop-grad approach (e.g., BYOL, SimSiam) is effectively also doing feature decorrelation, similar to BarlowTwins, VICReg. The authors start with Tian et al. (arXiv:2102.06810)'s results and show that the stop-grad will satisfy W(t)^TW(t) = F(t)F(t)^T at large step t. The authors propose a novel method that explicitly uses this term as a regularizer. This can be seen as a soft version of BarlowTwins that require decorrelation but is not necessarily full rank. Experiment results (100ep ImageNet pretraining, linear probe) show that the proposed method outperforms BarlowTwins and VICReg.
strengths_and_weaknesses: Strengths:
1. The proposed method is well motivated by the theory, which is an extension of Tian's (arXiv:2102.06810) results.
2. The ablation study is convincing. It proves the effectiveness of the proposed regularizer term. 
3. The authors provided all experimental details, including pseudo-code and hyperparameters tuning.

Weaknesses
1. The authors claim that they extend the theory. However, most of the equations are already known in the literature. For example, any linear system under SGD will satisfy eq14; see arXiv:1905.13655, arXiv:2110.09348, arXiv:2201.12680. Therefore, I don't find Section 3.2 supports the claim that "we theoretically show that the use of stop-gradient and a projector implicitly provides a constraint leading to similar feature decorrelation."
2. The primary connection between the theory and the proposed method is that W^TW becomes a diagonal matrix and will not collapse to a low rank. The authors claim that they empirically observe this happens for BYOL, as shown in supplementary material section C. However, there is no study on the dynamics of this term in the proposed method. The authors need to show that this term follows similar behavior.
3. The authors did an extensive study on the projector design. With various projector/predictor dimension settings, the proposed method demonstrates superior performances over BarlowTwins and VICReg. However, this study is less significant considering the main claim. The projector design is hyperparameter tuning. It would be better to provide a clear comparison between optimal settings.
questions: 1. What's the consideration for using a different projector in BarlowTwins and VICReg? The hyperparameters in these methods are optimized for the original projector size. In fact, the accuracy could be much higher if a different set of hyperparameters were used for such a projector head. I   suggest not to claim the advantage of a proposed method by beating a weakened baseline.

2. What's the consideration of "partition"? This is not discussed in the theory part or the motivation. How to understand this term?
limitations: N/A
review_title: null

Meta-review: metareview: The paper connects non-contrastive SSL methods (such as BYOL/SimSiam) with feature de-correlation approaches (e.g., Barlow-Twins and VICReg) via theoretical analysis and empirical validations. The paper is well-motivated and the problem to be addressed is important for the community. The experiments seem to be extensive. 

There are thorough discussions between authors and reviewers. Reviewers now agree that the paper can be accepted, while the supports remain limited.  

There are certain issues that AC strongly advises the authors to address in their camera ready version. One example is to make sure the authors clearly explain which part is proved and which part is observed empirically (e.g., $W_p^\top W_p \rightarrow I$). For empirical observation, please clearly state the experimental setup. AC suspects that this empirical observation ($W_p^\top W_p \rightarrow I$) heavily depends on whether the underlying feature $F$ has been standardized and the high-dimensional output of the projector (i.e., vectors tend to be orthogonal in high-dimensional space). 
confidence: Certain
Decision: Accept