Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: *Summary*
The paper provides a new perspective on the BYOL self-supervised learning method. First, the paper introduces an upper-bound objective, BYOL', that is easier to analyze than BYOL because it is composed of two well understood losses: an alignment loss and cross-model loss. Further, it shows empirically that optimizing BYOL' is similar to optimizing BYOL. Second, the paper introduces the RAFT method which maximizes the alignment loss instead of minimizing it. The paper proves that under some assumptions, such as a linear predictor function, optimizing BYOL' is equivalent to RAFT. Based on this analysis, the paper explains why the predictor function is essential for BYOL and why it is hard to achieve convergence.

*Quality*
I really like the analysis of the paper. The paper provides a mix of theoretical and empirical argument for understanding BYOL, and introduces a new method called RAFT. The main drawback of the paper is that it limits the empirical analysis to a single and much simpler experimental setup using CIFAR10 and resnet18. I believe that since BYOL's significance is an empirical one and is mainly established on Imagenet, any empirical analysis of BYOL in other simpler settings is quite limited.

*Clarity*
The authors have done a very good job in writing this paper. The logic, presentation and results are quite clear to understand.

*Originality*
I find the paper quite interesting and original in its analysis. I especially like the analysing BYOL through the BYOL' upper-bound.

*Significance*
I think the results of the paper could have been quite more significant if applied on other experimental setups. While I understand working with SOTA models can be computationally expensive, the main argument of this line of work is empirical and it is hard to be convincing without more extensive empirical results.

review_title: Official Blind Review

final_score: rating: 3: Clear rejection
review_content: **Summary**: the paper aims to explain the success of BYOL, a recently proposed contrastive method that mysteriously avoids the trivial constant solution without requiring negative samples. The paper proposes a new loss named RAFT. Compared to BYOL, RAFT is more general since it subsumes a variation of BYOL as its special case, and contains a cross-model term to be maximized which regularizes the alignment loss and encourages the online encoder to "run away" from the mean teacher.

The paper claims this cross-model term encourages disparity, which could help demystify why BYOL does not collapse to a trivial solution. However, the cross-model term itself cannot prevent outputs from collapsing, as explained below.

**Question 1**: my main concern is the effectiveness of the cross-model loss: I disagree that the cross-model loss prevents collapsing representations. I think the authors may be confusing contrasting two samples ("cross-sample") and contrasting two functions ("cross-model"):
- The cross-model loss is essentially the L2 distance between two functions, which is the average squared error between two model outputs on the *same sample.*
- The common contrastive loss, which contrasts outputs from the same model on *different samples*.

For example, suppose MT is a constant function at the $t_{th}$ iteration (i.e. the function outputs some constant $c$ for all input), then the online encoder could be updated to be another constant as far away from $c$ as possible, i.e. the cross-model loss is maximized, however we still have the sample collapsing issue. As a side note, a constant function also achieves a perfect alignment loss.
More concretely, consider $f(x) = Wx +b$ where $W$ is initialized to be the all-0 matrix, i.e. $f(x) = b$ is a constant function. Then for all future updates, learning $f$ only updates $b$ but not $W$ (since there's no gradient on $W$), and therefore $f$ will remain a constant function, i.e. it always collapses the points. One may argue that it is wrong to choose $W = 0$, but the point is, the success of BYOL needs more careful analysis of the optimization process, which cannot be addressed by the cross-model loss term itself.

**Question 2**: section 3 phrases the need of a predictor as a disadvantage of BYOL, however RAFT also requires a predictor head to achieve good classification performance. Studying the effect of the predictor is an interesting direction and will make the paper much stronger, as the authors also point out in the conclusion.

**Other comments**:
- Table 3: why are there no results for BYOL'-MLP? Comparing RAFT-NP to BYOL'-NP, there doesn't seem to be a clear edge of RAFT, both in terms of the uniformity loss and the accuracy.
It would also be better to highlight the key results in the table. The current table is quite dense; adding more highlights and comments will have the reader understand what to take away from these results.
- Paper organization: a lot of material is deferred to the appendix, which makes the paper a bit hard to follow since the reader needs to jump back and forth. It would be better if results in the appendix are better summarized in the main text.
- The term BYOL' first appears at the end of the first paragraph on page 2 without a definition.
- Minor typo: there's an extra left parentheses in front of the second $f$ in equation (5); an extra comma after "distribution" in the first paragraph of section 3.
review_title: Studying an interesting problem, however the main claim is erroneous

final_score: rating: 3: Clear rejection
review_content: This paper analyses the recently proposed Bootstrap Your Own Latent (BYOL) algorithm for self-supervised learning and image representation. 
The authors first derive an alternative training procedure called BYOL' by computing an upper bound of the BYOL objective function.
After diverse analyses, the authors then introduce Run Away From Your Teacher (RAFT), where RAFT is another BYOL variant that resembles contrastive method by having an attractive and repealing term in the training objective. According to the authors, this decomposition allows for a better understanding of the training dynamics.

Finally, the authors made the following transitivity reasoning:
 - BYOL and BYOL' are almost equivalent
 - RAFT and BYOL' are shown to be equivalent under some assumptions.
Thus, conclusions that are drawn from analyzing RAFT should still hold while analyzing BYOL. They thus link the interest of BYOL's predictor and the EMA through the RAFT loss decomposition. 

I have multiple strong concerns regarding this paper. These concerns are both on the paper results, shortcuts in the analysis, and the writing style. 


Results:
--------------

 - In section 4, the authors introduce BYOL' as a variant of BYOL. To do so, they derive an upper bound on the BYOL loss, i.e. the L2 distance between the projection and the projector, and they try to minimize it. However, this approach disregards that BYOL does not minimize a loss (due to the stop gradient). In other words, the BYOL objective keeps evolving during training; the target distribution is non-stationary.  As mentioned in the BYOL paper: "Similar to GANs, where there is no loss that is jointly minimized w.r.t. both the discriminator and generator parameters; there is therefore no a priori reason why BYOL’s parameters would
converge to a minimum of L_BYOL given the online and target parameters". Minimizing an upper-bound is at best insufficient, at worst a non-sense. The sentences, "minimizing L_{BYOL'}  would yield similar performance as minimizing L_{BYOL}" and "we conclude that optimizing L_{BYOL'} is almost equivalent to L_{BYOL}" are unfortunately wrong.  This is somewhat highlighted different qualitative results in Appendix F.1.b != F.1.d.
A better approach would be to ensure that the *gradients* go in a similar direction (so the training dynamics are similar rather than the objective function). However, even such a demonstration could be insufficient due to compounding factors in the training dynamics. 
 - The 1-1 mapping between BYOL' and RAFT rely on three hypotheses. While (i) and (ii) are reasonable, hypothesis (iii) is quite strong, and more importantly, neither elaborated nor discussed. In other words, I am unable to validate/invalidate the interest of the theoretical results. Would it be possible to measure the normal gradient empirically? To bound it? 
 - In section 3, i would recommend the author to mention that multiple components were also in the BYOL paper; especially when writing "therefore, we conclude the predictor is essential to the collapse prevention of BYOL."
 - Although I acknowledge that self-supervised learning requires heavy computational requirement, and few teams may run experiments on ImageNet. Yet, I would recommend the authors to not use CIFAR10 as the dataset has multiple known issues (few classes, small images, few discriminative features). Other variants such at STL or ImageNete can be trained on a single GPU over a day, and are less prone to misinterpretation in the results. Besides, I want to point out that BYOL was not correctly tuned: the experiments are based on a different optimizer (Adam vs LARS) and no cosine decay were used for the EMA, while these two components seem to be critical, as mentioned in BYOL and arxiv:2010.1024. 

Overall, I have a serious concern about the paper's core contributions. However, there are still some good elements in the paper that I think are under-exploited:
 - RAFT is itself an original, new and interesting algorithm. The potential link to BYOL is indeed an interesting lead, but in its current state, I would make it a discussion more than a key contribution.
 - Table D.3 shows that RAFT/BYOL' does not collapse without predictors when \beta is high. Albeit providing low accuracy, a non-collapse is quite surprising. Unfortunately, the authors leave it for future work


Shortcuts:
--------------
I was surprised by multiple shortcuts in the reasoning process or undiscussed conclusions:
 - The authors mention that the predictor is a dissatisfactory property of BYOL. Could they elaborate? This is actual the key component of the method (if not the only one!), and such pro/cons could be detailed in light of other methods. 
 - In section 4.1, the authors mention that: similar accuracies and losses are sufficient somewhat confirm that BYOL and BYOL' are similar. Two completely different methods may have the same errors while being radically different... 
 - In Section 4.2, the authors mention that "Based on the form of BYOL, we conclude that MT is used to regularize the alignment loss". However, there is no experiments to try to contradict/validate this claim. Differently, the EMA may ease the optimization process or it may have different properties. Even if I understand the logic behind this statement, I regret that the authors do not try to confort it. 
- In section 4.2, the authors mention that there exist multiple works (while only citing one...) demonstrating that EMA is "roughly" equivalent to sample averaging and may encourage diversity. While this is sometimes true in specific settings (cf. markov game and fictitious play), this is also known to ease optimization (cf. target network in DQN). Stating that RAFT is better than BYOL because it better leverage the EMA target is tricky without proper analysis.
- Albeit understandable, the transitivity between BYOL and RAFT is difficult to defend due to multiple approximations and hypothesis. Therefore, it is of paramount importance that the approximations and hypothesis are validated, which is not sufficiently done in the paper. 


Writing: 
--------------
 - Although papers' writing quality remain subjective, I tend to expect a formal language. I kind of feel ill-at-ease when reading sentences including "BYOL works like a charm", "disclosing the mistery", "to go harsher", "bizarre phenomon". Other sentences also expresses judgement such as "inconsistent behavior", "dissatisfactory property of BYOL" or "has admirable property" without proper argumentation. 
 - It is non-trivial to follow the different version of the algorithms... which are defined in the appendix. Please consider renaming BYOL'. 
 - A related work section would have been useful to put in perspective BYOL that are theoretically motivated e.g. AMDIM, InfoMin, other self-supervised learning methods without negative example, e.g. DeepCluster, SwAV. Section 2 is more about the background, not related work.
 - there are a few confusions in the notation, \alpha \beta have different meaning across equations (Eq 7 vs 8)
 - In section 3, random is ill-defined. In Cifar10, random should be 10%, I assume that you refer to random projection. Please clarify.  
 - Figure 1 is clear, and I recommend to keep it as it is.
 - From my perspective, the mathematical explanation in Section 5 is quite obfuscated, and I would recommend a full rewriting.
 - Please avoid unnecessary taxonomy, e.g. uniformity optimizer, effective regulizers and others.
 - In conclusion, you mentioned some results about the projector. However, you never detail them in the paper. Please, do not discuss unpublished results.

Overall, I had difficulties following the paper: I keep alternating between the appendix, previous sections, and the text. Again, the phrasing makes me ill-at-ease.


Conclusion:
--------------------
I have some serious concerns about the core results of the paper. Importantly, Theorem 4.1 follows a misinterpretation of the BYOL training dynamics. From my perspective, there are too many unjustified claims, and I cannot recommend paper acceptance. However, there is some good idea in the paper, and I strongly encourage the authors to study RAFT independently of BYOL in the future.
review_title: RAFT

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper mainly proposes an objective that incorporates the target network in BYOL in a opposite way that encourages the prediction of the online network to be far away from the target network.

Concerns:

1.  It is overly claimed that "we unravel the puzzle of how BYOL avoids representation collapse". For example, the authors fail to capture some intrinsic properties of BYOL, such as the role of prediction head (MLP + BN).  The theorem 5.1 can only deal with the prediction head that is linear. The authors could refer to the [1] for more insights.

2. The experimental results do not support the claim that "RAFT is a conceptually non-collapsing algorithm". In table 1, for results equipped with q_{w}-MLP, RAPT with better acc (71.31) in fact does not have smaller uniform loss. Instead, its alignment loss is smaller (which brings the acc improvement). So, the RAFT does not actually always enlarge the uniformity as claimed.
 
3. The implementation of BYOL in this paper regarding Cifar10 is not convincing. [2] also use the resnet18 as the encoder and it achieves the accuracy with 91+ in Cifar10. However, the reproduced result in this paper is only around 70. 

4. BYOL proves its own effectiveness in ImageNet. To make fair comparisons, the authors shall conduct experiments in the same dataset. Otherwise, the claim regarding to the BYOL might not be solid.

5. It is acceptable that running away from the mean teacher increases the difficulty of alignment. But it is unclear to the reviewer why it can produce global uniformity in the representation space?

[1] Tian, Yuandong, et al. "Understanding Self-supervised Learning with Dual Deep Networks." arXiv preprint arXiv:2010.00578 (2020).

[2] Ermolov, Aleksandr, et al. "Whitening for Self-Supervised Representation Learning." arXiv preprint arXiv:2007.06346 (2020).
review_title: Review2

Meta-review: Most of the reviewers and AC found many claims of this submission unsubstantiated. 
Decision: Reject