Run Away From your Teacher: a New Self-Supervised Approach Solving the Puzzle of BYOL
Authors: Haizhou Shi
Dongliang Luo
Siliang Tang
Jian Wang
Yueting Zhuang
Venue:  (2021)
Keywords: representation learning
self-supervised learning
contrastive learning
regularization
theory

Abstract: Recently, a newly proposed self-supervised framework Bootstrap Your Own Latent (BYOL) seriously challenges the necessity of negative samples in contrastive-based learning frameworks. BYOL works like a charm despite the fact that it discards the negative samples completely and there is no measure to prevent collapse in its training objective. In this paper, we suggest understanding BYOL from the view of our newly proposed interpretable self-supervised learning framework, Run Away From your Teacher (RAFT). RAFT optimizes two objectives at the same time: (i) aligning two views of the same data to similar representations and (ii) running away from the model's Mean Teacher (MT, the exponential moving average of the history models) instead of BYOL's running towards it. The second term of RAFT explicitly prevents the representation collapse and thus makes RAFT a more conceptually reliable framework. We provide basic benchmarks of RAFT on CIFAR10 to validate the effectiveness of our method. Furthermore, we prove that BYOL is equivalent to RAFT under certain conditions, providing solid reasoning for BYOL's counter-intuitive success.

Under review as a conference paper at ICLR 2021
RUN AWAY FROM YOUR TEACHER: A NEW SELF- SUPERVISED APPROACH SOLVING THE PUZZLE OF BYOL
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, a newly proposed self-supervised framework Bootstrap Your Own La- tent (BYOL) seriously challenges the necessity of negative samples in contrastive learning frameworks. BYOL works like a charm despite the fact that discards the negative samples completely and there is training objective. In this paper, we suggest understanding B YOL from the view our newly proposed interpretable supe--upe learning framework, Run Away From your Teacher (RAFT). RAFT optimizes two objectives at the same time: views of the same data similar representations and (ii) run- ning away from the model's Mean Teacher (MT, the exponential moving average of the history models) instead of BYOL's running towards it. The second term of RAFT explicitly prevents the representation collapse and thus makes RAFT a more conceptually reliable framework. We provide basic benchmarks of RAFT on CIFAR10 10 to validate the effectiveness of our method. Furthermore, we prove that BYOL equivalent to RAFT under certain conditions, providing solid reasoning for BYOL's counter-intuitive success.
to prevent collapse in its
(i) aligning
INTRODUCTION
Recently the performance gap between self-supervised learning and supervised learning has been narrowed thanks to the development of contrastive learning (Chen et al. 2020b a Tian et al. 2019
Chen al. 2020b (ohnn 2016 Zhuang et al., 2019; He et al. 2020) Oord et al., 2018 Hadsell
et al.) 2006). Contrastive learning distinguishes positive pairs of data from the negative. It has been shown that when the representation space no-normalieed, i.e. a hypersphere, optimizing the contrastive loss is approximately equivalent to optimizing the alignment of positive pairs and the uniformity of the representation distribution at the same time (Wang & Isola 2020). This equivalence conforms to intuitive understanding. One easily imagine failed method when only optimize either of the properties: aligning the positive pairs without uniformity constraint causes representation collapse, mapping different data all to the same point; scattering the data uniformly in the representation space without aligning similar ones yields no more meaningful representation than random.
The proposal of Bootstrap Your Own Latent (B fiercely challenges our consensus that negative samples are necessary contrastive methods (Grill al. 2020). BYOL trains the model (online network) to predict its Mean Teacher (moving average of the online, refer to Appendix B.2 on two augmented views of the same data (Tarvainen & Valpola 2017). There is no explicit constraint on uniformity in BYOL,, while the expected collapse never happens, what's more, it reaches the SOTA performance on the downstream tasks. Although BYOL has been empirically proven to be an effective self-supervised learning approach, the mechanism that keeps it from collapse remains unrevealed. Without disclosing this mystery, it would be disturbing for us to adapt BYOL to other problems, let alone further improve it. Therefore solving the puzzle of BYOL is urgent task.
In this paper, we explain how BYOL works through another interpretable learning framework which leverages the MT in the exact opposite way. Based series of theoretical derivation and em- pirical approximation, we build new self-supervised learning framework, Run Away From your Teacher (RAFT), which optimizes two objectives at the time: (i) minimize the representation

---

Under review as a conference paper at ICLR 2021
distance between two samples from positive pair and (ii) maximize the representation distance between the online and its MT. The second objective of RAFT incorporates the MT in a way exactly opposite to BYOL, and it explicitly prevents the representation collapse by encouraging the online to be different from its history (Figure 2a). Moreover, we empirically show that the second objective of RAFT is more effective and consistent regularizer for the first objective, which makes RAFT more favorable than BYO,. Finally, we solve the puzzle of BYOL by theoretically proving that BYOL special form of RAFT when certain conditions and approximation hold. This proof explains why collapse does not happen in B BYO,, and also makes the performance of B BOLL' an approximate guarantee of the effectiveness of RAFT.
The main body of the paper is organized in the same order of how we explore the properties of BYOL and establish RAFT based on them (refer Appendix A for more details). In section 3 we investigate the phenomenon that BYOL fails to work when the predictor is removed. In section 4 we establish two meaningful objectives out of BYL by upper bounding. Based on that, we propose RAFT due to its stronger regularization effect and its accordance with our knowledge. In section 5 we prove that, as a representation learning framework, BYOL is a special form of RAFT under certain achievable conditions.
In summary, our contributions are listed as follows:
We present a new self-supervised learning framework RAFT that minimizes the alignment and maximizes the distance between the online network and its MT. The motivation of RAFT conforms our understanding of balancing alignment and uniformity of the repre- sentation space, and thus could be easily extended and adapted to future problems.
We equate two seemingly opposite ways of incorporating MT in contrastive methods under certain conditions. By doing so, unravel the puzzle of how B YOL avoids representation collapse.
2 BACKGROUND AND RELATED WORK
2.1 TWO METRICS OPTIMIZED IN CONTRASTIVE LEARNING
Optimizing contrastive learning objective has been empirically proven have positive correlations
with the downstream task performance (Chen et al. 2020b a Tian et al. 2019 Chen et al. 2020b
Sohn 2016 Zhuang et al., 2019 He et al., 2020 Oord al. 2018). Wang Isola (2220) puts the contrastive learning under the context of hypersphere and formally showcases that optimizing the contrastive loss (for preliminary of contrastive learning, refer to Appendix [. equivalent to opti- mizing metrics of the encoder network when the size of negative samples K is sufficiently large: the alignment of the two augmented views of the same data and the uniformity of the representation population. We introduce the alignment objective and uniformity objective follows.
Definition 2.1 (Alignment loss) The alignment loss fatign (J; Ppon) of the function f over positive- pair distribution Ppos. is defined as:
Lalign (f: Ppos) & E (( [1f(z1) - 1(22
where the positive pair (x1, are two augmented views of the same input data x ~ X, i.e. (I1, x22 (t1 (z), (z)) and t1 T1.t2 To are two augmentations. For the sake of simplicity, we omit Pos and use Caliga (f) in the following content.
Definition 2.2 (Uniformity loss) The loss of uniformity Lunitorm (f: X) of the encoder function f over data distribution X is defined
Luniform (f: A log B(aa))=0 [01115(4---001)))
(2)
where f > is a fixed parameter and empirically set to t = 2. To note here, the vectors in the representation space are automatically Ig-normalized, ie. f(r) P f(x)/ / [ 2. as we limit the representation space to hypersphere following Wang & Isola (2020) and Grill et al. (2020) and the representation vectors in the following context are also automatically L normallzeed, unless specified otherwise. Wang & Isola (220)) has empirically demonstrated that the balance of the alignment loss
2

---

Under review as a conference paper at ICLR 2021
and the uniformity loss is necessary when learning representations through contrastive method. The rationale behind it is straightforward: Lalign provides the motive power that concentrates the similar data, and Luniform prevents it from mapping all the data to the same meaningless point.
2.2 BYOL: BIZARRE ALTERNATIVE OF CONTRASTIVE
A recently proposed sule--upe representation learning algorithm BYOL hugely challenges the common understanding. that the alignment should be balanced by negative samples during the con- trastive learning. It establishes two networks, online and target, approaching to each other during training. The online is trained to predict the target's representations and the target is the Exponential Moving Average (EMA) of the parameters of the online. The loss of BYOL at every iteration could be written as
LBYOL A E(2.1.t)) [Ilo (((=((((())))))) fe fe(t2()))11
where two vectors in representation space are automatically (z-normalized. fo is the online encoder network parameterized by and qu is the predictor network parameterized by w. x X is the input sampled from the data distribution X' and t1 (x), t2(x) are two augmented views of where l ~ Ti, l2 To are two data augmentations. The target network fe is of the same architecture as So and is updated by EMA with controlling what degree the target network preserves its history
+ T + (1-
From the scheme of BYQL training, it seems like there is no constraint on the uniformity, and thus most frequently asked question about BYO is how prevents the representation collapse. Theoretically, we would expect that when the final convergence of the online and target is reached, CBYOL degenerates to Lalign and therefore causes representation collapse, while this speculation never happens in reality. Despite the perfect SOTA performance of BYOL, there is one inconsistency not to be neglected: it fails with representation collapse when the predictor is removed, which means % for any given This inconsistent behavior of B YOL weakens its reliability and further poses questions on future adaptation of the algorithm. The motivation of understanding and even solving this inconsistency is the start point of this paper.
ON-AND-OFF BYOL: FAILURE WITHOUT PREDICTOR
We start by presenting dissatisfactory property of BOL: its success heavily relies on the existence of the predictor qw- The experimental setup of this paper is listed in Appendix The performance of BYOL original model, whose predictor qu a two-layer MLP with batch normalization, evaluated on the linear evaluation protocol (Kolesnikov et al. 2019 Kornblith al.) 2019 Chen et al. 2020a
He et al. 2020 Grill et al., 2020) reaches 68.08 0.84%. When the predictor is removed, the performance degenerates 20.92 1. 1.29%, which is even lower than the random baseline's 's 42. 74 0.41%. We examine the speculation that the performance drop is caused by the representation collapse both visually (refer to Appendix [ET) and numerically. Inspired by Wang & Isola (2220) we use Luntomm(oo;
to evaluate to what degree the representations are spread on the hypersphere and Calign (qw o ff) to evaluate how the similar samples are aligned in the representation space. The results in Table that with the predictor, BYOL optimizes the uniformity of the representation distribution. On the contrary, when taken away the predictor, the alignment of two augmented views is overly optimized and the uniformity of the representation deteriorates (Figure therefore we conclude the predictor is essential to the collapse prevention in BYOL.
One reasonable follow-up explanation on the efficacy of the predictor may consider its specially designed architecture or some good properties brought by the weight initialization, which makes it hard to understand the mechanism behind it. Fortunately, after replacing the current predictor, two-layer MLP with batch normalization (loffe & Szegedy (2055), with different network architec- tures and weight initializations, we find that there is no significant change either on linear evaluation protocol or on the model behavior during training (Table I for detailed training trajectory, refer to Figure 4.. We first replace the complex structure with linear mapping qu (-) W(-). This replace- ment provides a naive solution to representation collapse: W = I, while it never converges to this apparent collapse. Surprisingly enough when go harsher this linear predictor by initializing
3

---

Under review as a conference paper at ICLR 2021
Table 1: Evaluation results of BYOL variants on CIFARIO after 300 epochs of training, used as evidence supporting the proposal of RAFT. () (1, are set in BYOL' and RAFT. Lalign (9w fo) and uuiiform (fo) are evaluated by averaging the last 10 epochs of training. We highlight the overly optimized Lalign and failed Luniform in this table. For the accuracy on the linear evaluation protocol, we also only highlight the ones that underperform the random baseline.
Model
Qu
Calign (qw fo)
Luniform (fa)
Linear Evaluation Protocol(%)
Rand- BeleeBsslinee
W
78.09 x 10
-0.51
42.74 0.41
BYOL
MLP
25.03 x 10
-222
64.32 = 0.89
BYOL'
MLP
22.09 10-4
-2.07
69.21 = 1.01
RAFT
MLP
18.90 x 10-4
-2.04
71.31 = 0.75
BYOL-LP
W
7.71 x 10-4
-2.16
67.64 0.90
BYOL'-LP
W
7.32 10
-2.19
68.61 + 0.73
RAFT-LP
W
7.42 10
-2.23
67.55 + 0.55
BYOL-NP
I
1.94 X 10-10 10
-0.14
20.92 1.29
BYOL'-NP
I
1.35 X 10-10
-0.10
16.92 + 1.05
RAFT-NP
I
16.07 10-4
-0.006
11.72 0.05
YOL-LP
W
7.83 10
-.92
67.63 = 1.27
TanB YOL'-LP
W
7.52 10
-2.19
67.66 = 0.76
TanRAFT-LP
W
7.61 X 10
-2.13
69.31 + 0.87
W with the apparent collapse solution 1, the model itself seems to have a self-recovering mecha- nism even though it starts off poor position: the loss quickly approaches to 0 and the uniformity deteriorates for 10-20 epochs and suddenly it deflects from the collapse and keeps on the right track. We conduct theoretical proof that randomly initialized linear predictor prevents the (more strict form of) representation collapse by creating infinite non-trivial solutions when the convergence is achieved (refer to Appendix []). while we fail to correlate the consistently optimized uniformity with the presence of the predictor, which indicates that deeper rationale needs to be found.
4 RUN AWAY FROM YOUR TEACHER: MORE EFFECTIVE REGULARIZER
4.1 DISENTANGLE THE BYOL LOSS BY UPPER BOUNDING
Analyzing LaYOL is hard, since it only has one single mean squared error term and there are many factors entangled within it, e.g., two augmented views of the same data, predictor, and the EMA updating rule. Inspired by the Variine decomposition on squared loss (Geman et al. 1992). extract the alignment loss by subtracting and adding the same term qu (fo(tz(z)) and further yield the upper bound of LBYOL- For details, please refer to Appendix G
Definition 4.1 (Cross-model loss) The cross-model loss Leross-motel (f X) of the function f and g over the data distribution X is defined
Lenoes model model (J X) A E
Definition 4.2 (BYOL' loss) The BYOL' loss LBYOL is defined as
LEVOL aCCatign o fu; Pros) + 2LLemes-mmdelggg o Se, Je; X2)
(6)
where a. > are constants, Ppus is defined Eq. and Xs = T2(22) is the distribution of the Calign (gw fe) to denote Catign(gu fo; Ppos)
augmented data. For the sake of simplicity, the following content. For the sake of symmetry, we use Leroes-model o fa) to denote (m/ fo. fe, X2) Lenoss-motal (gw Se, X2) to compute the cross-model loss.
4

---

Under review as a conference paper at ICLR 2021
encoder
predictor
fo (x,)
Ow (f (x1)
X1
fo
min
(x
(x2)
input
LRAST
Lavo
ONLINE
max
min
82
X2
(x,)
MEAN
Figure I: Framework diagram of RAFT and BYOL. The online network is composed of an encoder fe and extra predictor qw- The Mean Teacher fe is the EMA of the encoder fo- In BYOL, the loss is computed by minimizing the distance between the prediction of one view æ1 and another view 22's 's representation generated by the MT. In RAFT, we optimize two objectives together: (i) minimize the representation distance between two samples from a positive pair and (ii) maximize the representation distance between the online network and its MT.
Theorem 4.1 (Cayor is an upper bound LBYOL.) Lavor: is an upper bound of Lavou. if we ig- nore the scalar multiplication. Concretely speuking, for any given constants B >> 0, we huve
Lavor. S (- 1 +
Proof Please refer to Appendix G
Ideally, minimizing LAYOr would yield similar performance as minimizing LBYOL. exemplify the legitimacy of LBYOL' by setting (0, 1). In Table 1 the performance of BYOL and BYOL' are close to each other with respect to three metrics: alignment, uniformity, and downstream linear evaluation protocol, regardless of the form of predictors. When the predictor is linear map- ping, the performance differences between them are subtle. Besides, when the predictor removed, the representation collapse also happens to BYOL'. So conclude that optimizing LAYOL is al- most equivalent to LEYOL- In spite the performance similarity, L is of a more disentangled form than LAYOI. and therefore we focus on studying the former instead of the latter.
The new objective consists two terms: the first term Lalign minimizes the representation distance between samples from a positive pair and has already been shown crucial to the successful trastive methods (Wang & Isola 2020). Intuitively, it provides the motive power to concentrate sim- ilar data in the representation space. Based the form of BYOL', conclude that MT is used to regularize the alignment loss. This perspective of two terms regularizing each other is crucial to our analysis and improvement of the original BYOL framework. Understanding why BYOL works with- out collapse is approximately equivalent to understanding how minimizing Leros-model (0 o fo. f() effectively regularizes the alignment loss, or even actively optimizes the uniformity.
4.2 RAFT: RUN AWAY FROM YOUR TEACHER
The major difficulty of correlating Leross model with Luniform is that their optimization intentions are only irrelevant, but somewhat opposite. Minimizing the cross-model loss asks the network to produce close representations for certain inputs, while optimizing the uniformity loss requires to produce varying representations. The disparity residing in the form pushes us to question the original motívation of BYOL: do we really want the online network to approach to the Mean Teacher? To test our suspicion, we minimize (Calign(qee fo) Leross-mmeel (4w o fe- fs)] instead of [Catig (gro fo) Leross-model qw fe. f.)], and we find it works as well. This bizarre phe- nomenon will be explained in Section []] Removing the predictor, we observe that although mini- mizing [Catign (fo) Leross model (fo, fe) fails yield better representation than the random baseline,
5

---

Under review as a conference paper at ICLR 2021
represenlattin space representation evolution purshing force from MT butputs of online outputs of MT
RAFT-NP
BYOL' -PP
uniformity optimizer
effective regularizer alignment
others
(a)
(b)
Figure 2: Analysis on the legitimacy of RAFT and why it's more favorable. (a) Diagram demon- strating how RAFT conceptually works: if two samples' updating directions are opposite, MT helps pushing them away at the next several iterations. Here z; To (Jo()))), z = fe(x;), i = 1,2. (b) Objective categories diagram with respect the effect constraining the alignment loss. In trastive methods, the most favorable objective actively optimizes uniformity, including both BYOL' and RAFT when there exists predictor. The secondly favorable objective is the effective regularizer of alignment. RAFT remains to restrain alignment loss without predictor, while BYOL fails to do so, which implies that RAFT more unified objective.
it prevents the ment loss, while minimizing
alignment loss, i.e. it works an effective regularizer for the align- (J= J&) does not.
Based the conclusion above and law of Occam's Razor, propose new self-supervised learn- ing framework, Run Away From your Teacher (RAFT), which optimizes two learning objectives simultaneously: (i) minimize the alignment loss of samples from a positive pair and (ii) maxi-
mize the distance between the online network and its MT (refer to Figure and Algorithm I.
Definition 4.3 (RAFT loss) The RAFT loss LRAFT is defined as
RAPF
aCalign o fe; Ppos) -Boross-model (Aw fe. N2)
(8)
where a, are constants and other components follows Definition 4.2
Compared to BYOL and BYOL', RAFT better conforms to our knowledge and is conceptually non-collapsing algorithm. There has been a lot work demonstrating that weight averaging is roughly equal to sample averaging (Tarvainen & Valpola 2017). thus if two samples' representa- tions are close to each other at the beginning and their initial updating directions are opposite, then RAFT consistently separates them in the representation space. All the forms of loss terms could be classified into three categories: uniformity optimizer, effective regularizer for alignment loss, and others (refer to Figure 2b). According our experiments, when the predictor is removed, run- ning away from MT remains an effective regularizer for the alignment loss while BYOL's running towards MT fails do so. thus RAFT is of more unified and consistent form. In summary, our pro- posed learning framework RAFT is completely based on the intention of solving the inconsistency of the predictor BYO,, and it's better than BYO in threefold:
Consistency. Compared to BYOL, our newly proposed method has an effective regularizer for the alignment loss regardless of the presence of predictor.
Interpretability. Mean teacher uses the technique of weight averaging and thus could be considered as an approximate ensemble of the previous versions of the model. Running away from the mean teacher intuitively encourages the diversity of the representation, which is positively correlated to the uniformity.
Disentanglement. The learning objective is decoupled into aligning two augmented views and running away from the mean teacher, and hence could be independently studied.

---

Under review as a conference paper at ICLR 2021
We will discuss the relationship between RAFT and BYOL' in the next section, and find BYOL' a special form of RAFT under certain conditions, which makes the performance of BYOL' a guarantee of the effectiveness of RAFT. We provide benchmarks of alignment, uniformity, and downstream linear evaluation performance on CIFARIO (Table 3). We discover that balancing the alignment loss and the cross-model loss is not an easy job with the predictor taken away. The im- balance between the alignment loss and the cross-model loss would lead to representation collapse or over-regularized alignment where every data randomly projected. One interesting research di- rection is to study the efficacy of the predictor. The reason why it helps the two terms to achieve an equilibrium is left be answered.
UNDERSTANDING B YOL VIA RAFT
In Section 4.1 we derive upper bound LavOr of LavOI. and explicitly extract two terms Catign and Lermsss In BYOL', two terms are simultaneously minimized, while in RAFT, we minimize Lalign but maximize Ceross-mdell instead. To clearly distinguish the difference between the two objectives, we rewrite them as following:
L Oehoaigg Iw fo) + BLeuss model (he fo, fe); GAAFT = CoLatign Awo o [o) 3L (To o so, fe);
(10)
where a, B > 0 are constants.
In form, CRAFT and LEYor: seem to evolve in opposite optimizing direction on the second term, but the empirical study has shown that both of them work. How can two opposite optimization goals produce similar effect? Since RAFT is a conceptually working method, we analyze the mechanism of BYOL' by establishing the equivalence between the parameters of BYOL' and RAFT under mild conditions.
Theorem 5.1 (One-to-one correspondence between BYOL' and RAFT) There a one-to-one correspondence between parameter trajectories of BYOL' and RAFT when the following three con- ditions hold:
i. the representation space is a hypersphere;
ü. the predictor is linear transformation, i.e. qw -W))
ii only the tangential component of the gradient on the hypersphere is preserved. Proof We prove the theorem by construction. For the detail, please refer Appendix H
Remark The third condition conforms to the property of the hypersphere representation space and easy to achieve. One can preserve only the tangential gradient by slightly modifying the loss. For example, suppose the representation of the MT is v and the representation of the input is which are both normalized, the mooss loss || - =1I2 can be revised as || >= >z112/A, where A= =gs(( (z, 2)) stands for stopping gradient of the inner product (2, 2). Our experiments in Table demonstrates that the condition of the tangential component of the gradient doesn't 't turn any of the algorithms including BYO,, BYOL' and RAFT into a collapsed one.
In Theorem we show that optimizing LBYOL' with initial parameters (0(0), (())) is equivalent to optimizing GRAFT with initial parameters -W()) when the aforementioned three conditions are satisfied. This equivalence demonstrates that the final encoder network fe and for equal to each other. Therefore we conclude that, as representation learning framework, BYOL' equivalent to our newly proposed RAFT.
From a geometric point of view, the optimization process is the data points moving the represen- tation space under the guidance of the training loss. The loss function measures the potential energy of the parameters, and the gradient with regard to the data points the motive force. If the represen- tation space is hypersphere in BYOL, then the tangential force, i.e. the tangential component of the gradient, is the only key to scattering or concentrating the data points in the representation space. By the central symmetry of the hypersphere, clockwise and counterclockwise moving direc- tions are equivalent to some extent, for example, pushing a point by x/2 and pulling it by π 2 on the 2-dimensional sphere causes the same effect.
7

---

Under review as a conference paper at ICLR 2021
The equivalence between BYOL' and RAFT offers direct way to understand some strange phenomena we observe which are also reported the original B YOL paper. Firstly, the non-collapse of BYOL is explained, since the RAFT is an intuitively and practically working algorithm. The equivalence of BYOL' and RAFT when predictor is linear helps us understand why BYOL is an effective self-supe learning algorithm. It also explains our initial question why BYOL fails to avoid representation collapse without the predictor: removing the predictor means fixing W = I, which breaks the RAFT's designing principle of running away from the MT. Secondly, though the B YOLL' optimization procedure is of the form that two models approaching to each other, there has been no report of convergence in the original paper. The established equivalence perfectly explains it. RAFT incorporates the MT in an extremely dynamic way since it continuously varies from the history models, thus there would be no convergence of the data points. So does the parameters.
CONCLUSION AND FUTURE WORK
In this paper, we address the problem of why the newly proposed self-supervised learning framework Bootstrap Your Own Latent (BYOL) works without negative samples. By decomposing, upper bounding and approximating the original loss of BYOL, we establish another interpretable self- supervised learning method, Run Away From your Teacher. We show that RAFT contains explicit term that prevents the representation collapse and we also empirically validate the effectiveness of RAFT. By constructing a one-to- me-too-ne correspondence from RAFT to BOOL' (variant of BYOL), successfully explain the mechanism behind YOL that makes it work and therefore implies the huge potential of our proposed RAFT. Based the observation and the conclusion, here we have several suggestions for future work:
Theoretical guarantees of RAFT. Though we have intuitively explained why running away from the MT is an effective regularizer, we don't provide theoretical guarantees why optimizing RAFT would be favorable with respect the representation learning. In future, one can try to relate RAFT to the theory of Mutual Information (MI) maximization (Belghazi et al., 2018 Hjelm et al., 2018 Tschannen al. 2019). as the training objective of contrastive learning InfoNCE has been proven to be lower bound of the MI (Poole et al. 2019). One detail should be noticed when attempting to correlate RAFT with MI maximization. Even though RAFT an effective regularizer, it fails to yield good-quality representations when the predictor is removed, thus any theoretical proof on the effectiveness of RAFT should well explain the mechanism behind this extra predictor.
On the efficacy of the predictor. It has become a popular and almost standardized method to add an extra MLP on top of the network in contrastive learning methods (Chen et al.) 2020a b Grill et al. 2020), while most of the work adopts this method effect this MLP brings to the algorithm. In this paper, however, we find that this extra MLP may bring some unexpected properties to the original training objective: although the representations optimized by disparate motivations (in our paper, BYOL, running towards MT and RAFT running away from MT), the encoder network is trained be exactly the same. This observation indicates that the mechanism of the extra MLP to the network needs to be further studied.
special trick without considering the
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many con- sistent explanations of unlabeled data: Why you should average. ICLR, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540, 2018.
Pratik
Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018, 2019.
contrastive learning of visual representations. arXiv preprint MxXiv:2002.05709 2020a. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint wrXiv:2033.04297, 2020b.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma.
Neural computation, 4(1):1-58, 1992.
Jean-Bas Man-BBatieen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint 000:20006.07333 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR' 066), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 066,00 2018.
Karan Grewal, Phil Bachman, Adam
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.031665 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.v:14/2.69880 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recog- nition, pp. 1920-1929, 2019.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Samuli Laine and Timo Aila. Temporal ensembling for suni-supervised learning. arXiv preprint arXiv:1610.0222 2016.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08870, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 180. 2018.

---

Under review as a conference paper at ICLR 2021
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint arxiv:1955.06922, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information processing systems, pp. 1857-1865, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXin::906.005899 2019.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.1362) 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprint urXiv:2005. /0242, 2020.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6002-6012, 2019.
10

---

Under review as a conference paper at ICLR 2021
A MAIN THREAD OF PAPER
The proposal of RAFT based on a series of theoretical derivation and empirical approximation. Therefore the logic chain of our paper is fundamental the legitimacy of our explanation on BYOL and the superiority of our newly proposed RAFT. Here we organize our main thread in the same order as the sections, provide a clear view with readers.
In Section 3
As a learning framework, BYOL does not consistently work. It heavily relies on the exis- tence of the predictor. We want to understand why this inconsistency exists.
The architecture of the predictor doesn't 't affect the collapse of BYOL, the fact that the linear predictor qho()) (-)) prevents collapse will used as a crucial condition in Section
In Section
A new disentangled objective LBYOL = (aLaainn(Ww Sa) aa (Aw Se. Je) is established by upper bounding.
We showease that minimizing LBYOr' close to minimizing LBYol in terms of alignment, uniformity, and linear evaluation protocol, which indicates that understanding the behavior of optimizing BYOL's upper bound approximately equivalent understanding BYOL.
In Section 4.2
We find minimizing Lalign((ee fa) L Leross-odde fa. fe)] works well, which has the exact opposite way incorporating the cross-modes loss to BYOL'.
Based on the observation above, propose a new self-supervised learning approach Run (9w Away From your Teacher, which regularizes Latign (9w fe by maximizing Leross-mdell (9w
fe. fe). Compared with BYOL, RAFT accords more with our common understanding.
Additional experiments show that without predictor, BYOL' fails regularize Latign (Ja), let alone optimizing uniformity. On the contrary, although not able to actively optimize uniformity either, RAFT's maximizing Lercas-modal (fe. fe) continues be an effective reg- ularizer for Lalign (fe), which makes it more favorable (Figure 2b).
In Section 5
We prove that when the predictor is linear (g.- W) and the representation space is hypersphere where only the tangential component of gradient is preserved during training, minimizing femess (W for fc) and maximizing it obtain the same encoder fo-
Based the equivalence above, conclude that BYOL' is a special case of RAFT un- der conditions above. The equivalence established helps understanding several counter- intuitive behaviors of BYOL.

---

Under review as a conference paper at ICLR 2021
B BACKGROUND AND RELATED WORK
B.1 CONTRASTIVE LEARNING
Contrastive methods relies on the assumption that two views of the same data point share the infor- mation, and thus creates a positive pair. By separating the positives and the negatives, the neural network trained by the algorithm learns to extract the most useful information from the data and performs better on the downstream tasks. Typically, the algorithm uses the InfoNCE objective:
Commras (h, K) E
(w )omPpr (w.tt))
log
(11)
elx,,tt) + E g ch
{-]] ) -..
where (I, (x) are sampled from the positive pair distribution Ppos which is built by series of data augmentation functions [ref]. The negative samples {,,] are iid sampled for K times from the data distribution Xx; function h(x, y) measures the similarity between two input data (æ,, y)- Empirically for the sake of symmetry, the measurement function h(z, v) d(f(z), f()) has an encoder f()) and a similarity metric .)..,) evaluating how close the two representations are.
B.2 MEAN TEACHER
There is one type of semi-supervised learning method that BYOL constantly reminds people of, Mean Teacher (MT) (Tarvainen & Valpola 2017: Laine & Aila 2016). Like B YOL, MT is also of Teacher-Student (T-S) framework, where the teacher network is also the EMA of the student net- work. The additional consistency loss between the teacher and student is applied to the supervised signals. There has been a lot of work demonstrating the efficacy of (tthiwawrnn et al.) 2019 Novak et al. 2018: Chaudhari et al. 2019), among which the major conclusion states that the consis- tency loss between the student and its MT acts as a regularizer for better generalization. The proven properties of MT' might lead us to focus on how the online network's 's learning from MT effectively regularizes Lalign in BYO.. In this paper, however, we propose the opposite way of leveraging MT in contrastive methods.
C EXPERIMENTAL SETUP
Dataset Our main goal is to unravel the mystery why BYOL doesn't collapse during training and to solve the predictor-incon inconsistency. The most important metric is whether the algorithm collapses or not, and don't 't't target on developing more powerful self suff-super learning algorithm that surpasses SOTA on large dataset. In this repsect, we limit our experiments to the scope of the CIFARI0 dataset. Each image is resized from 32 x 32 to 96 96. This change is the consequence of the tradeoff between the effect of the data augmentation and batch size: larger size of the image would allow more subtle and informative data augmentation scheme while it will reduce the training batch size, which has already been empirically shown is harmful to the model performance.
Model architecture In our experiments, the model is composed of three stages: encoder fe that adopts the ResNet18 architecture (without the classifier on top); a projector ge that is comprised of linear layer with output size 512, batch normalization, rectified linear units (ReLU), and a final linear layer with output size 128; a predictor tu that is comprised of the same architecture as the projector but without the batch normalization.
Training We adopt the same data augmentation scheme that is used in Chen et al. (220a)) and Grill et al. (2020) and train the BYOL on the training set for 300 epochs with batch size 128 on 3 random seeds. The objective of training is specified accordingly and the model is trained on the Adam optimizer with learning rate x 10-1 (Kingma & Ba 2014). Unless stated otherwise, we update the target network with the EMA rate 4 x 10 without the cosine smoothing trick.
After training, we evaluate the encoder's performance on the widely adopted linear evaluation protocol: we fix the parameter of the encoder and we train another linear classifier on top of it using all the training labels for 100 epochs with learning rate x 10-4, The final classi- fication accuracy indicates to what degree the representations of the same class concentrate and the representations of the different class separate, and thus tells the quality of the representation.
Evaluation
12

---

Under review as a conference paper at ICLR 2021
D TABLES
Table 2: Look-up table for the models that appear in the paper.
Model Name
Description
BYOL-MLPP (BYOL)
BYOL with MLP-Predictor
BYOL-NP
BYOL with No Predictor
BYOL-LP
BYOL with Linear Predictor
BYOL-LPI
BYOL with Linear Predictor initialized with / BYOL with Linear Predictor, preserving only the Tangential gradient
TanBYOL-LP
PYOL'-MLPP (BYOL')
trained with LUYOL Lalign crros--models MLP Predictor
BYOL'-LP
trained with Lnvor. = Calign Ceross-model. linear predictor trained with LIYOL Calign Leross model No Predictor
BYOL'-NP
TanBYOL
BYOL' with Linear Predictor, preserving only the Tangential gradient
RAFT-MLPP (RAFT)
RAFT with MLP-Predice RAFT with No Predictor
RAFT-NP
RAFT-LP
RAFT with Linear Predictor
TanRAFT-LP
RAFT with Linear Predictor, preserving only the Tangential gradient
Table 3: Evaluation results of RAFT CIFAR1O. and represents the weight of Latign (To So and Ceross-model of. fe i.c. L = a Calign Auw fe) + Berressmmoell Tue fe. fe). All the quantifiable metries are evaluated after 300 epochs of training of the training set of CIFARIO. Compared to BYOL', our proposed RAFT is better in terms of the effectiveness of regularizing the Lalign.
Model
f
B
Lalign ( Aw o So)
Cunitorm(fa
Linear Evaluation Protocol (%%)
-and-Baeeline
7.81
10
-0.51
42.74 + 0.41
RAFT
MLP
-0.1
8.37
10 10
-2.00
65.53 = 0.99
MLP
1
-1
1.89
10-3
-2.04
71.31 = 0.75
MLP
-10
1.00 x 10-2
-0.29
25.88 + 0.42
RAFT-LP
W
I
-0.1
8.22 10
-1.61
52.57 + 2.72
W
I
-1
7.42 10-4
-2.25
67.55 + 0.55
W
-10
3.70 x 10
-2.15
66.10 0.82
RAFT-NP
-1
1.61 x 10-3
-0.01
11.72 + 0.05
T
-10
1.54 10-2
-0.99
32.13 + 0.52
I
-100
1.56 10-2
-1299
29.36 = 0.53
BYOL'NPP
I
1.35 X 10
-0.12
16.92 1.05
I
1
10
2.38 X 10
-0.88
24.42 + 1.14
T
1
100
4.55
10-8
-1.14
37.48 + 2.06
13

---

Under review as a conference paper at ICLR 2021
E ALGORITHMS
Algorithm 1: RAFT: Run Away From your Teacher
Inputs :
X. Tı, and T2
set of images and distributions of transformations model parameters and encoder
8 and fo
and qu
predictor parameters and predictor
& and fe optimizer
MT parameters and MT
optimizer, updates online parameters using the loss gradient total number of optimization steps and batch size target network update schedule and learning rate schedule
K and N
1K {Te k= and (me)
K
for k = to K do
{1; f=1 XW
sample batch of N images
for r; E B do
t Ti and t2 T2
// sample image transformations // reps for model // reps for // loss for alignment.
f ((((f((t(x:))) and 22 + (ff(tt((z).
2 (xx)) and 2 c fe((t((;))
4 |a | 12
1 2 |
Il 12 +1
1
11) // loss for cross-model
end
80 NV Σ 0gl. + dglp
11 compute the loss gradient w.r.t.0
0 optimizerree (0, 80,10
// update trainable parameters // update target parameters
12
E THE (1 Th
13 end
Output: encoder fo
14

---

Under review as a conference paper at ICLR 2021
F VISUALIZATION OF TRAINING EVOLUTION
F.1 REPRESENTATION DISTRIBUTION EVOLUTION OF DIFFERENT LEARNING ALGORITHMS
4000V4
-35
45
45
-55
46
(a) Supervised learning representation distribution evolution
<pati
08
(b) BYOL representation distribution evolution
1.4
50-
BYOL representation distribution evolution w/o predictor
60
-30
413
12
13
(d) BYOL' ACalimCaigg Lecoesemuaal representation distribution evolution
ewent
33
J45
24
1
20
61
10
26
66
(e) RAFT( (Calig Lecoss-moda) representation distribution evolution
Figure 3: Visualization of the representation distribution evolution on CIFARIO training set. We project the representation fz( to 2-D dimension using PCA (Wold et al. 1987 and then normalize unit sphere. The width of the circle shows the density of the data points that are projected to that particular position. Two dots residing two augmented views of the same data. (a) Supervised learning has no restriction on the uniformity of representation space. (b) BYOL with predictor evenly projects the data different positions. (c) BYOL w/o predictor tends to project the huge portion of the data to the same position. upper bound BYOL' also effectively disperses representations that minimizing/maximizing Lecoss-model has similar effect on the final representation distribution.
each side of the blue line across the circle represents
(d) BYOL's the sphere. (e) Our RAFT shows
15

---

Under review as a conference paper at ICLR 2021
F.2
EVOLUTION OF NUMERICAL METRIC DIFFERENT LEARNING ALGORITHMS
0.0
100
-1.0
Trahig
40
BYOL-HIFP
BFOLLP
-23
BYOI-LP
0
50
100
120 200
252
503
20
100
120
200
230
300
30
120
255
200
250
350
(a)
(c)
Figure 4: Training behaviors of BYOL with varying structures of the predictor. For detailed expla- nation on the model architecture, refer to Appendix (a) Evolution of training loss Lavor. When taken the predictor, the training loss quickly converges to 0 (red curve, BYOL-NP). Replacing the MI Mree-rre (blue curve, BYOL-MIPPP) with the linear predictor (green curve, BYOL-LP) will not cause the collapse even though an apparent solution for collapse. Furthermore, ini- tializing the linear predictor with / forces the loss quickly approaching to 0 at beginning, while it recovers from the seemingly collapse after 10-20 epochs of training (orange curve, BYOL-LPII, (b) Evolution of the representation uniformity. BYOL with predictor consistently optimizes the unifor- mity of the representation distribution even though the uniformity not explicitly included in loss term. One interesting fact to note here is that the uniformity loss is optimized with a constant rate with linear predictor (green curve, BYOL-LP; orange curve, BYOL-LPI) after certain phase of training. (c) linear evaluation protocol CIFARIO. Different structures of the predictor provide close performance on the downstream classification task.
2.50
2:00
2.00
5=1
1.50
1.50
1.50
E.25
125
1.25
0.7
075
100
0.75
0.50
050
0.50
0.50
150
350
200
160
150
9.6
-0.5
-05
-3.5
10
-3.0
-10
-2.
Letomby
-2.0
-30
3=01
-2.0
8:3
20
1030
p-
100
2- 100
-3.0
20
100
150
700
750
305
-30
53
100
350
300
250
760
-3.0
50
150
150
700
250
203
Tainnng #sochs
Traning epocte
Trahing
(a)
(b)
(c)
Figure 5: The evolution traces of Latign (qu for fe) and Cunifom((o) in BYOL' and our proposed RAFT. (a) Evolution trace of BYOL'NNP, Increasing 3 (weight of Lerus regularizer for Latign) does not prevent the failed regularization: Calign converges to quickly. (b) Evolution trace of RAFT-NP. Small value of doesn't effectively regularize Latign. but increasing the weight helps. In this respect, RAFT a more effective regularizer, while the uniformity optimization holds no huge difference from BYOL'-PP (c) Evolution trace of RAFT-LP. With the linear predictor, on the contrary to RAFT-NP, the uniformity is optimized consistently during training, which implies deeper rationale of the existence of predictor.
16

---

Under review as a conference paper at ICLR 2021
G PROOF OF B BOL UPPER BOUNDING
In this section, we provide how we derive the upper bound of LaYOL. For the sake of simplicity, without loss of rigor, we use t t (xr) to represent the transformed input x.
LAYOL B Ti
(fo(tt -
=E 1l qn (fo(22)) + qn (f - -
(12)
By applying the Sauchy-Schwardd inequality Eq. we yield:
LBYOL - +1+ + (B (flm)()((()))) - qio + AE [1le [fo - fe fe(
1 -1+ + 2) (Aulign(9w fo: Ppos) + codess-mdel o fox fc:
(13)
which stands for any O, where the positive-pair distribution Ppos is modeled by the chain rule of the conditional probability:
Ppec (x1, = X(x) (((r)-T((tt)l) (t) T2(t2/z).
For any given pair a, 3 > O, we let A = B//o and substitute back to Eq. 13 yielding
LAYOL S (1 +
a
Ppu)) Latign (ww o So: ) +
B Lenns Lenne madel o So. Je, 22)
8
a
1
1
(-
|aLCatignAww soi Ppos) + Beemsss (gw for fε, X]]
a
3
LBYOL'
(14)
B
and optimization objective, have
min 1 +
1 L + min LavoL'
(15)
Therefore we have proven that LavoL' as optimization objective is the upper bound of LBYOL-
To note here, one subtract and add different term (x1 form the alignment loss the side of MT fe.
LBYOL E (fl((()(())) + fe(22))l)
(16)
while it doesn't t help to solve the problem since the alignment constraint on the side of MT doesn't t generate gradients.
17

---

Under review as a conference paper at ICLR 2021
H PROOF OF ONE-TO-ONE CORRESPONDENCE BETWEEN BYOL' AND RAFT
Theorem (One-to-one correspondence between BYOL' and RAFT) There is a one-to-one cor- respondence between parameter trajectories of BYOL' and RAFT when the following three condi- tions hold:
the representation space is a hypersphere;
i. the predictor is linear transformation, i.e. qw(-)) W();;
only the tangential component of the gradient on the hypersphere is preserved.
Without losing generality, suppose that 21 (1 T2 = tz(x) where x is an arbitrary in- put and batch size is and (a, (3 (1,1). We set BYOL' and RAFT with initial parameters (0', (') (0(0) W()) and (0, W (010) -W( respectively. For convenience, assumes
the dot product "'" ignores the row layout or column layout in the chain rule of derivatives and we
define the following symbols:
22 fe(22),
(17)
2 W (x1),
(18)
22 W So (x2),
(19)
W fo(a)),
(20)
22 W (x2
(21)
Based on the notations defined, we rewrite the loss terms of BYOL' and RAFT as follows:
RYOL.' align
||
2/2 2
(22)
emse-model
BYOL'
=
1|25 Σ2
(23)
RAFT
align
||=1 22
(24)
RAFT emss-model
-1|
(25)
The two objectives are following:
L
BYOL' Lalig
CBYOL' cross-modell
(26)
CRAFT
Lalign
CAPPT
RAPT cross-model!
(27)
We claim that under the third condition, the following equations hold:
@LavoL
@L
@L
(28)
a0
0e
aw"
aw
where subseript denotes the tangential component of the gradient.
Firstly we show the equivalence with respect to 0. Differentiate Oij respectively, we obtain
CBYOL' align
Clign align
CRAFT with respect 0zz.
OCRYOL
0zk
0z
=
(z] - 22211 + (2) - x2211]
(29)
00'
00'
00,
alignn
CLPFT
2
(21 22)1 (3) 22)
021
022
(30)
00g
a0
00,;
align
2 (=1 2/111111111
0%1
0z1
(31)
00
36'
00,
CRAPT
0z1
022
align
2 ( 22)1
(32)
00.g
|1
300;
00:;
18

---

Under review as a conference paper at ICLR 2021
where (zi -21) 242,, (21 22) are vectors at the points 2/2 and 22 on the hypersphere and decompose the vector into the tangential (denoted by []) and normal component (denoted by L):
(< 22) (x) - 22) + (21 2))1] (2) - 22) [ 22)1 + (21 - 12)1
(33)
Generally, suppose 2 is a unit vector starting at the origin point, which is perpendicular to the unit hypersphere at the point 2, for any vector v starting the point 2, we have
T (0, 2, "I
Ta
(0,
(34)
Then we can compute the tangential component of the gradient:
( >)11 (zj z {zí - 2/2 =2) 2/2
= (z/2 z1) 2/21
(2) 22)1 (21 =2) (21 22, 22) 22
= (22, ø1) 22.
(35)
Because of the initialization,
21, 22 2/2 therefore we have
(21
2/2111
(2) 22)
(36)
02
0zh
821
0m2
(37)
08%
00%
38
00:j
So we show that
Ö CBYOL align 00
CRAFT 00:,
(38)
We differentiate L
BYOL'
RAFT
with respect to Oz Ogj respectively, obtain that
model
model
BYOL! d crass madel
of (x2)
=-2 =2211 W
(39)
08
38%
CRAFT cross madel 00.j
22)1 W. (12)
= 2
(40)
00,j
Similar to Eq.
35
we derive that
22
2/)1 =32 22)
(41)
Since 0' = =. ff (x2))/00 (fo(zz) /00r and W = -W we have that
a CBYOL'
8 CRAFT
(42)
00
30:
Therefore by Eq. 38 and Eq. 42 RAFT's updating of the parameter is equal to BYOL':
ae' |1
(43)
00
Also we differentiate
BYOL'
CRAFT with respect to W/G, Wi respectively, we obtain that align
align
d BYOL'
0zh
align
2 - -=2111
(44)
OW
aw
aW
1/
a CRAFT
dz1
822
aliign
2 (21 -22)1
(45)
aw,;
aW
19

---

Under review as a conference paper at ICLR 2021
Note that -21, 22 222 and similar to Eq. 35 easy to show that
(2) - 2/2 (21 22)11
(46)
Also,
02
221
(47)
OW 19
a
4zyy
022
(48)
aW
aW
So we have
CBOOL align aw 11
RAFT align
(49)
aW., -I
Differentiate >
CBOL''
RAFT
with respect to W Wa, respectively, we obtain that
model
OLBYOL' cros-model
Oz/
2 2/221111
(50)
aw ij
aw
d RAFT
222
model aw
2x2
22)11 aWss
(51)
Then have that
2 d BYOL'
CCRFFT cross-mode model
cross model
(52)
OW!
OW
By Eq. 49 and Eq. 52 we prove that the cross-model loss of BYOL' generates the opposite gradient
to RAFT. namely,
@L
OLRFFT
(53)
OW
OW
Therefore by the two main conclusions Eq.
43 and Eq.
53 for BYOL' with parameters (0', W') (0(0) W respectively, we have
(0(0), W()) and RAFT with parameters (0, W
(0(()) = 000) 11 11 [LuooL
W Wr(()) . &Buyoo
BYOL:
1/ OW
00 10-0000
|1 W' WK(0)
(54)
RAFT: (00)-00)) 000) - [1 @LRFFT
@CRAFT
W( = (0) -1 aW
(55)
00
P=g(o)
We derive that ((1) (((1), W() -W(1), and furthermore, g(k) (((A), ((A)
WV(A)
at any iteration k. In this way, we establish an one-to-one correspondence between the parameter trajectories of BYOL! and RAFT in training, referred to H:
H: RAFT((..)) + BYOL(
(56)
20

---

Under review as a conference paper at ICLR 2021
I NON-TRIVIAL SOLUTIONS CREATED BY PREDICTOR
Suppose inputs 21 = 1 (x) and 22 = (x) is n-dimensional.. And in linear model, Se. Je and tu is parameterized by matrices (0tj)nxxs (Eighaxn and (Wij) mx) respectively.
The objective is
LaVOI (x2. [[om((a((((()) - Je(c))))11
(2X [lW02] f021
(57)
Differentiate |[0zz Ex2l with respect to Oij and Wij. we have
a [[[Wrr1 - Ex² 12]
WA. (6x1) 22.
00.5
08,
θ (O..))x))) - Cn,
2 (Wx,.(011) -x,
8ig
k=1
2T (xi)
k=1
2 (WTT) x lij
(58)
① [| - Ex2 [2]
m
W (0.1) x22
aWij
θ (0x)) - +2
2 (f.ri) [x,
OW,
2T' (7x1) ((==))
k=1
2T,0,,
[T(0x:) "'
(59)
where Te = W (0x1) - LA. a1) T= (Ti, T "m.....T))"""" W(On)) Eor,, and Wk,s Ek. are the k-th of W and respectively.
So
a [| f
θ [[[ - Ex2 112 [T()))]]
(WTT) x
(60)
00
OW
Let
θL 0. 0, aW
(61)
00
we have that
OE (g [[] 0x; - Ex2
0 @| 0x
Ex2ll3
0
(62)
00
08
OE
[[ (1.. - Ex22222
Ol aw
Exall
0
(63)
OW
When the weight of target converge, we have ((t) = ((+++)) in the updating rule,
((k+1) = + (1 Th ((())
((4) -((++)) (A)
(64)
21

---

Under review as a conference paper at ICLR 2021
Substituting by 8, obtain
(xX,717) [W] (WOx: - 0x (lo,, [WOrr] 0x22) æj 0T] =0
(65)
(66)
Let (1 = B, (127 B, we have that
cq sylW (W8A) W OB
W8A0 ABAT
WO OBA- -0
(67)
To solve Eq. ?? (which is called Sylvester's 's equation), we using the Kronecker product notation and the vectorization operator vec, we can rewrite the equation in the form
(Im W (BA @ In) vec 8 vec 0
(68)
So it has non-trivial solution 0 if and only if (Im & W - (BA-l)" @ I,) has a non-trivial null space. An equivalent condition to having non-trivial null space is having zero an eigenvalue. Let W has eigenvalues common with BA then we have non-trivial solution of 0, which exactly the prevention for collapse.
22


Reviews:
[{"review_id": "p40Znp-nO1_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly proposes an objective that incorporates the target network in BYOL in a opposite way that encourages the prediction of the online network to be far away from the target network.\n\nConcerns:\n\n1.  It is overly claimed that \"we unravel the puzzle of how BYOL avoids representation collapse\". For example, the authors fail to capture some intrinsic properties of BYOL, such as the role of prediction head (MLP + BN).  The theorem 5.1 can only deal with the prediction head that is linear. The authors could refer to the [1] for more insights.\n\n2. The experimental results do not support the claim that \"RAFT is a conceptually non-collapsing algorithm\". In table 1, for results equipped with q_{w}-MLP, RAPT with better acc (71.31) in fact does not have smaller uniform loss. Instead, its alignment loss is smaller (which brings the acc improvement). So, the RAFT does not actually always enlarge the uniformity as claimed.\n \n3. The implementation of BYOL in this paper regarding Cifar10 is not convincing. [2] also use the resnet18 as the encoder and it achieves the accuracy with 91+ in Cifar10. However, the reproduced result in this paper is only around 70. \n\n4. BYOL proves its own effectiveness in ImageNet. To make fair comparisons, the authors shall conduct experiments in the same dataset. Otherwise, the claim regarding to the BYOL might not be solid.\n\n5. It is acceptable that running away from the mean teacher increases the difficulty of alignment. But it is unclear to the reviewer why it can produce global uniformity in the representation space?\n\n[1] Tian, Yuandong, et al. \"Understanding Self-supervised Learning with Dual Deep Networks.\" arXiv preprint arXiv:2010.00578 (2020).\n\n[2] Ermolov, Aleksandr, et al. \"Whitening for Self-Supervised Representation Learning.\" arXiv preprint arXiv:2007.06346 (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "bLQo-ZBk1eq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nThe paper provides a new perspective on the BYOL self-supervised learning method. First, the paper introduces an upper-bound objective, BYOL', that is easier to analyze than BYOL because it is composed of two well understood losses: an alignment loss and cross-model loss. Further, it shows empirically that optimizing BYOL' is similar to optimizing BYOL. Second, the paper introduces the RAFT method which maximizes the alignment loss instead of minimizing it. The paper proves that under some assumptions, such as a linear predictor function, optimizing BYOL' is equivalent to RAFT. Based on this analysis, the paper explains why the predictor function is essential for BYOL and why it is hard to achieve convergence.\n\n*Quality*\nI really like the analysis of the paper. The paper provides a mix of theoretical and empirical argument for understanding BYOL, and introduces a new method called RAFT. The main drawback of the paper is that it limits the empirical analysis to a single and much simpler experimental setup using CIFAR10 and resnet18. I believe that since BYOL's significance is an empirical one and is mainly established on Imagenet, any empirical analysis of BYOL in other simpler settings is quite limited.\n\n*Clarity*\nThe authors have done a very good job in writing this paper. The logic, presentation and results are quite clear to understand.\n\n*Originality*\nI find the paper quite interesting and original in its analysis. I especially like the analysing BYOL through the BYOL' upper-bound.\n\n*Significance*\nI think the results of the paper could have been quite more significant if applied on other experimental setups. While I understand working with SOTA models can be computationally expensive, the main argument of this line of work is empirical and it is hard to be convincing without more extensive empirical results.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JGO_OAVdEQU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the recently proposed Bootstrap Your Own Latent (BYOL) algorithm for self-supervised learning and image representation. \nThe authors first derive an alternative training procedure called BYOL' by computing an upper bound of the BYOL objective function.\nAfter diverse analyses, the authors then introduce Run Away From Your Teacher (RAFT), where RAFT is another BYOL variant that resembles contrastive method by having an attractive and repealing term in the training objective. According to the authors, this decomposition allows for a better understanding of the training dynamics.\n\nFinally, the authors made the following transitivity reasoning:\n - BYOL and BYOL' are almost equivalent\n - RAFT and BYOL' are shown to be equivalent under some assumptions.\nThus, conclusions that are drawn from analyzing RAFT should still hold while analyzing BYOL. They thus link the interest of BYOL's predictor and the EMA through the RAFT loss decomposition. \n\nI have multiple strong concerns regarding this paper. These concerns are both on the paper results, shortcuts in the analysis, and the writing style. \n\n\nResults:\n--------------\n\n - In section 4, the authors introduce BYOL' as a variant of BYOL. To do so, they derive an upper bound on the BYOL loss, i.e. the L2 distance between the projection and the projector, and they try to minimize it. However, this approach disregards that BYOL does not minimize a loss (due to the stop gradient). In other words, the BYOL objective keeps evolving during training; the target distribution is non-stationary.  As mentioned in the BYOL paper: \"Similar to GANs, where there is no loss that is jointly minimized w.r.t. both the discriminator and generator parameters; there is therefore no a priori reason why BYOL’s parameters would\nconverge to a minimum of L_BYOL given the online and target parameters\". Minimizing an upper-bound is at best insufficient, at worst a non-sense. The sentences, \"minimizing L_{BYOL'}  would yield similar performance as minimizing L_{BYOL}\" and \"we conclude that optimizing L_{BYOL'} is almost equivalent to L_{BYOL}\" are unfortunately wrong.  This is somewhat highlighted different qualitative results in Appendix F.1.b != F.1.d.\nA better approach would be to ensure that the *gradients* go in a similar direction (so the training dynamics are similar rather than the objective function). However, even such a demonstration could be insufficient due to compounding factors in the training dynamics. \n - The 1-1 mapping between BYOL' and RAFT rely on three hypotheses. While (i) and (ii) are reasonable, hypothesis (iii) is quite strong, and more importantly, neither elaborated nor discussed. In other words, I am unable to validate/invalidate the interest of the theoretical results. Would it be possible to measure the normal gradient empirically? To bound it? \n - In section 3, i would recommend the author to mention that multiple components were also in the BYOL paper; especially when writing \"therefore, we conclude the predictor is essential to the collapse prevention of BYOL.\"\n - Although I acknowledge that self-supervised learning requires heavy computational requirement, and few teams may run experiments on ImageNet. Yet, I would recommend the authors to not use CIFAR10 as the dataset has multiple known issues (few classes, small images, few discriminative features). Other variants such at STL or ImageNete can be trained on a single GPU over a day, and are less prone to misinterpretation in the results. Besides, I want to point out that BYOL was not correctly tuned: the experiments are based on a different optimizer (Adam vs LARS) and no cosine decay were used for the EMA, while these two components seem to be critical, as mentioned in BYOL and arxiv:2010.1024. \n\nOverall, I have a serious concern about the paper's core contributions. However, there are still some good elements in the paper that I think are under-exploited:\n - RAFT is itself an original, new and interesting algorithm. The potential link to BYOL is indeed an interesting lead, but in its current state, I would make it a discussion more than a key contribution.\n - Table D.3 shows that RAFT/BYOL' does not collapse without predictors when \\beta is high. Albeit providing low accuracy, a non-collapse is quite surprising. Unfortunately, the authors leave it for future work\n\n\nShortcuts:\n--------------\nI was surprised by multiple shortcuts in the reasoning process or undiscussed conclusions:\n - The authors mention that the predictor is a dissatisfactory property of BYOL. Could they elaborate? This is actual the key component of the method (if not the only one!), and such pro/cons could be detailed in light of other methods. \n - In section 4.1, the authors mention that: similar accuracies and losses are sufficient somewhat confirm that BYOL and BYOL' are similar. Two completely different methods may have the same errors while being radically different... \n - In Section 4.2, the authors mention that \"Based on the form of BYOL, we conclude that MT is used to regularize the alignment loss\". However, there is no experiments to try to contradict/validate this claim. Differently, the EMA may ease the optimization process or it may have different properties. Even if I understand the logic behind this statement, I regret that the authors do not try to confort it. \n- In section 4.2, the authors mention that there exist multiple works (while only citing one...) demonstrating that EMA is \"roughly\" equivalent to sample averaging and may encourage diversity. While this is sometimes true in specific settings (cf. markov game and fictitious play), this is also known to ease optimization (cf. target network in DQN). Stating that RAFT is better than BYOL because it better leverage the EMA target is tricky without proper analysis.\n- Albeit understandable, the transitivity between BYOL and RAFT is difficult to defend due to multiple approximations and hypothesis. Therefore, it is of paramount importance that the approximations and hypothesis are validated, which is not sufficiently done in the paper. \n\n\nWriting: \n--------------\n - Although papers' writing quality remain subjective, I tend to expect a formal language. I kind of feel ill-at-ease when reading sentences including \"BYOL works like a charm\", \"disclosing the mistery\", \"to go harsher\", \"bizarre phenomon\". Other sentences also expresses judgement such as \"inconsistent behavior\", \"dissatisfactory property of BYOL\" or \"has admirable property\" without proper argumentation. \n - It is non-trivial to follow the different version of the algorithms... which are defined in the appendix. Please consider renaming BYOL'. \n - A related work section would have been useful to put in perspective BYOL that are theoretically motivated e.g. AMDIM, InfoMin, other self-supervised learning methods without negative example, e.g. DeepCluster, SwAV. Section 2 is more about the background, not related work.\n - there are a few confusions in the notation, \\alpha \\beta have different meaning across equations (Eq 7 vs 8)\n - In section 3, random is ill-defined. In Cifar10, random should be 10%, I assume that you refer to random projection. Please clarify.  \n - Figure 1 is clear, and I recommend to keep it as it is.\n - From my perspective, the mathematical explanation in Section 5 is quite obfuscated, and I would recommend a full rewriting.\n - Please avoid unnecessary taxonomy, e.g. uniformity optimizer, effective regulizers and others.\n - In conclusion, you mentioned some results about the projector. However, you never detail them in the paper. Please, do not discuss unpublished results.\n\nOverall, I had difficulties following the paper: I keep alternating between the appendix, previous sections, and the text. Again, the phrasing makes me ill-at-ease.\n\n\nConclusion:\n--------------------\nI have some serious concerns about the core results of the paper. Importantly, Theorem 4.1 follows a misinterpretation of the BYOL training dynamics. From my perspective, there are too many unjustified claims, and I cannot recommend paper acceptance. However, there is some good idea in the paper, and I strongly encourage the authors to study RAFT independently of BYOL in the future.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hbfpvs6x3b9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: the paper aims to explain the success of BYOL, a recently proposed contrastive method that mysteriously avoids the trivial constant solution without requiring negative samples. The paper proposes a new loss named RAFT. Compared to BYOL, RAFT is more general since it subsumes a variation of BYOL as its special case, and contains a cross-model term to be maximized which regularizes the alignment loss and encourages the online encoder to \"run away\" from the mean teacher.\n\nThe paper claims this cross-model term encourages disparity, which could help demystify why BYOL does not collapse to a trivial solution. However, the cross-model term itself cannot prevent outputs from collapsing, as explained below.\n\n**Question 1**: my main concern is the effectiveness of the cross-model loss: I disagree that the cross-model loss prevents collapsing representations. I think the authors may be confusing contrasting two samples (\"cross-sample\") and contrasting two functions (\"cross-model\"):\n- The cross-model loss is essentially the L2 distance between two functions, which is the average squared error between two model outputs on the *same sample.*\n- The common contrastive loss, which contrasts outputs from the same model on *different samples*.\n\nFor example, suppose MT is a constant function at the $t_{th}$ iteration (i.e. the function outputs some constant $c$ for all input), then the online encoder could be updated to be another constant as far away from $c$ as possible, i.e. the cross-model loss is maximized, however we still have the sample collapsing issue. As a side note, a constant function also achieves a perfect alignment loss.\nMore concretely, consider $f(x) = Wx +b$ where $W$ is initialized to be the all-0 matrix, i.e. $f(x) = b$ is a constant function. Then for all future updates, learning $f$ only updates $b$ but not $W$ (since there's no gradient on $W$), and therefore $f$ will remain a constant function, i.e. it always collapses the points. One may argue that it is wrong to choose $W = 0$, but the point is, the success of BYOL needs more careful analysis of the optimization process, which cannot be addressed by the cross-model loss term itself.\n\n**Question 2**: section 3 phrases the need of a predictor as a disadvantage of BYOL, however RAFT also requires a predictor head to achieve good classification performance. Studying the effect of the predictor is an interesting direction and will make the paper much stronger, as the authors also point out in the conclusion.\n\n**Other comments**:\n- Table 3: why are there no results for BYOL'-MLP? Comparing RAFT-NP to BYOL'-NP, there doesn't seem to be a clear edge of RAFT, both in terms of the uniformity loss and the accuracy.\nIt would also be better to highlight the key results in the table. The current table is quite dense; adding more highlights and comments will have the reader understand what to take away from these results.\n- Paper organization: a lot of material is deferred to the appendix, which makes the paper a bit hard to follow since the reader needs to jump back and forth. It would be better if results in the appendix are better summarized in the main text.\n- The term BYOL' first appears at the end of the first paragraph on page 2 without a definition.\n- Minor typo: there's an extra left parentheses in front of the second $f$ in equation (5); an extra comma after \"distribution\" in the first paragraph of section 3.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers and AC found many claims of this submission unsubstantiated. 
Decision: Reject