SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Manu Srinath Halvagal $^{1,2,*}$
Axel Laborieux $^{1,*}$
{firstname这个名字}@fmi.ch
<sup>1</sup> Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland
* These authors contributed equally.
Non-contrastive self-supervised learning (SSL) methods like BYOL and SimSiam rely on asymmetric predictor networks to avoid representational collapse without negative samples. Yet, how predictor networks facilitate stable learning is not fully understood. While previous theoretical analyses assumed Euclidean losses, most practical implementations rely on cosine similarity. To gain further theoretical insight into non-contrastive SSL, we analytically study learning dynamics in conjunction with Euclidean and cosine similarity in the eigenspace of closed-form linear predictor networks. We show that both avoid collapse through implicit variance regularization albeit through different dynamical mechanisms. Moreover, we find that the eigenvalues act as effective learning rate multipliers and propose a family of isotropic loss functions (IsoLoss) that equalize convergence rates across eigenmodes. Empirically, IsoLoss speeds up the initial learning dynamics and increases robustness, thereby allowing us to dispense with the exponential moving average (EMA) target network typically used with non-contrastive methods. Our analysis sheds light on the variance regularization mechanisms of non-contrastive SSL and lays the theoretical grounds for crafting novel loss functions that shape the learning dynamics of the predictor's spectrum.
SSL has emerged as a powerful method to learn useful representations from vast quantities of unlabeled data [1-8]. In SSL, the network's objective is to "pull" together its outputs for two differently augmented versions of the same input, so that they learn representations that are predictive across randomized transformations [9]. To avoid the trivial solution whereby the network output becomes constant, also called representational collapse, SSL methods use either a contrastive objective to "push" apart representations of unrelated images [2, 3, 10-13] or other non-contrastive strategies. Non-contrastive methods comprise explicit variance regularization techniques [6, 7, 14], whitening approaches [15, 16], and asymmetric losses as in Bootstrap Your Own Latent (BYOL) [1] and SimSiam [5]. Asymmetric losses break symmetry between the two branches by passing one of the representations through a predictor network and stopping gradients from flowing through the other "target" branch. How this architectural modification prevents representational collapse is not obvious and has been the focus of several theoretical [17-21] and empirical studies [22-24]. A significant advance was provided by Tian et al. [17] who showed that linear predictors align with the correlation matrix of the embeddings, and proposed the closed-form predictor DirectPred based on this insight. However, previous analyses assumed a Euclidean loss at the output [17-19, 21] except [20], whereas practical implementations typically use the cosine loss [1, 5] which yields superior performance on downstream tasks. This difference raises the question whether analysis based on the Euclidean loss provides an accurate account of the learning dynamics under the cosine loss.
In this work, we provide a comparative analysis of the learning dynamics for the Euclidean and cosine-based asymmetric losses in the eigenspace of the closed-form predictor DirectPred. Our analysis shows how both losses implicitly regularize the variance of the representations, revealing a connection between asymmetric losses and explicit variance regularization in VICReg [7]. Yet, the learning dynamics induced by the two losses are markedly different. While the learning dynamics of different eigenmodes decouple in the Euclidean case, dynamics remain coupled for the cosine loss.
Moreover, our analysis shows that for both losses, the predictor's eigenvalues act as learning rate multipliers, thereby slowing down learning for modes with small eigenvalues. Based on our analysis, we craft an isotropic loss function (IsoLoss) for each case that resolves this problem and speeds up the initial learning dynamics. Furthermore, IsoLoss works without an EMA target network possibly because it boosts small eigenvalues, the purported role of the EMA in DirectPred [17]. In summary, our main contributions are the following:
- We analyze the SSL dynamics in the eigenspace of closed-form linear predictors for asymmetric Euclidean and cosine losses and show that both perform implicit variance regularization, but with markedly different learning dynamics.
- Our analysis shows that predictor eigenvalues act as learning rate multipliers which slows down learning for small eigenvalues.
- We propose isotropic loss functions for both cases that equalize the dynamics across eigenmodes and improve robustness, thereby allowing to learn without an EMA target network.
2 Eigenspace analysis of the learning dynamics
To gain a better analytic understanding of the SSL dynamics underlying non-contrastive methods such as BYOL and SimSiam [1, 5], we analyze them in the predictor's eigenspace. Specifically we proceed in three steps. First, building on DirectPred, we invoke the neural tangent kernel (NTK) to derive simple dynamic expressions of the predictor's eigenmodes for Euclidean and cosine loss. This formulation uncovers the implicit variance regularization mechanisms that prevent representational collapse. Using the eigenspace framework, we illustrate how removing the predictor or the stop-gradient results in collapse or run-away dynamics. Finally, we find that predictor eigenvalues act as learning rate multipliers for their associated mode, thereby slowing down learning for small eigenvalues. We derive a modified isotropic loss function (IsoLoss) that provides more equalized learning dynamics across modes, which showcases how our analytic insights help to design novel loss functions that actively shape the predictor spectrum. However, before we start our analysis, we will briefly review DirectPred [17] and the NTK [25], a powerful theoretical tool linking representational changes and parameter updates. We will rely on both concepts for our analysis.
2.1 Background and problem setup
We begin by reviewing DirectPred [17] and defining our notation. In the following, we consider a Siamese neural network  $z = f(\boldsymbol{x};\boldsymbol{\theta})$  with output  $\boldsymbol{z} \in \mathbb{R}^{M}$ , input  $\boldsymbol{x} \in \mathbb{R}^{N}$ , and parameters  $\boldsymbol{\theta}$ . We further assume a linear predictor network  $W_{\mathrm{P}} \in \mathbb{R}^{M \times M}$  and use the same parameters for the online and target branches as in SimSiam [5]. We denote pairs of representations as  $z^{(1)}, z^{(2)}$  corresponding to pairs of inputs  $x^{(1)}, x^{(2)}$  related through augmentation and implicitly assume that all losses are averaged over many augmented pairs. The asymmetric loss function (Fig. 1a), introduced in BYOL [1], is then given by:
$$
\mathcal {L} = d \left(W _ {\mathrm {P}} \boldsymbol {z} ^ {(1)}, \operatorname {S G} (\boldsymbol {z} ^ {(2)})\right),
$$
where SG denotes the stop-gradient operation, and  $d$  is either the Euclidean distance metric  $d(\pmb{a},\pmb{b}) = \frac{1}{2}\|\pmb{a} - \pmb{b}\|^2$  or the cosine distance metric  $d(\pmb{a},\pmb{b}) = -\frac{\pmb{a}^\top\pmb{b}}{\|\pmb{a}\|\|\pmb{b}\|}$ . We refer to the corresponding loss functions as  $\mathcal{L}^{\mathrm{euc}}$  and  $\mathcal{L}^{\mathrm{cos}}$  respectively.
DirectPred. Tian et al. [17] showed that a linear predictor in the BYOL setting aligns during learning with the correlation matrix of representations  $C_z \coloneqq \mathbb{E}_x\left[zz^\top\right]$ , where the expectation is taken over the data distribution. Since the correlation matrix is a real symmetric matrix, one can diagonalize it over  $\mathbb{R}$ :  $C_z = U D_C U^\top$ , where  $U$  is an orthogonal matrix whose columns are the eigenvectors of  $C_z$  and  $D_C$  is the real-valued diagonal matrix of the eigenvalues  $s_m$  with  $m \in [1, M]$ .
a
b
Figure 1: (a) Schematic of a Siamese network with a predictor network and a stop-gradient on the target network branch. The target network can be a copy (SimSiam [5]) or a moving average (BYOL [1]) of the online network. In either case, the target network is not optimized with gradient descent. (b) Visualization of learning dynamics under the Euclidean distance metric showing learning update directions along two eigenmodes, with the light cloud representing the distribution of the representations  $z$ , the darker cloud representing the predictor outputs  $W_{\mathrm{P}}z$ , and the dotted circle indicates the steady state  $\lambda_{1,2} = 1$ , reached during learning. All eigenvalues converge to one. (c) Same as (b), but for the cosine distance. The dotted line indicates the steady state  $\lambda_1 = \lambda_2$ .
C
Cosine distance
Given this eigendecomposition, the authors proposed DirectPred, in which the predictor is not learned via gradient descent but directly set to:
$$
W _ {\mathrm {P}} = f _ {\alpha} \left(C _ {z}\right) = U D _ {C} ^ {\alpha} U ^ {\top}, \tag {1}
$$
where  $\alpha$  is a positive constant exponent applied element-wise to  $D_{C}$ . The eigenvalues  $\lambda_{m}$  of the predictor matrix  $W_{\mathrm{P}}$  are then  $\lambda_{m} = s_{m}^{\alpha}$ . We use  $D$  to denote the diagonal matrix containing the eigenvalues  $\lambda_{m}$ . While DirectPred used  $\alpha = 0.5$ , the follow-up study DirectCopy [18] showed that  $\alpha = 1$  is also effective while avoiding the expensive diagonalization step. While Tian et al. [17] based their analysis on the Euclidean loss  $\mathcal{L}^{\mathrm{euc}}$ , most practical models, including Tian et al.'s large-scale experiments, relied on the cosine similarity loss  $\mathcal{L}^{\mathrm{cos}}$ . This discrepancy raises the question to what extent setting the predictor to the above expression is justified for the cosine loss. Empirically, we find that a trainable linear predictor does align its eigenspace with that of the representation correlation matrix also for the cosine loss (see Fig. 4 in Appendix A).
Neural tangent kernel (NTK). The NTK is a powerful analytical tool characterizing the learning dynamics of neural networks [25, 26]. Here, we recall the definition of the empirical NTK [26] corresponding to a single instantiation of the network's parameters  $\theta$ . If  $|\mathsf{D}|$  denotes the size of the training dataset,  $\mathcal{L}: \mathbb{R}^M \to \mathbb{R}$  an arbitrary loss function,  $\mathcal{X}$ , the training data concatenated into one vector of size  $N|\mathsf{D}|$ , and  $\mathcal{Z} = z(\mathcal{X})$ , the concatenated output of size  $M|\mathsf{D}|$ , then the empirical NTK is the  $(M|\mathsf{D}| \times M|\mathsf{D}|)$ -sized matrix:
$$
\Theta_ {t} (\mathcal {X}, \mathcal {X}) = \nabla_ {\boldsymbol {\theta}} \mathcal {Z} \nabla_ {\boldsymbol {\theta}} \mathcal {Z} ^ {\top},
$$
and the continuous-time gradient-descent dynamics [26] of the representations  $z$  are given by:
$$
\frac {\mathrm {d} \boldsymbol {z}}{\mathrm {d} t} = - \eta \Theta_ {t} (\boldsymbol {x}, \mathcal {X}) \nabla_ {\mathcal {Z}} \mathcal {L}. \tag {2}
$$
In other words, the empirical NTK links the representational dynamics  $\frac{\mathrm{d}z}{\mathrm{d}t}$  under gradient descent on the parameters  $\theta$ , and the "representational gradient"  $\nabla_{\mathcal{Z}}\mathcal{L}$ .
2.2 Implicit variance regularization in non-contrastive SSL
As a starting point for our analysis, we first express the relevant loss functions in the eigenbasis of the predictor network. We do this using a closed-form linear predictor as prescribed by DirectPred. In the following, we use  $\hat{z} = U^{\top}z$  to denote the representation expressed in the eigenbasis.
Lemma 1. (Euclidean and cosine loss in the predictor eigenspace) Let  $W_{\mathrm{P}}$  be a linear predictor set according to DirectPred with eigenvalues  $\lambda_{m}$ , and  $\hat{z}$  the representations expressed in the predictor's eigenbasis. Then the asymmetric Euclidean loss  $\mathcal{L}^{\mathrm{euc}}$  and cosine loss  $\mathcal{L}^{\mathrm{cos}}$  can be expressed as:
$$
\mathcal {L} ^ {\mathrm {e u c}} = \frac {1}{2} \sum_ {m} ^ {M} \left| \lambda_ {m} \hat {z} _ {m} ^ {(1)} - \operatorname {S G} \left(\hat {z} _ {m} ^ {(2)}\right) \right| ^ {2}, \tag {3}
$$
$$
\mathcal {L} ^ {\cos} = - \sum_ {m} ^ {M} \frac {\lambda_ {m} \hat {z} _ {m} ^ {(1)} \operatorname {S G} \left(\hat {z} _ {m} ^ {(2)}\right)}{\| D \hat {z} ^ {(1)} \| \| \operatorname {S G} \left(\hat {z} ^ {(2)}\right) \|}. \tag {4}
$$
for which we defer the simple proof to Appendix B. Rewriting the losses in the eigenbasis makes it clear that the asymmetric loss with DirectPred can be viewed as an implicit loss function in the predictor's eigenspace, where the variance of each mode naturally appears through the  $\lambda_{m}$  terms. In the following analysis, we will show how the learning dynamics implicitly regularize these variances  $\lambda_{m}$ . From Eq. (3) we directly see that  $\mathcal{L}^{\mathrm{euc}}$  is a sum of  $M$  terms, one for each eigenmode, which decouples the learning dynamics, a fact first noted by Tian et al. [17]. In contrast, the form of  $\mathcal{L}^{\mathrm{cos}}$
yields coupled dynamics due to the  $\| D\hat{z}^{(1)}\| = \sqrt{\sum_k(\lambda_k\hat{z}_k^{(1)})^2}$  term in the denominator. This coupling arises from the normalization of the representation vectors to the unit hypersphere when calculating the cosine distance. The normalization effectively removes one degree of freedom and, in the process, adds a dependence between all the representation dimensions (Fig. 1b and 1c).
To get an analytic handle on the evolution of the eigen-representations  $\hat{z}$  as the encoder learns, we first note that if training were to update the representations directly, instead of indirectly through updating the weights  $\theta$ , they would evolve along the following "representational gradients":
$$
\nabla_ {\hat {z} ^ {(1)}} \mathcal {L} ^ {\mathrm {e u c}} = \left(D \hat {\boldsymbol {z}} ^ {(1)} - \hat {\boldsymbol {z}} ^ {(2)}\right) D, \tag {5}
$$
$$
\nabla_ {\hat {z} ^ {(1)}} \mathcal {L} ^ {\cos} = - \frac {D \hat {\boldsymbol {z}} ^ {(2)}}{\| D \hat {\boldsymbol {z}} ^ {(1)} \| \| \hat {\boldsymbol {z}} ^ {(2)} \|} + \frac {\left(D \hat {\boldsymbol {z}} ^ {(1)}\right) ^ {\top} \hat {\boldsymbol {z}} ^ {(2)}}{\| D \hat {\boldsymbol {z}} ^ {(1)} \| ^ {3} \| \hat {\boldsymbol {z}} ^ {(2)} \|} D ^ {2} \hat {\boldsymbol {z}} ^ {(1)}. \tag {6}
$$
In practice, however, representations of different samples do not evolve independently along these gradients, but influence each other through parameter changes in  $\theta$ . This interdependence of representations and parameters are captured by the empirical NTK  $\Theta_t(\mathcal{X},\mathcal{X})$  (cf. Eq. (2)). Because the NTK is positive semi-definite, loosely speaking, gradient descent on the parameters changes representations "in the direction" of the above representational gradients.
To see this link more formally, we express the NTK in the eigenbasis as  $\hat{\Theta}_t(\mathcal{X},\mathcal{X}) = \nabla_\theta \hat{\mathcal{Z}}\nabla_\theta \hat{\mathcal{Z}}^\top$  where  $\hat{\mathcal{Z}} = \hat{z}_t(\mathcal{X}) = U^\top z_t(\mathcal{X})$ . Since we are concerned with the learning dynamics in this rotated basis, we will rewrite Eq. (2) for continuous-time gradient descent for a generic loss function  $\mathcal{L}$  as:
$$
\frac {\mathrm {d} \hat {z}}{\mathrm {d} t} = - \eta \hat {\Theta} _ {t} (\boldsymbol {x}, \mathcal {X}) \nabla_ {\hat {\mathcal {Z}}} \mathcal {L}. \tag {7}
$$
Note, that structurally these dynamics are the same as the embedding space dynamics in Eq. (2) but merely expressed in the predictor eigenbasis (see Lemma 2 in Appendix B for a derivation). Although  $\hat{\Theta}_t$  changes over time and is generally intractable in finite-width networks, it is positive semidefinite. This property guarantees that the cosine angle between the representational training dynamics under the parameter-space optimization of a neural network  $\frac{\mathrm{d}}{\mathrm{d}t}\hat{\mathcal{Z}}\propto -\hat{\Theta}_t\nabla_{\hat{\mathcal{Z}}}\mathcal{L}$  and the dynamics that would result from optimizing the representations  $\frac{\mathrm{d}}{\mathrm{d}t}\hat{\mathcal{Z}}\propto -\nabla_{\hat{\mathcal{Z}}}\mathcal{L}$  is non-negative:
$$
\left\langle - \nabla_ {\hat {Z}} \mathcal {L}, \frac {\mathrm {d} \hat {\mathcal {Z}}}{\mathrm {d} t} \right\rangle = \eta \left\langle \nabla_ {\hat {Z}} \mathcal {L}, \hat {\Theta} _ {t} \nabla_ {\hat {Z}} \mathcal {L} \right\rangle \geq 0.
$$
In other words, the representational updates due to network training lie within a 180-degree cone of the dynamics prescribed by Eqs. (5) and (6). This guarantee makes it possible to draw qualitative conclusions about asymptotic collective behavior, e.g., whether a network is bound to collapse or not, from analyzing the more tractable dynamics that follow the representational gradients  $\frac{\mathrm{d}}{\mathrm{d}t}\hat{\mathcal{Z}}\propto -\nabla_{\hat{\mathcal{Z}}}L$  of the transformed BYOL/SimSiam loss. For ease of analysis, we now consider linear networks with Gaussian i.i.d inputs, an important limiting case amenable for theoretical analysis [27]. In this setting
the empirical NTK becomes the identity and the simplified representational dynamics are exact, allowing us to fully characterize the representational dynamics for  $\mathcal{L}^{\mathrm{euc}}$  and  $\mathcal{L}^{\mathrm{cos}}$  in the following two theorems. In the proofs for these theorems, we show that the assumption of Gaussian inputs can be relaxed further.
Theorem 1. (Representational dynamics under  $\mathcal{L}^{\mathrm{euc}}$ ) For a linear network with i.i.d Gaussian inputs learning with  $\mathcal{L}^{\mathrm{euc}}$ , the representational dynamics of each mode  $m$  independently follow the gradient of the loss  $-\nabla_{\hat{z}}\mathcal{L}^{\mathrm{euc}}$ . More specifically, the dynamics uncouple and follow  $M$  independent differential equations:
$$
\frac {\mathrm {d} \hat {z} _ {m} ^ {(1)}}{\mathrm {d} t} = - \eta \frac {\partial \mathcal {L} ^ {\mathrm {e u c}}}{\partial \hat {z} _ {m} ^ {(1)}} (t) = \eta \lambda_ {m} \left(\hat {z} _ {m} ^ {(2)} - \lambda_ {m} \hat {z} _ {m} ^ {(1)}\right), \tag {8}
$$
which, after taking the expectation over augmentations yields the dynamics:
$$
\frac {\mathrm {d} \hat {z} _ {m}}{\mathrm {d} t} = \eta \lambda_ {m} (1 - \lambda_ {m}) \hat {z} _ {m}. \tag {9}
$$
We provide the proof in Appendix B and appreciate that  $\frac{\mathrm{d}}{\mathrm{d}t}\hat{z}_m$  has the same sign as  $\hat{z}_m$  whenever  $\lambda_{m} < 1$  and the opposite sign whenever  $\lambda_{m} > 1$ . These dynamics are convergent and approach an eigenvalue  $\lambda_{m}$  of one, thereby preventing collapse of mode  $m$ . Since the eigenmodes are orthogonal and uncorrelated, and the condition simultaneously holds for all modes, this ultimately prevents both representational and dimensional collapse [28]. Since the eigenvalues also correspond to the variance of the representations, the underlying mechanism constitutes an implicit form of variance regularization. Finally, we note that the above decoupling of the dynamics for the Euclidean loss has been described previously in Tian et al. [17].
Nevertheless, the representational dynamics are different for the commonly used cosine loss  $\mathcal{L}^{\mathrm{cos}}$ .
Theorem 2. (Representational dynamics under  $\mathcal{L}^{\mathrm{cos}}$ ) For a linear network with i.i.d Gaussian inputs trained with  $\mathcal{L}^{\mathrm{cos}}$ , the dynamics follow a system of  $M$  coupled differential equations:
$$
\frac {\mathrm {d} \hat {z} _ {m} ^ {(1)}}{\mathrm {d} t} = \eta \frac {\lambda_ {m}}{\| D \hat {z} ^ {(1)} \| ^ {3} \| \hat {z} ^ {(2)} \|} \sum_ {k \neq m} \lambda_ {k} \left(\lambda_ {k} \left(\hat {z} _ {k} ^ {(1)}\right) ^ {2} \hat {z} _ {m} ^ {(2)} - \lambda_ {m} \hat {z} _ {m} ^ {(1)} \hat {z} _ {k} ^ {(1)} \hat {z} _ {k} ^ {(2)}\right), \tag {10}
$$
and reach a regime in which the eigenvalues are comparable in magnitude. In this regime, the expected update over augmentations is well approximated by:
$$
\frac {\mathrm {d} \hat {z} _ {m}}{\mathrm {d} t} \approx \eta \lambda_ {m} \cdot \mathbb {E} \left[ \frac {\hat {z} _ {m} ^ {2}}{\| D \hat {\boldsymbol {z}} \| ^ {3}} \right] \cdot \mathbb {E} \left[ \frac {\hat {z} _ {m}}{\| \hat {\boldsymbol {z}} \|} \right] \cdot \sum_ {k \neq m} \lambda_ {k} (\lambda_ {k} - \lambda_ {m}), \tag {11}
$$
where we have assumed averages over augmentations. See Appendix B for the proof. Theorem 2 states that  $\frac{\mathrm{d}}{\mathrm{d}t}\hat{z}_m$  has the same or different sign as  $\hat{z}_m$  depending on the sign of the aggregate sum  $\sum_{k\neq m}\lambda_k(\lambda_k - \lambda_m)$ . This relation suggests that a steady state is only reached through mutual agreement when the non-zero eigenvalues are all equal. In contrast to the Euclidean case, there is no pre-specified target value (see Fig. 5 in Appendix A). Thus, the cosine loss also induces implicit variance regularization, but through a markedly different mechanism in which eigenmodes cooperate.
2.3 Stop-grad and predictor network are essential for implicit variance regularization.
We now extend our analysis to explain the known failure modes due to ablating the predictor or the stop-gradient for each distance metric. When we omit the stop-grad operator from  $\mathcal{L}^{\mathrm{euc}}$ , we have:
$$
\mathcal {L} _ {\mathrm {n o S G}} ^ {\mathrm {e u c}} = \frac {1}{2} \| W _ {\mathrm {P}} z ^ {(1)} - z ^ {(2)} \| ^ {2} \quad \Rightarrow \quad \frac {\mathrm {d} \hat {z} _ {m}}{\mathrm {d} t} = - \eta (1 - \lambda_ {m}) ^ {2} \hat {z} _ {m}, \tag {12}
$$
so that  $\frac{\mathrm{d}}{\mathrm{d}t}\hat{z}_m$  and  $\hat{z}_m$  always have opposite signs (see Appendix C for the derivation). This drives the representations toward zero with exponentially decaying eigenvalues, causing the notorious representational collapse [5]. Omitting the stop-grad operator from  $\mathcal{L}^{\mathrm{cos}}$  yields a nontrivial expression for the dynamics causing the largest eigenmode to diverge (see Appendix C). Interestingly, this is different from the collapse to zero inferred for the Euclidean distance.
Similarly, when removing the predictor network in the Euclidean loss case, the dynamics read:
$$
\mathcal {L} _ {\mathrm {n o P r e d}} ^ {\mathrm {e u c}} = \frac {1}{2} \| \boldsymbol {z} ^ {(1)} - \operatorname {S G} (\boldsymbol {z} ^ {(2)}) \| ^ {2} \quad \Rightarrow \quad \frac {\mathrm {d} \hat {z} _ {m}}{\mathrm {d} t} = 0, \tag {13}
$$
meaning that no learning updates occur. When the predictor is removed in the cosine loss case, the dynamics are:
$$
\mathcal {L} _ {\text {n o P r e d}} ^ {\cos} = - \frac {\left(\boldsymbol {z} ^ {(1)}\right) ^ {\top} \operatorname {S G} \left(\boldsymbol {z} ^ {(2)}\right)}{\| \boldsymbol {z} ^ {(1)} \| \| \operatorname {S G} \left(\boldsymbol {z} ^ {(2)}\right) \|} \Rightarrow \frac {\mathrm {d} \hat {z} _ {m}}{\mathrm {d} t} = \eta \sum_ {k \neq m} \left(\mathbb {E} \left[ \frac {\hat {z} _ {k} ^ {2}}{\| \hat {\boldsymbol {z}} \| ^ {3}} \right] \mathbb {E} \left[ \frac {\hat {z} _ {m}}{\| \hat {\boldsymbol {z}} \|} \right] - \mathbb {E} \left[ \frac {\hat {z} _ {m} \hat {z} _ {k}}{\| \hat {\boldsymbol {z}} \| ^ {3}} \right] \mathbb {E} \left[ \frac {\hat {z} _ {k}}{\| \hat {\boldsymbol {z}} \|} \right]\right). \tag {14}
$$
As we show in Appendix C, these dynamics also avoid collapse. However, the effective learning rates become impractically small without the eigenvalue factors from Eq. (11). We summarized the predicted dynamics of all settings in Table 1. Thus, our analysis provides mechanistic explanations for why stop-grad and predictor networks are required for avoiding collapse in non-contrastive SSL.