SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Summary of eigendynamics as predicted by our analysis for linear networks.
Loss	d\(\hat{z}_{m}\)/dt \(\propto\)	Predicted dynamics
\(\mathcal{L}^{\text{euc}}\)	\(\lambda_{m}(1-\lambda_{m})\)	λs converge to 1, large ones faster.
\(\mathcal{L}^{\text{euc}}_{\text{noSG}}\)	\(- (1 - \lambda_{m})^{2}\)	All λs collapse.
\(\mathcal{L}^{\text{euc}}_{\text{noPred}}\)	0	No learning updates.
\(\mathcal{L}^{\text{euc}}_{\text{iso}}\)	\((1 - \lambda_{m})\)	λs converge to 1 at homogeneous rates.
\(\mathcal{L}^{\text{cos}}\)	\(\lambda_{m}\sum_{k\neq m}\lambda_{k}(\lambda_{k}-\lambda_{m})\)	λs converge to equal values.
\(\mathcal{L}^{\text{cos}}_{\text{noSG}}\)	Appendix C	All λs diverge.
\(\mathcal{L}^{\text{cos}}_{\text{noPred}}\)	Appendix C	λs converge to equal values at low rates.
\(\mathcal{L}^{\text{cos}}_{\text{iso}}\)	\(\sum_{k\neq m}\lambda_{k}(\lambda_{k}-\lambda_{m})\)	λs converge to equal values at homogeneous rates.
2.4 Isotropic losses that equalize convergence across eigenmodes
In Eqs. (9) and (11) the eigenvalues appear as multiplicative learning rate modifiers in front of the difference terms that determine the fixed point. Hence, modes with larger eigenvalues converge faster than modes with smaller eigenvalues, reminiscent of previous theoretical work on supervised learning [27]. We hypothesized that the anisotropy in learning dynamics could lead to slow convergence for small eigenvalue modes or instability for large eigenvalues. To alleviate this issue, we designed alternative isotropic loss functions that equalize relaxation dynamics for all eigenmodes by exploiting the stop-grad function. Put simply, this involves taking the dynamics from Eqs. (8) and (10), removing the leading  $\lambda_{m}$  term, and deriving the loss function that would result in the desired dynamics. One such isotropic "IsoLoss" function for the Euclidean distance is:
$$
\mathcal {L} _ {\mathrm {i s o}} ^ {\mathrm {e u c}} = \frac {1}{2} \| \boldsymbol {z} ^ {(1)} - \operatorname {S G} \left(\boldsymbol {z} ^ {(2)} + \boldsymbol {z} ^ {(1)} - W _ {\mathrm {P}} \boldsymbol {z} ^ {(1)}\right) \| ^ {2}. \tag {15}
$$
We note that this IsoLoss has the same numerical value as  $\mathcal{L}^{\mathrm{euc}}$ , but the gradient flow is modified by placing the prediction inside the stop-grad and also adding and subtracting  $z^{(1)}$  inside and outside of the stop-grad. The associated idealized learning dynamics in our analytic framework are given by:
$$
\frac {\mathrm {d} \hat {z} _ {m}}{\mathrm {d} t} = \eta (1 - \lambda_ {m}) \hat {z} _ {m}, \tag {16}
$$
where the  $\lambda_{m}$  factor (cf. Eq. (9)) disappeared (Table 1). Similarly, for the cosine distance,
$$
\mathcal {L} _ {\mathrm {i s o}} ^ {\cos} = - \left(\boldsymbol {z} ^ {(1)}\right) ^ {\top} \operatorname {S G} \left(\frac {\boldsymbol {z} ^ {(2)}}{\| W _ {\mathrm {P}} \boldsymbol {z} ^ {(1)} \| \| \boldsymbol {z} ^ {(2)} \|}\right) + \frac {1}{2} \operatorname {S G} \left(\frac {\left(W _ {\mathrm {P}} \boldsymbol {z} ^ {(1)}\right) ^ {\top} \boldsymbol {z} ^ {(2)}}{\| W _ {\mathrm {P}} \boldsymbol {z} ^ {(1)} \| ^ {3} \| \boldsymbol {z} ^ {(2)} \|}\right) \| W _ {\mathrm {P}} ^ {1 / 2} \boldsymbol {z} ^ {(1)} \| ^ {2} \tag {17}
$$
is one possible IsoLoss, in which  $W_{\mathrm{P}}^{1/2} = U D^{1/2} U^{\top}$  with the square-root applied element-wise to the diagonal matrix  $D$ . While this IsoLoss does not preserve numerical equality with the original loss  $\mathcal{L}^{\cos}$ , it achieves the desired effect of removing the leading  $\lambda_{m}$  learning-rate modifier (cf. Table 1).
Figure 2: Evolution of representations (top) and eigenvalues (below) of  $W_{\mathrm{P}}$  throughout training with different loss functions. The representational trajectories correspond to training with  $M = 2$  for visualization and the points signify the final network outputs. The eigenvalues were computed with dimensions  $N = 15$  and  $M = 10$ . (a) Omitting the stop-grad leads to representational collapse in the Euclidean case (top), and diverging eigenvalues for the cosine case (bottom). (b) No learning occurs without the predictor with the Euclidean distance, but learning does occur with the cosine distance, although at low rates. Note the change in scale of the time-axis. (c) Optimizing the BYOL/SimSiam loss leads to isotropic representations under both distance metrics. (d) Optimizing IsoLoss has the same effect, but with uniform convergence dynamics for all eigenvalues for both distance metrics.
3 Numerical experiments
To validate our theoretical findings (cf. Table 1), we first simulated a small linear Siamese neural network as shown in Fig.1a, for which Theorems 1 and 2 hold exactly. We fed the network with independent standard Gaussian inputs, and generated pairs of augmentations using isotropic Gaussian perturbations of standard deviation  $\sigma = 0.1$ . We then trained the linear encoder with each configuration described above. Training the network with  $\mathcal{L}_{\mathrm{noSG}}^{\mathrm{euc}}$  resulted in collapse with exponentially decaying eigenvalues, whereas  $\mathcal{L}_{\mathrm{noSG}}^{\mathrm{cos}}$  succumbed to diverging eigenvalues as predicted (Fig. 2a). Training without the predictor caused vanishing updates for  $\mathcal{L}_{\mathrm{noPred}}^{\mathrm{euc}}$  and slow learning for  $\mathcal{L}_{\mathrm{noPred}}^{\mathrm{cos}}$ , in line with our analysis (Fig. 2b). Optimizing  $\mathcal{L}^{\mathrm{euc}}$ , the representations become increasingly isotropic with all the eigenvalues  $\lambda_{m}$  converging to one (Fig. 2c, top), whereas optimizing  $\mathcal{L}^{\mathrm{cos}}$  also resulted in the eigenvalues converging to the same value, but different from one (Fig. 2c, bottom). The anisotropy in the dynamics of different eigenmodes noted above is particularly striking in the case of the Euclidean distance (Fig. 2c). Training with  $\mathcal{L}_{\mathrm{iso}}^{\mathrm{euc}}$  and  $\mathcal{L}_{\mathrm{iso}}^{\mathrm{cos}}$  resulted in similar convergence properties as their non-isotropic counterparts, but the eigenmodes converged at more homogeneous rates (Fig. 2d). Finally, we confirmed that these findings were qualitatively similar in the corresponding nonlinear networks with ReLU activation (see Fig. 6 in Appendix A). Thus, our theoretical findings hold up in simple Siamese networks.
Figure 3: Learning dynamics for a ResNet-18 network trained with different loss functions. (a) Evolution of the eigenvalues of the representation correlation matrix during training for closed-form predictors as prescribed by DirectPred (left) and IsoLoss (center). Right: Standard BYOL with the nonlinear trainable predictor. For clarity, we plot only one in ten eigenvalues. Both  $\mathcal{L}^{\mathrm{euc}}$  and  $\mathcal{L}_{\mathrm{iso}}^{\mathrm{euc}}$  drive the eigenvalues to converge quickly and remain constant thereafter with relatively small fluctuations (note the logarithmic scale). BYOL results in the eigenvalues being spread across a large range of magnitudes. (b) Linear readout validation accuracy for  $\mathcal{L}^{\mathrm{euc}}$  and  $\mathcal{L}_{\mathrm{iso}}^{\mathrm{euc}}$  during the first 500 training epochs. IsoLoss accelerates the initial learning dynamics as predicted by the theory. (c) Same as in (a) but for the cosine distance.  $\mathcal{L}^{\mathrm{cos}}$  recruits few large eigenvalues, but drives them gradually to the same magnitude, whereas  $\mathcal{L}_{\mathrm{iso}}^{\mathrm{cos}}$  quickly recruits all eigenvalues and causing them to converge to an isotropic solution. In contrast, BYOL recruits eigenvalues in a step-wise manner. (d) Same as (b) but for the cosine distance.
3.1 Theory qualitatively captures dynamics in nonlinear networks and real-world datasets.
To investigate how well our theoretical analysis holds up in non-toy settings, we performed several self-supervised learning experiments on CIFAR-10, CIFAR-100 [29], STL-10 [30], and TinyImageNet [31]. We based our implementation<sup>1</sup> on the Solo-learn library [32], and used a ResNet-18 backbone [33] as the encoder and the cosine loss, unless mentioned otherwise (see Appendix D for details). As baselines for comparison, we trained the same backbone using BYOL with the nonlinear predictor and DirectPred with the closed-form linear predictor. We recorded the online readout accuracy of a linear classifier trained on frozen features following standard practice, evaluated either on the held-out validation or test set where available.
We found that the eigenvalue dynamics of the representational correlation matrix in the ResNet-18 closely mirrored the analytical predictions for the closed-form predictor. For Euclidean distances (Fig. 3a), the eigenvalues for DirectPred and IsoLoss converged to a small range of values around one. However, the dynamics for BYOL with a learnable nonlinear predictor deviated significantly with the eigenvalues distributed over a larger range. Consistent with our analysis, IsoLoss had faster initial dynamics for the eigenvalues which also resulted in a faster initial improvements in model performance (Fig. 3b). The faster learning with IsoLoss was even more evident for the cosine distance (Fig. 3c). Surprisingly, BYOL, which uses a nonlinear predictor also closely matched the predicted dynamics in the case of the cosine distance. Furthermore, the dynamics showed a stepwise learning phenomenon wherein eigenvalues are progressively recruited one-by-one, consistent with recent findings for other SSL methods [34]. Finally, IsoLoss exhibited faster initial learning (Fig. 3d), in agreement with our theoretical analysis. Thus, our theoretical analysis accurately predicts key properties of the eigenvalue dynamics in nonlinear networks trained on real-world datasets.
$^{1}$ Code is available at https://github.com/fmi-basel/implicit-var-reg
3.2 IsoLoss promotes eigenvalue recruitment and works without an EMA target network.
To further investigate the impact of IsoLoss on learning, we first verified that it does not have any adverse effects on downstream classification performance. We found that IsoLoss matched or outperformed DirectPred on all benchmarks (Table 2) when trained with an EMA target network as used in the original studies. Yet, it performed slightly worse than BYOL, which uses a nonlinear predictor and an EMA target network. Because EMA target networks are thought to amplify small eigenvalues [17], we speculated that IsoLoss may work without it. We repeated training for the closed-form predictor losses without EMA to test this idea. We found that  $\mathcal{L}_{\mathrm{iso}}^{\mathrm{cos}}$  was indeed robust to EMA removal. However, it caused a slight drop in performance (Table 2) and a notable reduction in the recruitment of small eigenvalues (see Fig. 7 in Appendix A). In contrast, optimizing the standard BYOL/SimSiam loss  $\mathcal{L}^{\mathrm{cos}}$  with the symmetric linear predictor was unstable, as reported previously [17]. Finally, we confirmed the above findings also hold for  $\alpha = 1$  (cf. Eq. (1)) as prescribed by DirectCopy [18] (see Table 3 in Appendix A). Thus, IsoLoss allows training without an EMA target network.
Table 2: Linear readout validation accuracy in  $\%$  ± stddev over five random seeds. The  $\dagger$  denotes crashed runs, known to occur with symmetric predictors like DirectPred [17]. Starred values * were taken from the Solo-learn library [32].
Model	EMA	CIFAR-10	CIFAR-100	STL-10	TinyImageNet
BYOL	Yes	92.6*	70.5*	91.7 ± 0.1	38.3 ± 1.5
SimSiam	No	90.7 ± 0.2	66.3 ± 0.4	87.5 ± 0.7	39.8 ± 0.6
DirectPred (α = 0.5)	Yes	92.0 ± 0.2	66.6 ± 0.5	88.8 ± 0.3	40.1 ± 0.5
No	12.1 ± 1.3†	1.6 ± 0.6†	10.4 ± 0.1†	1.3 ± 0.2†
IsoLoss (ours)	Yes	91.5 ± 0.2	69.0 ± 0.2	89.0 ± 0.3	44.8 ± 0.4
No	91.5 ± 0.2	64.3 ± 0.3	87.4 ± 0.1	40.4 ± 0.4
The above result suggests that IsoLoss promotes the recruitment of small eigenvalues in closed-form predictors. Another factor that has been implicated in suppressing recruitment is weight decay [18]. To probe how weight decay and IsoLoss affect small eigenvalue recruitment, we repeated the above simulations with EMA and different amounts of weight decay. Indeed, we observed less eigenvalue recruitment with increasing weight decay for DirectPred (Appendix A, Fig. 8a), but not for IsoLoss (Fig. 8b). However, for IsoLoss larger weight decay resulted in lower magnitudes of all eigenvalues. Hence, IsoLoss reduces the impact of weight decay on eigenvalue recruitment.
We provided a comprehensive analysis of the SSL representational dynamics in the eigenspace of closed-form linear predictor networks (i.e., DirectPred and DirectCopy) for both the Euclidean loss and the more commonly used cosine similarity. Our analysis revealed how asymmetric losses prevent representational and dimensional collapse through implicit variance regularization along orthogonal eigenmodes, thereby formally linking predictor-based SSL with explicit variance regularization approaches [6, 7, 14]. Our work provides a theory framework which further complements the growing body of work linking contrastive and non-contrastive SSL [24, 35-38].
We empirically validated the key predictions of our analysis in linear and nonlinear network models on several datasets, including CIFAR-10/100, STL-10, and TinyImageNet. Moreover, we found that the eigenvalues of the predictor network act as learning rate multipliers, causing anisotropic learning dynamics. We derived Euclidean and cosine IsoLosses, which counteract this anisotropy and enable closed-form linear predictor methods to work without an EMA target network, thereby further consolidating its presumed role in boosting small eigenvalues [17].
To our knowledge, this is the first work to comprehensively characterize asymmetric SSL learning dynamics for the cosine distance metric widely used in practice. However, our analysis rests on several assumptions. First, the analytic link through the NTK between gradient descent on parameters and the representational changes is an approximation in nonlinear networks. Moreover, we assumed Gaussian i.i.d inputs for proving Theorems 1 and 2. Although these assumptions generally do not
hold in nonlinear networks, our analysis qualitatively captures their overall learning behavior and predicts how networks respond to changes in the stop-grad placement.
In summary, we have provided a simple theoretical explanation of how asymmetric loss configurations prevent representational collapse in SSL and elucidate their inherent dependence on the placement of the stop-grad operation. We further demonstrated how the eigenspace framework allows crafting new loss functions with a distinct impact on the SSL learning dynamics. We provided one specific example of such loss functions, IsoLoss, which equalizes the learning dynamics in the predictor's eigenspace, resulting in faster initial learning and improved stability. In contrast to DirectPred, IsoLoss learns stably without an EMA target network. Our work thus lays out an effective framework for analyzing and developing new SSL loss functions.
This project was supported by the Swiss National Science Foundation [grant numbers PCEFP3_202981 and TMPFP3_210282], by EU's Horizon Europe Research and Innovation Programme (grant agreement number 101070374) funded through SERI (ref 1131-52302), and the Novartis Research Foundation. The authors declare no competing interests.
[1] Jean-Bastien Grill, Florian Strub, Florent Alché, Coretin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[5] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
[6] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
[7] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.
[8] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pages 456-473. Springer, 2022.
[9] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[10] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
[27] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120 [cond-mat, q-bio, stat], December 2013. URL http://arxiv.org/abs/1312.6120.
[28] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[30] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[31] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[32] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1-6, 2022. URL http://jmlr.org/papers/v23/21-1155.html.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. arXiv preprint arXiv:2303.15438, 2023.
[35] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:2206.02574, 2022.
[36] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.11508, 2022.
[37] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self-supervised learning by characterizing idealized representations. Advances in Neural Information Processing Systems, 35:11279-11296, 2022.
[38] Ashwini Pokle, Jinjin Tian, Yuchen Li, and Andrej Risteski. Contrasting the landscape of contrastive and non-contrastive learning. arXiv preprint arXiv:2203.15702, 2022.
Figure 4: Eigenspace alignment between  $C_z = \mathbb{E}_x\left[zz^\top\right]$  and a linear predictor  $W_{\mathrm{P}}$  trained with gradient descent. Following [1], we measure eigenspace alignment as the cosine between  $\pmb{u}_i$  and  $W_{\mathrm{P}}\pmb{u}_i$  for every eigenvector  $\pmb{u}_i$  of  $C_z$ . (a) Measured alignment for every eigenvector of  $C_z$  ordered by sorted eigenvalue indices over training epochs, when the network is trained with  $\mathcal{L}^{\mathrm{euc}}$ . (b) Same as (a) but for training with  $\mathcal{L}^{\mathrm{cos}}$ .
a
b
Figure 5: Comparison between the dynamics under Euclidean and cosine asymmetric losses for different initializations in a network with  $M = 2$  output neurons. (a) Observed dynamics of the eigenvalues in the two-neuron toy network under three different initializations. Both eigenvalues always converge to 1 regardless of the initialization. (b) Same as (a), but for the cosine distance. Under different initializations, the two eigenvalues converge to arbitrary, but equal, values.
Table 3: Linear readout validation accuracy in  $\% \pm$  stddev over five random seeds. The  $\dagger$  denotes crashed runs, known to occur with symmetric predictors like DirectPred [1].
Model	α	EMA	CIFAR-10	CIFAR-100	STL-10	TinyImageNet
DirectCopy	1	Yes	91.3 ± 0.2	68.7 ± 0.3	89.3 ± 0.2	45.3 ± 0.3
No	12.1 ± 0.6†	1.5 ± 0.5†	10.3 ± 0.1†	0.6 ± 0.1†
DirectPred	0.5	Yes	92.0 ± 0.2	66.6 ± 0.5	88.8 ± 0.3	40.1 ± 0.5
No	12.1 ± 1.3†	1.6 ± 0.6†	10.4 ± 0.1†	1.3 ± 0.2†
IsoLoss (ours)	1	Yes	91.5 ± 0.2	69.3 ± 0.2	89.1 ± 0.3	45.6 ± 0.9
No	91.4 ± 0.2	63.0 ± 0.3	86.9 ± 0.4	38.5 ± 0.5
0.5	Yes	91.5 ± 0.2	69.0 ± 0.2	89.0 ± 0.3	44.8 ± 0.4
No	91.5 ± 0.2	64.3 ± 0.3	87.4 ± 0.1	40.4 ± 0.4
Figure 6: Same as Fig. 2 but with a ReLU nonlinearity on the embeddings. We observe learning dynamics qualitatively similar to the linear network.
Figure 7: Eigenvalue dynamics for learning under IsoLoss  $(\mathcal{L}_{\mathrm{iso}}^{\mathrm{cos}})$  with and without the EMA target network. Removing the EMA results in markedly different dynamics with fewer eigenmodes recruited during training.
b
Figure 8: Effect of weight decay on eigenvalue recruitment for DirectPred and IsoLoss. (a) Evolution of eigenvalues during learning under DirectPred  $(\mathcal{L}^{\mathrm{cos}})$  with EMA and different amounts of weight decay. Decreasing weight decay correlates with the number of eigenvalues recruited during learning. (b) Same as (a) but for IsoLoss  $(\mathcal{L}_{\mathrm{iso}}^{\mathrm{cos}})$ . All eigenvalues are recruited independent of the strength of weight decay. However, the magnitude of the eigenvalues inversely correlates with the magnitude of weight-decay.
Lemma 1. (Euclidean and cosine loss in the predictor eigenspace) Let  $W_{\mathrm{P}}$  be a linear predictor set according to DirectPred with eigenvalues  $\lambda_{m}$ , and  $\hat{\mathbf{z}}$  the representations expressed in the predictor's eigenbasis. Then the asymmetric Euclidean loss  $\mathcal{L}^{\mathrm{euc}}$  and cosine loss  $\mathcal{L}^{\mathrm{cos}}$  can be expressed as:
Proof. Under DirectPred, the predictor is a symmetric matrix with eigendecomposition  $W_{\mathrm{P}} = UDU^{\top}$ . Since  $U$  is an orthogonal matrix, we also have  $UU^{\top} = I$  so that we can simplify the losses as follows:
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {e u c}} = \frac {1}{2} \| W _ {\mathrm {P}} \boldsymbol {z} ^ {(1)} - \operatorname {S G} (\boldsymbol {z} ^ {(2)}) \| ^ {2} \\ = \frac {1}{2} \| U D U ^ {\top} z ^ {(1)} - \operatorname {S G} (U U ^ {\top} z ^ {(2)}) \| ^ {2} \\ = \frac {1}{2} \| D \hat {\pmb {z}} ^ {(1)} - \mathrm {S G} (\hat {\pmb {z}} ^ {(2)}) \| ^ {2} \\ = \frac {1}{2} \sum_ {m} ^ {M} | \lambda_ {m} \hat {z} _ {m} ^ {(1)} - \mathrm {S G} (\hat {z} _ {m} ^ {(2)}) | ^ {2} \\ \end{array}
$$