SlowGuess/NeurIPS_2023_OCR huggingface dataset:
One line of research on MSE-NC in unconstrained/layer-peeled models aims to derive global minimizers of optimization problems associated with DNNs [17, 18, 48]. In particular, Tirer et al. [48] showed that global minimizers of the MSE loss with regularization of both  $\mathbf{H}$  and  $\mathbf{W}$  exhibit NC. Moreover, Ergen & Pilanci [17] showed that NC emerges in global minimizers of optimization problems with general convex loss in the context of the 2-layer-peeled model. In comparison to our work, these contributions do not consider the training dynamics of DNNs, i.e., they do not discuss whether and how the model converges to the optimal solution.
Another line of research on MSE-NC explicitly considers the dynamics of the unconstrained features models [21, 38]. In particular, Han et al. [21] considered the gradient flow of the unconstrained renormalized features along the "central path", where the classifier is assumed to take the form of the optimal least squares (OLS) solution for given features  $\mathbf{H}$ . Under this assumption, they derive a closed-form dynamics that implies NC. While they empirically show that DNNs are close to the central path in certain scenarios, they do not provide a theoretical justification for this assumption. The dynamics considered in their work is also distinct from the standard gradient flow dynamics of DNNs considered in our work. On the other hand, an earlier work by Mixon et al. [38] considered the gradient flow dynamics of the unconstrained features model, which is equivalent (up to rescaling) to the end-of-training dynamics (20) that we discuss in Sections 4.2 and 5.2. Their work relies on the linearization of these dynamics to derive a certain subspace, which appears to be an invariant subspace of the non-linearized unconstrained features model dynamics. Then they show that minimizers of the loss from this subspace exhibit NC. We note that, in terms of our paper, assuming that the unconstrained features model dynamics follow a certain invariant subspace means making assumptions on the end-of-training invariant (27). In comparison to these works, we make a step towards realistic DNNs dynamics by considering the standard gradient flow of DNNs simplified by Assumption 3.2 on the NTK structure, which is supported by the extensive research on NTK alignment [7, 9, 44, 45]. In our setting, the NTK captures the dependence of the features on the training data, which is missing in the unconstrained features model. Moreover, while other works focus only on the dynamics that converge to NC, we show that DNNs with MSE loss may not exhibit NC in certain settings, and the invariant of the dynamics (26) characterizes the difference between models that do and do not converge to NC.
Notably, works by Poggio & Liao [41] adopt a model different from the unconstrained features model to analyze gradient flow of DNNs. They consider the dynamics of homogeneous DNNs, in particular ReLU networks without biases, with normalization of the weights matrices and weights regularization. The goal of weights normalization in their model is to imitate the effects of batch normalization in DNNs training. In this model, certain fixed points of the gradient flow exhibit NC. While the approach taken in their work captures the dependence of the features on the data and the DNN's parameters, it fundamentally relies on the homogeneity of the DNN's output function. However, most DNNs that exhibit NC in practice are not homogeneous due to biases and skip-connections.
NC and local elasticity. A recent extensive survey of NC literature [33] discussed local elasticity as a possible mechanism behind the emergence of NC, which has not been sufficiently explored up until now. One of the few works in this research direction is by Zhang et al. [53], who analyzed the so-called locally-elastic stochastic differential equations (SDEs) and showed the emergence of NC in their solutions. They model local elasticity of the dynamics through an effect matrix, which has only two distinct values: a larger intra-class value and a smaller inter-class value. These values characterize how much influence samples from one class have on samples from other classes in the SDEs. While the aim of their work is to imitate DNNs' training dynamics through SDEs, the authors
do not provide any explicit connection between their dynamics and real gradient flow dynamics of DNNs. On the other hand, we derive our dynamics directly from the gradient flow equations and connect local elasticity to the NTK, which is a well-studied object in the deep learning theory.
Another work by Tirer et al. [50] provided a perturbation analysis of NC to study "inexact collapse". They considered a minimization problem with MSE loss, regularization of  $\mathbf{H}$  and  $\mathbf{W}$ , and additional regularization of the distance between  $\mathbf{H}$  and a given matrix of initial features. In the "near-collapse" setting, i.e., when the initial features are already close to collapse, they showed that the optimal features can be obtained from the initial features by a certain linear transformation with a block structure, where the intra-class effects are stronger than the inter-class ones. While this transformation matrix resembles the block-structured effect matrices in locally-elastic training dynamics, it does not originate from the gradient flow dynamics of DNNs and is not related to the NTK.
Proof of Theorem 4.1. We will first derive the dynamics of  $h_s(x_i^c)$ , which is the  $s$ -th component of the last-layer features vector on sample  $x_i^c \in X$  from class  $c \in [C]$ . Let  $\mathbf{w} \in \mathbb{R}^P$  be the trainable parameters of the network stretched into a single vector. Then its gradient flow dynamics is given by
$$
\dot {\mathbf {w}} = - \nabla_ {\mathbf {w}} \mathcal {L} (f) = - \sum_ {k = 1} ^ {C} \sum_ {i ^ {\prime} = 1} ^ {N} \left(f (X) _ {k i ^ {\prime}} - \mathbf {Y} _ {k i ^ {\prime}}\right) \nabla_ {\mathbf {w}} f (X) _ {k i ^ {\prime}}, \tag {31}
$$
where  $\nabla_{\mathbf{w}}f(X)_{ki'} \in \mathbb{R}^P$  is the component of the DNN's Jacobian corresponding to output neuron  $k$  and the input sample  $x_{i'}^{c'}$ . Since entries of  $f(X)$  can be written as
$$
f (X) _ {k i ^ {\prime}} = \sum_ {s ^ {\prime} = 1} ^ {n} \mathbf {W} _ {k s ^ {\prime}} \mathbf {H} _ {s ^ {\prime} i ^ {\prime}} + \mathbf {b} _ {k} = \sum_ {s ^ {\prime} = 1} ^ {n} \mathbf {W} _ {k s ^ {\prime}} h _ {s ^ {\prime}} \left(x _ {i ^ {\prime}} ^ {c ^ {\prime}}\right) + \mathbf {b} _ {k}, \tag {32}
$$
$$
\dot {\mathbf {w}} = - \sum_ {k = 1} ^ {C} \sum_ {i ^ {\prime} = 1} ^ {N} \sum_ {s ^ {\prime} = 1} ^ {n} (f (X) _ {k i ^ {\prime}} - \mathbf {Y} _ {k i ^ {\prime}}) \nabla_ {\mathbf {w}} \left(\mathbf {W} _ {k s ^ {\prime}} h _ {s ^ {\prime}} \left(x _ {i ^ {\prime}} ^ {c ^ {\prime}}\right) + \mathbf {b} _ {k}\right). \tag {33}
$$
By chain rule, we have  $\dot{h}_s(x_i^c) = \langle \nabla_{\mathbf{w}}h_s(x_i^c),\dot{\mathbf{w}}\rangle$  . Then, taking into account that
$$
\left\langle \nabla_ {\mathbf {w}} h _ {s} \left(x _ {i} ^ {c}\right), \nabla_ {\mathbf {w}} \left(\mathbf {W} _ {k s ^ {\prime}} h _ {s ^ {\prime}} \left(x _ {i ^ {\prime}} ^ {c ^ {\prime}}\right) + \mathbf {b} _ {k}\right) \right\rangle = \mathbf {W} _ {k s ^ {\prime}} \left\langle \nabla_ {\mathbf {w}} h _ {s} \left(x _ {i} ^ {c}\right), \nabla_ {\mathbf {w}} h _ {s ^ {\prime}} \left(x _ {i ^ {\prime}} ^ {c ^ {\prime}}\right) \right\rangle , \tag {34}
$$
and that  $\langle \nabla_{\mathbf{w}}h_s(x_i^c),\nabla_{\mathbf{w}}h_{s'}(x_{i'}^{c'})\rangle = \Theta_{s,s'}^h (x_i^c,x_{i'}^{c'})$  by definition of  $\Theta^h$  , we have
$$
\dot {h} _ {s} \left(x _ {i} ^ {c}\right) = - \sum_ {k = 1} ^ {C} \sum_ {i ^ {\prime} = 1} ^ {N} \sum_ {s ^ {\prime} = 1} ^ {n} \left(f (X) _ {k i ^ {\prime}} - \mathbf {Y} _ {k i ^ {\prime}}\right) \mathbf {W} _ {k s ^ {\prime}} \Theta_ {s, s ^ {\prime}} ^ {h} \left(x _ {i} ^ {c}, x _ {i ^ {\prime}} ^ {c ^ {\prime}}\right). \tag {35}
$$
Now by Assumption 3.2 we have  $\Theta_{s,s'}^h = 0$  if  $s \neq s'$ . Therefore, the above expression simplifies to
$$
\begin{array}{l} \dot {h} _ {s} \left(x _ {i} ^ {c}\right) = - \sum_ {i ^ {\prime} = 1} ^ {N} \Theta_ {s, s} ^ {h} \left(x _ {i} ^ {c}, x _ {i ^ {\prime}} ^ {c ^ {\prime}}\right) \sum_ {k = 1} ^ {C} \left(f (X) _ {k i ^ {\prime}} - \mathbf {Y} _ {k i ^ {\prime}}\right) \mathbf {W} _ {k s} \\ = - \sum_ {i ^ {\prime} = 1} ^ {N} [ \mathbf {W} ^ {\top} (\mathbf {W H} + \mathbf {b 1} _ {N} ^ {\top} - \mathbf {Y}) ] _ {s i ^ {\prime}} \Theta_ {s, s} ^ {h} (x _ {i} ^ {c}, x _ {i ^ {\prime}} ^ {c ^ {\prime}}). \\ \end{array}
$$
To express  $\dot{\mathbf{H}} = \left[\dot{h}_s(x_i^c)\right]_{s,i} \in \mathbb{R}^{n \times N}$  in matrix form, it remains to express  $\Theta_{s,s}^h (x_i^c,x_{i'}^{c'})$  as the  $(i',i)$ -th entry of some matrix. We will separate the sum into three cases: 1)  $i = i'$ , 2)  $i \neq i'$  and  $c = c'$ , and 3)  $c \neq c'$ . According to Assumption 3.2, the first case corresponds to the multiple of identity  $\kappa_d\mathbb{I}_N$ . The second corresponds to the block matrix of size  $m$  with zeros on the diagonal,
which can be written as  $\kappa_c(\mathbf{Y}^\top \mathbf{Y} - \mathbb{I}_N)$ . The third matrix equals to  $\kappa_n(\mathbf{1}_N\mathbf{1}_N^\top - \mathbf{Y}^\top \mathbf{Y})$ . Therefore we can express the dynamics of  $\mathbf{H}$  as follows:
$$
\begin{array}{l} \dot {\mathbf {H}} = - \left[ \mathbf {W} ^ {\top} \left(\mathbf {W H} + \mathbf {b} \mathbf {1} _ {N} ^ {\top} - \mathbf {Y}\right) \right] \left[ \kappa_ {d} \mathbb {I} + \kappa_ {c} \left(\mathbf {Y} ^ {\top} \mathbf {Y} - \mathbb {I}\right) + \kappa_ {n} \left(\mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} - \mathbf {Y} ^ {\top} \mathbf {Y}\right) \right] \\ = - \left(\kappa_ {d} - \kappa_ {c}\right) \mathbf {W} ^ {\top} \left(\mathbf {W H} + \mathbf {b 1} _ {N} ^ {\top} - \mathbf {Y}\right) \\ - \left(\kappa_ {c} - \kappa_ {n}\right) \mathbf {W} ^ {\top} \left(\mathbf {W H Y} ^ {\top} \mathbf {Y} + m \mathbf {b 1} _ {N} ^ {\top} - m \mathbf {Y}\right) \\ - \kappa_ {n} \mathbf {W} ^ {\top} \left(\mathbf {W H 1} _ {N} \mathbf {1} _ {N} ^ {\top} + N \mathbf {b 1} _ {N} ^ {\top} - \frac {N}{C} \mathbf {1} _ {C} \mathbf {1} _ {N} ^ {\top}\right). \\ \end{array}
$$
Now we notice that  $\mathbf{H}\mathbf{Y}^{\top}\mathbf{Y} / m$  is the matrix of stacked class means repeated  $m$  times each and  $\mathbf{H}\mathbf{1}_N\mathbf{1}_N^\top / N$  is a matrix of the global mean repeated  $N$  times. Therefore, we have
$$
\mathbf {W H Y} ^ {\top} \mathbf {Y} + m \mathbf {b 1} _ {N} ^ {\top} - m \mathbf {Y} = m \mathbf {R} _ {\text {c l a s s}},
$$
$$
\mathbf {W H 1} _ {N} \mathbf {1} _ {N} ^ {\top} + N \mathbf {b} \mathbf {1} _ {N} ^ {\top} - \frac {N}{C} \mathbf {1} _ {C} \mathbf {1} _ {N} ^ {\top} = N \mathbf {R} _ {\mathrm {g l o b a l}}
$$
according to the definitions of global and class-mean residuals in (18) and (17).
The expressions for the gradient flow dynamics of  $\mathbf{W}$  and  $\mathbf{b}$  follow directly from the derivatives of  $f(X)$  w.r.t.  $\mathbf{W}$  and  $\mathbf{b}$ . This completes the proof.
B.2 Proof of Theorem 5.1
Proof of Theorem 5.1. Recall from (23) in Section 5.1 that we have the following decomposition
$$
\mathbf {H Q} = \sqrt {m} \left[ \mathbf {H} _ {1}, \mathbf {H} _ {2} \right], \quad \mathbf {H} _ {1} = \frac {1}{\sqrt {m}} \mathbf {H Q} _ {1}, \quad \mathbf {H} _ {2} = \frac {1}{\sqrt {m}} \mathbf {H Q} _ {2}
$$
with orthogonal  $\mathbf{Q} = [\mathbf{Q}_1, \mathbf{Q}_2] \in \mathbb{R}^{N \times N}$ . We now artificially add  $\mathbf{QQ}^\top (= \mathbb{I}_N)$  to the dynamics (19) in Theorem 4.1 and obtain
$$
\left\{ \begin{array}{l l} \dot {\mathbf {H}} \mathbf {Q} = & - \left(\kappa_ {d} - \kappa_ {c}\right) \mathbf {W} ^ {\top} \left(\mathbf {W H Q} + \mathbf {b 1} _ {N} ^ {\top} \mathbf {Q} - Y \mathbf {Q}\right) \\ & - \left(\kappa_ {c} - \kappa_ {n}\right) m \mathbf {W} ^ {\top} \left(\frac {1}{m} \mathbf {W H Q Q} ^ {\top} \mathbf {Y} ^ {\top} \mathbf {Y Q} + \mathbf {b 1} _ {N} ^ {\top} \mathbf {Q} - Y \mathbf {Q}\right) \\ & - \kappa_ {n} N \mathbf {W} ^ {\top} \left(\frac {1}{N} \mathbf {W H Q Q} ^ {\top} \mathbf {1} _ {N} \mathbf {1} _ {N} ^ {\top} \mathbf {Q} + \mathbf {b 1} _ {N} ^ {\top} \mathbf {Q} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {N} ^ {\top} \mathbf {Q}\right) \\ \dot {\mathbf {W}} = & - (\mathbf {W H Q} + \mathbf {b 1} _ {N} ^ {\top} \mathbf {Q} - Y \mathbf {Q}) \mathbf {Q} ^ {\top} \mathbf {H} ^ {\top} \\ \dot {\mathbf {b}} = & - (\mathbf {W H Q} + \mathbf {b 1} _ {N} ^ {\top} \mathbf {Q} - Y \mathbf {Q}) \mathbf {Q} ^ {\top} \mathbf {1} _ {N}. \end{array} \right. \tag {36}
$$
Let us simplify the expression. Since  $\mathbf{Q}_1 = \frac{1}{\sqrt{m}}\mathbb{I}_C\otimes \mathbf{1}_m$  and  $\mathbf{Q}_2 = \mathbb{I}_C\otimes \tilde{\mathbf{Q}}_2$ , we have
$$
\mathbf {1} _ {N} ^ {\top} \mathbf {Q} = \sqrt {m} \left[ \mathbf {1} _ {C} ^ {\top}, \mathbb {O} \right], \quad \mathbf {Y} \mathbf {Q} = \sqrt {m} \left[ \mathbb {I} _ {C}, \mathbb {O} \right]. \tag {37}
$$
Plugging (37) into (36), we see the dynamics can be decomposed into
$$
\left\{ \begin{array}{l l} \dot {\mathbf {H}} _ {1} = & - \left(\kappa_ {d} - \kappa_ {c}\right) \mathbf {W} ^ {\top} \left(\mathbf {W} \mathbf {H} _ {1} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \mathbb {I} _ {C}\right) \\ & - \left(\kappa_ {c} - \kappa_ {n}\right) m \mathbf {W} ^ {\top} \left(\mathbf {W} \mathbf {H} _ {1} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \mathbb {I} _ {C}\right) \\ & - \kappa_ {n} N \mathbf {W} ^ {\top} \left(\frac {1}{C} \mathbf {W} \mathbf {H} _ {1} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \\ \dot {\mathbf {H}} _ {2} = & - \left(\kappa_ {d} - \kappa_ {c}\right) \mathbf {W} ^ {\top} \mathbf {W} \mathbf {H} _ {2} \\ \dot {\mathbf {W}} = & - m \left(\mathbf {W} \mathbf {H} _ {1} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \mathbb {I} _ {C}\right) \mathbf {H} _ {1} ^ {\top} - m \mathbf {W} \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} \\ \dot {\mathbf {b}} = & - m \left(\mathbf {W} \mathbf {H} _ {1} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \mathbb {I} _ {C}\right) \mathbf {1} _ {C}. \end{array} \right. \tag {38}
$$
To further simplify (38), we define the following quantities
$$
\mu_ {\text {s i n g l e}} := \kappa_ {d} - \kappa_ {c}, \quad \mu_ {\text {c l a s s}} := \mu_ {\text {s i n g l e}} + m \left(\kappa_ {c} - \kappa_ {n}\right), \quad \mathbf {R} _ {1} := \mathbf {W H} _ {1} + \mathbf {b 1} _ {C} ^ {\top} - \mathbb {I} _ {C}. \tag {39}
$$
Notice that  $\mu_{\mathrm{single}}$  and  $\mu_{\mathrm{class}}$  are the two largest eigenvalues of the block-structured kernel  $\Theta_{s,s}^{h}(X)$  (see Table 1 for the eigendecomposition of a block-structured matrix), and  $\mathbf{R}_1$  is a matrix of the stacked class-mean residuals, which is also defined in (17). The dynamics (38) simplifies to
$$
\left\{ \begin{array}{l l} \dot {\mathbf {H}} _ {1} = & - \mathbf {W} ^ {\top} \left(\mu_ {\text {c l a s s}} \mathbf {R} _ {1} + \kappa_ {n} N \left(\frac {1}{C} \mathbf {W H} _ {1} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right)\right) \\ \dot {\mathbf {H}} _ {2} = & - \mu_ {\text {s i n g l e}} \mathbf {W} ^ {\top} \mathbf {W H} _ {2} \\ \dot {\mathbf {W}} = & - m \left(\mathbf {R} _ {1} \mathbf {H} _ {1} ^ {\top} - \mathbf {W H} _ {2} \mathbf {H} _ {2} ^ {\top}\right) \\ \dot {\mathbf {b}} = & - m \mathbf {R} _ {1} \mathbf {1} _ {C}. \end{array} \right. \tag {40}
$$
It remains to simplify the expression for  $\dot{\mathbf{H}}_1$ . By using the relation
$$
\frac {1}{C} \mathbf {W H} _ {1} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top} = \frac {1}{C} \mathbf {R} _ {1} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}, \tag {41}
$$
we can deduce that the dynamics for  $\dot{\mathbf{H}}_1$  in (40) can be expressed as (recalling that  $N = mC$ )
$$
\dot {\mathbf {H}} _ {1} = - \mathbf {W} ^ {\top} \mathbf {R} _ {1} \left(\mu_ {\text {c l a s s}} \mathbb {I} + \kappa_ {n} m \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right). \tag {42}
$$
We notice that  $(\mathbb{I}_C + \frac{\kappa_n m}{\mu_{\mathrm{class}}}\mathbf{1}_C\mathbf{1}_C^\top)^{-1} = \mathbb{I}_C - \alpha \mathbf{1}_C\mathbf{1}_C^\top$ , where  $\alpha := \frac{\kappa_n m}{\mu_{\mathrm{class}} + C\kappa_n m}$ . Then we can derive the invariant of the training dynamics by direct computation of the time-derivative  $\dot{\mathbf{E}}$ , where
$$
\mathbf {E} := \frac {1}{m} \mathbf {W} ^ {\top} \mathbf {W} - \frac {1}{\mu_ {\text {c l a s s}}} \mathbf {H} _ {1} \left(\mathbb {I} _ {C} - \alpha \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \mathbf {H} _ {1} ^ {\top} - \frac {1}{\mu_ {\text {s i n g l e}}} \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} \tag {43}
$$
Since  $\dot{\mathbf{E}} = \mathbb{0}$ , we get that the quantity  $\mathbf{E}$  remains constant in time. This completes the proof.
B.3 Proof of Theorem 5.2
We divide the proof into two main parts: the first one shows the emergence of NC1, and the second one shows NC2-4.
(NC1). Following the analysis in Section 3, the dynamics eventually enters the end of training phase (see Section 4.1). Then the dynamics in Theorem 5.1 simplifies to the following form:
$$
\left\{ \begin{array}{l} \dot {\mathbf {H}} _ {1} = \mathbb {O} \\ \dot {\mathbf {H}} _ {2} = - \mu_ {\text {s i n g l e}} \mathbf {W} ^ {\top} \mathbf {W} \mathbf {H} _ {2} \\ \dot {\mathbf {W}} = - m \mathbf {W} \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} \\ \dot {\mathbf {b}} = \mathbb {O} \end{array} \right. \tag {44}
$$
As we note in Section 4, this dynamics is similar to the gradient flow of the unconstrained features models and is an instance of the class of hyperbolic dynamics, which is discussed in Section 5.2. During this phase the quantity
$$
\tilde {\mathbf {E}} := \mu_ {\text {s i n g l e}} \mathbf {W} ^ {\top} \mathbf {W} - m \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} = m \mu_ {\text {s i n g l e}} \left(\mathbf {E} + \frac {1}{\mu_ {\text {c l a s s}}} \mathbf {H} _ {1} \left(\mathbb {I} - \alpha \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \mathbf {H} _ {1} ^ {\top}\right) \tag {45}
$$
does not change in time. Hence we can decouple the dynamic using the invariant as follows:
$$
\left\{ \begin{array}{l} \dot {\mathbf {H}} _ {2} = - \mu_ {\text {s i n g l e}} \left(\tilde {\mathbf {E}} + m \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top}\right) \mathbf {H} _ {2} \\ \dot {\mathbf {W}} = - \mathbf {W} \left(\mu_ {\text {s i n g l e}} \mathbf {W} ^ {\top} \mathbf {W} - \tilde {\mathbf {E}}\right) \end{array} \right. \tag {46}
$$
Since  $\mathbf{E}$  is p.s.d (or zero, as a special case),  $\tilde{\mathbf{E}}$  is p.s.d as well, and the eigendecomposition of the invariant is given by  $\tilde{\mathbf{E}} = \sum_{k}c_{k}v_{k}v_{k}^{\top}$  for some coefficients  $c_{k}\geq 0$  and a set of orthonormal vectors  $v_{k}\in \mathbb{R}^{n}$ . Then we also have  $\mathbf{H}_2\mathbf{H}_2^\top = \sum_{k,l}\alpha_{kl}v_kv_l^\top$ , where  $\alpha_{kl}$  are symmetric (i.e.  $\alpha_{kl} = \alpha_{lk}$ ) and  $\alpha_{kk}\geq 0$  for all  $k = 1,\ldots n$  (since  $\mathbf{H}_2\mathbf{H}_2^\top$  is symmetric and p.s.d.). Note that coefficients  $c_{k}$  here are constant while coefficients  $\alpha_{kl}$  are time-dependent. Let us then write the dynamics for  $\alpha_{kl}$  using the dynamics of  $\mathbf{H}_2\mathbf{H}_2^\top$ :
$$
\left(\mathbf {H} _ {2} \dot {\mathbf {H}} _ {2} ^ {\top}\right) = - \tilde {\mathbf {E}} \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} - \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} \tilde {\mathbf {E}} - 2 \left(\mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top}\right) ^ {2} \tag {47}
$$
Then for the elements of  $\alpha$  we have:
$$
\dot {\alpha} _ {k l} = - \alpha_ {k l} \left(c _ {k} + c _ {l}\right) - 2 \sum_ {j} \alpha_ {k j} \alpha_ {j l} \tag {48}
$$
For the diagonal elements  $\alpha_{kk}$ , this gives:
$$
\dot {\alpha} _ {k k} = - 2 c _ {k} \alpha_ {k k} - 2 \sum_ {j} \alpha_ {k j} ^ {2} \tag {49}
$$
Since  $c_{k}\geq 0,\alpha_{kk}\geq 0$  and  $\alpha_{kj}^2\geq 0$  , we get that
$$
\alpha_ {k k} \xrightarrow [ t \rightarrow \infty ]{} 0 \quad \forall k \tag {50}
$$
And, therefore, all the non-diagonal elements also tend to zero. Thus, we get that
$$
\mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} \underset {t \rightarrow \infty} {\longrightarrow} \mathbb {O} \tag {51}
$$
$$
\mathrm {H} _ {2} \xrightarrow [ t \rightarrow \infty ]{} 0 \tag {52}
$$
Now we notice that from the expression for  $\mathbf{H}_2$  in (24) it follows that  $\mathbf{H}_2 = \mathbb{O}$  implies variability collapse, since it means that all the feature vectors within the same class are equal. Indeed,  $\mathbf{H}^{(c)}\tilde{\mathbf{Q}}_2 = \mathbb{O} \in \mathbb{R}^{n \times (m-1)}$  means that there is a set of  $m-1$  orthogonal vectors, which are all also orthogonal to  $[h_i(x_1^c), \ldots, h_i(x_m^c)]$  for any  $i = 1, \ldots, n$ , where  $x_i^c$  are inputs from class  $c$ . However, there is only one vector (up to a constant) orthogonal to all the columns of  $\tilde{\mathbf{Q}}_2$  in  $\mathbb{R}^m$  and this vector is  $\mathbf{1}_m$ . Therefore,  $[h_i(x_1^c), \ldots, h_i(x_m^c)] = \gamma \mathbf{1}_m$  for some constant  $\gamma$  for any  $i = 1, \ldots, n$ . Thus, we indeed have  $h(x_1^c) = \cdots = h(x_m^c)$ , which constitutes variability collapse within classes.
(NC2-4). Set  $\beta = \frac{1}{C}$ . We first show that zero global feature mean implies  $\mathbf{b} = \beta \mathbf{1}_C$ . At the end of training, since  $\mathbf{R}_1 = \mathbb{0}$ , we have
$$
\mathbf {W} \mathbf {H} _ {1} + \mathbf {b} \mathbf {1} _ {C} ^ {\top} = \mathbb {I} _ {C} \tag {53}
$$
On the other hand, zero global mean implies  $\mathbf{H}_1\mathbf{1}_C = C\langle h\rangle = \mathbb{0}$ . Then multiplying (53) by  $\mathbf{1}_C$  on the right, we get the desired expression for the biases. Given the zero global mean, we have
$$
\frac {1}{m} \mathbf {W} ^ {\top} \mathbf {W} - \frac {1}{\mu_ {\text {c l a s s}}} \mathbf {H} _ {1} \mathbf {H} _ {1} ^ {\top} - \frac {1}{\mu_ {\text {s i n g l e}}} \mathbf {H} _ {2} \mathbf {H} _ {2} ^ {\top} = \mathbf {E} - \frac {\alpha m C ^ {2}}{\mu_ {\text {c l a s s}}} \langle h \rangle \langle h \rangle^ {\top} = \mathbf {E} \tag {54}
$$
By the proof of NC1,  $\mathbf{H}_2\to \mathbb{O}$ . Together with the assumption that  $\mathbf{E}$  is proportional to the limit of  $\mathbf{W}^{\top}\mathbf{W}$  (or zero, as a special case), we obtain
$$
\mu_ {\text {c l a s s}} \mathbf {W} ^ {\top} \mathbf {W} - m \mathbf {H} _ {1} \mathbf {H} _ {1} ^ {\top} \rightarrow \gamma \mathbf {W} ^ {\top} \mathbf {W} \tag {55}
$$
for some  $\gamma \geq 0$ . Note that since  $\mathbf{H}_1\mathbf{H}_1^\top$  is p.s.d. this implies  $\tilde{\lambda}_c \coloneqq \mu_{\mathrm{class}} - \gamma \geq 0$ . By multiplying the left and right with appropriate factors, we have
$$
\left\{\begin{array}{l}\mathbf {H} _ {1} ^ {\top} \left(\tilde {\lambda} _ {c} \mathbf {W} ^ {\top} \mathbf {W} - m \mathbf {H} _ {1} \mathbf {H} _ {1} ^ {\top}\right) \mathbf {H} _ {1} \rightarrow \mathbb {O}\\\mathbf {W} \left(\tilde {\lambda} _ {c} \mathbf {W} ^ {\top} \mathbf {W} - m \mathbf {H} _ {1} \mathbf {H} _ {1} ^ {\top}\right) \mathbf {W} ^ {\top} \rightarrow \mathbb {O}.\end{array}\right. \tag {56}
$$
Consequently (according to (53))
$$
\left\{\begin{array}{l}\tilde {\lambda} _ {c} \left(\mathbb {I} _ {C} - \beta \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) ^ {2} - m \left(\mathbf {H} _ {1} ^ {\top} \mathbf {H} _ {1}\right) ^ {2} \rightarrow \mathbb {O}\\\tilde {\lambda} _ {c} \left(\mathbf {W W} ^ {\top}\right) ^ {2} - \left(\mathbb {I} _ {C} - \beta \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) ^ {2} \rightarrow \mathbb {O}\end{array}\right. \tag {57}
$$
Since both  $\mathbf{WW}^{\top}$  and  $\mathbf{H}_1^\top \mathbf{H}_1$  are p.s.d., we have