SlowGuess/NeurIPS_2023_OCR huggingface dataset:
unless  $x$  and  $y$  are aligned. In that case, the optimal value is  $\frac{b^T b}{y^T y}$ , and we require that  $a, b$  are aligned and  $\| y \| / \| x \| = \| b \| / \| a \|$  for the problem to be feasible. Moreover, the matrix  $W^{*} X$  at the optimal  $W^{*}$  is non-negative and thus  $W^{*} X = \sigma(W^{*} X)$ . If the matrix  $X$  is orthogonal, then the rows of the optimal  $W^{*}$  are either 0, or aligned with one of the columns of  $X$ .
Proof sketch of Lemma 6. The optimization problem is separable with respect to the rows of  $W$ , hence we can solve for each row separately. As the problem is convex and has linear constraints, the KKT conditions are necessary and sufficient for optimality. By solving such conditions explicitly and computing the objective value, the result follows. The details are deferred to Appendix A.
Now, we can plug (4) into the outer optimization problem, where we solve for the output matrix  $A$  given the constraint on singular values. We only optimize over the numerator of (4), as the denominator does not depend on the variable.
Lemma 7. Let  $X$  be a fixed entry-wise non-negative rank-2 matrix (the rank-1 case is treated separately later) with columns  $x, y$  and  $A = \sigma(WX)$ . The minimum of the following optimization problem over this non-negative matrix  $A$  with columns  $a, b$ :
$$
\min  _ {A} b ^ {T} b x ^ {T} x - 2 a ^ {T} b x ^ {T} y + a ^ {T} a y ^ {T} y, \quad \text {s . t .} \mathcal {S} (A) = \left\{s _ {1}, s _ {2} \right\} \tag {5}
$$
is  $\frac{1}{2} (s_1^2 + s_2^2)(\tilde{s}_1^2 - \tilde{s}_2^2) + \frac{1}{2} (s_1^2 - s_2^2)(\tilde{s}_1^2 - \tilde{s}_2^2)$ , where  $\{\tilde{s}_1, \tilde{s}_2\} = \mathcal{S}(X)$ .
Proof sketch of Lemma 7. The optimization problem (5) is non-convex and has non-linear constraints. It is still solvable using the KKT conditions, however some care is required as such conditions are no longer necessary and sufficient. Using the Linear Independence Constraint Qualification (LICQ) rule, we handle the necessity. Then, by solving for KKT conditions and comparing the objective value with solutions for which the LICQ condition does not hold, we obtain the desired description of the optimal solution. The details are deferred to Appendix A.  $\square$
At this point, we are ready to obtain the result of Lemma 5. If  $\tilde{s}_1, \tilde{s}_2 \neq 0$  then, by combining the result of Lemma 7 with (4), the proof is complete. If  $\tilde{s}_2 = 0$ , we can optimize (4), given  $s_2 = 0$  and the extra constraint  $\frac{a^T a}{b^T b} = \frac{x^T x}{y^T y}$ , which follows from the requirement on the rank of  $A$  and the fact that  $X$  has rank 1. The claim then follows from a direct computation of the objective value using the constraint. The case  $\tilde{s}_1 = \tilde{s}_2 = 0$  gives that  $X = 0$ , and is trivial.
So far, we have reduced the optimization over  $\{W_{l}\}_{l = 2}^{L - 1}$  to the optimization over the singular values of  $\{\sigma (H_l)\}_{l = 2}^L$ . We now do the same with the first layer's matrices  $W_{1},H_{1}$ . For this, we again split the optimization of  $\frac{\lambda W_1}{2}\| W_1\| _F^2 +\frac{\lambda H_1}{2}\| H_1\| _F^2$  given  $\{s_1,s_2\} = S(\sigma (W_1H_1))$  into two nested sub-problems: the inner one performs the optimization for a fixed output  $H_{2}\coloneqq W_{1}H_{1}$ ; the outer one optimizes over the output  $H_{2}$ , given the optimal value of the inner problem.
The inner sub-problem is solved by the following Lemma, which describes a variational form of the nuclear norm. The statement is equivalent to Lemma C.1 of [28].
Lemma 8. The optimization problem
$$
\min  _ {A, B; C = A B} \frac {\lambda_ {A}}{2} \| A \| _ {F} ^ {2} + \frac {\lambda_ {B}}{2} \| B \| _ {F} ^ {2} \tag {6}
$$
is minimized at value  $\sqrt{\lambda_A\lambda_B}\| C\| _*$ , and the minimizers are of the form  $A^{*} = \gamma_{A}U\Sigma^{1 / 2}R^{T}$ ,  $B^{*} = \gamma_{B}R\Sigma^{1 / 2}V^{T}$ . Here, the constants  $\gamma_{A},\gamma_{B}$  only depend on  $\lambda_{A},\lambda_{B}$ ;  $U\Sigma V^T$  is the SVD of  $C$ ; and  $R$  is an orthogonal matrix.
To be more concrete, we apply Lemma 8 with  $A = W_{1}$  and  $B = H_{1}$  (so that  $C = W_{1}H_{1} = H_{2}$ ). Thus, we are only left with optimizing  $\| H_2\|_*$  given  $S(\sigma(H_2)) = \{s_1, s_2\}$ . We solve this problem in the following, much more general lemma, which might be of independent interest. Its proof is deferred to Appendix A.
Lemma 9. Let  $L \geq 2$  be a positive integer. Then, the optimal value of the optimization problem
$$
\min  _ {H} \| H \| _ {S _ {\frac {L}{2}} ^ {\frac {L}{2}}} ^ {2}, \quad s. t. \mathcal {S} (\sigma (H)) = \left\{s _ {1}, s _ {2} \right\} \tag {7}
$$
equals  $(s_1 + s_2)^{\frac{2}{L}}$ . Here  $\| \cdot \|_{S_p}$  denotes the  $p$ -Schatten pseudo-norm.
Remark 10. As a by-product of the analysis in the proof of Lemma 9, we obtain a characterization of the minimizers of (7), which will be useful in Step 2. Let  $H^{*}$  be an optimizer of (7). Then,  $\sigma(H^{*})$  has orthogonal columns. For  $L > 2$ , its negative entries are set uniquely so that  $H^{*}$  has rank 1. For  $L = 2$ , its negative entries are arranged as follows. All the rows of  $\sigma(H^{*})$  which contain exactly one positive entry on the first column will result in rows of  $H^{*}$  that are multiples of each other. Similarly, all the rows of  $\sigma(H^{*})$  which contain exactly one positive entry on the second column will result in rows of  $H^{*}$  that are multiples of each other. The zero rows of  $\sigma(H^{*})$  remain zero rows in  $H^{*}$ . Moreover, the  $l_{2}$  norm of the negative entries of the rows of  $H^{*}$  with positive entry in the first column equals the  $l_{2}$  norm of the negative entries of the rows of  $H^{*}$  where the positive entry is in the second column. Finally, taking any two rows of  $H^{*}$  with counterfactual positioning of the signs of the entries, the product of negative entries does not exceed the product of positive entries.
Armed with Lemma 4, 5 and 9, we reformulate the  $L$ -DUFM objective (1) only in terms of  $\{s_{l,1}, s_{l,2}\}_{2 \leq l \leq L}$ . We then split this joint problem into two separate, identical optimizations over  $\{s_{l,1}\}_{2 \leq l \leq L}$  and  $\{s_{l,2}\}_{2 \leq l \leq L}$ , which are solved via the lemma below (taking  $x_l := s_{l,1}^2$  and  $x_l := s_{l,2}^2$ , respectively). Its proof is deferred to Appendix A.
Lemma 11. The optimization problem
$$
\min  _ {x _ {l}; 2 \leq l \leq L} \frac {\lambda_ {W _ {L}}}{2 \left(x _ {L} + 2 \lambda_ {W _ {L}}\right)} + \sum_ {l = 2} ^ {L - 1} \left(\frac {\lambda_ {W _ {l}}}{2} \frac {x _ {l + 1}}{x _ {l}}\right) + \sqrt {\lambda_ {W _ {1}} \lambda_ {H _ {1}}} \sqrt {x _ {2}} \tag {8}
$$
is optimized at an entry-wise positive, unique solution if
$$
\lambda_ {H _ {1}} \prod_ {l = 1} ^ {L} \lambda_ {W _ {l}} <   \frac {(L - 1) ^ {L - 1}}{2 ^ {L + 1} L ^ {2 L}}, \tag {9}
$$
and at  $(x_{2},\ldots ,x_{L}) = (0,\ldots ,0)$  otherwise. When (9) holds with equality, both solutions are optimal. Here,  $0 / 0$  is defined to be 0.
Note that (9) is equivalent to (2) when  $n = 1$ . Furthermore, if the optimization problem (8) has a single non-zero solution, then the optimal  $s_{l,1}^{*}$  and  $s_{l,2}^{*}$  are forced to coincide for all  $l$ . From this, we obtain the orthogonality of  $\sigma(H_{l}^{*})$  for  $l \geq 2$ . Lemma 6 then gives that the optimal  $H_{l}^{*}$  for  $l \geq 3$  must be orthogonal as well, since it is non-negative and equal to  $\sigma(H_{l}^{*})$ . Finally, the statement about  $W_{l}^{*}$  for  $l \geq 2$  also follows from Lemma 6. This concludes the proof of DNC2-3 for  $n = 1$ . As a by-product of Lemma 8, we additionally obtain a precise characterization of the optimal  $W_{1}^{*}, H_{1}^{*}$ .
Step 2: DNC1 for  $n > 1$ . Now, we assume  $n > 1$  and the conclusion of Step 1. We show that DNC1 holds for the optimizer of (1). We give a sketch here, deferring the proof to Appendix A.
We proceed by contradiction. Assume there exist  $\tilde{H}_1^*, W_1^*, \ldots, W_L^*$  which achieve the minimum of (1), but  $\tilde{H}_1^*$  does not exhibit DNC1. Note that the objective in (1) is separable w.r.t. the columns of  $H_1$ . Thus, for both classes  $c \in \{1, 2\}$ , the partial objectives corresponding to the columns  $\{h_{c,i}^{(1)}\}_{i=1}^n$  of  $\tilde{H}_1^*$  must be equal for all  $i$ . Indeed, if that is not the case, we can exchange all the columns of  $\tilde{H}_1^*$  with the ones that achieve the smallest partial objective value and obtain a contradiction. Now, we can construct an alternative  $\bar{H}_1^*$  such that, for each class  $c$ , we pick any  $h_{c,i}^{(1)}$  and place it instead of  $\{h_{c,j}^{(1)}\}_{j \neq i}$ . By construction,  $\bar{H}_1^*$  exhibits DNC1 and it is still an optimal solution. At this point, let us construct  $H_1^*$  from  $\bar{H}_1^*$  by taking a single sample from both classes. We claim that  $(H_1^*, W_1^*, \ldots, W_L^*)$  is an optimal solution of (1) with  $n = 1$  and  $n \lambda_{H_1}$  as the regularization term for  $H_1$ , while keeping  $d_1, \ldots, d_L$  and  $\lambda_{W_1}, \ldots, \lambda_{W_L}$  the same. To see why this is the case, consider an alternative  $G_1^*$  achieving a smaller loss. Then,  $(G_1^* \otimes \mathbf{1}_n^T, W_1^*, \ldots, W_L^*)$  would achieve a smaller loss than  $(\bar{H}_1^*, W_1^*, \ldots, W_L^*)$ , which contradicts its optimality.
Recall that  $H_1^*$  is an optimal solution of (1) with  $n = 1$ . By Step 1, the columns  $x, y$  of  $\sigma(H_2^*)$  are such that  $x, y \geq 0$ ,  $x^T y = 0$ , and  $\|x\| = \|y\|$  (here, by the notation  $x \geq 0$  we mean an entry-wise inequality). Assume  $\sigma(H_2^*) \neq 0$ . First we show that  $x, y$  and thus the whole  $\sigma(H_2^*)$  are uniquely determined by  $(W_1^*, W_2^*, \ldots, W_L^*)$ . For this we use the uniqueness of the scaling obtained in Lemma 11 and the DNC3 property for the  $(\sigma(H_2^*), W_2^*)$  pair. Then, by Lemma 9, all the matrices  $A_2$  such that  $\sigma(A_2) = \sigma(H_2^*)$  and  $\|A_2\|_* = \|H_2^*\|_*$  have columns of the form  $x - ty, y - tx$  for  $t \in [0,1]$ . Let  $A_2^t$  be a matrix of that form for a particular  $t$ . A series of steps which include computing the SVD of  $A_2^t$  and the usage of Lemma 8 reveal that, among  $0 \leq t \leq 1$ , there is a single  $t^*$  and  $A_2^* = A_2^{t^*}$  such that  $W_1^*$  solves the optimization problem (6) in Lemma 8 if  $H_2 = A_2^t$ . Thus, having fixed  $A_2^*$  that can be the output of the first layer, this in turn gives the uniqueness of the unconstrained features  $H_1$  solving (6) while  $W_1 = W_1^*$ ,  $H_2 = H_2^* = A_2^*$ . However, as  $\tilde{H}_1^*$  is not DNC1 collapsed by assumption, there are multiple possible choices for  $\tilde{H}_1^*$  and then  $H_1^*$ , which gives a contradiction with the uniqueness of  $H_1$  solving (6).
Step 3: DNC2-3 for  $n > 1$ . Having established DNC1, we obtain DNC2-3 by reducing again to the case  $n = 1$ , which allows us to conclude the proof of the main result. We finally remark that  $W_1^*$  and  $H_1^*$  do not need to be orthogonal, due to the  $\Sigma^{1/2}$  multiplication in the statement of Lemma 8. This means that  $W_1^*$  and  $H_1^*$  may not exhibit DNC2-3 (as stated in Theorem 3).
4.2 Numerical results
DUFM training. To support the validity of our theoretical result and to demonstrate that the global optimum in the DUFM can be found efficiently, we conduct numerical experiments in which we solve (1) with randomly initialized weights using gradient descent. In Figure 2, we plot the training progression of the DNC metrics for two choices of the depth:  $L = 3$  (first row) and  $L = 6$  (second row). In both cases, we use  $n = 50$  training samples and constant layer width of 64. The learning rate is 0.5, though smaller learning rates produce the same results. We run full gradient descent for  $10^{5}$  steps, and the weight decay is set to  $5 \cdot 10^{-4}$  for all the layers. The plots are based on 10 experiments and the error bands are computed as one empirical standard deviation to both sides. To measure the DNC occurrence, we plot the following metrics as a function of training. For DNC1, we define  $\Sigma_{W}^{l} = \frac{1}{N}\sum_{c=1}^{2}\sum_{i=1}^{n}(h_{c,i}^{(l)} - \mu_{c}^{(l)})(h_{c,i}^{(l)} - \mu_{c}^{(l)})^{T}$  as the within-class variability at layer  $l$  and  $\Sigma_{B}^{l} = \frac{1}{2}\sum_{c=1}^{2}(\mu_{c}^{(l)} - \mu_{G}^{(l)})(\mu_{c}^{(l)} - \mu_{G}^{(l)})^{T}$  as the between-class variability, where  $\mu_{G}^{(l)}$  is the global feature mean at layer  $l$ . Then, the DNC1 metric at layer  $l$  is  $\| \Sigma_{W}\Sigma_{B}^{\dagger}\|_{F}^{2}$ , with  $\dagger$  being the pseudo-inverse operator. This is an upper bound on the trace of the same matrix, which is sometimes used in the literature for measuring NC1. An ideal DNC1 would show 0 on this metric. For DNC2, our metric at layer  $l$  is given by  $\frac{s_{l,1}}{s_{l,2}}$  (recall that these are the singular values of  $\sigma(H_l)$ ), and we consider the feature matrices both after and before the ReLU activation. An ideal DNC2 would show 1 on this metric. For DNC3, we consider the weighted average sine of the angles between each row of  $W_{l}$  and the closest column of  $\sigma(H_l)$ ; the weights are the  $l_2$  norms of the rows of  $W_{l}$ . To avoid
Figure 2: Deep neural collapse metrics as a function of training for the DUFM objective (2). The first (second) row corresponds to  $L = 3$  ( $L = 6$ ). The metrics corresponding to DNC1 and DNC3 are very small, and the metric corresponding to DNC2 is close to 1, which means that the solution found by gradient descent exhibits neural collapse in all layers, as predicted by Theorem 3.
numerical issues, we exclude rows of  $W_{l}$  whose  $l_{2}$  norm is smaller than  $10^{-6}$ . An ideal DNC3 would show 0 on this metric.
The plots in Figure 2 closely follow the predictions of Theorem 3 and, taken as a whole, show that DUFM training via gradient descent clearly exhibits deep neural collapse, for all the three metrics considered, and for both depths. More specifically, DNC1 is progressively achieved in all layers, with the layers closer to the output being faster. DNC2 is achieved for all  $\sigma(H_l)$  with  $l \geq 2$  and  $H_l$  for  $l \geq 3$  (though we didn't include this in plots to avoid too many lines) and, as predicted by our theory, it is not achieved on  $H_2$  (the dotted line in the middle plot) and on  $H_1$  (the blue line). Interestingly, for these numbers of layers, SGD does not find solutions which would exhibit DNC2 on  $H_1$ , even though such solutions exist. Hence, gradient descent is not implicitly biased towards them, see Appendix C for a more detailed discussion. DNC3 is achieved on all layers - even on the first, which is not covered by our theory and where the effect is the most pronounced; the deeper layers achieve values of DNC3 of order  $10^{-2}$ , which translates into an angle of less than one degree between weight rows and feature columns. Finally, we highlight that the training loss matches the optimal value computed numerically via (17) (the corresponding plot is reported in Appendix C), and hence the solution found by gradient descent is globally optimal. Additional ablation studies concerning the number of layers, weight decay and width of the layers are reported in Appendix C.
ResNet20 training with pre-trained DUFM. We support the validity of DUFM as a modeling principle for deep neural collapse via the following experiments. We first train an  $L$ -DUFM for  $L = 3,5$  with  $K = 2$  and  $n = 5000$  to full optimality, as in the previous paragraph. Then, we replace  $H_1^*$  with the output of a randomly initialized backbone, we fix the weight matrices in the DUFM, and we train end-to-end to convergence on the classes 0 and 1 of CIFAR-10. In Figure 1, we report the DNC metrics of our 3-DUFM head, as the training of the backbone ResNet20 progresses. While at the start of the training the features  $H_1$  are random and, therefore, the model does not exhibit DNC, the ResNet20 recovers well the DNC properties after training. This suggests that the crucial assumption of DUFM – the unconstrained features, indeed holds in practice. The plots are based on 7 experiments with confidence bands of one empirical standard deviation to both sides. Additional details as well as the experiments with 5-DUFM and VGG13 and DenseNet121 are discussed in Appendix D.
In this work, we propose the deep unconstrained features model (DUFM) – a multi-layered version of the popular unconstrained features model – as a tool to theoretically understand the phenomenon of deep neural collapse (DNC). For binary classification, we show that DNC is globally optimal in the DUFM. This provides the first rigorous evidence of neural collapse occurring for arbitrarily many layers. Our numerical results show that gradient descent efficiently finds a solution in agreement with our theory – one which displays neural collapse in multiple layers. Finally, we provide evidence in favor of DUFM as a modeling principle, by showing that standard DNNs can learn the unconstrained features that exhibit all DNC properties.
Interesting open problems include (i) the generalization to multiple classes  $(K > 2)$ , (ii) the analysis of the gradient dynamics in the DUFM, and (iii) understanding the impact of biases on the collapsed solution, as well as that of the cross-entropy loss. As for the first point, a crucial issue is to control  $\| W_1H_1\|_*$ , since the inequality  $\| \sigma (M)\|_*\leq \| M\|_*$  does not hold for general matrices  $M$ . In addition, while Lemma 5 still appears to be valid for  $K > 2$  (and it would be an interesting stand-alone result), its extension likely requires a new argument.
M. M. is partially supported by the 2019 Lopez-Loreta Prize. The authors would like to sincerely thank Eugenia Iofinova, Bernd Prach and Simone Bombari for valuable feedback to our manuscript.
[1] Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Topological, Algebraic and Geometric Learning Workshops, 2022.
[2] Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data. arXiv preprint arXiv:2301.00437, 2023.
[3] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. In Proceedings of the National Academy of Sciences (PNAS), volume 118, 2021.
[4] Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.09028, 2022.
[5] Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
[6] Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and 12 normalization with improved out-of-distribution detection in deep neural networks. Transactions on Machine Learning Research (TMLR), 2022.
[7] X. Y. Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations (ICLR), 2022.
[8] Hangfeng He and Weijie J Su. A law of data separation in deep learning. arXiv preprint arXiv:2210.17020, 2022.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (ICCV), 2015.
[10] Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. In Technometrics, volume 12, 1970.
[11] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384, 2022.
[31] E Weinan and Stephan Wojtowitsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning, 2022.
[32] Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction-repulsion-balanced loss for imbalanced learning. In Neurocomputing, 2023.
[33] Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. In Research, volume 6, 2023.
[34] Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[35] Yongyi Yang, Jacob Steinhardt, and Wei Hu. Are neurons actually collapsed? on the fine-grained structure in neural representations. In I Can't Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification (NeurIPS 2022 Workshop), 2022.
[36] Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. arXiv preprint arXiv:2209.09211, 2022.
[37] John Zarka, Florentin Guth, and Stéphane Mallat. Separation and concentration in deep networks. In International Conference on Learning Representations (ICLR), 2021.
[38] Jiayao Zhang, Hua Wang, and Weijie Su. Imitating deep learning dynamics via locally elastic stochastic differential equations. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[39] Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. arXiv preprint arXiv:2301.01100, 2023.
[40] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse undermse loss: Global optimality with unconstrained features. In International Conference on Machine Learning (ICML), 2022.
[41] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[42] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Proof. We will split the proof into two distinct cases:  $n = 1$  and  $n > 1$ . The proof for  $n > 1$  will require the statement to hold for  $n = 1$ , so we start with this case. The explicit form of the problem goes as:
$$
\min  _ {H _ {1}, W _ {1}, \dots , W _ {L}} \frac {1}{2 N} \| W _ {L} \sigma \left(W _ {L - 1} (\dots W _ {2} \sigma \left(W _ {1} H _ {1}\right)\right) - I _ {2} \| _ {F} ^ {2} + \sum_ {l = 1} ^ {L} \frac {\lambda_ {W _ {l}}}{2} \| W _ {l} \| _ {F} ^ {2} + \frac {\lambda_ {H _ {1}}}{2} \| H _ {1} \| _ {F} ^ {2}. \tag {10}
$$