The Prevalence of Neural Collapse in Neural Multivariate  Regression
Authors: George Andriopoulos
Zixuan Dong
Li Guo
Zifan Zhao
Keith W. Ross
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Collapse
Multivariate Regression
DNN
Unconstrained Feature Model
Primary area: optimization_for_deep_networks
TL;DR: We provide theoretical and experimental analysis for Neural Regression Collapse which is prevalent among multivariate regression tasks.

Abstract: Recently it has been observed that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. We empirically show that multivariate regression, as employed in imitation learning and other applications, exhibits Neural Regression Collapse (NRC), a new form of neural collapse: (NRC1) The last-layer feature vectors collapse to the subspace spanned by the $n$ principal components of the feature vectors, where $n$ is the dimension of the targets (for univariate regression, $n=1$); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the last-layer weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends on the covariance matrix of the targets. After empirically establishing the prevalence of (NRC1)-(NRC3) for a variety of datasets and network architectures, we provide an explanation of these phenomena by modeling the regression task in the context of the  Unconstrained Feature Model (UFM), in which the last layer feature vectors are treated as free variables when minimizing the loss function. We show that when the regularization parameters in the UFM model are strictly positive, then (NRC1)-(NRC3) also emerge as solutions in the UFM optimization problem. We also show that if the regularization parameters are equal to zero, then there is no collapse. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. This extension is significant not only because it broadens the applicability of neural collapse to a new category of problems but also because it suggests that the phenomena of neural collapse could be a universal behavior in deep learning.

The Prevalence of Neural Collapse in Neural Multivariate Regression
George Andriopoulos Zixuan Dong 2.4+
Li Guo
Zifan Zhao
Keith Ross!
New York University Abu Dhabi 3 New York University Shanghai
SFSC of Al and DL, NYU Shanghai
New York University
Abstract
Recently it has been observed that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. We empirically show that multivariate regression, as employed in imitation learning and other applications, exhibits Neural Regression Collapse (NRC), new form of neural collapse: (NRCI) The last-layer feature vectors collapse to the subspace spanned by the n principal components the feature vectors, where is the dimension of the targets (for univariate regression, = 1); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the last-layer weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends on the covariance matrix of the targets. After empirically establishing the prevalence of (NRC1)-(NR33 for variety of datasets and network architectures, we provide an explanation of these phenomena by modeling the regression task in the context of the Unconstrained Feature Model (UFM), in which the last layer feature vectors are treated as free variables when minimizing the loss function. We show that when the regularization parameters in the UFM model are strictly positive, then (NRCI)-(NRC)) problem. We also show that if the regularization parameters there no collapse. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. This extension is significant not only because it broadens the applicability of neural collapse to new category of problems but also because suggests that the phenomena of neural collapse could a universal behavior in deep learning.
also emerge as solutions in the UFM optimization equal to zero, then
Introduction
Recently, insightful phenomenon known neural collapse (NC) [Payyan et al. 2020 has been empirically observed during the terminal phases of training in classification tasks with balanced data. NC has three principal components: (NC1) The features of samples within each class converge closely around their class mean. (NC2) The averages of the features within each class converge to form the vertices of simplex equiangular tight frame. This geometric arrangement implies that class means are equidistant and symmetrically distributed. (NC3) The weight vectors the classifiers in the final layer align with the class means of their respective features. These phenomena not only enhance our understanding of neural network behaviors but also suggest potential simplifications in the architecture and the training of neural networks.
"Equal contribution.
Corresponding author: keithwroas@nyu.edi
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A
Before Collapse
After Collapse
21
Figure I: Visualization of the neural regression collapse. The red dots represent the sample features, the blue arrows represent the row vectors of the last layer weight matrix, and the yellow plane represents the plane spanned by the principal components of the sample features. Here the target dimension is n 2. The feature vectors and weight vectors collapse to the same subspace. The angle between the weight vectors takes specific forms governed by the covariance matrix of the targets.
The initial empirical observations of NC have led to the development of theoretical frameworks such the layered-peele model (Fang et al. 2021 and the unconstrained feature model (UFM) [Mixon et al. 2020). These models help explain why NC occurs in classification tasks theoretically. By allowing the optimization freely adjust last-layer features along with classifier weights, these models provide important insights into the prevalence of neural collapse, showing that maximal class separability is natural outcome for a variety of loss functions when the data balanced [Han et al.)
2021
Poggio and Liao 2020 Zhou et al.) b0.
Regression in deep learning is arguably equally important as classification, as it serves for numerous applications across diverse domains. imitation learning for autonomous driving, regression is employed to predict continuous control actions (such speed and steering angles) based on observed human driver behavior. Similarly, regression is used in robotics, where the regression model is trained to imitate expert demonstrations, In the financial sector, regression models extensively used for predictive analytics, such as forecasting stock prices, estimating risk, and predicting market trends. Meteorology also heavily relies regression models to forecast weather conditions. These models take high-diman nigh-dimensional inputs from various sensors and satellites to predict multiple continuous variables such as temperature, humidity, and wind speed. Moreover, many reinforcement learning algorithms include critical regression components, where regression is employed to predict value functions with the targets being Monte Carlo bootstrapped returns.
While NC has been extensively studied in classification, to our knowledge, its prevalence and implications in regression remain unexplored. This paper investigates new form of neural collapse within the context neural multivariate regression. Analogous to the classification problem, we introduce Neural Regression Collapse (NRC): (NRC1) During training. the last-layer feature vectors collapse to the subspace spanned by the n principal components the feature vectors, where n is the dimension of the targets (for univariate regression, = 1); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends the square-root of the covariance matrix of the targets. A visualization of NRC shown in Figure !
Employing six different datasets including three robotic locomotion datasets, two versions of an autonomous driving dataset, and an age-prediction dataset and Multi-Layce Perceptron (MLP) and ResNet architectures, we establish the prevalence of NRCI-NRC3, This discovery suggests universal geometric behavior extending beyond classification into regression models, simplifying our understanding of deep learning
generally.
To help explain these phenomena, we then apply the UFM model neural multivariate regression with an L2 loss function. In this regression version of the problem, the optimization problem aims to minimize the regularized mean squared error over continuous-valued targets. We show that when the regularization parameters in the UFM model are strictly positive, then (NRCI)-NRC3) also emerge

---

as solutions in the UFM optimization problem, thereby providing mathematical explanation of our empirical observations. Among many observations, we discover empirically and theoretically that when the regression parameters are zero or very small, there is no collapse; and if we increase the parameters a small amount above zero, the (NRCI)-(NRC) geometric structure emerges.
To the best of our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. By demonstrating the prevalence of neural collapse in regression tasks, we reveal that deep learning systems might inherently simplify their internal representations, irrespective of the specific nature of the task, whether be classification or regression.
2 Related work
Neural collapse (NC) was first identified by Papyan et al. (2020) as symmetric geometric structure observed in both the last layer features and classification vectors during the terminal phase of training of deep neural networks for classification tasks, particularly evident in balanced datasets. Since then, there has been a surge of research into both theoretical and empirical aspects of NC.
Several studies have investigated NC under different loss functions. For instance, (Han et al. 2021
Poggio and Liao 2020 Zhou al. 2022a have observed and studied neural collapse under the
Mean Squared Error (MSE) loss, while papers such as (Zhou et al. 2022b Guo et al. [024] have demonstrated that label smoothing loss and focal loss also lead to neural collapse. In addition to the last layer, some papers (He and Su 2023, Rangamani et al. 2023] have also examined the occurrence of the NC properties within intermediate layers. Furthermore, beyond the balanced case, researchers have investigated the neural collapse phenomena in imbalanced scenarios. (Fang et al. 202T) identified a phenomenon called minority collapse for training on imbalanced data, while [Hong and Ling 2023 Thrampoulidis et al. 2022 Dang et al. 2023 offer more precise characterizations of the geometric structure under imbalanced conditions.
To facilitate the theoretical exploration of the neural collapse phenomena, Fang et al. 2021 Mixon
al. 2020) considered the unconstrained feature model (UFM). The UFM simplifies a deep neural network into an optimization problem by treating the last layer features as free variables to optimize over. This simplification is motivated by the rationale of the universal approximation theorem (Hornik et al. 1989]. asserting that sufficiently weer-parameterized neural networks can be highly expressive and can accurately approximate arbitrary smooth functions. Leveraging the UFM, studies such as
[Zhu et al. 2 Zhou et al. 2022a| Thrampoulidis et al. 2022 Tirer and Bruna 2022. Tirer et al. 2023 Ergen and Pilanci 2021 Wojtowytsch al.) [0200 have investigated models with different loss functions and regularization techniques. These studies have revealed that the global minima of the
empirical risk function under UFMs align with the characterization of neural collapse observed by
[Papyan et al.) 2020). Beyond the UFM, some work (Tirer and Bruna, 2022 Sūkenik et al. [2024] has extended the model to explore deep constrained feature models with multiple layers, aiming to investigate neural collapse properties beyond the last layer.
In addition to its theoretical implications, NC serves as a valuable tool for gaining deeper insights into DNN models and various regularization techniques [Guo et al. 2024 Fisher et al. [024]. It provides crucial insights into the generalization and transfer learning capabilities of neural networks
Huui et al. 2022 Kothapalli 2022, Galanti et al., [02T], inspiring the design of enhanced model
architectures for diverse applications. These include scenarios with imbalanced data (Yang et al..
2022
Kim and Kim) and contexts involving online continuous learning [Seo et al., 2024].
Despite extensive research on the neural collapse phenomena and its implications in classification, to the best of our knowledge, there has been no investigation into similar issues regarding neural regression models. Perhaps the paper closest to the current work is [Zhou et al. 2022a]. which applies the UFM model to the balanced classification problem with MSE loss. Although locused on classification, [Zhou et al. 2022a] derive some important results which apply to regression as well as to classification. Our UFM analysis leverages this related paper, particularly their Lemma B.
Prevalence of neural regression collapse
We consider the multivariate regression problem with M training examples yx),, = 1,...,, M}. where each input x; belongs to RRd and each target vector y: belongs to R For the regression
3

---

task, the deep neural network (DNN) takes input an example E RD and produces output y f(x) R For most DNNs, including those used in this paper, this mapping takes the form (h+(x) + b. where ha(-) : IR E2 is the non-linear feature extractor consisting of several nonlinear layers, W is n x d matrix representing the final linear layer in the model, and E _ is the bias vector. For most neural regression tasks, n << d, that is the dimension of the target space is much smaller than the dimension of the feature space. For univariate regression, n 1. The parameters 6, W, and b are all trainable.
fo w. (x)
We train the DNN using gradient descent to minimize the regularized L2 loss:
M Σ
(l,,,.,((..) - y-llē + Ae 2 1111
+ Aw 2
min M.W.M b 2M
IWI]
where || 112 and || || denote the L--norm and the Frobenius norm, respectively. As commonly done in practice, in experiments set all the regularization parameters to the same value, which we refer to as the weight-decay parameter AWD, that is, we set d = Aw = AWDD
3.1 Definition of neural regression collapse
In order to define Neural Regression Collapse (NRC), let E denote the n x n covariance matrix corresponding the targets {y,, L..... ..M:M); Σ MY Y) Y)",, where Y [y yy... = .y...).. and y = M Di=1 yr. Throughout this paper, we make the natural assumption that Y and Σ have full rank. Thus E is positive definite. Let Amin > 0 denote the minimum eigenvalue of Σ.
Denote H := [h, ... h where h; is the feature vector associated with input x,, that is, h, (xx).. Further denote the normalized feature vector h; h [hh.|-1 Of course, W, H, and b are changing throughout the course of training. For any p g matrix C and any p-dimensional vector v, let (roj(v(()) denote the projection of onto the subspace spanned by the columns of C. Let HCCA, be the d x n matrix with the columns consisting of the n principal components of H.
We say that Neural Regression Collapse (NRC) emerges during training if the following three phenomena occur:
1 h (roj (r,j(h,HHrC.... 2
NRC1
Σ
0.
M
M - projjjhhh: 1 +
NRC2 = M
Σ
There exists a constant y E (0, Amin) such that:
WWT
NRC3
n
0.
|WT
NRCI + 0 indicates that there is feature-vector collapse, that is, the d-dimensional feature vectors h, i 1,..., , M, collapse much lower n-dimensional subspace spanned by their n principal components. In many applications, = 1, in which case the feature vectors are collapsing to a line in the original d-dimensional space. NRC2 + indicates that there is form of self duality, that is, the feature vectors also collapse to the n rz-dimensional space spanned by the rows of W. NRC3 + indicates that the last- weights have a specific structure within the collapsed subspace. In particular, it gives detailed information about the norms of the row vectors in W and the angles between those row vectors. NRC3 - indicates that angles between the rows W are influenced by 1/2 If the targets are uncorrelated so that Σ and ///2 are diagonal, then NRC3 + 0 implies that the rows in W will be orthogonal. NRC3 + 0 also implies a specific structure for the feature vectors, discussed in Section
3.2 Experimental validation of neural regression collapse
In this section, we validate the emergence of NRC1-NRC3 during training across various datasets and deep neural network (DNN) architectures.

---

Datasets. The empirical experiments in this section are based on the following datasets:
The Swimmer, Reacher, and Hopper datasets are based on MoJoCo [Todorov et al. 2012] Brockmma et al. 2016 Towers et 2023]. a physics engine that simulates diverse continuous multi-joint robot controls and has been a canonical benchmark for deep rein- forcement learning research. In our experiments, we use publicly available expert datasets (see appendix 1 Each dataset comprises raw robotic states as inputs (x; 's) and robotic actions as targets (y: 'ss. order to put these expert datasets in an imitation learning context, we reduced the size of the dataset by keeping only small portion of the episodes.
The CARLA dataset originates from the CARLA Simulator, an open-source project de- signed to support the development of autonomous driving systems. We utilize dataset Codevilla et al. (2018) sourced from expert-driven offline simulations. During these simula- tions, images (x, 8) from cameras mounted on the virtual vehicle and corresponding expert driver actions as targets (y;'s) are recorded as human drives in the simulated environment. We consider two dataset versions: 2D version with speed and steering angle, and ID version with only the speed.
The UTKFace dataset [Zhang et al. 2017 is widely used in computer vision to study age estimation from facial images of humans. This dataset consists of about 25,000 facial images spanning wide target range of ages, races, and genders.
Table I summarizes the six datasets, with the dimensions of the target vectors y ranging from one to three. The table also includes the minimum eigenvalue of the associated covariance matrix Σ and the Pearson correlation values between the i-th and j-th target components for i * j. When is no correlation value; when n 2, there is one correlation value between the two target components; and when n 3, there three correlation values among the three target components. From the table, we observe that the target components in CARLA 2D and Reacher are nearly uncorrelated, whereas those in Hopper and Swimmer exhibit stronger correlations.
1. there
Table I: Overview of datasets employed in our neural regression collapse analysis.
Dataset
Data Size
Input Type
Target Dimension n
Target Correlation
Amin
Swimmer 1,000
state
-0.244
0.276
Reacher
1,000
raw state
-0.00933
0.0097
Hopper 10,000
raw state
[-0.215, -0.090, 0.059]
0,215
Carla ID
600,000
RGB image
NA
208.63
Carla 2D
600,000
RGB image
-0.0055
0.156
UTKface
25,000
RGB image
NA
1428
Experiment Settings. For the Swimmer, Reacher, and Hopper datasets, we employed a four-layer MLP (with the last layer being the linear layer) as the policy network for the prediction task. Each layer consisted of 256 nodes, aligning with the conventional model architecture in most reinforcement learn- ing research [Tarasov et al.] 2022]. For the CARLA and UTKFace datasets, we employed ResNet18 and ResNet34 al. [2016], respectively. To focus behaviors associated with neural collapse and minimize the influence of other factors, we applied standard preprocessing without data augmentation.
All experimental results are averaged over at least 2 random seeds and variance is displayed by shaded area. The choices of weight decay employed during training varied depending on the dataset. Also, the number of epochs required for training depends on both the dataset and the degree of weight decay. In particular, we used a large number of epochs when using very small weight decay values. appendix A provides the full experimental setup.
Empirical Results. Figure presents the experimental results for the six datasets mentioned above. The results show that the training and testing errors decrease as training progresses, as expected. The converging coefficient of determination (/e2) also indicates that model performance becomes stable. Most importantly, the figure confirms the presence of NRC1-NRC3 across all six datasets. This indicates that neural collapse is not only prevalent classification but also often occurs in multivariate regression.
5

---

Reacher
Swimmer
Hopper
0.10 10a
0.18
Model Performance NRC1-3
NICC
NICC
10-1-
NRCZ
NAC1
20-4
MRC2
MRCI
0.16
0.5
0.05
0.5
0.14
0.12
0.0
0.00
0.0
0.03
0.375M
1.12MM 1.5M
150K
225K
300K
50K
75K
100K
0.75
0.35
0.02
Trann MSE
0.50
Trin NSE
0.5
0.30
Train MSE
M5
Test MSE
MSE
0.01
0.5
0.00
0.0
0.375M O. 1.75M 1.155M 1.5M
75K
150K
225K
300K
25K
50K
75K
100K
Epoch
Epoch
Epoch
Carla 20
Carla
UTKFace
0.6
NRC1-3
0.2
0.04
0.5
NRC3
0.4
0.1
0.02
0.2
0.2
Model Performance
0.0
0.0
0.0
20
40
60
80
25
50
75
100
25
50
75
100
100
www MMM
1.0
75
Train MSE
200
Trum MSF
Test MSE
Test MSE
50
R²
500
M
100
25
0.0
20
25
100
25
50
100
Epoch
Epoch
Epoch
Figure 2: Prevalence of NRCI-NRC3 in the six datasets. Train/Test MSE and the coefficient of determination (R²) are also shown.
Reacher 2)
Swimmer 2)
Hopper (n
0.6
0.6
0.75
PC1
EUR
0.4
0.4
0.50
PC2
PC3
0.2
0.2
0.25
PC4
PC5
0.0
0.0
0.00
0.75M
1.5M
0.5M
1M
0
100K
200K
Epoch
Epoch
Epoch
Carla (n 2)
Carla (0 1)
UTKFace 1)
1.0
1.0
0.75
EVR
0.50
0.5
0.5
0.25
0.00
0.0
50
50
100
0
50
100
Epoch
Epoch
Epoch
Figure Explained Variance Ratio (EVR) for the first principal components (PC).
also experimentally analyze the explained variance ratio (EVR) of principal components to further verify the collapse to the subspace spanned by the first components. In Figure 3 we investigate the EVR of the first 5 principal components of during the training process. For all datasets, there is significant variance for all of the first components after short period of training; for other components, there is very low or even no variance. This also supports that a perfect collapse occurs in the subspace spanned by the first n principal components.
Our definition of NRC3 involves finding a scaling factor y for which the property holds. Figure illustrates the values of NRC3 function of for W obtained after training. We observe that each

---

dataset exhibits unique minimum value of y. More details about computing NRC3 can be found in
appendix A.3
Reacher
Swimmer
Hopper
Carla 2D
Last Epoch NRC3
0.20
NRC3
0.10
NRC3
0.30
NRC3
0.015
NRC3
0.25
0.010
0.10
0.05
0.20
0.005
0.05
0.15
0.00
0.00
0.000
0.749
0
0.691
1
0.011
0.149
V/Amin
m/n
VIAmin
V/Amin
Figure The optimal value of for NRC3.
Figure 5 investigates neural regression collapse for small values of the weight-decay parameter Aw (appendix A..4 contains results on all 6 datasets.) We see that when Aw = 0, there is no neural regression collapse. However, we increase AWD by small amount, collapse emerges for all three metrics. Thus we can conclude that the geometric structure NRC1-3 that emerges during training is due to regularization, albeit the regularization can be very small. In the next section, will introduce mathematical model that helps explain why there is no collapse when Aw O and why it quickly emerges as AWD D is increased above zero.
Train NSE
Test MSE
NRC1
NRC2
NRC3
10a
10"
10-3
Swinner
Train MSE
Bost MSE
NPCC1
10-3
NRC3
100
-3-3
10-7
10
10
S
10-9
0.5M
100
1.00
Carla 20 MSE
Test MSE
80
0.2
Aea
40
0.50
NRC2
NRC3
0.1
0.1
Jao S0-4
20
0.00
0.0
0.0
50
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 5: Phase change in neural collapse for small weight-decay values
Unconstrained feature model
As discussed in the related work section, the UFM model has been extensively used help explain the prevalence neural collapse in the classification problem. In this section, we explore whether the UFM model can also help explain neural collapse in neural multivariate regression.
Specifically, we consider minimizing L(H, W. b), where
dw 2
WH + b1 Y C(H, W. 2M
|W
2M
where 17, = [1 and λи Aw are non-negative regularization constants.
The optimization problem studied here bears some resemblance to the standard linear multivariate regression problem. we view the features h, 1,..., M, as the inputs to linear regression, then ý: := Wh; + b is the predicted output, and ||y: - ý.1l is the squared error. In standard linear regression, the h are fixed inputs. the UFM model, however, not only are we optimizing over the weights W and biases b but also over all the "inputs" H.
7

---

is global minimum. Thus, all vectors the sphere given by ||wl| AH ( are optimal solutions. Furthermore, h; 1,..., M, are all in the one-dimensionall subspace spanned by w. Thus the optimal solution of the UFM model provides theoretical For the case of classification, regularization is needed in the UFM model to prevent the norms of H and/or W from going to infinity in the optimal solutions. contrast, the UFM regression model, the norms in the optimal solutions will be finite even without regularization. However, as regularization typically used in neural regression problems to prevent overfitting, it is useful to include regularization in the UFM regression model as well.
4.1 Regularized loss function
Throughout this subsection, we assume that both Aw and λи are strictly positive. We shall consider the Aw λи case subsequently. We also make number of assumptions in order to get distracted by less important sub-cases. Throughout we assume n S d, that is, the dimension of the targets is not greater than the dimension of the feature space. As stated in a previous subsection, for problems of practical interest, we have πt << d. Recall that Σ is the covariance matrix of the target data. Since E is a covariance matrix and is assumed have full rank, it is also positive definite.
therefore has positive definite square root, which we denote by Σ Let Amax A 2 A2 2 ... An = Amin > O denote the n eigenvalues of Σ. We further define the n X n matrix
A = 1/2 Veln:
(2)
where = λH Also for any p X q matrix C with columns C1 ......,
Cq. denote [C], to be the p x matrix whose first j columns are identical to those in C and whose last j columns are all zero vectors, i.e., [C], [ci C2 Cj 0..-0),
0). . All proofs are provided in the appendix.
Theorem 4.1. Any global minimum [W. H. b) for 7 takes the following form: If c Amax. then for any semi-ortiin orthioorttgonall matrix R,
W =
AH Aw
1/4
[A2/2 1,.R R.
H H=
Aw w |
[X Y), b=
(3)
where j* = max X3 2 c). If c. >>> Amaxx then (W, H. b) 0.. Furthermore, if (W, H) is a critical point but not global minimum, then it is strict saddle point.
Theorem 4.1 has numerous implications, which elaborate on below.
4.2
dimeeninalo univariate case
In this subsection, we highlight the important special case 1, which often arises in practice (such as with Carla ID and the UTKface datasets). When 1, simply the scalar o², which the variance of the one-dimensional targets over the M samples. Also, W is row vector, which we denote by w. Theorem 4.1 for n provides the following insights:
1. Depending whether 0 < <. o² or not, the global minimum takes on strikingly different forms. In the case, c > o², corresponding to very large regularization parameters, the optimization problem ignores the MSE and entirely focuses on minimizing the norms | P I = 0, ||w] = O.
and [ 12. giving
2. When 0 < c < o², the optimal solution takes a more natural and interesting form: For any unit vector RRd, the solution (H, w, b) given by
0 c/2 1)e,
VAw T (Y Y). VAHO w
w
H
b = 7
(4)
explanation for NRCI-NRC2. (NRC3 is not meaningful for the one-dimensional case.) Note
that the h, 's have a global zero mean and the norm of h; is proportional [y: Øl.
4.3 General n-dimensional multivariate case
In most cases of practical interest, we will have < Amin. that [A A/122 1/2 in Theorem 4.1

---

Corollary 4.2. Suppose 0 < <c< Amnn Then the global minima given by 3 have the following properties:
(i) All of the d-dimensional feature vectors h; i = ,,...,M, M, lie in the n-dimensional subspace
spanned by the n rows of W. (ii) WWT
V
Aw
AH.
[/ vcIn), (iii) AH
H |
Aww W
(iv) CH,,W, = nc/2+ /2 (v) WH + b1 - --=V=<>> /2
From Theorem 4.1 and Corollary 4.2 we make the following observations:
1. Most importantly the global minima in the UFM solution match the empirical properties (NRCI)-(NRC)) observed Section 3. In particular. the theory precisely predicts NRC3, with = This confirms that the UFM model is an appropriate model for neural regression.
2. Unlike the one-dimensional case, the feature vectors are no longer colinear with any of the rows of W. Moreover, after rotation and projection (determined by the semi-orthogonal matrix R.), the angles between the target vectors in Y Y do not general align with the angles between the feature vectors in H. However, the target components are uncorrelated. so that Σ is diagonal, then A is also diagonal and there is alignment between H and Y Y.
Theorem 4.1 also provides insight into the "strong regularization" case of > Amin- In this case, the rows of W and the feature vectors H in the global minima belong to subspace that has dimension even smaller than n, specifically, dimension j* < n. To gain some insight, assume that the target components are uncorrelated so that Σ is diagonal and Xラ- = oz. i.e., of is the variance of the j-th target component. Then for target component for which
of. the corresponding row in W will be zero and the component prediction will be y/") 5(6) for all examples i l,...,,. M. For more details, we refer the reader to Section D.I in the appendix.
4.4 Removing regularization
In the previous theorem and corollary, we assumed the presence or L2 regularization for W and H, that is, we assumed Xw > O and λи > 0. Now we explore the structure of the solutions to the UFM when Aw = λи = 0. In this case, the UFM model is modeling the real problem with Aw equal to or close zero. The loss function becomes:
1 2M
L(W, H)
WW Y
For this case, we do not need bias since we can obtain zero loss without it.
Theorem 4.3. The solution (W, H) is a global minimum if and only if W is any x d full rank matrix and
H -= + (Ia- w))
(6)
where W is the pseudo-invers inverse of W and Z is any x M matrix. Consequently, when there is no regularization, for each full-rank W there is an infinite number of global minima (W, H) that do not collapse to any subspace of Rd..
From Theorem 4.3 when there is no regularization, the feature vectors do not collapse. Moreover, any full rank W provides an optimal solution. For example, for n 2, the two rows of W can have any angle between them except angle and angle 180. This is very different from the results we have for Åh. Aw > 0, in which case W depends on the covariance matrix Σ. Note that if we set λи Aw and let λи - then the limit of W still depends on Σ. Thus there a major discontinuity in the solution when λH, Aw goes to zero. We also observed this phase shift in the experiments (see Figure (5). We can therefore conclude that neural regression collapse is not an intrinsic property of neural regression alone. The geometrie structure of neural regression collapse is due the inclusion of regularization in the loss function.
4.5 Empirical results with UFM assumptions
We also provide empirical results for the case when we train with the same form of regularization as assumed by the UFM model. Specifically, we turn off weight decay and add an 1.2 penalty on the last-layer features h, i M, and on the layer linear weights W. Additionally, we omit the

---

ReLU activation function in the penultimate layer, allowing the feature representation produced by the feature extractor to take any value, thus reflecting the UFM model. For these empirical results, when evaluating NRC3, rather than searching for in the definition of NRC3, we use the exact value of given by Theorem 4.1
that is, 7 Xw AH
C.
Train MSE
NRC1
NRC2
NRC3
0.044
0.25
0.6
0.8
0.20
Swinmer
Train MSE
2
0.4
0.6
Aan 20
20
0.038
0.036
0.2
0.05
0.034
0.0
0.00
5K
10K
0
5K
10K
5K
10K
5K
10K
100
0.30
Carla 20
80
0.6
0.20
App
60-
NRCC
0.4
0.20
Aapm
0.10
20
0.10
0.00
50
50
50
50
Epoch
Epoch
Epoch
Figure 6: Empirical results with UFM assumption where L2 regularization on H and W are used instead of weight decay.
Figure 6 illustrates training MSE and NRC metrics for varying values of (For simplicity, we only considered the case where Aw λи Appendix (Bliiixxccontins results on remaining datasets.) As we are considering different model and loss function for these empirical experiments, convergence occurs more quickly and so we train for a smaller number of epochs. We can conclude that the UFM theory not only accurately predicts the behavior of the standard L2 regularization approach with weight-decay for all parameters (Figure 5). but also accurately predicts the behavior when regularization follows the UFM assumptions (Figure(6). (b.
Conclusion
We provided strong evidence, both empirically and theoretically, of the existence of neural collapse for multivariate regression. This extension is significant not only because it broadens the applicability of neural collapse neural collapse could be a universal behavior deep learning. However, it is worth acknowledging that while we have gained better understanding of the model behavior of deep regression models in the terminal phase of training, we have not addressed the connection between neural regression collapse and model generalization. This crucial aspect remains an important topic for future research.
category of problems but also because suggests that the phenomena of
10

---

Acknowledgments and Disclosure of Funding
This work is submitted in part by the NYU Abu Dhabi Center for Artificial Intelligence and Robotics, funded by Tamkeen under the Research Institute Award CG010.
This work is partially supported by Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning at NYU Shanghai. Experimental computation was supported in part through the NYU IT High-Performance Computing resources and services.
References
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. In International Conference on Robotics and Automation (ICRA), 2018.
Hien Dang. Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data. arXiv preprint arXiv:2301.0044 2023.
Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In International Conference on Machine Learning. pages 3004-3014 PMLR, 2021.
Cong Fang. Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, 118(43))e2003011 1 18, 2021.
Quinn Fisher, Haoming Meng. and Vardan Papyan. Pushing boundaries: Mixup's influence neural collapse. arXiv preprint arXiv:2402.06/71, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9, 2020.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:21/2./5/22 2021.
Quentin Gallouédec, Edward Beeching, Clément Romac, and Emmanuel Dellandréa. Jack of All Trades, Master of Some, a Multi- Transformer Agent. arXiv preprint 4rXiv:2402.09844, 2024. URL
arxiv orgo httg//bb/2402..000 09844
Li Guo, Keith Ross, Zifan Zhao, Andriopoulos George, Shuyang Ling, Yufeng Xu, and Zixuan Dong. Cross entropy versus label smoothing: A neural collapse perspective. arXiv preprint arXiv:2402.03979, 2024.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.0200 2021.
Hangfeng He and Weijie J Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36))e221104112)) 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross-entropy loss with imbalanced data. arXiv preprint arXiv:2309.0977 09725, 2023.
Kurt Horiik Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint 0rX1v:2202.08384, 2022.

---

Hoyong Kim and Kangil Kim. Fixed non non-negative orthogonal classifier: Inducing zero-mean neural collapse with fea-ture dimension separation.
Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.040% 2022.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint arXiv:2101.00072, 2020.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference Machine Learning, pages 28744. PMLR, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Minhyuk Seo, Hyunseo Koh, Wonje Jeung, Minjae Lee, San Kim, Hankook Lee, Sungjun Cho, Sungik Choi, Hyunwoo Kim, and Jonghyun Choi. Learning equi-angular representations for online continual learning. XrXiv preprint arXiv:2404.0166 2024.
Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. Advances in Neural Information Processing Systems, 36, 2024.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offiine RI. Workshop: Offline RL as a "Launchpad", 2022. /ttppttts://opeere..ee
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural- geometry. Advances in Neural Information Processing Systems, 15-27225-272333 2022.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, pages 21478-21505. PMLR, 2022.
Tom Tirer, Haoxiang Huang, and Jonathan Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning, pages 14301-34329, PMLR, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033, 2012. doi: 10. 109//ROS 2012.6386109.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo VicezzVicente,, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023.
org/record/8127005
Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012.054220 2020.
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in neural information processing systems, 15:37991-38002, 2022.
12

---

Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regresion by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In International Conference Machine Learning, pages 27 27179-27200 PMLR, 2022a.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. Advances in Neural Information Processing Systems, 35:3 199-131771100, 2022b.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820- 22834,, 2021.
13

---

A Experimental details for Section 3.2
A.1 MuJoCo
For Reacher and Swimmer environments, the datasets come from an open-source repository Gal louédec et al. 2024] and contain expert data collected by policy trained by PPO (Schulman et al. 2017]. The hopper dataset part the D4RL datasets [Fu et al., 2020). well-acknowledgee bench- all model hyperparameters and experimental settings used in section 3.2 In experiments, we train the models long enough so that the model weights converge. We provide more details below about the MuJoCo datasets employed and some hyperparameter settings depending on each dataset.
mark for offline reinforcement learning research. Sable[]sumaarzae
Table 2: Hyperparameter settings for experiments with weight decay on MuJoCo datasets.
Hyperparameter
Value
Number of hidden layers
3
Model Architecture
Hidden layer dimension
256
Activation function
ReLU
Number linear projection layer (W)
Epochs
1.5e6, Reacher
le6, Swimmer
2e5, Hopper
Batch size
256
Optimizer
SGD
Learning rate Wcight dccay
lc-2
Training
1.5c-3, Reacher
le-2, Swimmer
le-2, Hopper
Seeds
0, 1.2
Compute resources
Intel(R) Xeon(R) Platinum 8268 CPU
Number of CPU compute workers
4
Requested compute memory Approximate average exccution time
16 GB
16 hours
MuJoCo environment descriptions We use expert data obtained from Gallouédec et al. [2024) and Fu et al. [2020] for the Reacher, Swimmer, and Hopper environments. Reacher is robot arm with two joins; the goal of this environment is to control the tip of this arm to reach a randomly generated target point in 2-dimensional plane. Swimmer is
robot with three different body parts connected by two rotors; the goal of Swimmer is to move forward on a 2-dimensional plane as fast possible. Similarly, Hopper is 2-dimensional one-legged robot with four body parts, and the goal is to hop forward as fast as possible. All three simulated robots are controlled by applying torques the joints connecting the body parts. Those torques therefore the actions. In creating the datasets, online reinforcement learning was used to find expert policies [Gallouédec et al.) 2024 Fu et al., 2020). To generate the offline expert datasets, the expert policy is then applied to the environment to generate episodes consisting of states x; and actions (that is, targets) y.. The state x; includes robot positions, and angle, velocity, and angular velocity of all robot joints, and the targets y; include the torques on joints.
Low data regime Using regularized regression to train a neural network with expert state-action data is often referred to imitation learning. In this paper, we follow the common practice of using relatively small MLP architectures for the MuJoCo environments [Tarasov et al. 2022]. In imitation learning, it is desirable to learn good policy with as little expert data as possible. We therefore train the models with subsets of the expert data in the datasets for each of the three environments. Specifically, we use 20, and episodes (complete expert demonstrations) for Reacher, Swimmer, and Hopper, respectively. This corresponds to 1,000, 1,000, and 10,000 data points for the three environments, respectively. For each environment, we also take a subset of the full validation (test) datasct and keep the number of data 20% of training data size. As using fewer full expert
14

---

demonstrations for Swimmer, we increase the weight decay value to further mitigate overfitting in this case.
A.2 CARLA and UTKface
The Carla dataset collected by recording surroundings via automotive cameras, while a human driver operates vehicle in simulative urban environment [Codevilla et al. 2018). The recorded images are states x; of the vehicle and the expert control from the driver, which includes speed and steering angles, serves actions y; [0, 85) [-1, 1] in the dataset. A well-trained model on this dataset is expected to drive the vehicle safely in the virtual environment. The UTKfaae dataset consists of full-face photographs x; of humans whose ages range from to 1 116 (Zhang et al. 2017 The goal of this dataset is to accurately predict the age y, of the person in each photo.
In both cases, ResNet network [He et al., 2016) is employed the model backbone to extract image features. And the full dataset is used for training both models, as learning a good feature extractor from visual inputs requires large number of images [He et al. 2016 Sun et al. 2017]. To adapt ResNet architecture, a native of classification tasks, to regression tasks, we replace the last layer classifier with a fully connected layer to map learned features to the continuous targets. Depending on the task complexity, we select ResNet18 8 for Carla and ResNet34 for UTKface. The experimental setup for CARLA ID/2D and UTKKfaee datasets are summarized in Table 3
Table 3: Hyperparameters of ResNet for Carla and UTKface datasets.
Hyperparameter
Value
Backbone of hidden layers
ResNet18, Carla
Architecture
ResNet34, UTKface
Last layer hidden dim Final layer activation function
512
ReLU
Epochs
100
Batch size
512
Optimizer
SGD
Momentum
0.9
Learning rate
0.001
Training
Multistep aalm
0.1
Seeds
0,
Compute resources
NVIDIA A100 8358 80GB
Number of compute workers
8
Requested compute memory
200 GB
Approximate average execution time
hours
A.3
Computing NRC3
For univariate regression, note that WW7 /|/l|B and 1 - =0 where is vector of the final linear layer of the model; a is the standard deviation of the one-dimensional targets. Thus, NRC3 is trivially zero. Alternatively, to align with the theory in Section 4.2 one may define one-dimensional NRC3 as:
NRC3 = || 72 1113-11111
1/23 -
for some 71 E (0, Amiu) and 7/2 > 0. However, this is also trivially true, e.g. for any n (0, (note that Amin = 0²) and 72 = || ( 1/2)-1 after parameters w become stable. Therefore, we found NRC3 for univariate regression to be not meaningful, and therefore omitted the corresponding plots.
For multivariate regression, we run all experiments long enough in order to ensure that the training has entered the terminal phase of training as measured by 1t (see Figure 2}. After training, we extract the W matrix and identify that minimizes NRC3 for that specific W. This 7 was then used to compute the NRC3 metric for all W matrices during training, resulting in the NRC3 curves shown in Figure Figure visualizes NRC3 as function of y for the final trained W.
15

---

In Appendix we show that under condition that is satisfied if AWD is reasonably large, non- normalized version of NRC3, see (32). convex and it has unique minimum. Since we employ relatively large weight decay for experiments in Figure 2 the condition of Theorem F.1 is satisfied, and thus Figure R displays unique optimal y for all datasets.
A.4 Results for small weight decay
Figure and Figure 8 include results on studying small weight decay values for all datasets. When weight decay approaches zero, NRC1-3 typically become larger, compared with NRC1-3 obtained with larger weight decay values.
Particularly, when there is weight decay, observe that NRC1-3 has strong tendency to converge (There is a relatively small amount of collapse since gradient descent tends to seek solutions with small norms.), while the test MSE increases small MuJoCo datasets. Theorem 4.3 provides some insight: when there is no regularization, there is an infinite number of non-collapsed optimal solutions under UFM; whereas Theorem 4.1 shows that when there regularization, all solutions collapsed. When there is regularization, we are seeking small norm optimal solution, which leads NRC1-3.
16

---

Reacher
Swimmer
Hopper
Train MSE
10-2
10-1
10-3
10
0.75M
1.5M
0.5M
1M
100K
200K
Test NSE
10-
0.2
0.20
0.1
0.15
0.0
0.75M
1.5M
0.5M
1M
100K
200K
1.00
1.0
0.6
0.4
0.4
0.25
0.75M
1.5M
0.5M
1M
100K
200K
0.75
NRC1
0.50
0.5
0.25
0.00
0.0
0.00
0.75M
1.5M
0.5M
1M
100K
200K
1.0
1.0
1.0
NRC2
0.5
0.5
0.5
0.0
0.0
0.0
0.75M
1.5M
0.5M
1M
100K
200k
10
10-1
NRC3
0.05
10-2
10-2
0.00
0.75M
1.5M
0.5M
1M
100K
200K
Awo Awn Se 3
Awo Awn Se -3
Awo Se
Awn
Asn =
Aun m
Figure 7: Train/Test MSE, R², and NRC1-3 under different weight decays for MuJoCo datasets.
17

---

Carla
Carla 1D
UTKFace
1000
Train MSE
50
500
25
20
40
25
100
25
50
75
100
100
1000
Test MSE
200
75
500
100
AAAu
40
100
25
50
100
1.00
1.0
0.75
R2
0.4
0.50
0.5
0.25
20
40
60
25
100
25
50
75
100
1.0
0.6
NRC1
0.4
0.5
0.50
0.25
0.0
20
40
100
25
50
100
0.2
0.15
NRC2
0.1
0.10
0.0
20
40
60
25
75
100
25
50
75
100
Epoch
Epoch
0.2
NRC3
0.1
App le 2
Avo
Avo = 1e-2
Avo =
App
Avo 56-2
40 Epoch
App 18 -1
Se -4
Avp
5e
Figure 8: Train/Test MSE, R², and NRC1-3 under different weight decays for Carla and UTKFace datasets.
18

---

B Additional experimental results
In this section, delve into additional experiments aimed at further exploring the phenomena of neural regression collapse.
experiments that align with the UFM assumption and verify the theoretical NRC1-3. In addition to the L. regularization both H and W, the model thus is empowered with more expressive learned feature H, e.g. removing ReL.U in the penultimate layer to allow negative feature values and incorporating more training data. Complete results for all six datasets are shown in Figure 9 As we can see, NRC1-3 do not converge to very low values as is when regularization is stronger. This confirms our NRC theory in Section + and also corresponds to the results observed in Figure 7 and Figure X where we apply weight decay to all model parameters in practice.
Complete experiments under UFM assumption As studied in Section 4.5
Train MSE
NRC1
NRC2
NRC3
1.0
Reacher
MSS
Train MSE
0.75
0.050
App 2
10
N8C1
NRC2
0.01
0.50
0.025
Anp 2
0.000
SK
10K
0.045
Swinner
Train MSE
0.040
NAC1
0.5
MRC3
0.2
0.1
Jup Ia 1m-3
Ave
20
0.035
Anp
0.0
0.0
0.0
10k
5K
10k
Hopper
Train MSE
0.3
MRC1
0.5
0.02
0.04
Awo 1
Ave la 1
Awo
0.1
0.0
0.0
0.00
5K
10K
5K
10K
SK
10K
Carta 20 MSE
100
0.75
0.2
MRC
0.50
NRC2
NRC3
0.1
Jue 3
0.25
Ave 1e-4
0.1
0.0
50
50
50
50
0.2
Car 1D
Train MSE
1000
Apo 11--1
NRC1
NRC2
500
Aao 100-
2ac #0
0.0
50
100
50
100
50
100
UTKFace Train MSE
1000
Jac 11
500
NRC1
NRC2
0.2
0.7
46 550
00
50
100
100
50
100
Epoch
Epoch
Epoch
Figure 9: Empirical results with UFM assumption where L2 regularization on H and W are used instead of weight decay for all six datasets.
Norms of H and W As demonstrated in Corollary 4.2 jii), the norms of the last layer weight matrix and the feature matrix depend on the ratio of regularization parameters λH/Aw- Figure we empirically demonstrate how the regularization parameters impact the norms of the last layer weight matrix and the feature matrix. Specifically, we fixed Aw = 0.01 and varied the value of λH- We observe that with increasing λи. the feature norm monotonically decreases, and the norms of the rows of the weight matrix monotonically increase, which is consistent with our thcoretical result.
10
Connection to whitening In statistical analysis, whitening (or sphering) refers to a common preprocessing step to transform random variables to orthogonality. whitening transformation (or sphering transformation) is linear transformation that transforms set of vectors of random variables with a known covariance matrix into a new set of vectors of random variables such that the components of the transformed vectors are uncorrelated and have unit variances. The transformation
19

---

h-norm
wl-norn
A=m1e-1
Reacher
Ar=
0.2
Awmle-3
Au=le-d
0.1
Av=
500
Epoch
Epoch
Epoch
0.3
A=mle-1
Swinner
Au=le
0.2
0.2
Au=1e-3
0.1
Au-le-s
500
1000
500
1000
500
1000
Epoch
Epoch
Epoch
Figure 10: Comparison of the norms of W and H with fixed Aw and varying λμ The columns from left to right represent the model's average feature nomm and the norms for w and W2, respectively.
Random initialized model
Madel after convergence
0.4
0.20
1.00
0.3
0.15
0.10
Prediction ereer
Prodicton emrrr
0.00
-0.1
-0.05
-025
-0.2
-0.10
-9.50
-0.3
-0.15
-0.75
-0.4
-0.4
-0.2
0.0
0.2
0.4
-0.20
-0.2
-0.1
0.0
0.1
0.2
1.00
Prediction error gl)
Prodiction errar gli)
Figure 11: Residual errors (2)) versus (1)) for both the randomly initialized model and the trained model after convergence on the Reacher dataset. The color of the points indicates the ratio 2(2) (2))/201 (1)1)
is called "whitening" because it changes the input vector to white noise. Due to rotational freedom, there are infinitely many possible whitening methods, and consequently there is a diverse range of whitening procedures in use such as PCA whitening, Cholesky whitening, and Mahalonobis or zero-phase component analysis (ZCA) whitening among others. In the latter, the matrix used for the procedure of whitening is WCCA = [ where Σ is the covariance matrix the original data. Interestingly, W2CA is obtained as the whitening transformation that minimizes the MSE between the original and the whitened data, which is appealing since in many applications, desirable to remove correlations with minimal additional adjustment, with the aim that the transformed data remains as similar as possible to the original data.
From Corollary 4.2(v), v), we get the residual error of the regression model, E E RRXM which can be formulated as
= -= /1 Y).
The residual error matrix is proportional to the CCChwintened centered target matrix. we denote the ZCA-whitened target matrix by
2CA = -12 -Y
we have that
M M-IZZCA = I,
As consequence, the residual error matrix can represented as -CZZZCA from which significant implication arises. After the model converges, the residual error matrix will be mean zero white noise with sample covariance matrix given by M-'EE" = cI,, i.e. the errors are uncorrelated across the n target dimensions and each has variance equal to
20

---

For any given sample, upon examining individual dimensions, it becomes apparent that the j-th dimension of the residual error, elt), is proportional to the j-th dimension of the standardized target variable 2(i) We trained a 4-layer MLP model on the Reacher dataset for which the target variable is 2-dimensional to validate the Above-mentioned properties. We created scatter plots of the residual error g(2) versus e(2) for both the randomly initialized model and the trained model after convergence. Figure I1 illustrates these scatter plots, with the color of the samples indicating e(2) /2(1, As observed from the plot, after the model converges, the residual errors reduce to white noise. Additionally, from the right plot (for the model after convergence), we observe that the plot exhibits circular pattern where the color of the samples gradually changes as they move from one quadrant another. This indicates the consistency between e(2) e(2)/E() /e(!) and 2(2) (1(), which is
consistent with the result in Corollary 4.2 v).
21

---

C
Supplementary lemmas
Let us recall the form of the objective:
Hh 2M
dw
C(H. W. b) 2M
|WH+ b1 -Y +
H 1r
|WI
where 1/r - ................................ 1] and λH > O regularization constants.
In Lemma C.1 we demonstrate that if (H, w,b) is critical for then W can be written as closed-form function of H and the residual error. In an analogous way, H can be written as closed-form function of W and the residual error. Furthermore, b y, where is the mean of the targets. In addition, we provide the identity that connects the matrix norms of the two, see (iii) below.
Lemma C.1. i) /f (H, W, b) is critical point of 7 then
H - WT - (WH+ Y -)Y,,
Aw W
M (WH + Y Y)H?
y.
ii) (H, W. b) is a critical point of for fixed (H, W), b= minimizes L(H, W b).
m Aн H MXw || P.
Proof. i) To prove the first part of the lemma, we will proceed by equating to zero the gradients w.r.t. the variables the optimization objective Those can be written in the form of matrix in the following way:
UC = WT 1
dH
(WH + b1 Y) +
H, M
(8)
ÖH M
OC 1
(WH + b1 - Y)H + Aw W,
(9)
aW M
OC 1
(WH + b1 Y)L
(10)
M
We set 0c = 0 in (10) and observe that ab
1 Y WH)1M
Y1M
HIM
b
W
M y Wh.
(11)
M
M
recalling that y M- -= and = M-1 ΣM h;
We set ac OW = 0 in and observe that
1 M
Aw W
WH +b YYH
(12)
We set 0C 0 in X and observe that
2H
HиH = W (WH b1 Y),
(13)
Agh W (Wh, +b yy),
Vi = 1,.,, M,
(14)
Ahh W (wh (WWhb-y)) y).
(15)
We derived 15) by summing both sides of (14 over i, and subsequently dividing them by M.
Substituting b for Wh, see [II. we get
h
b b=
(16)
Thus, combining 12 13
and (66) completes the first part of the proof of
ii) If (H, W, b) is critical point of (7). noting that for fixed (H, W), the objective C(H, W, convex w.r.t b, readily yields that b minimizes L(H, W. b).
22

---

iii) Let now verify that λμ
H MMw W F. (H, W. is a critical point, then 0C
WT UL OW 0.
H
(17)
0H
Recalling the first-order gradients of w.rtt H and W respectively, see and (9. and substituting those (77. implies
AH
W
WWH + Y Y) +
H
H w
Y)H Aw W WH+
M
M
M
which gives
AH
HH = Aw Www
M
By definition,
λH
AH M
λH (r(HH" Awtr(W -w M
HH
M
and this establishes the assertion Aн | M.w [
Next, we touch upon various implications of Theorem 4.1 in the case when 0 < c < Amin. so that [A 1//2 = 1/2, where
A = Σ 1/2 cln.
(18)
For convenience, let us break the form of a global minimum (H, W, b), see 3 into three parts.
λA Aw
1/4
w
A A/
(19)
Aw
1/4
H
2
2 Y),
(20)
b y,
(21)
where R E Rnxd is semi- orthogonal, i.e., RR =R" In the following lemma, we demonstrate that the residual error is rescaled version of the centered targets, the value of the loss function the
global minimum can be computed directly by invoking the value of and the matrix norm A 1/2 Lemma C.2. Suppose 0 < c < Amin- For a global minimum (H, W, b) of we have that the residual error takes the following form:
-Y), Moreover, the value of the loss function L at the global minimum is calculated
WH + -Y-
C, = 2 + Voll(=1/2 ~Vln)
ne
Proof. By 19
21 for the first assertion,
WH Y A 1/RRR A 1/2 1 1 - -Y
[1 I (Y
=-VGE"/2111 -Y),
noting, for the first equality, that RRT I, A1/2A1/2 1/2 A. For the third equality, see (18) Therefore,
(WH+Y
Y)(WH Y +
Y)T
= cl,-
(22)
M
Using Lemma C.I
jii) and 22 we deduce
dy ir(A) VollA A 1/
C(H. W) which completes the proof of the lemma.
>w//
Aw dw
The proof of Corollary 4.2 directly follows:
23

---

C.1 Proof of corollary 4.2
(i) is derived in the proof of Lemma C.1 see 14 and (66). is also easy derive them from the
form of H as given in (3). (ii) follows by the form W, see By Lemma C.I (iii and Lemma (iii)-(v) follow immediately.
D Proof of theorem 4.1
The proof of Theorem 4.1 leverages their lemma in our notation.
Zhou et al. 2022a Lemma B.1]. For clarity, we now restate
Lemma D.1. [Zouu al. 2022a
Lemma B.1] For d, M with d 2 and Y == =- - Y Y
RM
with SVD given by Y U LEV
Ei=1 oruny where 01 2 02 on 0 are the singular
values, the following problem
min HERAX MT WERaxd
C(H, W.
a strict saddle function with no spurious local minima, in the sense that i) Any local minimum (H, W. y) of 7 is a global minimum of 7
with the following form
WH US VMWw AM I
Correspondingly, the minimal objective value of (7) is
C(H, W. y)
2 Σ (n o;)
+VV+MM MAw Σ 1=1
Tii
where 7/s := == Aw) ) the i-th diagonal entry of (f M
ii) Any critical point (H, W. y) that is not local minimum is strict saddle point with negative curvature, i.e., the Hessian at this critical point has least one negative eigenvalue.
Let - USVT E.= a,u,o denote the compact SVD of Y RRxM where 01 02 2 2 On > are the singular values, and Σ E Ruxn is diagonal, containing the aforementioned singular values. Furthermore, U E gax= V E Mxx are orthogonal and semi- orthogonal respectively, i.e. UUT = U II and VTV I, respectively. For the proof, recall the value of c = dwdH
Proof of Theorem 4.1
Let (H, w, be a global minimum of ① By Lemma D.1 (H, W, has
the following form:
WH US
VM
(23)
In light of Lemma
C.1 and the identity AE
| = MAw W 1.. from 23 we have that
W-
AH
1/4
UPS
R.
(24)
MMw
MAw)
1/4
RT VMC.IEVT
H
(25)
for E paxd such that RR = I,, Furthermore, using the SVD of = UEVT,
Y
Σ
U
T =
UT,
M
VM
VM
which deduces 1/2 U UT.
Since UT U-1 this further yields
VM
12 VCInl = Σ vMc, V-1
(26)
which implies that the matrices M / Vei,) and E v Mel, are similar. result, they have the same cigenvalues. The n matrix the left-hand side of (26) has eigenvalues given by
24

---

VMA. MM, 1,..., n, where X is the i-th eigenvalue (in descending order) of E whereas VMc, i = 1..... n are the (ordered) diagonal elements of the x matrix on the right-hand side of (26). So,
VA
for all i = 1,...,
(27)
M
Case I: 0 <c< Amin then by 27 is the case that o; MMc Vi, and thus |Σ VMcIn]
VM By (6)
Uy Mcl.
Ve
(28)
and thus (4 becomes
λH
W
Ve,,
for = UR E Rxd semi-orthogonal. The first assertion of the theorem follows by recalling the definition of = 11 VCl.
For the second assertion of the theorem, it remains to observe that
= -MUE-
VMUVT.
So, VT = ---1/20
1- and from 25 we get
1/4
H
Aw
R
R
1/1
V
Aw
1/4
R R - -Y)
HH
Aw
W
E 1 Y),
λH
where the second equality follows from (88
Case II: If c > Amax- then by (27). it is the case that 0; < / Me, Vi, and thus - -M = 0. By
(24 and 25 (H, W, y) (0, 0. y) desired.
Case I11: If Amin < C < Amax. by (27). it is the case that
lo; VMc - Joi Mcc, if < j* otherwise,
00
where j* max Xi c), and thus (2) MMc
=P -PS--V /Mclj.
By 26
U E v MMdn
1/2 = M
[5//2 - Veln
$
U,
(29)
and thus 24 becomes
1/4
W
Σ 1/2
Val
R.
Aw
j+
for R : UR E xd semi- orthoonlho..nal. The first assertion of the theorem follows by recalling the definition of 1/2 Vel,
For the second assertion of the theorem, it remains to observe that
Σ |-
= VM VUVT.
So, VT = M-1/2u
J 1/2 IÝ, and from 25 we get
Aw
1/4
1/2 vMcln
H
R
M
E
UT 11111-11
ÀH
Aw
1/4
R
11/2
va,1
| 1
- - Y)
ÀH
Aw
W 1//1-11(
Y),
XH
25

---

where the second equality follows from 29
D.1 Examples for theorem 4.1 (uncorrelated target components)
In this subsection, we examine closely the case when n = 3 and the target components are uncorre- lated. This simplifies considerably the problem as now the covariance matrix Σ is diagonal matrix with entries given (in order) by oj. where of denotes the variance of the j-th target component, for
j 1, 3. The unique positive definite and symmetric matrix A 1/2, see 18
is given by
(a
0
A 1/2
(30)
03
vc)
Without loss of generality, assume that max
02 03 min O.
If 0 < < o
min oz. by Theorem 4.1
j* = 3, and therefore any global minimum
H. W. b) of (
takes the following form:
1/4
Aw w Σ 1 Y),
W
A //RR,
H
b = y,
Aw
for any semi-orthogonal matrix E Baxd The form of 1/2, see (30). readily yields
0 - 1)ee, ej: c1/2
w
λH
j j= 1,233
c.f., 4 where (e, j 1,2,3) 3) is any collection of vectors lying in Ied such that ey is unit vector, for all = 2,2,3, and e₁ is orthogonal to Cg for all 1: o K. To interpret the landscape of global minima in the case when the target components uncorrelated, the UFM "forces" the angle between the weight matrix rows to be x/2 (fixes the weight matrix rows be orthogonal). Then, the configuration of the bwy 's is exactly as in the I-dimensional target case, that is those are restricted to lie on spheres of certain radiuses. The feature vector h, that corresponds to the i-th training example is then the 3-dimensional subspace spanned by Wi, W2 and W3.
If o2 min,
j* 2. We analyze the latter, in which case << oi. < oz but > oz. By Theorem 4.1 any global minimum (H, W, b) of 7 takes the form below:
< 2 maxx by Theorem 4.1 j*
HA Aw
1/4
Aw w /=//1-1 YY), λA
W
12
y,
for any semi-orthogonal matrix R E 1xddd The form of [A2/2| see 30) readily yields
T
j= 1,2,
c1/2
0,
c.f., 4
where e1, 02 E Rdd unit vectors orthogonal each other. It is also worth mentioning
that
dw
h;
[ w/ 1
-y:
dw
(y -5(1)
(y!
-5(2))
T1 w
W
Hy
02
for all ;= 1,..., M. In other words, for fixed the feature vector h, lies span ( wff w
Moreover, the previous formula indicates that the inner product between h, and w/ is proportional to the j-th standardized target component.
26

---

The analysis of the case j* = i.e., < of but e > o2. 03 is analogous, therefore we just record the form of the w, 's and h,'s, omitting any further details:
w
1)e,
w = w = O,
c2/2
(1) - 5(1)
dw
h;
Wi.
HH
for all i = 1,... M. In other words, for fixed i, the feature vector h; is colinear with wf.
If с > max = of. by Theorem
4.1
H W. = (0,0.y)
E
Proof of theorem 4.3 (no regularization)
We first show
min L(W H 0
Clearly L(W, .H) 0 for all W and H. Now consider any fixed x d matrix W with rank W) Since W has rank n, (Wh h E [d R Thus there exists h, E Rd such that Wh; y: for all 1..... M. Let minw H L(W, H) O.
[hi- ... hm]. For this W and H we have L(W, H) = 0. Thus
Now let W be any d matrix with full rank n, and consider the set of H that satisfy L(W, H) w.r.t this W. This is a standard least squares problem for which the solution is well known:
H = w Y + (Ls W WtW
(31)
where W is the pscudo-inverse of W and Z is any x M matrix.
To complete the proof, we need to show that if rank W) n. then W cannot be part of an optimal solution for T.W,, H). Suppose rank (W) < n. Wh y only has solution if y lies in the column O only if y all lie in the column space of W. Since rank( Y) n and rank( () < n, there will be least one y; that not in column space W. Thus for this y; there is no h; such that Wh; y.. Thus for this y, will have (y, Wh,) )a > 0. implying W cannot part of an optimal solution W, H.
space of W. Thus L(H, W)
Finally, we note from 66 that a column h in H is the sum of two vectors, with the first vector lying in the row space of W and the second vector lying in the (orthogonal) null space of W. Since W has full rank, this implies that the columns of H can span all of R.
F Proof of the uniqueness of
Mathematically, we can show that, under a condition that is satisfied if AW is reasonably large,
NRC3(7) = ww 11/2 2//211
(32)
given in the definition of NRC3, without normalizing the Gram matrix of W and 11/ 1/2 I... convex and it has a unique minimum.
Theorem F.1. If
tr( tr(1//2) > tr( WWF)
NYC3(y), as given in 32 is minimized at
ir E (12 tr IT
Moreover.
NRC3()") = (WW tr 1w(W
tr( 1/(2
I,
11/2
I,
n
n
27

---

Proof. Expanding the squared matrix norn appearing in the definition of (RC3()), it is necessary and sufficient to minimize
f(>) : tr(WW*) tr(x1//) + +17,
which has first and second derivatives given by
f(()) = 1r(WWT) - 1r12)/
+
71/2
f"()) 1r(21/2 tr (wWW
2/3/2
Since tr (r(2 > tr( tr(WWT), by the 2nd derivative test, f. and consequently NRC3(7), is convex with unique minimum achieved at
tr Σ (1/22) tr WWWT
28

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract, we claim to empirically demonstrate the phenomenon of Neural Regression Collapse as discussed in Section 3.2 and then we provide a theoretical explanation from the perspective of the Unconstrained Feature Model, see Section 4 and Appendix B Thus, we show that the phenomena of neural collapse could universal behavior in deep learning both empirically and theoretically.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
we point out that our explanation of neural regression collapse Justification: In Section 5
doesn't t have implications for model generalization.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low or images are taken in low lighting. Or a speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
29

---

Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
In Section 4 and Appendix B
we provide full set of assumptions and
proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Sections 3.2 and Appendix A we provide a comprehensive description of the experimental setup, including detailed information on the datasets used, model architectures, and hyperparameter settings. This detailed disclosure ensures that other researchers can reliably reproduce the experimental results and validate the claims made in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution should either be way to access this model for reproducing the results way to reproduce the model (e.g.., with an open-source dataset or instructions for how to construct the dataset).
new model (e.g.., large language model), then there
30

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e. to registered users), but should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We upload the code with environment in the supplemental materials. The datasets used are all open-source datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cer
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g.., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ //nips. t/nips.cc/publl// wwtppbbiccguudesse. hetoli/c/giidessdeeeuubiiiis wwweeeSuummsssoooooooooo
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (e.g.,, data splits. hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Sections 3.2 and Appendix A we provide comprehensive description of the experimental setup, including detailed information on the datasets used, model architectures, and hyperparameter settings. More details can be found in the code in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
31

---

Justification: All the results are reported in terms of learning curves, and each figure includes many plots, so error bars are not reported. But we do run the experiments multiple times and observe very similar performance in terms of NRC, testing loss, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g... Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative eτror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix A we provide the details for computation resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We are sure that the research presented in this paper adheres to the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g... if there is a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
32

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper mainly focuses showing and understanding the neural regression collapse phenomena observed practical neural networks and unconstrained feature models. No potential negative societal impact is expected of this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g... disinformation, generating fake profiles, surveillance), faimeess considerations (e.g.., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g... gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g... code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the owner of assets are properly cited the reference and main body of our paper.
Guidelines:
33

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g.., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g... website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "MTQS75WtAn", "reviewer": "Reviewer_vXK1", "summary": "The authors explore a new notion of neural collapse which has been formulated to accommodate multivariate regression tasks. NC was originally introduced and recognized as an artifact of multi-class classification tasks. While there has been extensive research into the phenomena of neural collapse for classification, this paper is interesting in that it is the first theoretical study of NC for regression problems. The results are novel and the methodology is sound. \n\nThe paper is nice because it expands the view of NC in deep learning to cover regression as well as classification. The authors shed further light on this phenomenon and their results fit nicely in the existing literature. Being a new viewpoint (e.g. regression analysis) on a well-studied phenomenon (e.g. neural collapse), there is some question as to whether their formulation is the right approach. For me, their definition seems compelling. \n\nEssentially, the idea is that instead of looking at the class means or class features and the simplex formed by those vectors in embedding space, you instead look at the $n$ principal components of the sample features in embedding space where $n$ is the dimension of the target space (e.g., n=1 for univariate regression). In the same way that NC for classification says the class feature means collapse to an ETF, for the regression setting, the sample features collapse to the $n$ principal components in embedding space.", "questions": "There is extensive analysis of the neural collapse to the $n$ principal components. Did you do any work to justify that $n$ is the right number of principal components to assume? The embedding dimension could be quite large in comparison. How would your analysis change if you let $n$ be different from the dimensionality of the targets? For example, what happens to your NC conditions when you take $n=1$ for one of your robotic datasets? Or set $n=3$ for I realize this doesn't lend itself to as clear an interpretation (if any) but to play devil's advocate it would be helpful to see that analysis, particularly in the form of some explained variance ratio. For classification tasks, the structure of the ETF during neural collapse is very natural. For a regression task, particularly when introducing a new definition, it requires an additional justification or at least an additional analysis. \n\nHigh level question out of curiosity: There are standard ways to convert a (univariate) regression task into a multi-class classification task (e.g. via quantile regression). Similarly, there are standard ways to express a classification task as a regression taks. How would the standard notion of NC for classification compare with your formulation of NC for regression between these two formulations? It would be interesting to see how analaglous or stable this proposed definition is under such a re-framing.\n\nline 71 (clarification): you state that when the regression parameters are zero or very small, there is no collapse. Does that mean all NC1 - NC3 fail? or just some?\n\nline 77: you mention that this framing of NC for regression can lead to more explainable models and potentially more efficient training processes. Can you reasonably justify this claim? How exactly? Do you have any references or reasonable examples (albeit from the classification setting) that you can point to and how those assumptions would compare to a regression task?\n\nIn the related work (lines 88-92) you discuss previous work concerning NC for classification tasks when the classes are imbalanced. In the imbalanced classification setting, some of the original NC properties no longer hold and/or they must be reformualted to account for the class imbalance. How does one translate the notion of balanced dataset for regression task? This aspect seems to have been ignored entirely. Can you add some more detail justifying why it's been ignored or how it relates to your current framing?\n\nline 150: you state \"typically $d=256$ or larger and typically $n \\leq 5$. Typically with respect to what? \n\nline 199-201: you state \"This indicates that NC is…a fundamental property of multivariate regression\" This feels like an overstatement. This paper is examining 6 datasets over two model classes. The results are certainly very promising but the world of multivariate regression and model architectures can be vastly larger.\n\nIn Figure 2: NRC3 states that there exists a constant \\gamma \\in (0,\\lambda_min) such that the derived quantity goes to zero. What value of \\gamma are you using in the plots for NRC3 here? According to Figure 3 it seems that there is a unique (?) optimal value for \\gamma which minimized NRC3\n\nline 202: you mention that each dataset (excluding CARLA1D and UTKFace) exhibit a unique minimum value of NRC3 for the range of \\gamma's explored. Why is that the case? I may have missed this in the exposition and theory.\n\nIn Figure 3: Why are CARLA1D and UTKFace not included? Presumably bc these have n=1. One could still evaluate \\lambda_{max}. What do those corresponding plots look like for these univariate datasets?\n\nIn Figure 4: you only show results for CARLA2D and Swimmer datasets. Where the other datasets not included in your experiments? It would be nice to see comparable results in the appendix at least, particularly, if you're claiming the ubiquity of these results for multivariate regression tasks.\n\nline 207: you claim the geometric structure you propose NC1-NC3 is due to the regularization and, in fact, is not exhibited without at least some regularization. Particularly, when there is no regularization why does NRC occur (in any criteria)? And do the models correspondingly not converge? Or do they? Can you include the Train MSE (and/or Test MSE) in Figure 4 as you do in Figure 5? This point about the regularization constant seems important but admittedly remains unclear to me. You mention that this is addressed later (e.g. section 4.4), but the subsequent section is focused only on the UFM model. There is an underlying assumption that the UFM model encapsulates the NRC setting similar to what has been demonstrated for classification NC. As for intuition, I agree. But a rigorous logical connection for the regression case remains unclear to me. \n\nIn Figure 5: I assume when \\lamba_H=\\lambda_W=0, we would see the training diverge or at least the NRC values diverge? Is that true? What happens in this setting that connects with what we see in Figure 4. \n\n(typos/nits)\nIn Figure 4: I would change the labeling of the y-axis to denote the quantity being measured (e.g. NRC1 - NRC3) and use the plot titles to indicate the dataset. However, I guess this would turn your 2x3 plot into a 3x2 plot and not be the most efficient use of space. Perhaps you can do something similar to how you've displayed Figure 5.\n\nFigure 4 and Figure 5 seem comparable to me (an exploration of the NRC1-NRC3 values for small weight decay regularization constants). But why is there an inconsistency between the datasets examined. For Figure 4 you look at CARLA2D & Swimmer and for Figure 5 you look at Swimmer and Reacher. Why? This is even more glaring because the remaining datasets aren't included in the appendix either. Is that because the results aren't as good?\n\nFigure 5: the caption is not very descriptive. I recommend adding more detail here.", "limitations": "Yes, the authors have addressed the limitations", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and well motivated. This is an interesting problem and it's nice to see such a well thought out first attempt at extending an important phenomenon (i.e. neural collapse) to a more broad task setting. The theory is solid and well supported with clear and careful proofs.", "weaknesses": "It would be nice to see more comprehensive and consistent experiments (I've outlined suggestions below). The authors are introducing a new concept; or more specifically, they are reframing a well-known concept to a new setting (i.e. neural collapse to neural regression collapse). It is of course impossible to include all possible datasets and architectures but when claiming such a novel and fundamental property of deep learning there is an additional onus on the authors to be as comprehensive as possible.\n\nOverall, the primarily weaknesses are surrounding the experiments which good but lacking. There are a few things missing from the experiments that I would like to see and would alleviate some of my confusion around these results:\n1. None of the plots in Figures 1-6 include error bars. Were multiple seed trials run for these experiments? This seems to be a glaring omission\n2. Indicate the value of \\gamma that is being used for the NRC3 plots in Figure 2.\n3. in Figure 3, is this unique (?) minimizer for \\gamma/\\lambda_{max} stable or consistent across multiple trials with differing seeds and initializations?  \n4. the representation of datasets is not comprehensive and inconsistent. I understand not all plots can be contained in the main paper but it would be nice to see them in the Appendix. For example Figure 4 and 5 (which examine similar quantities) have Figure 4 looking at CARLA2D & Swimmer and Figure 5 looking at Swimmer and Reacher. Figure 3 does not include CARLA1D or UTKFace datasets. Why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your very detailed and insightful review, and for all the positive comments you made regarding our work. We also fully agree that the presentation of the experimental results can be improved. During the rebuttal week, we have worked very hard, running additional experiments and reorganizing our figures to respond to your comments. We believe our experiments are now “comprehensive and consistent”. We promise to include _all_ additional results below in the future revision.\n\nWe ran all experiments with 3 random seeds and plot the variance across seeds as the shaded area. As shown in all figures, there is little change across seeds, confirming that NRC consistently emerges in regression.\n\nConcerning your question about 𝜸 in Figure 2: We ran all experiments for long enough to ensure the training enters TPT measured by $\\mathbf{R^2}$ (See Figure A).  After training, we extracted the $W$ matrix and identified 𝜸 that minimizes NRC3 for that specific $W$. This 𝜸 was then used to compute the NRC3 metric for all $W$ matrices during training, resulting in the NRC3 curves shown in Figure 2. Figure 3 visually shows NRC3 as a function of 𝜸 for the final trained $W$. Mathematically, we can show that if $\\lambda_{WD}$ is sufficiently large, NRC3(𝜸), as given in the definition of NRC3, is convex and has a unique minimum.  Empirically, our experiments show that 𝜸 w.r.t. minimum NRC is consistent across different seeds. \n\nWe very much liked your suggestion to examine the explained variance ratio. The results are striking as shown in Figure C: for all datasets, there is significant variance for the first $n$ components; for other components, there is very low or even no variance. We also examine different definitions of NRC1 with $n$ varying from 1 to 4 below to justify the choice of $n$ being equal to the target dimension. \n||NRC1_pca1|NRC1_pca2|NRC1_pca3|NRC1_pca4|\n|:-:|:-:|:-:|:-:|:-:|\n|Reacher|1.01e-2|4.02e-4|2.55e-6|2.27e-12|\n|Swimmer|5.48e-1|3.96e-11|1.10e-11|5.02e-12|\n|Hopper|0.617|0.558|0.499|0.129|\n\nConcerning your question about converting a univariate regression problem to a classification problem, and vice-versa. We believe this is an interesting direction for future work. Of course, if $n$ is large, it would become difficult to convert from regression due to quantization and the resulting classification dataset would likely be highly imbalanced. In classification, as you state, the dataset can be balanced or imbalanced, and there are NC studies for both. If we were to convert a univariate regression dataset to a classification dataset using quantization, then the dataset would be balanced if there are the same number of $y_i$ values in each quantile. As this property is likely unrealistic for most datasets, then we can conclude neural regression has a stronger connection to classification with imbalanced datasets. \n\nWe believe that the UFM theory provided in the paper helps to explain many properties of neural regression. In terms of efficiency, experiments in Figure D show that we can fix W to a random matrix such that $WW^{\\top}=\\Sigma^{1/2}$ as suggested by Theorem 4.1, and use this fixed matrix throughout training. This approach significantly reduces the number of weights to be optimized and results in similar performance and NRC metrics, paralleling a well-known approach in NC for classification.\n\nThe number of features in the penultimate layer $d$ is typically 256 or larger for deep learning. In multivariate regression, the target dimension $n$ is typically small in comparison. For example, in most MuJuCo environments, often used to study imitation and reinforcement learning, satisfy $n \\leq 6$.\n\nWe agree that “NC is…a fundamental property of multivariate regression” is a bit of an overstatement. We will rewrite this to say “often occurs in neural multivariate regression”. \n\nWe ignored the plots for 𝜸 and NRC3 for univariate regression with $n=1$. For $n=1$, $\\lambda_{max} = \\sigma^2$, the variance of the 1D targets. This is just a scalar value for each of the $n=1$ datasets and does not seem to lead to an insightful plot. \n\nConcerning your question about which NRC metrics fail when regression parameters are zero or very small. Figure A and B show when weight decay approaches zero, NRC1-3 typically become larger, compared with NRC1-3 obtained with larger weight decay values. \n\nParticularly, when there is no weight decay, we run more training epochs to verify the asymptotic behavior of NRC1-3 and test MSE in Figure A. We observe that NRC1-3 has a strong tendency to converge (There is a relatively small amount of collapse since gradient descent tends to seek solutions with small norms.), while the test MSE increases on small MuJoCo datasets. Theorem 4.3 provides some insight: when there is no regularization, there is an infinite number of non-collapsed optimal solutions under UFM; whereas Theorem 4.1 shows that when there is regularization, all solutions are collapsed. When there is regularization, we are seeking a small norm optimal solution, which leads to NRC1-3. \n\nThe main assumption in the UFM model – for both classification and regression – is that the neural network is capable of mapping any set of training inputs to any set of feature vectors. Based on this assumption, the UFM model leads to a new optimization problem for classification and regression. We believe the UFM model for regression makes the same logical connection that is made for classification. \n\nWe have included UFM experiments with no regularizer in Figure B. Similarly as in Figure A, NRC1-3 do not converge to very low values as is when there is regularization. As indicated in section 4.5, Figure 5/B is intended to validate the results under the UFM framework. To align with the UFM assumption, we remove ReLU in the penultimate layer such that the learned feature can be any real number instead of being restricted to positive values.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lk768zMePn", "reviewer": "Reviewer_6Hcy", "summary": "This paper studies the neural collapse phenomena in neural multivariate regression. The authors rigorously analyze the neural collapse behavior using a simplified model that only includes the last two layers. It was shown that in the multivariate regression task, the last layer of the neural network would collapse to a certain structure that aligns with the principle components of the last layer features. Moreover, the authors highlight the importance of regularization in the prevalence of neural collapse and argue that small regularization may lead to a non-collapsed solution. Experimental results on various real-world datasets are presented to support the theoretical findings.", "questions": "1. Since the neural collapse is primarily for the terminal phase of training, it is helpful to include $R^2$ of each experiment to confirm the network has entered the terminal phase of training.\n\n2. Is there any specific reason for not regularizing on the bias term in the UFM (Equation 1)? \n\n3. In theorem 4.3, it was shown that without regularization, there are an infinite number of global minimums that are not collapsed solutions. However, it was well known that linear regression trained with gradient descent is implicitly biased towards minimum norm solution. Therefore I am curious about if collapse will happen in the optimization result on UFM.\n\n4. Discussing the difference with [1] is beneficial, it shows neural collapse happens in classification tasks without any regularization, while the current work shows this is not the case in regression.\n\n[1] An unconstrained layer-peeled perspective on neural collapse", "limitations": "The authors have addressed the limitations properly.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presented study is novel, it discovers the neural collapse phenomenon in multivariate regression, extends the boundary of neural collapse, and provides new understanding of neural multivariate regression. The theoretical results and experimental results are solid and well-organized.", "weaknesses": "My major concern is the significance and potential impact of the results: \n\n1. Neural collapse under MSE loss has been extensively studied in the literature already [1]. The analysis technique in this paper is not new, and the only difference is in the distribution of Y.\n\n2. The authors don't provide enough evidence to support the potential impact of neural collapse in multivariate regression. For example, does neural collapse imply better generalization or robustness in regression [2]? Does fixing the last layer to be neural collapse help the training [3]?\n\nOverall I would encourage the authors to add more discussion about the practical message to increase the impact of the current paper.\n\n[1] Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path\n[2] Prevalence of Neural Collapse during the terminal phase of deep learning training\n[3] A geometric analysis of neural collapse with unconstrained features", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed review and helpful comments. We also thank you for recognizing that the “study is novel,” “provides a new understanding of neural multivariate regression,” and that the “theoretical results and experimental results are solid and well-organized”.  \n\nWe agree, however, that we didn’t sufficiently emphasize the significance and impact of the results. Indeed, neural collapse for the classification problem under MSE loss has been extensively studied. We will update our paper to fully describe our contribution with respect to earlier works on MSE loss for classification. Indeed in our proofs, we leveraged important existing results from the UFM theory for MSE loss. But the regression problem remains substantially different from the MSE classification problem, a difference which is highlighted by a very different definition of neural collapse. Classification with a balanced data set gives rise to an ETF geometric structure, whereas multivariate regression gives rise to a distinctly different structure defined by our NRC1-3 definitions which involve subspaces spanned by principal components and the covariance matrix of the targets.  This new definition requires an entirely new empirical analysis on entirely different datasets. And although the mathematical UFM derivations for regression mirror those for classification, the qualitative nature of the final results - involving the covariance matrix of the targets and their eigenvalues - is substantially different than those for classification. \n\nIn terms of impact, the NRC properties offer valuable insights into model training for regression tasks. For example, we can fix the last layer to neural collapse solutions by setting the weight matrix $W$ to be a random matrix which satisfies $WW^{\\top}=\\Sigma^{1/2}$ as suggested by Theorem 4.1. By training the model with the weights $W$ kept frozen, we can significantly reduce both the number of parameters and the computational burden, while still maintaining comparable performance in terms of training/testing MSE and NRC metrics, as shown in Figure D. We will include this figure and the corresponding discussion in the revised version. Thank you for your suggestion. We believe that the empirical and theoretical results in this paper will guide the future design of multivariate regression problems.\n\nThank you for your suggestion to investigate R2 during the terminal phase of training. In Figure A of the supporting PDF we provide the $\\mathbf{R^2}$ results, which confirm the network has entered the terminal phase of training.\n\nWe omitted regularization of the bias term for the UFM for two reasons. First, in practice, it rarely improves performance. Second, it allows for a cleaner presentation of the theorem statements and proofs. It could be easily added without changing the main takeaway points. \n\nThank you for your interesting comment about no regularization. We agree that gradient descent often exhibits an implicit bias towards a minimum norm solution. We have added the curve corresponding to $\\lambda_{H}=\\lambda_{W} = 0$ in Figure B, which indeed shows to be the case, with the NRC metric typically decreasing as training progresses. This is consistent with the observation in the layer-peeled paper, as you indicated. We will highlight this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ap24BYeVQ4", "reviewer": "Reviewer_T8Jo", "summary": "This paper investigates Neural Regression Collapse (NRC), a new form of Neural Collapse observed in multivariate regression tasks. NRC is characterized by three phenomena: (NRC1) last-layer feature vectors collapsing to the subspace spanned by the principal components of feature vectors, (NRC2) feature vectors collapsing to the subspace spanned by weight vectors, and (NRC3) the Gram matrix for weight vectors converging to a form dependent on the targets' covariance matrix. Empirical evidence from various datasets and architectures supports NRC's prevalence. The Unconstrained Feature Model (UFM) explains these phenomena, indicating NRC emerges with positive regularization parameters. This study extends Neural Collapse to regression, suggesting a universal deep learning behavior.", "questions": "Could the authors explain in detail how Eq. (24) and Eq. (25) are derived in the appendix?", "limitations": "see Weaknesses", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the significant issue of Neural Collapse in regression tasks, extending its understanding beyond classification and suggesting a universal behavior in deep learning models.", "weaknesses": "in line 601, for $\\left[ \\tilde{\\Sigma} - \\sqrt{Mc} \\right]_{+}^{\\frac{1}{2}}$, there is no \\( \\mathbf{I}_n \\) in the matrix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the detailed review and helpful comments. We also thank you for recognizing NRC as a “new form of Neural Collapse observed in multivariate regression tasks” and that “this study extends Neural Collapse to regression, suggesting a universal deep learning behavior”.\n\n**Weaknesses:**\n\nIndeed, in line 601, in the matrix that you quoted, multiplication of $\\sqrt{Mc}$ by $\\mathbf{I}_n$ is needed. We have spotted this typo in our manuscript and corrected it.\n\n**Questions:**\n\nFinally, let us now elaborate on the derivation of Eq. (24) and Eq. (25) in the Appendix. The proof of Theorem 4.1 leverages [Zhou et al., 2022a, Lemma B.1]. For clarity, we have restated their lemma in our notation, see Lemma D.1 in Appendix D. In order to derive $WH$, Eq. (23) in our work, Zhu et al., first showed that, if $w_i$ and $h^i$ denote the $i$-th column of $W$ and $H^T$ respectively, then, when $w_i\\neq 0$ and $h^i\\neq 0$ (if this is not the case, then $w_i=h^i=0$), we have that\n$$\nw_i = u_i ||w_i||, \\qquad h^i = v_i ||h^i||, \\qquad \\lambda_H ||h^i||^2 = M \\lambda_W ||w_i||^2, \\qquad \\sqrt{\\frac{M \\lambda_W}{\\lambda_H}} ||w_i||^2=\\sigma_i - \\sqrt{M c},\n$$\nwith $\\sigma_i\\ge \\sqrt{M c}$, for all $i=1,...,n$, where $\\sigma_i$ is the $i$-th singular value of $\\tilde{Y}$ and $u_i$, $v_i$ are the corresponding left and right singular vectors. Rewriting the equation above (see Eq. (22) in [Zhou et al., 2022a, Lemma B.1]) in matrix notation readily yields Eq. (24) and Eq. (25) in the Appendix. We will include this clarification in the Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suR24qFGIm", "reviewer": "Reviewer_8FCG", "summary": "The paper introduces a novel extension of neural collapse to neural regression collapse, demonstrating that a similar phenomenon exists in multivariate regression. It treats the last layer feature vectors as free variables when minimizing the loss function and derives results similar to those of traditional neural collapse.", "questions": "- If the final layer regression directly uses the features $H$ learned by the preceding neural network through weights $W$, and $H$ already has good properties, can it be understood that the regression model has no actual significance?\n- Does the regression here only refer to linear regression? Can nonlinear forms, such as GP regression, also be considered?\n- In line 222-225, why the norms in UFM regression cases are finite?\n- In Figure 5, it seems that as the value of the regularizers decreases, the effectiveness of NRC decreases while the performance of MSE improves. What is the reason for this? Additionally, could the experimental results from Section 4.4 (removing regularization), be added to the figure for comparison?\n- In the experiments validating NRC, the authors used six datasets. However, when testing the effect of the regularizer, did you only select one of these datasets? How does it perform on the remaining datasets?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and has a good structure. \n\nThe authors provide a detailed explanation of NRC following the framework of NC. They validate NRC through experiments on multiple datasets and discuss the role of regularizers. \n\nThe paper also includes comprehensive proofs, demonstrating NRC's performance under the UFM setting.", "weaknesses": "- It seems that Neural Multivariate Regression simply turn the final layer's classification into a regression loss? And the two are just different in form, but ultimately can both be written as MSE loss, making little difference in analyzing the NC phenomenon.\n\n- Since NC is a model-agnostic framework, after changing the task, how are the conclusions of NRC different from those of traditional NC? I think this should be further clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the detailed review and helpful comments as well as for recognizing that our work “derives results similar to those of traditional neural collapse”. \n\nWe acknowledge that the analysis for NRC bears some similarities with traditional NC for classification tasks with MSE loss. In our proofs, we leveraged some existing results from UFM theory for MSE loss. However, the regression problem is substantially different from the MSE classification problem, resulting in a very different definition of neural collapse. In classification with balanced data, neural collapse refers to the convergence of the last layer features and weight matrix to an ETF geometric structure. In contrast, in multivariate regression, the feature embeddings converge to an n-dimensional subspace spanned by the principal components of the features, which converge to the subspace spanned by the row vectors of the weight matrix $W$. Moreover, in regression tasks, the weight matrix satisfies distinct properties that depend on the covariance matrix of the target variables and their eigenvalues. Therefore, the qualitative nature of the final results is substantially different from those for classification. In addition, the NRC properties provide valuable insights for model training in regression. As demonstrated below, we also present training with a fixed $W$ based on the neural collapse solution, which improves training efficiency without compromising performance. We anticipate that further exploration will reveal additional implications, which we leave for future work.\n\n**Questions:**\n\n* Indeed, the weights $W$ and features $H$ in the final regression layer have desirable properties, as highlighted by Theorem 4.1. Consequently, instead of learning the final regression layer, we can design the weight matrix $W$ as a random matrix such that $WW^{\\top}=\\Sigma^{1/2}$ as suggested by Theorem 4.1. By training the model with the weights $W$ kept frozen, we can significantly reduce the number of parameters and the computational complexity of training, while still achieving comparable performance in terms of both training/testing MSE and NRC metrics, as illustrated in Figure D.\n\n* This paper focuses on deep regression with a final linear layer, where the network is highly overparameterized and capable of learning a wide range of representations. In lines 217-221, we discussed the resemblance of the UFM with standard (multivariate) linear regression. On a high level, one can regard the features $h_i$, $i=1,...,M$, as the inputs to linear regression. Then, the objective becomes the minimization of the squared loss between the predicted outputs $Wh_i+b$ and the targets $y_i$. In standard regression, the $h_i$’s are fixed inputs, whereas in the UFM, we are optimizing over the weights $W$, the biases $b$ and most importantly over the “inputs” $H$.  As for GP regression, this is indeed an interesting research direction, but one that is certainly very rich and beyond the scope of the current paper.\n\n* The norms of $W$ and $H$ have different training dynamics under classification and regression tasks. In classification tasks with CE loss and no regularization, once features can be perfectly separated (during TPT), the training objective will always decrease if we fix the direction of H and W and only increase their norms (similar to temperature scaling), with the optimum attainable only at infinity. However, with MSE loss without regularization, the loss function increases towards infinity as the norm of the predictor Wh_i approaches infinity. The MSE loss function therefore forces the norms of the predictors to be finite on its own without the need of regularization. The actual optimal solutions for W and H are characterized in Theorem 4.3 (no regularization case) which shows that there are actually an infinite number of finite-norm solutions.\n\n* In Figure 5, the train MSE decreases when decreasing the regularization constants, which mirrors the decrease of the train MSE in Figure 4, when decreasing $\\lambda_{WD}$.  This observation aligns with modern machine learning practices, where training MSE tends to be lower with no or weaker regularization. Moreover, Figure 5 exhibits a phase change in NRC, with NRC being more pronounced with stronger or higher regularization constants. Following your suggestion, we have added the experimental results corresponding to no regularization to Figure 5; see Figure B in the rebuttal PDF for these additional results.\n\n* Figure 4/A has been updated to include experiments on all six datasets, providing a comprehensive validation of the impact of regularization. We have also updated Figure B to include the Reacher, Swimmer, Hopper, and CARLA2D datasets. Due to space limitations, CARLA1D and UTKFace are not included in Figure B and will be added in the future revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Since 2020, when [Papyan et al., 2020] published their seminal paper on neural collapse, there has been a flurry of activity in the area, with at least a dozen papers on the topic of neural collapse published in major machine learning venues. To our knowledge, this entire stream of important research is focused on the classification problem. 

As discussed in the introduction of our paper, arguably, regression is at least as important as classification in modern machine learning. To our knowledge, our paper under review to NeurIPS is the first paper to examine regression, including multivariate regression, in the context of neural collapse. The paper first proposes a brand-new definition for neural collapse for regression with a very different geometric description than the ETF definition appropriate for classification. The paper then presents extensive experimental results establishing that multivariate regression indeed exhibits neural collapse as defined by this new definition. In this rebuttal, we supplement our original experimental results with a suite of new experimental results, as requested by the reviewers, further confirming the prevalence of neural collapse in regression. Our submission also considers an approximation in which the neural network is assumed to be capable of mapping any set of training inputs to any set of feature vectors, the so-called Unconstrained Feature Model (UFM), which has also been used to analyze neural collapse in classification. Under the UFM model, we derive the explicit solutions for the optimal features, last-layer linear weights, predictors, and MSE training error, and show that the theoretical results match the empirical results, providing evidence that the UFM model can help explain the emergence of neural collapse in regression. 

The paper was reviewed by four expert reviewers, and all four of them appeared to be positive about the paper. Reviewer vXK1 writes: “this paper is interesting in that it is the first theoretical study of NC for regression problems. The results are novel and the methodology is sound.” The reviewer goes on to say, “the paper is nice because it expands the view of NC in deep learning to cover regression as well as classification. The authors shed further light on this phenomenon and their results fit nicely in the existing literature…there is some question as to whether their formulation is the right approach. For me, their definition seems compelling.” The same reviewer also writes, “this is an interesting problem and it's nice to see such a well thought out first attempt at extending an important phenomenon (i.e. neural collapse) to a more broad task setting. The theory is solid and well supported with clear and careful proofs”. Reviewer 6Hcy writes, “the presented study is novel, it discovers the neural collapse phenomenon in multivariate regression, extends the boundary of neural collapse, and provides a new understanding of neural multivariate regression. The theoretical results and experimental results are solid and well-organized.” The reviewer T8Jo writes, ”the paper addresses the significant issue of Neural Collapse in regression tasks, extending its understanding beyond classification and suggesting a universal behavior in deep learning models”. And reviewer 8FCG writes, “the paper introduces a novel extension of neural collapse to neural regression collapse” and further writes, “the paper is well-written and has a good structure,” and “the paper also includes comprehensive proofs, demonstrating NRC's performance under the UFM setting”.

Although there is a consensus that the paper is novel and interesting, the reviewers have raised many important questions which is perhaps why they originally scored the paper as a “marginal accept” rather than a “strong accept”. Since receiving the reviews, we have worked hard at running complementary experiments (as shown in Figures A, B, C, and D in the attached PDF file) and writing the rebuttals, addressing all of their comments and suggestions. We hope that the reviewers and the area chair will find our rebuttal satisfactory.

Meta-review: This paper presents a study of the neural collapse (NC) phenomenon in a new setting that has not been studied before; that of multivariate regression tasks. While parts of the analysis are similar to previous studies of NC under a MSE loss, the task studied here differs and thus the structural properties that characterize NC in this case are new. The authors provide these characterizations, provide empirical evidence, and a theoretical study under the unconstrained feature model. All reviewers believe this is a worthy contribution for the conference, and I agree.
Decision: Accept (poster)