Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the understanding of neural collapse (NC) phenomena of the learned last-layer features and classifiers observed in deep learning classifiers. Most of the existing work provides analysis under the so-called unconstrained features model where the features are viewed as free optimization variables. Motivated by the fact that dee layers cannot arbitrarily modify intermediate features that are far from being collapsed, this paper proposes a slightly different model by forcing the features to stay in the vicinity of predefined features. This is achieved by adding a regularizer to the existing unconstrained features model. The authors provide a perturbation analysis for the new problem with an additional penalty on the distance to predefined features and show that within-class variability of the output features is reduced compared to the predefined input features. 
strength_and_weaknesses: ## Strength:
- This paper provides a new model based on the existing unconstrained features model for understanding neural collapse. 
- The new model could potentially provide a further justification for neural collapse observed in practical networks.
- The authors provide analysis based on gradient flow and perturbation analysis for the global solutions of the new model. The analysis shows a reduction in the within-class variability of the output features compared to the predefined input features. 
## Weakness: 
- The major concern is the rationality behind the proposed new model. In particular, the new model, as shown in eq. (2), is the same as the existing unconstrained features model but with an additional regularizer on the distance to predefined features $H_0$. Why this model can approximate what happened in practice? What do these predefined features $H_0$ represent? If these $H_0$ represent the features before the last-layer features $H$, then in practice, $H$ is obtained as a linear transformation (and plus nonlinear functions) of $H_0$, which may be far away as simply adding regularizer on the distance between $H$ and $H_0$. In any case, more discussions should be provided to support this new model. 
- The analysis in Section 3 uses a new measure for neural collapse, $\tilde NC_1$, and shows the decrease of $\tilde NC_1$. But the experimental results are plotted in terms of $NC_1$. Does a decrease of $\tilde NC_1$ imply a decrease of $NC_1$?
- The results in Section 3 only show a decrease in neural collapse, but do not provide a specific amount of decrease. 
- The results in Section 4 assume $\beta$ is much larger than 1, which implies that the learned features will be very close to $H_0$. The main result in Theorem 4.1 also suggests this, as eq. (5) implies that the collapsing gap is approximately preserved, i.e.,  $\delta H \approx \delta H_0$. This is not close to what we observed in practice as in Figure 2, where the features become collapse very quickly across layers.   
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-organized and easy to follow. But the notations in section 4 are a little bit confusing. For example, $H_0$, $H^*$, and $\hat H^*$ all refer to the same one. The perturbation result is new and provides some new insights about neural collapse. 
summary_of_the_review: This paper studies a clear and important question about approximate neural collapse observed with practical networks. However, there is a lack of sufficient description of why the proposed approach can be used to explain the approximate neural collapse. For example, it is not clear to me what the term $||H - H_0||_F^2$ represents in a practical network.  It also appears that the proposed approach can not capture the observed phenomena in practice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies neural collapse in DNNs and proposes a modification to the unconstrained features model (UFM) that has been studied by several previous works. In particular, the authors seek to explain a question on the variability of class-wise features (NC1) under the MSE loss. Previous works using the UFM conclude that the features collapse to their class-means, leading to zero variability among the features of examples belonging to a class. Here, the authors, motivated by the fact that in practical neural networks, the last layer features are not truly unconstrained (thus they are a function of the input examples), they propose adding to the UFM a penalty on the features deviating from a fixed features matrix. For this model, the authors theoretically prove that an NC1 metric decreases monotonically along the gradient flow path. Further, with the so-called perturbation analysis, approximate values of the deviation in the optimal feature are given when the fixed feature deviates by a small amount. This result leads to some useful insights on the sensitivity of the optimal features under near-collapse to regularization hyperparameters. Some empirical evidence is provided that correlates with certain insights gained from the theoretical study.
strength_and_weaknesses: Strengths:
-----------------------------------------------

1. The motivating hypothesis is interesting- whether the widely used UFM is sufficient for addressing the NC1 behavior in DNNs
2. The authors propose a simple modification to the UFM to penalize deviation from a fixed feature matrix. A justification on its usefulness is provided by connecting the effect to that of finite depth neural networks not permitting the last layer features to wander freely. This can prove to be a useful model for relaxing the infinite approximation capability of neural networks that is proposed as a heuristic justification for studying the unconstrained features model.
3. The authors prove two main properties of the optimum of the proposed modified UFM under MSE loss: the monotonic reduction in NC1, effect of regularization on the deviation. These are consistent with observed behavior in DNN training.

Weaknesses:
-----------------------------------------------

1. It is not clear that the hypothesis of NC1 not being 0 is completely justified- can this be proved in a simple model, where when SGD is performed, the “final” solution does not exhibit exact collapse? Is the experimental observation strong enough to eliminate other effects such as not being deep in the terminal phase of training? For example, NC1 is Fig. 3 top row has a value less than 1e-2. For practical purposes it might be sufficient to say that the features have collapsed. Are there significant and specific consequences of the metric not being much smaller?
2. I think more extensive and careful experimentation can make a stronger case for the proposed model to be useful as a strictly better model than the simple UFM. 
3. On the layer-wise experiment: this is an interesting study. However, from the details in the appendix, it seems like each layer is trained for only 3 epochs- if run longer, the layer would have likely achieved smaller NC1, taking the setting closer to near-collapse. Any specific reason for stopping at 3 epochs? Also, what is the reason for using Adam rather than SGD for this experiment? Perhaps it is better to show this result with MSE loss rather than CE to be consistent with the theoretical study.
4. On the effect of regularization, how consistent are the empirical observations on different architectures and datasets? Is the effect of needing to train for more epochs when using a smaller regularization taken into account in the interpretation? Specifically, when using 0 WD, typically the training needs to be much longer. Results shown in Fig 3 will be more convincing if the training length aspect is eliminated.

Miscellaneous queries, comments:

1. Theorem 4.1: the result is up to O(norm squared of deviation in optimal solutions with and without perturbation). However, this applies to the statement approximating the actual deviation in the optimal solutions. Is this reasonable?
2. Just before the statement of Cor 2.2, H0 is stated as “already collapsed”. It is better to clarify that this H0 is in fact the ETF/OF solution itself in addition to being collapsed.


clarity,_quality,_novelty_and_reproducibility: The paper is in general clearly written, theoretical results are coherently stated and experimental section is explained sufficiently. 
summary_of_the_review: Overall I think the topic is relevant, the author's idea of modeling non-perfect NC is interesting and the paper is contributing something new to the recent literature on neural collapse. I have some question about the actual implications of the result, the practical relevance of the model and the experiments as discussed above. My first impression is positive and looking forward to the discussion.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors analyze properties of a sparsity regularized matrix factorization where one of the factor matrices $H$ is further regularized to be close to a given matrix $H_0$:
\begin{align} \min_{W,H}f(W,H) = \lVert Y-WH \rVert^2 +\lambda_W\lVert W\rVert^2 +\lambda_H \lVert H\rVert^2 + \beta \lVert H-H_0\rVert\end{align}
The properties of the solvers of the above objective shall give insight into properties of neural collapse, describing the phenomenon that neural networks tend to resemble nearest centroid-classifiers after training, where the class centroid is computed in the penultimate layer. The matrix $Y\in\\{ 0,1 \\}^{K\times Kn}$ indicates here the one-hot encoded targets for $K$ classes and $n$ samples per class. $H\in\mathbb{R}^{d\times Kn}$ is in this case related to the $d$-dimensional output of the penultimate layer and the matrix $H_0$ indicates architectural restrictions on what kind of representations can be learned. The matrix $W$ is in return related to the weight matrix connecting the penultimate layer with the last layer.

The main result (Thm 1) indicates (approximately) that when $\lambda_W>0$ the within-class scatter decreases and the between-class scatter increases with $\beta\rightarrow 0$.
Experiments evaluate how the weight decay on specific layers influences the neural collapse behavior on Cifar 10.
strength_and_weaknesses: The main weakness of this paper and the referenced paper on which the authors build (Tirer&Bruna, 2022) is that most of the results and contributions already have been studied in the field of matrix factorization. This whole field is entirely neglected and properties of the objective solvers are derived over approximations of approximations, that just as well follow directly from the theory of matrix factorization (using maybe even a more realistic explanation, see below). Considering this, the contributions of this paper seem thin. 

I think that the introduced neural collapse score is intuitive and that it could be useful. I also think that there is some insight gained from the experiments confirming that sparsity in the learned transformed feature representation induces neural collapse. Although I have some doubts whether the experiments actually correspond to the theory.

# Assumptions and Approximations
There are quite a few assumptions that make the derived theory less impactful. First of all, analyzing the matrix factorization model as a substitute for neural network optimization is very doubtful. After all, the targets (class labels) are not directly approximated by a linear function, but by the softmax of a linear function. Then, cross-entropy is typically used instead of MSE. In addition, the authors make the assumption that each class has exactly $n$ samples. To make the $l_2$-norm regularized matrix factorization model of Tirer & Bruna more "realistic", the authors introduce the regularization term $\lVert H-H_0\rVert^2$. This is an approximation for the fact that $H$ can not be chosen freely, but is instead a function of the input, given by the neural network's representations learned as the output of the penultimate layer. So, this penalty term is quite far from what's happening in reality and I also don't see how the provided analysis of this objective can actually provide insight.

To prove Theorem 3.1, the gradient flow is analyzed of, again, an approxiation $\frac{dH_t}{dt} = -Kn\nabla\mathcal{L}(H_t)$. In fact, we only have   $\frac{H_t-H_0}{t} = -Kn\nabla\mathcal{L}(H_t)$. 

# The Results from a Matrix Factorization Perspective
I think that the most interesting result of this paper is the analysis of the within and between class scatter in dependence of the sparsity penalization of $H$. The theorems in this paper don't show exact properties but rely on multiple approximations. The approximate result can be also directly derived from the theory of matrix factorization. Considering that we actually analyze now $\lVert Y-WH\rVert$, then  after the penultimate layer, most architectures use a ReLU activation. Hence, $H$ is nonnegative. If we restrict $H$ to be a one-hot encoded matrix, that is $H\in\\{0,1\\}^{d\times Kn}$ and $\lVert H_{\cdot i}\rVert=1$, then the objective $\min_{W,H}\lVert Y-WH\rVert$ is _equivalent_ to $k$-means on the matrix $Y$, where $W$ indicates the centroids. Hence, trivially, this objective would minimize the within-class scatter and maximize the between-class scatter. A similar result can however be achieved if we introduce a sparsity constraint on $H$. the nonnegative, sparse matrix $H$ indicates in this case a fuzzy clustering, where the nonzero elements in $H_{k j}$ indicate a degree with which the  sample $x_j$ belongs to class $k$. The higher the sparsity regularization weight, the fewer classes are selected for a sample to belong to, and the closer we get to the $k$-means clustering.

# Experiments
The experiment results are depicted in Figure 3. A network is trained layer-wise on Cifar 10. The sparsity constraint on $H$ is simulated by a weight decay on all weights between layers, except for the last one. This is however not the same as sparsity in the learned representations given by $H$, which is the output of the penultimate layer. The experiments indicate that a higher weight decay on "H" increases the neural collapse metric on Cifar-10, which is an interesting result, but it only loosely connects to the theory.

## References
Udell, Madeleine et al. “Generalized Low Rank Models.” Found. Trends Mach. Learn. 9 (2016): 1-118. (see in particular "2.2 quadratically regularized PCA" and "3.2 Examples" )
clarity,_quality,_novelty_and_reproducibility: Clarity and quality of the paper decrease with the repetitively made approximations, and the inflated math. The theorems are quite difficult to read (for example Thm. 4.1, what is $F_{k,k}$ in Thm. 4.2) and the meaning of these theorems is also not obvious to me. As already pointed out, the novelty is also low because most of the derived intuitions/relations about solvers of the matrix factorization objective are already known. Reproducibility is ok.
summary_of_the_review: The authors provide support to the hypothesis that sparsity in the learned representations of neural networks, given by the output of the penultimate layer, induces/strengthen the behavior of neural collapse. The provided theory is only loosely connected to the reality of neural network training and is also not novel in the light of matrix factorization theory. Experiments are also just loosely simulating the sparsity in learned representations, which makes the results overall not very convincing.

# After Rebuttal Thoughts
I did not find the rebuttal very convincing. The theoretical analysis is still weak. Even when the network is trained with MSE loss, the objective function would not be the considered objective, but softmax would be applied on the matrix product. Generally, the nonlinear parts of the objective and the learned function, represented by $H$ are neglected. The analysis of the considered objective does still not deliver more knowledge than what is known from matrix factorization theory. The use of gradient flow is not a contribution in itself, it doesn't really fit here (see my criticism on approximations). The addition of the penalty term is emphasized as a contribution, but the anlysis focuses on the case where the weight of the penalty term goes to zero. Hence, I keep my score and I strongly encourage to have a look into the connection of NMF with clustering methods to improve further work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new model for neural collapse (NC) which penalizes the degree to which the learned features diverge from some initial value. This model, it is argued, more closely aligns with the evolution of features from one layer to the next in a neural network. This modification to the unconstrained features model, coupled with the assumption that the weight vector is optimal with respect to the features, yields a gradient flow which also exhibits neural collapse. Finally, a perturbation analysis is applied to the model when the previous-layer features are not collapsed. The principal take-away from this analysis is that it is intra-class variation, rather than inter-class variation, of the previous-layer features which drives the deviation from collapse of the output features. 
strength_and_weaknesses: Strengths
	
- The model studied in this paper is more realistic than previous works that have completely unconstrained features.
- The paper is clearly written.
- The modified trace term measuring neural collapse defined in Section 3 is well justified, and I appreciated the comparison against prior works to highlight the necessity of $\widetilde{NC}_1$ in Theorem 3.1 
- The perturbation analysis in Section 4 provides a nice intuition for how non-collapsed features in early layers of the network can slow down convergence to NC in later layers.
- The experiments are easy to interpret and provide a sensible evaluation of the theoretical predictions.
- The visualization of the operator $F$ studied in Theorems 4.1 and 4.2 is helpful to ground the discussion around these results.

Weaknesses/Questions:
- I’m unsure how realistic the gradient flow in (4) is as a model for feature learning in neural networks.
In moving from the penalty on the distance $\|H - H_0\|$ to the gradient flow, the analysis seems to implicitly change the interpretation of $H_0$ as the output of a previous layer to the value of the current features.
- Further, this requires that the weights always be optimal with respect to the current features. This assumption seems unlikely to be true in neural networks, and it’s not clear to me how the approximation error and resulting dynamics attained by suboptimal weights would relate to the “central path” described in this paper. 
- The conclusions from Theorem 4.2 seem to implicitly assume a relatively small number of classes in order for the contribution of the inter-class blocks of $\mathbf{F}$ on the off-diagonal to be negligible. I would be interested in seeing (at least in the appendix) a discussion of how the relative contribution of inter- and intra-class interactions vary as a function of the number of classes. In particular, how do the spectra of the blocks translate to the spectrum of the overall linear operator $F$? While I appreciate the empirical example in Figure 1, I would have liked to see a more detailed theoretical analysis of the relationship between the sub-blocks and the overall matrix.

Minor errata:
“Values” is misspelled twice as “vlaues”.

clarity,_quality,_novelty_and_reproducibility: The paper is reasonably clearly written, though as I have commented previously does have a number of typos. I have not had an opportunity to review the proofs in depth, but I didn't notice anything obviously incorrect at first glance. My main concern with the paper is the relevance of the theoretical model to the real-world phenomenon it is seeking to describe. As is often the case in theoretical deep learning papers, it is difficult to determine whether the qualitative agreement of experiments with theory is due to the correctness of the theory as a model for the empirical phenomenon or due to the theory being constructed so as to describe observations of situations that are similar to the experiments being conducted. 
summary_of_the_review: This paper presents an interesting perspective on neural collapse by studying the behaviour of networks that are near, but do not precisely exhibit, feature collapse. The theoretical model used in this analysis comes closer to capturing the limited ability of neural networks to arbitrarily change the feature representation, but still makes strong assumptions on e.g. the optimality of the weights $W$ over the course of training and the evolution in the value of the initial earlier layer features. The perturbation analysis results are intriguing and provide some intuition for how features may exhibit slow convergence to total collapse, but the closed-form solution obtained by these results is quite unwieldly and difficult to interpret except through analysis of sub-matrices. Overall, I think this paper makes a step towards studying neural collapse in a more realistic regime, but there remains a wide gap between the theoretical model studied here and practical settings which limits the significance of the contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work investigates the neural collapse phenomenon under a modified version of a previously studied unconstrained feature model in which the features remain close to predefined values. 

* Strengths are the progression over simplistic models and contributing to the theoretical description of neural collapse. 
* Weaknesses are insufficient justification for the proposed model and assumptions, limited explanatory power of the results, and concerns on limited innovation. 

After the discussion period some concerns could be clarified, but other concerns remained, particularly about certain parts of the analysis, the justification of the proposed model, and the significance of the contribution. I find that the article contains promising ideas but also share some of the reviewers' concerns and find it would have been stronger by more clearly motivating and further developing the results in relation to practical settings. 


justification_for_why_not_higher_score: The authors provided responses to the initial reviews, which however did not sufficiently resolve some of the reviewers' concerns. 
justification_for_why_not_lower_score: NA
Decision: Reject