Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations
Authors: Yongyi Yang
Jacob Steinhardt
Wei Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Collapse
Representation Learning
Neural Networks

Abstract: Recent work has observed an intriguing "Neural Collapse" phenomenon in well-trained neural networks, where the last-layer representations of training samples with the same label collapse into each other. This suggests that the last-layer representations are completely determined by the labels, and do not depend on the intrinsic structure of input distribution. We provide evidence that this is not a complete description, and that the apparent collapse hides important fine-grained structure in the representations. Specifically, even when representations apparently collapse, the small amount of remaining variation can still faithfully and accurately captures the intrinsic structure of input distribution. As an example, if we train on CIFAR-10 using only 5 coarse-grained labels (by combining two classes into one super-class) until convergence, we can reconstruct the original 10-class labels from the learned representations via unsupervised clustering. The reconstructed labels achieve $93\%$ accuracy on the CIFAR-10 test set, nearly matching the normal CIFAR-10 accuracy for the same architecture. Our findings show concretely how the structure of input data can play a significant role in determining the fine-grained structure of neural representations, going beyond what Neural Collapse predicts.


Under review as a conference paper at ICLR 2023
ARE NEURONS ACTUALLY COLLAPSED? ON THE FINE- GRAINED STRUCTURE IN NEURAL REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has observed an intriguing "Neural Collapse" phenomenon well- trained neural networks, where the last-layer representations of training samples with the same label collapse into each other. This suggests that the last-layer representations are completely determined by the labels, and do not depend on the intrinsic structure of input distribution. We provide evidence that this is not a complete description, and that the apparent collapse hides important fine-gras tne-grained structure in the representations. Specifically, even when representations apparently collapse, the small amount of remaining variation can still faithfully and accurately captures the intrinsic structure of input distribution. As an example, if we train on CIFAR- 10 using only 5 graiiee labels (by combining two classes into one super-class) until convergence, we reconstruct the original 10-class labels from the learned representations via unsupervised clustering. The reconstructed labels achieve 93% accuracy the CIFAR-10 test set, nearly matching the normal CIFAR- 10 accuracy for the same architecture. Our findings show concretely how the structure of input data can play significant role in determining the fine-grained structure of neural representations, going beyond what Neural Collapse predicts.
INTRODUCTION
Much of the success of deep neural networks has, arguably, been attributed their ability to learn useful representations, or features, of the data (Rumelhart et al.) 1985). Although neural networks are often trained to optimize single objective function with no explicit requirements on the inner representations, there is ample evidence suggesting that these learned representations contain rich information about the input data (Levy & Goldberg 2014 Olah et al. 2017). As a result, formally characterizing and understanding the structural properties of neural representations is of great theoretical and practical interest, and can provide insights on how deep learning works and how to make better use of these representations.
One intriguing phenomenon recently discovered by Papyan et al. (2000) is Neural Collapse, which identifies structural properties of last-layer representations during the terminal phase of training (i.e. after zero training error is reached). The simplest of these properties is that the last-layer representations for training samples with the same label collapse into a single point, which referred to as "variability collapse (NCI)." This is surprising, since the collapsed structure is not necessary to achieve small training or test error, yet it arises consistently in standard architectures trained on standard classification datasets.
series of recent papers were able to theoretically explain Neural Collapse under simplified model called the unconstrained feature model or layer-peeled model (see Section 2 for a list of references). In this model, the last-layer representation of each training sample treated as free optimization variable and therefore the training loss essentially has the form of a matrix factorization. Under variety of different setups, was proved that the solution to this simplified problem should satisfy Neural Collapse. Although Neural Collapse is relatively well understood in this simplified model, this model completely ignores the role of the input data because the loss function is independent of the input data. Conceptually, this suggests that Neural Collapse is only determined by the labels and may happen regardless of the input data distribution. Zhu et al. (202I) provided further empirical support of this claim via random labeling experiment.

---

Under review as a conference paper at ICLR 2023
super-ciass supenciess 1
supen-cess
50
supei-ciass 4
25
original class original
-50
-3
-75
-50
50
Figure 1: Fiie--griinee clustering structure of the last-layer representations of ResNet- 18 trained on Coarse CIFAR-10 (5 super-classes. Left figure: PCA visualization for all training samples. Right figure: t-SNE visualization for all training samples in super-class 4 (which consists of original classes 4 and 9).
the other hand, conceivable that the intrinsic structure of the input distribution should play role in determining the structure of neural net representations. For example, if class contains a heterogeneous set of input data (such as different subclasses). it is possible that their heterogeneity is also respected in their feature representations (Sohoni et al.) (020).. However, this appears to contradict Neural Collapse, because Neural Collapse would predict that all the representations collapse into the same point long they have the same class label. This dilemma motivates to study the following main question in this paper:
How reconcile the roles of the intrinsic structure of input distribution the explicit structure of the labels in determining the last-layer representations in neural networks?
Our methodology and findings. To study the above question, we design experiments manually create a mismatch between the intrinsic structure of the input distribution and the explicit labels provided for training standard classification datasets and measure how the last-layer representations behave in response to our interventions. This allows us to isolate the effect of the input distribution from the effect of labels. task), alter its labels in two different ways, resulting in coarsely-labeles and finely-labeled version:
illustrative example, the CIFAR-10 dataset (a 10-class classification
combine every two class labels into one and obtain 5-class task (see Figure 2 for an illustration);
Coarse CIFAR-10:
Fine CIFAR-I0: split every class label randomly into labels and obtain a 20-class task.
We train standard network architectures (c.g. ResNet, DenseNet) using SGD on these altered datasets.
Our main findings are summarized below.
First, both the intrinsic structure of the input distribution and the explicit labels provided in training clearly affect the structure of the last-layer representations. The effect of input distribution emerges earlier in training. while the effect of labels appears at a later stage. For example, for both Coarse CIFAR-10 10 and Fine CIFAR-10 10, at some point the representations naturally form 10 clusters according to the original CIFAR-10 10 labels (which comes from the intrinsic input structure), even though 5 or 20 different labels are provided for training. Later in training (after 100% training accuracy is reached), the representations collapse into or 20 clusters driven by the explicit labels provided, predicted by Neural Collapse.
Second, even after Neural Collapse has occurred according to the explicit label information, the seem- ingly collapsed representations corresponding to each label can still exhibit fine- ime-grained structures determined by the input distribution. As illustration. Figure visualizes the representations from the last epoch of training a ResNet-18 on Coarse CIFAR- 10. While globally there are 5 separated clusters predicted by Neural Collapse, if we in each cluster, it clearly consists of two
2

---

Under review as a conference paper at ICLR 2023
subclusters which correspond to the original CIFAR-10 classes. We also find that this phenomenon persists even after a very long training period (e.g. 1,000 epochs), indicating that the effect of input distribution is not destroyed by that of the labels, at least not within normal training budget.
To further validate our finding that significant input information is present in the last-layer representa- tions despite Neural Collapse, we perform simple ciisterrand-Liedd Probe (CLP) procedure the representations from ResNet-18 18 trained on Coarse CIFAR-I0 10, in which we use an unsupervised clustering method to reconstruct the original labels, and then train a linear classifier on top of these representations using the reconstructed labels. We find that CLP can achieve > 93% accuracy on the original CIFAR-J 10 test set, matching the standard accuracy of ResNet- 18, even though only coarse labels are provided the entire time.
Takeaway. While Neural Collapse is intriguing phenomenon that consistently happens in well- trained neural networks, we provide concrete evidence showing that it is not the most comprehensive description of the behavior of last-layer representations in practice, it fails to capture the possible fine-grained properties determined by the intrinsic structure of the input distribution.
RELATED WORK
The Neural Collapse phenomenon was originally discovered by Papyan et al. (2020). and has led to a series of further investigations.
A number of papers Fang et al. (2021) Lu & Steinerberger (2020); Wojtowytsch et al.
(020);
Mixon et al. (2022): : Zhu et al. (2021 I ): Ji al. (2021); Han et al. (2021 i Zhou et al. (2022) Tirer
Bruna (2022); Yaras et al. (2022) studied simplified "unconstrained feature model", also known as
"layer-peeled model", and showed that Neural Collapse provably happens under a variety of settings.
This model treats the last-layer representations of all training samples as free optimization variables. By doing this, the loss function no longer depends on the input data, and therefore this line of work is unable to capture any effect of the input distribution
the structure of the representations. Ergen & Pilanci 2021 Tirer& & Bruna 2022); Weinan & Wojtowytsch (2022) considered more complicated models but still did not incorporate the role of the input distribution.
Hui et al. (2022) studied the connection of Neural Collapse to generalization and concluded that Neural Collapse occurs only on the training set, not on the test set. Galanti al. (2021) found that
Neural Collapse does generalize to test samples as well as new classes, and used this observation to
study transfer learning and few-shot learning.
Sohoni et al. (020) observed that the last-layer representations different subclasses within the same class are often separated into different clusters, and used this observation to design an algorithm for improving group robustness. However, it is unclear whether their observation happens in the Neural Collapse regime, while we find that fine- grained structure representations can co-exist with Neural Collapse. Furthermore, [Sohoni et al. (2020) looked at settings in which different subclasses have different accuracies and attributed the representation separability phenomenon to this performance difference. On the other hand, we find that representation separability happens even when there is no performance gap between different subclasses.
PRELIMINARIES AND SETUP
This section introduces the notation used throughout this paper, as well as a more detailed explanation of Neural Collapse and clarification of the experiment setup.
For classification task dataset, denote it by D {( 2k In k- where (æ Mk. ) E IRd x [C] is a pair of input features and label, n is the number of samples, d' is the input dimension, and is the number of classes.
For a given neural network, we denote the last-layer representation by H E Roxd ie. the hidden representation before the final linear transformation, where d is the last-layer dimensionality. For an original class c E [C], we denote the number of samples in class e by ne, and the last-layer representation of k-th sample in class by h (c)
3

---

Under review as a conference paper at ICLR 2023
3.1 PRELIMINARIES OF NEURAL COLLAPSE
Neural Collapse characterizes phenomena, named NC1-NC4, Here we introduce the first two which concern the structure of the last-layer representations.
NC1, or variability collapse, predicts that the variance of last-layer representations of samples within the same class vanishes as training proceeds. Formally, it can be measured by NC1 = Tr <w
(Papyn et al. 2020 Zhu et al. 2021), which should tend to O. The Ew and ΣB are defined as
Ew
h
me) (n1 1l.)
The
cE|C]
and
Eg T
Σ (Me cE(C]
mc) (Me uG)
(2)
(c) where Me En= k== g are the class means and PG 1 Ex= b4 is the global mean.
NC2 predicts that the class means converge to special structure, i.e. their normalized covariance converges to Simplex Equiangular Tight Frame (ETF). This can characterize by
MM
VC-1 (1- +0
NC2
during training, where M is the stack of centralized class-means, whose c-th row is pe MG. and 1c Rº is an all-one vector in length C, the identity matrix.
3.2 EXPERIMENT SETUP
In our experiment, we explore the role of input distribution and labels through assigning coarser or finer labels each sample, and then explore the structure of last-layer representation of model trained on the dataset with coarse or fine labels and see to what extent the information of original labels are preserved.
The coarse labels are created in the following way. First choose a number Ö which is divisible by C, and create coarse labels by
Uh yk mod Č;
(4)
which merges the classes whose index have the same modulus w.r.t. Ö and thus create Ö super-classes. Since the original index of classes generally has no special meanings, this process should act similar randomly merging classes We say the samples with the same coarse label belongs to the same super-class, and call the dataset D {(1x, ūx))== the coarse dataset, which the dataset we used to train the model. Figure 2 provides an illustration of the coarse labels on CIFAR-10 with Ò which we call Coarse CIFAR-10.
Comme airplene dog
automobie & tog
aed S home
cat da ship
deer truck
airplane & dog
automobile hrg
Bnd hores
cat ship
desr & truck
Figure 2: Dataset illustration of Coarse CIFAR-10.
To create fine labels, we randomly split each class into two sub-classes. Specifically, the fine labels created by
dh = + 3C,
(S)
where OR is the fine label of sample and C' is the number of original classes and / is Bernoulli Variable. This process result in dataset D = { (x) ) with 2C classes. Same as before, we call D the fine dataset.
Wee adopt this determined process for simplicity and reproducibility. However, do provide additional results with random merging in appendix.
4

---

Under review as a conference paper at ICLR 2023
10
200
Formbct ooopects
Figure 4: An illustration of predicted class dis- tance matrix heatmaps of Coarse CIFAR- 10, each row and column represents original class. (a): input distribution dominates the last-layer representations. (b): If Neural Collapse domi- nates the last-layer representations.
Figure The value NC₁ and NC2 w.r.t. num- ber of training epochs.
EXPLORING THE FINE-GRAINED REPRESENTATION STRUCTURE WITH COARSE CIFAR-10
In this section, we experiment with coarsely labeled datasets, using Coarse CIFAR-10 as an illustrative example. Specifically, the model is trained on the training set of Coarse CIFAR- 10 for certain number of steps that is sufficient for the model to converge. We then take the last-layer representations of the model during training and explore their structure.
In order to make an exhaustive observation, the experiment is repeated using different learning rates and weight-decay rates. Specifically, choose the learning rate of training in {10-1, 10-2, 10-33 3 and weight-decay rate in (5 x 10 10-3, 5 x 10-1, x 10-5} and test all 9 possible combinations of them. The experiment is also conducted on multiple datasets and network architectures. Due space limit, in this section we focus on experiment of ResNet- 18 on Coarse CIFAR-10, where the original C' = 10 and the number of coarse labels is chosen as = 5. For training hyper-parameterd combinations, referred as config #1 and
we
only report results for two representative syper-parameet config #2. The learning rate of config #1 is set to 0.01 and weight-decas rate is 5 x 10-4, and for config #2 the learning rate is 0.01 and weight decay rate x 10-3 Both settings reach training error quickly during training. We defer complete experiment results to appendix.
As a preliminary result, we first show that under our experiment settings, Neural Collapse does happen, i.c. the representation does converge 5 clusters and the class- means form Simplex ETF structure. Specificallyy, we measure NC; and NC₂ defined in Section 33.[for for both config #1 and config #2, with replaced by Ö since we are calculating it on coarse dataset. The results are shown in Figure which matches previous results in Papyan al. (2020); Zhu al. (202I), indicating that Neural Collapse does happen under both settings.
4.1 CLASS DISTANCE
We first look the average square Euclidian distance of last-layer representations between each two original classes. Formally, we calculate a class distance matrix D E RCCC' whose entries are
Drj
h
(6)
n.m
for all i,j E [C], where (e) represents the last-layer representation the k-th sample of super-class
Since the model is trained on coarse dataset, Neural Collapse asserts that for every original class pair i.j in the same super-class (including the of i j), the class distance D,,, should be very small. In Coarse CIFAR-10, this will result three darks lines (let darker color represents lower value) in the heatmap of D since each super class contains original classes, illustrated in Figure (b).
5

---

Under review as a conference paper at ICLR 2023
For example, in Coarse CIFAR-10 the original class "airplane" and "dog" both belong the super class "airplane & dog", therefore per Neural Collapse's prediction, their last-layer representations would collapse each other, making the average square distance extremely small compared to other entries. In contrast, if the last-layer representations perfectly reflects the distribution of input, i.e. original classes, the class distance matrix should be diagonal matrix shown in Figure (a), because the last-layer representation of samples in each original class only collapse to the class-mean of this original class.
Figures 5 and [6] display the heatmap of class distance matrix D, arranged by number of training epochs. From the results there are two surprising observations can be made: Firstly, there are indeed three dark lines at the final stage, however, the three dark lines does not show up simultancously. and the central line represents the samples in the same original classes shows up earlier. The second observation is, even in the final stage where the training error is zero, the three lines can be not of the same degree of darkness, with the central line shallower, and this is especially apparent Figure 6
Those observations suggests that, the actual behaviour of the last-layer representations is between cases predicted in Figure 4 (a) and (b): input distribution and training label both have an impact the distribution of the last-layer representations, and although the time of them to show up different, both them can be present even after reaching zero training error for long time. These observations suggest both the existence of Neural Collapse and the inadequacy of Neural Collapse to completely describe the behaviour of last-layer representations.
Figure 5: The heatmap of class distance matrix of config #1. From left to right: number of epoch 20, 120, 240 and 350.
Figure 6: The heatmap of class distance matrix of config #2. From left to right: number of epoch 20, 120, 240 and 350.
4.2 VISUALIZATION
In this subsection, we take closer look at the last-layer representations of the model at the end of training by reducing the dimensionality of the last-layer representations through t-SNE (Van der Maaten & Hinton 2008) and visualize them. Specifically, we visualize each super-class separately, but color the samples whose original labels are different with different colors.
The visualization results of config #1 and config #2 are displayed in Figures and g It can be observed that, there is distinguishable difference in the distribution original labels in both config #1 and config #2. This suggests that, the imput distribution information, i.c. the original label information, well preserved in the last-layer representations under both settings, albeit for config #2 it seems not be distinguished through distance matrix in Section 4.1
Training extremely long. In order to explore if the fine fine-graing ene-grained structures are still preserved even after extremely long time of training, we further train the model with config #2 to 1,000 epochs. The heatmap of the distance matrix is presented in Figure 9 We also present the t-SNE visualization result, but only put the result of the first super-class in Figure TO due space limitation.

---

Under review as a conference paper at ICLR 2023
300
onginal CRU
Enginal cas
1a
orgiaal
-200
-00
-00
33
30
dass
20
supercuu
55
100
-30
das
30
120
Figure 7: The t-SNE visualization of config #1. Each grid is super-class.
150
otgraldasa a
orky futl cliss
organel class
30
Org CEI
-30
erginal cam
-200
-10
-15
110
-15
dlos
-100
-10
supercins
V
106
Figure 8: The t-SNE visualization of config #2. Each grid is super-class.
4.3 LEARNING CIFAR-10 FROM 5 COARSE LABELS
As the results in Section 4.2 suggest, even after the training accuracy has reached 100% for long time, the samples within each super- class still exhibit a clear structure per their original class, and those structures act clusters after reducing dimensionality. Inspired by this observation, we perform a -nntter-ad-LLinaat evaluate what extent the original class information is preserved the last-layer representations. In CLP, we use the representations learned on Coarse CIFAR-10 10 reconstruct 10 labels and run a linear probe with the reconstructed labels. Specifically, we first use t-SNE to reduce dimensionality to and then use KMeans to find clusters of the representations within each super-class. We use the clusters as reconstructed labels to do lincar probe. In linear probe, train linear classifier top of the previously learned representation H on the training set with reconstructed labels and evaluate the learned linear classifier on original test set. Notice that because we don't know the mapping of reconstructed classes to true original classes, we permute each possible mapping and report the highest performance. We also train a linear probe with original training labels comparison. The results of config #1 and config #2 are shown in Figures and [2 respectively
(CLP) test to
23
dimensionality-reduee
Figure 9: The heatmap of
distance matrix ResNet- 18 trained on Coarse CIFAR- 10 for 1000 epochs.
In most cases, the performance of CLP on original test set is com- parable with linear probe trained on true original labels or even with models originally trained on CIFAR-10. Notice that the representation Hl is obtained through the model trained with coarse labels, and the label reconstruction only uses information of and the number of original classes. This means on original test set even we only have access to almost only the information of coarse labels. This result further confirms that the input distribution can play an important role in the last-layer representations.
original cas
-50
-20
Figure 10: The t-SNE vi- sualization of the last-layer representations of the first super-class of ResNet-18 trained on Coarse CIFAR-10 for 1 1,000 epochs.
achieve very high performance
7

---

Under review as a conference paper at ICLR 2023
bicycle
16.1
90.0
insortycle truck
00.0
70.0
00.0
50.0
cluster and Inear probe linear
06.0
10.0
original
-2
epprtts
Figure 13: The t-SNE visualization of the last-layer representations of super- class "vehicles I" of ResNet-18 trained on CIFAR-100 with original super- classes.
Figure 11: The CLP result of config #1. "original test" the highest test set accuracy achieved by ResNet18 trained original CIFAR-10 with the same training hyper-parameters of config #1.
100.00
crab
60.00
lobster
snall tgr
50.00
40.02
cluster Inear probe
30.00
linear probe with qlial label original best
-30
0
30
100
150
200
250
300
250
-20
nimber of aporis
Figure 12: The CLP result of config #2. "original test" is the highest test set accuracy achieved by ResNet18 trained on original CIFAR-10 with the same training hyper-parameters of config #2.
Figure 14: The t-SNE visualization of the last-layer representations of super-class "non-inseet invertebrates" of ResNet- 18 trained on CIFAR-100 with original super-classes.
How DOES SEMANTIC SIMILARITY AFFECT THE FINE- GRE-GRAINNED STRUCTURE?
In our experiments with Coarse CIFAR-10, each coarse label is obtained by combining two classes regardless of the semantics. The fact that the neural network can separate the two classes in its representation space implies that the network recognizes these two classes as semantically different (even though they are given the same coarse label). In this section, explore the following question: the sub-classes a super-class have semantic similarity, will the representations still exhibit a fine-grained structure distinguish them? Intuitively, if the coarse label provided is "natural" and consists of semantically similar sub-classes, it is possible that the neural network will not distinguish between them and just produce truly collapsed representations.
We take an initial step towards this question by looking at ResNet- 18 trained on CIFAR- 100 using the official 20 super-classes (each super-class contains 5 sub-classes) labels. Unlike randomly merging classes as we did in Section 4 the official super-classes of CIFAR- are natural, merging classes with similar semantics (for example, "beaver" and "dolphin" both belong to "aquatic mammals"). This offers perfect testbed for our question.
We find that ResNet- 18 indeed is not able distinguish all sub-classes in its representation space, but can still produce separable representations if some sub-classes within a super-class are sufficiently different. Interestingly, the notion of semantic similarity of ResNet- 18 turns out to agree well with that of humans. Figures and classes. From the visualizations, although there are not as clear clusters as for Coarse CIFAR-10, the representations do exhibit visible separations between certain sub-classes. In Figure 13 "bicycles" and "motorcycles" are entangled together, while they are separated from "bus". "pickup truck", and "train", which is suman-interpretabbl In Figure [4] "crab" and "lobster" mixed together, which both aquatic and belong to malacostraca, while the other three are not and have more differentiative representations.
and 14 show the t-SNE visualizations of representations from two super-
8

---

Under review as a conference paper at ICLR 2023
These results confirm the intuition that the fine-graine seee-grriedd structure in last-layer representations is affected by, or even based on, the semantic similarity between the inputs.
THE FINE-GRAINED REPRESENTATION STRUCTURE ON FINE CIFAR-10
In this section, we consider finely-labeled dataset. We construct a fine version of CIFAR-10 with the process described in Section 3.2 and call it Fine CIFAR-10. Figure and and Figure T6 presents the results of class distance matrices for config and config #2, which are arranged by the number of training epochs.
can be observed that at the early stage training. there are three dark lines, which indicates the last-layer representations training. this 10-class relationship is reserved and is especially significant config #2, although with lighter color. This suggests that there exists fine-grained structure even at the terminal phase. Those observations are consistent with the observations made in Section and supports our conclusion.
converging towards 10 clusters instead of 20 first. At the end of
Figure 15: The heatmap of class distance matrix of config #1 on Fine CIFAR-10. From left to right: number of epoch 20, 120, 240 and 350.
Figure 16: The heatmap of class distance matrix of config #2 Fine CIFAR-10. From left right: number of epoch 20, 120, 240 and 350.
DISCUSSION
In this paper, we initiated the study of the role of the intrinsic structure of the input data distribution on the last-layer representations of neural networks, and in particular, how to reconcile it with the Neural Collapse phenomenon, which is only driven by the explicit labels provided in the training procedure. Through a series of experiments, we provide concrete evidence that the representations can exhibit clear fine-grained structure despite their apparent collapse. While Neural Collapse an intriguing phenomenon and deserves further studies to understand its cause and consequences, our work calls for more scientific investigations the structure neural representations that go beyond Neural Collapse.
We note that the fine-grained representation structure we observed depends on the inductive biases of the network architecture and the training algorithm, and may not appear universally. In our experiments on Coarse CIFAR-10, we observe the fine-grained structure for ResNet and DenseNet, but not for VGG (see appendix for extended results). We also note that certain choices of learning rate and weight-decay rate lead to stronger fine- fine-grained grained structure than others. We leave a thorough investigation of such subtlety for future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tolga Ergen and Men Pilanci. Revealing the structure of deep neural networks via convex duality. In International Conference on Machine Learning, pp- 3004-3014, PMLR, 2021.
Cong Fang. Hangfeng Hc, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, 118(43):21033011 18, 2021.
Tomer Galanti, András György, and Mareus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15122, 2021.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. arXiv preprint urXiv:2/06.02073 2021.
Like Hui. Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08338, 2022.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled
perspective on neural collapse. arXiv preprint arXiv:21 10.02796, 2021.
Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Fighteenth Conference on Computational Natural Language Learning, pp. 171-180, Ann Arbor, Michigan, June 2014. Association for Computational Linguistics. doi: 10.31155vw/W111166111 URL http://aaaaanoooooooooooooo
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint 2tt::0//2.08665. 2020.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-13, 2022.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: http:/iiilllllu2//////aaaee/i/ie
10.2391 1/13tt//iisill..000000.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences. 117(40): 24652-24663, 2020.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. subelass left behind: Fine- fine-grained robustness in coarse- gratined classification problems. Advances in Neural Information Processing Systems, 13:19999119352, 2020.
Tom Tirer and Joan Bruna, Extended unconstrained features model for exploring deep neural collapse. arXiv preprini arXiv:2002.08087 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Weinan and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning. pp. 270-290. PMLR, 2022.
Stephan Wojtowytsch al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:20/2.05422 2020.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. arXiv preprint arXiv:2209.097 09211, 2022.
10

---

Under review as a conference paper at ICLR 2023
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. arXiv preprint arXiv:2203.012238 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Nearal Information Processing Systems, 34:29820-22935 2021.

---

Under review as a conference paper at ICLR 2023
A (NEW) EXPERIMENT ON COARSE CIFAR-10 WITHOUT WEIGHT DECAY
In order to study whether weight decay is crucial for the fine- ame-ggained structure, we perform an additional experiment Coarse CIFAR-10 when weight decay is O. We present the final class distance matrices (see Section 4.1) for three different learning rates 0.1, 0.01, and 0.001 in Figure 17 Except when the learning rate is 0.001, the training error reaches O. We see that the fine-grained representation structure can still emerge without weight decay.
Figure 17: The heatmap of class distance matrix when weight decay is 0. From left right: learning
rate = 0.1, 0.01, 0.001.
B (NEW) EXPERIMENT ON MIXTURE OF GAUSSIANS WITH COARSE LABELS
In this section, reproduce the fine-grained representation structure in simple synthetic setting, ie., classifying mixture of Gaussians using a 2-layer MIP, and perform some preliminary quantitative studies in this setting. This allows us to preform controlled experiments by varying different characteristics of the data distribution, architecture, and algorithmic components, and it can serve as a starting point for future theoretical work.
Formally, we create dataset from a mixture of Gaussians, where the input from the e-th cluster generated from ((u((), I), and each cluster mean f(()) is drawn i.i.d. from o20, The class label of each datapoint is the index of the cluster it belongs the separation between each two clusters is, and the more likely it is to observe fine-grained representation structure when given coarse labels.
The larger o² is, the larger
We perform the same coarsening process described in Section 3.2 (by combining two classes into one super-clase and train a 2-layer MLP on the coarsely labeled dataset. We measure the significance of the fine- see--graind structure using the ratio between [the average squared distance between representa- tions in the same super-class representations in the same subclass), which we call the Mean Squared Distance Ratio (MSDR). Mathematically, it defined as
different sub-classes) and [the average squared distance between
average igf j in same super-class (Di.}
MSDR
average; (D.+)
where Dr.j is defined in 6). larger MSDR means that the fine- Ane-grained structure is more pronounced, while MSDR x indicates no fine-grained structure.
By varying training and data-generating parameters, we investigate factors that impact the significance of the fine- tne-grained structure. Specifically, we vary the input dimension, hidden dimension in the network, and weight decay rate, and plot how MSDR scales with o². The results are shown in Figures 18 to 20 Note that in each figure we use two different scales in the x-axes to differentiate the cases of o² 2 and o² > 2.
From the figures, observe that both input and hidden dimensions exhibit a clear positive correlation with MSDR. On the other hand, the weight decay rate does not have an impact on MSDR this setting.
Training details. We generate data from 8 clusters, each having 500 samples. We train the model with gradient descent for 1,000 steps. The results are averaged over 10 runs. When varying one
12

---

Under review as a conference paper at ICLR 2023
- 96
4. 1_hou-
Lrout - 120
Loo. 122
moLt 220
now. 250
15.0
d. maut - 512
maF 512
1.rp - 1924
RDW 1034
14300
30000
7320
75
0.0
0.25
0.50
0.75
1,00
1.25
3.50
1.75
7.00
15
Verfance
Verkence
Figure 18: Mean Squared Distance Ratio variance o² for different input dimensions. Red lines the left end are cases where the training accuracy does not reach 100%.
12
120C
10
4_hidde
4_hidee1
100C
d_hidlen
800
600
273
Figure 19: Mean Squared Distance Ratio vs. variance o² for different hidden dimensions. Red lines on the left end are cases where the training accuracy does not reach 100%.
10
Waight dEcay 02-co
Wagght d8:2y 02-00
weight decay 1s:07
weight decay 12:07
#eight dacay 10:00
Wegght dec2y 12.59
wagght desay lu43
18:50
weight decey 1m:4
300C
weiight decay 1n03
R00
Sagar
400
200
0.50
0.70
1.50
1.50
2.73
Venincce
Vanaace
Figure 20: Mean Squared Distance Ratio vs. variance x² for different weight decay rates. Red lines on the left end are cases where the training accuracy does not reach 100%.
13

---

Under review as a conference paper at ICLR 2023
hyper-parametes other hyper-parameters set to their default values: dipput 512, dhidden 512, weight decay = O.
B.1 MODELING SEMANTIC SIMILARITY
In Section 5 showed that the emergence of fine-grained representations depends the semantic similarity between sub-classes, Now we take step in investigating this question by creating "similar" and "dissimilar" sub-classes in the Gaussian mixture model considered in this section.
In particular, we use the same data-generating process described above, except that half of the super-classes will be altered so that they consist of "similar" sub-classes, and we say that the other super-classes consist of "dissimilar" sub-classes. The way to generate similar sub-classes it to first sample (0, I) and then generate two means (e()), pe(e)) ~ N(, T2).. Therefore, we can vary 72 to control the level of similarity between similar sub-classes.
We use default hyper- parameters described above, fix d² 4, and vary 72. Figure shows the Mean Squared Distance Ratio for similar and dissimilar subclasses, respectively. We see that fine-grained structure within a super-class does require sufficient dissimilarity between its sub-classes, which agrecs with our observation from Section on CIFAR-100.
1.0
Lisdimitor
Sam
Simiar
M Woleeyy My ba Mereedee
Vear Siaatt betavee Ritt
Mlaa Sqaame betnovo Ritte
1.00
100
80
25:
20-
Figure 21: Mean Squared Distance Ratio vs. sub-variance 72. The setting is described Ap- pendix B.I
C COMPLETE COARSE CIFAR- 10 EXPERIMENT RESULTS
In the following sections, we provide extended experiment results. As mentioned Section 3.2 we permute learning rate in (10-1 10-2 10-3) and weight decay rate in (5x 10-3 5 10 1, 5 10-9 5). Generally, the results will be shown 3 3 table, of which each grid represents the result of one hyper-parameter combination, with each row has the same learning rate and each column has the same weight decay rate.
In this section, repeat the experiments in Section 4 with all learning rate and weight-decay rate combinations. Firstly, we present the training statistics (accuracy, loss) of all hyper-parameterss in be observed that all hyper-parameter groups achieved very low training error except the first one (weight decay 5 x 10 learning rate = 10-1). ). In fact, the last two syper-parameet combinations (learning rate 10-3 weight decay E (5 x 10-4 5 10-5)) didn't achieve exactly 0 training error (their training error are < 0.5% but not exactly 0), and the other hyper-parameter combinations all achieved exactly training error.
Figures 22 and 23 as an reference.
C. CLASS DISTANCE
Here present the visualization of the heatmap of class distance matrix D which is defined in Section 4.1. We choose 4 epochs to show the trend during training. The results are presented in Figures 24 to 27 whose epoch numbers 20, 120, 240 and 349 respectively.
14

---

Under review as a conference paper at ICLR 2023
wigit Belp 30:0
140
023
2/0
160
260
130
80
200
180
20
306
130
300
1.4
3.4
180
20
34
Figure 22: Training and test error during training. Figure 23: Training and test loss during training.
-80
230
5.
08
50
15
25
50
75
25
56
100
280
Figure 24: The heatmaps of class distance matri- ces of different epoch 20.
Figure 25: The heatmaps of class distance matri- ces of different syper-parameter combinations at epoch 120.
combinations at
C.2 VISUALIZATION
In this section, present the t-SNE visualization result of ResNet-18 on Coarse CIFAR-10 in Figures 30 The results are divided into three groups, each of which has the same learning rate and the format each group the same as Figures andd g
C.3
The Cluster-and ana-err-an---nnrrr Linear-Probe test results of ResNet-18 18 trained on Coarse CIFAR-10 with all hyper- parameter combinations presented in Figure
15

---

Under review as a conference paper at ICLR 2023
15
13
130
Figure 26: The heatmaps of class distance matri- Figure 27: The heatmaps of class distance matri- of different hyper-parameter combinations at epoch 240.
of different hyper-parameter combinations epoch 349.
25
-25
30
Clacs
clas
super cina
Super Clso
I.pe cpprc
super class
spee data
100
Geag Cliss
orageatcko Aha unigealcces
50
53
-300
-10
-55
116
-W0
B
166
100
75
30
-20
-200
Saler 0305
Super Clos.
d600 0355
Figure 28: Visualization last layer representations of ResNet- 18 trained on Coarse CIFAR-10 10 with learning rate O.I. Each represents hyper-parameter combination and each column represents a super-class. The weight decay rates from top to bottom are 5 x 10-3, 5 10-4,5 5, x 10-5
16

---

Under review as a conference paper at ICLR 2023
Ong IN dlac
-20
-20
-40
orgins clasc orginn cascs
50
15
-00
-20
-3
300
10
oeighn Lhay-4
-100
cless
suxe dess
900 doss
Super cosd
Super class
oPpes COULD
enssellddss erggall
OOEEN CORCS
ang rail dbock
-20
-00
-0
clas1
-00
-T6C
100 -100
335
-25
-00
-20
100 -15
100
Figure 29: Visualization of last layer representations of ResNet-18 18 trained Coarse CIFAR-10 with learning rate = 0.01. Each row represents a hyper-parameter combination and each column represents super-class. The weight decay rates from top bottom 5 10-3, 3, 10-4, x 10
100
atgral-daard onginalckas
onggiall caa
orginall Fu
42
orginal chau- orgirali
29
20
-10
44
orginal Fass orginal casa
-100
-3
100
120
OPggN Cleo
oigena Clos
cao orgine clasi
-20
-40
-50
orighal
-60
super Ciss
3128 Cass
sazer das
5228 den
auzer cam
472 Clao
50
20
-40
erianal clusc
320
-20
20
100
-30
20
202
-56
-230
-20
Figure 30: Visualization of last layer representations of ResNet- 18 trained on Coarse CIFAR-10 with learning rate 0.001. Each row represents a hyper-parameter combination and each column represents super-class. The weight decay rates from top bottom are 10-3 5.5x 10-4 10-5
17

---

Under review as a conference paper at ICLR 2023
Sed0
30.5
50.00
50.00
130
20.00
730
200
15
Figure 31: The result of -eniee-rr----inea--PPe
test. the figure"CLP" refers to Cluster-and- Linear-Probe, "LPO" refers to linear probe with original labels and "OT" refers to the test set ac- curacy of model trained original CIFAR-10.
Figure 32: The heatmaps class distance matri- ces of different hyper-paramete combinations on Fine CIFAR-I0 1-10 at epoch 20.
Poott entry
ghg deny Se.t
gli Oeby
144
Figure 33: The heatmaps of class distance matri- ces of different hyper-parameter combinations on Fine CIFAR-10 10 epoch 200.
Figure 34: The heatmaps of class distance matri- ces of different hyper-parameter combinations on Fine CIFAR-10 epoch 350.
D COMPLETE CLASS DISTANCE RESULT OF FINE CIFAR-10
In this section, we provide the visualization of the class distance matrix of Fine CIFAR-10 with all
hyper-parameter combinations, which has been partially displayed in Section 6 As before, multiple epochs during training are selected to display a evolutionary trend of the class distance matrices. The results presented in Figures 32 to 34 whose epoch numbers 20, 200 and 350 respectively.
18

---

whose epoch number are Under review as a conference paper at ICLR 2023
1.113
Bm 1000
len 3900
000 31495
20C
Kin Km--00000
300
OC 201
OCT 2004
NC*
Figure 35: The heatmaps of class distance matri- ces of different hyper-parameter combinations on Coarse CIFAR-100 at epoch 20.
Figure 36: The heatmaps of class distance matri- ces of different hyper-parameter combinations on Coarse CIFAR-100 epoch 200.
E EXPERIMENT OF RESNET-18 ON COARSE CIFAR-100
In this section, report extended experiment results Coarse CIFAR-IO0 100. The same with the case of Coarse CIFAR-10, we construct CIFAR-100 through the label coarsening process described in Section 3.2 and choose Ö = 20, so that every 5 original classes are merged into one super-class. We repeat most of experiments in Section
E.1
CLASS DISTANCE
The heatmaps of distance matrices are presented in Figures 35 36 and 38 20, 200 and 350 respectively.
E.2
VISUALIZATION
In this section, present the t-SNE visualization result of ResNet-18 18 Coarse CIFAR-100. We only put the result for config #1 here as a demonstration Figure 37
E.3
Notice that we omit the visualization result of Coarse CIFAR-100 since there are too many fig- ures. We present the Sluster-and-Linearr Probe results to reflect the clustering property of last-layer representations learned on Coarse CIFAR-100. The CLP results are presented in Figure
F RANDOM COARSE CIFAR-10
In this section, as mentioned Section 3.2 we make our experiment more complete by performing a random combination of labels CIFAR-
R- 10 rather than using determined coarsening process in the main paper. The dataset construction is almost the same as the process of assigning coarse labels described in Section 3.2 except here randomly shuffle the class indices before coarsening them.
The class distance matrices of three difference epochs are shown in Figures 40 to 42 From the results see, although there longer three dark lines, for each row there generally dark
19

---

Under review as a conference paper at ICLR 2023
Figure 37: Visualization of last layer representations of ResNet- 18 trained Coarse CIFAR-100 under config #1. Each grid represents a super-class and each color in grid represents sub-class.
$0.00
70:00
1.0
50.00
2.00
20
102
200
11.00
20
780
19:00
2000
56.08
50.00
30.00
200
Dec 421
22.4
Figure 39: The result of Cluster-and-Lineer-PPooo test. In the figure"CLP" Linear-Probe, "LPO" refers to linear probe with original labels and "OT" refers to the test set ac- curacy of model trained on original CIFAR-10.
Figure 38: The heatmaps of class distance matri- ces of different hyper-parameter combinations Coarse CIFAR-100 100 at epoch 350.
"CLP" refers to Cluster-and-
20

---

Under review as a conference paper at ICLR 2023
15
200
150
19
56
26
379
215
190
135
Figure 40: The heatmaps of class distance matri- ces of different hyper-parameter combinations on Random Coarse CIFAR-10 at epoch 20.
Figure 41: The heatmaps of class distance matri-
ces of different hyper-parameter combinations on Random Coarse CIFAR-10 at epoch 200.
velctt Srtay +140
decy Sed44
Bct
10
15
16
780
140
135
180
190
110
136
150
Figure 43: The heatmaps of class distance ma-
Figure 42: The heatmaps of class distance matri- ces of different Random Coarse CIFAR-10 10 epoch 350.
trices of different hyper-parameter combinations with DenseNet-121 on Coarse CIFAR-10 at epoch 20.
combinations on
blocks, represents the original classes belongs to the same super-class, and the same observations in Section cann still be made here.
G EXPERIMENT WITH DENSENET
We also perform experiments with different neural network structures for completeness. In this section, we show the result with DenseNet- 121 on Coarse CIFAR-10. The experiments with DenseNet is supportive observations in the main paper.
21

---

Under review as a conference paper at ICLR 2023
deyy
17
140
130
15
13 11 50
19
16
26
13
205
125
130
Figure 44: The heatmaps of class distance ma- trices of different hyper-parameter combinations with DenseNet- 121 on Coarse CIFAR-10 at epoch 20.
Figure 45: The heatmaps of class distance ma-
trices of different hyper-parameter combinations with DenseNet-121 on Coarse CIFAR-10 at epoch 20.
G.1 CLASS DISTANCE
The class distance matrices of three epochs during training are presented in Figures 43 to 45
G.2
CLUSTER-AND-LLNEARRPPOO
The Beeeter-and-Lnnea--PPe test results are presented Figure 46
H EXPERIMENT WITH VGG
We also extend our experiments to VGG-18. Interestingly, VGG to some extent is a counter example of the observations made in the main paper: it only displays Neural Collapse, and can not distinguish different original classes within one super-class, even in an early stage of training. The reason why VGG is abnormal requires further exploration.
The class distance matrices of three epochs with VGG during training are shown in Figures 47 to It can be observed that the three dark lines appears almost at the same time and always be of nearly the same darkness. This represents the trend predicted by Neural Collapse (Figure 4 (a)), but rejects the prediction made by (Figure (b)).
22

---

Under review as a conference paper at ICLR 2023
WP/ Office 5o:43
#EOgH Rmtry S4.M4
wWW acow So.dd
135
0
spach
56
13.00
73/00
200
1b0ch Ind
Figure 46: The result of Cluster-and-Linear-Poon test with DenseNet-122 121. In the figure"CIP"" refers to Cluster-and and-aineaal Probe. "LPO" refers to lin- ear probe with original labels and "OT' refers the test set accuracy of model trained on original CIFAR-10.
Figure 47: The heatmaps of class distance ma-
trices of different hyper-parameter combinations with VGG-18 18 on Coarse CIFAR-10 10 at epoch 20.
350
75
15
75
10
15
140
1.09
400
Figure 48: The heatmaps of class distance ma- trices of different hyper-parameter combinations
Figure 49: The heatmaps of class distance ma-
trices of different hyper-parameter combinations with VGG-18 on Coarse CIFAR-10 10 at epoch 350,
with VGG-18 on Coarse CIFAR-10 at epoch 200,
23


Reviews:
[{"review_id": "Jux9jSLgUX", "reviewer": "Reviewer_jgRV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The Neural Collapse phenomenon indicates that the last-layer representation of training samples with the same label would collapse into each other in well-trained networks. It means that the last-layer representation would only be determined by the labels, regardless of the input data distribution. This paper suggests that for practical networks with only approximate neural collapse, the small amount of remaining variation can still capture the intrinsic structure of input distribution. The contributions mainly involve the following two observations. First, the authors observe that the effect of input distribution appears earlier in training, while the effect of labels emerges at the terminal phase of training. Second, experimental results indicate that the collapsed representation corresponding to each label can still keep fine-grained structures determined by the input distribution. ", "summary_of_the_review": "This paper studies the role of input data distribution on the last-layer representation of neural networks. I have mixed feelings about the recommendation. Given the recently observed neural collapse phenomena indicating the last-layer representation would only be determined by the labels, this paper provides a very interesting observation that the small amount of remaining variation can still capture the intrinsic structure of input distribution. This result could be of interest to the community. But on the other hand, as mentioned before, a similar phenomenon seems to have already been discovered in the literature. I look forward to the authors’ response.", "strength_and_weaknesses": "## Strength:\n- This paper tackles an important and very interesting problem about whether and how the input distribution would affect the structure of the last-layer representation. \n- It reveals that for practical networks with only approximate neural collapse, the small amount of remaining variation can still capture the intrinsic structure of input distribution. In particular, the last-layer representations of different subclasses within the same class are separated into different clusters according to the input distribution.\n- This paper provides rich visualizations to evaluate the observed phenomena. For example, the authors use the heatmap of the class distance matrix to measure the clusters and capture the change during training. The Cluster-and-Linear-Probe(CLP) method is performed to validate the fine-grained structures determined by the input distribution.\n\n## Weaknesses:\n- As mentioned in Section 2, the phenomenon that last-layer representations of different subclasses within the same class are often separated into different clusters has already been discovered in the literature, such as by Sohoni et al. (2020). The only difference in this work seems to be that this phenomenon is still observed when the network is trained extremely long (such as 1000 epochs). As this is the main contribution of this paper, it seems the contribution or novelty is limited, though personally, I think the result is very interesting. \n- Following the above point, the level of neural collapse mainly depends on two factors: the training epochs and the network capacity (width and depth). The current experiments are conducted with ResNet-18 and include the setting with a sufficient number of training iterations. What if one uses a much larger network? Do we still observe similar additional structures within the neural collapse representations? The features tend to be more collapsed with a larger network.   \n- The two settings config #1 and config #2 only differ in the weight decay parameters, but learn different representations as shown in Figure 5 and Figure 6. Could the authors comment on why would the weight-decay rate be a critical hyper-parameter for the final fine-grained structure of last-layer representation?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JulZFgvosSA", "reviewer": "Reviewer_fK87", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically studied the previously observed phenomenon of \"neural collapsing\", which suggests that the last layer representation collapse for each class. This paper investigated this problem with ResNet-18 on variants of dataset created from CIFAR-10/100 and found that even though the phenomenon does occur, finer grained information still presents in the representation. For example, when trained with 5 \"super-class\" labels created by combining two classes from CIFAR-10, the 10-class structures can still be found in the representations, and a simple kmeans clustering can separate the 10 classes with high accuracy even after 1000 epochs of training (into the neural collapse regime).", "summary_of_the_review": "This paper studies a clear question and make a simple answer. My main concern is that this paper does not have enough materials to support a full conference paper. I've suggested a few directions to expand the studies that could potentially make this paper more solid in the section above. \n\n---------------------------------\nThanks to the authors for the reply and additional synthetic experiments. I'm willing to raise the score a bit, but since there is no rating between 3 and 5, I'm keeping my current rating.", "strength_and_weaknesses": "**Strength**: This paper studies a simple question (does the representation depends on input structure in the neural collapse regime) and shows a clear answer (yes it does).\n\n**Weakness**:\n\n1. While the main conclusion in this paper is interesting and clear, I don't think this paper has enough material to make a full paper. One potential direction to enrich the results is to make the study more systematic. For example, currently there is an anecdotal observation on CIFAR-100 that when the sub-classes are \"semantically similar\" enough, the sub-class structures could be blurred when training with superclass labels. It could make the paper more solid if systematic and quantitative results are presented with studies over different network architectures, training algorithms and hyperparameters and different type of datasets. A related question would be are there other input structures other than the fine-grain labels recoverable from the representation in the neural collapse regime. Another potential direction is to formally analyze the reason behind such phenomenon. \n\n2. It is well known that a lot of information is retained in the final layer representations and sometimes even in the logit vectors after a model is trained with a classification loss. There is a whole line of research on recovering instance based information or even reconstructing the inputs that is missing from the related work section. e.g.\n\n    - Mahendran et al., Understanding deep image representations by inverting them.\n    - Dosovitskiy et al., Inverting visual rep- resentations with convolutional networks.\n    - Nash et al., Inverting supervised representations with autoregressive neural density models.\n    - Teterwak et al., Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers.\n    - Rombach et al., Making Sense of CNNs: Interpreting Deep Representations and Their Invariances with INNs.\n  \n3. The experiments in Section 5 show that when learning with super classes constructed from CIFAR-100 according to the semantic labels, some of the fine-grain class structures are missing in the learned representations. To what extent does this depends on the superclasses being semantically constructed? Because the original experiments was on cifar-10 forming a super class with 2 classes. It would be helpful to have a more controlled baseline with 5-class formed superclasses as well -- for example, on cifar-100 with randomly formed superclasses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IKI8TPYvPR", "reviewer": "Reviewer_MGwH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the question of whether neural collapse is solely dependent on the output labels as suggested in prior work, or if it still contains information about the input distribution. To answer this question, the authors train models on Cifar10 and Cifar100 using coarse-grained and fine-grained labels, and measure if the original labels are still recoverable from the last layer activations even after neural collapse. They find that the original labels can be reconstructed through a Tsne and clustering mechanism, but less so when the class groupings are semantically similar. ", "summary_of_the_review": "The work provides some new insights into the phenomenon of neural collapse, but some missing hyperparameter settings raises doubt over whether the observations are an artifact of the training procedure, thus it is marginally below the acceptance threshold in my opinion. Including those hyperparameter results will likely increase my score.", "strength_and_weaknesses": "### Strength:\n\n- The paper tackles an interesting question related to the phenomenon of neural collapse, which is whether information about the input distribution is still carried in the last layer activations after neural collapse. The experiments with coarse-grained and fine-grained class labels are well-designed to answer this question, and the visualizations and CLP accuracy provide compelling evidence for the role of the input distribution even with neural collapse. \n\n### Weakness:\n\n- I think a wider set of hyperparameter settings would be needed to fully complete the story. For example, using a learning rate decay schedule and a setting with 0 weight decay would have been good to see. This would help answer whether the input information would be erased when the network is able to converge to a lower loss, which would tell us if this phenomenon is inherent in neural network training / neural collapse, or if it is only an artifact of strong regularization in the terminal phase of training.\n\n- An explanation or intuition for why config #1 and #2 generate different behaviors in different experiments would be a valuable discussion to include.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E2sYqcaXk9p", "reviewer": "Reviewer_VqKm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study starts from the view that neural collapse is a phenomenon only concerned with the explicit labels of the dataset. This study suggests that intrinsic structure of input distribution should also play a role in the last-layer representation structure. The authors construct two experimental settings where the input distribution is not consistent with the label, namely Coarse CIFAR-10 and Fine CIFAR-10, to verify the insights proposed in this paper.  ", "summary_of_the_review": "The study has a clear motivation. But its novelty and significance are very limited. ", "strength_and_weaknesses": "Strengths: \n\nThe motivation of this paper is interesting. The study on the representation structure is very important. The paper has a good presentation. \n\nWeaknesses:\n\n(1).\tThe core insight of this paper is that the input distribution should play a role in the neural network representation structure, which is ignored by neural collapse. At the first glimpse, it is interesting and brings something new over neural collapse. However, we should note that neural collapse is an ideal state. It is a target/goal of the optimization in a neural network. It is theoretically proved to be the global optimality (0/minimal loss) of unconstrained feature model or layer-peeled model. But in real implementations, we almost cannot realize this ideal state, and an optimized model almost cannot attain 0 training loss. Even in the neural collapse study (Papyan et al., PNAS), the within class variance approaches to 0, but is not 0. So, NC is a limit, instead of a real state. \n\nLet us assume that neural collapse indeed happened in the coarse label training. The within class variance would be 0, and all features in the same coarse class would be exactly the same. In this case, it is impossible to reconstruct the original classes by clustering. When the within class variance is not 0 (no matter how many coarse labels), it is not surprising that the representation structure is affected by the input distribution, and we can cluster the representation into the classes of input distribution.\n\nAs a conclusion, the main view of this paper that input distribution also affects the representation structure is correct. But is has no conflict with neural collapse. Neural collapse is just a limit. In a real neural network, we cannot infer that label is the only factor to decide the representation structure. So, this study does not bring something new or inspiring. The novelty and significance are limited. \n\n(2).\tI do not see any instruction from this study. The view that input distribution also affects the representation structure comes with no surprise. The paper verifies this view by coarse and fine label training. But how does this work instruct us for training a neural network? \n\n(3).\tThe main question claimed in Introduction that “how can we reconcile the roles of the intrinsic structure of input distribution vs. the explicit structure of the labels in determining the last-layer representations in neural networks?” is not proper. First, as given in (1), neural collapse has no conflict with the intrinsic structure effect. I do not think we need to reconcile the two roles. Second, the authors seem not to answer this question. As given in (2), this paper does not offer any guidance. I do not see any solution that can “reconcile the two roles”. \n\n(4).\tReference is not sufficient. Many related neural collapse studies are not cited and discussed with. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject