Linguistic Collapse: Neural Collapse in (Large) Language Models
Authors: Robert Wu
Vardan Papyan
Venue: NeurIPS 2024 poster (2024)
Keywords: neural collapse
uniformity
large language models
LLM
GPT
language modeling
geometry
unconstrained features
generative model
transformer
attention
causal
autoregressive
Primary area: optimization_for_deep_networks
TL;DR: Analysis of the development of neural collapse in (large) language models and its relationship with generalization.

Abstract: Neural collapse ($\mathcal{NC}$) is a phenomenon observed in classification tasks where top-layer representations collapse into their class means, which become equinorm, equiangular and aligned with the classifiers.
These behaviors -- associated with generalization and robustness -- would manifest under specific conditions: models are trained towards zero loss, with noise-free labels belonging to balanced classes, which do not outnumber the model's hidden dimension.
Recent studies have explored $\mathcal{NC}$ in the absence of one or more of these conditions to extend and capitalize on the associated benefits of ideal geometries.
Language modeling presents a curious frontier, as \textit{training by token prediction} constitutes a classification task where none of the conditions exist: the vocabulary is imbalanced and exceeds the embedding dimension; different tokens might correspond to similar contextual embeddings; and large language models (LLMs) in particular are typically only trained for a few epochs.
This paper empirically investigates the impact of scaling the architectures and training of causal language models (CLMs) on their progression towards $\mathcal{NC}$.
We find that $\mathcal{NC}$ properties that develop with scale (and regularization) are linked to generalization.
Moreover, there is evidence of some relationship between $\mathcal{NC}$ and generalization independent of scale.
Our work thereby underscores the generality of $\mathcal{NC}$ as it extends to the novel and more challenging setting of language modeling.
Downstream, we seek to inspire further research on the phenomenon to deepen our understanding of LLMs -- and neural networks at large -- and improve existing architectures based on $\mathcal{NC}$-related properties.
Our code is hosted on GitHub: [`https://github.com/rhubarbwu/linguistic-collapse`](https://github.com/rhubarbwu/linguistic-collapse).

Linguistic Collapse: Neural Collapse in (Large) Language Models
Robert Wu
Vardan Papyan University of Toronto, Vector Institute vardan papyan@utoronto.co ca
University of Toronto, Vector Institute rupertocs. toronto. edu
Abstract
Neural collapse (NC) is phenomenon observed in classification tasks where top-layer representations collapse into their class means, which become equinorm, equiangular and aligned with the classifiers. These behaviours associated with generalization and robustness would manifest under specific conditions: models are trained towards zero loss, with noise-free labels belonging to balanced classes, which do not outnumber the model's 's hidden dimension. Recent studies have ex- plored NC the absence of one or more of these conditions extend and capitalize the associated benefits of ideal geometries. Language modelling presents curi- ous frontier, as Iraining by token prediction constitutes classification task where none of the conditions exist: the vocabulary is imbalanced and exceeds the embed- ding dimension; different tokens might correspond to similar contextual embed- dings; and large language models (LLMs) particular are typically only trained for a few epochs. This paper empirically investigates the impact of scaling the architec- tures and training of causal language models (CLMs) their progression towards NC. We find that NCC properties that develop with scale (and regularization) are linked to generalization. Moreover, there is evidence of some relationship between NC and generalization independent of scale. Our work thereby underscores the generality of NCC extends the novel and more challenging setting of language modelling. Downstream, we seek to inspire further research on the phenomenon deepen understanding of LLMs and neural networks at large and improve existing architectures based on AC-related properties. Our code is hosted GitHub: https //gi htt.tttttt.tt/t/////////. com http:///whoaaaaooo//iiiiiiioooaa
R² 0.92
R² 0.84
0.69
R² 0.89
16.389
17.984
Val. Loss
3-
17.489
796
2
15.746
-8
6
22 0.25 0.50 0.75 1.00
0.1 0.2 0.3
Variability Collapse
Hyperspperical Uniformity (CA(C))
Uniform Duality (cosine) (1/WC²))
Classifier Agreement (linear vs. NCC) (NCC))
(log CDNV) (N02)
Figure 1: Simultaneous development of the four neural collapse (NC) 1 properties in 230 causal language models trained on TinyStories [2]. alongside improvement in generalization (i.e. validation performance). Left to right: N/11) within-class (representation) variability collapse; G.CC2) hyper- spherical uniformity of class means; LNNCs) uniform duality between class means and corresponding classifiers; and M/C4) agreement between token (maximum prior) classifiers and implicit nearest- class centre classifiers. Coloured by model size and annotated with coefficient of determination (R²).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
1.1 Neural Collapse
learning phenomenon known as neural collapse (NC) emerges during the terminal phase of training (TPT) deep neural networks with cross-entropy (CE) loss for classification tasks. [[] It was originally characterized the co-occurrence of the following properties in model's top-layer (also known as last-layer) representations (also known as features or embeddings) and linear classifier weights:
(WC1) Within-class variability collapse: Top-layer representations collapse to their class means.
(WC2) Convergence to simplex ETF: Class means tend towards equinorm and equiangular vectors when centred about the global average. The resulting geometry known as simplex equiangular tight frame (ETF) maximizes pairwise angles and distances.
(WC3) Convergence to self-duality: Linear classifier weight vectors converge to their corre- sponding top-layer class mean vectors, and thus also form a simplex ETF.
(WC,) Nearest decision rule: Linear classifiers approximate a nearest-class centre (NCC) classi- fier: top-layer representations predict the class with the closest mean (implied by NNC1-3).
These behaviours, often associated with improved generalization and robustness 335 (among other benefits, such as those discussed in $1.4) traditionally manifest under the following conditions:
Rq1) Few classes: The number of classes upper-bounded by the embedding dimension plus one: C S d + 1; this is required to construct a perfect simplex ETF.
Rq2) Balanced classes: The number of samples is equal across classes: N. No. Ve d.
Rq3) Noise-free labels: Identical (or very similar) embeddings should belong to the same class.
Rq4)
Sufficient training (TPT): The model is trained past zero error, towards zero loss.
Absent these conditions, does not typically anticipate NC. Since then, follow-up studies have extended beyond and proposed techniques of quantifying or achieving NC (discussed in Section 2
1.2 (Large) Language Models
NC is a phenomenon observed specifically in classification tasks. While not traditionally thought of classifiers, language models including large language models (LLMs) learn to model aleatoric uncertainty, which can be viewed as stochastic token prediction 66. For instance, masked language models (MLMs) such BERT 7 predict one or several masked tokens within an input sequence based on the surrounding context. Likewise, autoregressive or causal language models (CLMs) such as GPT g perform next-token prediction (NTP) in a sequence given the context of previous tokens. Most of these models are essentially Strained by token classification on their vocabularies. This parallel also drawn by raises few natural questions:
1. Does the pre-training stage of language model exhibit NC?
2. How do scaling and other training configurations influence NC (L)LMs?
3. To what extent is ACC in (L)LMs correlated with their generalization abilities?
4. Do such correlations, between NC and improved generalization, persist independent of the (potential confounders of) model size and training?
To address these, we first examine the specific settings of training CLMs as they are opposed (-) to the traditional prerequisites (RI $I.I for NC to manifest.
-Rq1) Many classes:
The unique tokens found in language modelling vocabularies are vast, usually numbering in the tens of thousands and far exceeding the hidden dimension [10].
RRq2) Imbalanced classes: The distribution of tokens in natural language is typically very
imbalanced 1I [2], as is the case in TinyStories 2 the dataset we use (Appendix Figure
4 It has an average of 16K samples per class but standard deviation of over 32K.

---

RRq3) Ambiguous contexts: There may exist very similar or even identical contexts followed by different tokens in the natural language data [3]. For instance, over half of the sequences in TinyStories [2 lead with "Once upon a time", but only three-quarters of these follow with comma ("Once upon a time," "). In other words, there almost certainly some ambiguity to contend with in our context embeddings.
RQq4) Undertraining: Most practical language models (including LLMs) are not trained for more than few epochs [14 15). Further optimization typically renders diminishing returns in improving evaluation performance [6] long before any possible TPT.
1.3 Contributions
We train suite of Transformer-basee [17 CLM across grid of model widths, depths, and training epochs on the TinyStories dataset [2] to assess the degrees to which NC properties develop and how they relate to generalization performance. Our findings (summarized in Figure [ reveal:
Emergence of NC with scale: As we scale model size and training, the properties of NC emerge; within-class variability (AC,) and interference are reduced while hyperpsherical uniformity (GNC2,,, uniform duality (L,,C3), and classifier agreement (NC44) improve.
Progression towards hyperspherical uniformity: Class means, while unable to form simplex ETF (NC2), nonetheless tend towards uniform dispersion hypersphere, geometry theorized by [8] and formalized by 19 as hyperspherical uniformity (GNC2).
Tendency towards uniform duality: Direct alignment (self-duality) between class means and classifiers (NCs) does not appear to develop with scale. Instead, the variability of (mis)alignment across classes decreases with width and training, suggesting its minimization which we term uniform duality (UNNC3) may be more cohesive NC..
The developments of NC properties are correlated with improved validation performance. We show these correlations persist even when fixing the (potential confounders of) model architecture and training by simply varying the random seeds for initialization and data shuffling. This suggests that NCC is not simply side-effect of training, but possibly factor of generalization in its own right.
Correlation between NC and generalization:
1.4 Significance
Recently, methods building NCC have found use in deep learning at large. We highlight areas such learning (22/24) meta-
as federated learning (20], graph neural networks [21] incremental/continn
learning 24 25
out-s--disti detection 226.28 privacy [29 30), learning theory xl 36
transfer learning 3
5
37 40), and even LLM prompting 41.. With our results, we aim to extend
insights from such contributions and other related works to the autoregressive language modelling domain and ultimately assist researchers in improving and interpreting their (L)LMs.
2
Related Works
NCC was initially observed in image classification tasks such as on CIFAR-1O/100 42 and ImageNet
43
Since then, the phenomenon has been further studied both theoretically and empirically [5] [8]
35
66), with several works venturing into settings without some of the traditional prerequisites
( (-Rq1-4,
1.2) and proposing adaptations of the analysis framework optimization procedures:
large number of classes (-Rq1) NC established the simplex ETF as an optimal configuration. However, a perfect simplex ETF (NC2) requires that the number of classes C' not exceed d + 1 where d is the embedding dimension. This requirement that be sufficiently large impractical when the classes number beyond the thousands For instance, GPT-2 (68) and Llama 3.1 [69 have vocabularies of around 50K and 128K tokens, respectively.
In such scenario, one might still expect class means to be uniformly distributed on d-dimensional formalize this as hyperspherical uniformity within generalized neural
hypersphere 18] 19
I 'Our models range from 3.4M (small) to 205M (large), so we inclusively use "CLM" instead of "LI.M.. Following [67]. might describe such setting (C > d + model "in superposition".
3

---

collapse (GNC) framework, which [9] then empirically demonstrate. These latter two works mention language modelling as applicable for GNC: ( even framed token prediction as a classification task, just we do. We remark however that both earlier works simulate large number of classes by drastically shrinking the embedding dimension. contrast, we study realistic NTP, using the full class space (vocabulary) with an imbalanced token distribution and ambiguous samples.
Class imbalance (-Rq2) NC traditionally assumed that classes were cumple-balanced Since then, follow-up works have investigated the effect of class imbalance on the prevalence of NC. [47] studied the layer-peeled model (LPM) and discovered that minority collapse occurs when classes imbalanced across two equally-sized groups of classes; a threshold for minority collapse was later characterized by [35]. [52] showed that NCC still occurs in such an LPM when the classifier initialized fixed ETF. 70 introduced simplex-encoded-laie
interpolation (SELI) but noted that more severe imbalance worsens even this geometry. Recently, feature regularization has been employed to induce NC and improve generalization dlass-imbllancer settings [58 61
Multi-label classification (-Rq3) In some problems, one might encounter mixed or multi-label samples, be they natural or due to noise or augmentation 72). .NC was also recently studied for such data by 773. who observed that multi-label class means arrive at the average of their labels' respective means. They also devise an augmented CE loss function to accommodate such samples.
Likewise, most of our ambiguous samples could be considered multi- or soft-label: identical (or very similar) context samples with different hard token labels (-Rq3). Under popular CLM pre-training (teacher-forcing with CE loss), this effectively precludes the prospect of achieving zero classification error and potentially introduces irreducible noise.
A recent study showed that memorization of noisy labels likely leads to degradation of ACC 660
Early stages of training (-Rq4) [53 studied AC in small ResNet 77 models that had not yet
converged (similar to most LLMs). They show that within-class variability drops and "plateaus" (NC,) earlier than other NC metrics, a result that we also observe ($4.T Figures 77.
Natural language processing (NLP) An earlier study reported that the ratio of within-class to between-class covariances of word embeddings increases with model depth [75] (6)., seemingly at odds with NCC, It can, however, be distinguished from literature studying layer-wise NC in that does not centre the mean representation vectors (i.e. subtract the global mean).
19 fine-tuned BERT using their hyperspherical uniformity gap (HUG) objective binary classification tasks from the GLUE benchmark (77). (78) conducted a tangentially-relate inves- tigation of convolutional neural networks for few-class semi-supervised clustering in which they identify NC.. Our work is distinct from these in several ways: a) our class space far exceeds our embedding dimension (C >> d + 1) because we classify the full token vocabulary; b) we analyze embeddings at a loken-level granularity rather than at the sequence level; and c) our NTP task is causal (context-depp deeeedeed)))) opposed to the per-sample independence of their few-category classification.
more related work studied feature collapse in individual word representations [79], but the authors note that their analysis is limited to shallow NLP modelling more rigid and tabular text data.
3 Methodology
Below we describe the training setup for our CLMs (.3.) procedures 3 for collecting top-layer embeddings (33. 3.7. and measurements of AC and generalization ($3.4 3.5
3.6
3.7 3.8
3.1 Dataset and Language Models
TinyStories 2 is synthetic dataset generated by GPT-3.5 and GPT-4 using around 1500 English words a child might use. NTP is performed by sampling from the token vocabulary V = [1.29233]
3 "Leveraging our generic. NCC package: htttps:/ggiteee /ttpb/ttttttuutt. homm/hhbbarbww/nuuraalaaaoo cttp://ooooral-cclppsepee 'TinSStorie was developed and evaluated as a faithful emulation of large language modelling, so we chose it for experimentation to train hundreds of CLMs and analyze embeddings at low cost. See Appendix

---

which for our purposes can therefore be framed as classification among C = 29.233 classes Following the Style preprocessing regime [8] raw sequences are packed into S chunks of size
T, providing N = S(T 1) token samples for training Details are listed in Appendix A
We use 30 CLM architectures based GPT Neo [0]. configured similarly 2
They vary in width (embedding dimension) d E {64, 128, 256, 512, 768, 1024] and depth (number of self-attention layers) {1, 8, 12). Our models were trained by teacher-forcing") using CE loss. For each architecture, we trained multiple models for 1, 3, and 10 epochs ablated over weight decay factors 0.0005 51 and B 0.1 81 Further details are listed in Appendices B
3.2
Context Embeddings
Base CLMs perform next-token prediction: given a sequence tokens 3ia E V top-layer context embedding h ((11:)) IRd is used to predict the next token 2'41 V where 1 S T. A classifier for class with weights w. and bias b. would make maximum prior (MAP) predictions
+++ = argmax (we, h((1a)) be-
cEV
Class embedding means For each token class we are interested in accumulating the mean embedding fe_ E IRd across sequences and their contexts æiat where the next token 2+41 ground-truth (f < T) and equal to
(x()) is
h (
(6) 2441
c)).
N
t=1
where N₂ is the number of samples of class and I is the (binary) indicator function. We also utilize Me-A
their unweighted average ç == Ee[Me,. and subsequently the globally-centred means P. ll Alla"
Class embedding variances In a second pass, accumulate the sample variance norms 10
oe = 1
(z))))
Me 1el11(1f (e) = =c)
3.3 Homogeneity and Variability
For some collapse measures (such as (G)NC, and NC3), we are primarily interested in the variation rather than the average of pairwise relationships. To that end, also include in our analysis the coefficient of variation (CoV) of several measurements, which is the ratio their standard deviations to their means: d-)//(()). We can interpret this normalized measure of variability.
3.4 Signal-to-Noise Ratio - NC1
The ability to disambiguate between classes depends on the ratio of within-class to between-class variabilities. Building upon foundational works 855 86), NC originally measured variability through an inverse signal-to-no ratio (SNR), whose minimization constitutes within-class variability collapse 1 We instead employ class-distanee normalized variance (CDNV) similar to [3:
0² +
0e.e
Ve # d.
Me
2 21|
Mer
Our metric differs in that we divide by an additional power || -1 of the mean distance norm. This further downweights the CDNV within well-separated class pairs in favour of emphasizing more
$ Althhough the GPT-Neo 80 tokenizer has over 50K tokens, only a subset vocabulary appears in TinyStories.
'We cannot the first ground-truth nor the last predicted token in any chunk.
labels for context as opposed to predicted tokens. SSimilar to many causal L.Mss [4), classifiers do not include additive biases, b 0.
"Paralielized training using sequenees' ground-tre
Different from most literature where classes were halanced and weighting is already equal.
1h sample variance is computed across all unnormalized dimensional entries.
5

---

mutually noisy pairs. We found this augmented CDNV with k = 2 to be especially useful in our setting of many imbalanced and variably confusable token classes.
These pairwise measures constitute the off-diagona [T entries of symmetric matrix in RC whose an inverse SNR. Within-class variability collapse is then re-characterized by the minimization of this quantity: Oe.c' - 0, Ve & c'. This alternative convergence is empirically faithful , but more robust and numerically stable 3
average
3.5 Geometric Structures - (G)NC22
The separability of our representations also depends on the geometric structures found in our embed- dings. [ characterize NC2 as convergence to a simplex equiangular light frame (ETF) [87] 88
Equinormness
Such a near orthooormal configuration firstly implies that class means are equinorm:
log [r,
مll2 log l
حll2 - 0, Ve + c'.
(5)
We measure CoV in the logarithms of class mean norms to assess the degree of "equinormness".
Equiangularity NC₂ also entails that class means are equiangular about their centre jı: pairwise distances and angles between their class means should be maximized and similar. Following M. we measure interference (sometimes known as similarity or coherence [89 90).. Its minimization,
(p Pe - Yc * d,
(6)
together with equinormness (Equation constitute convergence geometry is not ultimately attainable since there are too many classes (C' > d + 1), it can still be meaningful to measure model's tendency towards one. As with CDNV noise (Equation 4 pairwise interferences form off-diagona entries in a symmetric matrix in RCXC The minimization of CoV in interferences therefore expresses the degree of "equiangularity".
simplex ETF. Although this
Hyperspherical uniformity A relaxation from the ETF is hyperspherical uniformity (GNC2), with eqiunorm (Eq. 5 means μe uniformly distributed on the d-dimensional hypersphere [18 19 We likewise gauge the angular uniformity with pairwise interactions through some distance kernel
Σ
K (Prr) + min Σ E exce
A...A epc
3.6 Alignment Between Classifiers and Class Embedding Means (4)NC3
The linear classifiers [we]C] lie in a dual vector space to that of the class means { While convergence self-duality (NCC) was initially measured as distances (I] between class means and classifiers (Equation we follow [ to inspect class- wise cosine similarities
We
Pio
- 1, Ve
(8)
For intuition analogous to that for equinormness and equiangularity ($3.5), we also measure uniform duality (UNCs) as the minimization of the CoV of these similarities (Appendices O)
3.7 Agreement of the Classifiers- - NC4
Finally, NC,, is described as the simplification (or approximation) of the linear classifier's MAP prediction behaviour (Equation [
to that of the implicit nearest-class centre (NCC) classifier:
argmax (we, h)) + b. -> argmin
|| Mell2, Vh.
cEVV
cEV
HTee main diagonal of CDNVs is undefined (due to the denominator) and ignored.
12ehe main diagonal of interferences is equal to 1 (maximal coherence self- self-similarity. Foollowing [19] we employ logarithmic inverse distance kernel Hog (a. b) log ||a b15² for its ability to emphasize gaps hetween small distances while scaling down the effect of larger distances.
"Dot-pree product would be confounded by norms and therefore inappropriate for similarity up to rescaling.

---

We calculate agreement as the proportion of validation samples on which the classifiers agree: T6
Toi-1 (+)
11:t mells)
2
argmin
h
(10)
cEV
3.8 Probing Relationship Between NCC and Generalization
To isolate the effect of NC on generalization independent of model scaling and training (if it exists), we selected a two-layer 768-wide architecture of which to train twenty more instances with weight decay / 0.0005, each with a different data shuffling seed. We then followed the remainder of the pipeline described above to collect and analyze embeddings with respect to NC. Finally, we performed a permutation test with 10 trials determine the statistical significance any correlation between NCC and generalization that remains when we hold constant all factors but shuffling seeds.
4 Experimental Results
In this section, we present the results from our empirical study on scaling and generalization:
(WC1) Within-class variability is reduced across model scale (more so by width than depth) and training (up to 6 epochs), and is tightly correlated with validation performance.
(WC2) Equinormness//quianggllln
shows some improvement with scale, training, and perfor- mance. Hyperspherical uniformity (GNC2) also improves but more clearly and consistently,
(WC3) Our models fail to achieve self-duality: class means do not align with classifiers. However, uniform duality (UNC3) is correlated with model width, training, and performance.
(WC,) Larger
trained models exhibit closer agreement between their linear and implicit NCC classifiers. Agreement is also associated with validation performance.
4.1 Within-Class Variability Collapse- Mollapse-ACh NC1
Scaling our models dramatically reduces normalized variance, which is further aided by more training epochs and stronger weight decay (Appendix Figs. 6 7 These noise reductions tightly associate with generalization (Fig. left, "WC,"". The relationship is most apparent at scale.
4.2
Geometric Structures - NC)
4
19:137
R²
0.86
R² 0.79
R² 0.84
16.389
og # Parameters
17.984
Loss
3
17/189
16.796
2
16.020
15.746
1
15.053
0.125 Log Mean Norms (CoV) (WC2)
0.075
0.100
4000 6000 8000 10000 0.25
0.50 0.75 1.00
Interference (CoV) (NC22) Log Distances (CoV) (G.CC2)
Figure 2:
Validation loss correlated with all three measurements: (left) equinormness (NC2) ex- pressed as variation in logarithmic norms; (centre) equiangularity (NC2) as variation in interference; (right) hyperspherical uniformity (GNC2) as variation in logarithmic pairwise distances.
Equinormness Logarithmic class mean grow with model width and training (Appendix Fig. [8) and subtly with depth (Appendix Fig. (9. Meanwhile, the variation of these (logarithmic) norms consistently decreases (improving equinormness) with scale (Appendix Figs. [0] IT). Both trends correlate with improved generalization (Appendix Fig. 20
In practice, we use an equivalent decomposition (Fq.
16 Note that agreement is equivalent accuracy.

---

Equiangularity Scaling model dimensions reduces average interference (Appendix Figs. 12
13
down an empirical plateau of approximately 10 10-3, which is more apparent less-trained However, the variation of interference rises and persists when scaling (Appendix Figs. 14
suggesting that interference becomes more concentrated between some pairs of classes. These results could be due to various factors, including but not limited to unfriendly conditions of language modelling ($ (.2) or the impossibility of a perfect simplex ETF ($35).
Appendix Figure 2 shows only rough performance correlation with average interference (i.e. coherence) and more definite - albeit still noisy - one with the variation of interference (i.e. equiangularity). In other words, the limited trends we observed toward simplex ETF suggest that the association of NC2 with generalization begins break down when C
Hyperspherical uniformity Logarithmic distances drop more gradually and consistently with scale (Appendix Figs. T6 17). implying this quantity is more robust or may not face the same geometric barriers seen in conventional interference (Appendix Figs. 14 15). Variation of these logarithmic distances is also consistently reduced with scale (Appendix Fig. 19)
And finally, generalization has much stronger correlations with logarithmic distances than it has with regular interference (Fig. 2) validating the applicability GNC [19] when C' > d + 1.
4.3 Classifier (Misjalignment and Duality (2))N33
Model width (d) is correlated faintly with the average similarity between class means and their respective classifiers (Appendix Fig. 23). but strongly with variation in similarity (Appendix Fig, 25).
The relationships to generalization follow the same pattern (Fig. 3 suggesting that uniform duality (UNC3) might serve better NC property than self-duality (NCa overall; discuss this in 5..
4
19.137
R² 0.31
R² 0.69
18.389
Log # Parameten
17.984
Loss
17.489
Val.
16.796
16.020
15.746
1
15.053
0.2
0.3
0.4
0.5
1.0
1.5
2.0
Similarity (Cosine) (Avg) (ACa)
Similarity (Cosine) (CoV) (UNCCs)
Figure 3: Validation loss shows negligible relationship with self-duality (WCa, left) and some correlation with uniform duality (UNCs, right). In other words, UNC₂ develops with scale and correlates with generalization much better than NCC.
4.4 Classifier Agreement-NCA.
The linear and NCC classifiers agree far more examples than random chance, and model scaling encourages this agreement (Appendix Figs. 29 30). Increasing width for certain depths happens to plateau even regress the agreement rate, but this limitation is overcome with further training. And finally, agreement is a strong indicator of generalization (Fig. right, NC.).
5 Analysis
We find that NC generally promoted by model size and training and correlated with generalization (validation performance). We also discer some of this correlation independent of scale ($5.1).

---

Table 1: Permutation test of NC measurements with respect to validation loss. Twenty-one (21) identical two-layer 768-wide models were trained with different data shuffling seeds and permuted with 10 trials. The p-values below 0.05 (bolded) show those properties to be statistically significant.
Property
Measurement
R Correlation (1)
p-value (L)
NC1
Within-Class Variability Collapse
0.192011
0.0485
NC2
Equinormness
0.026174
0.4870
NCC
Equiangularity
0.218574
0.0317
GNC₂
Hyperspherical Unitormity
0.487935
0.0002
NCa
Self-Duality
0.322210
0.0063
UNC3
Uniform Duality
0.000036
0.9784
NC4
Classifier Agreement
0.490278
0.0001
5.1 Neural Collapse's Relationship with Generalization
Table presents the correlation scores of NC metrics with generalization alongside their associated p-values from the permutation tests described in $3.8 The majority of the correlations are statistically significant (p 0.05) independent of scale, affirming that AC correlated with generalization.
5.2 Duality of Duality
The dichotomy of self-duality (NC3) and uniform duality (UNC3) is rather conspicuous. Our main experiments find that NC, does not consistently develop with scale while UNC3 does (Fig. [3) However, within fixed scale, the opposite is true, implying that may be confounded by model capacity while NC3 is a subtle and fine-grat fine-grained indicator of generalization.
5.3 The Effect of Weight Regularization
Our models trained with either weight decay factor exhibited very similar patterns in the emergence of NC (or lack thereof), but the more aggressive factor 3 = 0.1 resulted in stronger development of NC properties than with 3 0.0005 (Appendices) E G
M N.P. These findings empirically affirm / 0.1 weight decay as sensible for CLM pre-training 81, and coneur with [51)
on the pivotal role that appropriate regularization plays in the emergence of NC.
6 Limitations
Neural collapse While to the best of our knowledge, no previous work has studied realistic stochastic token prediction, it is possible that the quantities that we measure are not perfectly suited for ACC in language modelling. As we described in I1.I the NC framework does not translate neatly to the language modelling space due to many adverse conditions, so full convergence to ACC in the TPT was highly improbable. This paper leaves much room for future work better adapt ACC for next-token prediction, which we discuss further in Section 7
Language modelling Our work focused on autoregressive pre-training in its most basic form. We did not conduct experiments into encoder, multi-modal, or instruction-tuned models. Post-training techniques such as supervised fine-tuning, reinforcement learning with human feedback [91] or direct preference optimization [92] are also out-of-scope. This paper uses validation CE loss as the sole indicator of performance, leaving out any downstream task evaluations.
Confounder model scale The models that used in our permutation test ($5.1 Table are only of a single small architecture trained for one epoch with relatively weak weight regularization (3 = 0. 0.0005). Therefore, our experimental results on scale-independent links between NCC and generalization may not necessarily translate to larger models. Further investigation on (foundation) L.IMMs orders of magnitude larger than our CLMs trained with modern NLP methods would provide more robust insight into any direct correlations.

---

7 Discussion
Layer/depth-wise neural collapse evolve as a function of model depth
Past works have established that properties resembling NC
4
36
53
56,
60) 93/104] Layer-wise NC - sometimes and related phenomena at intermediate layers remain
[9
dubbed deep neural collapse (DNC) an interesting subtopic. We leave their study and induction in CLMs (like (105) as future work.
153 59
Learning to collapse Given the evidence for the development of NC and associated generalization under various loss functions [19 49 (60, [000] [06) in other domains, NLP researchers may still benefit from analyzing, adapting or training towards NC. As alluded to carlier, the simplex ETF and even the CE loss may not be truly optimal for this problem setting, so we anticipate future works to both construct more amenable geometries with better-suited objectives and then capitalize on their benefits downstream. As discussed in $2 there is an abundance of literature in NC, some of which could potentially adapt NC to be useful for NLP; we hope inspire more such investigations.
Interpretability At a high level, the number and density of clusters for a token can reflect its learned meanings and uses. This would be particularly useful as LLMS adapt to ever-evolving language and further expansion into non- non-English domains. Our formulae (Section 3) and results (Section 4
expose the pairwise token class interactions in noise, interference, classifier duality, and classifier agreement in the top-level features. Similarly to works in other domains 2 55 72 [09], these NC
metrics can serve a form of low-level interpretability to aid understanding certain behaviours of (L) Between tokens, one can often discern how related or interchangeable words are based on their pair wise interactions, or how antithetical unrelated they are based on orthogonality. For example, we present a rudimentary analysis of homonyms and English first names in Appendix Q
Fairness Foundation models are ubiquitous for their comprehensive capabilities and adaptability.
As previous work discussed class imbalance [35] 58 61 62), our work may extend these strategies to measure and perhaps promote fairness in foundation LLMs, some of which are designed to be multilingual or multicultural. For example, [T contemporarily explores the use of UNCC to
mitigate biases in BERT-base based (7] models.
While NC itself would not lead unfairness, its potential interpretability may, in theory, enable an (adversarial) agent to measure and optimize for (un)fairness as they manipulate an LLM.
LLM Evaluations Researchers in NLP and multimodal settings are ultimately interested in mea- suring model performance on practical tasks; notable benchmarks include GLUE (77). MMLU 11I, and BIG-bench (112). However, several contemporaries (1331115) have demonstrated that models' downstream capabilities are roughly correlated with their abilities to effectively compress their pre- training data. Based on their findings, our application of the NC framework to the pre-training stage of CLMs against validation CE loss should be an appropriate first step in this intersection. Looking forward, we anticipate exciting analysis for language modelling tasks or benchmarks, especially on creativity and retrieval for natural language understanding and generation.
Conversely, some form of ACC could be an alternative evaluation. Although it would be prohibitively expensive to measure NC on the vast and sometimes obscure pre-training data of most frontier production LLMs, doing
small of in-distribution data (i.e. test set) would be realistic.
8 Conclusion
In this paper, we apply the neural collapse (NC) framework to the next-token prediction problem. where models are undertrained and next-tokens variably drawn from numerous and imbalanced token classes. We leverage canonical and more recent metrics to demonstrate that ACC emerges as we scale the size and training of hundreds of causal language models. Our results show correlation between. NC and generalization, a relationship that persists even when the model scale is fixed.
In the short term, our work presents rudimentary techniques to analyze and interpret token-level properties of (L)LMs. We anticipate future work to suitably adapt NCC (and related frameworks) to the still-fresh frontier of autoregressive language modelling. Researchers could then effectively capitalize on previous learnings from NCC to better understand and improve the pre/post-training processes of increasingly complex and large language (multimodal) models.
10

---

Acknowledgements
We thank Ellioo Creager, David Glukhov, Daniel Johnson Jivan Waber and Colin Raffel for their helpful feedback and stimulating discussions. Aditya Mehrotra and Yu Bo Gao provided technical assistance in our implementations. We acknowledge the support of the Natural Sciences and Engineer- ing Research Council (NSERC) of Canada (www www.r-eee---oo- maser--cssng ge ca/). This research was enabled in part by resources from Calcul Québec (www. www.eulleeeee.ccccmomom..mmm.. ca), the Digital Research Alliance of Canada (www. all alliancecan. ca), and the Vector Institute (www. vectorinstitute ai
References
[1] Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40) 24663, sep 2020. doi:
URL https: //www//wwwwww/ pnas
_RLhttps
org/ arg/doi/ful111..1 1073 http://ww....//pssa.//a..o
pnas http:/pps...025511111111 2015509117
[2] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL. httpptt/aaaaaii..ottt. org/ abs/ htt///bbb/20000000 20...00007777 07759
[3] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning, 2022. URL htt.:hpss//aa....oooaabaa/////// 15121v2
[4] Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks, 2022. URL Uttpttss://axxx org/ abs 22000000882
[5] Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:// openreview. wtw.:/neeoooooooddttoooppppppppe
[6] Emily M. Bender, Timnit Gebru, Angelina MeMillan-Major, and Shmargaret Shmitchell, On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021
ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 4503833097 doi: 10.1145/3442 010.:3444218834449922 URL htttps://doo org/ 10. 1145/ 3445922
[7] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://aaxiv org/ htggaab//8810.044
[8] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
[9] Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number classes, 2023. Utthttps://aaxi org/
abs/ /2300 05351
[10] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank mnn language model, 2018. URL httl:t://////iiv.ortt org.abs...rg................... 1711 03953
[11] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379 423, 1948. doi: 15380/.1538----11 7884-3355.144 1948.b011338...
[12] P Sargant Florence. Human behaviour and the principle of least effort. The Economic Journal, 60(240):808-81) 1950.
[13] Daniel Jurafsky and James H Martin. Speech and language processing: An introduction natural language processing, computational linguistics, and speech recognition, 2009.
[14] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: //tp///a/vvv//////////eaa//aaaaa 0
11

---

[15] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https //arxiv.org/abbll..... 200bb//222211555661
[16] Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023. /tRL/tttss////xiiv..t
16264v4
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[18] Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss, 2021. URL https://arxis: wwwp:/r//////////eeeeo 08465v2
[19] Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. Generalizing and decou- pling neural collapse via hyperspherical uniformity gap, 2023. (Rtpss//://aa org/
1203. 06484v2
[20] Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5319-5329, 2023.
[21] Vignesh Kothapalli, Tom Tirer, and Joan Bruma. A neural collapse perspective on feature evolution in graph neural networks, Advances in Neural Information Processing Systems, 36, 2024.
[22] Yibo Yang. Haobo Yuan, Xiangtai Li, Jianlong Wu, Lefei Zhang, Zhouchen Lin, Philip Torr, Dacheng Tao, and Bernard Ghanem. Neural collapse terminus: A unified solution for class incremental learning and its variants, 2023. URL https://arxiv org/ /b08 01746v1
[23] Qinhao Zhou, Xiang Xiang, and Jing Ma. Hierarchical task-incremental learning with feature- space initialization inspired by neural collapse. Neural Processing Letters, 55(8): 1081 1-10827, 2023.
[24] Hang Ran, Weijun Li, Lusi Li, Songsong Tian, Xin Ning. and Prayag Tiwari. Learning optimal inter-class margin adaptively for few-shot class-incremental learning via neural collapse- based meta-learning. Information Processing & Management, 61)) 2024. ISSN 0306-4573, doi: http:///w...rr///0000o1//....mmp scciaccc/Arrtccee////////00/20/0
URL fttpp:://www.cceeceediiee com/
[25] Saaketh Medepalli and Naren Doraiswamy. On the role of neural collapse in meta learning models for few-shot learning, 2023. URL /ttp:t/ss:/: wtt.aaawwwwa.aaaaaaaaaaaaa
00451v2
[26] Jarrod Haas, William Yolland, and Bernhard Rabus. Linking neural collapse and 12 normal- ization with improved cut-of-distribution detection in deep neural networks, 2023. URL actps://arxive org/ abs /209 0
[27] Mouin Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi. Neco: Neural collapse based distr--iisttibuttinn detection, 2024. URL https
//arxiv org/ atte///ss//31....eee 06823
[28] Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. Epa: Neural collapse inspired robust -istt--distiiuutinn detector, 2024. URL ULL.:trpee//aarir.ro////////0000
[29] Donghao Li, Yang Cao, and Yuan Yao. Neuromixgdp: A neural collapse-inspired random mixup for private data release. In Conference on Parsimony and Learning, pages 480-514 514. PMLR, 2024.
12

---

[30] Chendi Wang, Yuqing Zhu, Weijie J. Su, and Yu- Yu-Xiang Wang. Neural collapse meets differ- ential privacy: Curious behaviors of noisygd with near-perfect representation learning, 2024.
URL httph:tttt/rrrteeeee httpa::/aaxvvv..rre/..
org/ / abs// 2405. 2
[31] Tolga Ergen, Arda Sahiner, Batu Ozturkler, John Pauly, Morteza Mardani, and Mert Pilanci. Demystifying batch normalization in relu networks: Equivalent convex optimization models and implicit regularization, 2022. URL https:/t
org/ htt////ss///./.../// 2103 01499v3
[32] Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 3004-3014, PMLR, 18-24 Jul 2021. URL https::/proceeddinn mlr press/ #139/ergen201
[33] Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung-Hsu Chou. Neural (tangent kernel) collapse. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informa- tion Processing Systems, volume 36, pages 16240-16270. Curran Associates, Inc.,
2023. URL http:t::////rreeedns....mo neurips cttppppp.ccppppppeeeeeeee Saapperrfilsspppaaee ftllss/paperr2222/1122 3477 34FTaa0eessaaaaaalclc.66lll2eee3
wwwnceenweeoecooooooocooooooeeee pdf'
[34] Matus Telgarsky. Feature selection with gradient descent on two-layer networks in low-rotation regimes, 2022. URL
[35] Wanli Hong and Shuyang Ling, Neural collapse for unconstrained feature model under cross- entropy loss with imbalanced data, 2023. URL
09725v2
[36] Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, and Aukosh Jagannath. Higg-mimssnniih---mmnnsssnnn sgd aligns with emerging outlier eigenspaces, 2023. URL /RRL:ittsss//rriiioo///////////
03010v1
[37] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collac for understand- undderssad ing generalization in deep learning, 2022.
for
[38] Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop Pre-training: Perspectives, Pirfalls, and Paths Forward at ICML 2022, 2022. URL VrK/pKwOhT_
[39] Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre- trained models are from neural collapse on the target dataset informs their transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5549-5555 2023.
[40] Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu Understanding and improving transfer learning of deep models via neural collapse. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL https://
openreview www.
[41] Didi Zhu, Zexi Li, Min Zhang, Junkun Yuan, Yunfeng Shao, Jiashuo Liu, Kun Kuang, Yinchuan Li, and Chao Wu. Understanding prompt tuning for v-l models through the lens of neural collapse, 2023. htLps:t/::
org/ abs 2306. 15955v3
[42] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. https://:///ww cs foronto.edu. wttp:eetwe.e.eeeeaaaaaaaaaetrrre 2009--R pdf
[43] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[44] Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features, 2020. URL httpt:t////aaat httggabb//011..11mm
13

---

[45] Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network clas- sifiers trained with the square loss, 2020. URL httpss:/arxii, http://wwww....rra/////0../...00 00072v
[46] Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers, 2021. URL /RLLpttss://aaxxii.ooo abs/ 05420v3
[47] Cong Fang, Hangfeng He. Qi Long, and Weijie Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(4))e210300111 2021.
[48] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu
A geometric analysis of neural collapse with unconstrained features, 2021. URL https
// aaaxiv.orr////....v............r wtt.oorg//bb///so/// 2ttp///5555000055111
[49] X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2022. URL httppps:///peeeeeeiiee http://wwwwieeeeeeeeeeeeeeeeeeee http///orummiwwwwwud///aa/a
[50] Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35:1 1547-11560, 2022.
[51] Akshay Rangamani and Andrzej Banburski-Fahey, Neural collapse in deep homogeneous classifiers and the role of weight decay. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 4243 4247, 2022. doi: 10.1 109/
SCASSP43922.2022 202.9.77777.....7.
[52] Yibo Yang. Shixiang Chen, Xiangtai Li. Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do really need learnable classifier the end of deep neural network?, 2022. URL https://arxivv org/ abs///bs//////////////////////// 09081v3
[53] Tom Tirer and Joan Bruna Extended unconstrained features model for exploring deep neural collapse. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21478-21508 PMLR, 17-23 Jul 2022. URL
mlr. wwtps//eveettieeeeeeeeeeeeeeeeee
html
[54] Peng Wang. Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu Linear convergence analysis of neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
[55] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective, 2022. URL /ttts://t/:/ttii/ org/ abs/ 2210. 0219922
[56] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28729-28745, PMLR, 23-29 Jul 2023. URL httpp:::/ppocceednggg mlr preess/v202/raagaammi2222 html
[57] Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning, pages 34301-34329, PMLR, 2023.
[58] Hien Dang, Tho Tran, Stanley Osher, Hung Tran-The, Nhat Ho, and Tan Nguyen. Neural collapse in deep linear networks: From balanced to imbalanced data, 2023. URL https //arxiv mraxivvoor//ann wttggabs//33/11110 00437v5
[59] Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. Advances Neural Information Processing Systems, 36, 2024.
14

---

[60] Due Anh Nguyen. Ron Levie, Julian Lienen, Eyke Hüllermeier, and Gitta Kutyniok. Memorization-dilann dilation: Modeling neural collapse under noise. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URL. https::/ net/
f forum2Ad=c..wxqmmDD.2
[61] Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jía. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19550-19560, 2023.
[62] Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neu- ral collapse deep long-tailed learning. International Conference on Artificial Intelligence and Statisticcs, pages 1 11534-12 1154.. PMLR, 2023.
[63] Peifeng Gao, Qiangian Xu, Peisong Wen, Huiyang Shao, Zhiyong Yang, and Qingming Huang. A study of neural collapse phenomenon: Grassmannian frame, symmetry and generalization, 2023. URL https:://///iiii..ooomm org
08914v2
[64] Mufan Bill Li. Mihai Nica, and Daniel M. Roy. The neural covariance sde: Shaped infinite depth-and-width networks at initialization, 2023, URL httpp://arxiv..oo http:://aaxvvvvrrrreee/ wwwgg/bs/205
02768v3
[65] Zhanxuan Hu, Yichen Wang. Hailong Ning, Yonghang Tai, and Feiping Nic. Neural collapse inspired semi-supervised learning with fixed classifier. Information Sciences, 667:1 20469, 2024.
[66] Gao Peifeng, Qianqian Xu, Yibo Yang, Peisong Wen, Huiyang Shao, Zhiyong Yang. Bernard Ghanem, and Qingming Huang. Towards demystifying the generalization behaviors when neural collapse emerges, 2024. URI htt.htttpp::/ppereeiiee www..neeooooummdd///vssaree
[67] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds. Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. URI. https //transformer- circuits. pubp 2tt///222///// wwtoooooooelliieeeeee heeeel/indxx... http//dddeee.mlll
[68] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
[69] A.
Dubey et al. (101 additional authors). The Ilama 3 herd of models. 2024. URL https /arxiiv..a Seexiv.oog//bbee wrgg/bs////// 21783 All authors were affiliated with Meta.
[70] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbal- ance trouble: Revisiting coural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225-27233 2022.
[71] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
[72] Quinn Fisher, Haoming Meng, and Vardan Papyan. Pushing boundaries: Mixup's influence on neural collapse, 2024. URL httpp:pss//..xx...o///bbbs////e/
[73] Pengyu Li, Xiao Li, Yutong Wang, and Qing Qu. Neural collapse in labti learning with pick-all-label loss, 2024. httptttts//: org/
15903v4
[74] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[75] David Mimno and Laure Thompson. The strange geometry of skip gram with negative sampling. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference Empiricul Methods in Natural Language Processing, pages 2873-2878, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1/D11863////DI1-13300
15

---

[76] Kawin Ethayarajh. How contextual are contextualized word representations? compara the geometry of bert, elmo, and gpt-2 embeddings. 2019. URL /tL.ttts///axxi..oorr/////////// 00512
[77] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. 2019.
URL https: /ttps: org/ abs/ 1804. 07461v3
[78] Jia Hui Feng, Edmund M-K Lai, and Weihua Li. A study of neural collapse for text classifica- tion. In International Conference on Deep Learning Theory and Applications, pages 126-142. Springer, 2023.
[79] Thomas Laurent, James H. von Brecht, and Xavier Bresson. Feature collapse, 2023. URL 16162v1
/ttpp://arxiv..rrg///aa
[80] Sid Black, Leo Gao, Phil Wang. Connor Leaby, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorfine March 2021. URL https //doi. 10. 22tp///ooo.. 2tt11/////2eooooo02775555 If you use this software, please cite it using these metadata.
[81] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[82] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https 01068v4
//arxiv. org/
[83] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https:///xxiiii org hwtp//as//20 2302. 13971v1
[84] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock. Teven Le Scao, Thibaut Lavril. Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv. www.//gg//////o..ee//..o/...eee/ 0310.0082200
[85] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, (12:1199-88, 1936.
[86] C Radhakrishna Rao. The utilization of multiple measurements in problems of biological classification. Journal of the Royal Statistical Society. Series B (Methodological), 10(2): 159- 259, 1948.
[87] Thomas Strohmer and Robert W Heath Jr. Grassmannian frames with applications to coding and communication. Applied and computational harmonic analysis, 14(3):257-22 2003.
[88] Shayne FD Waldron. An introduction to fanite tight frames. Springer, 2018
[89] David L Donoho, Michael Elad, and Vladimir N Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Transactions on information theory, 52(1))6-18, 2005.
[90] Joel A Tropp. Just relax: Convex programming methods for identifying sparse signals noise. IEEE transactions on information theory, 10(3):1030- 2006.
[91] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. htLhsttp::1 http:///axvvvvoo www. 0ttps///999.085800
16

---

[92] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 23728-53744 Curran Associates, Inc., 2023. URL httpbtttps://procceedngg neurnngg...nuriii neur ips cc/paper_.... files/ / 4022:711164666006666666666666662 Ceraper--oofeeenceee pdf
[93] Vardan Papyan. Traces of class/rosss- structure pervade deep learning spectra, 2020. URL http://aarxiv.orgaaaa//saabaa///
[94] Christopher R. Hoyt and Art Owen. Probing neural networks with t-sne, class-specific projections and guided tour, 2021. (tRps:t:s://// //axipps///rxvv.... /ttp/vxxvvv..///i///ee/////aavva www. arrg/bbs/2107 /bbb///0...257711
[95] John Zarka, Florentin Guth, and Stéphane Mallat. Separation and concentration in deep networks, 2021. URL https: www.av..rrgabb////1/0/ 10424v2
[96] Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Alexander Cloninger, Timothy Doster, Tegan Emerson, Manohar Kaul, Ira Ktena, Henry Kvinge, Nina Miolane, Bastian Rieck, Sarah Tymochko, and Guy Wolf, editors, Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022, volume 196 of Proceedings of Machine Learning Research, pages 37-47, PMLR, 25 Feb-22 Jul 2022. URL https //proceedings..o .aaaangsmlle mlr wwwpppprssssv.eeee ben- shaul 22a. html
[97] Hangfeng He and Weijie Su, A law of data separation deep learning, 2022. URL //axxiv brrrxivv.orraae abs/ 2210. 17020v2
[98] Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks, 2023. URL/tttss//arxxv http::////xavvooog..///// hatg//bb/2000
02760v1
[99] Wojeiech Masarezyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Mit oš, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 76772-76805. Curran Associates, Inc., 2023. URL http:: http:t:sppppppooeeieeneeeeeee//e httpp//eeeeeeeeeereppepoop.ppppp cctppppr.ppppppppceeeeeeeeee p02///110/22000233355550001f1112 Papeao---aaaee--eaeeeeeeeee
[100] Daniel Beaglehole, Peter Sūkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse, 2024. URL
org/
abs bba/2202.115 13728v2
[101] Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Under- standing deep representation learning via layerwise feature compression and discrimination, 2024. URL
2311. 02960v2
[102] Sicong Wang, Kuo Gai, and Shihua Zhang. Progressive feedforward collapse of resnet training, 2024. URL
org absp//bs////0//// 00985v1
[103] Connall Garrod and Jonathan P. Keating. Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model, 2024. URL https://arxiv org/ www.a///44.. 06106v1
[104] Emanuele Zangrando, Piero Deidda, Simone Brugiapaglia, Nicola Gugliclmi, and Francesco Tudisco. Neural rank collapse: Weight decay and small within-class variability yicld low-rank bias, 2024. URL httLhhtps::/ : //axiiv.////iiv///aa org/ abs/ 03991v1
[105] Jiachen Jiang, Jinxin Zhou, and Zhihui Zhu. On layer-wise representation similarity: Appli- cation for multi-exit models with a single classifier, 2024. /RtL/tttss///rxiv.ooo
2406. 14479
[106] Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. Research, 6:0024, 2023.
17

---

[107] Tong Liang and Jim Davis. Inducing neural collapse to fixed hierarchy- awwar frame for reducing mistake severity. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1443-1452, 2023.
[108] Guglielmo Bonifazi, lason Chalas, Gian Hess, and Jakub Lucki. Can we understand plasticity through neural collapse?, 2024. URL UR/.happss//rrttt..r/////// wttp://wwr..abb//...00222
[109] Li Guo, Keith Ross, Zifan Zhao, George Andriopoulos, Shuyang Ling, Yufeng Xu, and Zixuan Dong. Cross entropy versus label smoothing: A neural collapse perspective, 2024. URL
attps://arxive http://aaxxvv..oab///s/ess wwwg/ba/////0/00 ottp/a2aa//2000003333332222222 03979v2
[110] Jingxuan Xu, Wuyang Chen, Linyi Li, Yao Zhao, and Yunchao Wei. Collapsed language models promote fairness, 2024. URL https: https://arxiv.. www.aawwwwi......aaaaoaaaa wttpa//..rggbb//////// httpabb//241..0000 04472
[111] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representationss (ICLR), 2021.
[112] BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
/ wtt.:/iiew..tttttoommmoddaalaaad id-uyTL5Bvos]
[113] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective, 2024. URL Btt.Lattts.//axxx.oor//aaaa 15796
[114] Yuzhen Huang, Jinghan Zhang. Zifei Shan, and Junxian He. Compression represents intelligence linearly. In First Conference on Language Modeling, 2024. URL https
//openreview.o wttpleewweeeeeeeeeeeeeeeeeeeeeee
[115] Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. Entropy law: The story behind data compression and Ilm performance, 2024.
[116] Stephen Merity, llya Sutskever, Simon Korblithh and Nikhil Goyal. Pointer sentinel mixture models. arXiv preprint arXiv: 2016.
[117] Yao Zhu, Zhen Yu, Chao Zhang, Yijia Wu, et al. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.0588 2015.
[118] Common Crawl. Common crawl, 2023. URL. https://comoncrawl.. org 2023-10-30.
Accessed:
[119] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset diverse text for language modeling. 2020. URL https //arxiv..m org abs /atp///1....00000
[120] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2024.
http:tttss:/aaxiv.oorr wwt.://gg..........////// 17493
[121] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024. URL https://aaxx: org/abs/2404.0 01413
[122] Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. Bfloat16 processing for neural networks. In 2019 JEEE 26th Symposium Computer Arithmetic (ARITH), pages 88-91, 2019. doi: 10.1109/ARITH.211
18

---

A Dataset
TinyStories is dataset of short children's stories generated by GPT-3.5 and GPT-4 (2), released with the POLA-Sharing- 1.0 licence. We trained and evaluated models on their first version, as described:
The 2,141,709 stories are split into 2,119,719 train and 21,990 validation stories.
Their experimental setup [2] called for the GPT-2 [68] tokenizer, of which only subset vocabulary V [1,, 29233] appears in TinyStories.
Following the GPT-style ceacher-forcing regime for training/evaluation g raw sequences (stories) from the train set are packed (by two preprocessing workers) into 229,367 (S) chunks of 2048 (7) tokens each. This setup provides 469,514,245 (N) ground-ruut 17 token samples for training.
Samples (log scale)
10?
106
105
0
100 Top 500 (1.71%) Most Frequent Tokens in TinyStories
200
300
400
500
Figure 4: The 500 most frequent classes from TinyStories 2 exhibit significant sample imbalance. Despite the synthetic nature of TinyStories, such distribution typical of natural language [11 12]
A.1 Alternative (Real) Datasets
The study of NC in causal language modelling at the token level would be unreasonably expensive, the motivation WikiText [116). BookCorpus [17), CommonCrawi [188), or most subsets from the Pile [119] are much complex and broad to be effectively compressed by CLMs of the scale that we work with.
small dataset is clear. However, most commonly used text datasets such as
WikiText-2 and WikiText- 103 present significant drawbacks for our experiments. Both datasets contain considerable amount of low-quality data that does not concentrate essential linguistic structures such as grammar, vocabulary, facts, and reasoning. WikiText-2 has a similar empirical vocabulary to TinyStories under the GPT-Neo [80] tokenizer (27K vs. 29K) but only has around 34K rows of training data compared to 2.1M in TinyStories. Our small-scale NC experiment on WikiText-2 revealed that the models were very brittle and prone overfitting. On the other hand, WikiText- 103 is comparably sized to TinyStories but utilizes around 44K unique tokens. Our CLMs trained Wiki'Text- 103 struggled produce coherent sentences, likely due the excessive breadth and information, as noted by the authors of TinyStories. Beyond these two, we were unable to find any real datasets that both followed established scaling laws [14] [5 for CLMs our scale and simple enough to suit the analysis of NC.
A.2
On the Use of TinyStories
According to its authors, TinyStories 2 is explicitly designed to preserve the essential elements of natural language, such as grammar, vocabulary, facts, and reasoning, while being smaller and more refined in terms of its breadth and diversity. Unlike large corpora that overwhelm small language models (SLMs) due to their excessive breadth and diversity, Tiny Stories offers a concentrated dataset that hones in vocabulary, consisting of approximately 1500 words that a child would use, and in its 29K empirical vocabulary under the GPT-Neo tokenizer.
linguistic structures and reasoning capabilities. This is evident in its small
17 N S(T cannot the first ground -ouuth the last predicted token in any chunk.
19

---

Despite its concentrated nature, TinyStories enables models trained on it to produce grammatically correct, factual, and reasonable stories. Additionally, these models can be finetuned on specific instructions found in the IinyStories-Instruct dataset. The authors of TinyStories also demonstrate that their models can creatively produce stories dissimilar enough to their training data, indicating a balanced capability for generalization and creativity.
One particular advantage of TinyStories is the small vocabulary relative to total training tokens, rendering a reasonable number of classes with higher average token counts. This is relevant because the possibility of NC and CLM's 's ability to compress language data into distinct geometries depend partially on the ratios between embedding dimension, vocabulary size, and average token frequency. Conveniently, frequency analysis of the overall dataset produced a distribution (Figure ( similar to real human language, so TinyStories should provide a good balance for an initial study of this NC.
Additionally; TinyStories has more regular structure as GPT-3.5/4 was instructed produce children's 's stories with certain themes and forms with a conservative vocabulary. We believe this would reduce the amount of clustering noise from the breadth of information and structures in real general data, and allow our smaller CLMs to exhibit some clear trends toward NC.
Furthermore, TinyStories was created using GPT 3. 3.5/4, advanced language models with significantly larger architectures trained on orders of magnitude more tokens; this should help minimize the effect of the synthetic nature of the generated dataset. We also considered a possible effect of model collapse as a result of training on synthetic data [120] and follow-up work [12]] suggest that a single iteration of data generation (as generated TinyStories) has very negligible model collapse.
B Model Architectural Details
Table 2: Sample architectural configuration for a 12-layer 1024-dimensional causal language model (CLM) based 2 and GPT-Neo [80]. Shallower models have configurations with attention_ _o__ere and attention_ types truncated. Narrower models adjust hidden_size to their width (d). All other configuration values the same across models.
SETTING
VALUE
activation_ ourutttionn_unctinnn
function
gelu new
architectures
attention, _ropout
attention_
global, local, global, local, [[global, local], 6]
attention_tyye typess
bos_token_ire id
50256
embed_dropout
eos_token_idd id
50256
false
hidden_size
1024
initializer_rang
0.02
intermediate_size
null
layer_noe bhyyer_norm_esillon
1e-05
max__position_emmdddine
2048
model_type
gpt_neo
hum_headee
16
num num_layere layers
resid dropout
summary activatiionn
Sinmary_activation
null
funmaary_iirst_dropoo
0.1
cammaryyproj_tt__abee
true
summary_type
cls_index
cimmary_use_prog
true
torch_dtypp
float32
transformers_
4.28.1
use__ache
true
bocab_size
50257
window_aize
256
20

---

C
Optimization
The training was performed using an adaptation of an open-source causal language modelling script from Huggingface:
http:///wwwo.aoaaaaaaaa///aaaaaa clm.
Each model was trained single NVIDIA A100 (40GB) GPU for up hours per epoch.
Learning rates set by linear schedule based the number of steps with warm-up.
Training was performed in bfloat16 [12 mixed precision.
The results presented in this work are from two sets of models trained with weight decay 3 0.0005 [51] and / 0.1 [8]] A previous set of models was trained without weight decay and the results are very similar to 3 = 0.0005.
Table 3: Batch sizes used to train models single NVIDIA A100 (40GB) GPU. Width (d) and
depth (L) correspond to hidden_sizee size and length of attent i on layers, respectively, in Table
DEPTH (
WIDTH (d) 7
128
256
512
768
1024
1-layer
16
16
16
16
16
16
2-layer
16
16
16
16
16
8
4-layer
8
8
8-layer
8
8
8
4
4
12-layer
4
4
4
4
Log Normalized Variance (NC1)
4.0
19.137
-2
18.389
3.5
Log # Parameters
Validation Loss
17.984
3.0
-4
17.489
2.5
16.796
-6
2.0
16.020
1.5
15.746
-8
1.0
15.053
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
Training Epochs
Training Epochs
Figure 5: Average (logarithmic)) class-distance normalized variance (CDNV.NCi) (left) and valida- tion (cross-entropy) loss (right) with respect to training epochs.
D Embeddings Collection & NC Analysis
Codes for (post-)training analysis are hosted on GitHub:
Main code http:////tttaaacc.//aaaaaaaaaaat
Auxillary package: http://///thhh..ooooo/aaaaarrrrr
One pass over the train set for embeddings collection can take up to hours on single NVIDIA 1100 (40GB) GPU. Analysis of single metric for given model takes less than 5 minutes.
21

---

E Within-Class Variability Collapse with Scale - NC
Log Normalized CDNV
-2
5
(=1
[=2
-4-
L=4
L=8
-6
-7
[=12
-8
-8
64 256 512 768 1024
64 256
512 Width (Hidden Dim.) Epochs
768 1024
64 256
512 768 1024
Width (Hidden Dim.) Epochs
Width (Hidden Dim.)
10 Epochs
Log Normalized CDNV
5
L-1
L=2
-6-
L=4
-7
-7
L=8
L-12
-6
-8
-8
256 512 768 1024
256 5i2 768 1024 Width (Hidden Dim.) 6 Epochs
256 512 768 1024
Width (Hidden Dim.) Epochs
Width (Hidden Dim.) 10 Epochs
Figure 6: Average (logarithmic) diss-dist normalized variance (CDNV) is reduced (WC1) when scaling width (d) and across training for (left) through 10 (right) epochs with weight decays 3 0.0005 (top) and B = 0.1 (bottom).
Log Normalized CDNV
-2
d=64
d=128
-4
d=256
d=512
-7
d=768
d=1024
-8
2 4
12
2
4
8
12
2
4
12
Depth (Num. Layers) 1 Epochs
Depth (Num. of Layers) Epochs
Depth (Num. Layers)
10 Epochs
Log Normalized CDNV
-5
-2
d=
d28
-6
d=256
d=512
d=768
-8
-8
d= 1024
12 4
1
12
2 4
12
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) 10 Epochs
Figure 7: Average (logarithmic) class-distance normalized variance (CDNV) is reduced (NC,) when scaling depth (L) and across training for (left) through 10 (right) epochs with weight decays 3 = 0.0005 (top) and 3 = 0.1 (bottom).
22

---

Mean Norms Growth with Scale - (Related to 2VC2)
Log Mean Norms (Avg)
3.75
3.5
--- L=1
3.50
[=2
3.0
L=4
3.25
+ L=8
2.5
3.00
L=12
2.5
2.0
2.75
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024
64
256 512 768 1024
Width (Hidden Dim.) 3 Epochs
Width (Hidden Dim.) Epochs
Log Mean Norms (Avg)
4.00
(=1
3.75
L=2
3.0
3.5
+-- L=4
[=8
2.5
3.25
L=12
3.0
3.00
2.0
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024 Width (Hidden Dim.) Epochs
Figure 8: Logarithmic class mean norms grow when scaling width (d) and across training for (left)
through 10 (right) epochs with weight decays 3 0.0005 (top) and 0.1 (bottom).
Log Moan Norms (Avg)
3.75
3.5
d=
3.50
d=128
d=256
3.25
d=512
3.00
2.5
d=1024
2.0
2.75
12
4
12
12
4
8
12
12
8
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
Epochs
Log Log Mean Norms (Avg)
4.00
d=64
3.75
d=128
3.0
3.5
d=256
3.50
d1512
2.5
3.25
d 768
d=1024
4
12
12
Depth (Num. Layers) Epochs
Depth (Num. Layers)
Depth (Num. of Layers)
Epochs
10 Epochs
Figure 9: Logarithmic class mean norms grow when scaling depth (L) and across training for (left)
through 10 (right) epochs with weight decays 3 = 0.0005 (top) and / = 0.1 (bottom).
23

---

G Equinormness with Scale - (9)NC2
Log Mean Norms (CoV)
1e-1
1e-1
1.0
-
L=2
L
L
L=12
0.6
64 256 512 768 1024
64 512 768 1024
64
512 768 1024
Width (Hidden Dim.) Epochs
Width (Hidden Dim.) 3 Epochs
Width (Hidden Dim.) 10 Epuchs
Log Moan Norms (CoV)
1e-1
le-1
1e-2
+ L=1
1.2
L=2
L=1
1.0
+ L=8
L-12
0.6
6-
256 512 768 1024 Width (llidden Dim.) 1 Epochs
64 256 512 768 1024 Width (Hidden Dim.) 6 Epochs
64 256 512 768 1024
Width (Iiidden Dim.)
10 Epochs
Variation in (logarithmic) norms decreases (NC2) when scaling width (d) in models trained for 1 (left) through 10 (right) with weight decays B = 0.0005 (top) and B = 0.1 (bottom). Note that the degree of equinormness eventually plateaus for sufficiently deep and trained models.
Figure 10:
Log Moan Norms (CoV)
1e-1
1e-1
d=64
d=128
d=256
1.0
d-512
0.8
0.8
768
d=1024
0.6
1
4
2
12
2
4
12
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
10 Epochs
Log Mean Norms (CoV)
1.0
d=64
1.2
d=128
0.8
d=256
d=512
d=768
0.6
d=1024
12
12
Depth (Num. of Layers)
Depth (Num. Layers) Epochs
Depth (Num. of Layers)
1 Epochs
10 Epochs
Figure 11:
Variation in (logarithmic) norms decreases (NC₂) when scaling depth (L) in models
trained for (left) through 10 (right) with weight decays 3 = 0.0005 (top) and B = 0.1 (bottom).
24

---

H
Interference with Scale - (Related to NC2)
1e-2
1e-2
1e-2
Interference (Avg)
L=1
L=2
L-4
L-8
[=12
33
-4
64 256 512 768 1024
64 256 512 768 1024
64 256 512 768 1024
Width (Hidden Dim.)
Width (Hidden Dim.) 3 Epochs
Width (Hidden Dim.) 10 Epochs
1 Epochs
le 2
1c
Interference (Avg)
00.55
-1-
L=2
-0.50
L=4
-2
L-8
-0.5
L-12
-3-
-1.0
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024 Width (Hidden Dim.) 6 Epochs
64 256 512 768 1024 Width (Hidden Dim.) 10 Epochs
Figure 12: Average interference decreases (to some extent) when scaling width (d) in models trained for 1 (left) through 10 (right) with weight decays 0.0005 (top) and 3 0.1 (bottom).
1e-2
1e-2
1e-2
Intorference (Avg)
d=64
-1-
d=128
d=256
-2
d-512
768
-3
4
-6
d=1024
12
4
1 2
12
12 4
Depth (Num. of Layers)
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
Epochs
10 Epochs
1e-1
1e-1
Interference (Arg)
d=64
0.25
d288
-0.5
0.50
d=256
d=512
0.75-
d=768
-1.0
d-1024
11.0
12
i2
1
Depth (Num. of Layers)
Depth (Num. of Layers)
Depth (Num. Layers)
1 Epochs
6 Epochs
10 Epochs
Figure 13: Average interference decreases (to some extent) when scaling depth (L) in models trained for (left) through 10 (right) with weight decays 3 = 0.0005 (top) and B = 0.1 (bottom).
25

---

I Equiangularity with Scale
(Against NC22)
1e4
1e3
1e3
Interference (CoV)
1.0
7
L-1
8
L-2
0.8
6-
L=4
6
1=8
0.6
5
L=12
4
4-
64
256
512
768 1024
64 256
512
768 1024
64 256
512 768 1024
Width (Hidden Dim.)
Width (Hidden Dim.)
Width (Hidden Dim.) Epochs
Epochs
3 Epochs
1e3
1e3
Interference (CoV)
7
L=1
L=2
L-4
5-
L
0.6
L=12
256
768 1024
64 256
1024 Width (Hidden Dim.) 6 Epochs
512
256 512 768 1024
Width (Hidden Dim.) Epuchs
Width (Hidden Dim.) 10 Epochs
Figure 14: Variation in interference roughly increases when scaling width (d) in models trained for 1 (left) through 10 (right) with weight decays 3 = 0.0005 (top) and 3 0.1 (bottom). Note this trend is against equiangularity, affirming the traditional NC2 be less useful than GNC2 [T]]
1e4
1e3
1e3
Interference (CoV)
1.0
7
d=64
d-128
0.8
6
d-256
d=512
0.6
5
d=768
4
-.-
d=1024
2
8
12
2
12
2
4
8
12
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
Epochs
Interference (CoV)
d=64
d=128
0.8
d=256
d=512
0.6
d=768
d= 1024
1
1
2
Depth (Num. Layers)
Depth (Num. of Layers)
Depth (Num. of Layers)
Epochs
Epochs
10 Epochs
Figure 15: Variation in interference increases when scaling depth (T.) in models trained for (left)
through 10 (right) with weight decays / 0.0005 (top) and = 0.1 (bottom). Note this trend is against cquiangularity, affirming the traditional /CCp to be less useful than GNC, [19].
26

---

Logarithmic Distances with Scale - (Related to GNC22
10
-3.0 1e-1
1e-1
Log Distances (Arg)
3.2
L-1
-3.0
L=2
-- L=4
-3.2
-3.3
L=8
-3.2
L-12
-3.4
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024 Width (Hidden Dim.) 3 Epochs
64 256 512 768 1024
Width (Hidden Dim.) Epochs
1e-1
1e-1
1e-1
Log Distances (Avg)
3.2
L=1
3.0
-.- L
-3.3
L
-3.3
L=8
-3.2
L=12
-3.4
-3.4
-3.4-
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024 Width (Hidden Dim.) 6 Epochs
256 512 768 1024
Width (Hidden Dim.) 10 Epochs
Figure 16: Average logarithmic distance decreases when scaling width (d) in models trained for 1 (left) through 10 (right) with weight decays 3 0.0005 (top) and 0.1 (bottom).
1e-1
Log Distances (Avg)
-3.0
332
d=64
-3.0
---
d-128
dm256
-3.3
d=512
-3.2
d=768
d=1024
334
3.4
4
2
4
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
Depth (Num. of Layers)
Epochs
10 Epochs
18-
P-1
le-1
Log Distances (Avg)
332
-3.2
d=64
-3.0
d=128
33.
d=256
-3.3
d=512
-3.2
d 768
3.4
-3.4
d=1024
-3.4
12
4
8
12
2
4
8
12
4
8
Depth (Num. Layers) Epochs
Dupth (Num. Layers)
Depth (Num. Layers) 10 Epochs
Epochs
Figure 17: Average logarithmic distance decreases when scaling depth (L) in models trained for (left) through 10 (right) with weight decays 3 = 0.0005 (top) and B = 0.1 (bottom).
27

---

K Hyperspherical Uniformity with Scale - GNC2
Log Distances (CoV)
10-
1e-1
1.00
8
L-1
L=2
0.75
6
- L=4
4-
L=8
0.50
4
L-12
3-
64 256
512
768 1024
64 256
512 768 1024
64 256
512 768 1024
Width (Hidden Dim.) Epochs
Width (Hidden Dim.)
Width (Hidden Dim.) Epochs
3 Epochs
1e-1
Log Distances (CoV)
1.00
L=1
5
5-
-.- L
0.75
4-
L=8
0.50
3
3
-.. L=12
2
64 256 512
768
1024
64
256
5i2
768
1024
64
256
512 768 1024
Width (Hidden Dim.)
Width (Hidden Dim.)
Width (Ilidden Dim.)
Epochs
6 Epochs
10 Epochs
Figure 18: Variation in logarithmic distances decreases when scaling width (d) in models trained for 1 (left) through 10 (right) with weight decays 0.0005 (top) and 3 0.1 (bottom). This consistent trend towards hyperspherical uniformity affirms that GNC2 [9 is more useful than NC2.
1e-1
1e-1
Log Distances (CoV)
1.00
d=
5
d-128
0.75
d-256
4-
d=512
0.50
d=768
d-1024
2
8
12
4 Depth (Num. of Layers) Epochs
2
12
1
2
4
8
12
Depth (Num. Layers) Epochs
Depth (Num. of Layers)
10 Epochs
le-
Log Distances (CoV)
1.00
d=64
d=128
0.75
4
d=256
d1512
0.50
3
768
d=1024
2
1
12
2
12
4
8
12
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
Epochs
Figure 19: Variation in logarithmic distances decreases when scaling depth (T. in models trained
for 1 (left) through 10 (right) with weight decays 3 0.0005 (top) and 8 0.1 (bottom). This consistent trend towards hyperspherical uniformity affirms that GNC2 [19] is more useful than A/2.
28

---

L Correlations of (g)NC, with Generalization Performance
19.137
R² 0.86
R² 0.86
18.389
Log # Parameters
17.984
Sel Loss
17.489
16.796
16,020
15.746
15.053
2.0
2.5
3.0
3.5
4.0
0.06
0.08
0.10 0.12
Log Mean Norms (Avg) (AC2)
Log Mean Norms (CoV) (AC22
Figure 20: Generalization (validation loss) shows some correlation with logarithmic mean norms in both their average (left) and variations (i.e. equinormness, NNCh)) (right).
19.137
R² = 0.56
R² - 0.79
R 0.74
18.380
(metes
Loss
3-
17.480
16.796
16.020
15.
-0.10
-0.05
0.00
4000
6000
8000
10000
1.38
1.39
1.40
Interference (Avg) (N/2))
Interference (CoV) (NC2)
Simplex ETF Error (AR2))
Figure 21: Generalization (validation loss) correlated with average interference (left) and its variation (i.e. equiangularity. .CC2) (centre). We also computed the empirical measure from (right).
4
19.137
18.389
Log # Parameter
17.984
Loss
3
17.489
16.796
2
16.020
0.82
R² = 0.84
15.746
15.053
0.34
-0.32
-0.30
0.25
0.50
0.75
1.00
Log Distances (Avg) (g.V'2)
Log Distances (CoV) (CNCC2)
Validation loss shows some correlation with average (logarithmic) kernel distances and with their variation (i.e. hyperspherical uniformity, G2C22) (right).
Figure 22:
29

---

M Self-Duality with Scale - (Against NC3)
Self-duality (NC3) was originally the convergence of classifiers means up to rescaling [T]
we
Rc - 0, vc
(11)
weel2
Instead, we use class-wise cosine similarity (Equation
8) and its variation (UNC3).
Similarity (Cosine) (Avg)
1e-1
1e-1
18
2.5
L=1
3.5
L=2
2.0
2.5-
3.0
L
L=12
1.5
2.5
2.0
256 512 768 1024 Width (Hidden Dim.) Epochs
256 512 1024 Width (Hidden Dim.) Epochs
64 256 512
1024 Width (Hidden Dim.)
10 Epochs
Similarity (Cosine) (Avg)
1e-1
1e-1
L=1
3.5
3.5
2.0
L=8
L-12
1.5
3.0
64 256 512 768 1024 Width (Hidden Dim.) Epochs
64 256 5i2 768 1024 Width (Hidden Dim.) Epochs
64 256 512 768 1024 Width (Hidden Dim.) 10 Epochs
Figure 23: Average classifier alignment increases NC3 when training for (left) through 10 (right) with weight decays 3 0.0005 (top) and | = 0.1 (bottom). However, we see no meaningful trend when scaling width d, suggesting that NC3 does not coalesce with language modelling training.
Similanty (Cosine) (Avg)
1e-1
2.5
d-64
3.0
3.5
d=128
2.0
3.0
d-256
d=512
d= 768
1.5
2.5
d=1024
2.0
12
4
8
12
12
4
8
12
2
4
12
Depth (Num. of Layers) Epachs
Depth (Num, of Layers) Epochs
Depth (Num. of Layers)
10 Epochs
Similarity (Cosine) (Arg)
10 1
18
1e
4.0
2.5
d=64
d=128
3.5
3.5
d5256
2.0
d-512
d= 768
3.0
d= 1024
Depth (Num. of Layers) Epochs
Depth (Num. of Layers) Epochs
Depth (Num. Layers)
10 Epochs
Figure 24: Average classifier alignment increases NC3 when training for (left) through 10 (right) meaningful trend
with weight decays 8 0.0005 (top) and 3 = 0.1 (bottom). However, when scaling depth L, suggesting that NC3 does not coalesce with language modelling training.
30

---

N
Uniformity Duality with Scale - -Va2
Similarity (Cosine) (CoV)
(=1
2.0
1.4
1.0
[=2
1.2
L=4
1.5
* L=8
[=12
1.0
64
256
512 768 1024
64 256
512 768 1024
64 256 512 768 1024
Width (Hidden Dim.) Epochs
Width (Hidden Dim.) 3 Epochs
Width (Hidden Dim.)
10 Epochs
Similarity (Cosine) (CoV)
1.0
1.0
L-1
- L=2
L=4
1.5-
L=8
L-12
0.6
0.6
1.0
64 256
512
768
1024
64 256
512
768 1024
64 256 512 768 1024
Width (Hidden Dim.) Epochs
Width (Hidden Dim.) Epochs
Width (Hidden Dim.)
10 Epochs
Figure 25: Variation in classifier alignment decreases when scaling width (d) in models trained for 1 (left) through 10 (right) with weight decays 0.0005 (top) and 0.1 (bottom).
Similarity (Cosine) (CoV)
d=64
2.0
1.4
1.0
d=128
d=256
0.8
1.5
d=512
1.0
d=768
0.8
0.6
d=1024
1.0
2
12
1
4
Depth (Num. of Layers)
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
1 Epochs
10 Epochs
Similarity (Cosine) (CoV)
1.0
1.0
d=64
2.0
d 128
0.8
0.8
d-256
1.5-
d=512
d=768
0.6
d= 1024
2 4
12
12
12
1
4
12
Depth (Num. of Layors) 1 Epochs
Depth (Num. Layers) Epochs
Depth (Num. Layers) 10 Epochs
Figure 26:
Variation in classifier alignment increases when scaling depth (L) in models trained
for 1 (left) through 10 (right) with weight decays 3 = 0.0005 (top) and 3 = 0.1 (bottom). This negative trend of ULVC3 in more learnt models (right) suggests that the link of (2)WC3 with scale and performance is still weak.
31

---

Correlations of (U)NC3 with Generalization Performance
19.137
R² 0.69
R 0.31
R 0.67
380
17.964
Loss
17.469
16.000
15.746
2.5
5.0
7.5
10.0
0.2
0.3
0.4
1.395
1.400
1.405
1.410
Similarity (Dot Prod.) (Avg) (ACs)
Similarity (Cosine) (Avg) (A2g)
Self-Duality Error (AC2s)
Figure 27: Generalization (validation loss) correlated with average dot- dot-product similarity (for inter- pretability purposes only) (left) and cosine similarity (classifier alignment, NC3)) (centre). We also computed the empirical measure from [5] (right).
4
19.137
R
0.69
R² 0.75
18.389
Log # Parameters
17.984
Loss
17.489
Val.
16.796
2
16.020
15.746
15.053
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.5
Similarity (Cosine) (CoV) (UWCss
Similarity (Dot Prod.) (CoV) (UNC))
Figure 28: Generalization (validation loss) correlated with variation in dot-product similarity (for interpretability purposes only) (left) and cosine similarity (uniform duality, (NCCs) (right).
32

---

P
Classifier Agreement -
-C
For computational reasons, we compute Equations 9
ussing a simple decomposition:
argmin
| Mell2 = argmin cEV
(11b01 + |l 2 2//, Me.
(12)
OEV
where [1, B) and c E with batch size B and vocabulary V.
Classifier Agreement (%)
L=1
+ L=2
2-
L=4
L=8
1-
+ L=12
512 768
512 768 1024
512 768 1024
Width (Hidden Dim.) 1 Epochs
Width (Hidden Dim.) 3 Epochs
Width (Iiidden Dim.)
10 Epochs
Classifler Agreement (9%
1e
1e
1e
L-1
L=2
2.
L=4
2-
--- L=8
1-
L=12
64 256 512 768 1024
64 256 512 768 1024 Width (Hidden Dim.) 6 Epochs
64 256 512 768 1024 Width (Hidden Dim.) 10 Epochs
Width (Hidden Dim.) Epochs
Figure 29: Classifier agreement improves when scaling width (d) in models trained for I (left) through 10 (right) with weight decays 3 = 0.0005 (top) and 0.1 (bottom).
Classifier Agreement (%)
le
3
d=64
d=128
2.
2
d=256
d=512
1-
d=768
d=1024
2 4
12 Depth (Num. of Layers) Epochs
12
12 4
12
Depth (Num. of Layers) Epochs
Depth (Num. of Layers)
10 Epochs
Classifler Agreement (%)
d=64
d=128
2.
d=256
2
d=512
d= 768
d=1024
12
12
Depth (Num. of Layers) 1 Epochs
Depth (Num. Layers) Epochs
Depth (Num. of Layers)
10 Epochs
Figure 30: Classifier agreement improves when scaling depth (L) in models trained for 1 (left)
through 10 (right) with weight decays 3 = 0.0005 (top) and = 0.1 (bottom).
33

---

Examples for Interpretability
This section presents token-wise interpretability results from top-layer embeddings from our most learned models (trained for 10 epochs). Our largest one is publicly available: https
//huggingfan ace. ht///aabrwww//yyyytotsssssssssea 12x_0ee__xx11222_111
Paiviss Grometies (020162))
Pairwise Gosmetries (04x1024)
Pairvise Goometites (0Bx1024)
Parrwise Geometrics (12x1
Inlerhesnaa (NC2)
SAIIAAL DiALzNN GACCO
Kerrel butreoo
Figure 31: Pairwise interactions between token mean embedding vectors across models of fixed width (d 1024) and increasing depths (L= = 2, 4, 8, 12). Interference (NC2) decreases on average but only slightly becomes more uniform (top-right, blue). In contrast, logarithmic kernel distances (GNC2) decrease and become more evenly spread, with some outlier pairs (bottom-left, green).
Class Mean Norms of Homonyms (TinyStories 12x1024)
120
Class Mean Embestding Norms
100
80
60
40
20
wavo
well
close
ring
light
spring
leff
baf
bowl
bank
lead
[AVERAGE]
data
match
Vocabulary (GPT2 Tokenizer)
Figure 32: Under -1tooriees-2x1224.1.. these fifteen homonyms have much shorter mean embedding norms (i.e. closer to the global centre) than the average token. This is expected since homonyms typically present conflicts and interference.
34

---

Within-Class Variability of English Given Names (TinyStoriis 12x1 024)
0.000175-
Token Pairwine CDNV
0.000150-
0.000125
0.000100
0.000075
0.000050
0,000025
0.000000
Elizabeth Charlum S Mathing Robory
Kally fessica Donald
Ryan
Mary Jeunifer
Janob
Andrane William
Yevn
Tommas Steve
Paul M
Luke Jamus
Gouroe
John
Jim
FiGuT Harry Datbll Robia
(AVERAGE)
Tim
Vocabulary (GPT2 Tokenizer)
Interference of English Given Names (TinyStories 12x1024 024)
Average Token Pairwisn Interference
0.000
-0.002
-0.004
-0.006
-0.008
Donald
Martin Daniel
Robin
Pand
Harry
Hearry
Michant
Jamess
Ayan
Eric
Eliabbeth both Robery
Alico Thomas Charles les
So Sarah
Mary
Anthom snthooy Androw
Kelly
Britn
Latrr
William
Lake
Jesoica Matthow
Jacsh
Kyto
Anna
lAVERAGE
Timi
Vocabulary (GPT2 Tokenizer)
Figure 33: Under TinyStories- 111sstrrress11xx0244 10L, the average within-class variability (top) and inter- ference (bottom) of some English first names far below those of the average token. This might be because names are distinct and are not typically used in the same contexts as other words (aside from articles). The only names have CDNV close that of the average token are "Anna" and "Tim". Note that the positive interference of the average token (right) is not a typo.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims of this paper listed the abstract and introduction are simply the emergence of the NVC phenomena and its relationship with generalization in CLMs, which are the scope and results of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the experiments themselves (e.g. small models, limited sample size) as well as the possibility that our formulations could be improved in the future (there could be more suitable metrics). See Sections 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
36

---

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The equations and equivalent decompositions used are all clear and numbered in the paper. Their sources are all provided as references. We do not introduce new proofs or non-trivial expressions. See Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Our methods for training the models, collecting embeddings, and analyzing metrics are clearly described in the paper, with supporting details in Appendices A B Links code are provided in the Abstract and Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
37

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Links to the main code and auxiliary library are provided in the Abstract and Section E] respectively. Main code repository includes training, collection, and analysis scripts, as well as a README briefly documenting how to use them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
sss.mmolissiyycoooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https / wtppps.ccppublic.. conooliccggideee /tt.://uuddeeooeemimiiicccci wwiobbmmssiooooiiioiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
D provide the necessary information on model, data, training, and collection specifications reproduce our experiments.
Justification: Appendices
A B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We trained multiple copies of the same models and observed patterns consistent with the ones presented in our paper. We also state the number of data points and permutation
trials we performed for our significance test. See Figure I and Sections 4 and 5
38

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details on the number of datalozdder/peproocessng works, the specific GPU model used and the amount of RAM and training/collection time required. See Appendices
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: fttttp:://neurigge http.///cb..///hiccccieeeeeeeeee
Answer: [Yes]
Justification: We have read the Code of Ethics and determined that our paper does not violate the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
39

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes discussion on how NC might be extended to measure fairess in LLMs. Naturally this may inform positive adversarial manipulations of LLMs.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We do not use pretrained models or scraped data. Our dataset is synthetic, public and transparent with no risks. The training procedure we use is the standard causal language modeling regime.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
40

---

The pre-training dataset and its licence are provided in Appendix A and the authors of basic GPT Neo architectures are credited in Appendix B
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /daaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Timestamped CSV files of results that we reported in our paper are included in the repository linked in the Abstract. A link to a reference model is provided in Appendix Q
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not work with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
41

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not work with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
42


Reviews:
[{"review_id": "gDAyITviu6", "reviewer": "Reviewer_NZtr", "summary": "This paper investigates neural collapse (NC) -- properties of the penultimate feature representation of DNN -- in causal language models (CLM). Previous NC studies have primarily focused on classification problems with balanced classes with few labels compared to the feature dimensionality. This paper finds that NC properties in CLM are correlated with generalisation. This finding is based on extensive experimental evaluations on the TinyStories dataset.", "questions": "See \"Weaknesses.\"", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The NC analysis is interesting and it's noteworthy that properties of NC appear to correlate with generalisation in CLMs, which differ in important ways from other classification settings where NC has been studied.\n\n* The empirical results seem to paint a clear trend in terms of NC and generalisation. \n\n* The presentation is generally quite good, with contributions and significance clearly spelled out.", "weaknesses": "* I’m somewhat unclear on the motivation for using TinyStories, a purely synthetic dataset. While I understand the benefit of using a small dataset to ease the computational burden of the experiments, I would have liked to see some further experiments on real data confirming the observed trends. For example, you could sample a small dataset of human-written stories.\n\n* I would have liked to see more practical guidance on the value of the proposed metrics. For example, it’s not clear if they provide any value in terms of assessing model fit beyond what a held-out validation would (more cheaply) provide.\n\n* In several places, the wording is unusual / confusing. For example, L47-L48 says that \"[...] LLM learn to model aleatoric\nuncertainty, which can be viewed as stochastic token prediction,\" which is a peculiar/wordy way to describe a standard MLE procedure.\n\n* While the experiments are fairly comprehensive, the technical novelty is relatively low. The paper mostly consists of methods from prior works being applied to a new classification setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I’m somewhat unclear on the motivation for using TinyStories, a purely synthetic dataset. While I understand the benefit of using a small dataset to ease the computational burden of the experiments, I would have liked to see some further experiments on real data confirming the observed trends. For example, you could sample a small dataset of human-written stories.\n\nThis crucial question arose during our research, as using a synthetic dataset may limit the generality of our results to human-written data (see our lengthy explanation in the general rebuttal). We acknowledge the importance of validating our findings with real data and are aware of follow-up efforts to scale our experiments with appropriately sized models and datasets.\n\n> I would have liked to see more practical guidance on the value of the proposed metrics. For example, it’s not clear if they provide any value in terms of assessing model fit beyond what a held-out validation would (more cheaply) provide.\n\nNC has been studied in and in relation to many areas of machine learning. We defer the vast related works and applications to the Related Works and Discussion sections of our manuscript, while our response here will therefore be tailored towards practical consequences and applications of studying NC in CLMs.\n\n1. Better optimization strategies: LLMs are typically pre-trained with CE loss in teacher-forcing. It stands to reason that works listed in the “Learning to collapse” Discussion paragraph can be applied or inspire better objective functions towards potentially better geometric configurations for language models. Hierarchical structures like those presented by Liang & Davis (2023). The implication is that future LLMs could be trained in fewer steps or to better performance.\n2. Interpretability of contextual token-wise behaviors: our measurements provide insight into behaviors of individual token embeddings. Token-wise NC3 can reveal how confident an LLM is about certain tokens in their top-layer classifiers. For example, the number and density of clusters for a particular token can shed light into its various meanings and uses. This would be particularly useful as LLMs adapt to ever-evolving use of language and further expansion into non-English domains.\n3. Interpretability of pair-wise interactions: our measurements primarily center around pair-wise relationships between tokens in a vocabulary. The pair-wise arrangements are critically important to modeling interference between tokens because their context vectors live on a lower-dimensional hypersphere. For example, you can tell how related or interchangeable two words are based on their noise and interference (NC1,2), or how antithetical or unrelated they are based on orthogonality (NC2).\n4. Interpretability for fairness: natural language is inherently imbalanced. Individual or pair-wise analyses of representations can reveal inequalities within the vocabulary or across topics. These insights can guide researchers and engineers to address inequities in a targeted and measurable way.\n5. Interpretability for ethics: if there are concerns about biases or safety issues in general language modeling, NC analysis can aid researchers in interpreting how LLMs could pose risks to users, and how to safely mitigate risks without degrading model quality.\n6. Continual learning: [19, 20, 21] cited studied NC in (task-)incremental learning problems and provided insights or improvements. Since an LLM (especially a foundation model) is to continually learn language patterns and develop capabilities, NC can provide carefully structured configurations that allow for graceful lifelong learning.\n\nOf course it remains to be seen in exactly what form or role NC will play in these areas of further studies. It is also probable there are benefits of NC analysis beyond our knowledge. Our work here is to take the first step in extending the body of NC work into the more irregular and demanding settings of causal language modeling.\n\nWe are grateful to the reviewer for probing this response. We will further expand our Significance and Discussion (sub)sections to explicate these practical guidelines.\n\n> In several places, the wording is unusual / confusing. For example, L47-L48 says that \"[...] LLM learn to model aleatoric uncertainty, which can be viewed as stochastic token prediction,\" which is a peculiar/wordy way to describe a standard MLE procedure.\n\nWe agree with the reviewer and will revise L47-48 to “[…] learn to stochastically predict tokens to generate text.” We also identified the following opportunities for rewording or clarification:\n\n1. L107-108 will be adjusted: “[…] classes number in the tens of thousands.”\n2. L128 will be clarified such that ambiguous contexts *are* “soft-label” samples.\n3. L230-231 will be rewritten: “These noise reductions are associated with generalization (Fig. 1, left, “N C1”); this relationship grows stronger with model size.”\n\nShould the reviewer feel there are more instances that we did not cover, we will happily review further during the discussion period. And of course, we will perform more passes on this paper to improve the writing for future revisions.\n\n> While the experiments are fairly comprehensive, the technical novelty is relatively low. The paper mostly consists of methods from prior works being applied to a new classification setting.\n\nThis is a fair point, that we mostly applied previous expressions and methods to the causal language modeling setting. Our contribution focuses on applying the canonical NC framework to more adverse conditions (larger and imbalanced vocabulary, ambiguous contexts, undertraining), the effort of which is reflected in our work. We hope that this first attempt at analyzing NC in this area inspires further work that better adapts to the ever-changing landscape.\n\n---\n\nWe are grateful for the helpful comments. Should our response warrant it, we would greatly appreciate the reviewer raising the score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWRTisgCkz", "reviewer": "Reviewer_Nr3s", "summary": "The final layers of classifiers show a property called neural collapse (NC), which are seen as beneficial to model performance. The authors study it’s appearance in causal language models (CLMs), and point out that CLMs do not respect those conditions (CLM are trained on noisy, unbalanced data, made on more tokens than the models have dimensions, and training is stopped before loss reaches 0). \nThey then provide a theoretical framework to adapt the NC properties to CLMs (arguing for two relaxations from past works: measuring hyperspherical uniformity, and uniform duality), and consider show that despite expectations, evidence of NC exists, and is stronger for better performing models.", "questions": "considering CLMs to be classifiers is technically sound, nonetheless unlike with classification tokens can mean different things (ex:homonyms) these might also explain divergence from the classic NC model. Did you observe any such groupings or effects?", "limitations": "Limitations are to my knowledge well adressed", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Provided measured and well formulated novel results regarding the impact of scale, training, number of parameters on the different components of generalisation. While a highly technical read which required a lot of concentration, all the required information for understanding the concepts are introduced. Conclusions are measured and entirely based on evidence from theory and empirical evidence, yet discussion manages to highlight the implication of such research. Exciting mathematical tools I hope to use in the future.", "weaknesses": "Apart from evaluation loss, model performance metrics from commonly used LLM benchmarks are not provided, making high level comprehension of the technical observations more complicated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Apart from evaluation loss, model performance metrics from commonly used LLM benchmarks are not provided, making high level comprehension of the technical observations more complicated.\n\nLLM researchers are indeed interested in performance metrics beyond the cross-entropy (CE) evaluation loss that we use. However, as our work concerns the measurement of NC under model pre-training only, other benchmarks (particularly those on downstream tasks) would not necessarily be appropriate as they measure specific capabilities rather than generic stochastic token prediction; therefore, they would be out-of-scope for our work.\n\nFurthermore, according to several recent works on LLM evaluations (https://arxiv.org/abs/2403.15796, https://arxiv.org/abs/2404.09937, https://arxiv.org/abs/2407.06645), downstream capabilities of LLMs are roughly correlated with their abilities to compress their pre-training data. Based on these findings, we find that CE loss is the most sensible metric with which to measure generalization.\n\nWe thank the reviewer for raising this question as we see fit to include a short discussion on this and potential future directions towards the end of our manuscript (citing the aforementioned works).\n\n> Considering CLMs to be classifiers is technically sound, nonetheless unlike with classification tokens can mean different things (ex:homonyms) these might also explain divergence from the classic NC model. Did you observe any such groupings or effects?\n\nAppreciate the question! We inspected token-wise NC values in our largest and most-trained model (12-layer d=1024, 10 epochs). Based on your suggestion, we chose 15 homonyms and found that most have much shorter mean vector norms (meaning they’re closer to the global center) than the average token. This makes sense as homonyms present conflicts and interference.\n\n| GPT-Neo Token ID | Token Text | Scaled Norm |\n| ---------------- | ---------- | ----------- |\n| 808              | row        | 102.9727    |\n| 1806             | ring       | 63.4789     |\n| 2971             | light      | 74.4070     |\n| 4053             | well       | 55.3457     |\n| 4475             | date       | 111.7086    |\n| 8664             | bat        | 83.0259     |\n| 9464             | left       | 81.7467     |\n| 15699            | match      | 114.7236    |\n| 16469            | spring     | 80.0322     |\n| 17796            | bank       | 90.8248     |\n| 19204            | wave       | 43.5028     |\n| 19836            | close      | 60.6004     |\n| 22043            | fair       | 57.2620     |\n| 28230            | lead       | 102.0583    |\n| 36859            | bowl       | 88.7034     |\n| AVERAGE          | AVERAGE    | 106.8762    |\n\nWe also observed that the individual variability and interference of some English first names all other tokens were far below the average. This is also intuitive as names are distinct from one another and aren't typically used in the same contexts as other words (aside from articles).\n\n| GPT-Neo Token ID | Token Text | CDNV         | Interference |\n| ---------------- | ---------- | ------------ | ------------ |\n| 7371             | Donald     | 8.176141e-05 | -0.008308    |\n| 7554             | John       | 0.000108     | -0.007270    |\n| 11006            | David      | 9.249406e-05 | -0.006167    |\n| 12041            | Paul       | 9.661650e-05 | -0.006768    |\n| 13256            | Michael    | 8.857495e-05 | -0.006625    |\n| 14731            | James      | 0.000101     | -0.006610    |\n| 14967            | Tim        | 0.000164     | -0.004741    |\n| 17121            | William    | 9.332073e-05 | -0.005867    |\n| 18050            | Jim        | 0.000109     | -0.006896    |\n| 18308            | Harry      | 0.000110     | -0.006749    |\n| 19156            | Robert     | 8.133316e-05 | -0.006438    |\n| 19206            | Steve      | 9.622102e-05 | -0.006541    |\n| 19962            | Daniel     | 0.000114     | -0.006963    |\n| 20191            | George     | 0.000108     | -0.006674    |\n| 20508            | Andrew     | 9.280709e-05 | -0.006036    |\n| 21868            | Ryan       | 8.332534e-05 | -0.006594    |\n| 22405            | Thomas     | 9.526886e-05 | -0.006400    |\n| 23865            | Kevin      | 9.392182e-05 | -0.005817    |\n| 24119            | Mary       | 8.846451e-05 | -0.006322    |\n| 24761            | Brian      | 8.554082e-05 | -0.006003    |\n| 24778            | Martin     | 9.663536e-05 | -0.007183    |\n| 25004            | Eric       | 9.731254e-05 | -0.006565    |\n| 25372            | Matthew    | 8.109680e-05 | -0.005544    |\n| 28711            | Charles    | 7.878443e-05 | -0.006371    |\n| 29284            | Sarah      | 7.926711e-05 | -0.006366    |\n| 30730            | Luke       | 0.000100     | -0.005830    |\n| 31160            | Anna       | 0.000175     | -0.005005    |\n| 32476            | Henry      | 0.000109     | -0.006685    |\n| 32697            | Anthony    | 6.594844e-05 | -0.006055    |\n| 34831            | Kelly      | 8.580151e-05 | -0.006017    |\n| 40656            | Robin      | 0.000118     | -0.006808    |\n| 42516            | Kyle       | 8.159300e-05 | -0.005333    |\n| 43187            | Jennifer   | 8.750914e-05 | -0.006629    |\n| 43568            | Elizabeth  | 6.347037e-05 | -0.006462    |\n| 43687            | Laura      | 8.584216e-05 | -0.005922    |\n| 44484            | Alice      | 9.208282e-05 | -0.006416    |\n| 45572            | Jessica    | 8.405318e-05 | -0.005614    |\n| 46751            | Jacob      | 9.124901e-05 | -0.005349    |\n| AVERAGE          | AVERAGE    | 0.000177     | 0.000519     |\n\nThere are probably thousands of groupings or linguistic relationships one could observe in the NC measurements, so we’ll leave such interpretability to future applications.\n\n---\n\nWe thank the reviewer for the thought-provoking questions. We hope our responses adequately address them and warrant a slightly higher score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z83CKev7aU", "reviewer": "Reviewer_XbAS", "summary": "This work focuses on studying the Neural Collapse phenomenon in the context of language model training. The author first introduces the original NC properties, explaining how such metrics may not apply to the case of LLM training given 1) the ambiguity of language next token prediction, 2) large number of possible tokens , 3) token imbalance and 4) under-parametrization or undertraining of LLM models.  \n\nHaving established this, the authors introduce new metrics based on the NC ones. Further, the authors analyze the relationship between these metrics and validation loss on a range of models trained on TinyStories dataset. In this regard, 1) collapse of token representation ($ NC_1 $), 2) Hyper-spherical uniformity which represents maximal separation ($ G-NC_2 $), 3) alignment between token feature mean and token classifier ($ U-NC_3 $) and 4) NCC accuracy of features ($ NC4 $) have positive (and stronger) correlation with validation loss.", "questions": "1) Is it true that NC properties are attributed to better generalization for classification ? Could the authors please provide some references.\n2) Have the authors considered using smaller or simpler language datasets to combat the problems with large numbers of classes or have models be trained for a longer time in order to see better convergence results for the NC metrics as illustrated in the appendix ?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The connection established between Neural Collapse and LLM training is an interesting point. The introduction of the new NC metrics are not only viable for LLMs but also for other classification and training regimes that suffer from similar problems as the language domain. \n\n\nFurther the experiments on TInyStories are performed on a large number of models to establish a connection between the metrics and validation loss and the results could be used to develop evaluation metrics for LLM training.", "weaknesses": "1) **(Major)** I appreciate the extensive experimental work on establishing a connection between the proposed NC metrics and validation accuracy; however, I believe the coefficient of determinations provided in Table 1 suggest a low correlation between NC properties and validation loss. While I see the higher correlation for NC1 and NC4 in Fig 1, I’m having a hard time convincing myself that other factors such as model size and architecture don’t play a role in improving the loss.\n2) **(Major)** While I understand the attempt at making the autoregressive next token prediction of LLMs comparable to a simple classification setup, I do not think I agree with the “not soft label, not multi label” argument. As the authors suggest, the language format is rather ambiguous. Considering a simple next word prediction, “I went to the ___ “ can be followed by a variety of valid words. Same applies to token baked predictions. I believe not having a mechanism to account for this and dismissing the multi-lableness or soft-lablenss of language removed much of the uniqueness and difficulty of dealing with LLM training. \n3) **(Major)** Considering how languages themselves have an imbalanced nature, I wonder whether the suggested ETF of hypersphere features and weights geometric structure are actually optimal for LLMs. As an example, in the case where for a CIFAR10 classifier, we train the model under artificial imbalance, I would understand why symmetric features could potentially help improve test accuracy given the balanced nature of the validation set. However, I have a hard time being convinced the same is true in language. \n4) **(Minor)** I find it hard at times to follow the paper’s ideas. Particularly with regards to section 4, I believe a bit more structure or separation of ideas could help readers better digest the conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **(Major)** I appreciate the extensive experimental work on establishing a connection between the proposed NC metrics and validation accuracy; however, I believe the coefficient of determinations provided in Table 1 suggests a low correlation between NC properties and validation loss. While I see the higher correlation for NC1 and NC4 in Fig 1, I’m having a hard time convincing myself that other factors such as model size and architecture don’t play a role in improving the loss.\n\nTo clarify, we don't claim that size and training don't contribute to reducing the loss. Our principle observation is that scaling improves both performance (as expected) and NC, and that the two are associated (shown in Figure 1).\n\nTo address the concern that scaling alone is the confounding factor, we conducted a seed sweep ($\\S$ 4.5) to test an independent correlation. Table 1 shows that most correlations between NC and generalization are statistically significant. Equinormness is not correlated but only accounts for some of NC, a property that is superseded by GNC2 anyway. The only real exception is therefore UNC3, which seems to be entirely confounded by model scale. (Although this does highlight the effect of NC3 on performance independent of scale.)\n\n> **(Major)** While I understand the attempt at making the autoregressive next token prediction of LLMs comparable to a simple classification setup, I do not think I agree with the “not soft label, not multi label” argument. As the authors suggest, the language format is rather ambiguous. Considering a simple next word prediction, “I went to the ___ “ can be followed by a variety of valid words. Same applies to token baked predictions. I believe not having a mechanism to account for this and dismissing the multi-lableness or soft-lableness of language removed much of the uniqueness and difficulty of dealing with LLM training.\n\nThanks for the detailed comment. Firstly, we agree with the reviewer that there are indeed ambiguous samples, particularly with short sentences, such as the example “I went to the ___.” These cases can resemble soft label data examples, and we will edit the relevant paragraph in the Related Works to reflect this.\n\nWe note however that ambiguous contexts are typically shorter and found at the beginning of sequences. For longer sequences, the probability of two identical contexts followed by a different word is low. Additionally, longer contexts outnumber shorter ones in the training data, especially as LLMs (such as Llama 3.1) handle longer contexts. Shorter contexts could even be excluded or treated as outliers. Thus, we conclude that these anomalies likely represent a minority of the data and aren't a significant issue overall. Nonetheless, we take the first step in NC at the next-token level, and future work might incorporate multi-label (arXiv:2310.15903) and mixup (arXiv:2402.06171) NC.\n\n> **(Major)** Considering how languages themselves have an imbalanced nature, I wonder whether the suggested ETF of hypersphere features and weights geometric structure are actually optimal for LLMs. As an example, in the case where for a CIFAR10 classifier, we train the model under artificial imbalance, I would understand why symmetric features could potentially help improve test accuracy given the balanced nature of the validation set. However, I have a hard time being convinced the same is true in language.\n\nOur primary objective is to investigate the structures that form in standard LLM training. We recognize that the convergence geometry in current LLMs may not be optimal, likely due to using CE loss (adapted for balanced data) on an imbalanced dataset. We don't not claim the simplex ETF or hypersphere are optimal; rather, we focus on characterizing the patterns that emerge at scale in relation to baseline geometries. However, recent works have proposed methods for addressing data imbalance (arXiv:2301.00437v5, arXiv:2301.01100, and MLR: Liu et al. (2023)). We hope our insights may lead to adaptations of such methods to optimize these structures that address the imbalanced nature of language\n\n> **(Minor)** I find it hard at times to follow the paper’s ideas. Particularly with regards to section 4, I believe a bit more structure or separation of ideas could help readers better digest the conclusions.\n\nWe sympathize with this sentiment. We'll move 4.5 through 4.7 (including Table 1) into a separate section.\n\n> Is it true that NC properties are attributed to better generalization for classification? Could the authors please provide some references?\n\nThe original NC showed some relationship to generalization. In particular, we highlight the results in transfer learning (arXiv: 2112.15121) and a generalization bound that can estimate test performance (arXiv: 2202.09028).\n\nWe aren't aware of other works that explicitly studied the relationship between NC and generalization. However, our results appear to reinforce the generalization narrative as we observe correlations long before the terminal phase of training, some of which are independent of scale. We hope to inspire future works that might definitively answer this open question.\n\n> Have the authors considered using smaller or simpler language real datasets to combat the problems with large numbers of classes or have models be trained for a longer time to see better convergence results for the NC metrics as illustrated in the appendix?\n\nThank you for your insightful suggestion. We considered using smaller or simpler language data to potentially improve convergence. However, we focus on large-scale language modeling scenarios to ensure our findings are robust and reflective of contemporary practices. This would render our results applicable and generalizable to real-world applications. The synthetic nature of TinyStories is a caveat to our study (as the reviewer implies), but we provide a lengthy explanation for its use over real data in the general rebuttal above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1G5fq0U1kj", "reviewer": "Reviewer_3JhA", "summary": "This paper empirically investigates the emergence of Neural Collapse (NC) properties during the training of causal language models. NC is a phenomenon observed in the top layer of deep-nets trained on one-hot classification problems, where the last-layer class-mean embeddings become equinorm, have maximal angular separation and align with their respective last-layer classifier. However, NC only emerges when: 1) the models are trained beyond the zero training error regime, 2) the number of classes $C$ is larger than the last-layer embedding dimension $d$, 3) the classes are balanced (they have the same number of samples in the training set).\n\nConsidering causal language training as the classification of contexts across $C$ words in the vocabulary set, this work explores the extent to which NC emerges in language models, given that none of the three conditions above hold for language models: 1) language models are typically trained for a few epochs, 2) the number of classes $C$ is large, 3) the words frequency in the training set is heavily imbalanced. Since the original NC geometry is not achievable, particularly without condition 2, the authors define *generalized* NC metrics to measure the geometrical properties of the last-layer embeddings and classifiers. They train NeoGPT with varying numbers of layers and width $d$, and analyze the correlation between the (generalized) NC metrics and validation loss across different training regimes.", "questions": "1. Is your setup only specific to causal language modeling? Particularly, a) Is any part of the formulation or results influenced by the causal/autoregressive training, or is it simply related to the different nature of language datasets in general, such as the large number of classes and the possibility of having multiple labels for a given training sample?, b) Do you expect similar behavior if the experiments were conducted using other language training methods, such as masked language modeling?\n\n2. I don’t understand what you mean by *ambiguous* samples, ``which are not soft- or multi-label’’ (line 128). When a context appears several times in the training set, each time followed by a different next token, this training sample has a soft label, where the label (next token) can take on different values with some non-zero probability for each. \n\n3. (Related to the previous question) What is classification error (line 130) in the context of next-token prediction and what do you mean by \"irreducible noise’’? How do you define the error for contexts that appear several times with different next tokens/labels?\n\n4. Intuitively, why do you expect that a model minimizing UNC3 has better generalization? Minimizing NC3 means the classifiers and mean vectors are aligned which is consistent with the classification objective. However, minimizing UNC3 (CoV of the term in Eq (8)), only implies the degree of misalignment between the classifiers $w$ and mean embeddings $\\mathbf{\\mu}$ is uniform across classes/samples. How does this connect to generalization?\n\n5. Do you observe a decrease in the NC metrics and validation loss as you train the models longer (e.g., 1 vs 10 epochs)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem setup is interesting as it attempts to extend previous observations in deep learning to the trending language models, despite their distinct configurations. The scope of the study, its connection to previous works, and the limitations of the setup are well-presented in the paper.", "weaknesses": "It is not clear to me whether the suggested NC metrics are suitable for the language setup. For instance, since in a language dataset, a given fixed context might be followed by different next tokens, a model that is accurate in predicting the labels on the training set cannot achieve near-zero NC1. Or regarding UNC3, see question 4 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I don’t understand what you mean by *ambiguous* samples, ``which are not soft- or multi-label’’ (line 128). When a context appears several times in the training set, each time followed by a different next token, this training sample has a soft label, where the label (next token) can take on different values with some non-zero probability for each.\n\nThe reviewer is correct that such ambiguous contexts do resemble soft-label data examples, and we will edit the relevant paragraph in the Related Works to reflect this.\n\n> It is not clear to me whether the suggested NC metrics are suitable for the language setup. For instance, since in a language dataset, a given fixed context might be followed by different next tokens, a model that is accurate in predicting the labels on the training set cannot achieve near-zero NC1. Or regarding UNC3, see question 4 below.\n\nWe agree that such ambiguous (soft-label) samples such as “Once upon a time ___.” do exist. However, we note that such contexts are more often shorter subsequences typically found in the beginnings of sequences. In longer sequences, the probability of having two identical contexts followed by a different word is diminishingly small. Additionally, longer contexts significantly outnumber the shorter sequences in the training dataset, especially when data and models work with ever longer contexts; this is progressively true in models with increasingly long context windows, such as Llama 3.1 (https://ai.meta.com/research/publications/the-llama-3-herd-of-models/). In the abstract, shorter (ambiguous) contexts could even be excluded or relegated as outliers in the data. Therefore, we conclude that it is likely not an issue overall.\n\nOn the other hand, we believe it would still be interesting to extend our work to incorporate soft labels and multi-label approaches. This could be explored using recent advancements in multi-label neural collapse (https://arxiv.org/abs/2310.15903) and mixup neural collapse (https://arxiv.org/abs/2402.06171). We hope that our work serves as a strong first step in modeling NC at the next-token level.\n\n> Is your setup only specific to causal language modeling? Particularly, a) Is any part of the formulation or results influenced by the causal/autoregressive training, or is it simply related to the different nature of language datasets in general, such as the large number of classes and the possibility of having multiple labels for a given training sample?, b) Do you expect similar behavior if the experiments were conducted using other language training methods, such as masked language modeling?\n\nWe appreciate the questions. We don’t believe our setup is strictly specific to causal language modeling.\n\n1. The adverse conditions we list in our Introduction are mostly due to the imbalanced nature of tokens in natural language data. Intuitively, they would apply beyond autoregressive modeling.\n2. We expect to see similar results in masked language modeling. However, it may depend on which tokens are frequently masked in the data (if a specific scheme is used). We hypothesize that one could conduct experiments with similar results on bidirectional encoders such as BERT or T5 (or their derivatives) but we imagine they’d present additional challenges such as lower token sample efficiency of the MLM paradigm, or the simultaneous prediction or fill-in of multiple dependent/correlated token predictions.\n\n> (Related to the previous question) What is classification error (line 130) in the context of next-token prediction and what do you mean by \"irreducible noise’’? How do you define the error for contexts that appear several times with different next tokens/labels?\n\nThe loss is token cross-entropy and the error we use is the average mis-classification rate over all tokens. We do not treat ambiguous samples differently and instead defer to the diminishing proportion argument previously made regarding soft-labels. Irreducible noise refers to some minimum loss and error that can't be reduced by more scaling simply because of soft-label contexts.\n\n> Intuitively, why do you expect that a model minimizing UNC3 has better generalization? Minimizing NC3 means the classifiers and mean vectors are aligned which is consistent with the classification objective. However, minimizing UNC3 (CoV of the term in Eq (8)), only implies the degree of misalignment between the classifiers w and mean embeddings μ is uniform across classes/samples. How does this connect to generalization?\n\nOur initial manuscript didn't include our reasoning behind UNC3 and we neglected to provide it in our Appendix, so we thank the reviewer for raising this oversight.\n\nThe goal of (self-)duality is to minimize the angles between mean vectors and their corresponding classifiers. This can be measured through the expectation (over class or token pairs) of the squared angle between vectors: $\\mathbb E[\\theta^2]$.\n\n$$\\mathbb E[\\theta^2] = \\mathbb E[\\theta]^2 + \\text{Var}[\\theta]$$\n\nNC3 measures the average angle $\\mathbb E[(\\theta)]$, whereas our newly introduced UNC3 measures the variances in the angles $\\text{Var}[\\theta]$. Achieving duality ultimately requires minimizing both terms.\n\nOnce again, we appreciate this comment and will add a short discussion on duality decomposition in $\\S$ 3.6 and $\\S$ 4.6.\n\n> Do you observe a decrease in the NC metrics and validation loss as you train the models longer (e.g., 1 vs 10 epochs)?\n\nYes. Appendices D, E, F, G, H, I, J, L, and M all show trends towards NC across training (left-to-right) in sufficiently large models. We show more explicit scatter plots (red = more parameters) in the 1-page PDF supplied in the general rebuttal. That PDF also includes validation loss with respect to training. We will also add these figures to the Appendix of our main manuscript.\n\n---\n\nOverall, we appreciate the constructive comments of the reviewer. Should our response address the concerns raised, would you consider raising the score? Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## The Choice of TinyStories

The study of NC in causal language modeling at the token level would be very expensive, so the motivation to use a small dataset is clear. However, most commonly used text datasets such as WikiText, BookCorpus, CommonCrawl, or most subsets from the Pile are much too complex and broad to be effectively compressed by CLMs of the scale that we work with.

WikiText-2 and WikiText-103 present significant drawbacks for our experiments. Both datasets contain a considerable amount of low-quality data that doesn't concentrate on essential linguistic structures such as grammar, vocabulary, facts, and reasoning. WikiText-2 has a similar empirical vocabulary to TinyStories under the GPT-Neo tokenizer (27K vs. 29K) but only has around 34K rows of training data compared to 2.1M in TinyStories. Our small-scale NC experiment on WikiText-2 revealed that the models were very brittle and prone to overfitting. On the other hand, WikiText-103 is comparably sized to TinyStories but utilizes around 44K unique tokens. Our CLMs trained on WikiText-103 struggled to produce coherent sentences, likely due to the excessive breadth and information, as noted by the authors of TinyStories. Beyond these two, we were unable to find any real datasets that both followed established scaling laws (Kaplan et al., Hoffman et al.) for CLMs at our scale and are simple enough to suit the analysis of NC.

This is where the TinyStories dataset becomes invaluable. Their manuscript (Eldan & Li) from last year informs much of our reasoning. This is an excerpt from their manuscript:

> We introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still **produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.**

According to its authors, it’s explicitly designed to preserve the essential elements of natural language, such as grammar, vocabulary, facts, and reasoning, while being smaller and more refined in terms of its breadth and diversity. Unlike large corpora that can overwhelm small language models (SLMs) due to their excessive breadth and diversity, TinyStories offers a concentrated dataset that hones in on core linguistic structures and reasoning capabilities. This is evident in its small word vocabulary, consisting of approximately 1500 words that a child would use, and in its 29K empirical vocabulary under the GPT-Neo tokenizer.

Despite its concentrated nature, TinyStories enables models trained on it to produce grammatically correct, factual, and reasonable stories. Additionally, these models can be finetuned on specific instructions found in the TinyStories-Instruct dataset. The authors of TinyStories also demonstrate that their models can creatively produce stories that are dissimilar enough to their training data, indicating a balanced capability for generalization and creativity.

One particular advantage of TinyStories is the small vocabulary relative to total training tokens. The consequence is a reasonable number of classes with higher average token counts. Conveniently, frequency analysis of the overall dataset produced a distribution (Figure 4 in first draft) that is similar to real human language. This is relevant because the ability to measure NC and a CLM’s ability to compress language data into distinct geometries depends partially on the ratios between embedding dimension, vocabulary size, and average token counts. TinyStories provides a good balance for an initial study in this phenomenon.

Additionally, TinyStories has more regular structure as GPT-3.5/4 were instructed to produce children’s stories with certain themes and forms with a limited vocabulary. We believed that this would reduce the amount of clustering noise from the very broad information and structures in real general data, and allow our smaller CLMs to exhibit clear trends towards NC.

Furthermore, TinyStories was created using GPT 3.5/4, which are advanced language models with significantly larger architectures trained on orders of magnitude more tokens, helping minimize the effect of the synthetic nature of the generated dataset. We also considered a possible effect of model collapse as a result of training on synthetic data but Shumailov et al. and some follow-up works suggest that a single iteration of data generation (as generated TinyStories) has a very negligible model collapse.

With all that said, we are deeply grateful to the reviewers who raised this issue. We will include the above explanation at length in our Appendix B and a summary in $\S$ 3.1 where we introduce TinyStories.

### References

- Kaplan et al. (2020): https://arxiv.org/abs/2001.08361
- Hoffman et al. (2022): https://arxiv.org/abs/2203.15556
- Eldan & Li (2023): https://arxiv.org/abs/2305.07759
- Shumailov et al. (2023): https://arxiv.org/abs/2305.17493

Meta-review: The paper explores the phenomenon of neural collapse (NC) in the context of causal language models (CLMs). The authors aim to understand how NC properties, which have been predominantly studied in well-behaved classification settings that are trained on noisy, unbalanced data with a large and often imbalanced vocabulary. The paper applies the concept of NC to a new domain, CLMs, which differs significantly from the classification settings where NC has been traditionally studied, which is novel and valuable. The authors provide a comprehensive empirical evaluation, demonstrating a correlation between the proposed NC metrics and generalization in CLMs. One main concern raised by the reviewers is the use of a synthetic dataset (TinyStories), which may limit the generalizability of the findings to real-world human-written language data.
Decision: Accept (poster)