Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
Authors: Nathan Godey
Éric Villemonte de la Clergerie
Benoît Sagot
Venue: COLM (2024)
Keywords: saturation, softmax bottleneck, anisotropy, pretraining, small language models
TL;DR: Language models relying on less than 1000 hidden dimensions tend to saturate in performance due to the softmax bottleneck.

Abstract: Recent advances in language modeling consist in pretraining highly parameterized neural networks on extremely large web-mined text corpora. Training and inference with such models can be costly in practice, which incentivizes the use of smaller counterparts. However, it has been observed that smaller models can suffer from saturation, characterized as a drop in performance at some advanced point in training followed by a plateau. In this paper, we find that such saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution. This mismatch affects the performance of the linear prediction head used in such models through the well-known softmax bottleneck phenomenon. We measure the effect of the softmax bottleneck in various settings and estimate that models based on less than roughly 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, which leads to reduced evaluation performance.

Published as a conference paper at COLM 2024
Why do small language models underperform? Studying LM Saturation via the Softmax Bottleneck
Nathan Godey!2, Éric de la Clergerie' & Benoît Sagot! Inria Paris, Sorbonne Université
Paris, France
aethan.godyyennria.f fr
Abstract
Recent advances in language modeling consist in pretraining highly pa- rameterized neural networks on extremely large web-mined text corpora. Training and inference with such models can be costly in practice, which incentivizes the use of smaller counterparts. However, has been observed that smaller models can suffer from saturation, characterized as drop in performance at some advanced point in training followed by a plateau. In this paper, we find that such saturation can be explained by mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution. This mismatch affects the performance of the linear prediction head used in such models through the well-known softmax bottleneck phenomenon. We measure the effect of the softmax bottleneck in various settings and estimate that models based on less than roughly 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, which leads to reduced evaluation performance.
Introduction
The representation degeneration problem is a common phenomenon that affects self- supervised learning methods used for textual data (Gao et al. 2019 Lai et al. (002),, among other modalities (Jing et al. 2022 Godey et al., 2024). This phenomenon consists in the emergence of degenerated structures in the intermediate latent spaces of language models throughout training. In particular, many observations on the intermediate representations of Language Models (LMs) have shed light on their low angular variability (or anisotropy) by showing that cosine-similarity between pairs of intermediate embeddings tend to be unexpectedly high (Zhou et al., 2021 Rajaee & Pilehvar (002)). Other works have identified outlier dimensions that emerged during training (Puccetti et al. (022). However, these observations were mostly made on relatively small-scale models of dimensions comparable to BERT (Devlin et al., 2019) or models from the GPT-2 family (Radford et al. (001)).
These models are usually composed of neural network fo that takes sequences of tokens (y<) E [1, vji as inputs and produces a relatively low-dimensional contextual represen- tation in IRd, where the hidden dimension of the model. They then rely on language modeling head that produces logits for contextual token probabilities. A common choice for the language modeling head is a linear layer with parameter W Vxd where is the number of possible tokens. The resulting next-token probability distribution is then given by:
p(y.) = (WWfoly)))
where o is the softmax function.
In the language modeling field, the current trend consists scaling up the generative pretraining approach introduced with GPT-2, which implies training neural models made of several billions parameters on gigantic web-mined text corpora (Brown et al., 2020
Touvron et al. 2023 Almazrouei et al. 2023: Jiang et al. 2023). However, training and

---

Published as a conference paper at COLM 2024
serving such highly parameterized models raises energy and hardware-related problematics, which motivates for looking into achieving similar performance levels with smaller models (Sardana & Frankle 2023).
Nevertheless, the evaluation of the Pythia model suite (Biderman et al. 2023) has shown that training small models on very large corpora could lead to saturation, the form of a performance degradation in late pretraining. In this paper, we explore this saturation phenomenon through the lens of representation degeneration, and find that both phenomena strongly correlate. We further demonstrate that representation degeneration strongly occurs in the language modeling head of small models, and we theoretically and empirically show how linear language modeling head can represent a performance bottleneck for architectures based on small hidden dimensions.
Overall, our contributions can be summarized as:
We characterize the performance saturation of small language models through evaluation and extrapolation of the scaling laws;
We find that the representations of smaller models degenerate concurrently with this saturation. We shed light on rank saturation, i.e. the explosion of the entropy of singular value distributions of small LM prediction heads;
We empirically verify that the rank of the target contextual distribution is usually high. Moreover, we observe that regardless of the expressiveness of the output representations of a model, linear head W substantially affects performance when rank(W) 1000;
We theoretically quantify the performance limitation induced by a low-rank linear language modeling head.
Related Works
Small LMs & Saturation Biderman et al. (2023) train Pythia, a suite of models of vari-
ous sizes 300B tokens from the Pile (Gao et al. 2020), and release the weights for exhaustive number of checkpoints during training. They notice that smaller models suffer
performance decrease on the Lambada dataset (Paperno et al., 2016) in late training. The scaling laws (Kaplan et al. 2020 Hoffmann et al. 2022) predict that training smaller models on large corpora suboptima in terms compute. However, recent initiatives (Zhang) let al. 2024 Faysse et al. 2024 Team et al. 2024)) have pretrained smaller language models
on large datasets, motivated by inference cost reduction (Sardana & Frankle 2023).
Softmax Bottleneck The concept of softmax bottleneck was introduced in Yang et al. 20018 where the authors show that model using hidden dimension inferior to the rank of the contextual probability matrix cannot predict correctly in every context. They then hypothesize that this rank is relatively high in natural language and propose alternative method for the predictive layer of language models. Subsequent works have explored negative effects of the softmax linear layer on language modeling performance (Chang &
McCallum 2022) and possible alternatives (Lin 2021 Kanai et al. 2008). We extend this line of work by quantifying the critical dimensionalities involved in the softmax bottleneck.
Representation Degeneration is a phenomenon in which pretrained models tend to adopt low-entropy singular value distributions (Jing et al.) 2022). In language modeling, represen- tation degeneration takes the form of anisotropy (Ethayarajh) 2019 Rajaee & Pilehvar 2021 and was proven to be related with the Zipfian shape of token distribution (Gao et al. 2019 Biš et al. 2021). We study this phenomenon along training and its relation with saturation.
Data Dimensionality and Performance Sharma & Kaplan 2022 link the scaling laws observed across pretrained models to data dimensionality, through the lens of Intrinsic Dimension (Camastra & Staiano, 2016). While they show that Singular Value Decomposition (SVD) is not suited for studying the dimensionality of the data manifold in the universal
2

---

Published as a conference paper at COLM 2024
approximation paradigm, argue that it is well-suited, to certain extent, when studying the performance of a linear classifier limited by the dimensionality of input representations.
3 Language Model Saturation
We first verify that we can indeed observe and quantify performance saturation for the Pythia checkpoints, as they are the only released intermediate checkpoints for a wide range of model sizes. We measure the cross-entropy of Pythia checkpoints 50k tokens randomly sampled from their pretraining dataset, ie. The Pile (Gao et al. 2020).
par 12
4.00
pa 256
4.25
Finel checkpoint checkpoint
Cross- antropy loss
Power law (tinal)
3.80
pa
3.75
entropy
3.60
3.40
poM 768
3.25
TXOM
400M
2.75
144
2.25
10a
1030
102t
1022
Estimated FLOPS
(a) Loss saturation
(b) Loss extrapolation
Figure Performance of Pythia models on the Pile. On the left, we compare training dynamics of models from 14M (top) to 410M (bottom) parameters, displaying darker shades as we approach the minimal value. On the right, we fit a power law on larger models and find that final checkpoints of smaller models underperform compared to predictions.
In Figure 1a we clearly see that models up to 410M parameters suffer from the saturation phenomenon, characterized as an increase of the in-domain loss in advanced training stages.
In Figure 1b. we fit a scaling law in the style of Hoffmann al. 2022 on data points from models ranging from 410M parameters, only optimizing for model-related constants (A and a) while reusing all other values (B 410.7, B = 0.28, 1.69). We recall the relation between parameter count N and token count T given in Hoffmann et al. 2022):
L(N,T) = N
+
T8
We find that optimal parameters are A = 119.09 and a = 0.246. We display the fitted curves for token counts that correspond best and final checkpoints. We observe that the final checkpoints underperform the extrapolation by 8% in average. The Joss-minimzzing (best) checkpoints, which are expected fall short of the extrapolation due their incomplete learning rate cooldown, only underperform it by roughly 4%.
similar performance saturation is also observed on datasets used for evaluation in the LM Evaluation Hareess (Gao et al. 2023), as shown in Table
Checkpoint
Lambada (ppl.)
Lambada
StoryCloze
Wikilext (ppl.) 4
SciQ 1
ARC-e
Best Final 32.9
24.6
40.3
59.6
30.47
79.6
46.5
38
57.2
33.4
73.4
43.2
Table 1: Zero-shot performance of 1yythia160MM best and final checkpoints on evaluation datasets. Unless specified, we report accuracy all tasks.
3

---

Published as a conference paper at COLM 2024
4 Performance Saturation is Rank Saturation
4.1 Anisotropy Scale
Anisotropy is common form of representation degeneration that has been observed among various small language models. It consists in reduced angular variability of the representation distribution at given layer. Previous works (Ethayarajh) 2019 Godey et al. 2024) notice that almost all layers of small Transformers language models are anisotropic. A common measure for anisotropy in set H of vector representations is the average cosine-similarity:
A(H) = |H|2 - [H|
Σ
M,jeHHiii 11hil12 1
However, it remains unclear whether anisotropy affects models with over billion parame- ters. In order to address this question, we compute average cosine-similarity intermediate representations across layers in suites of models; namely GP7T-2 (Radford et al. 2019), OPT (Zhang et al. 2022),, Pythia (Biderman et al. 2023), and Gemma Team et al. 2024). We use hypothesize that the domain of this dataset
subsample of The Pile (Gao et al. 2020), includes or matches the comain of the pretraining datasets used in these suites.
Avenge corne-zrmiin zimiant
Neerot cosne-dmirrr
0.-
0.0
0.2
0.8
2.5
0.5
1.0
0.0
0.2
0.
0.6
0.5
1.0
Layer
Layer
(a) Pythia
(b) GPT-2
1.0
109
0.0
parsmotess
0-6
0.
0.2
0.6
6.8
1.0
0.0
0.2
04
26
0.5
1.0
depth
depth
(c) Gemma
(d) OPT
Figure 2: Anisotropy in function of layer depth (i.e. order in the forward pass).
In Figure we observe that most layers of Transformers models are anisotropic some extent, regardless of the scale. Nevertheless, there seems to be a dichotomy in the last layer, where models are either nearly isotropic or highly anisotropic. Interestingly, we notice that the dichotomy aligns with the one of the saturation phenomenon for the Pythia suite, where only models containing 160M or fewer parameters seem affected by last-layer anisotropy.
We thus decide to study the training dynamics of anisotropy for the Pythia suite, and compare them with the saturation phenomenon in Figure
Fiiurre 3 illustrates neat correlation between the emergence of the performance saturation phenomenon and the appearance of anisotropy in the last-layer representations of the models. It also shows that anisotropy increases abruptly around the saturation point during training. Moreover, we see here that on specific in-domain corpus, the models quickly lose performance at saturation and never seem to fully recover from this explosion.

---

Published as a conference paper at COLM 2024
170
160
Wiltrra word
Jamn man
Witrert word rpl
145-
MLN
140
120
140
162
80
300
120
Baining staps 04
Training Stops
Taining steps 10)
(a) 14M
(b) 31M
(c)
70M
2.0
0.8
boger Argoorrs
Wilhott
20
40
60
80
150
120
140
25
50
75
100
125
Triiiing scepo
stops
(d) 160M
(e) 410M
Figure 3: Evolution of the language modeling performance on the Wikipedia test set from the LM Evaluation Harness (Gao al. 2023) and last-layer anisotropy of Pythia models along training (color).
4.2 Singular Values Saturation
Average cosine-similarity is valuable measure of the uniformity of a distribution, but in- cluding other metrics can help to better capture the complexity of some manifolds (Rudman et al.) 2022). Moreover, it only focuses on the output embeddings of the language models, and not on their weights. In this section, we extend our analysis by studying the singular value distributions of the language modeling heads, link our empirical observations to our theoretical findings. In Figure 4 we display the singular value distributions of the final predictive layer weights W along training.
320
0.8
100
20
0.
13
01
2 12200600455 09609483893
9. 4 0000000 2.9.00
(a) 14M
(b) 31M
(c) 70M
635
625
82
815
(d) 160M
(e) 410M
Figure 4: Evolution of the singular value distributions of the LM heads of Pythia models during training, normalized by the maximum singular value.

---

Published as a conference paper at COLM 2024
Figure 4[sheds light on specific pattern of spectral saturation, roughly co-occurring with the performance saturation phenomenon. shows that the singular value distribution pro- gressively flattens during training, and nearly reaches uniformity before abruptly evolving towards a spiked distribution with high maximal singular value, relatively to the other ones.
In order to quantify this behavior more accu- rately, we use singular entropy metric, computed as the Kullback-Leible divergence between the normalized singular value distribution and the uniform distribution.
KC.oolll
Figure 5 shows that singular distributions evolve differently for models using less than 410M parameters than for the larger ones. The heads of small models see their singular value distributions become increasingly uniform, up to a point where they degenerate abruptly, which again correlates with the LM performance drop. The singular value distributions of larger models tend to be more stable, and do not dis- play clear monotonic patterns throughout train- ing.
100
10e
125
Training steps (k)
Figure 5: Training dynamics of the singu- lar entropy, for different Pythia models.
The Softmax Bottleneck & Language Dimensionality
5.1 Inherent Dimensionality of Natural Language
Intuitively, the saturation of the singular values distribution observed only for smaller models in Section 4.2 questions the dimensionalities involved the optimization of the LM head. In this section, we propose to empirically measure critical value for the rank of the LM head, and to estimate the dimensionality of the contextual probability distribution the head's outputs are supposed match.
In order to empirically measure the effect of the rank of the linear head, we propose to train rank-constrained heads on pretrained contextual representations from seahll--prammttrizeed language models. In order to control the maximum rank r, we consider heads of the form W AB e RR where the coefficients of A E Rxr and B E RRrx are drawn from / (0, 1) (d being the hidden dimension of the model). The rank such W matrices is limited by the parameter E [1,d], which we sweep over wide range of values.
We freeze the language models and train the rank-constrained heads on their output rep- resentations roughly 150M tokens, while adjusting the learning rate to the trainable parameter count (more details in Amppendix B).
55
50
Finel val
3.0
45
40
Pythis-410m
35
Pythia-14 1b
Llame-7a
2.0
10a
10a
10?
Head bottlenack dimension
Head
(a) Accuracy
(b) Cross-entropy
Figure 6: Performance of several models the bottleneck dimension of the head increases.

---

Published as a conference paper at COLM 2024
we observe that perplexity starts to noticeably decrease when the rank of the language modeling head W is inferior to 1000, regardless of the model size. This hints that the head is not major performance bottleneck for models with greater hidden dimensions, but that it may hurt performance for models with smaller ones independently of the quality of the output representations.
In Figure 6
Another interesting factor to estimate is the dimensionality inherent to the data itself. To avoid possible effects related to specific inductive biases, we train naive 5-gram language models on several datasets of varying coverage (IMDb (Maas et al. 2011), Wikitext (Merity et al. 2016), and The Pile (Gao et al. 2020)), using two tokenizers of varying vocabulary sizes (30k tokens for Llama-2 and 50k tokens for Pythia). Given C observed 5-grams, we
consider the matrices W E RCxV where each row is probability distribution over possible tokens in a given 4-token context, and compute their singular value distributions, as in Terashima et al. (2003). In In[Figure> 7 we report W-error, the minimal approximation error Soantt-oun--iiii theorem (see Lemma 5.2)
W for a matrix of rank d as predicted by the
normalized by the Frobenius norm of W:
||aa4::11 ll2
W-error(d)
[|]
1.0
Wikieett
IMD6
0.6
0.6
0.2
0.2
0.0
0.0
5000
10000
15200
20000
25000
5200
10000
15000
20000
25000
(a) Llama-2 tokenizer
(b) Pythia tokenizer
Figure 7: W-error d increases, for different tokenizers and datasets. We observe that while W-error can be halved using 1000 or 2000 dimensions, it only becomes negligible after 10,000-15,000 dimensions.
We find that the estimated rank of W is nog-negligible with respect to the usual magnitude of hidden dimensions. In the next section, we analyze the connection between the dimen- sionality of an ideal linear language modeling head and performance from a theoretical perspective.
5.2 A Theoretical Bottleneck
In this section, we aim identifying a formal link between the inherent dimensionality of the contextual distribution and the performance bottleneck that can be attributed to the lower dimensionality of the output representations of language model. To that end, we conceptualize a language modeling head optimized on ideal contextual representations, and we explore the relationship between its spectral properties and the performance gap induced when training low-rank head on the same representations.
Let's consider set T of sequences (y:)ic(1.ly)) of elements taken from a vocabulary of size V, representing the pretraining data. We consider a function 4² that perfectly (e.g. in a bijective way) represents given context V-i single real vector of infinite dimension. As we do not focus on Ø², we can simplify the notations by introducing the contextual representations x = (y(
7

---

Published as a conference paper at COLM 2024
The task of the linear language modeling head can be formalized as optimization problem on the matrix W: W WeRVxN Vx20 yET
argmin Σ I(W,xx,
(1)
where C is the cross-entropy objective defined using the softmax function as:
((W,,,y) log(c((x)l))
In practice, neural language model Φ3 produces contextual representations x; (o(y<i) of dimension d e IN The linear language modeling head Wg RVdd is trained concurrently with do with the same objective as in Equation
We focus on the maximal expressiveness of a lower-dimensional head: when provided with perfect contextual representations x,, what is the maximal performance level of a linear language modeling head of maximal rank This question can be put in mathematical terms:
W argmin Σ C s.t. rank(W) S d WERVx YET
(2)
Lemma 5.1 shows that by approaching W directly, we can asymptotically expect to close the performance gap.
Lemma 5.1 (proof in Appendix A.1) Let's consider Rxoo M E H the matrix unit sphere for the Frobenius norm [[, and & R such that W W + eM. When e + 0:
[CW X L(W", ,1)l 0(s)
Hence, our problem is linked to a low-rank matrix approximation (Kumar & Schneider 2017), which has direct connections with spectral theory. In our case, can use the SokarrtYoung-Miee theorem.
Lemma 5.2 (Eckart-Young-Mins theorem) Let's consider (0;) the singular values of W in decreasing order, and Ma the set of matrices in R V xms of rank d < V = rank(W"). Then:
V Σ
min WaEMa
|[W, W
of
i=d+1
Combining all of the above yields Theorem 5.3
Theorem 5.3 (proof in Appendix A.2) Let's consider (0,) the singular values of W in decreasing order. Then, when d + V, the loss gap induced by d-dimensional bottleneck on the linear LM head follows:
(N Σ of 1=dd
Σ Ec(WS L(W" ((," 0 VCT
=
These properties shed light on how the dimensionality of the ideal language modeling head impacls the performance when the LM head low-rank. However, the relation obtained in Theorem 5.3 is not particularly strong, as discussed in Amppendix A.2
In Figure we compare the results of the head bottleneck experiment of the Pythia- model in SScciin 5. to the W-error on the head of the same model as the bottleneck dimension evolves. shows that the loss gap grows slowly with the W-error, implying that even when the allowed rank would lead to poor approximation of W, the performance can still remain acceptable. We notice that the performance starts decreasing when the W-error outgrows 0.6.
8

---

Published as a conference paper at COLM 2024
6 Discussion
One way to address the problem at hand could be to train shallow small language models, in- creasing hidden dimension at the expense of other hyperparameters, such as layer count or feed-forward dimension. However, we believe that such research directions may not be promis- ing in this context. Previous works have exten- sively explored and optimized the hyperparam- eter choices for various architecture sizes. The impact of width and depth has been extensively studied (Merrill et al. 2022 Tay et al. 2022 Petty et al., 2023), often showcasing the importance of depth in final performance and generalization capabilities.
3.8
3.6
w/ bottleneck
3.2
2.8
0.2
0.4
0.6
W-error
Figure 8: Final loss with trained rank- constrained heads (mimicking Wi), as function of the theoretical W-error for rank d the head of the Pythia-1B model.
Another possible way forward consists in imple- menting more expressive softmax alternatives (Yagg et al. 2018 Chang& & McCallum 2022) in the context pretraining small language mod- els on large datasets. We leave the exploration of such techniques for future work.
We also believe that further exploration of the specific nature of the singular components after the collapse we describe in Section 4.2 could improve our understanding of LM saturation. We hypothesize that the resulting dominating components are correlated with token frequency, based on previous works that link anisotropy with token frequency (Gao et al. 2019 Ethayarajh 2019 Bis et al. 2021) and show the importance of token frequency in the head mechanism (Meister et al. 2023)..
Beyond the scope of this article, we argue that our work demonstrates that last-layer anisotropy is symptomatic of performance saturation, and is thus likely not a desirable property of language models. We also advocate that this work paves the way towards better understanding of the structure of the contextual probability distribution, which could also enhance our interpretation of the scaling laws.
Conclusion
Small language models can be affected by performance saturation during training. We find that this phenomenon can be explained by an inherent difficulty in mapping a low- dimensional output representation space to high-rank contextual probability distribution through linear language modeling head. Indeed, we show theoretical link between the performance gap induced by smaller hidden dimension and the spectral properties of the contextual probability distribution.
We empirically confirm that the rank of such a mapping can be expected to be relatively high compared to regular hidden dimension choices. Moreover, we conduct experiments to measure the impact of constraining the rank of the LM head on the performance of large language model. Our results show that performance noticeably drops when using a hidden dimension smaller than roughly 1000. We further analyze the saturation phenomenon through the lens of spectral analysis and find that the emergence of last-layer anisotropy that only affects small models can be correlated with saturation. We also show that the LM heads of small models concurrently suffer from spectral saturation, i.e. uniformization of singular values that leads to a degenerated state.
Our work paves the way for better understanding of the consequences of the softmax bottleneck on language modeling, and for the conception of language models that better embrace the complexity of the target probability distribution.

---

Published as a conference paper at COLM 2024
Limitations
The main limitation of this article is the relatively small amount of saturated language models we studied. As is the only suite of language models trained in the range of interest to release extensive amount of intermediate checkpoints, we could only observe the training dynamics of small Pythia models. Although we observe strong last-layer anisotropy for the smallest GPT-2 model, we cannot tell with certainty whether it suffered from saturation. The OPI-125m model does not display a strong last-layer anisotropy, which could indicate that it was not affected by the saturation phenomenon.
Nevertheless, we argue that this paper does not show that all small models should suffer from saturation, but rather that the saturation of small language models is symptomatic of a limitation that may affect language models that are based on a relatively small hidden dimension. Furthermore, we do not state that there is a causality relationship between degeneration and low hidden dimension choices, but rather expose a strong correlation be- tween both phenomenon that can be explained through the prism of our softmax bottleneck analysis.
Another limitation of this work is the loose nature of the mathematical connection that establish between the dimensionality of the ideal language modeling head and the conk-constrained performance (cf. Theorem 5.3 Moreover, can also be argued that considering ideal x; representations is ill-defined notion. We argue that the reasoning behind Theorem could be applied to any contextual representations, as the ideal nature of x is not necessary in the demonstrations. The word ideal reflects that our observations hold for x representations obtained from any underlying model, to an extent that depends on the structure that these representations impose on the W matrix for a given training set T.
Acknowledgements
We thank Song Duong for carefully reviewing this article and for his valuable suggestions.
This work was funded by the last author's chair in the PRAIRIE institute funded by the French national agency ANR as part of the "Investissements d'avenir" programme under the reference ANR-19-P3IA-0001.
This work was granted access to the HPC resources of GENCI operated by IDRIS (allocation 2023-AD01001360RR1
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guil- herme Penedo. The falcon series of open language models, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models acToss training and scaling. In International Conference on Machine Learning, pp. 2397-2430, PMLRR 2023.
Daniel Bis, Maksim Podkorytov, and Xiuwen Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur,
Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Pp. 5117-5130, Online, June 2021. Associ- ation for Computational Linguistics. doi: 10.18653 /10///11222 wwwp://.aacmmmiin.noo URL https://aclanthhooogy org/2221 naacl-man weaec--main.ce 403
10

---

Published as a conference paper at COLM 2024
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Francesco Camastra and Antonino Staiano. Intrinsic dimension estimation: Advances and open problems. Information Sciences, 328:26-41, 2016. ISSN 0020-0255. doi: https:/ /doi. ogg///00.066/nnss2215....0000000 URL //ww:/.///wwww/ www.scedieeecctteececceeeeeeeeee D11/5002002555006617
Haw-Shiuan Chang and Andrew McCallum. Softmax bottleneck makes language models unable to represent multi-mode word distributions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8048-8073, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ w1t//////2222.aclloogg URL https:://allatthlllgg. http://ccaaaooooooorr//////222/
org www.8222.cccc2. acl 554.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653 (0.88633/v//NI/ 11/1//11111111. URL httpps://claanthologgy org/N19-1423)
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJJCNLP), pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ v1 10153//11119110000 DD19-1066 http::/t: httttss:/aclaathhll.oo org/D19-1006
Manuel Faysse Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Croissantllm: A truly bilingual french-english language model, 2024.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2019. URL httt:tttt////eeeeeeeee htt.lttee...eeeeoeeeeeeee/ee//or
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo. http://dd.rrrreerrrr///////////.
Nathan Godey, Éric de Clergerie, and Benoît Sagot. Anisotropy inherent to self-attention in transformers, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc,
11

---

Published as a conference paper at COLM 2024
Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. /tthss:::///pmm
Sekitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi. Sigsoftmax: Reanalysis of the softmax bottleneck. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Kesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc,, 2018 URL httpss:/pprcceeddingg metpeeddininnnnniiii.see. hct.iipscccpppprrrr files/paper/20 http:/20001//11edddccc/// 0tt.a///c0.......999999990000060
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXio, /200
2020. /tt////ttss//ppp...iaiiiccooooot wwwantisscoolar.ccomm org/
/appi
CorpusID:21086 1095
N. Kishore Kumar and J. Schneider. sureree on low rank of matrices. Linear and Multilinear Algebra, 65(11) 2017. doi: 10.008//000011007/201116777777
URL StLhhtts:: http:://ooooooo///// 1080000088 087. 2016. 1267104
Wen Lai, Alexandra Chronopoulou, and Alexander Fraser. Mitigating data imbalance and representation degeneration in multilingual machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14279-14294, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v/ 22///222...nmm. 1i22...inning-emnlpp org/2023. fiwdiigiiinnnsii cetegs--mnll.... 953
URL https://aclannthllun
Ying-Chen Lin. Breaking the softmax bottleneck for sequential recommender systems with dropout and decoupling, 2021.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Compulational I. Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June Associatoon for Computational Linguistics.
URL httphttp/////w.aci http://wwww.acceeeeeerrr//atttt/ wttpr//annnoooooo//111-1111
Clara Meister, Wojciech Stokowiec, Tiago Pimentel, Lei Yu, Laura Rimell, and Adhiguna Kuncoro. A natural bias for language generation models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 243-255, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: v0.1663/v1/20233.ad URL httpp:://aclatthllllge/ooo/ org/ 2223. acl short.22
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
William Merrill, Ashish Sabharwal, and Noah A. Smith. Saturated transformers are constant- depth threshold circuits. Transactions of the Association for Computational Linguistics, 10:843- 856, 2022. doi: 11162/taclla.0093. URL httppps//ac...atoooooo..oo//////
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 152-1133, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http:// /ww. www..aclweb.brrr wra.eb..og//ann htw../ww..gg//antooolg////111111
12

---

URL Published as a conference paper at COLM 2024
Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen. The impact of depth and width on transformer language model generalization, 2023.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell'Orletta. Outlier di- mensions that disrupt transformers are driven by frequency. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pp. 1286-1304, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL httiss//acclanthooog..orrr wtw.g///22...nnng-eeeeeee
.3
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Sara Rajaee and Mohammad Taher Pilehvar. A cluster-based approach for improv- ing isotropy in contextual embedding space. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meefing of the Associa- fion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Pp. 575-584, Online, August 2021, As- sociation for Computational Linguistics. doi: 11116653/11/221acc--s https://aclanthooogg..orgg http:///221....l.shehe acl-sh 73
Sara Rajaee and Mohammad Taher Pilehvar. An isotropy analysis in the multilingual BERT embedding space. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 1309-1316, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 21118653/v12022. findings-acl.103.
hattps://aacanttiiei wwwp:/y...rr//2222/..// findings acl. 103
William Rudman, Nate Gillman, Taylor Rayne, and Carsten Eickhoff. IsoScore: Measuring the uniformity of embedding space utilization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 3325-3339, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 20366633v11 http//12.2222iiiii// findings-acl.262. URL 2022. fww.ingsdiggasee acl. 262
org/
Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal Accounting for inference in language model scaling laws, 2023.
Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1-34, 2022. URL
http://r.rrrggpppp////////////// 1111
html
Yi Tay, Mostafa Dehghani, Dehghani.Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations, 2022. URL
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, l'aige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic
13

---

Published as a conference paper at COLM 2024
Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fermando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024.
Shiro Terashima, Kazuya Takeda, and Fumitada Itakura. A linear space representation of language probability through svd of n-gram matrix. Electronics and Communicatioons in Japan (Part III: Fundamental Electromic Science), 86(8): 2003. doi: wi com /ttp.//oo/oo///////01100000 10t.:/...000/....11.10.
1002/ecjc.10106. URL https: https:://onnneeeeee
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference Learning Representations, 2018. URL
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. Frequency-based distortions in contex- tualized word embeddings. CoRR, abs/2214.08465, 2021. URL tttps://arxxi / 2104. 08465
14

---

Published as a conference paper at COLM 2024
A
Proofs
A.1
Lemma 5.1
The proof is mainly based on calculations and limited development:
x, L(W", ,
(Xp(( 1y,)
exp((W x)),,)
log Ejev exp (WW;))) + log
Ejev exp (Wxx()
() + +log Σjev exp (xp((("'x)) x²)) (MMM)))))))) )₁)
Σjev expvexp( (W))
I Ljev (expe(xxxx))) (Mx;)))
() + +log
exp (W=x)); + +0(e)
expexx((M))))
<(Mx;) y
+o(e)
ΣjEV exp( (W*xx;))
Σ Σjev exp( (epp((M)) (Mx;)))
(Mx,) )v, +
+0(e)
Ejev exp( (W*x;);)
The continuous function M - | - yy; + Lee (Mx)))
is bounded on the compact
exp((W">)])
matrix unit sphere (i.e. where ||M]| = 1), which ends the proof.
This result could also be proven using differentiability argument, but prefer to display a more precise relation between the loss gap and the error on the W matrix approximation, stressing out its quasi-linear nature. This formulation will hopefully pave the way for further exploration of this relation in future works.
Remark
A.2 Theorem 5.3
Let us note Wa the best approximation of W of rank d with respect to the Frobenius norm. By definition of W we have that:
<( L(W", ,i YET
< E[W C(W",xi,y (3)
VET
The Sckart-Young-Mirske theorem tells us that when V,
|[a
Wi =
Σ 07 00
By defining e = Wa- - W we can apply Lemma 5.1 and show that:
(1)
IC Mi)
Σ
of
i-d11
From Equation (3) we have that:
(v
Σ C L(W',, ","",,,,,10 xx) =0 NET
L(W*
Σ
Σ
v
of
i-di1
15

---

Published as a conference paper at COLM 2024
By definition of W and W we also have that:
0 E C(W", xf VET
Σ ((W,, - L(W", xf, YET
which ends the proof.
Remark The bound used in Equation (3) can be rather loose in practice. We can think of no particular reason why approaching W directly should be the optimal way to minimize the loss on T. Hence, the presented result should be taken carefully, and we leave the refinement of such analysis for future work.
B Hyperparameters
B.1 Constrained head experiments (Figure
We freeze the pretrained weights in the Transformer layers, and we train each rank- constrained head (i.e. in the form W = AB with r as the inner dimension of the matrix product) for various values of GPUs for the Pythia models and A100 GPUs for Llama-7B. We use the hyperparameters from Biderman et al. ),023), except for the batch size which set to 256 as it fits our hardware setup better. As the trainable parameter count evolves with r, we search for the best-performing learning rates among values ranging from 1 10-3 to 5. 10-2.
150M tokens sampled from The Pile using 4 V100
We report the chosen learning rates in Figure
Pythia-410m
Pythia-1b Ulam-7bb
10-2.
Learning rate
H0 X X
10-3
32
256
512
768
1024
1536
2048
3072
4096
Rank constraint
Figure 9: Chosen peak learning rates used for the rank-constrained head experiments for each model.
16


Reviews:
[{"review_id": "0DaRhswHLi", "reviewer": "Reviewer_G2Lq", "summary": "This paper studies relevant properties that lead to performance saturation in small language models. A particular attention is guided towards the rank of linear layer. Through experiments and some theoretical justification, the rank of the linear layer casts an impact on explaining the limited performance of small language models.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for its highly valuable comments, and especially for its care about the theoretical aspects of the article.\n\n### 1. *The explanation does not lead to new methodology design*\nOur paper is framed as an extensive study of the connection between the softmax bottleneck and the representation degeneration issue. The drawn conclusions show that there is, to a certain extent, an intrinsic limitation to the performance a language model using a small hidden dimension can achieve. Designing a novel methodology to go around this limitation is definitely a path future works should explore, but as shown in the literature, it represents a difficult problem that probably could lead to a full paper on the subject if such a method was designed. We believe that describing this limitation in depth can pave the way for such methods, and represents an interesting contribution as such.\n\n### 2. *The theory seems not to add new insights about the phenomenon*\nThe goal of the theoretical section is to show that the connection between both phenomena is theoretically grounded. Hence, while we agree that we get no novel insight from this section, we argue that it strengthens our argument and shows its general nature.\n\n### 3. Mathematical issues\nIn a revised version of the paper, we adjusted and corrected some parts of the theoretical part. Notably, we removed lemma 5.1 as it is not necessary for the conclusion, we unified the notations and explained them in details from the start of the article.\n\nLemma 5.2 can indeed be proven by invoking the smoothness of the loss function, likely by using the $C^1$ nature of cross-entropy to argue that it is Lipschitz continuous, as regular continuity seems insufficient e.g. if the gradient in $W^*$ is undefined. We will add a remark in the proof to underline this. However, our proof relies on limited development, as we wanted to display a more precise relation between the loss gap and the error on the W approximation, and its quasi-linear nature. This formulation will hopefully pave the way for additional exploration of this relation in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study here is self-contained and explanatory with both empirical and theoretical evidences.\n\nPresentation is relatively good.\n\nAs discussed in the related work, the discovery of a tie between the softmax bottleneck to the performance saturation is new to the community.", "reasons_to_reject": "This study seems to explain a phenomenon, while the explanation does not lead to new methodology design. The discussion at the end of the paper is rather weak on the encouragement of more expressive softmax alternatives.\n\nThe theory seems not to add new insights about the phenomenon and the theory itself is very standard in a simplified model. The proof is not very rigorous though, especially the usage of small o and big O notations is not well explained.\n\nPresentation can be improved by providing a clearer connection between Sections 3 - 5.", "questions_to_authors": "Dimension of $W$ in Lemmas 5.1, 5.2 and Theorem 5.3 does not match with previous definitions.\n\nLemma 5.1 seems to be the definition of a global minimizer. Why there is the limit of $\\epsilon \\to 0$ in Lemma 5.2 (resp. $d \\to D$ in Theorem 5.3)? Isn't Lemma 5.2 simply the continuity of the loss function?", "extra_scores": {}}, {"review_id": "FXN5YtDNZ0", "reviewer": "Reviewer_sGwj", "summary": "The paper delves into a 'why' question concerning LM saturation through the softmax bottleneck. The key finding reveals that models with fewer than 1000 hidden dimensions, often representing the context, tend to adopt degenerate latent representations during late pretraining  characterised by a drop after a plateau, resulting in reduced performance. The evaluation of these findings was conducted empirically, and theoretical analysis indicates that the quantification of the performance limitation is induced by a low-rank linear language modeling head.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to reviewer sGwj for its time and effort, and for raising interesting points about generalization.\n\n### *Answering the why question in a general way*\nAlthough our paper has a strong empirical stance, we provide several elements in an attempt to prove the general nature of our observations:\n- An extensive analysis of the intrinsic rank of contextual probability manifolds (Figure 7), that allows an estimation of the dimensionality of target probability distributions in several natural language scenarii.\n- Rank-constrained experiments, performed on several models of different sizes and initially trained on different data (Figure 6).\n- A theoretical exploration (section 5.2) that proves that the connection between the softmax bottleneck and the achievable performance affects all language models trained using cross-entropy on a linear logit language modeling head, especially in the low-dimensional case.\n\n### *What would be the immediate impact of the study and analysis?*\nThere can be several immediate impacts, among others:\n- Including last-layer anisotropy monitoring when training language models to detect saturation;\n- Further exploration of mitigation strategies for the softmax bottleneck issue, specifically designed for language models, with potential performance benefits;\n- Incentivizing the use of larger hidden dimension choices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written for a knowledgable reader. It is difficult to answer a `why' question and the paper showcases a good attempt to characterise a learning problem and provides empirical justifications and theoretical analysis.", "reasons_to_reject": "It is very difficult to provide an answer/justification to `why' question with language modelling that can be generalisable. But that is the nature of the artificial neural network empirical research paradigm. We still don't know why small models worked well on some problems and did much worse on other problems. Even when it worked well on some problems, it might not have worked well for every instance of that problem.", "questions_to_authors": "What would be the immediate impact of the study and analysis?", "extra_scores": {}}, {"review_id": "n7YBbmXWjo", "reviewer": "Reviewer_vM6v", "summary": "The paper provides some experimental and theoretical evidence that the softmax bottleneck explains degenerate behaviour that has been observed when training LLMs with fewer parameters. In particular, it empirically shows that: \n1. as the input dimension $d$ of the final classification layer shrinks, training loss tends to worsen and plateau late in training, and\n2. as this happens, the average cosine similiarity between vectors in this layer suddenly spikes, and\n3. singular value distributions also become suddenly high entropy (that this corresponds to 2 is not surprising), and \n4. The the softmax bottleneck (where $d$ is much smaller than the number of classes $n$) theoretically relates to the observed problem with singular value distributions.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We warmly thank the reviewer for its thorough feedback.\n\n### 1 - *The writing of the paper is quite frankly a mess*\nBased on the feedback of other reviewers, our understanding of the writing issue rightly identified by reviewer vM6v is that the paper is hard to follow *for readers that are less knowledgeable in this specific subfield*. We thus propose improvements for the camera-ready version that should make the article more inclusive:\n- **The representation degeneration problem** : We immediately provide two references that explore this specific problem in-depth, and we proceed to explain what it consists in. We will insert an explanation between the first two sentences of the papers, to make the connection between representation degeneration and anisotropy clearer: ”It [the rep. deg. problem] consists in the emergence of degenerated structures in the intermediate latent spaces of language models throughout training. In particular, many observations…”.\n- **Low angular variability** : We agree that readability could be improved by expanding the phrase, which we will do in the revised version: “...have shed light on their low angular variability, by showing that cosine-similarity between pairs of  intermediate embeddings tend to be unexpectedly high.”\n- In general, realizing that the page limit was greater than 8 (which we initially thought) let us reorganize and expand some paragraphs further. Although we cannot share the new version because of the conference guidelines, we obtained positive feedback from readers on the writing quality of this revision. The theoretical section was also reorganized and detailed in a new version of the article. \n\n### 2. *I would be surprised if the relevant threshold wasn't dependent on other factors*\nThe models we experimented with in Figure 6 are: the 410m and 1B models from the Pythia suite, which were trained on The Pile with a vocabulary of 50k tokens, and the 7B Llama-2 model, which was trained on another unspecified dataset using a vocabulary of 32k tokens. We chose The Pile for the low-rank head training as it is expected to match the pretraining dataset of Llama-2 closely (and exactly for the Pythia models). \n\nIn all three cases, we see that the performance on The Pile validation data seems to noticeably deteriorate around a rank constraint of *roughly* 1000. Hence, we believe that the 1000 number is a good *estimation of the range* where the head rank starts being a significant in-domain performance bottleneck.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I don't work on this problem, but my outsider view is that there is enough here that it will probably be of interest to those trying to figure out how to build effective compact models. I think the evidence presented here is circumstantial (there might be other ways to explain all of these things happening at once), but that seems ok from a scientific perspective; other people might be able to follow up by exploiting these observations or by coming up with better explanations for them.\n- The figures are quite clear, and make good use of a color gradient to illuminate relationships between variables. Most of what I understood from the paper was due to the figures.", "reasons_to_reject": "1. The writing of the paper is quite frankly a mess. It appears to have been written in a big hurry, and some parts are difficult to follow, especially at the beginning. For example, at first I thought I would follow the paper pretty well, having understood the abstract, and having myself worked on the softmax bottleneck and on training dynamics. But I found the first paragraph (and much of the introduction) to be utterly incomprehensible, winging about obscure terms like \"The representation degeneration problem\" and \"low angular variability\" as if everyone working on LMs knows what these are. I think I followed the paper pretty well after a few pages in, after the central ideas were explained and it became clear that these terms were just needless technical mumbo-jumbo. But this is a problem, because as it stands, I think most readers won't bother reading past the first page if the paper remains in its current form. I certainly wouldn't have if I hadn't been reviewing it. The proof is written as a bunch of math with no intuitive explanation, including of how it connects to the rest of the exploration; you have to squint a bit to see the relationship, or else take it on faith. \n\n2. The paper makes an overly strong, unqualified claim that \"a linear head W substantially affects performance when rank(W) < 1000\". While this seems to be empirically true _for the models examined in this paper_ (which, as I understand, are all trained on the same data, only varying the number of parameters), I have a hard time believing that 1000 is some kind of magic number that is applicable in all circumstances. I would be surprised if the relevant threshold wasn't dependent on other factors like the size of the training data and vocabulary. I think this claim should be softened to account for these unknowns. \n\ntl;dr: the paper exhibits a severe case of the Curse of Knowledge (https://en.wikipedia.org/wiki/Curse_of_knowledge). Fortunately this can be fixed with better writing. I really think a rewrite is in order before it is published.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9isn337FmB", "reviewer": "Reviewer_gtfa", "summary": "The paper find that there are strong correlation between the following things\n1. Loss saturation (i.e., instability of the loss after training for a long time)\n2. Performance drop in the downstream tasks\n3. Anisotropy distribution in the embedding space\n4. Low effective dimension in the word embedding space (i.e., problem caused by softmax bottleneck)", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the very positive feedback from reviewer gtfa. \n\n## Reasons to Reject\n1. We agree that the causality is not perfectly established in the paper, as it is very hard to derive clear causality from such highly-parameterized models for which reproducing pretraining is rather costly and conducting ablations is rather costly. We never explicitly mention such a causality, and describe correlations wherever required in the paper. Based on your comment, we will include a causality/correlation statement in the Discussion section of our final version.\n3. As some other reviewers have pointed out, it may be tough for less knowledgeable readers to understand the paper as it is in its current form. As we described in the other answers, we have worked on a revised version of the paper in which the writing has been improved for publication to a broader audience.\n\n## Questions\n1. The purpose of Figure 1b is to show that if we extrapolate the expected performance of smaller Pythia models from the final performance of larger ones, we observe that models ranging from 14m to 160m parameters perform worse than expected. However, one could argue that extrapolating the larger models performance from the smaller models would lead to the opposite conclusion (i.e. large models underperform). Hence, we add another extrapolation (using the *best* checkpoints) to show that smaller models were closer to the performance predicted by the scaling laws mid-training, before the saturation phenomenon. We will add a sentence to make this clearer in the corresponding paragraph.\n2. In our revised version, Figure 7 is after section 5.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Disclaimer: I did some research in the related topics previously, so I would like to admit that my evaluation might contain some positive biases toward this research direction. Too few papers like this one that studies these fundamental and practical problems. I believe this kind of study is very important and won't be outdated in the near future. \n\n1. Overall, the presentation is clear and easy to read for me (maybe partially due to my prior knowledge in this field).\n2. The correlations are very strong. I reviewed several analysis papers for language models before and it is very rare that you can observe such strong correlations.\n3. The connection between loss saturation and anisotropy distribution / softmax bottleneck is a novel discover.\n4. The authors designed several good experiments and theoretical analysis to quantify the seriousness of the problems.\n5. Build the connections between several previous findings or research directions quantitatively, including softmax bottleneck, loss saturation, and anisotropy representation.", "reasons_to_reject": "I think the rejection reasons are fairly minor to me. You don't have to spend time writing the rebuttal. \n\n1. In the title, the why question seems to suggest this paper build causality relation between observations. It would be better to emphasize in the paper that correlation is not causality. In addition, it would be even better if the authors can find whether the loss saturation causes the anisotropy representation or the anisotropy representation causes loss saturation, and why.\n\n2. Except for the connection between loss saturation and anisotropy distribution, the rest of the empirical and theoretical analyses is somewhat novel but not super novel (although showing them quantitatively is pretty useful and the execution is done very well).\n\n3. I am not sure how easy to understand this paper for the researchers who do not study related problems before.", "questions_to_authors": "Small presentation suggestions:\n\n1. The purpose of Figure 1 (b) is not very clear to me. Maybe you could explain more about what we can learn from this result. \n\n2. When I first read Figure 7 and the definition of W-error(d), I don't understand what you tried to analyze. I understand it after reading Theorem 5.3. Thus, I suggest to move Figure 7 and its corresponding explanation to the end of section 5.2 (after your theoretical results).", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers highlight that this paper makes an empirical and theoretical account linking {correlation between loss saturation, anisotropy, effective low rank of the embedding matrix}. There was some concern about clarity and "actionability" which (to my reading) authors adequately addressed in rebuttal period. In short, although this doesn't propose new methodologies, it identifies a theoretical and empirical problem (as well as showing ways for identifying it in models), which opens up a field a future innovations.
Decision: Accept