Superposition Yields Robust Neural Scaling
Authors: Yizhou Liu
Ziming Liu
Jeff Gore
Venue: NeurIPS 2025 oral ()
Keywords: LLM
superposition
neural scaling law
Primary area: deep_learning
TL;DR: Neural scaling law in LLMs is explained through representation interference due to superposition

Abstract: The success of today's large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decreases as a power law with model size, remains unclear. We propose that representation superposition, meaning that LLMs represent more features than they have dimensions, can be a key contributor to loss and cause neural scaling. Based on Anthropic's toy model, we use weight decay to control the degree of superposition, allowing us to systematically study how loss scales with model size. When superposition is weak, the loss follows a power law only if data feature frequencies are power-law distributed. In contrast, under strong superposition, the loss generically scales inversely with model dimension across a broad class of frequency distributions, due to geometric overlaps between representation vectors. We confirmed that open-sourced LLMs operate in the strong superposition regime and have loss scaling inversely with model dimension, and that the Chinchilla scaling laws are also consistent with this behavior. Our results identify representation superposition as a central driver of neural scaling laws, providing insights into questions like when neural scaling laws can be improved and when they will break down.

Superposition Yields Robust Neural Scaling
Yizhou Liu, Ziming Liu, and Jeff Gore Massachusetts Institute of Technology {liuyz, zmliu, gore)&mit. edu
Abstract
The success today's large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decreases as a power law with model size, remains unclear. We propose that representation superposition, meaning that LLMs represent more features than they have dimensions, can be a key contributor to loss and cause neural scaling. Based Anthropic's toy model, we use weight decay to control the degree of superposition, allowing us to systematically study how loss scales with model size. When superposition is weak, the loss follows power law only if data feature frequencies are power-law distributed. In contrast, under strong superposition, the loss generically scales inversely with model dimension across broad class of frequency distributions, due to geometric overlaps between representation vectors. We confirmed that open-sourced L.MMs operate in the strong superposition regime and have loss scaling inversely with model dimension, and that the Chinchilla scaling laws are also consistent with this behavior. Our results identify representation superposition as central driver of neural scaling laws, providing insights into questions like when neural scaling laws can be improved and when they will break down
Introduction
The remarkable success of large language models (LLMs) has been driven by the empirical observa- tion that increasing model size, training data, and compute consistently leads to better performance [14. Across wide range of tasks including language understanding [15 6), math [7-10,, and code generation [T 12 larger models achieve lower loss, higher accuracy, and greater generaliza- tion abilities 13) This consistent trend, known neural scaling laws, has been observed across multiple model families and architectures, fueling the development of increasingly large models 24 These scaling laws have not only shaped the current strategies for building better models but have also raised fundamental questions about why such simple and universal patterns emerge in complex learning systems.
The pawer-law loss with model size plays a central role in both the practical design and the theoretical understanding of large-scale machine learning systems, yet its origin remains inconclusive 3 14|25 Various explanations have been proposed, drawing from statistical learning theory and empirical phenomenological models, including improved function or manifold approximation in larger models [14 [1]. and enhanced representation or skill learning in larger models [19222]. In the limit of infinite data, many of these explanations predict power-law decay of loss with model size, provided the underlying data distribution also follows power law. The scaling exponents are sensitive to the properties of the data distribution. Moreover, the connection between these mechanistic explanations and the behavior of actual LLMs needs further exploration.
'Code available at ott.eerr//gttt..ooooo.o..t/ttpps
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

a
Rep. 1
Rep. 1
Superposition
Hidden space
No superposition
Rep. 2
Rep. 2
b
Weak superposition
d
Strong superposition
Actual LLMs
100
100
-1.0
Loss
Feature
Loss
importance:
10-1
Linear decay
1
Power law decay
Exponential decay
10-2
10-1
101
102
10
10
102
10a
104
Model dimension,
Model dimension,
Figure 1: Superposition leads robust and fast power-law loss decay with model size. (a) Illustration of superposition where a three-dimensional space can at most represent three features without any interference (overlap). (b) Toy model results the regime of weak superposition, where we set data dimension = 10240 (number of features). The toy model will be introduced; more details are in Appendix D.I (c) Illustration of superposition: there are more features than the dimension of the space. (d) The same toy models in the strong superposition regime show lower losses, which power laws with model dimension and have exponents close to (color coding same as panel b). The gray points from actual LLMs, which have similar power-law exponent near 1.
When considering LLMs specifically, becomes clear that representation or embedding can limiting factor, which is closely related to phenomenon called superposition [26 27 yet this aspect has not been thoroughly studied. LLMs must learn embedding vectors for tokens, process these representations through transformer layers predict the next token, and use final projection (the language model head) to generate the output. Conceptually, fitting functions or manifolds and learning skills grammars primarily tasks of the transformer layers, while representation is more directly tied to the embedding matrix and the language model head. To represent more than fifty thousand tokens
abstract concepts within hidden space of at most few thousand dimensions, the quality of representations is inevitably constrained by the model dimension or width, contributing to the final loss. Although models can represent
features than their dimensionality would suggest through mechanism known as superposition (27), prior works on neural scaling laws seem fall the weak superposition regime implicitly [15-20], which may be less relevant the regime where LLMs operate. This gap leads us to study
Question: How will superposition influence the loss scaling with model dimension (width)?
Varying the degree of superposition and data structure, when is the loss power law? And if the loss is power law, what will the exponent be?
We adopt a toy model construction similar to 27 to study how superposition affects neural scaling laws. the toy model, representations are learned by recovering data, each composed of multiple latent features. These features in data have different frequencies of occurrence, reflecting their relative importance. Weak superposition means that only the most frequent features are perfectly represented, while the others are ignored. As illustrated in Figure Ta. the first three of six features are represented

---

Toy model of representation learning via data recovery
Hi Iis
hidden
data space
space
weight matrix
data space y
T
h
W
data dimension n
weight matrix
m. model
dimension
Loss
<lly x
ReLU(WW
h
W
y ReLU(WW"
b No superposition
wwT
C Superposition
WWT
1.0
W1
m
0.
W1
05
0.0
-0.5
W3
-10
W
-10
Freq.
W
Freq.
W2
Rank
Rank
Figure 2: Toy model of superposition. (a) Architecture and loss of the toy model. (b and c) A row of the matrix W, denoted by W,, is the representation of feature i. (b) No superposition represented the most frequent features, i.e., the first three (n = 6 features in m = 3 dimensional space), without interference. In the frequency- the ith row vector's 's norm ||W]|2. (c) With superposition, features are all represented, while the representations W; overlap.
plot, height means feature i's frequency Ps, and color means
in the diree-dimeniioall space without interference, and the remaining three are omitted. We find that the weak superposition regime, the scaling of loss with model dimension depends sensitively on how feature frequency decays with rank: the loss follows power law with model size only if the feature frequencies themselves follow power law, provided that m is sufficiently large (Figure By contrast, strong superposition allows many more features to be represented, albeit with overlap in the representation (Figure [ In this regime, the model displays a robust behavior: loss scales inversely with model dimension across different data frequency distributions (Figure Remarkably, we find that actual LLMs follow similar scaling. We summarize our contributions as
Main results/messages
Loss in the weak superposition regime depends summing frequencies of ignored features, which is a power law frequencies follow power law. In the strong superposition regime, loss arises from the interference between representations and can have robust "one over width" scaling because of the geometry.
LLMs exhibit strong superposition and agree quantitatively with toy model predictions.
The rest of the paper will elaborate on the takeaways. In Section we introduce the toy model, describe the data sampling procedure, and explain how we control the degree of superposition. Section 3 presents the detailed results. In Section 4 we compare our findings to related works. Finally, Section summarizes our conclusions and discusses limitations and future directions.
2 Methods
To understand the relationship between superposition and data structure, we need a toy model to represent data features simple enough yet not simpler - two key principles need to reflected, (i) there are more features to represent than the dimension of the model, and (ii) features occur in data with different frequencies. Later, we will discuss how the loss due to representation studied here may affect the overall final loss in LLMs.
3

---

We adopt the toy model of superposition from Anthropic 27 (an autoencoder) with minor modifica- tions (Figure 21) Input z E R" is vector with data dimension being the number of atomic (or irreducible) features. Each element 25 in a: is interpreted as the activation of this sample at feature i, which follows
J; = U, U3 Bernoulli(p:) & v; ~ 2O,
(I)
Here, u, sampled from Bernoulli distribution controls whether the feature i is activated, and uy sampled from a uniform distribution controls the activation strength once feature is activated. All samples are ii.d. The frequency of feature to appear in the data is p,. Without loss of generality, we make the indices of features the same as their frequency or importance rank. The data structure is then about how p; decreases with rank i. The expected number of activations in one input will be referred to as activation density: E' = Ex= P;- The model learns hidden representations by recovering the data, which cannot be done perfectly because the model dimension is much smaller than the number of possible features in the data n. The trainable parameters are a weight matrix W E x and bias vector b R". The weight matrix embeds data into hidden space with dimension m, h = W x, with m < nt. In practice, we fix n as a large number and change the model dimension m. We use W read out the embedding, where y ReI.U(Wh b). The loss is defined as the difference between the recovered y and the original x, L = {lly - 2: =112)a, where ()) means average over distribution.
We can now formally introduce superposition. Note that W is the representation of feature in the hidden space, where we use W; to denote the ith row of the W matrix. We emphasize the following
Key concepts
Feature frequency: p: is the probability that feature is activated (non-zero) in a sample, which is assumed to decrease with i.
Sparsity: We say features are sparse when E/n is small.
The feature is represented (in the hidden space) when W; non-zero.
No superposition ideally means the first m rows of W form an orthogonal basis (i.e., the first m most important features represented perfectly) and the rest of the rows are zero (i.e., the rest of the features ignored or lost), as illustrated in Figure 2p. Superposition means that there are more than m rows in W with non-zero norms (Figure 2+)
We next summarize important facts of this toy model 27
Preliminaries
Superposition cannot lead to lower losses in linear model (without ReLU function) due to the large amount of interference. ReLU and negative biases can cancel off interference, realizing error correction. With this non-linearity, superposition can be preferred when feature frequencies are more even (better not ignore features) and features are sparse in data (error correction is easier).
We can see that in Figure where features are sparse in data, the losses in the strong superposition regime are indeed much smaller than those in the weak superposition regime across several feature frequency distributions.
one regime is more preferred, we want to approach it more quickly in training. it is not preferred, we also want to study the scaling behaviors scientifically in that regime. To this end, we introduce decoupled weight decay (or growth) term in training to tune the degree of superposition:
Wi, W.,, 7 O, 1), 7 < <,
W.++1
We, W,,
where 7/ is the learning rate and Wi,e is the ith row of the weight matrix at step f (vector operations are
element-wisst For weight decay 7 < O. the update corresponds to gradient descent on (/)W.,llla -1), 1)2, encouraging unit-norm rows. We implement this weight decay in AdamW (8 optimizer with warm-up and cosine decay learning rate schedule (details in Appendix At each training step, we sample new data.

---

b
Weight decay +
Weight decay
2.0
1.0
Density
104
10a
Rep. fraction, $122
Data
m/n
1.8
10
100
1.6
10g m
0
0.5
Row norm
Row norm
1.4
Freq,
10
10-1
1.2
10-3
10-3
0.0
1.0
10a
103
10
103
1
Rank
Rank
Weight decay, Y
Figure 3: Weight decay can tune the degree of superposition. (a) Positive weight decay ( = 1 in the figure) has |W near 0 or 1, with frequent features more likely to be represented (color means |[W]l22 in frequency-rank plots). Negative weight decay (y = -)) has |Will around 1. We show results when a 1, decays lead to strong superposition, and large weight decays lead to no superposition (#1/2 s m m/n). More data Appendix D.3
100, yet the claim is generally true. (b) For all models, small weight
We find that the weight decay can robustly control superposition. We first see that important features tend to be represented (associated ||Will2 > 0), and norms of W; become bimodal, clustering near or (Figure 3R). This allows us to define the fraction of represented features as
1/ = i [Will2 > 1/2)//m,
namely, the fraction of rows with norm larger than 1/2 We found that weight decay can tune superposition for all models we trained, with small weight decay giving strong superposition, i.e., 11/2 2 m/n, and large weight decay corresponding to weak superposition, i.e., Φ1/2 ~ m/n (Figure 3p). The ability of weight decay to tune superposition is robust feature frequency distributions (Appendix D.3). We can then systematically study scaling behaviors in different regimes.
The toy model differs from LLMs in architecture, data, and loss. Since focus representations rather than next-token prediction, we omit transformer layers. Conceptually, LLMs map a document to a token, with inputs and outputs in different spaces, while the toy model operates within single shared space. Despite this, the toy model captures key aspects language structure through engineered sparsity and feature importance, making its data structure aligned with that of LLMs at high level. While LLMs use cross-entropy loss and the toy model uses squared error, we can show that this does affect the scaling behaviors (Appendix A.2). Thus, the toy model suitable abstraction for studying representation-linie scaling.
3 Results
For a systematic scan, we set p; x 1/12 in this section and can vary the data exponent a to change affect the
how skewed p; is The activation density E scaling (Appendix D.4p. We fix data dimension n = 1000, vary model dimension m from 10 to 100, and sweep weight decay y from 1 to 1. We fit final test losses as a power law, L x 1 1/m0m and call O. the model exponent. More details on hyperparameters are in Appendix B.2
whose value be shown
3.1 Weak superposition
We seek to understand when loss follows a power law with model dimension, and what determines the exponent when it does in the weak superposition regime. Consider an idealized case where the top $1/2n most frequent features are perfectly represented (no overlap), where /1/2 2 the fraction of
In theory, we should use as threshold. choice, 1/2, may minimize misclassifications since norms are near 0 or 1. Our result is robust to this threshold since norms are very concentrated. "The word phrase frequency in natural language follows Zipf's law, which is power law (a 1).
5

---

b
2.0
1.0
10
Model exponent, am
Dlaa X
Weight decay, V
1.5
1.0
0.5
10
Loss
exponent a
1.0
a-1 1
0.0
10"²
0.5
a
Data
0.5
0.5
Theory
(Xm
10-3
0.0
0.0
1.0
100
101
102
1.0
2.0
Number of represented
Data exponent,
Figure 4: Loss at weak superposition can be well described by the frequency sum of ignored features. (a) Observation and theory weak superposition (i.e., Equation (4) function of number of represented features, #1/2 agree when weight decay 7 is positive. (b) For those closest to the ideal no superposition case, expect am standard errors. Details in Appendix D.5
1, which is close measured values. Error bars
represented features. The optimal biases are b; 0 for i S 01/221 and b; = (xra) for i 01/2 The loss can then be written as:
27
Σ
Σ ((0 :p: p (1
Σ
(4)
i>01/an
1201/28
1201/27
The last approximation is right when pı < for >> / and pt terms are negligible. We use the definition of u, where 2). giving (u?) 4/3. We to estimate the summation, yielding an expression that depends on the number of represented, $1 and the data exponent, a (Appendix D.5). We find that, in the weak superposition regime, the actual losses closely match this prediction (Figure 4b). Focusing on cases closest to the ideal no- superposition scenario, where 01/22
the integral Joyan p.di
and the first features such cases occur when a >> 1 and yield yield model exponent am ~ a (Figure 4 This matches the theoretical expectation that f p.dí oc m -a+1 when >>>> m and > 1. Thus, in the weak superposition regime, loss scaling well described by the contribution of unlearned features, that is, the total frequency of features not represented by the model.
represented, we observe that
We can now answer our Question the weak superposition regime.
Result 1: "Power law in, power law out" in the weak superposition regime
The loss is governed by sum of frequencies of less frequent and not represented features. Ideally, there are model dimension m most important features being represented. feature frequencies follow power law, P; x 1/1" with 1, the loss or the summation starting at m will power law with m with exponent a
This finding of the specific toy model agrees with previous works with very different settings [5] where some power-law skill importance or spectrum is assumed.
3.2 Strong superposition
We next turn to the strong superposition regime. Consider case where only feature j is activated. The output y; has activation ~ W, W,, leading to a loss that scales squared overlaps (W; W2) due to the definition of loss. The loss arises from the non-zero overlaps between representation vectors. We cannot solve the weight matrix W in this regime. The section goes back and forth between theoretical ansatz and experimental observations to understand the high-level behaviors.
We start by considering relatively even feature frequencies, where trained W, are expected to be isotropic. One simplest theoretical ansatz of isotropic vectors is i.i.d. vectors uniformly on the unit sphere. In 1m the squared overlap of two such random vectors follows Beta( I m=1 distribution,

---

Var(squared overlap)
Density
2.5
Ratio random case
1.0
2.0
1.5
0.0
0.6
0.8
1.0
1.2
x Axponeeil a
b
Row norm
0.5
1.0
10
Freq.
norm
0.5
10-3
0.0
0.0
10
10
102
10
10a
102
Rank
Model dimension, m
2.0
1.0
Mean square overlap
10
1/m
Model exponent, amm
2(0 - 1)
eaec e exponent, a
Weight decay, V
1.5
2
0.5
1.0
0.0
Om
0.5
-05
10-2
0.0
0
1.0
10a
102
1
2
Model dimension, m
Data exponent, a
Figure 5: Loss scaling at strong superposition is explained via geometry. (a) The row norm distribution bimodal around 1. (b) The more frequent the features are, the more likely their norms are greater than 1. (c) Variance of squared overlaps for features with ||Will2 is smaller than that of random unit vectors, i.e., 2(m-1)
+2) Overlaps are calculated using directions Wa/l|Wall2- We show the measured variances (7 -0.55) divided by the above theory value for random vectors. (d) The features with ||Walla > 1 have 1/m mean squared overlaps, where we plotted all the data when < O. (c) At strong superposition (ry < 0), am = 1 if feature frequencies are flat (a small) due to isotropic vector geometry. But am 2(a 1) if the feature frequencies are skewed (a large). Error bars are standard errors. More details Appendix D.6
and therefore has mean 1/m and variance vn [m= 2/m². The squared overlaps for isotropic
2(m-1)
random vectors typically obey 1/m scaling.
The actual trained W; have structures whose norms are bimodal near (Figure 5 and more important features tend to have larger vector norm (Figure We want understand how such structure will change the scaling of overlaps. It turns out that for better error correction (using bias cancel interference), the model needs minimize the maximum overlap rather than the sum of squared overlaps. Consider y unit vectors E 1 with v 2 m. It can be shown that 29
V m
max W |
K.
(5)
m(v 1)
The lower bound, k 1/m when v > m. The bound is met when the vectors form an equal angle tight frame (ETF) 330 32] which has variance in absolute overlaps and appears in contexts such as quantum measurements
(36 and neural collapse [3 38). ETFs real spaces can only exist We find that the W; with ||Wll associated with important features tend be ETF-like (Figure 5) and d): the variance squared overlaps smaller than that of random vectors and can be near 0 for even feature frequencies (small a); the mean of squared overlaps collapse on 1/m Rs x Being ETF or ETF-like can help error correction and reduce loss values, but would not change the typical scaling with if the number of vectors is much larger than m.. Similar to ETFs, whose number of vectors is bounded, the number of vectors W, with Will m² /2 (Appendix D.6,, and the less important features tend not to be represented (norm lower than 1).
< m2(m+1) 30 32
around
7

---

Mean square overlap 8
b
wikitext
bookcorpus
10-1_
Loss (model part)
100
c4
pile
2.0
10
1.0
10-3
10-1
00.91#0.04
opt
Qwen gpt2 pythia
10"4
10-4
10-3
10a
10a
Inverse model dimension, 1/m
Model dimension, m
Figure 6: Superposition may explain the neural scaling law observed in actual LLMs. We evaluate four open-sourced model classes, Opt [39]. GPT2 [40], Qwen 41,, and Pythia 22
which have
model sizes from around 100M to 70B (evaluation details in Appendix C). (a) We found the mean square overlaps of W:ll|W;l roughly follow 1/m scaling, where W is the language model head. (b)
The model class is reflected by color as panel a, while we use shapes for evaluation datasets 443
The loss related model size is fitted power law, yielding empirical am 0.91 + 0.04 close to 1. More analysis in Appendix D.7
Vectors of these less important features cannot be explained with simple theoretical ansatz. Yet, combining the lessons from random vectors and ETF, expect the squared overlaps to scale 1/m robustly for isotropic vectors (confirmed in Appendix (D6). Considering all the overlaps when feature frequencies are even, then predict the loss to scale 1/m, which is true (Figure 5e).
When the feature frequencies are skewed (large a), we find that the model exponent O'm increases with and becomes greater than 1. Vectors being non-isotropic, i.c., important features having much smaller overlaps, may lead to larger a'm- To illustrate this idea, we conjecture an extreme situation where the m² /2 most important features can be ETF-like and contribute negligible loss compared to the less important ones. the worst case (Appendix A. the less important features lead to loss proportional Ei=m /2 ~ m 2(a-1) i.e., O'm = 2(xx - 1), which is close to observations (Figure 5b). The real configuration of W, is more complicated than this simple conjecture, requiring advanced future studies on when a'm loses robustness and how it depends feature frequencies sensitively then. To conclude the section, we have
Result 2: Geometric origin of 1/m loss scaling (a.m = 1) at strong superposition
For even feature frequencies, vectors W; tend to be isotropic in space with squared overlaps scaling like 1/m when n > m, leading the robust 1/m power-law loss. For skewed feature frequencies, representation vectors are heterogeneous in space, making loss sensitive feature frequencies, where might need power-law frequencies have power-law losses.
3.3
LLMs
Finally, we explore how findings might be relevant real LLMSs 39.49 we treat tokens as atomic features, with data dimension n equal to the vocabulary size. The model dimension for LLMs is known. We analyze the language model head, denoted by the weight matrix W. Through the norm and interference distributions of the rows of W, we claim LLMs are in superposition (Appendix D.7p. we measure token frequency. it follows power law with exponent close to 1 (Appendix D.7j. We conclude, based on the knowledge from toy models, that LLMs operate in superposition regime, and expect loss to related squared overlaps ~ 1/m. We next calculated the mean squared overlaps of normalized rows W W/ 2, and found they roughly obey 1/m scaling (Figure 6p). We argue that cross-entropy loss, given that the overlaps are small in absolute value, can be expanded and approximately scales as the mean square overlaps (Appendix A.2). We therefore expect the loss of representation-limnte L.MSs to have 1/m scaling. I.LM losses are close to linear function of 1/m (Appendix D.7). Yet when + the extrapolation of losses does not
As naive mapping,

---

hit O. The non-zero intersection can be due to intrinsic uncertainty in language. Increasing model sizes decreases "wrong" interferences but cannot eliminate uncertainty in the data. So, as in previous papers where loss is decomposed into model size part, dataset size part, and a constant 3 loss values by the following,
we fit our
Cmm/m/m 9m Lmm>
(6)
where the model size part Cm/m²= is universal (model size is a function of m), and A.m contains loss irrelevant model size, depending the evaluation dataset and model class. The fitting yields
am = 0.91 + 0.04 (Figure(ōb) We inferred from the Chinchilla models [3] that due to model size N m 2.2.52.. 2....0.03 (Appendix(D.) am (2.52+0.03) x aN 0.88 where aN 0.35 47 is the power-law exponent of loss with model size. The exponents am from LLMs are close to 1.
We highlight the finding as
Result 3: Superposition is an important mechanism behind LLM neural scaling laws
LLMs operate in the strong superposition regime. The squared overlaps of token representa- tions scale as 1/m, token frequencies are flat (a = 1), and the model size relevant loss scales closely to 1/m, agreeing with the toy model prediction.
4 Related works
Neural scaling laws were first characterized empirically (2]. demonstrating that for LLMs, the cross- entropy loss improves predictably as a power-law with increased model size (parameters), dataset size, or compute, over multiple orders of magnitude. This finding is built on earlier observations (e.g. (48) that deep learning performance scales in smooth power-law fashion with data and model growth. Many works showed the surprisingly universal nature of such scaling behaviors across architectures and tasks [244), directing further development of LLMs.
Several heuristic toy models have been proposed to explain neural scaling laws. One common view is that models aim to fit data manifolds or functions, and the scaling exponents depend strongly on the structure of the data [1,15] Another group of models assumes the network learns discrete features or skills 19 20 whose importance follows a power-law distribution, giving results the same as ours in the weak superposition regime. One toy model predicts that loss scales inversely with model width (25, arguing that parameters independently perform the same task with noise, and the scaling follows from the central limit theorem. However, this model applies in the overparameterized cases and may be less relevant to LLMs.
More formal approaches rely on similar heuristics. The scaling behavior depends on how the dataset size and model size approach infinity. When the dataset is fixed and model size grows to infinity, the system is variance-limited, and loss scales as 1 1/m by central limit theorem arguments [15). When the dataset size grows infinity first, the loss scaling enters the resolution-limitee regime. In linear models or kernel methods, this leads to a. = a [15 seemingly consistent with our weak superposition regime. Here, a' is the exponent of the power-law decay of kernel eigenvalues, which can be seen abstract feature importance. Considering neural tangent kernels, a' depends on both data and the model configuration. Our work may be framed as mechanistically showing that a' = a (a is the intrinsic data exponent) when models have no superposition, and a' is something else when models have strong superposition, which is new. The iooouution-limieen regime has also been described as fitting the data manifold [15].
Our toy model is based on Anthropic's model of superposition 27 (an autoencoder), with modifica- tions to the data sampling. The original study explored how data structure influences superposition but did not explicitly control it. Related models have appeared in compressed sensing 49 (3 and neural information processing [54. (55), yet with distinct contexts and objectives. Besides representation, people also studied calculation in superposition [56] 57
5 Discussion
Our work is built on observations of the toy model and analysis without rigorously solving the toy model. We are thus limited to explaining deeper behaviors in the toy model. Our analysis of LLMs

---

suggests they are in the strong superposition regime, but the underlying reasons were not studied in detail. We believe one reason that features are sparse in language, as the number of tokens required to predict one token is much less than the total number of tokens. The softmax function may also be important since it is strong at error correction, giving superposition an advantage.
Neural scaling laws also include scaling laws with dataset size and with training steps, which we did not study, At each step, a fixed number of new data points are used for optimization. So, we expect the scaling with the total data amount and that with training steps will be the same, similar to the results at weak superposition 20].. However, in the strong superposition regime, data or training step scaling is related to angle distribution and how angles between representations evolve, which cannot be easily explained without rigorous solving.
We focused on representation loss, yet LLMss should also have losses due to parsing or processing in the transformer layers. We imagine that the loss associated with model size be written
Cm/m am [m(()= + Se(O),
where is the depth of the LLM, fm and ft are two functions capturing the loss due representation and parsing, respectively. A future direction is to study the parsing-limice limited scaling (i.e. [(( function) independently. It is also plausible that the observed scaling of inference time (58] is connected to this sursing-limited regime. We here write the equality because E depends on m in LLMS (3]442) Given model size N, m and l are constrained (roughly, N oc m² There is an optimal m-f relationship such that the loss fm (m) + fe( can be minimized given N [59]. At this optimal m-l relationship, (m) and fe) should be balanced. Therefore, we expect fe( to be similar to fm (m). And if (m) ~ 1/m due to superposition and fe( is similar, we can measure an empirical a'm 2 from data, which true. Or, if the width-limited loss is much larger, we can also observe that the total loss due to model size has am ~ 1. We conclude that superposition in any case is an important mechanism underlying neural scaling laws.
Beyond explaining existing phenomena, our results may offer guidance for future LLM development and training strategies:
Assuming our explanation of width scaling is correct, we ask can change the loss scaling with width to be faster than power laws, or to have larger exponents? The answer is no for natural languages but may be yes for domain tasks with super skewed feature frequencies. Another question is when the scaling law will stop? Based on our naive comnection between features and tokens, the answer is that when the model dimension reaches the vocabulary size, the loss limited by width will deviate from a power law and vanish. However, the vocabulary size may set a lower bound for the true number of independent things in language, then the power law with width may continue for a longer time.
Recognizing that superposition benefits LLMs, encouraging superposition could enable smaller models to match the performance of larger ones (with less superposition) and make training more efficient. Architectures such as nGPT (60, which constrain hidden states and weight matrix rows to the unit sphere (promoting superposition), demonstrate improved performance. Optimizers that stabilize training without weight decay have also shown promising results [61]. potentially due to enhanced superposition. Yet, these improvements may be related to altering coefficients in the neural scaling laws rather than the exponents. We also acknowledge that encouraging superposition may cause difficulties for the mechanistic interpretation of models and AI safety (7 62).
As side note, with the same pre-training loss, LLMs with different degrees of superposition may exhibit differences in emergent abilities such as reasoning or trainability via reinforcement learning [63] requiring future studies.
In conclusion, we studied when loss can be power law and what the exponent should be with different data properties and degrees of superpasition. We found that geometric interference at strong superposition may explain the L.M neural scaling laws observed 3. Our results contribute to a deeper understanding of modern artíficial intelligence systems, which also open various directions for future research. We hope our insights will support the continued development and training of more capable and efficient LLMs.
10

---

Acknowledgments and Disclosure of Funding
We are grateful for feedback and suggestions from Yasaman Bahri, Cengiz Pehlevan, Surya Ganguli, Blake Bordelon, Daniel Kunin, and the anonymous reviewers. The authors acknowledge the MIT Office of Research Computing and Data for providing high performance computing resources that have contributed to the research results reported within this paper. J. G. thanks the Sloan Foundation for funding. The authors declare no competing interests.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
[3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203 2022.
[4] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint urXiv:2010. /4701, 2020.
[5] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[6] Haotong Qin, Ge-Peng Ji, Salman Khan, Deng-Ping Fan, Fahad Shahbaz Khan, and Luc Van Gool. How good is google bard's visual understanding? an empirical study on open challenges. arXiv preprint arXin:2307..15116 2023. https//axii..rrg/a//207...133
[7] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quan- titative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857, 2022.
[8] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211 1..9008.. 2022.
[9] Stephen Wolfram. Wolframlalpha as the computation engine for gpt models, 2023. http://wwww..rraaa.ccoo//lllaaa/
[10] Tricu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, :7777425) 2024.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar 107::2107..0374, 2021.
[12] GitHub. Github copilot: Your ai pair programmer, 2022.
[13] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 12. 1446, 2021.
[14] Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1-34, 2022.

---

[15] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 11127)xe2311877121, 2024.
[16] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR. 2020.
[17] Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. Journal of Statistical Mechanics: Theory and Experiment, 2025(8):084002, 2025.
[18] Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.1688 2022.
[19] Mareus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021.
[20] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 01228699-28722 2023.
[21] Ziming Liu, Yizhou Liu, Eric J Michaud, Jeff Gore, and Max Tegmark. Physies of skill learning. arXiv preprint arXiv:2501 12391, 2025.
[22] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
[23] Ari Brill. Neural scaling laws rooted in the data distribution. arXiv preprint arX1v:2412.07942. 2024.
[24] Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020 1020011) 2020.
methods: empirical data versus
[25] Jinyeop Song, Ziming Liu, Max Tegmark, and Jeff Gore. A resource model for neural scaling law. arXiv preprint arXiv:2402. 05 /64, 2024.
[26] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
[27] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Doddss Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish. Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv:1711.01001, 2017.
[29] Lloyd Welch. Lower bounds on the maximum cross correlation of signals (corresp.). IEEE Transactions on Information Theory, 20(3):397-399, 2003.
[30] Peter G Casazza and Gitta Kutyniok. Finite frames: Theory and applications. Springer Science Business Media, 2012.
[31] Thomas Strohmer and Robert Heath Jr. Grassmannian frames with applications to coding and communication. Applied and computational harmonic analysis, 14(3):257-275. 2003.
[32] Matthew Fickus, Dustin G Mixon, and Janet C Tremain. Steiner equiangular tight frames. Linear algebra and its applications, 436(5):1014-102) 2012.
[33] Joseph M Renes, Robin Kohou,, A Scott, and Carlton M Caves. Symmetric informa- tionally complete quantum measurements. Journal of Mathematical Physics, 45(6):217--2180, 2004.
12

---

[34] Yizhou Liu and John B. DeBrota. Relating measurement disturbance, information, and orthogo- nality. Phys. Rev. A, 104:052216, Nov 2021.
[35] Yizhou Liu and Shunlong Luo. Quantifying unsharpness of measurements via uncertainty. Phys. Rev. A, 104:052227, Nov 2021.
[36] Yizhou Liu, Shunlong Luo, and Yuan Sun. Total, classical and quantum uncertainties generated by channels. Theoretical and Mathematical Physics, 213(2):6113-133, 2022.
[37] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24552-2466 2020.
[38] Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206. 0.000, 2022.
[39] Susan Zhang. Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00065 2022.
[40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[41] An Yang, Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2XXv:24422115115, 2024.
[42] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2430. PMLR, 2023.
[43] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, arXiv preprint arXiv:1609.000 07843, 2016.
[44] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00022 000027, 2020.
[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-66 67, 2020.
[46] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pages 19-27, 2015.
[47] Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102. 2024.
[48] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint 171i: 2017.
[49] David L Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):
1306, 2006.
[50] Emmanuel J Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IFFE Transactions on Information Theory, 5292),489-509 2006.
13

---

[51] Richard G Baraniuk. Compressive sensing, IEEE Signal Processing Magazine, 24(4):1 121, 2007.
[52] Surya Ganguli and Haim Sompolinsky. Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. Annual review of neuroscience, 35(1):485- 508, 2012.
[53] Madhu S Advani and Surya Ganguli. Statistical mechanics of optimal convex inference in high dimensions. Physical Review X, 6(3):03 1034,, 2016.
[54] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (6583))6607-609) 1996.
[55] Behtash Babadi and Haim Sompolinsky. Sparseness and expansion in sensory representations. Neuron, (3(5):221-1228 2014.
[56] Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation superposition. arXiv preprint arXiv:2408. 05451, 2024.
[57] Micah Adler and Nir Shavit. On the complexity of neural computation in superposition. arXiv preprint arXiv:2409. 15318, 2024.
[58] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
[59] Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. The sopth-to-witth interplay in self-attention. arXiv preprint arXiv:2006. 12467, 2020.
[60] Iyy Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. ngpt: Normalized transformer with representation learning on the hypersphere. arXiv preprint arX00:2410.01131, 2024.
[61] Yizhou Liu, Ziming Liu, and Jeff Gore. Focus: First order concentrated updating scheme. arXiv preprint arXiv:2501. 122.4423... 2025.
[62] Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety-a review. arXiv preprini arXiv:2404. 14082, 2024.
[63] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
14

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We start our Discussion section by discussing limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
15

---

Justification:
We provide all details sufficient to reproduce.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We submit our code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wwciii/cggiieeeeeee
/
http://oooouummiiiioeeoe ssionooiicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We explained these details in our Appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We included error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the Appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We followed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is theoretical, with no obvious social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not have such risks.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited the datasets and LLMs analyzed.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy (https://neut) /cC//aareeecccc/2222//////////// for what should or should not be described.
20

---

A Theoretical analysis
A.1 Toy model loss
We provide a simple analysis for toy model loss scaling. The expected loss in the weak superposition regime is well explained by Equation (4). We do not need to repeat here.
In the strong superposition regime, we consider an even special data sampling, where data æ is sampled such that each data point has and only has one activated feature. The frequency for feature i to be activated is still p.. After determining which feature is activated, say i, we still sample 2; as U; from (0,, 2). This sampling is different from the experiments. Yet, since learned that activation density does not affect scaling exponent (Figure I4), we expected this analysis to predict at least the scaling exponent. Under the assumptions, we have
n Pr
L
Σ ReLU?(W, Was + b,,) (ReLU(W, Wey + b,) uz)²) j7t
(8)
We are unable to solve for the optimal W and such that this loss L, minimized. Yet, it is easy to see that we want |1 to be close to 1, W; W, to be as small as possible, and by to be small negative values of the same order of magnitude as W; W,, such that the interference terms ReLU(W;- Wey; + by) may vanish and the recovered feature value ReLU(W; Wwy: b.) can be close to the real one v;.
For convenience, based on the observation that the vector norms are bimodal around in the strong superposition regime, we define strongly represented features as those that have |[Wil22 > which are more frequent and ETF-like, and weakly represented ones for those with [ < 1. We can quantify the fraction of strongly represented as
Φ = ((i: : [Wil2 > 1)1/n,
which significantly exceeds m/n and is around the ETF expectation m² /2n (Figure 16p.
We now review our conjectured extreme configuration, consisting strongly represented and weakly represented features. The first d most important features are considered to be strongly represented, whose absolute overlap with any other representation scales as /1/m. The rest of the features are weakly represented and squeezed into small angle such that they have small overlaps with the strongly represented, while they can have large overlaps with each other. With such configuration, the first d n terms in the summation of Equation (8) will scale as 1/m since each term (..).; scales as 1/m. The rest of the terms in Equation (8). in the worst scenario that the terms (...) ).,, do not decrease obviously with m, will be proportional to Σi= Pt. If the strongly represented features dominate, we can have 1/m scaling for the loss. On the contrary, when the weakly represented dominate, we expect the loss to have scaling like Limo,n Pi. More specifically, if pt 1/8a (a 1) and we use m² /2 to approximate o,n, the loss scales as 1 1/m2(a-1))
A.2 Cruss-entropy loss
We provide the reason why cross-entropy loss also scales squared overlaps. We consider the last hidden state after going through the normalization layer is W,/ W//[Wl such that the output should be the ith token. By constructing such an example, ignore possible loss due to parsing but focus the loss just due to representation. The loss from this data point is
ellWalaa
L = - In E, W.WWl/ In 1+
Σ jyfi
WwW.//Wwlll-- (Wall2
(10)
We assume that W:- W5/llWWlle is much smaller than 1 since we know the overlap scale as 1/m. We then approximate the loss via Taylor expansion
L - In [ (n 1)e--WWa 1.. +
Σ [. W./ + (W; W./lIW/lb///ee-111..l (11)
In the first thought, the summation EjSt W W, should be since there positive and negative overlaps distributed evenly if the vectors span the whole space. But in language, sentence can
21

---

have different continuations, connecting different tokens. For example, both putting "cats" or "dogs" after "I like" are legit. The existence of data "I like" then will tend squeeze different tokens closer to each other. The summation W; W should be small positive constant <,, related to the correlation in data. The reason we keep the second-order term clear now as they are the lowest order terms related to model sizes. We keep expanding the In function and have
ED lIWalla
W. W3 |Wlll
L (n 1) WWllla 1lW-llll
e-IlWillz
(12)
|Wil12
jØL
The part related to the model size is mainly
Lm
1 Σ 2
W W
e ||alla,
(13)
37 |Will2
In this construction, one can see that once ||W is sufficiently large, the loss can be arbitrarily low, which does not happen in reality. The reason still related the intrinsic uncertainty language data. If one sentence can have different continuations, we need in the hidden space, a region that can lead to large probabilities over different tokens. However, when the norm is too large, one will find that the hidden space is sharply separated each hidden state yields high probability only on one token. We then expect the norm ||W]|2 to be as large as possible such that ne ||Willa is small while ||Will2 is upper bounded by intrinsic data uncertainty. Therefore, || should not depend on model size much (verified in Appendix D.7). The loss related to model size Lm. then scales as 1/m since the cosine similarity scales V1/m and Lm is related to the squared cosine similarity in the lowest order approximation.
B Toy model training
In this Appendix, we explain how we trained the toy models and obtained raw data. There are two classes of toy models trained. The first one large toy model with data dimension n 10240, which is reported in Figure and Figure 9 show scaling behavior across around two orders of magnitude. The other toy model class is small toy models fixing n = 1000, such that we can scan more hyperparameters. Figures
and use small toy models.
B.1 Large toy models
We implemented neural network experiment to study the scaling of feature representation and recovery. The toy model is defined as a two-layer neural network with ReLU activation (see Figure 2
The hyperparameters are given as follows.
Data dimension n: 10240
Model dimension m: Varied exponentially from 2 to 200
Batch size: 2048 (tested up to 8192, which does not affect final loss)
Total training steps: 20000 (tested up to 80000, which does not affect final loss)
Learning rate: Initially set to 0.02, scaled according to hidden dimension
Weight decay: -1.0 for strong superposition, and 0.1 for weak superposition
Device: Training performed using one V100 GPU, with floating-poin precision (FP32)
Data points were synthetically generated at each training step according to Equation I to simulate feature occurrence frequencies. We considered three distributions with activation density E 1:
Exponential: P4 x -4/400
Power-law: P; x 112
Linear: Pt i
22

---

We employed the A optimizer with distinct learning rates and weight decay settings for the weight matrix W and bias vector Specifically, for weight matrix W, learning rate was scaled as Ir x (8/ //m))00 jo.25 with specified weight decay. And for bias vector b, a learning rate of 2.0/m was used with no weight decay. A cosine decay learning rate schedule with warm-up phase (5% of total steps) was implemented. At each training step. input data batches were dynamically generated based on the selected probability distribution. The final test loss calculated across newly sampled data with size being 100 times the batch size.
The model and optimizer were compiled and executed on a CUDA-enabled GPU for efficient training. After training, weight matrices W and training losses were stored and analyzed.
Final outputs, including weight matrices and training loss histories, were saved in PyTorch format for subsequent analysis and visualization.
This setup provided structured exploration of feature representation scaling under varying dimen- sions and distributions, crucial for understanding superposition and scaling laws neural networks.
The code can be found in exp-17.pp
B.2 Small toy models
We conducted numerical simulations using neural network model designed for feature recovery. The objective was to analyze the model's behavior across various conditions involving feature frequency skewness (controlled by data exponent a),, model dimensions, and weight decay parameters.
In the small toy models reported in Figures
and we set the hyperparameters as
Feature dimension n: Fixed at 1000.
Hidden dimension m: Varied logarithmically between 10 and 100, across 6 distinct sizes, i.e., m 10, 15, 25, 39, 63, 100.
Batch size: 2048.
Training steps: 20000 steps for each condition.
Learning rate: Initialized at 1 10-2, .dynamically adjusted using cosine decay scheduling with a warm-up phase 2000 steps.
Weight decay: Explored systematically from -1.0 1.0, in increments of 0.22 approximately (10 discrete values).
Data exponent a: Ranged linearly from 2, with 17 discrete steps.
In Figure 14
we fix data exponent a 1 while scan 9 activation densities linearly from 1 to the maximal value Σ 1/i. All other settings are the same.
Synthetic data was generated for each batch based on a power-law probability distribution, defined as:
pi
< jar
where E (1,2,....,
with the condition E.P: E. Each element of the batch data randomly activated based on this probability, then scaled by uniform random value between 0 and 2.
At each training step, input batches were regenerated, and the learning rate was updated following the cosine decay schedule described above.
The training performance was evaluated using Mean Squared Error (MSE) loss computed between the network output and input batch data every step. Final weights were saved for further analysis. The final test loss is calculated across newly sampled data with a size being 100 times the batch size.
The simulations were performed in parallel using 96 CPU cores, where each core executed one distinct parameter combination defined by the weight decay and data exponent values. Or, in Figure 14 the parameter combination is defined by weight decay and activation density values.
Loss histories and trained weight matrices were saved separately for post-experiment analysis. Files were systematically indexed to indicate the corresponding experimental parameters. This detailed setup facilitated a comprehensive investigation of model behavior under diverse training and data distribution conditions.
23

---

The code can be found in exp-10 10. py. exp-10-3.1 py, and exp- 15. 15.p7 py.
C
LLM evaluation
C.1 Overlap analysis
We analyzed the row overlaps of the language model head weight matrices among various large language models (LLMs) investigate the geometric properties of their hidden spaces.
We selected models from the following families, varying widely in parameter count:
OPT (from OPT- 1P5m to OPT-66b)
Qwen2.5 (from 0.5B to 72B)
GPT-2 (GPT2, GPT2-Medium, GPT2-Large, GPT2-XL)
Pythia (from 70m to 12B)
Weights were downloaded directly from Hugging Face model repositories. For each model, the weight matrix language modeling head was normalized by its row norms:
W
W, + [ + c 10-9.
where e is for numerical stability.
We computed the pairwise absolute cosine overlaps between normalized vectors using batch-wise computations for efficiency. The overlap between embedding vectors W; and W, is given by:
W; W
overlap W,, Wr)
I. l12
To handle large embedding matrices efficiently, overlaps were computed in batches (size of 8192
vectors).
We calculated two key statistics for the overlaps within each model:
Mean Overlap: The average of absolute overlaps for all unique vector pairs:
mean mean_oveerla = Σiey overlap(W., W,) n(n - 1)/2
Overlap Variance: Calculated as:
Eiey (overlap (W,, W2) meann_ovelapp" n(n 1)/2
variance_overlap
From these values, we can calculate mean square overlaps mean mean_overlapp + variance_overte _overllp.
The calculations were accelerated using GPU resources (CUDA-enabled) efficiently handle computations involving extremely large matrices.
Results including mean overlaps, variances, and matrix dimensions were recorded for comparative analysis across model sizes and architectures.
The code is in overlap-0.pp
C.2 Evaluation loss
This experiment aims to evaluate multiple large language models (LLMs) efficiently using model parallelism and dataset streaming techniques. The models were assessed on standard text datasets to measure their predictive performance systematically.
Models were selected from Hugging Face and evaluated using a model-parallel setup:
OPT series
24

---

Qwen2.5 series
GPT-2 series
Pythia series
We used the following publicly available datasets for evaluation:
Wikitext- Standard English language modeling dataset.
P0ke-10k: A subset of The Pile, designed for diverse textual data.
C4: Colossal Clean Crawled Corpus, containing large-scale web text.
BookCorpus: Large-scale collection of books used for unsupervised learning.
Datasets were streamed directly, efficiently sampling 10000 text segments with maximum sequence length of 2048 tokens (~ 10? tokens).
Texts from datasets were tokenized using the respective model-specific tokenizers. Tokenization involved truncation and manual padding uniform batch lengths. Specifically, padding tokens assigned an ID of 0, and label padding utilized special token (-100) to ensure they did not contribute to loss computations.
Each model was loaded using Hugging Face's AutoModelForCausalLL. with model parallelism enabled, allowing the evaluation of large models that exceed single-GPU memory limits. Evaluations were conducted in batches, employing a DataLoader with custom collate function for optimized memory
The model's predictive performance was assessed by computing loss values internally shifted by the Hugging Face library, suitable for causal language modeling.
Model parallelism was implemented to efficiently distribute computations across multiple GPUs, leveraging CUDA-enabled hardware.
For each model and each dataset,
evaluation and save the evaluation losses.
Random seeds and deterministic sampling ensured reproducible dataset selections, though explicit seed settings were noted as commented options within the implementation.
Evaluation results, including loss metrics and potentially intermediate model states, systemati- cally stored for detailed post-analysis.
The code is in cali-1.pp
C.3
Token frequency
The purpose of this analysis is compare token frequencies generated by different tokenizers across several widely-used textual datasets. Understanding these frequencies helps assessing the representational capacity and efficiency of tokenizers used by various large language models.
We considered the same four datasets mentioned for LLM evaluation. And we use four different tokenizers from the four model classes we evaluated.
Each tokenizer processed textual data from the specified datasets, streaming data directly to efficiently handle large-scale inputs. A target of 1,000,000 tokens per tokenizer-datte pair was set to ensure sufficient statistical representativennss.
For each dataset-tokenizer combination:
1. Text samples were streamed directly from the datasets.
2. Text was tokenized without adding special tokens (e.g., EOS).
3. Token frequencies were counted and accumulated until the target token count (1 million tokens) was reached.
4. Token frequencies were saved as JSON files for subsequent detailed analyses.
Token frequency data was systematically stored for each tokenizer and dataset combination, enabling comparative analyses of token distributions. The data files provide foundational insights into tokenizer efficiency and coverage across diverse textual domains.
25

---

2 2
10
10
Figure 7: Row norm distributions for large toy models. There are 6 rows of panels. Rows and correspond to the power-law feature frequency. Rows 3 and 4 correspond to the exponential feature frequency. Rows 5 and correspond to exponential feature frequency. In the two rows that correspond to the same feature frequency, the upper row is at strong superposition and the lower one weak superposition. The columns from left to right correspond to different model dimensions from small to large.
The code is in ceeoo-free--.100
D Figure details and supplementary results
Here, we show how to process the raw data obtained from toy models or LLMs to generate results in the main text. Some supplementary analysis is also conducted support the main text arguments.
D.1 Figure! 1
The toy models reported in Figure 1 are large toy models with data dimension n = 10240 explained in Appendix B.I After obtaining the final losses, directly plot them with respect to the model dimension m. Error bars are calculated as the standard deviation of losses over 100 batches. The error bars smaller than the dots (Figure I b and d).
When we are fitting the loss in log-log as a line, we choose the linear part to fit. the loss versus model dimension curve is obviously not line, we fit the whole curve line and output the R² value as a measure of how non-linear is. Specifically, we fit the last five points for the plwer-law decay feature in the weak superposition regime (yellow data in Figure Пb). Other cases in the weak superposition regime are fitted to line with all data. the strong superposition regime, when feature frrquency decreases from the third point (yellow and green Figure TH). And for exponential decay feature frequencies, fit all the data with line. In the strong superposition regime, the measure model exponent Oen close to 1: 1.01 + 0.05 (exponential decay). 1.0 + 0.1 (power-law decay), and 0.89 + 0.05 (linear decay).
power law
linear function, we fit the data line starting
The LLM data are copied from Figure Gb, with slope -0.91 0.04 being close as well. We will explain details about Figure later.
26

---

b
1.00
0.75
1.0
11
Loss
0.50
0.5
0.25
0.00
0.0
10a
102
10a
0.0
0.5
1.0
Model dimension, m
Loss by unlearned
C
d
1.00
Mean square overlap
109
0.75
10-1
0.50
10
0.25
10
0.00
10
102
103
10
102
103
Model dimension,
Model dimension, m
Figure 8: Large toy models agree with theoretical expectations. We use blue, yellow, and green for exponential, power-law, and linear feature frequencies, respectively. Dots correspond to the weak superposition regime, and squares the strong superposition regime. (a) The fraction of represented features is 1 for strong superposition, and is close to m/n (black dashed line). number of activated but unlearned features well describes the loss weak superposition. The dashed line is where the actual loss is the same as the predicted onc. (c) In the strong superposition regime, the number of strongly represented features is much larger than m but bounded by some value around m² 2/. The slowly growing dashed line is m/n and the fast growing dashed line is min m² n2/2n/ /2n]. (d) The mean squared overlap of strongly represented features is close n², which is close to 1 1/m, given that the number of strongly represented features is much larger than m.
(b) The expected
We also output the weight matrix W for these large toy models (Figure 7 They follow the same pattern that in the weak superposition regime, row norms are bimodal and are either close to 0 or 1, making 0.5 a good separation point for measuring how many features are represented. And in strong superposition regime, the row norms are distributed near 1, and is good separation point for the two peaks, i.e., peak greater than refers to strongly represented features which are more important, and the peak smaller than I corresponds to the weakly represented.
We can analyze the large toy model in the same way what has been done in Figure and The fraction of represented features /1 is calculated, which is in the strong superposition regime, while it is close to m/n in the weak superposition regime (Figure Ba).
With the measured 11/2, can estimate the loss due to unlearedd features, (v²) Ei=o Pi. This theoretical value agrees well with the actual loss in the weak superposition regime (Figure (8).
In the strong superposition regime, the fraction strongly represented features calculated, agreeing with the expectation that the number of strongly represented features is much larger than m but bounded by some value around m² /2 (Figure ge).
At the end, we see that the mean square overlap of the strongly represented is close to the characterized value x2 (Figure 8d), which scales as 1/m since the number of the strongly represented is much larger than m.
27

---

b
2.0
Model exponent, dm
2.0
Data exponent,
1.5
squared
0.9
1.5
1.0
law
1.0
0.8
powwe expootr
0.5
linear
0.5
0.0
0.0
0.7
Strong
Weak superposition
superposition
Weight decay, Y
Figure 9: Rich scaling phenomena arise when we change the degree of superposition and data structures. (a) In the same experiments (n 10240) in Figure where p; cxp(-i/400) (blue), n (green), we use
1/12.2 (yellow), and y 0.1 for weak superposition. R-squared value of the fitting is used measure how likely the fitted part is a power law (Figure TO). R-squared values closer to mean that the data are more similar a power law. We found that strong superposition, power laws are robust across different underlying feature distributions. Yet, at weak superposition, only power-law feature frequencies can lead to power-law losses. (b) When changing superposition by weight decay and varying feature frequency decay by a given x 1/2%, we found roughly three distinct behaviors. For (b), 1000 and 10, 15, 25, 39, 63, 100.
- to reach strong superposition and
2.0
1.0
0.8
Data exponent,
1.5
0.6
Squara
1.0
0.4
0.5
0.2
0.0
0.0
1
Weight decay, y
Figure 10: R squared values for fitting loss as a power law with model dimension. Data are from the small toy models with data dimension 1000.
In Figure I we set weight decay = -1 to have strong superposition and = 0.1 to have weak superposition. We compute squared values from linear fits log-log plots to quantify scaling behavior, assessing how closely the loss follows power law to model dimension. We can see that strong superposition, the losses are close to power laws, regardless of the underlying feature frequencies, yet the loss is a power law at weak superposition if the feature frequency p; is power law with rank (Figure 9p). For a systematic scan, we next set ps x 1/1a and can vary the data exponent a to change how fast p; decays consistently. Assuming power-law form for the final test loss, x 1/m"", we extract the model exponent Omm from the empirical fit. We the loss with a power law in all cases. The fitted am reveals how fast losses decay, even in the regime where the loss should not a power law. Roughly, three distinct patterns emerge: (1) under weak superposition (positive 7. yellow box in Figure 9p). O'm. is small, indicating slow loss decay; (2) under strong superposition and a wide range of small data exponents (red box), O. remains robustly near 1; (3) for strong superposition with large data exponents (blue box), a'm increases with By interpreting these three patterns, we aim to understand when loss follows power law with model dimension. and what determines the exponent when it does.
Figure D reports the R² values from the fitting, where the raw data comes from training large toy models (Appendix B.I When we are fitting the loss in log-log line, choose the linear part to
28

---

fit. If the loss versus model dimension curve is obviously not line, we fit the whole curve as a line and output the R² value as a measure of how non-linear it is. Specifically, we fit the last five points for the power-law decay feature case in the weak superposition regime (yellow data in Figure [p). Other cases in the weak superposition regime are fitted to line with all data. In the strong superposition regime, when feature frequency decreases as power law or as linear function, we fit the data as line starting from the third point (yellow and green in Figure [T).
And from the raw data of small toy models (Appendix B.2), we can fit the model exponent Om directly and plot as a function of y and a as in Figure Dp.
The fitting in Figure 99 does not care whether the loss versus model dimension curve is a power law or not. We provide the R squared values for the fitting here (Figure I0). The closer R squared values are to 1, the better the data can be thought to be a power law (a line in log-log plot). In the strong superposition regime, the R squared values suggest the data are close be power-law. While in the weak superposition regime, data may not be power-law, especially when y is too large. When a is smaller than 1, it is not power law in theory. The R squared values are not too small since the loss decay is very slow, and a line in log-log plot is still a good approximation. When a > 1 and y x 1, the number of represented features can be smaller than m or even non-increasing. Too large weight decay still makes the configuration of the representation be in no superposition. However, it destroys some feature representations that can exist, making the configuration far from the ideal case where m features are represented. So, we may not see power laws when weight decay is too strong.
D.2 Figure 2
Figure Zintroduced the toy model and the concept of superposition without real data. The W matrix we used show superposition in Figure is obtained by optimizing the square of off-diagonal terms of the normalized W, i.e., each row is normalized to have norm first.
D.3 Figure 3
In Figure 3 we reported results from the trained small toy models with data dimension n 1000, whose detailed hyperparameters are in Appendix B.2
We showed results at data exponent a in Figure 3 The results are obtained at m 100, 7 for panel a, and at different m and y for panel b. We showed that the more frequent features tend to have larger norms or be better represented. And the norm distribution very bimodal. We here show that it is true that the norm is around 1 or 0 for various a and model sizes m and degrees of superposition (Figures [ and 12h. The fraction of represented, 61/2, can be calculated directly.
Here, we provide the heat map of ø1 2 at different m as a function of a and (Figure 13). The pattern is robust, suggesting weight decay is good tool to change the degree of superposition regardless of data properties and model sizes.
D.4 Sparsity does not affect scaling behaviors in our tests
We studied the effect of the number of expected activated features or activation density E, which was set to 1. By fixing data exponent 1, which will be shown to be relevant to natural language, we can scan different superposition degrees and activation densities. Since p; I is required. which is equivalent to p1 < 1, we have E -= 1/2". setting the upper bound for our scanning. We found that loss is approximately proportional to activation density É (Figure 14p). This fact suggests that the power law exponent should not change, which we confirmed (Figure 14p). Under controlled superposition degree, activation density linearly increases loss and thus does not affect the scaling exponents in our experiments.
Once obtaining the small toy models scanning activation density and keeping a = 1, we can plot the loss as function of activation density E in Figure 14 The linear fitting is also straightforward. We chose one y to show the main text. Here, we present the whole picture that, with any weight decay tested, the model exponent is robust to the change of activation density (Figure 15).
29

---

2x300
10"
101
30
203
1al
140
101
101
101
101
1471
Sat
10²
101
2014
b0m1
10n1
1011
101
107
10-1
10"
10
104
107
101
10-1
101
101
102
$8.18
2K.10"
10
1014
1014
Sonk
1.0
107
10-1
10-1
8ix 200
10a
101
107
127
161
:0P
101
20"
101
upt
101
boe
10
10
1:7
167-
101
301
50
:074
107
101
101
107
1.0
204
161
107
up4
101
10f
10.3
207
t01
101
10F
Figure 11: Row norm distribution at weak superposition (y= 0.55) shows that the rows either close to zero or have norm close to 1, making 0.5 a good separation. The rows panels from top to down correspond to 17 from 0 to 2. And the 6 columns from left to right correspond to m 10, 15, 25, 39, 63, 100.
30

---

20
70
20
22
300
10-
20
30
0.0
310
2.5
1n
10
Figure 12: Row norm distribution strong superposition (7 -0.55) shows that the rows have norm close to And density at is very low, making good separation point for two groups row norms. The 17 rows of pancls from top to down correspond to 17 from 0 to 2, And the 6 columns from left to right correspond to m 10, 15,25, 25, 39, 63, 100.
31

---

15
15
LS
05
10
08
10
Figure 13: Fraction of represented features as function of (x-axis) and a (y-axis). The 6 columns from left to right correspond to m = 10, 15, 25, 39, 63, 100. The colorbar is Φ1/2 where purple means 1 and white means 0.
a
b
10a
4
1,00222
Activation density, E
Weight decay, Y
0.5
3
Lo5g
Losa
100
1,150.1
1,12:0.00
1-1220.06
0.5
10-1
-1.0
0
10a
102
Activation density, E
Model dimension,
Figure 14: Activation density does not affect scaling exponents in our tests. (a) Loss is roughly proportional to activation density given the degree of superposition (m = 63, = 1000). (b) So, E will only affect the coefficient but the exponent when considering the power law with model dimension. We plot the evidence R I at strong superposition.
1.5
1.0
Model exponent, am
0.5
1.0
0.0
0.5
-055
0.0
-1.0
2.5
5.0
7.5
Activation density,
Figure 15: Model exponent is robust to activation density at different levels of superposition. The colorbar encodes weight decay as the main text. Error bars are standard errors.
32

---

Strong rep. fraction, &
1.0
2.0
1.5
0.5
1.0
0.5
0.0
0.0
10
102
Model dimension,
Figure 16: Fraction of strongly represented features (|Will2 > 1) at strong superposition (7 = -- is around min m² /2n} (fast increasing dashed line), which is much larger than m (slowly increasing dashed line). Colorbar means a as the main text.
D.5
Figure 4
In Figure we plot the raw losses from small toy model experiments (hyperparameters Ap- The theoretical value of loss is approximated by integral fmp,,2 1/id which
pendix B.2
@ 1-aa
nl-a" a ##1,
n 1-a
L
n
(14)
In @1 1.
In
To quantify how much the learned weight matrix deviates from the ideal no superposition structure, we construct a reference matrix and compute norm difference. Specifically, we first create an n-by-n zero matrix called base, and then insert an identity matrix of size m in its top-left corner. This padded identity matrix serves as reference for the perfect recovery of the first m features. We then compute the matrix product WWT from the learned weights and compare using the matrix 2-norm. The resulting value reflects the ambiguity or interference in the learned representations. We store this norm in the ambiguity tensor at the location indexed by the current task and model width. Given a weight decay and data exponent, we have 6 ambiguity values since we have 6 m values. calculate maximum ambiguity among these models, and choose the 9 cases with the smallest maximum ambiguity to plot in Figure 4b. One can see that when weight decay near 0.5, the models are closest to the ideal no superposition case where the first m features are represented perfectly. Smaller weight decay may not be sufficient to climinate superposition, and larger weight decay can suppress features that, principle, can be represented perfectly.
this reference
D.6 Figure 5
For convenience, based on the observation that the vector norms are bimodal around in the strong superposition regime, we define strongly represented features as those that have ||W]|2 2 > 1, which more frequent and ETF-like, and weakly represented ones for those with ||i]| 2 < 1. We can quantify the fraction of strongly represented as
Φ1 = [( |Will2 1}/nn,
(15)
which significantly exceeds m/n and around the ETF upper bound m²2 n22 (Figure 16 The group of vectors |WWll2 with norm greater than the strongly represented features then roughly agree with ETF properties: small variance, 1/m mean squared overlaps, and a limited number of vectors.
sttudies the results from small toy models (Appendix B focusing on the strong superposition regime. For the strongly represented fraction, Φ1, we can directly compute based on definition and the obtained weight matrices. We showed row norm distribution
15, 1, and 7 -0.78 Figure panels a and b. Here, we provide more data show that is natural separation point in norm to determine which are strongly represented and which weakly represented (Figure
33

---

2.0
Mean square overlap
1.5
10-1
1.0
0.5
10-2
0.0
10
102
Model dimension,
Figure 17: The mean squared overlap over all the W; vectors in the strong superposition regime
(y -1). Although the value may be higher than 1/m (the dashed line), the scaling is robustly 1/m. Colorbar means a as the main text.
Once select the rows with norm greater than 1, we can calculate their mean and variance of squared overlaps based on normalized rows / WW,l2 (Figure c and d). We argue that after training, the vectors will be more similar to ETFs than to random initialization. This is studied via the variance of overlaps. ETFs, theory, have zero variance. We find that the majority of the overlap variances are much smaller than the random initialization, especially when features have similar frequencies, which agrees with the expectation. The cases where the actual variance greater than that of the random vectors have large a, roughly correspond to the cases where configuration no longer dominates. This intuitive that when a is too large, the heterogeneity of overlaps will become large- - it is better to let more frequent features occupy larger angle space. We argue that the large variance at large a does not mean the configuration tends to random, but tends to be something more closely related to the frequency distribution of the features.
deviates from - ETF-like
Our explanations based on the strongly and weakly represented features capture the basic trend that when a is getting large, the more important features will have larger angle space and the loss decay will be more related to the data exponent. However, this theory is oversimplified, where the strongly represented all have small overlaps and the weakly represented all have large overlaps. The real situation may be more like the angle occupied by one feature decreases continuously as the frequency decreases. As suggested by Figure 5]. overlap variance within the strongly represented is greater when a is larger. is large, we cannot use simple theoretical expectations like ETFs but have to solve the toy model.
more precise about the overlap distribution as well as the exponent am when
We also provide evidence that overlaps of all the vectors (Figure 17 We see that some of the mean square overlaps are larger than 1/m instead of being on the line I/m. However, all the mean values follow 1/ m scaling even for large a cases where the vectors are no longer isotropic. We emphasize that for even frequencies and isotropic vectors, since squared overlaps scale as 1/m, the loss should scale as 1/m.
After fitting am of the trained small toy models (Appendix B.2). we plotted the a.m corresponding to the second the fourth smallest weight decays in Figure 5p. We also copied from Figure 4 and plotted the ideal weak superposition case in Figure5p. One question we had is that if m is always greater than n, in our analysis, all vectors can be strongly represented, what should a'm be? We trained the small toy models again as in Appendix [B.2 with m from 50 to 150. We found that in the strong superposition regime, the am is still around when increases little while smaller than 2(a 1) when a is larger than 1.5 (Figure [8}. When m² /2n > 1 always true, the vectors can be put into configuration where all overlaps are small and scale as /1/m, m. such that Om should be closer strongly represented have nearly uniform absolute overlaps is oversimplified. In the real situation, more frequent features have smaller or even faster decaying overlaps. Therefore, when a is too large, weighted sum of squared overlaps, weighting the more frequent features more, can decrease faster than the average decaying speed 1/m. Again, we need solve the toy model faithfully to uncover the rigorous relation between Om and and argue the robustness of am from theory.
smaller than 1.5, and on still
However, mentioned before, our picture that the
34

---

1.0
Model exponent, am
0.5
0.0
-0.5
-1.
0
1
2
Data exponent,
Figure 18: Small toy models with m from 50 to 150 (such that m2 /2 > ni) in the strong superposition regime yield similar om around when is small and slightly smaller Om (smaller than 2(a 1)) when a large. We copied Figure 5 and made the points transparent for comparison. The non- from small toy models with m from 50 150. Om 1 is the horizontal line, is the slowly increasing dashed line. Error bars standard errors. The colorbar encodes weight decay in the main text.
transparent points
O'm 2(a - 1) is the fast increasing dashed line, and am
100
10-1
Density
10-2
10-3
10
0
2
4
Row norm
Figure 19: Row norm distribution of the language model head of 125M.
D.7
Figure 6
After obtaining the overlaps as described in Appendix C.I we directly plot the raw data in Figure 6a. The data are quite noisy, and we did not fit the data with line.
We argued that the LLMs are the strong superposition regime since all tokens are represented. Figure 19 shows typical row norm distribution of LLM (opt. 125M parameters [9] We showed the mean, minimum, and maximum row norms of all the L.LSs studied in Figure 20 From the non-zero minimum norms and the fact >>> m, confirm LLMS are in strong superposition. As mentioned in the analysis in Appendix [A.2] we argue that the row norm of LLMs should not depend but controlled more by the intrinsie data property of language, which is also verified to be valid (Figure 20p.
We obtain the evaluation loss of each model each dataset as described in Appendix loss values by the formula.
CP2
We fit our
Cm/ /m Pm Tam:
where C.n/ Cm/ma mm am is universal and L is constant depending the dataset and model class. There are total 16 different L, since we have different model classes and datasets. In our fitting model, there are in total 18 parameters. We use Adam to minimize the mean square error between the predicted loss by the above function and the real loss. All losses obtained are used optimization. The code is in nonl comlinearfin it-3..t... ipynb.
provide the raw data, losses, as a function 1/m (Figure 21 1/m in one model class and with the same dataset. And the slope of the line seems to be universal.
The losses looks like line with
35

---

b
2
opt
4
Row norm
Row norm
Qwen
2
0 -h
10
10
1000
10
10
10
10
10
Number of parameters
Number of parameters
d
3
Row norm
Row
2
gpt2
1
pythia
10a
10a
10
10
10
10
10
10
Number parameters
Number parameters
Figure 20: LLMs are in strong superposition based on the non-zero norms the representation vectors. (a) OPT models. (b) Qwen2.5 models. (c) GPT-2 models. (d) Pythia models. The dots with error bars are mean values, and the error bar is the standard deviation of norms. Red small points the maximum values, and dark small points are the minimum values. The mean norm value characteristic row norm does not depend on model size much as expected (Appendix(A) (A.2..
wikitext c4
bookcorpus
pile
Loss
2
opt
Qwen
gpt2
pythia
le-3
0.0
0.5
1.0
1.5
2.0
Inverse model dimension, 1/m
Figure 21: Raw evaluation losses as function of inverse model dimension.
These two points support us in proposing the formula above, where Cm //mmmma is universal. The intersections are different depending on the dataset and model class, corresponding different Ly
We obtained the token frequencies as described in Appendix C.3 the token frequency and obtain the frequency-rant plot. We sample 1000 (this number does not matter once it is large, 10000 gives the same result) points uniformly in the 1o (Rank), and fit the frequency-rant as power law, line in log-log plot. Results show that the data exponent fitted close to regardless of the dataset or the tokenizer (Figure (2))
Given the raw data, we sort
We study the relationship between model dimension and model size N (number of parameters). For the four open-sourced models we analyzed [39.42] we can see that N m especially when m large. If we fit the N-m relation by power law while assuming universal exponent but different
36

---

a- 1.006 0.003
0.990 0.003
0.985 0.003
1.106 = 0.005
legtc/reqeency
bglo/reepency
Ignotheepuencyy
log10lRark]
log 101kkakkk
log10(tanki
kgg30(aank)
log1iliequency
log10i (ilmeeueenyy
log reqquencyy
-4
1
log10Rankk
k0g10Rankk
log10 Dannk)
kgg00(aak))
1.035 0.003
1.000 0.003
0.996 0.004
1.113 0.005
loc1Ofresuency!
10treeuennyy)
1cc10/Fequuncy))
log 1otrrequency)
-6
kgg10/aan]]
log [010RRank]
log10(Rank]
kog (0(0ak))))
1.013 0.003
0.997 0.003
1.053 0.003
1.112 0.005
loplllreeuue (evereecn)y)))
log o0f eequerccy)
loplolrreeeeng
-2
-0
log10(Rankk)
log30(Rank)
Figure 22: Taking tokens atomic features, their frequencies indeed follow power law, and the measured data exponent a is close to 1. The four rows from top to bottom correspond to four tokenziers, Pythia, OPT. Qwen2.5, and GPT-2, respectively. And the four columns from left to right correspond to four datasets analyzed, wikitext, C4, the Pile, and Bookcorpus, respectively.
a
b
10a1
10
opt Qwen
Chinchilla
Model size, N
1010
gpt2 pythia
Model size,
10
10
3.0
2.5240.03
10a
108
103
104
10
Model dimension,
Model dimension,
Figure 23: The model size is approximately a power law with model dimension. (a) The four model (b) The Chinchilla models
classes we analyzed
coefficients depending the model class, we obtain exponent of 2.51 (Figure 23 For the Chinchilla model reported [3], we find N is also close to power law with the model dimension, and the fitted exponent is 2.52 + 0.03 (Figure 23p).
37


Reviews:
[{"review_id": "xjkBlgnz7h", "reviewer": "Reviewer_Y6XA", "summary": "The paper argues that superposition is an important mechanism underlying observed neural scaling laws. To show this, the authors consider a toy setting adapted from the original toy models of superposition work and study how the level of superposition, controlled by a specifically defined weight decay term, influences scaling behavior. The key finding is that under weak superposition, power-law scaling occurs only if the features themselves are power-law distributed (model exponent is $\\alpha_m \\approx \\alpha - 1$, where $\\alpha$ is a data exponent), while under strong superposition, power-law scaling emerges largely independent of the feature distribution (model exponent, however, still depends on the data exponent, following $\\alpha_m \\approx \\max(1,\\ 2(\\alpha - 1))$). Beyond the toy setting, the paper also analyzes token representations in the last-layer weight matrix of practical LLMs and shows that they exhibit power-law scaling with a model exponent $\\alpha_m \\approx 1$, consistent with the toy example in the strong superposition regime with $\\alpha \\approx 1$. This connection is then further extended and tied back to the scaling behavior of the cross-entropy loss in LLMs.", "questions": "All questions and concerns are detailed in the Strengths and Weaknesses section. The main weaknesses 2-5 will have the greatest impact on my final evaluation after the rebuttal.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses an important problem: understanding the underlying reasons behind neural network scaling laws is highly relevant in the current era of large models.\n2. The idea of superposition and its role in scaling laws is interesting and, as far as I know, novel. \n3. The paper focuses on a well-motivated toy example and makes a meaningful connection to practical models.\n4. The authors make a commendable effort to provide a thorough analysis, linking theoretical and intuitive ideas with empirical evidence.\n\nMain weaknesses:\n1. The writing of the paper is very dense. In particular, the monolithic 4.5-page results section was difficult to read and navigate, and I found it challenging to precisely assess the claims due to how scattered they are throughout the text. I believe the paper would benefit from a significant rewrite to improve clarity and structure. Additionally, dedicating more space in the main text to discuss how the proposed explanation of scaling laws connects to existing theories (it is very important for placing the work in context) and comment on how this explanation could inspire new training methods or architectures (this point is mentioned in the abstract but discussed only in the appendix) would benefit the paper.\n2. The choice of weight vector norms as indicators of features being represented should be more thoroughly explained, and their distributions along with the threshold values used should be commented on in more detail:\n    - Initially, it was not clear to me what \"feature being represented\" means and that it is different from a feature being correctly predicted. Because of this, I was confused during the first two sections about the idea of learning more features than dimensions, since learning more features should directly harm the quality of how well each of them is learned. The second section of the paper should explicitly discuss that representing a feature is different from correctly predicting it, and that superposition is a balance between representing more features and having destructive interference between them during prediction.\n    - The choice of the 0.5 threshold for defining whether a feature is represented or not is based on the empirical norm distribution shown in Figure 3a. However, as far as I understand, this distribution may vary significantly depending on the level of superposition (e.g., the norms are not close to one in the distribution shown in Figure 6b). Could you please clarify how the norm distribution changes across different superposition regimes, and whether the 0.5 threshold is meaningful in all cases, not only in the weak superposition setting?\n    - Could you please comment on why the vector norms are distributed bimodally in the case of strong superposition shown in Figure 6b? Given the form of the weight decay term, one might expect the raw norms to be close to 1. Why is the density low near 1? Intuitively, since the feature frequencies change smoothly and monotonically, one might expect feature representation behavior to also change smoothly. Why do features instead fall into two distinct modes? Is this an artifact of the particular weight decay formulation? And if so, why should we expect this behavior to reflect what happens in practical LLMs?\n3. I believe there is an important issue in the assumption made in equation (4). Specifically, $y_i$ are assumed to be equal to $0$ for features $i > m$. However, based on the original toy models of superposition work, for the unlearned features, $y_i = b_i$ where the learned bias term $b_i$ is the expected value of $x_i$. \n4. The assumption that strong superposition is characterized by ETF-like geometry is not convincingly validated. The main evidence I gathered from the paper comes from Appendix F.6 and Figure 15, specifically the observation that the variance of feature overlap is smaller for strongly represented features than that of $m^2 / 2$ random vectors. However, this alone is not particularly convincing. Do weakly represented features behave differently? Is the number of strongly represented features actually equal to $m^2 / 2$ in practice? Not according to Figure 6c. In general, for such an important claim, I would expect a more rigorous and convincing analysis.\n5. There is no clear explanation for the change in scaling observed with strong superposition as data exponents increase. The intuitive explanation that higher-frequency feature vectors become more separable (collapsing lower frequency vectors and increasing their errors due to feature overlap) is compelling, but seems to be inconsistent with Figures 6d and 16, where increasing data exponents actually increases feature overlap for strongly represented features and decreases feature overlap for weakly represented ones.\n\nAdditional concerns, comments, and questions:\n1. Line 62: The weak superposition regime is not defined at this point.\n2. Lines 88/95: It seems to me that “data” is usually used to refer to a dataset or data distribution, so it was a bit confusing that “input data” and “one data” referred to a single sample.\n3. Figure 3a: The small subplots are difficult to parse and should be commented on in more detail.\n4. Equation 2: Does this equation take into account the loss gradients or only the weight decay term?\n5. Figure 4b: A diverging color map centered at $\\alpha_m=1$ would benefit the figure. Currently, it is difficult to see that the values in the red box are close to 1.\n6. Figure 4b caption: What does $m = 10 \\sim 100$ mean here? Which model size is used?\n7. Line 184: The word “nonlinearity” appears twice.\n8. Figure 6c caption: “which is much larger than m (slowly increasing dashed line)” — I think it should be $m/n$.\n9. Figure 6e: The second increasing line is not defined. Which weight decay values are used in these experiments? Where are the points for high weight decay values close to 1?\n10. Why, in the analysis of token representations of LLMs, is the weight matrix of the last layer used rather than the embedding matrix? (Or are they tied in the considered models?)", "was_revised": true, "final_justification": "**Initial submission.** The paper presented a novel and interesting view on scaling laws based on the idea of feature superposition. However, it had issues with text structure and clarity (Weaknesses 1, 2, 3), and some empirical results were not fully supportive of the claims (Weaknesses 4, 5).\n\n**After discussion.** The authors clarified unclear points (Weaknesses 2, 3) and provided additional discussion on the claims that were not fully supported by the results, offering a reasonable explanation for the potential reasons behind the discrepancies between the model and the findings (Weaknesses 4, 5). They also promised to improve the text structure and incorporate the additional discussions in to the paper. \n\n**Final recommendation.** The rebuttal and discussion addressed most of my concerns. While some questions remain open, the authors provided reasonable discussion on all of them. I believe that after incorporating these discussions and improving the structure of the text, the paper would make a strong contribution. Therefore, I now agree with other reviewers, will increase my score and recommend the paper for acceptance.", "rebuttal": "We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.\n\n* Main weakness 1: We are sorry for the inconvenience due to our presentation. We wanted to include too many points in a limited space (9 pages), which led to dense writing and inadequate details for each point. To solve this problem, we plan to emphasize the core results in the main text by adding more details and moving less important results like Figure 7 to the Appendix. We can also add subsections and use the Theorem environment to clearly list our assumptions and results to improve readability. We are willing to incorporate more suggestions from the reviewers, if any.\n\n* Main weakness 2:\n    * Yes, features being represented are those having non-zero representation vectors. Features being well-represented means they can be correctly recovered or predicted. We can highlight this by using a \"Definition\" environment.\n    * Based on the definition of \"represented or not\", the threshold should be 0. In practice, we need to use a (small) non-zero threshold. We chose 0.5 since it is equally far away from 1 and 0, potentially minimizing misclassification. But this choice is very conservative since 0.5 is not small. When $\\gamma \\geq 0$, the norm distribution concentrates near 0 or 1 (Figure 11), and choosing a threshold from 0.05 to 0.95 does not change the outcome. When $\\gamma < 0$, all norms are around 1 but may have large variance (Figure 12). In this case, choosing 0.5 as a threshold will give a conservative lower bound for the number of features being represented. Our goal is to establish a relation between $\\gamma$ and superposition. If a lower bound is close to 1, it suffices to say that $\\gamma<0$ can robustly generate strong superposition. We should add a remark on this issue. We can also provide a scan or sensitive study of the choice of thresholds in the Appendix.\n    * The number of ETF-like vectors has a bound $\\sim m^2/2$. When $m^2/2<n$, the less important features cannot have small overlaps and have to decrease their norms to have less interference. The two distinct modes are being in ETF-like configuration (with mutually small overlap) or not. This particular behavior is irrelevant to LLMs since $m^2/2>n$ is always true. Please see Figure 19, LLM embedding vector norms is not bimodal. Then, LLM loss should be dominated by the strongly represented features (ETF-like), more likely leading to $\\alpha_m = 1$, which seems to be true. We scan all kinds of behaviors of toy models, then identify which regime LLMs are in, and use the mechanism found from the toy model in that regime to explain the LLM behaviors.\n\n* Main weakness 3: We are sorry for our oversight. The assumption is that all biases are zero. In our experiments, since both $m$ and $n$ are large, the expected value of $x_i~(i>m)$, proportional to the probability of $p_i$, is extremely small, approximately satisfying the assumption.\n\n* Main weakness 4: \"ETF-like\" has two main points: (i) Overlaps distributed closely around $\\kappa$ in Eq. (5); (ii) The number of such vectors has an upper bound. We verified point (i) by Figure 6d (i.e., the mean of overlaps agrees with $\\kappa$) and Figure 15 (i.e., the variance is small at least for small $\\alpha$). And yes, weakly represented features behave differently, as they have much larger mean overlap (compare Figure 16 and Figure 6d). We verified point (ii) in Figure 6c, where the bound is not exactly $m^2/2$ but around. This discrepancy is part of why we refer to the vectors as being \"ETF-like\" (we do not claim they form an ETF). What we can rigorously say is that when $\\alpha = 0$, the optimal configuration is an ETF. However, when $\\alpha$ increases, one would expect important features to occupy larger angular space and break the evenness. Yet, we still observe that the vectors are similar to ETF for a wide range of $\\alpha$ and yield a robust $\\alpha_m \\approx 1$. In our Discussion, we admit our main weakness is that we cannot analytically solve the toy model. What we did was to connect empirical findings and simple theoretical examples to generate a zeroth-order understanding.\n\n* Main weakness 5: This problem is also due to the fact that we cannot solve the toy model now. Specifically, we found that solving the directions of the embedding vectors is unexpectedly hard. What we are actively trying is to assume random directions and solve vector norms and biases. We are curious how many phenomena can be explained by such an ansartz. What we presented in the paragraph starting the line 217 is a hypothesized explanation for the transition observed. Some results in Figure 16 are confusing to us, too. The proposed picture seems to agree more with observations in the small $\\alpha$ regime, but not the regime after the transition. Our understanding is that when $\\alpha$ is large, the overlap variance is also getting large (e.g., Figure 15). So, the decreasing mean overlap over weakly represented features does not mean they are not squeezed. The weakly represented ones may be squeezed into different positions in the space (and have lower mean but higher variance) such that the most important features can have lower overlap with each other. As long as each weakly represented feature contributes to constant loss (e.g., has one large overlap), our proposed explanation may be valid. Again, the study of direction configurations is hard in high-dimensional space for both theory and experiment. We should highlight that our explanation for the transition is one hypothesis. We believe that the empirical results are correct and worth sharing, and many theoretical studies can be done later by us or others with various tools.\n\n* Additional concerns 1: We defined \"the absence of superposition\" in line 59. We will rewrite to make it clear what \"weak superposition\" is.\n\n* Additional concerns 2: We will use \"input sample\" or \"input\".\n\n* Additional concerns 3: Thank you, we will explain more in the caption.\n\n* Additional concerns 4: Only weight decay. Since we used AdamW, weight decay is a separate line of code.\n\n* Additional concerns 5: Thanks for the suggestion! This would help a lot.\n\n* Additional concerns 6: $m = 10, 15, 39, 63, 100$. They are roughly uniform in the log scale. Will make this clear.\n\n* Additional concerns 7: Thanks! We will revise.\n\n* Additional concerns 8: You are right, we will revise.\n\n* Additional concerns 9: The second line is copied from Figure 5b. Therefore, we can summarize all behaviors of $\\alpha_m$ in different regimes on one plot. We tested 10 $\\gamma$ evenly from $-1$ to $1$. The points with high weight decay have $\\alpha_m \\ll 1$. And the reason is provided around line 166: if weight decay is too large, the number of learned features will be fewer than $m$, which leads to slow decay of loss and is not ideal.\n\n* Additional concerns 10: For models with sizes smaller than 3 billion, the embedding and LM head matrices are usually tied. Larger models usually do not tie the matrices. Analyzing either matrix leads to the same conclusion. However, analyzing the embedding matrix will lead us to a lengthy discussion on how the representation error would propagate through transformer layers, which may distract from the focus of this paper. We can provide the analysis result of the embedding matrices in the Appendix."}, {"review_id": "HIi24VelpU", "reviewer": "Reviewer_H5Wk", "summary": "This paper proposes an explanation for the power-law scaling behavior of LLMs' loss with model size, connecting it to superposition—the phenomenon where neural networks represent more features than their model dimensions by allowing some degree of interference (non-orthogonality) between feature representations. The authors build upon a previously introduced single-hidden-layer toy model, originally designed to demonstrate superposition, modifying it by incorporating a weight decay (or growth) term into the loss. This modification enables control over the degree of superposition, allowing its effects to be studied independently of data properties. The authors find that in a weak superposition regime, where only the most important (frequent) features are represented without interference, loss scales as a power law with model size only if the data distribution itself follows a power law. However, in a strong superposition regime, where all features are represented with interference and this interference dominates the loss, the loss scales as a power law with model size regardless of the data distribution. They provide a geometric explanation for this behavior in their toy model: in the strong superposition regime, the optimal representational strategy to minimize interference approximates an equal-angle tight frame. Under these conditions, the squared dot product between representations scales inversely with model size according to a power law with exponent 1. The authors then analyze the unembedding layers of several publicly available LLMs within a framework analogous to their toy model, finding evidence that these models operate in the strong superposition regime. This suggests that the mechanism identified in the toy model might also dominate the loss behavior of real LLMs, potentially explaining the empirically observed power-law scaling with model size. Furthermore, the empirically observed scaling exponents closely match the predicted value of 1. The authors also demonstrate alignment with the Chinchilla scaling laws by converting Chinchilla’s parameter-count-based term into a model-width-based term (consistent with their single-hidden-layer framework), again obtaining an exponent close to 1.", "questions": "The points below are intended for discussion—feel free to consider them low priority and/or disregard some of them. The questions are intuition-driven, so please forgive any lack of rigor.\n\n- I would like to better understand your thinking behind the following phrasing:\n\n  > *\"Conceptually, fitting functions or manifolds and learning skills or grammars are primarily tasks of the transformer layers, while representation learning is more directly tied to the embedding matrix and the language model head.\"*\n\n  Defining representation learning strictly as learning token embeddings seems unusual to me. Aren’t intermediate activations also representations, capturing increasingly abstract and composed features (at least up to the middle layers)? Would it perhaps be beneficial to introduce a more specific term for what you call \"representation loss,\" clarifying that you specifically refer to non-contextual representations?\n\n- Given the compositional process across layers, which yields a combinatorially increasing number of features far exceeding the vocabulary size, should we expect loss due to superposition (geometric interference) to play an even more dominant role in what you've termed the _parsing loss_? Would this make weak superposition virtually impossible in LLMs?\n\n- If we do not assume the Linear Representation Hypothesis to hold, would this have any direct implications for your theory? In its current form I guess it would not, given that the non-contextual representations (token embeddings) you analyze are linear by definition, but I'm curious whether you foresee any implications arising from such a scenario—perhaps specifically for the parsing loss.\n\n- Since natural language data often follow power-law distributions, other theories relying on this assumption should also be considered plausible explanations for LLM scaling behavior. It would be beneficial to see a discussion regarding how compatible or complementary your theory of neural scaling with model size is relative to previously proposed theories.\n\n- What do you consider the most likely approach for falsifying your theory?\n\n- Considering that superposition has mostly been presented as a negative phenomenon and a significant interpretability challenge for neural networks, I suggest discussing the potentially adverse implications of your proposal to encourage superposition as a strategy for improving performance. Especially given that AI safety is becoming increasingly prominent, this concern merits acknowledgment and thoughtful discussion.\n\n- Typo in the caption of Figure 6: \"at stong superposition\" → \"at strong superposition\".", "limitations": "The authors should discuss the potential negative implications for AI interpretability and safety arising from their suggestion to promote superposition in neural networks as a means of improving performance.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n- A compelling and somewhat intuitive argument presenting a novel connection between two highly relevant and interesting neural network phenomena: superposition and scaling laws.\n- A useful and straightforward extension of the superposition toy model that allows interventions to induce or inhibit superposition, resulting in observations largely independent of data distribution or model size. This differentiates it from previous explanations of neural scaling laws that relied on data-distribution assumptions (e.g. uniform on a d-dimensional manifold [15]; or power-law distributed quanta [19]).\n- A rigorous, experiment-rich investigation of the toy model, revealing interesting behaviors across varying data distributions and degrees of superposition, consistent with a geometric explanation of power-law scaling under optimal arrangement conditions to minimize interference in superposition.\n- Insightful experiments with real LLMs that provide empirical support for the proposed theory.\n- The work goes beyond a purely descriptive model, potentially offering actionable insights for improving LLM architectures and training strategies.\n- The authors suggest many different directions for future work based on their model and gained insights, showcasing that it can be an important contribution to the community even beyond the results presented here.\n- Clearly written and overall very well presented.\n\n**Weaknesses**:\n\nAny potential limitation I identified has already been acknowledged and thoroughly discussed by the authors and does not, in my opinion, remotely warrant rejection. See the Questions section for additional points intended for discussion.", "was_revised": true, "final_justification": "I will maintain my score. I found the paper to be technically solid, innovative, and potentially of significant impact in the field, both in advancing understanding and in driving improvement. Most of the weaknesses identified by other reviewers appear to stem from difficulties in reading the paper, which I believe will be addressed in the final version.", "rebuttal": "We are grateful for the reviewer's appreciation and helpful questions. We will reply to the Questions and Limitations point by point as follows.\n\n* Question 1 (Phrasing of a sentence): We are sorry that we misused the terminology \"representation learning\". Our representation-limited error is due to the imperfect token or concept embedding in a low-dimensional embedding space. In language, tokens are atomic features in our mind, and we can conceptually connect them to features in our toy model. And practically, token embeddings are easy to access and analyze. Yes, if the token embedding has some error, the more abstract representations in the middle layers will also have an error. We guess the error of composite features would have the same scaling with width as the atomic features. Our thinking behind the sentence is that there can be two sources of the total error, one is from the imperfect embedding itself (explicit focus of this paper), and the other is from imperfect calculation or transition acting on the embeddings (which is more related to the transformer layers). We propose keeping the phrase \"representation\" as it could apply to either components of error, although we could also replace it with \"embedding\" if this would avoid confusion.\n\n* Question 2 (Middle layer superposition): This is a great question! In the AI biology work by Anthropic, they found that in poem completion tasks, the model can plan ahead by representing the next whole sentence at the end of the prompt using the embedding dimension of one token. In the middle layers, the number of concepts to represent may be huge, especially when the context length is large. So, in our mind, for LLMs, superposition may be impossible to avoid. Yet, we do not have more rigorous arguments for this. Token embedding is a subproblem for representation in LLMs and should share the same scaling. There is definitely more to study in representations in the middle layers, and we can add discussions on this topic in the Discussion section. We also imagine that, in terms of new architectures, an adaptive hidden dimension may help performance: if one position needs a lot of previous tokens to understand or is needed by a lot of following tokens for their decoding, it should have a larger embedding dimension.\n\n* Question 3 (Linear representation hypothesis): Thanks for this question. If the linear representation hypothesis (LRH) holds, we expect the parsing loss to have a term that is the number of compositions times the error of one embedding. If LRH does not hold, there are a lot of possible situations. For example, the embedding may be on a manifold. But as long as the manifold is locally Euclidean, the 1/width scaling may still be robust (we are not sure). Another possibility when LRH does not hold is that there are infinitely many independent features, and LLMs need to keep embedding new independent features in the middle layers, which may increase the coefficient but not change the 1/width scaling. We are also sorry for the lack of rigor and preliminary thinking.\n\n* Question 4 (Previous theory): Thanks for mentioning this. Since different theories on neural scaling laws have different setups, we are not very confident in comparing them directly. As far as we can see, previous theories like the phenomenological quanta model and the formal kernel method have the same spirit as our no superposition result. Our finding in the strong superposition regime is therefore complementary to previous theories, suggesting a new mechanism due to the intrinsic model configuration rather than data properties.\n\n* Question 5 (Falsifying our theory): We think a direct experiment is to fix a large depth and only increase the width of LLMs, and study clearly the scaling of loss with width. If the scaling exponent obtained from this experiment is far from 1, then our mechanism cannot explain LLM behaviors. Another approach is about language itself. If one can argue that atomic features of language are not tokens but something much fewer than the number of tokens and embedding dimension, then our theory is false.\n\n* Question 6 (Interpretability and AI safety): Thank you for reminding us about interpretability and safety issues. We agree that encouraging superposition may lead to more powerful but less interpretable models and can have a negative impact on safety. We will add a discussion on these topics.\n\n* Question 7 (Typo): Thank you for the careful reading. We will revise.\n\n* Limitations: Please see reply to Question 6."}, {"review_id": "GDrrtbdl8D", "reviewer": "Reviewer_hFfh", "summary": "This work analyzes neural scaling laws in terms of the concept of superposition. The main body of the paper centers around a toy model for superposition first advanced by a team at Anthropic. In this model a small two-layer network is trained on a synthetic dataset where all features in the dataset can be directly controlled. The work identifies an optimization scheme such that strong superposition and weak supervision can be controlled via a weight decay parameter. Here, strong supervision is the state where the hidden activations of features fail to be fully orthogonal and the extent to which they are not orthogonal is determined by the frequency by which the feature appears. Weak superposition on the other hand is the situation where a subset of the features are represented by orthogonal vectors in the model and the rest are mapped to zero. The paper looks at how adjusting this hyperparameter, as well as the distribution of features impacts scaling constants in the model. They find two regimes. One, corresponding to strong superposition, where scaling closely follows a power law and one where it fails to follow a power law corresponding to weak superposition. The paper justifies this through an analysis of their toy model. The paper goes on to explore the statistics of large-scale LLMs and provides evidence that these fall into the strong superposition scheme.", "questions": "- The paper changes the hyperparameter settings so as to induce different kinds of superposition. The paper then observes different scaling properties. There is certainly a correlation here, but how do we know that superposition causes the scaling laws (it is possible that this falls out of the analysis and I am misremembering)?\n- It is this reviewer’s understanding that the paper does not consider the case where features are correlated. This is an essential feature of real-world data and it would be interesting to consider this. Have the authors thought about how this would change their model?\n- What does the statement ‘Conceptually, fitting…’ on Line 46 mean?", "limitations": "The paper does not address limitations. Limitations might include the fact that most of the analysis is done on a very simple model and would need to be supported by evidence in larger, more real-world datasets/models.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- **Analyzing the relationship between scaling laws and superposition is a natural research direction:** Scaling laws and superposition are both central ideas that have arisen through the development and study of LLMs. It is natural to ask in what ways they may be related. The paper lays out a nice framework with which to think about this.\n- **The analysis of the toy set-up is comprehensive and yields a nice story:** Though the toy model used in this paper is not a contribution of the work, the paper puts it to good use and it is a very helpful aid when reasoning through the ideas presented and discussed by the paper. The notation and descriptions of the experiments are reasonably easy to follow and do not ask too much of the reader’s mathematical background.\n\t\nWeaknesses\n- **The structure of the paper could be improved:** The paper is primarily one long analysis of the toy model. This analysis could easily be partitioned into sections that would help organize the content. For instance, the analysis of the weak superposition regime could be a section (or subsection), the analysis of the strong superposition regime could be a section, and the analysis of non-toy models could be a section. This would enable the main takeaways of the paper to be more easily extracted and would help readers that do not want to read through the paper linearly. It would also be helpful to provide a conclusion or discussion section which summarizes the results and puts them into context.\n- **The section on real-world models is weak when compared to the analysis of the toy set-up:** As noted above, the reviewer felt that the toy dataset section was helpful and made the paper much more interesting since it enabled a more detailed analysis both from the perspective of being able to finely control the whole system for experiments (including the data distribution) but also being able to theoretically justify various observations. However, this toy set-up is very far from real machine learning. The paper includes a short section in the end where real LLMs are analyzed for some of the same properties that were observed in the toy model, but the reviewer found this far less convincing than the rest of the paper. It is unreasonable to ask too much in this larger setting, but a more detailed analysis would be helpful. Furthermore, there may be small scale experiments that can be run that still use real data and our easier to analyze than large-scale LLMs.\n\t\nNitpicks\n- The paper uses the terminology ‘weak representations’ and ‘weak superposition’, the reviewer would suggest changing the former as these are not the same thing and may be confused with the current naming convention.\n- Line 40: ‘depend sensitively on’ $\\mapsto$ ‘are sensitive to’.", "was_revised": true, "final_justification": "The reviewer believes that this paper represents a promising direction of research with some interesting initial results. The primary aspects holding back the paper are presentation issues and lack of connection to real-world examples. The reviewer believes that these are both very addressable and hopes to see a new version of this work soon.", "rebuttal": "We thank the reviewer for the helpful questions and suggestions. We will reply to the weaknesses, questions, and limitations point by point as follows.\n\n* Weakness 1 (Structure of the paper): We are sorry for the confusion and inconvenience due to our presentation. The reviewer's suggestion is very helpful, which we will follow. We will also highlight our key definitions, assumptions, and takeaways by the Theorem environment. We attempted to include too many results in the main text. We will emphasize only the most important ones and save space to move Discussion back to the main text.\n\n* Weakness 2 (Weak LLM analysis): Yes, we totally agree with the reviewer, and we should acknowledge this limitation more. Our toy setup is far from actual LLMs, and we used the last paragraph of Section 2 to discuss key differences and why some details in LLMs may not matter. In Discussion (Appendix B), we also discussed other sources of error in LLMs and how to better interpret Figure 8b. Nevertheless, the connection between toy models and LLMs is not rigorous. Regarding the analysis of LLMs, we studied the embedding vectors, loss scaling with width, and token frequency distribution (Figure 22). We tried to analyze properties we have access to and that can be connected to the toy model in some way. The philosophy is that we study all kinds of possible behaviors in the toy model, identify the regime where LLMs are in, and connect LLM behaviors to mechanisms found via the toy model. We can move more details on LLM analysis from the Appendix to the main text to strengthen the last part. We are also willing to conduct mechanistic investigations on LLMs if there is any concrete quantity to look at.\n\n* Nitpick 1: Thank you for this point. We can use \"well represented\" vs. \"poorly represented\" or \"over represented\" vs. \"under represented\".\n\n* Nitpick 2: Thanks, we will revise.\n\n* Question 1 (Causality?): This is a conceptually important question. Let's think about another example first. Say we can change the temperature to make water ice or liquid. A behavior we found at the lower temperature state is a behavior of the ice. And a behavior at higher temperatures is that of the liquid state. By changing weight decay, we enter the state of \"weak superposition\" or \"strong superposition\". We can always say that the scaling law is a property of the strong superposition regime. To be rigorous, \"geometry of representations in the strong superposition regime yields a scaling law with width\" is the statement, and \"superposition causes scaling law\" is a simpler version of it.\n\n* Question 2 (Correlation of features): This is a great question. If we consider the correlation between different dimensions of the input vector, the \"independent atomic features\" should be the eigenvectors of the correlation matrix instead of the dimensions of the input vector. Theoretically, after rotating the weight matrices to another basis, the current analysis still follows. Yet, the non-linearity can introduce extra complexity, which we are not certain about and should study via experiments. We expect the scaling to be robust even if input dimensions are correlated. We should acknowledge this point in our Discussion. And if we consider the eigenvectors of the correlation matrix to be features, the feature frequencies can be different from those of the input dimensions. We measured the co-occurrence matrix of tokens in the Pile, varying context length from 3 to 512. The eigenvalues of the co-occurrence matrix have a scaling exponent $\\alpha$ from 1 to 1.2, which does not change much. So, we also do not expect that considering token correlation would change our conclusion.\n\n* Question 3 (Meaning of a sentence): We are sorry for the confusion, \"fitting function or manifold\" and \"learning skills\" refer to previous mechanisms proposed to explain the neural scaling laws. We will add citations near these terms. The sentence tries to say that some previous mechanisms conceptually describe what the transformer layers do, while there is another source of error -- representation or embedding -- that needs studying.\n\n* Limitations: We discussed some of our limitations in Appendix B Discussion. We will emphasize what the reviewer pointed out. And we will try to move some results to the Appendix such that we can move the Discussion section back to the main text."}, {"review_id": "fVQUZxpadW", "reviewer": "Reviewer_xNH3", "summary": "This paper explores why larger language models (LLMs) perform better by examining the role of representation superposition in neural scaling laws. The authors construct a toy model to study how superposition and data structure influence the scaling of loss with model size. They find that under strong superposition, where all features are represented but overlap, the loss scales robustly as an inverse power law with model dimension, largely independent of feature frequency distributions. In contrast, weak superposition shows more fragile scaling behavior. The study concludes that representation superposition is a key mechanism underlying observed neural scaling laws and suggests that promoting superposition could lead to more efficient training strategies and model architectures for LLMs.", "questions": "In my understanding, a larger dimension implies greater superposition, which also means more parameters. How can we be inspired to achieve better performance with fewer parameters?", "limitations": "The study focuses primarily on representation loss and does not explore other factors like dataset size or training steps in detail.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper provides a comprehensive analysis of how representation superposition affects neural scaling laws, offering new insights into why larger language models perform better.\n\n(2) It systematically studies the relationship between superposition and data structure, using both theoretical analysis and extensive experiments to support its conclusions.\n\n(3) The findings have practical implications for developing more efficient training strategies and model architectures, potentially leading to better performance with less computation.\n\nWeakness:\n\n(1) The paper relies on a simplified toy model, which may not fully capture the complexities of real-world large language models.\n\n(2) The theoretical analysis is not fully rigorous, leaving some deeper questions unanswered and requiring further investigation.\n\n(3) The study focuses primarily on representation loss and does not explore other factors like dataset size or training steps in detail.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the nice comments and helpful questions. We will reply to the weaknesses and limitations point by point.\n\n* Weakness 1: We agree, and we may emphasize this weakness more in our discussion of limitations. We tried to use the last paragraph in Section 2 to summarize some key differences between the LLMs and the toy model and justify why some details in LLMs can be ignored, which should also be highlighted or strengthened.\n\n* Weakness 2: This is true. The robustness of $\\alpha_m\\approx 1$ across $\\alpha$ and the transition of $\\alpha_m$ are observations, and we try to provide a picture of the underlying mechanisms with more details like vector norms and overlaps observed from the experiments. However, we are limited by the fact that we are not able to analytically solve the directions of the representation vectors. We are actively trying many theoretical analysis methods. One simplified analysis is to assume random directions of the embedding vectors and solve for norms and biases.\n\n* Weakness 3: Thank you for mentioning this. We had a brief discussion on other scaling laws in Appendix B. We are now actively studying the scaling with training steps. The scaling with dataset size should be the same as that with training steps in the online learning setup. The training dynamics in the weak superposition regime have been solved. Yet, the training dynamics in the strong superposition regime is hard to analyze.\n\n* Questions: Given the number of features, $n$, a larger dimension $m$ may lead to less superposition. When we talked about equal performance with fewer parameters, we were imagining a large model with weak superposition and a small model with strong superposition could have similar losses (which is true in the toy models). \n\n* Limitations: Please see reply to Weakness 3."}]

Author rebuttal:

Meta-review: This paper investigates the role of superposition in neural scaling laws. The authors introduce a toy model to examine how superposition and data structure affect the scaling of loss with model size. They find that under strong superposition where all features are represented but overlap, the loss scales robustly as an inverse power law with model dimension. In contrast, weak superposition leads to more fragile scaling behavior.

The reviewers agree that the findings are novel, well-motivated, and supported by careful experimentation. The paper is clearly written, and the analysis illuminates an important yet underexplored mechanism in neural scaling.
Decision: Accept (oral)