SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.2 Monogenic versus polygenic scaling curves
In our introduction of the Quantization Hypothesis in Section 2 and our multitask sparse parity study in Section 3 we modeled network performance on individual samples as benefitting from a single quantum - all samples belong to a single subtask, which is either solved or not solved in a binary fashion. In our model and on multitask sparse parity, scaling curves on individual examples all exhibit emergence - loss on individual examples undergoes a sharp transition at a particular scale of parameters or data. Do we observe this in large language models?
Inspecting a large number of per-token (per-sample) scaling curves, we observe a variety of scaling behaviors. On some samples, loss drops at a particular scale. More typically though, loss improves at multiple scales. If the Quantization Hypothesis is true and the effect of scaling is to simply add new quanta to the model, then for per-sample loss curves to show progress at multiple scales, those samples must benefit from multiple quanta additively. As first mentioned in Section 2, we borrow terminology from genetics and refer to prediction problems for which the model's performance is determined by a single quantum as monogenic (akin to when a single gene determines a trait) and as polygenic when multiple quanta influence performance (in analogy to when multiple genes contribute to a trait). In multitask sparse parity, all prediction problems are monogenic. In natural language, we observe that model performance on most tokens improves at multiple scales, suggesting that most tokens are polygenic, but we can find tokens for which loss drops as a single phase transition in scale. Polygenicity forms a spectrum: the smoothness of the loss curve can vary substantially between examples, presumably with some prediction problems using few quanta and others using many. In Figure 4, we show extreme examples of both monogenic and polygenic samples.
Note that our monogenic/polygenic taxonomy of model behaviors assumes that QH1 and QH2 are true. However, it could be the case that there isn't an underlying discreteness to what is learned, or that scaling totally changes what networks learn, rather than simply adding additional quanta. Whether scaling truly has the effect we described will have to be investigated in future studies of the internals of neural networks. We also note that it is possible that sharp transitions in the per-token loss curves could be due to noise – if we had multiple runs with different random seeds for each model scale, we could better test whether the mean loss across seeds decreases smoothly or if there is a genuine discreteness where gradual progress is impossible for apparently “monogenic” tokens.
5 The quanta of language modeling
We have conjectured that the internals and behavior of language models are decomposable into an enumerable set of modules and associated skills (quanta). What might these basic building blocks of LLMs be? In this section, we develop a preliminary method to discover quanta. In particular, we will attempt to cluster tokens in a language corpus according to what knowledge or skill LLMs use to predict those tokens from their context. Our goal is to find coherent clusters of language model behavior that each reveals some distinct skill that the model has learned. Note that in clustering tokens to discover quanta, we are making the likely unrealistic assumption that these tokens are monogenic – that there is only one quantum involved in predicting each token. Not also that these clusters of behavior will not give us a mechanistic understanding of the quanta, but simply provide examples of LLM skills which could be studied further in future work.
We propose the use of gradients to cluster next-token prediction samples, where a "sample" consists of a token and its context in some document. Given some model, we will cluster two samples together if the gradient of the model's loss on each sample w.r.t. the model's parameters is similar for the two samples. The intuition for using gradients is as follows: if a model uses the same internal module to generate its prediction on two samples, then the gradients for parameters within the module may be nonzero and similar for the two samples (and possibly  $\approx 0$  for parameters in irrelevant modules). If a model uses different modules to generate its prediction on different samples, then the gradients may not overlap. We therefore use gradient similarity as a proxy for mechanistic similarity – whether a model uses similar mechanisms/modules to generate its prediction on distinct samples. While crude, we find that gradients contain enough information to allow us to automatically discover many coherent clusters of LLM behavior using the following algorithm:
Quanta Discovery from Gradients (QDG): We will use spectral clustering on gradients to find clusters of samples whose gradient has nonzero cosine similarity. Given a set of samples  $(x_i, y_i)$  and a model  $f_\theta$ , we compute gradients for each sample  $g_i = \nabla_\theta L(f_\theta(x_i), y_i)$ . We then normalize these gradients  $g_i \mapsto \hat{g}_i$  so that  $\hat{g}_i \cdot \hat{g}_i = 1$ . Let  $A$  be a matrix whose rows are the normalized gradients:  $A_{i,} = \hat{g}_i$ . If we are clustering  $d$  samples and our model has  $n$  parameters,  $A$  has shape  $(d, n)$ . We compute an affinity matrix  $C = A A^T$ , a matrix of shape  $(d, d)$  where  $C_{ij} = \hat{g}_i \cdot \hat{g}_j$ , the cosine similarity between gradients  $g_i, g_j$ . From this, we compute an affinity matrix of the angular similarities  $\hat{C}$  (which take values in  $[0, 1]$ ) via  $\hat{C}_{ij} = 1 - \arccos(C_{ij}) / \pi$ . We then perform spectral clustering with  $\hat{C}$  to cluster samples.
QDG is expensive to compute for large models and for large numbers of samples. We therefore only apply it to the smallest model in the Pythia suite, which has 19m non-embedding parameters. We cluster 10000 tokens on which this model is confident and correct in its prediction, achieving less than 0.1 nats of cross-entropy. See Appendix C.1 for more detail.
We find that many, though not all, QDG clusters reveal some coherent model behavior. We show examples from clusters in Figure 1 and Figure 13. These clusters were found with the spectral clustering hyperparameter  $n\_clusters = 400$ . While most clusters involve the prediction of the same token, manually inspecting these clusters we find that they usually involve predicting the same token for a coherent reason, rather than being based merely on having the same output. We also find clusters for more abstract prediction rules. For instance, the quantum shown on the left column of Figure 1 is the skill of incrementing a numerical sequence, and the examples involve predicting a variety of different tokens representing numbers.
5.1 The natural distribution over language modeling quanta
In our model, some quanta are more frequently used than others. If these frequencies follow a power law in accordance with the Quantization Hypothesis, then we may expect QDG cluster sizes to be governed by a power law. The measured scaling exponent of  $\alpha_{N} = 0.083$  from Figure 3 implies a power law distribution over quanta with exponent  $-1.083$ . Do the cluster sizes follow this?
Figure 5 shows rank-frequency curves for clusters discovered with QDG for varying choices of n_clusters. These curves sort the clusters according to their size and then plot size against cluster index (rank). We plot rank-frequency curves for many choices of n_clusters since it is unclear a priori which n_clusters to use. When we measure the slope of the rank-frequency curve, we measure it from the envelope formed by the many rank-frequency curves, a practice which we
Figure 5: Left: angular similarity between model gradients for a variety of natural language samples. Samples are reordered according to their QDG cluster (with 400 clusters) to reveal the block-diagonal structure of the similarity matrix. We visualize a small part of the overall similarity matrix in this plot – note that not all clusters are as visibly distinct as the ones shown. Right: rank-frequency plot of QDG clusters. We measure the slope of the envelope of the rank-frequency curves from cluster rank 100-1000 to be  $\approx -1.24$ , which is a steeper than the slope of -1.08 expected from the measured parameter-scaling exponent from Figure 3, though within the margin of error given the uncertainty of our clustering methodology. See Appendix E for a discussion of the bias/uncertainty of our method.
discuss in Appendix E. Biases in the clustering algorithm and inherent noise in model gradients make clustering imperfect, and lead to high uncertainty of our the measured power law exponent. From our analysis in Appendix E, we think that extracting the power law exponent over quanta utilization frequency by measuring the slope of the rank-frequency curve should have uncertainty of at least 0.2. We also note that some rank-frequency curves don't look like a clean power law. In Appendix E, Figure 16 we find that we can get similar-looking curves in a toy model of this clustering process when the dimension and noise is high. Between ranks 100-1000, we measure a slope of  $\approx -1.24$ , about 0.16 off our expected slope of  $-1.08$ , and so within the margin of error. We are encouraged that the size of our discovered clusters seem to decay at a rate (very roughly) compatible with observed neural scaling exponents, in line with our theory. However, less naive clustering schemes, operating on more samples with more clusters, could be useful to sharpen this measurement.
Models of neural scaling: Several models of neural scaling laws have been proposed in prior work. Sharma and Kaplan [31] explain power law scaling w.r.t. model parameters using an argument from approximation theory, which relates neural scaling exponents to the dimension of the data manifold  $d$ . Michaud et al. [32] point out that effective dimension  $d$  could be generalized to the maximum arity of the target function's computation graph for sparse compositional problems. Bahri et al. [33] generalized the model of Sharma and Kaplan [31] to scaling w.r.t. dataset size, additionally relating scaling exponents to the power law spectrum of certain kernels. Maloney et al. [34] develop an exactly solvable random-feature model of scaling, from which they derive a joint parameter-data scaling law. Bordelon et al. [35] develop a model of data scaling for kernels, decomposing the generalization error into a sum over eigenmodes, whereas we decompose error into a sum over quanta. Arguably the closest prior work to ours is Hutter [36], who develops a model of data scaling wherein a discrete set of "features" must be learned. In this model, a feature is learned if it occurs at least once in the training set. If the features are Zipfian distributed, this produces power law scaling in expectation but with high variance. In our model, using a data threshold  $\tau \gg 1$  lowers the variance in the scaling curve, and we also considered scaling w.r.t. parameters and applied the model to real networks.
Understanding emergent abilities: Wei et al. [8] and Srivastava et al. [37] document examples of emergent abilities in large language models, though Schaeffer et al. [38] suggest that these examples
are an artifact of the metric used to evaluate performance. Arora and Goyal [39] develop a framework for the emergence of "skills", where predicting text requires combining multiple different skills from an underlying set of language skills.
Miscellaneous: The topic of phase transitions in machine learning is not new [40], but our work was strongly influenced by the recent work of Olsson et al. [25] who observe a phase change from the formation of induction heads and especially Nanda et al. [13] who conjecture that phase changes may be ubiquitous. Simon et al. [41] also exhibit a task where learning proceeds as a series of discrete steps. Chen et al. [42] develop a framework for understanding LLM "skills" in a hierarchy and for choosing data to more efficiently learn desired skills. Chan et al. [43] study how a Zipfian data distribution influences in-context learning.
Summary: The Quantization Hypothesis posits that for some types of prediction problems, models must learn a discrete (quantized) set of modules/knowledge/skills (quanta). When data is distributed in such a way that the "use frequencies" of these quanta follow a power law, then power law neural scaling can arise as models learn more and more quanta, with smooth scaling curves averaging over many small cases of emergence. We presented a toy dataset where neural scaling exhibits these properties. We then documented how language model scaling curves decompose, beyond simply how the mean loss scales. Lastly, we developed a method to discover quanta from the internal structure of trained models, from which we were able to enumerate a large number of skills of a small language model. The frequencies at which the quanta we discover are used for prediction in natural text seem to roughly track the power law our theory would predict, though this measurement is quite imprecise.
Limitations: While the Quantization Hypothesis appears to hold for our toy datasets, much work remains in investigating to what extent it holds for natural tasks like language modeling. Probably our riskiest assumption was that there is an underlying discreteness to everything that models learn. Gradual scaling seems typical in LLMs [38], and it could be more parsimonious to model neural scaling as an underlying smooth process rather than to assume that most tasks are highly polygenic with underlying discrete quanta. Note also that in our model of scaling w.r.t. parameters  $N$ , having more parameters merely increases the capacity of the network. In practice however, larger networks are more efficient learners [7], and one can trade off between parameters and data, whereas in our model parameters and data independently bottleneck the number of quanta that can be learned. Additionally, we modeled the quanta as being independent, where learning order is given just by the use frequencies, but it could make more sense to think of the quanta as living in a hierarchical dependency graph. Lastly, our QDG method is neither very principled nor scalable, and much better methods could likely be developed to discover quanta and study their statistics for larger models and across more samples.
Implications for emergence and forecasting: Srivastava et al. [37] find that on some tasks, neural scaling has high "linearity", with gradual improvements to scale, with other tasks displaying "breakthroughness", where performance improves sharply at some scale. In our model, high linearly would result from a task's relevant quanta being widely spread along the Q Sequence, and high breakthroughness would result from a task being monogenic or from the relevant quanta being close together in the Q Sequence. Our model also suggests that future capabilities could be forecasted if one could estimate the frequency at which that skill would benefit prediction in the training corpus.
Implications for mechanistic interpretability: If the Quantization Hypothesis is correct, then understanding a network reduces to enumerating its quanta. Having done this, the quanta could perhaps then be translated into a more interpretable format (something like code), studied in this format, and eventually executed in this format, rather than via the operation of the network.
Outlook: Lastly, our decomposition of networks into quanta is reminiscent of Minsky's Society of Mind [44] perspective that minds are decomposable into individually mindless "agents". If this decomposition is indeed possible, then the quanta (agents) become natural objects of study within networks. This mesoscale understanding of networks, in terms of the internal modules which collectively constitute their performance, could perhaps act like statistical physics for deep learning, allowing us to bridge our microscale understanding of low-level training dynamics and our macroscale understanding of model performance.
We thank Tamay Besiroglu, Neel Nanda, Tony Wang, David Bau, Ben Edelman, Brian Cheung, Wes Gurnee, Stephen Casper, Peter Hase, Davis Brown, Eleni Shor, Max Nadeau, and Xander Davies for helpful conversations and feedback. We thank Lauro Langosco for helping with code to visualize samples from The Pile. This work was supported by the Foundational Questions Institute, the Rothberg Family Fund for Cognitive Science, the NSF Graduate Research Fellowship (Grant No. 2141064), and IAIFI through NSF grant PHY-2019786.
A More general scaling laws
If one learns the first  $n$  quanta, reducing the loss from  $b_{k}$  to  $a_{k}$  ( $1 \leq k \leq n$ ), while the loss remains  $b_{k}$  for  $k > n$ , the expected loss is given by:
$$
L _ {n} = \sum_ {k = 1} ^ {n} a _ {k} p _ {k} + \sum_ {k = n + 1} ^ {\infty} b _ {k} p _ {k}. \tag {3}
$$
In the main text, we used  $a_{k} = a$  and  $b_{k} = b$  for our model. However, one can imagine a variety of other choices for  $a_{k}$  and  $b_{k}$ .
Case 1  $b_{k} = -\log p_{k}$  and  $a_{k} = 0$ , where  $p_{k} = k^{-(\alpha +1)} / \zeta (\alpha +1)$ . This baseline for  $b_{k}$  is the error of a model which outputs the token frequencies, independent of the context (assuming that quanta involve the prediction of a particular token). The expected loss is given by:
$$
L _ {n} = \sum_ {k = 1} ^ {n} 0 \cdot p _ {k} + \sum_ {k = n + 1} ^ {\infty} (- \log p _ {k}) \cdot p _ {k} \approx \frac {1 + \alpha + \alpha \log \zeta (\alpha + 1)}{\alpha^ {2} \zeta (\alpha + 1)} n ^ {- \alpha} + \frac {\alpha + 1}{\alpha \zeta (\alpha + 1)} n ^ {- \alpha} \log n, \tag {4}
$$
which contains a power law term  $n^{-\alpha}$  plus a log term  $n^{-\alpha}\log n$ . For very large  $n$ , the log term can be ignored, so  $L$  is still approximately a power law of  $n$  with exponent  $-\alpha$ , shown in Figure 6.
Figure 6: Comparing different scaling laws. Setting  $a_{k} = 0$ , we compare  $b_{k} = -\log p_{k}$  (solid lines) and  $b_{k} = 1$  (dashed lines) for different alphas. Although the  $b_{k} = -\log p_{k}$  case would cause an extra loss term  $n^{-\alpha}\mathrm{log}n$  in addition to the power law term  $n^{-\alpha}$ , the loss becomes a power law asymptotically when  $n$  becomes large.
Case 2  $b_{k} = -\log p_{k}$  and  $a_{k} = -\log (Cp_{k})$  ( $C > 1$ ), where  $p_{k} = k^{-(\alpha +1)} / \zeta (\alpha +1)$ . The expected loss is given by:
$$
L _ {n} = \sum_ {k = 1} ^ {n} (- \log (C p _ {k})) \cdot p _ {k} + \sum_ {k = n + 1} ^ {\infty} (- \log p _ {k}) \cdot p _ {k} \approx \frac {\log C}{\alpha \zeta (\alpha + 1)} n ^ {- \alpha} - \log C + \frac {1 + \alpha + \alpha \log \zeta (\alpha + 1)}{\alpha^ {2} \zeta (\alpha + 1)}, \tag {5}
$$
which is a power law  $n^{-\alpha}$  plus a constant.
B Additional results on multitask sparse parity
Training dynamics: When loss is broken down by subtask on multitask sparse parity, learning curves consist of many reverse-S shaped curves, and mean loss decreases smoothly as an average over these curves. In Figure 7, we show loss versus time for each subtask for training runs in both the single-epoch and multi-epoch regimes. In Figure 8 we show how convergence time for each subtask relates to the frequency of that subtask.
Figure 7: Training dynamics on the multitask sparse parity dataset consist of many "phase transitions" when decomposed by subtask – the loss curve for each subtask drops following an initial plateau of no apparent progress, in line with [28]. The mean loss decreases smoothly, averaging over these phase transitions in the model's performance on subtasks. We show curves for single-epoch training (top) and multi-epoch training on 5 million samples (bottom). The dashed red line indicates the early stopping point where mean test loss is minimized. For these runs,  $\alpha = 0.4$ .
Figure 8: Convergence time for each subtask versus the frequency of that subtask. We see that convergence time  $S_{k}$  on subtask  $k$  is  $S_{k} \propto p_{k}^{-0.81}$  rather than  $S_{k} \propto p_{k}^{-1}$  as we had expected. This leads to a steeper scaling w.r.t.  $S$  than expected from theory. For these experiments, we used  $\alpha = 0.4$ , and so we would have predicted  $\alpha_{S} \approx 0.29$  but instead we get  $\alpha_{S} \approx 0.45$ . We consider the model to have converged on a subtask once it gets mean test loss less than 0.1 bits on that subtask.
Scaling for varying  $\alpha$ : In Figure 10 we show scaling curves on multitask sparse parity in  $N, S, D$  for a variety of quanta distribution parameters  $\alpha$ . While all scaling curves appear to be power laws, the relationship between  $\alpha_{N}, \alpha_{S}, \alpha_{D}$  and  $\alpha$  is not precisely as predicted by theory:
1. Parameter scaling: We observe that the relationship between  $\alpha_{N}$  and  $\alpha$  deviates a bit from the prediction  $\alpha_{N} = \alpha$ , with  $\alpha_{N} < \alpha$  for small  $\alpha$  and  $\alpha_{N} > \alpha$  for large  $\alpha$ . Perhaps model size does not influence learning just by changing capacity, but also by affecting optimization.
Figure 9: Number of subtasks learned  $(n)$ , including subtasks learned after early-stopping would have terminated the training run, versus training samples  $D$  for a variety of  $\alpha$ . We see that the relation  $n \propto D^{1 / (\alpha + 1)}$  approximately holds, in line with theory. Deviation from theory for the scaling exponent of loss  $L$  w.r.t.  $D$  therefore likely originates from our failure to regularize network training, leading to early-stopping ending training before some subtasks can be learned.
2. Step scaling: We observe that  $\alpha_{S}$  is consistently higher than the theoretical prediction  $\alpha / (\alpha + 1)$ . In Figure 8, we saw that the number of steps to convergence for each subtask did not precisely follow  $S_{k} \propto p_{k}^{-1}$ , but was closer to  $S_{k} \propto p_{k}^{-0.81}$ . This means that many subtasks converge faster than we would expect, producing a steeper scaling curve.
3. Data scaling: We observe that  $\alpha_{D}$  is substantially higher than the theoretical prediction  $\alpha / (\alpha + 1)$  for small  $\alpha$ . We think this may be related to the fact that early-stopping cuts off training before all subtasks are learned as observed in Figure 7. In Figure 9, we show how the number of subtasks learned  $n$ , when we include subtasks learned after early-stopping, seems to be in line with theory:  $n \propto D^{1 / (\alpha + 1)}$ .
Better understanding the precise nature of power law scaling on multitask sparse parity is an interesting avenue for future work.
Figure 10: Scaling in parameters  $(N)$ , single-epoch training time  $(S)$ , and multi-epoch training samples  $(D)$  for varying quanta power law distribution parameter  $\alpha$  on multitask sparse parity. We notice that scaling curves in steps  $S$  are typically steeper than the  $\alpha_{S} = \alpha / (\alpha + 1)$  predicted from theory, and that for low  $\alpha$  the scaling curves in  $D$  also deviate from theory substantially.
C Additional results on language models
In Figure 11 we show how the distribution over losses changes across time during a training run, rather than across model scales like in Figure 3.
In Figure 13 we show additional examples from clusters discovered with QDG.
Figure 11: Left: Training curves (scaling w.r.t. steps  $S$ ) of mean test loss for Pythia models. We measure exponents  $\alpha_{S}$  between 0.037 and 0.06. Center: the distribution  $p(L)$  over time. Over time, models achieve  $\approx 0$  loss on an increasing fraction of tokens, similar to scaling in model size. Right: The distribution  $L \cdot p(L)$  over time.
C.1 Details of application of QDG to LLMs