SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eric J. Michaud* Ziming Liu, Uzay Grit, and Max Tegmark MIT & IAIFI
We propose the Quantization Model of neural scaling laws, explaining both the observed power law dropoff of loss with model and data size, and also the sudden emergence of new capabilities with scale. We derive this model from what we call the Quantization Hypothesis, where network knowledge and skills are "quantized" into discrete chunks (quanta). We show that when quanta are learned in order of decreasing use frequency, then a power law in use frequencies explains observed power law scaling of loss. We validate this prediction on toy datasets, then study how scaling curves decompose for large language models. Using language model gradients, we automatically decompose model behavior into a diverse set of skills (quanta). We tentatively find that the frequency at which these quanta are used in the training distribution roughly follows a power law corresponding with the empirical scaling exponent for language models, a prediction of our theory. $^2$
In the aggregate, larger neural networks trained on more data perform better than smaller neural networks trained on less data, in a predictable way. Across a range of studies, mean test loss has been observed to decrease as a power law in both the number of network parameters  $(L \propto N^{-\alpha_N})$  and the number of training samples  $(L \propto D^{-\alpha_D})$  [1, 2, 3, 4, 5, 6, 7]. Although aggregate performance changes smoothly with scale, when particular capabilities are examined, larger models often have emergent abilities, i.e., qualitatively different performance than smaller models [8, 9]. Understanding and reconciling both facets of scaling – the predictable power law decrease in loss and the emergence of new capabilities at scale – is of both theoretical and practical interest [10]. Understanding how scaling changes what neural networks learn is entangled with core questions: what are deep neural networks doing internally, and will they will continue to improve with scale?
Recent studies of the internals of neural networks have found a variety of impressive algorithms learned by gradient descent [11, 12, 13, 14, 15]. As more work is put into understanding the structures learned by neural networks (the task of mechanistic interpretability), we may find more and more circuits [11, 16] in models, intelligible internal algorithms for accomplishing prediction in specific contexts. Can such analysis be scaled up to frontier models [17]? Two assumptions which, if true, would make us more optimistic about mechanistically understanding large models include (1) decomposability/modularity/sparsity [18, 19, 20, 21] – that large models are decomposable into parts, and only a small number of these parts are relevant to the model's behavior on any given sample and (2) universality [22, 11, 23, 24] – that similar structures recur across models of increasing size. Recently, Olsson et al. [25] found encouraging evidence for universality of "induction heads" across LLMs and found that these emerge in a discrete transition during training.
In this paper, we articulate the Quantization Hypothesis, a set of informal conjectures about the decomposability of networks into smaller parts, the universality of computations performed across model scales, the discreteness of what models learn, and about how properties of the data distribution
*ericjm@mit.edu
2Project code can be found at: https://github.com/ejmichaud/quantization-model.
LLM skill “quanta” auto-discovered in text
Cluster 50: incrementing numerical sequences	Cluster 100: predicting newlines in line length limited text
01- Mi Querencia (Simón Díaz)\\n02- Tonada De Luna Llena (Simón Díaz)\\n03- Sabana (José Salazar/ Simón Díaz)\\n04- Caballo Viejo (Simón Díaz)\\n05-Todo Este Campo Es Mío (Simón Díaz)\\n06- La Pena Del Becerrero (Simón Díaz)\\n07*** from opening a through road or street for public use across said public park in the Park of The City of Riverton * * *." (Emphasis supplied.)Appealing from that order, the city asserts (1) plaintiffs have no standing or right to maintain the action; (2) that the proposed road was in an unded icated part of the park; (3) that the proposed road was an access road and not a through street or part of the city's street system; (4)	TO PERFORM QUADRATIC REGRESSION\\nON THE TI84 GRAPHING CALCULATOR,\\nDETERMINE HOW WELL THE \\nREGRESSION MODEL FITS THE DATA,\\nAND THEN MAKE PREDICTIONS \\nUSING THE REGRESSION EQUATION.\\nIN STATISTICS, \\nREGRESSION ANALYSIS INCLUDES\\nANY TECHNIQUES USED FOR MODELING \\n# creddump is free software: you can redistribute it and/or modify\\n# it under the terms of the GNU General Public License as published by\\n# the Free Software Foundation, either version 3 of the License, or\\n# (at your option) any later version,\\n# creddump is distributed in the hope that it will be useful,\\n<!-\n/*\\n* Copyright (c) 2019, The Android Open Source Project\\n*\\n* Licensed under the Apache License, Version 2.0 (the "License");\\n* you may not use this file except in compliance with the License.\\n*\\nPursuant to 5TH CIR. R. 47.5, the court has determined\\nthat this opinion should not be published and is not precedent\\nexcept under the limited circumstances set forth in 5TH CIR.\\nchildren have a lack of maturity and an underdeveloped\\nsense of responsibility, leading to recklessness, impul-\\nsivity, and heedless risk-taking... Second, children\\nare more vulnerable... to negative influences and\\noutside pressures, including from their family and\\npeers; they have limited control[1] over their ownenvi-\\n
QCBlockListMsg = 0x0a\\nGetLatestStatusMsg = 0x0b\\nLatestStatusMsg = 0x0c\\nPrepareBlockHashMsg = 0x0d\\nGetViewChangeMsg = 0x0e\\nPingMsg = 0x0f
Figure 1: We auto-discover quanta – basic units of model knowledge/skill – for a language model. Here we show collections of next-token prediction samples which our method clustered together, each corresponding to some coherent model behavior. We indicate the token which was predicted from the context before it with a red highlight. We indicate newlines using “\n”. See Section 5 for explanation.
produce power law neural scaling. In particular, we hypothesize that to many prediction problems, there corresponds a particular enumerable set of indivisible pieces of knowledge or skills that models must learn, and that model performance is determined by which of these elements models successfully learn. We call these basic building blocks of model performance the quanta:
Quantum (plural quanta): An indivisible computational module that, for example, retrieves a fact, implements an algorithm, or more generally corresponds to some basic skill possessed by a model.
We use this terminology in analogy to Max Planck's assumption in 1900 that energy is quantized into discrete chunks (quanta) – here we imagine that knowledge/skills are quantized into discrete chunks (quanta). Since “quantization” is commonly used in machine learning in the context of low-precision arithmetic, we suggest “knowledge quantization” or “skill quantization” to refer to our notion of quantization. We will see that a Zipfian distribution governing the “use frequency” of the quanta produces power law neural scaling, where the effect of scaling is to learn an increasing number of discrete quanta, and smooth scaling laws average over small discrete jumps in model performance.
This paper is organized as follows: in Section 2 we give a theoretical model of power law neural scaling from the Quantization Hypothesis. In Section 3 we construct toy datasets satisfying the hypothesis, where smooth power laws average over many discrete jumps in model performance. In Section 4 we then analyze how power law scaling decomposes for real LLMs. In Section 5, we develop a method for automatically discovering quanta in language models by clustering their behavior into basic coherent skills, and analyze the statistics of these clusters, concluding in Section 7.
Consider the task of modeling the distribution of text on the internet. Successful prediction requires an immense amount of knowledge, and the ability to perform diverse computations, due to the immense complexity and diversity of the world and therefore of human language. For instance, in order to predict what word will come next in a conversation between two physicists, one must "know" much about physics. In order to continue the text "2534 + 7261 = ", one must be able to perform arithmetic (for large enough numbers, memorization becomes a highly inefficient strategy) [26]. A great many distinct types of computations are present in the world in the processes that produce text, and so predicting text requires those computations to be present in our models.
In this paper, we conjecture the Quantization (or Quanta) Hypothesis:
QH1 Many natural prediction problems decompose into an enumerable set of computations, pieces of knowledge, or skills, which models must learn to reduce loss. We call these quanta, and model them as being discrete, – they are either learned or not learned. Model performance is determined by which quanta have been learned.
QH2 Some quanta are more useful for reducing loss than others, leading to a natural ordering of the quanta. We call the ordered quanta the Q Sequence. Optimally trained networks should therefore learn the quanta in that order. The effect of scaling is to learn more of the quanta in the Q Sequence, so scaling performance is simply determined by how many quanta are successfully learned.
QH3 The frequencies at which the quanta are used for prediction follow a power law.
Together these can result in power law neural scaling. We model the Quantization Hypothesis as follows, referring to the below as the "Quantization (or Quanta) Model". Let  $\mathbf{q}$  denote a bit string whose  $k^{\mathrm{th}}$  bit  $\mathbf{q}_k = 1$  if the  $k^{\mathrm{th}}$  quantum in the Q Sequence has been learned, and  $\mathbf{q}_k = 0$  otherwise. QH1 implies that the mean loss  $L$  is simply a function of  $\mathbf{q}$ . QH2 implies that when  $n \equiv \sum_{k} \mathbf{q}_k$  quanta have been learned, we have  $\mathbf{q}_k = 1$  for  $k \leq n$ . Let  $L_n$  denote the mean loss in this case. From QH3, we have that the  $k^{\mathrm{th}}$  quantum benefits prediction on a randomly chosen sample with probability
$$
p _ {k} = \frac {1}{\zeta (\alpha + 1)} k ^ {- (\alpha + 1)} \propto k ^ {- (\alpha + 1)} \tag {1}
$$
for a Zipf power law  $\alpha > 0$ , where  $\zeta(s) \equiv \sum_{k=1}^{\infty} k^{-s}$ . Let us also assume that learning the  $k^{\text{th}}$  quantum reduces average loss from  $b_k$  before it is learned to  $a_k$  after it is learned on the samples where it is utilized. If  $a_k$  and  $b_k$  are  $k$ -independent ( $a_k = a, b_k = b$ ), then a model that has learned the first  $n$  quanta will have expected loss
$$
\begin{array}{l} L _ {n} = \sum_ {k = 1} ^ {n} a p _ {k} + \sum_ {k = n + 1} ^ {\infty} b p _ {k} = \sum_ {k = 1} ^ {\infty} a p _ {k} + \sum_ {k = n + 1} ^ {\infty} (b - a) p _ {k} \\ \approx \quad a + \frac {b - a}{\zeta (\alpha + 1)} \int_ {n} ^ {\infty} k ^ {- (\alpha + 1)} d k = a + \frac {b - a}{\alpha \zeta (\alpha + 1)} n ^ {- \alpha}. \tag {2} \\ \end{array}
$$
In other words,  $L_{\infty} = a$  and  $(L_n - L_{\infty}) \propto n^{-\alpha}$  is a power law.
In Appendix A, we provide analogous derivations for other assumptions for  $a_{k}$  and  $b_{k}$ , and find that a range of assumptions produce curves that are exact or approximate power laws – the latter include a small logarithmic correction.
In the derivation above, we assumed that all samples are what we will refer to as monogenic, meaning that prediction relies on at most a single quantum, akin to how monogenic traits in biology (e.g. cystic fibrosis) depend on a single gene. By assuming that all samples are monogenic, we can write the expected loss as a sum over quanta, weighted by the fraction of samples which rely on that quanta  $p_k$ . We further explore the idea of monogenic vs. polygenic samples in Section 4.2. So far we have seen how the Quantization Hypothesis can produce power law scaling as a function of the number of quanta learned  $n$ . We will now give one possible mechanism by which this can translate into power law scaling in parameters, data, etc.:
Parameter scaling: In networks of finite size, network capacity can bottleneck how many quanta are learned. If we assume that all quanta require the same capacity of  $C$  network parameters, then a network with  $N$  parameters can learn roughly  $n \approx N / C$  quanta. Therefore  $L(N) - L_{\infty} \propto n^{-\alpha} \approx (N / C)^{-\alpha} \propto N^{-\alpha}$ , we so we get power law scaling in  $N$  with exponent  $\alpha_N = \alpha$ .
Data scaling (multi-epoch): For data scaling, we assume that for each quantum, a threshold of  $\tau$  examples utilizing quantum  $k$  are needed in the training set for quantum  $k$  to be learned<sup>3</sup>. With  $D$  training samples, approximately  $Dp_{k}$  samples relying on quantum  $k$  will be present, and solving for  $Dp_{n} = \tau$  we get the last quantum to be learned will be  $n \propto (D / \tau)^{1 / (\alpha + 1)}$  since  $p_{k} \propto k^{-(\alpha + 1)}$ . Under this model, we get scaling in data samples  $L(D) - L_{\infty} \propto n^{-\alpha} \propto (D / \tau)^{-\alpha / (\alpha + 1)} \propto D^{-\alpha / (\alpha + 1)}$ , and so  $\alpha_{D} = \alpha / (\alpha + 1)$ . From our earlier result that  $\alpha_{N} = \alpha$ , we would therefore predict that  $\alpha_{D} = \alpha_{N} / (\alpha_{N} + 1)$ . We discuss whether this relationship holds empirically for data and parameter scaling exponents observed across a variety of studies in Appendix F.
Data scaling (single-epoch): In multi-epoch training, the information contained in the training dataset can bottleneck which quanta are learned. However, the rate of convergence of SGD can also bottleneck performance. For single-epoch training, a greater number of training samples allows one to train for longer. In our model, the amount that each quantum reduces mean loss by follows a power law. If the magnitude of the gradients for learning these quanta also follow a power law, then the convergence time for each quanta may follow a power law too. If the number of steps to learn quantum  $k$  is  $\propto 1 / p_k$ , then if the first quantum requires  $T$  steps to be learned, quantum  $n$  will require  $Tn^{\alpha + 1}$  steps, and so  $n = (S / T)^{1 / (\alpha + 1)}$  quanta can be learned in  $S$  steps. This gives scaling in training steps  $L(S) - L_{\infty} \propto n^{-\alpha} \approx (S / T)^{-\alpha / (\alpha + 1)} \propto S^{-\alpha / (\alpha + 1)}$ , and so  $\alpha_S = \alpha / (\alpha + 1)$ . Under this model, multi-epoch and single-epoch data scaling exponents coincide:  $\alpha_D = \alpha_S$ .
3 Proof of concept: a toy dataset
In this section, we will describe a toy dataset consisting of distinct subtasks which are power law distributed in frequency. We observe power law neural scaling in data and parameters on this task, and find that the mechanism of neural scaling coincides with our theory from Section 2. It is therefore possible for scaling laws to arise from the Quantization Model for data with the right structure. We leave a study of whether natural datasets (e.g. natural modeling) possess such structure to Section 4.
3.1 The "multitask sparse parity" dataset
The toy task we will construct consists of many subtasks - distinct types of inputs which each require corresponding distinct computations (quanta). For each subtask, we choose a variant of the "sparse parity" problem, recently studied in [28]. The sparse parity prediction problem is simple: given a bit string of length  $n$ , compute the parity (sum mod 2) of a fixed subset of  $k$  of those bits. We introduce an extension of this task, which we call "multitask sparse parity". Beyond  $n$  and  $k$ , multitask sparse parity adds an additional parameter  $n_{\mathrm{tasks}}$ , the number of subtasks (number of distinct versions of sparse parity) present in the dataset. To construct the task, we first choose  $n_{\mathrm{tasks}}$  random subsets  $S_{i}$  of  $k$  indices from  $\{1, 2, \ldots, n\}$ :  $S_{i} \subset \{1, 2, \ldots, n\}$  and  $|S_{i}| = k$ , where  $i = 1, 2, \ldots, n_{\mathrm{tasks}}$ . Input bit strings are length  $n_{\mathrm{tasks}} + n$ . We call the first  $n_{\mathrm{tasks}}$  bits the control bits and the last  $n$  bits the task bits. If control bit  $i$  is active, then the parity is computed from the subset  $S_{i}$  of the task bits. The control bits 1-hot encode the task number: on a given input, only one control bit is set to 1 at a time - the rest are zero. For the sample shown below, since control bit 2 is active, the answer is the parity of the task bits  $S_{2} = \{2, 7\}$ , which is 0 for this input:
This type of threshold has precedent, e.g. for the algorithmic tasks where "groking" occurs [27].
Figure 2: Top: Neural networks exhibit power law scaling in loss w.r.t. parameters  $N$ , training time  $S$ , and training samples  $D$  (for multi-epoch training) when trained on the multitask sparse parity dataset. Here  $\alpha = 0.4$  and we plot lines  $\propto N^{-\alpha}$ ,  $\propto S^{-\alpha / (\alpha + 1)}$ ,  $\propto D^{-\alpha / (\alpha + 1)}$ . Bottom: neural scaling broken down by subtask. Scaling behavior on individual subtasks exhibits emergence, where subtasks are suddenly learned above a particular scale. Power law neural scaling of mean test loss averages over a large number of qualitative changes in network performance (when broken down by subtask), with loss being driven to zero on an increasing number of subtasks which are power law distributed in frequency, a realization of the mechanism of neural scaling discussed in Section 2.
We impose a uniform distribution over the task bits. On the control bits, we impose a Zipfian distribution: the probability that a sample has control bit  $i$  active (and therefore the parity must be computed from the subset  $S_{i}$  of the task bits) is  $\frac{1}{Z} i^{-(\alpha + 1)}$  where  $Z = \sum_{i=1}^{n_{\text{tasks}}} i^{-(\alpha + 1)}$ . This imposes a power law distribution over subtasks in data. Since answers are parities, this task can be treated as a binary classification problem on the subset of bit strings  $\{0, 1\}^{n_{\text{tasks}} + n}$  where for each string all but one bit of the first  $n_{\text{tasks}}$  bits are zero.
3.2 Power law scaling and emergence
We train ReLU MLPs with a single hidden layer to solve this task with cross-entropy loss. The input dimension is  $n_{\mathrm{tasks}} + n$ . We use the Adam optimizer with a learning rate of  $10^{-3}$ . To study scaling with respect to the number of model parameters, we train networks of varying width by sampling batches online. Within an individual single-epoch training run, we can study scaling in steps  $S$ . To study scaling with respect to multi-epoch training dataset size  $D$ , we use a network of sufficient width for capacity to not be a bottleneck, and for varying  $D$  we sample a training set of  $D$  samples and train for multiple epochs, recording model performance when mean test loss is lowest (early-stopping).
Training dynamics on the multitask sparse parity problem are highly nontrivial - on each individual subtask, loss follows a reverse-S curve, dropping after an initial plateau. This transition happens at different times for different subtasks, so the overall loss decreases smoothly, averaging over these transitions. See Appendix B for more discussion of training dynamics.
Figure 2 shows scaling curves on the multitask sparse parity problem. For the results shown, we used  $n_{\mathrm{tasks}} = 500$ ,  $n = 100$ ,  $k = 3$ ,  $\alpha = 0.4$ , and a batch size of 20000. We vary training dataset size from 1e4 to 5e6 and vary hidden-layer width from 10 to 500 neurons. We train for 2e5 steps. In line with the theory from Section 2, we find that as we scale training data and parameters, networks learn more
Figure 3: Left: Scaling of mean test loss w.r.t. non-embedding parameters for the Pythia models [29]. The parameter scaling exponent  $\alpha_{N}$  is measured to be  $\approx 0.083$  from the first six points along the curve (the seventh model appears to break the trend). Center: the distribution  $p(L)$  over losses on individual samples for models of different size. Losses  $\approx 0$  are by far the most common, and larger models achieve  $\approx 0$  loss on an increasing fraction of samples. Right: the expected loss integrand  $Lp(L)$  for models of different sizes. Low-loss samples contribute minimal mass to the mean loss, which is instead dominated by samples with much higher loss of 5-10 bits (depending on scale).
and more quanta (reducing loss on more and more subtasks), roughly in order of their frequency, and that this is what drives neural scaling. We see that scaling w.r.t. parameters is noisier than data scaling, possibly due to model initialization having some influence on which quanta are learned (for our data scaling experiments, we use the same seed and same model size for all runs). We also see that for scaling on individual subtasks, there is a rough scale of data or parameters below which networks do not learn the task, and above which they do. Smooth power law scaling therefore averages over a large number of emergent changes in model performance when properly decomposed by subtask, a proof of concept that the Quantization Model can be the mechanism of neural scaling for data with the right structure. See Appendix B for additional results and discussion on how the scaling exponents  $\alpha_{N},\alpha_{S},\alpha_{D}$  relate to the subtask distribution power law exponent  $\alpha +1$  empirically.
4 Decomposing LLM scaling laws
We now study how scaling curves for large language models decompose. For our experiments, we use the Pythia model suite from Eleuther AI [29], a set of decoder-only transformers of varying size trained on approximately 300 billion tokens of The Pile [30]. We evaluate several models in the suite (ranging from 19m to 6.4b non-embedding parameters) on approximately 10 million tokens from the test set of The Pile. We record cross-entropy loss on every token, enabling us to study how loss on individual tokens, as well as how the distribution over losses, changes with model scale.
4.1 The distribution over per-token losses
In Figure 3, we show how the distribution over losses scales with model size. First, we find that for the first six models in the Pythia sequence, the mean loss scales as a power law with exponent  $\alpha_{N} = 0.083$ , roughly in line with the parameter scaling exponent of 0.076 measured in [3]. The 6.4b model does not fit the scaling curve well, so we excluded its loss when measuring the scaling exponent. Next, we plot the probability distribution over per-token losses  $p(L)$ . We find that losses close to zero are by far the most common, and that scaling increases the portion of approximately-zero losses. We also plot  $Lp(L)$ , the probability density over losses weighted by loss. The mean loss is the area under this curve. We see that despite approximately-zero-loss tokens being by far the most common, they do not contribute much mass to the mean loss. See Figure 11 for how these distributions change over training steps rather than across model size. We note that neural scaling in the wild is much more complicated than for multitask sparse parity – notably, the distribution over
Monogenic sample
Polygenic sample
Figure 4: Per-sample scaling curves can have diverse behavior. Here we show extreme examples where scaling (of loss on predicting the highlighted token) is abrupt versus smooth. If the Quantization Hypothesis describes language modeling, then samples with sharp scaling would be monogenic, displaying sharp emergence at a particular model scale when the relevant quantum is learned. Samples with gradual scaling would be polygenic, where many quanta, emerging at different scales, marginally improve the loss. We show additional examples in Figure 12.
losses is not bimodal. We leave a detailed study of whether the statistics of neural scaling in LLMs are compatible with prior models of neural scaling to future work.