SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When applying QDG to language models, we use gradients within self-attention and MLP layers, but do not include embed, unembed, or layer norm gradients when we flatten and concatenate gradients into a vector  $g$ . We choose samples  $(x_i, y_i)$  for which our 19m-parameter model achieves a cross-entropy loss less than 0.1 nats. We filter based on this criteria since (1) we cannot cluster samples based on model mechanism if the model does not have such a mechanism for performing prediction correctly on those samples and (2) our intuition that samples with particularly low loss are more likely to be monogenic. We further exclude samples which can be solved via induction on the context, since such samples are quite common (possibly interfering with our task of finding diverse quanta) and since early experiments indicated that QDG had trouble clustering such samples together. We choose 10000 such samples to perform clustering on from the test set of The Pile. After computing the affinity matrix  $\hat{C}$ , we use the spectral clustering implementation from scikit-learn [45] with labels assigned via k-means.
D Quanta discovery on TinyStories
We also apply QDG to TinyStories-33M, a language model trained on the TinyStories dataset [46]. We consider only tokens on which TinyStories-33M achieves a loss less than 1 bit of cross-entropy. We apply QDG to 10000 such samples, clustering their gradients with spectral clustering with n_clusters  $= 400$ . We show some samples from the resulting clusters in Figure 14. Many of these clusters reflect some simple recurring pattern in the TinyStories dataset, like predicting "time" after "Once upon a", which many documents in the dataset start with. Some other clusters are more interesting however, like Cluster 11, which seems to involve predicting the correct noun in a sentence where that noun was referred to earlier in the sentence or in previous sentences.
<sup>4</sup>We exclude gradients for embed and unembed parameters because they are high dimensional and also because they may contain information more about the input and output rather than the computations the model performs internally. We exclude layer norm gradients because they appeared to contain less information about clusters in toy experiments.
<sup>5</sup>We filter (copying) induction problems by excluding samples where the token which is to be predicted is the last token in a trigram which occurred earlier in the context. This is not a very comprehensive filtering scheme.
Monogenic samples
"The law of unintended consequences and the history of previous military interventions in the region is not a recipe for political and economic stability," said Neil MacKinnon, global macro strategist at VTB Capital.
Polygenic samples
In general, the lesions of thoraco-cervical level were difficult to detect, because the appearance rate of SSEP peaks are reduced over the thoraco-cervical spine even in normal controls. In cases
Figure 12: Additional LLM scaling curves on individual samples which exhibit sharp vs smooth improvement. If the Quantization Hypothesis is true for language modeling, then we would interpret samples with sharp drops as "monogenic" and samples with gradual progress as "polygenic".
Opinion filed March 25, 1988. W.Y. Chalfant, of Branine, Chalfant & Hill, of Hutchinson, argued the cause and was on the brief for appellant, Hesston State Bank. Kenneth C. Jones, of Watson, Ess, Marshall
airline by revenue, dropped \(2.15, or 7.2 percent, to \)27.71 and Delta Air Lines lost $1. 16, or 5.7 percent, to \)19.11.\n\n\n
Stone said oil prices could start weighing on consumer spending down the road...
E The difficulty of estimating the power law exponent from clusters
In Section 5.1, when we looked at the distribution over elements in each cluster, we did not perfectly recover a Zipf distribution with exponent  $\approx 1.08$  that we expected from our theory. In this section, we describe the difficulty of accurately estimating such an exponent with our method.
E.1 QDG on multitask sparse parity
As a first experiment, we performed QDG on multitask sparse parity, where there is a known, artificially-imposed power law distribution over subtasks. We train a width-500 single-hidden-layer ReLU MLP on multitask sparse parity with  $\alpha = 0.4$  and with  $n = 100$ ,  $k = 3$ , and  $n_{\mathrm{tasks}} = 500$ . We then took 10000 samples which the network achieves  $\approx 0$  loss on (sampled from the Zipf distribution over subtasks with exponent 1.4). We compute gradients of cross-entropy loss w.r.t. all model parameters for these samples, and then perform QDG just like for LLMs. We show results in Figure 15. We plot the full similarity matrix where samples are ordered according to their a priori known subtask, rather than their cluster from QDG, and see a clear pattern where elements from the same subtask have on average higher angular similarity than elements between subtasks. However, from the rank-frequency plot of the clusters, we do not recover a slope of -1.4, but rather a lower slope of  $\approx -1.1$ . This shows that even when there is an exact decomposition of inputs into subtasks with a
Cluster 146: comma after day of month	Cluster 269: “s” after starting year of decade
Sam Willard\n\nSamuel Steven Willard (born September 9, \n\n215 U.S. 437 (1910)\nMECHANICAL APPLIANCE COMPANY\n\nv.\nCASTLEMAN\nNo. 48.\nSupreme Court of United States.\nArgued December 3, 1909.\nDecided January 3,	Romford Ice Arena\n\nRomford Ice Arena was an ice rink located in Romford in the London Borough of Havering, England. The venue was built in the 1980s\n\nAlthough the novel continues to be the dominant medium of the crime-my stery-detective narrative, short stories by these contemporary authors may be found in numerous anthologies of the genre published during the 1990s\n\nArmed with the new Shenwei SW26010 chips, a new supercomputer has surged to the top of the TOP500 list: Sunway TaihuLight. It is nearly three times as fast as Tianhe-2, being benchmarked in Linpack as being able to perform 93 quadrillion calculations each second (93 petaFLOPS). To put this achievement in context, modern desktop PCs are already more powerful than the top ranked supercomputers from the early 1990s\n\nEarly activism\nHe began a lifetime involvement with revolutionary politics in the late 1930s\nYesterday, visitor Greg Davidson commented that he was searching for songs played on the local forecast back in the late ‘80s\nIn 1954, the couple published Living the Good Life which inspired many young, educated Americans to create simpler, rural lifestyles and the back-to-the-land movement of the 1960s
Cluster 278: colon after CSS property	Cluster 292: protocol separator in URLs
\nickshaw_graph.detail \npointer-events: none;\nposition: absolute;\ntop: 0;\nz-index: 2;\nbackground: rgba(0, 0, 0, 0.1);\nbottom: 0;\nwidth:\n@ import '..././assets/sass/spin';\n\napp-header \nbackground-color: #282c34;\nmin-height: 100vh;\ndisplay:\See: http://jsfiddle.net/mWFGZ/1/\nhtml, body \nmargin: 0;\npadding:\n	# \#\n#
# \#\n#
# For questions please refer to:
# \#n
# https://When I run the below code, I am getting an error: MongoError: The dollar ($) prefixed field '\\push' in '\\push' is not valid for storage.\nI put this together based on the docs: https://Gruber, Martin A. Views of the National Zoological Park in Washington, DC, showing Exhibit. 1919. Retrieved from the Digital Public Library of America, http://But as citizens, our responsibility is to look beyond the anecdote. We journalists try to make sure that if a tree falls in the forest, it won't go un noticed. Still, if we get so taken by the trees that we don't see the forest, we'll all be lost.\n\nRx Smith is editor of the Times Union. Share your thoughts at http://
Figure 13: Additional examples of clusters of inputs discovered by QDG. Like in Figure 1, we used 10000 samples and n_clusters of 400.
known Zipf distribution over these subtasks, that we do not perfectly recover this Zipf distribution from QDG.
E.2 A toy model of QDG uncertainty and bias
A toy model: To understand the bias of spectral clustering, we develop the following toy model. We assume the dataset has  $N = 1000$  subtasks, each subtask containing  $n_i = \lfloor \frac{A}{i^\alpha} \rfloor (1 \leq i \leq N)$  tokens ( $A = 1000$ ). We use a Gaussian distribution  $\mathcal{N}(\mathbf{m}_i, \sigma^2 \mathbf{I}_{d \times d})$  to model gradients within a subtask  $i$ , where  $d$  is the embedding dimension,  $\sigma$  is the noise level, and  $\mathbf{m}_i$  is the Gaussian mean.  $\mathbf{m}_i$  itself is drawn from the standard Gaussian distribution  $\mathbf{m}_i \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_{d \times d})$ . We define the similarity between two vectors  $\mathbf{x}, \mathbf{y}$  to be  $\mathrm{sim} \equiv 1 + \frac{\mathbf{x}}{|\mathbf{x}|} \cdot \frac{\mathbf{y}}{|\mathbf{y}|}$ . We compute pairwise similarity between all  $\sum_{i=1}^{N} n_i$  tokens, and input the similarity matrix to the spectral clustering algorithm. We also need to specify the number of clusters  $k$ .
We have two hyperparameters in the toy model, the embedding dimension  $d$  and the noise level  $\sigma$ . We need to determine them such that this toy model can decently imitate LLM results (Figure 5). We fix  $\alpha = 1$ , sweeping  $d = \{30, 100, 1000\}$ ,  $\sigma = \{0, 0.5, 2.0\}$ , and  $k = \{100, 200, 500\}$ . As shown in Figure 16, the high-dimension ( $d = 1000$ ) large-noise ( $\sigma = 2.0$ ) scheme seem to best agree with the LLM results, since the  $k = 200$  curve can reproduce the sag and the cliff present in LLM curves.
Estimating  $\alpha$  from the frequency curve is hard, in fact, the slope depends on  $k$  and the region used to estimate it. However, we observe that different  $k$  curves form a clear envelope, whose slope is
Figure 14: Examples of clusters within the TinyStories dataset, discovered by QDG on the TinyStories-33M model. Here we just show samples from four out of  $400\mathrm{n\_clusters}$
robust in a reasonably wide region. The envelope slope seems to indicate  $\alpha$ . We fix  $d = 1000$  and  $\sigma = 2.0$ , sweeping  $\alpha = \{0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5\}$ . For each  $\alpha$ , we estimate the slope of the envelope. Although there is clear correlation between the estimated envelope and  $\alpha$ , if we use the envelope slope to estimate  $\alpha$ , the error is on the order of 0.2, as shown in Figure 17.
Figure 15: Similarity matrix and rank-frequency plots from QDG on multitask sparse parity. Despite sparse parity having a known decomposition into subtasks which are power law distributed in frequency, we do not recover this same power law from samples. We used  $\alpha = 0.4$  for the frequency distribution for an expected rank-frequency power law exponent of -1.4, but measure a rank-frequency envelope slope closer to -1.1.
F Parameter and data scaling exponents across studies
In Figure 18, we show  $\alpha_{N}$  and  $\alpha_{D}$  (or possibly  $\alpha_{S}$ , depending on the study) for a variety of prior studies of deep learning scaling, as compiled by Villalobos [47]. While the data is messy, it is intriguing that most of the Rosenfeld et al. [2] samples lie below the  $\alpha_{D} = \alpha_{N}$  line, as our model would predict. The scaling exponents from Hoffmann et al. [7] are slightly closer to our prediction than the relation  $\alpha_{D} = \alpha_{N}$ , which has been proposed by other models of neural scaling laws. Overall though, the existing empirical results are too messy to definitively support or contradict our model.
G Estimates of compute used for our experiments
Multitask sparse parity: Our training script takes roughly 1-4 hours (depending on network size) to perform a single-epoch training run on a GPU. When training multi-epoch on a fixed dataset, runs take typically between 3-10 minutes, with some outliers taking much longer. Our largest experiment was for Figure 10, where we trained networks of varying width on data with varying distributions over subtasks (with different power law exponents). 467 runs completed with a total running time of approximately 1450 hours. These experiments were run on a cluster with heterogeneous hardware. Available GPUs include NVIDIA A100, RTXA6000, QUADRORTX6000, GEFORCERTX2080TI, GEFORCERTX2080, GEFORCEGTX1080TI, titan-x, and Tesla-v100.
Pythia model scaling evaluations: We evaluated Pythia models on NVIDIA A100 80GBs. We do not have available the running time used when computing loss on approximately ten million tokens (for which we reported scaling statistics on), although it was likely less than an hour per model. The most expensive experiments were for Figure 11, where we evaluated the first four models in the Pythia suite across 143 checkpoints for a total of 572 evaluations. We likely used some hundreds of A100-hours for this, though possibly less than 100 hours.
QDG: We ran our QDG experiments on an NVIDIA A100 80GB. For the smallest Pythia model and for 10000 samples, it takes a few hours to compute the similarity matrix. We performed this computation only a handful of times.
Figure 16: To understand the bias of spectral clustering, we apply spectral clustering to a toy model with different embedding dimension  $d$ , noise scale  $\sigma$  and number of cluster  $k$ . The high-dimension  $(d = 1000)$  large-noise  $(\sigma = 2.0)$  scheme seems to best agree with the LLM results (Figure 5).
Figure 17: The difficulty of measuring  $\alpha$  from curves. We apply spectral clustering to a toy model with different  $\alpha$  and number of clusters  $k$ . For a fixed  $\alpha$ , different  $k$  curves define an envelope. One could use the envelope slope to infer  $\alpha$ , but this incurs errors around 0.2.
Figure 18: Parameter and data scaling exponents from various studies of deep learning scaling, compiled from the database of neural scaling laws from [47]. Our model of scaling predicts that  $\alpha_{D} = \alpha_{N} / (\alpha_{N} + 1)$ , indicated with the solid black line. Visible points are from [2, 3, 7, 5, 48]. [49] is above the visible window of the figure.