SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Allan Raventós* Mansheej Paul* Feng Chen Surya Ganguli
{aravento, mansheej, fengc, sganguli}@stanford.edu
Pretrained transformers exhibit the remarkable ability of in-context learning (ICL): they can learn tasks from just a few examples provided in the prompt without updating any weights. This raises a foundational question: can ICL solve fundamentally new tasks that are very different from those seen during pretraining? To probe this question, we examine ICL's performance on linear regression while varying the diversity of tasks in the pretraining dataset. We empirically demonstrate a task diversity threshold for the emergence of ICL. Below this threshold, the pretrained transformer cannot solve unseen regression tasks, instead behaving like a Bayesian estimator with the non-diverse pretraining task distribution as the prior. Beyond this threshold, the transformer significantly outperforms this estimator; its behavior aligns with that of ridge regression, corresponding to a Gaussian prior over all tasks, including those not seen during pretraining. Thus, when pretrained on data with task diversity greater than the threshold, transformers can optimally solve fundamentally new tasks in-context. Importantly, this capability hinges on it deviating from the Bayes optimal estimator with the pretraining distribution as the prior. This study also explores the effect of regularization, model capacity and task structure and underscores, in a concrete example, the critical role of task diversity, alongside data and model scale, in the emergence of ICL.
Pretrained transformers (PTs) can learn new tasks from just a few examples provided in the prompt without taking any gradient steps on those examples [1]. This ability, called in-context learning (ICL), has unlocked the widespread use of language models by making it efficient to adapt general purpose models to bespoke tasks without explicit training. Though remarkable, what makes ICL mysterious, and potentially harmful [2], is that the learning algorithm implemented by the PT in its forward pass is not built into its architecture or training process; instead it emerges from pretraining on large-scale data with a next token prediction objective. This raises a foundational question: can ICL really solve fundamentally new tasks that are very different from those seen during pretraining? If so, what learning algorithm does ICL implement? To answer these questions, we need to better understand how the different ingredients that go into pretraining influence this ability.
Towards this end, we explore how the diversity of tasks in the pretraining data affects the emergence of ICL. Prior work [3] has proposed that ICL works by performing Bayesian inference. During pretraining, transformers learn a prior over latent tasks represented in the pretraining data. When prompted with examples at inference time, they "retrieve" relevant pretraining tasks and generate subsequent tokens from the posterior distribution conditioned on the query and inferred tasks. This suggests that ICL performance on a new task is influenced by its similarity to tasks implicitly learned during pretraining. However, the distribution of tasks in our pretraining data,  $\mathcal{T}_{\mathrm{Pretrain}}$ , is usually a
*Equal Contribution. Code released at https://github.com/mansheej/icl-task-diversity
limited and unrepresentative subsample of the ideal distribution of tasks,  $\mathcal{T}_{\mathrm{True}}$ , that we want our model to be capable of learning in-context. For instance,  $\mathcal{T}_{\mathrm{True}}$  could be the set of all instructions we want an A.I. assistant to follow. But, large-scale language modeling datasets [4, 5] used to pretrain these models contain very few examples correctly formatted for ICL. Instruction finetuning (IFT) datasets [6-11] designed to ameliorate this are expensive to collect and thus contain tasks from just a few domains. Under the Bayesian framework, this distribution mismatch would cause the Bayesian estimator with a prior over the limited pretraining tasks,  $\mathcal{T}_{\mathrm{Pretrain}}$ , to perform suboptimally on tasks that are very different from those seen during pretraining. This motivates our question: can a model pretrained on a dataset with low task diversity nevertheless learn new, unseen tasks?
For general purpose language modeling, the size and complexity of  $\mathcal{T}_{\mathrm{Pretrain}}$  and the vague specification of  $\mathcal{T}_{\mathrm{True}}$  make this question challenging to analyze. So, following recent work [12-14], we study ICL for linear regression. Here, a task is a linear regression problem with a given latent regression vector; the PT must predict the target for a new data point from examples of data-target pairs provided in the prompt. Prior work [13] has shown that transformers that see an unlimited number of latent regression vectors during pretraining learn to perform ridge regression with the Bayes optimal ridge parameter. We instead consider the case where the pretraining task distribution,  $\mathcal{T}_{\mathrm{Pretrain}}$ , contains a limited and finite set of latent regression vectors (see Section 2 for details). To evaluate its ability to learn new tasks, the PT is tested on the ideal task distribution,  $\mathcal{T}_{\mathrm{True}}$ , which is a Gaussian distribution over all latent regression vectors. Studying this setting has two advantages: first, we can directly vary the task diversity in  $\mathcal{T}_{\mathrm{Pretrain}}$  by changing the number of unique latent regression vectors seen during pretraining. Second, we can calculate the optimal estimator that minimizes the pretraining loss—the Bayesian estimator with prior  $\mathcal{T}_{\mathrm{Pretrain}}$  as well as the optimal estimator for all tasks—the Bayesian estimator with prior  $\mathcal{T}_{\mathrm{True}}$ . This allows us to interpret the behavior of the PT by comparing its predictions to those of the optimal estimators under either task distribution. In our work, we vary the pretraining task diversity and probe the PT's ability to learn fundamentally new tasks in-context: does it behave like the optimal estimator for  $\mathcal{T}_{\mathrm{Pretrain}}$  and perform suboptimally on tasks from  $\mathcal{T}_{\mathrm{True}}$ , or does it align with the optimal estimator for  $\mathcal{T}_{\mathrm{True}}$  which can solve new, unseen tasks?
Contributions. Our contributions are as follows:
- We find that a transformer pretrained on data with low task diversity behaves like the Bayesian estimator with prior  $\mathcal{T}_{\mathrm{Pretrain}}$ ; it performs optimally on pretraining tasks but cannot learn new tasks in-context. However, as pretraining task diversity increases, the PT deviates from this Bayesian estimator, significantly outperforming it on new tasks, and at a large but still finite number of pretraining tasks, the PT's performance closely matches that of the optimal estimator on  $\mathcal{T}_{\mathrm{True}}$ .
- We identify a task diversity threshold for the emergence of ICL. Below this threshold, increasing the pretraining dataset size while keeping task diversity constant biases the PT towards the pretraining task distribution. Conversely, beyond this threshold, increasing the dataset size without increasing its task diversity improves the PT's performance on new, unseen tasks. This suggests that the PT's behavior undergoes a sharp algorithmic phase transition in the limit of many examples per task, aligning with the optimal estimators on  $\mathcal{T}_{\mathrm{Pretrain}}$  before the threshold and on  $\mathcal{T}_{\mathrm{True}}$  after it. We also examine this transition from the perspective of learning dynamics.
- We empirically show that increasing the task dimension at fixed SNR increases the task diversity threshold. However, the scaling of the PT's error with dimension is vastly superior to that of the optimal Bayesian estimator for  $\mathcal{T}_{\mathrm{Pretrain}}$ ; at a task diversity that is beyond the threshold at all dimensions we consider, the PT remains near-optimal with increasing dimension, whereas the optimal estimator for  $\mathcal{T}_{\mathrm{Pretrain}}$  grows progressively less similar to the optimal estimator for  $\mathcal{T}_{\mathrm{True}}$ .
- We show that increasing weight decay significantly decreases the task diversity threshold while increasing number of layers or embedding size increases the task diversity threshold. This elucidates the effect of regularization and model capacity on the emergence of ICL.
Overall these contributions suggest that the emergence of in-context learning in pretrained transformers cannot be fully explained by a theory of Bayesian inference on the pretraining distribution.
ICL of linear regression (schematic in Fig. 1). Each ICL task corresponds to a latent  $D$ -dimensional regression vector,  $\mathbf{w} \in \mathbb{R}^{D}$ . At inference time, the transformer takes as input a sequence of  $K$  data-target pairs,  $(\mathbf{x}_1, y_1, \dots, \mathbf{x}_K, y_K)$ , which are the in-context examples corresponding to this single task
Figure 1: Schematic for ICL of linear regression. (Left) A task corresponds to a latent regression vector,  $\mathbf{w}$  (purple line).  $(\mathbf{x}_1,y_1,\dots,\mathbf{x}_K,y_K)$  (black circles) is a sequence of in-context examples for this task. (Right) The PT,  $f_{\theta}$ , takes this as input and generates  $K$  outputs. The  $k$ th output,  $f_{\theta}(S_k)$ , is the prediction for the target of  $\mathbf{x}_k$  and depends only on the context  $S_k = (\mathbf{x}_1,y_1,\dots,\mathbf{x}_{k-1},y_{k-1},\mathbf{x}_k)$ .
w. For  $k \in \{1, \dots, K\}$ , the data are drawn i.i.d. from a  $D$ -dimensional standard normal distribution,  $\mathbf{x}_k \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_D)$ , and the targets are scalars given by  $y_k = \mathbf{w}^\top \mathbf{x}_k + \varepsilon_k$ . The  $\varepsilon_k$ 's are noise scalars drawn i.i.d. from a normal distribution with mean 0 and variance  $\sigma^2$ ,  $\varepsilon_k \sim \mathcal{N}(0, \sigma^2)$ . Let  $f_\theta$  be the PT with parameters  $\theta$ . Since we use a decoder-only transformer with a causal attention mask, for each  $k \in \{1, \dots, K\}$ , the transformer sees the context  $S_k = (\mathbf{x}_1, y_1, \dots, \mathbf{x}_{k-1}, y_{k-1}, \mathbf{x}_k)$  and based on this context, it makes a prediction,  $f_\theta(S_k)$ , for the target of  $\mathbf{x}_k$ . Thus, in each forward pass, the PT solves  $K$  linear regression problems each with the same latent regression vector but an increasing number of in-context examples.
Pretraining. The transformer is pretrained to minimize the next token prediction mean squared error (MSE) on sequences of data and target pairs. The latent regression vector for each sequence is drawn from the pretraining task distribution,  $\mathcal{T}_{\mathrm{Pretrain}}$ . This distribution has limited diversity as it is the uniform distribution over a finite set of  $M$  tasks,  $\mathcal{T}_{\mathrm{Pretrain}} = \mathcal{U}\{\mathbf{w}^{(1)}, \dots, \mathbf{w}^{(M)}\}$ . Each task in  $\mathcal{T}_{\mathrm{Pretrain}}$  is drawn i.i.d from a  $D$ -dimensional standard normal distribution,  $\mathbf{w}^{(i)} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_D)$ ,  $i \in 1, \dots, M$ . By increasing the number of tasks,  $M$ , in  $\mathcal{T}_{\mathrm{Pretrain}}$ , we can increase the diversity of the pretraining data. Since the transformer makes a prediction for every data point in the sequence, its loss,  $L^{\mathcal{T}_{\mathrm{Pretrain}}}$ , is just the MSE for each prediction, averaged over the predictions in the sequence:
$$
L ^ {\mathcal {T} _ {\text {P r e t r a i n}}} (\theta) = \underset { \begin{array}{l} \mathbf {x} _ {1}, \dots , \mathbf {x} _ {K} \sim \mathcal {N} \left(\mathbf {0}, \mathbf {I} _ {D}\right) \\ \varepsilon_ {1}, \dots , \varepsilon_ {K} \sim \mathcal {N} (0, \sigma^ {2}) \end{array} } {\mathbb {E}} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \left(f _ {\theta} \left(S _ {k}\right) - y _ {k}\right) ^ {2} \right]. \tag {1}
$$
Evaluation. We evaluate the PT's performance on tasks seen during pretraining by computing  $L^{\mathcal{T}_{\mathrm{Pretrain}}}$  using Eq. (1) but with new samples of data and noise. Since these are new instances of the task with new in-context examples, this evaluation corresponds to the test error of the PT. For a PT to successfully perform ICL of linear regression on new tasks, it must accurately predict the targets from the in-context examples for any task drawn from an ideal task distribution,  $\mathcal{T}_{\mathrm{True}}$ , over all latent regression vectors; in our case  $\mathcal{T}_{\mathrm{True}} = \mathcal{N}(\mathbf{0},\mathbf{I}_D)$ . We evaluate the PT's performance on new tasks by computing  $L^{\mathcal{T}_{\mathrm{True}}}$ , which follows Eq. (1) but where the tasks are sampled from the ideal task distribution:  $\mathbf{w} \sim \mathcal{T}_{\mathrm{True}}$  in the expectation.
Comparing the PT to optimal estimators. An advantage of studying ICL of linear regression is that we can calculate the ground truth optimal estimators that minimize the loss,  $L^{\mathcal{T}}$ , in Eq. (1) for both task distributions,  $\mathcal{T}_{\mathrm{Pretrain}}$  and  $\mathcal{T}_{\mathrm{True}}$ . The optimal estimator for the  $k$ th prediction,  $\hat{y}_k^T$ , that minimizes the  $k$ th term in the sum in  $L^{\mathcal{T}}$  is the Bayesian estimator with  $\mathcal{T}$  as the prior. This is given by the posterior mean of  $y_k$  conditioned on the context:  $\hat{y}_k^T = \mathbb{E}_{\mathcal{T}, \varepsilon_k} [y_k \mid S_k]$ , where the expectation is over the task distribution,  $\mathcal{T}$ , and the noise,  $\varepsilon_k$  (Appendix A.1).
For task distribution  $\mathcal{T}_{\mathrm{Pretrain}} = \mathcal{U}\{\mathbf{w}^{(1)},\dots,\mathbf{w}^{(M)}\}$ , the discrete minimum mean squared error (dMMSE) estimator is optimal. It is given by  $\hat{y}_k^{\mathrm{dMMSE}} = \left(\hat{\mathbf{w}}_k^{\mathrm{dMMSE}}\right)^{\top}\mathbf{x}_k$  where  $\hat{\mathbf{w}}_1^{\mathrm{dMMSE}} =$
Figure 2: ICL emerges in PTs beyond a threshold pretraining task diversity. We show all results on both tasks seen during pretraining (top row) and on new tasks (bottom row). The left column compares the normalized loss of transformers pretrained with increasing task diversity to that of dMMSE and Ridge. When the pretraining task diversity is small, the PT's performance matches that of dMMSE; it performs very well on tasks seen during pretraining but poorly on new tasks. As the pretraining task diversity increases, both dMMSE and PT approach Ridge. However, the PT approaches Ridge much faster, significantly outperforming dMMSE on new tasks (bottom left). In the middle and right columns, we compare the PT's predictions to those of dMMSE and Ridge respectively (Eq. (4)). We also increase the number of sequences per task at each level of task diversity by increasing the batch size while keeping total training steps fixed. This reveals a task diversity threshold between  $2^{14}$  and  $2^{15}$  pretraining tasks at which there is a phase transition in the behavior of the model. Below the threshold, increasing the dataset size leads to PTs with predictions more aligned with dMMSE on  $\mathcal{T}_{\mathrm{Pretrain}}$  (top middle). However, beyond this threshold (indicated by gray shading), increasing the dataset size leads to PTs more aligned with Ridge on all tasks (right).
$\frac{1}{M}\sum_{i = 1}^{M}\mathbf{w}^{(i)}$  and for  $k\in \{2,\dots,K\}$  (Appendix A.2)
$$
\hat {\mathbf {w}} _ {k} ^ {\mathrm {d M M S E}} = \sum_ {i = 1} ^ {M} \frac {\exp \left(- \frac {1}{2 \sigma^ {2}} \sum_ {j = 1} ^ {k - 1} \left(y _ {j} - \mathbf {w} ^ {(i) \intercal} \mathbf {x} _ {j}\right) ^ {2}\right)}{\sum_ {l = 1} ^ {M} \exp \left(- \frac {1}{2 \sigma^ {2}} \sum_ {j = 1} ^ {k - 1} \left(y _ {j} - \mathbf {w} ^ {(l) \intercal} \mathbf {x} _ {j}\right) ^ {2}\right)} \mathbf {w} ^ {(i)}. \tag {2}
$$
Intuitively,  $\hat{\mathbf{w}}_k^{\mathrm{dMMSE}}$  is just a weighted sum of the pretraining  $\mathbf{w}^{(i)}$ s with weight governed by the likelihood of observing targets  $\{y_{1},\dots,y_{k - 1}\}$  conditioned on inputs  $\{\mathbf{x}_1,\dots,\mathbf{x}_{k - 1}\}$  and the task being  $\mathbf{w}^{(i)}$ . A PT that minimizes the pretraining loss  $L^{\mathcal{T}_{\mathrm{Pretrain}}}$  will behave like this estimator.
For task distribution  $\mathcal{T}_{\mathrm{True}} = \mathcal{N}(\mathbf{0},\mathbf{I}_D)$ , the Ridge regression estimator with the ridge parameter set to the noise scale  $\sigma^2$  is optimal:  $\hat{y}_k^{\mathrm{Ridge}} = \left(\hat{\mathbf{w}}_k^{\mathrm{Ridge}}\right)^\top \mathbf{x}_k$ , where  $\hat{\mathbf{w}}_1^{\mathrm{Ridge}} = \mathbf{0}$  and for  $k = \{2,\dots,K\}$ ,
$$
\hat {\mathbf {w}} _ {k} ^ {\text {R i d g e}} = \left(\mathbf {X} ^ {\intercal} \mathbf {X} + \sigma^ {2} \mathbf {I} _ {D}\right) ^ {- 1} \mathbf {X} ^ {\intercal} \mathbf {y}, \tag {3}
$$
where  $\mathbf{X} = (\mathbf{x}_1^{\intercal},\dots,\mathbf{x}_{k - 1}^{\intercal})\in \mathbb{R}^{(k - 1)\times D}$  and  $\mathbf{y} = (y_{1},\ldots ,y_{k - 1})$  (Appendix A.3). A PT that performs optimally on new tasks will behave like this estimator. We can compare the behavior of the PT to that of the optimal estimators by computing the mean square difference of the predictions under a given task distribution  $\mathcal{T}$ . We write this as
$$
\Delta_ {\mathrm {P T}, \text {R i d g e / d M M S E}} ^ {\mathcal {T}} = \underset { \begin{array}{c} \mathbf {x} _ {1}, \dots , \mathbf {x} _ {K} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {D}) \\ \varepsilon_ {1}, \dots , \varepsilon_ {K} \sim \mathcal {N} (0, \sigma^ {2}) \end{array} } {\mathbb {E}} \left[ \frac {1}{K D} \sum_ {k = 1} ^ {K} \left(f _ {\theta} \left(S _ {k}\right) - \hat {y} _ {k} ^ {\text {R i d g e / d M M S E}}\right) ^ {2} \right]. \tag {4}
$$
3 Experiments and results
Unless specified otherwise, we study linear regression in  $D = 8$  dimensions with up to  $K = 16$  in-context examples and observation noise variance  $\sigma^2 = 0.25$ . We use either a base transformer
Figure 3: Increased pretraining steps reveals the same task diversity threshold for the emergence of ICL. Columns 1 and 2 in this figure are similar to the middle column in Fig. 2 and columns 3 and 4 correspond to the right column in Fig. 2, except here we increase the number of sequences per task by increasing the number of training steps while keeping batch size  $= 256$ . Both methods of increasing dataset size—increasing batch size in Fig. 2 and increasing training steps in this figure—reveal a transition in the behavior of the PT: beyond the task diversity threshold, ICL on new tasks emerges.
model with the GPT2 architecture [15] with 8 layers, 128-dimensional embeddings, and 2 attention heads or a small model with 4 layers, 64-dimensional embeddings, and 2 attention heads. We train with the Adam optimizer [16] and a one-cycle triangle learning rate schedule [17] with  $50\%$  warmup. The base model is trained with batch size 256 for 500K training steps, though these hyperparameters are varied in our experiments. We always sweep over a range of learning rates and choose the largest learning rate at which training is stable. For further details see Appendix B.
To pretrain a randomly initialized transformer on data with task diversity  $M$ , we first construct the pretraining task distribution,  $\mathcal{T}_{\mathrm{Pretrain}}$ , as described in Section 2. We then minimize the objective  $L^{\mathcal{T}_{\mathrm{Pretrain}}}$  in Eq. (1) using minibatch stochastic gradient descent. For each sequence in a minibatch, we sample a single task  $\mathbf{w}$  from  $\mathcal{T}_{\mathrm{Pretrain}}$ , as well as new samples of data,  $\{\mathbf{x}_i\}_{i=1}^K$ , and noise,  $\{\varepsilon_i\}_{i=1}^K$ , from their respective continuous distributions, to form a sequence  $(\mathbf{x}_1, \mathbf{w}^\top \mathbf{x}_1 + \varepsilon_1, \ldots, \mathbf{x}_K, \mathbf{w}^\top \mathbf{x}_K + \varepsilon_K)$ . If we train for  $N$  steps at batch size  $B$ , the transformer will see a total of  $NB$  unique sequences and roughly  $\frac{NB}{M}$  unique sequences for each latent task in  $\mathcal{T}_{\mathrm{Pretrain}}$ . By increasing either  $B$  or  $N$  at fixed  $M$ , we can increase the total size of the pretraining dataset (or number of sequences per task) while keeping the dataset diversity—the number of unique ws in  $\mathcal{T}_{\mathrm{Pretrain}}$ —fixed.
3.1 Task diversity threshold for the emergence of in-context learning
For Fig. 2, we pretrain our base transformer on datasets with increasing task diversity (on the x-axis) while keeping the total number of sequences seen during pretraining fixed ( $B = 256$ ,  $N = 500K$ ). We evaluate the PTs and both optimal estimators on tasks seen during pretraining drawn from  $\mathcal{T}_{\mathrm{Pretrain}}$  (Fig. 2 top left) and on new tasks drawn from  $\mathcal{T}_{\mathrm{True}}$  (Fig. 2 bottom left) and plot MSE normalized by task dimension— $L^{\mathcal{T}} / D$  from Eq. (1)). Since dMMSE is optimal on tasks from  $\mathcal{T}_{\mathrm{Pretrain}}$  (as discussed in Section 2), the green dMMSE markers denote the lowest possible loss the PT could achieve in this setting. In fact, the pretraining objective  $L^{\mathcal{T}_{\mathrm{Pretrain}}}$  explicitly encourages the PT to match dMMSE performance. On the other hand, Ridge is optimal on tasks sampled from  $\mathcal{T}_{\mathrm{True}}$  (Fig. 2 bottom left); the blue markers denote the lowest possible MSE the PT could attain on new tasks.
Low task diversity phase: the PT is Bayesian with respect to the pretraining distribution and cannot solve new tasks. At low pretraining task diversity— $M$  up to about  $2^{6}$ —the PT's MSE closely tracks that of dMMSE on tasks sampled from  $\mathcal{T}_{\mathrm{Pretrain}}$  (Fig. 2 top left); the PT performs optimally on tasks seen during pretraining. But it significantly underperforms on new tasks sampled from  $\mathcal{T}_{\mathrm{True}}$ , indicated by the gap in MSE between the PT and Ridge (Fig. 2, bottom left). In this regime, it behaves like the Bayesian estimator with prior  $\mathcal{T}_{\mathrm{Pretrain}}$ .
High task diversity phase: the PT is non-Bayesian with respect to the pretraining task distribution and can solve new tasks. At higher task diversities—above  $2^{6}$  pretraining tasks—the PT's MSE deviates from dMMSE and approaches Ridge under both  $\mathcal{T}_{\mathrm{Pretrain}}$  and  $\mathcal{T}_{\mathrm{True}}$ . Crucially, the PT starts to significantly outperform dMMSE on unseen tasks sampled from  $\mathcal{T}_{\mathrm{True}}$  (Fig. 2 bottom left) at the expense of not fully minimizing its training objective,  $L^{\mathcal{T}_{\mathrm{Pretrain}}}$  (gap between PT and dMMSE under  $\mathcal{T}_{\mathrm{Pretrain}}$ , Fig. 2 top left). This suggests that, a PT trained on a finite but large number of pretraining tasks can learn fundamentally new tasks in-context and this ability depends on its deviating from the optimal Bayesian estimator on the pretraining task distribution.
Figure 4: Learning dynamics of small PTs shows a transition at the task diversity threshold. We plot  $\Delta_{\mathrm{PT,Ridge}}^{\mathcal{T}_{\mathrm{Pretrain}}}$  vs training steps for small PTs. For the same  $M$ , learning curves for short (500K steps, left) or long (2M steps, center) training durations are similar, and for  $M > M^{*} \approx 2^{11.5}$  learning curves are similar to that of a model trained with infinite task diversity. Right: For  $M \leq 2^{10}$ ,  $t^{*}$  (the training step at which  $\Delta_{\mathrm{PT,Ridge}}^{\mathcal{T}_{\mathrm{Pretrain}}}$  is minimized) is well modeled by a scaling law  $t^{*} \propto M^{\alpha}$ . A linear fit of  $\log t^{*}$  vs  $\log M$  (dashed red line) gives  $\alpha \approx 0.47$ . But for  $M > 2^{10}$ ,  $\Delta_{\mathrm{PT,Ridge}}^{\mathcal{T}_{\mathrm{Pretrain}}}$  decreases through training;  $t^{*} = 2\mathrm{M}$ , is larger than predicted by the scaling law. This sudden break in the scaling law suggests a fundamental difference in the learning dynamics of models on either side of the threshold.