Bayesian scaling laws for in-context learning

## Abstract

Abstract In-context learning (ICL) is a powerful technique for getting language models to perform complex tasks with no training updates.
Prior work has established strong correlations between the number of in-context examples provided and the accuracy of the model’s predictions.
In this paper, we seek to explain this correlation by showing that ICL approximates a Bayesian learner. This perspective gives rise to a family of novel Bayesian scaling laws for ICL.
In experiments with GPT-2 models of different sizes, our scaling laws exceed or match existing scaling laws in accuracy while also offering interpretable terms for task priors, learning efficiency, and per-example probabilities.
To illustrate the analytic power that such interpretable scaling laws provide, we report on controlled synthetic dataset experiments designed to inform real-world studies of safety alignment. In our experimental protocol, we use SFT to
suppress an unwanted existing model capability and then use ICL to try to bring that capability back (many-shot jailbreaking). We then experiment on real-world instruction-tuned LLMs using capabilities benchmarks as well as a new many-shot jailbreaking dataset.
In all cases, Bayesian scaling laws accurately predict the conditions under which ICL will cause the suppressed behavior to reemerge, which sheds light on the ineffectiveness of post-training at increasing LLM safety. https://github.com/aryamanarora/bayesian-laws-icl

### 1 Introduction

Large language models (LLMs) can infer how to perform a task given only demonstrations and without additional training updates. This capability is known as in-context learning (ICL; Brown et al., ; Dong et al., ). Under ICL, task performance generally increases with the number of demonstrations,
though the precise relationship between these two quantities
is unclear. We call this relationship the ICL curve and seek to model it. Being able to predict the shape of the ICL curve would help us decide whether to do many-shot ICL Agarwal et al. () after testing only few-shot performance, predict potential alignment failures under many-shot jailbreaking (Anil et al., ), and decide how much fine-tuning we need in order to suppress ICL of undesirable behaviours.

The learning algorithm underlying ICL has been characterised as Bayesian by Xie et al. () and many later works (section 2). Drawing on this line of research, we use Bayes’ theorem to derive a family of Bayesian scaling laws for ICL (section 3) which model the ICL curve of an ideal Bayesian learner.

To evaluate the performance of our Bayesian laws, we model the ICL curve for gpt2 models trained on simple synthetic data following Xie et al. () as well as real-world LLMs tested on standard benchmarks (section 4.1). Compared to the power laws proposed by Anil et al. (), our Bayesian laws achieve lower error rates on both interpolation and extrapolation of the ICL curve, while also providing interpretable parameters for the prior over tasks, the efficiency of ICL, and per-example probabilities under different tasks.
In our second set of experiments (section 4.2), we present a case study using our Bayesian laws to model how post-training affects ICL of favoured and disfavoured behaviours. On toy models, we find that smaller amounts of post-training strongly change the prior over tasks but not the model’s knowledge of each task, and the amount of post-training needed to suppress ICL of disfavoured tasks increases with scale.

Finally, we present experiments on real-world LLMs ranging from 1B to 405B parameters (section 5). Our laws accurately predict the ICL behaviour of several models on both capabilities and safety benchmarks and a new many-shot jailbreaking dataset we introduce. We then compare Llama 3.1 8B Base and Instruct using one of our Bayesian scaling laws (section 5.2) and find that alignment merely reduces the prior probability of harmful behaviour but not its learnability under ICL. Our work thus introduces a tool for interpreting the task knowledge of LLMs using purely behavioural observations, which we hope is valuable for improving LLM alignment.

### 2 Related work

###### Understanding in-context learning.

LMs trained from scratch on controlled synthetic data have been variously claimed to approximate Bayesian learners Xie et al. (); Hahn & Goyal (); Zhang et al. (); Jiang (); Wies et al. (), gradient descent von Oswald et al. (); Ahn et al. (), or differing learning algorithms depending on the task, model scale, and training progress Akyürek et al. (); Garg et al. (); Bai et al. (); Shen et al. (); Falck et al. ().
Neverthless, no work has attempted to directly model the ICL curve on the basis of claims about the learning algorithm underlying ICL. In this work, we test the claims that LMs are Bayesian learners by deriving an expression for the ICL curve under Bayesian assumptions and seeing how well it models actual ICL behaviour.

###### Scaling laws.

Researchers have sought to characterise how LM loss and performance relates to model architecture, model scale, data scale, and training hyperparameters in order to predict and optimise training runs Kaplan et al. (); Hoffmann et al. (). LM scaling laws may also take into account data complexity Pandey () or use more expressive formulations for better extrapolation Alabdulmohsin et al. (); Caballero et al. ().
Power laws seem ubiquitous in describing LM behaviour and have recently been adopted to model the ICL curve under different model and data settings Anil et al. (); Liu et al. (); we use these power laws as baselines.

###### The ineffectiveness of post-training.

Much work has found that post-training, even when applied at scale, only changes LLM behaviour in ways that are superficial and easy to bypass (Qi et al., ; Zou et al., ; Shayegani et al., ; Carlini et al., ; Geiping et al., ; Jain et al., ; Prakash et al., ; Wei et al., ; Lee et al., ; Wei et al., ; Schwinn et al., ; Sheshadri et al., ).

Concerningly, ICL enables re-learning of behaviours that were suppressed with fine-tuning Wei et al. (); Xhonneux et al. (); Anil et al. (); Anwar et al. ().
Under a Bayesian view of post-training, it is possible that task priors are only reweighted while task knowledge is unchanged; our Bayesian scaling laws can test this hypothesis.

### 3 A Bayesian law for in-context learning

As discussed in section 2, there are many competing hypotheses about how ICL is learned and implemented in LMs. When training LMs on a variety of simple algorithmic tasks (e.g. linear regression, HMM next-emission prediction), many works find that ICL approximates a Bayesian learner (Xie et al., , inter alia).

If ICL is indeed Bayesian, we should be able to use Bayesian assumptions to exactly predict how prediction accuracy relates to number of in-context examples. This observation leads us to state some key assumptions necessary to frame ICL as Bayesian. Next, we use repeated application of Bayes’ theorem to model how ICL updates the task prior after encountering each new in-context example (section 3.1). Finally, we simplify our model to reduce parameter count and add an efficiency coefficient $K$ to take into account the effect of example length and informativeness (section 3.2). This results in a family of Bayesian scaling laws. We close the section by setting up some baselines and metrics for our experiments (section 3.3).

#### 3.1 Derivation

###### Definition 1 (Bayesian model of ICL) .

We define a Bayesian model of ICL as a tuple $\mathcal{M}=\langle\Sigma,\mathcal{T},\rho,\delta\rangle$, where

- •
$\Sigma$ is a finite alphabet of symbols $\sigma$.
- •
$\mathcal{T}=\{T_{1},\ldots,T_{M}\}$ is a set of tasks of size $M$.
- •
$\rho:\mathcal{T}\to[0,1]$ is the prior probability distribution over tasks, such that $\sum_{m=1}^{M}\rho(T_{m})=1$.
- •
$\delta:\mathcal{T}\times\Sigma\to[0,1]$ is a likelihood function, mapping a task $T_{m}\in\mathcal{T}$ and symbol $\sigma\in\Sigma$ to probability such that $\sum_{\sigma}\delta(T_{m},\sigma)=1$ for all $T_{m}\in\mathcal{T}$. This represents the conditional probability $p(\sigma\mid T_{m})=\delta(T_{m},\sigma)$.

Now let $D\in\Sigma^{n}$ be a string of $n$ symbols, i.e. a document. When processing this document, our Bayesian model of ICL $\mathcal{M}$ computes a posterior over tasks in accordance with Bayes’ theorem:

$$ $p(T_{m}\mid D)=\frac{p(D\mid T_{m})p(T_{m})}{\sum_{m=1}^{M}p(D\mid T_{m})p(T_{ m})}$ (1) $$

We enforce the condition that the probability of future symbols under this model depends entirely on the task posterior, i.e. $p(\sigma\mid D)=\sum_{m=1}^{M}p(\sigma\mid T_{m})p(T_{m}\mid D)$, and is thus independent of any other properties of the previously processed symbols.

The model we have defined represents initial uncertainty about the task at hand as the prior over tasks $p(T_{m})$, and its knowledge about the symbols associated with each task as $\delta$, the per-example probabilities. Due to the Bayesian update setup, as it sees more in-context examples, its posterior over tasks will converge to allocate all probability mass to the task under which those examples have the highest expected probability.(^1^11See the Bernstein–von Mises theorem and related discussion in Xie et al. ().)

We now derive a functional form for the ICL curve, relating number of in-context examples (i.e. the length of document $D$) to the expected probability of the next example ($p(\sigma\mid D)$).

###### Theorem 1 (Bayesian law for ICL) .

Given the following:

- •
$\mathcal{M}=\langle\Sigma,T,\delta\rangle$, is a Bayesian model of ICL;
- •
$\lambda:\sigma\to\mathbb{R}_{\geq 0}$, such that $\sum_{\sigma\in\Sigma}\lambda(\sigma)=1$, is a one-hot sampling distribution over $\Sigma$;
- •
$D\in\Sigma^{n}$ is a list of symbols sampled i.i.d. under $\lambda$, i.e. a document.

the next-example probability under the Bayesian model $\mathcal{M}$ given a document $D$ consisting of $n$ in-context examples sampled from $\lambda$ is

$$ $\mathbb{E}_{\sigma\sim\lambda}\left[p(\sigma\mid D)\right]=\frac{\sum_{m=1}^{M }{\mathbb{E}_{\sigma\sim\lambda}\left[p(\sigma\mid T_{m})\right]^{n+1}p(T_{m}) }}{\sum_{m=1}^{M}{\mathbb{E}_{\sigma\sim\lambda}\left[p(\sigma\mid T_{m}) \right]^{n}p(T_{m})}}$ (2) $$

where $p(T_{m})$ is the prior probability of task $T_{m}$, and the expectation $\mathbb{E}_{\sigma\sim\lambda}\left[p(\sigma\mid T_{M})\right]$ is computed over $\lambda$, the distribution the documents are sampled from.

###### Proof.

See appendix A.
∎

To model a particular distribution $T_{k}$ with this scaling law, we set $\lambda:=T_{k}$ and sample examples from $T_{k}$ to fit $\mathbb{E}_{\sigma\sim T_{k}}\left[p(\sigma\mid D)\right]$. To model multiple distributions together, we perform the same procedure on each distribution but share the priors $p(T)$ across distributions.

Our law has $M^{2}+M$ parameters to fit, where $M$ is the total number of distributions to model. $M^{2}$ of these terms are of the form $\mathbb{E}_{\sigma\sim T_{k}}\left[p(\sigma\mid T_{m})\right]$, i.e. the expected likelihood of an example sampled from $T_{k}$ under distribution $T_{m}$. The remaining $M$ terms are the prior probabilities $p(T_{m})$.

Figure: Figure 1: The sampling/scoring matrix $P$ (left) and our two approaches (middle and right) for reducing the number of unobserved parameters by tying values. Green boxes indicate observed values and red boxes indicate unobserved values.

###### Definition 1 (Bayesian model of ICL) .

###### Theorem 1 (Bayesian law for ICL) .

###### Proof.

#### 3.2 Making the Bayesian scaling law practical

We now describe some minor modifications to this law that simplify the model without harming empirical performance.

###### Reducing unobserved parameter count.

The initial formulation of the Bayesian law has a much larger parameter count than e.g. a power law. Instead of scaling quadratically with the number of distributions, we want the parameter count to scale linearly to make the comparison fair.

When fitting our Bayesian law to every task $T_{k}$, we must fit $M^{2}$ terms of the form $\mathbb{E}_{\sigma\sim T_{k}}\left[p(\sigma\mid T_{m})\right]$. This represents the probability of a sample from $T_{k}$ when scored under $T_{m}$.
When processing a series of examples sampled from task $T_{k}$, under an ideal Bayesian learner the task posterior converges to task $T_{k}$. Thus, asymptotically, the probability $\mathbb{E}_{\sigma\sim T_{k}}\left[p(\sigma\mid D)\right]$ converges to $\mathbb{E}_{\sigma\sim T_{k}}\left[p(\sigma\mid T_{k})\right]$. If we lay out a matrix $P\in\mathbb{R}^{M\times M}$ where $P_{i,j}=\mathbb{E}_{\sigma\sim T_{i}}\left[p(\sigma\mid T_{j})\right]$, this means, given the true ICL curve, we only observe the $M$ values along the diagonal.

To reduce the number of learned parameters that we cannot observe (and which can thus take on arbitrary values and lead to overfitting), we can tie some of the non-diagonal values in $P$. We propose two approaches to tying parameters: sampling-wise and scoring-wise. Under sampling-wise tying, we tie off-diagonal values in each column, and under scoring-wise tying we do the same but for rows.
We depict these two approaches graphically in Figure 1. Both approaches reduce the parameter count from $M^{2}+M$ to $3M$, and the number of unobserved parameters from $M^{2}-M$ to $M$, making the complexity of the law in line with that of a power law.

###### Multiple updates.

A key assumption in our law is that a Bayesian update only occurs after each in-context example is processed. In practice, LLMs process inputs token-by-token, and an in-context example may consist of multiple tokens. Examples may also vary in informativeness. To allow for flexibility in this regard, we multiply $n$ (number of in-context examples) by a learned ICL efficiency coefficient $K$ which modulates the strength of the Bayesian update.

###### Final scaling law.

We finally obtain the following functional form for the Bayesian scaling law:

$$ $\mathbb{E}_{\sigma\sim\lambda}\left[p(\sigma\mid D)\right]=\frac{\sum_{m=1}^{M }{(P_{\lambda,m})^{Kn+1}\rho_{m}}}{\sum_{m=1}^{M}{(P_{\lambda,m})^{Kn}\rho_{m}}}$ (3) $$

When fitting $M$ distributions, the total parameter count is $M^{2}+M+1$ for the original parameterisation of $P$, and $3M+1$ for sampling- and scoring-wise parameterisations. The only difference between the three variants of the Bayesian scaling law is how we tie values in $P$.

#### 3.3 Baselines

We compare our Bayesian scaling law with a few other functional forms; our choice of baselines is further justified in appendix B. Anil et al. () attempt to fit scaling laws to the curve relating number of in-context examples to negative log-likelihood. They use a power law and a bounded power law:

$$ $-\log{p_{\text{power}}(\sigma\mid D)}=Cn^{-\alpha}+K$ (4) $$

$$ $-\log{p_{\text{bounded}}(\sigma\mid D)}=C\left(1+\frac{n}{n_{c}}\right)^{- \alpha}+K$ (5) $$

Along with these, we benchmark the logistic function with input in log space as a baseline.

$$ $-\log{p_{\text{logistic}}(\sigma\mid D)}=\frac{C}{1+\left(\frac{n}{n_{c}} \right)^{-\alpha}}+K$ (6) $$

We list all the laws we study in Table 1 and report our procedure for fitting all laws in appendix D.

#### 3.4 Evaluation metrics

To evaluate how well a scaling law fits, we compute the normalised root mean-squared error (NRMSE). Given ground-truth values $\mathbf{y}=[y_{1},\ldots,y_{n}]$ and predicted values $\mathbf{\hat{y}}=[\hat{y}_{1},\ldots,\hat{y}_{n}]$,

$$ $\text{RMSE}(\mathbf{y},\mathbf{\hat{y}})=\sqrt{\frac{\sum_{i=1}^{n}{(y_{i}- \hat{y}_{i})^{2}}}{n}}\qquad\text{NRMSE}(\mathbf{y},\mathbf{\hat{y}})=\frac{ \text{RMSE}(\mathbf{y},\mathbf{\hat{y}})}{\frac{1}{n}\sum_{i=1}^{n}{y_{i}}}$ (7) $$

NRMSE is comparable across different populations, so we can use it to compare how good fits are between different models and datasets. We compute this metric on raw probabilities, not NLL. Finally, to establish statistical significance between the NRMSE of pairs of scaling laws, we simply run a paired $t$-test and report a significant comparison if the $p$-value is below 0.05.

### 4 Experiments on synthetic data (GINC)

We conduct a series of experiments comparing how well different scaling laws fit the ICL behaviour of toy transformer models trained from scratch on synthetic data. We use Xie et al. ()’s GINC dataset as our testbed for studying ICL in a controlled manner, pretraining LMs at various scales from scratch and observing their ICL behaviour before and after post-training. We report a summary of the results from this section in Table 1.

#### 4.1 Experiment 1: Can Bayesian scaling laws describe ICL on GINC?

Table: Table 1: Overview of scaling laws and their performance on GINC. Extrapolation is in the $10\%$ setting. Bold indicates lowest NRMSE or statistical insignificance when comparing to the lowest. See appendix F for more.

Xie et al. () introduce the GINC (Generative In-Context Learning) dataset as a synthetic testbed for studying ICL. GINC is created by sampling trajectories from a mixture of hidden Markov models that have sparse transition matrices.
Not only does training on GINC lead to ICL behaviour, but we also have knowledge of the ground-truth prior over the HMMs which we can use to sanity-check the inferred prior of our Bayesian scaling laws. Thus, we start by evaluating our laws in this controlled setting.

###### Data.

We create a GINC dataset with parameters specified in appendix D. The dataset consists of documents of length 10240 (including a prepended BOS token) sampled uniformly from 5 hidden Markov models. We also create a validation set of 50 documents of length 1024 sampled from the same GINC distribution.

###### Method.

We pretrain gpt2-architecture autoregressive language models with varying numbers of layers on GINC. We replicate the architecture and training setup in Xie et al. (). We chunk documents into sequences of length 1024, the maximum size of our context window. Our training objective is the next-token prediction task, minimising cross-entropy loss with teacher-forcing over all tokens.

$$ $\min_{\theta}\left\{-\mathbb{E}\left[\log{p_{\theta}(x_{i}\mid\mathbf{x}_{<i}) }\right]\right\}$ (8) $$

We provide additional details on model architecture and training hyperparameters in appendix D. For each of the model scales, we report pretraining losses on a training and validation set in Table 2(a).

###### ICL curve.

Following Xie et al. (), we evaluate the ICL ability of our GINC models on sequences that mimic the format of ICL examples in real-world LLM evaluation. Each evaluation document is a series of HMM trajectories of length $k$ all independently sampled from the same HMM and separated by the designated delimiter token. For each ICL example, we evaluate the probability of the gold $k$-th token at the $(k-1)$-th position; this forms our ICL curve.

Figure: (a) Train and validation losses of various sizes of gpt2 models pretrained on GINC. In all cases, we achieve better or similar val loss compared to those reported in Xie et al. ().
Refer to caption: x1.png

##### 4.1.1 Bayesian scaling laws outperform baselines

We now fit each of the scaling laws in Table 1 to the curve relating number of ICL examples to probability of the gold $k$-th token. Since only gpt models with at least 3 layers exhibit ICL on this task, we do not include scores for models with 1 or 2 layers when reporting averages. To compute statistical significance between pairs of models, we perform a paired $t$-test and report whether the $p$-value is below 0.05. We report detailed results in appendix F.

###### Interpolation error.

We fit each of the laws to all of the data and evaluate the fits, averaged over 5 random seeds. We plot average NRMSE for each law across model scales and trajectory lengths ($k$) in Figure 2(a), and report average NRMSE in Table 1. We find that the Bayesian (original) scaling law handily achieves statistically-significantly lower NRMSE than every other law, except for a non-significant comparison with our strong logistic baseline.

###### Extrapolation error.

Following Caballero et al. ()’s qualitative evaluation of extrapolation behaviour for model scaling laws, we perform a quantitative evaluation of extrapolation error. We take the first $10\%$ of the points in every ICL curve, fit each scaling law once, and report NRMSE on the remaining $90\%$ of the curve (which the laws were not fit to) in Table 1. Under this evaluation, the scoring-wise Bayesian scaling law achieves the best performance.

##### 4.1.2 Bayesian scaling laws have interpretable parameters

Figure: (a) Priors ($\rho$) of the Bayesian (scoring-wise) method. Longer trajectory lengths lead to inferred priors that are closer to uniform.
Refer to caption: x2.png

Now that we have confirmed that the Bayesian law is an accurate model of ICL behaviour, we can interpret the learned parameters of the Bayesian fits.
We plot some interesting parameters of the scoring-wise Bayesian law in Figure 3. We observe the following:

- •
The prior ($\rho$) distributions are somewhat noisy but roughly uniform, agreeing with the uniform pretraining distribution over the HMMs.
- •
ICL efficiency ($K$) roughly increases with model depth i.e. larger models have faster ICL, and with the length of each provided ICL example, i.e. more informative examples lead to faster ICL.

In general, we find that the scoring-wise Bayesian scaling law is the most in agreement with our knowledge about the pretraining distribution. On GINC, it seems that Bayesian scaling laws are interpretable and explain the shape of the ICL curve well, across a variety of model scales and ICL trajectory lengths.

#### 4.2 Experiment 2: Can Bayesian scaling laws model SFT on GINC?

The brittleness of post-training (section 2) shown through e.g. many-shot jailbreaking (Anil et al., ) raises the question: does post-training merely update model priors over subdistributions, or does it fundamentally change the knowledge models have about those subdistributions? We can operationalise this hypothesis with our Bayesian scaling laws by post-training various models with SFT, fitting the laws to their ICL behaviour, and examining whether parameters other than the prior ($\rho$) shift under post-training.

###### Data.

We fine-tune each model on samples taken only from HMM 0, on datasets equivalent in size to $\{1\%,2\%,5\%,10\%,20\%,50\%,100\%\}$ of the total number of pretraining examples.

###### Method.

We use the same next-token cross-entropy loss as in eq. 8 to perform supervised finetuning only on this positive subdistribution; see appendix D for hyperparameters. We fit a separate instance of the Bayesian law for each combination of model depth, example length, and # of SFT examples.

##### 4.2.1 SFT is more superficial with scale

Table 1 shows that the original Bayesian scaling law achieves the lowest average NRMSE, while scoring-wise beats all but the bounded power law. We present plots of some of the priors and the in-distribution symbol probabilities (i.e. the probability the model will converge to given infinite examples from a particular distribution) for the scoring-wise Bayesian scaling law in Figure 4.

In Figure 4(a), we can observe how the prior suddenly shifts to favour HMM 0 as SFT progresses with greater amounts of data. Notably, both the prior and the in-distribution scores (Figure 4(b)) change much more slowly for larger models, implying that SFT is less effective at larger scales at changing the knowledge the model possesses about subdistributions. Past a threshold, SFT seems to indeed change the model’s knowledge of the subdistributions (and not just its priors), but this threshold is higher for larger models.

#### 4.3 Experiment 3: DPO on GINC

###### Data.

We do the same as in the SFT experiment but with $\{0.1\%,0.2\%,0.5\%,1\%,2\%,5\%,10\%\}$ of the total number of pretraining examples. The prompt of each document is a single BOS token; the positive continuation is a sample from HMM 0 and the negative continuation is a sample from one of the other HMMs, taken uniformly.

###### Method.

DPO is a preference-learning RLHF method capable of directly optimising a language model without training a separate reward model Rafailov et al. (). Given a positive output $\mathbf{y_{w}}$ and a negative output $\mathbf{y_{l}}$, the training objective of DPO is

$$ $\min_{\theta}\left\{\mathbb{E}\left[\log\sigma\left(\beta\log\frac{p_{\theta}( \mathbf{y_{w}}\mid\mathbf{x})}{p_{\text{ref}}(\mathbf{y_{w}}\mid\mathbf{x})} \right.\right.\right.-\left.\left.\left.\beta\log\frac{p_{\theta}(\mathbf{y_{l }}\mid\mathbf{x})}{p_{\text{ref}}(\mathbf{y_{l}}\mid\mathbf{x})}\right)\right]\right\}$ (9) $$

In this case, the original pretrained model is referred to as $p_{\text{ref}}(\cdot)$, which we clone and optimise as $p_{\theta}(\cdot)$. We only update the parameters of $p_{\theta}(\cdot)$. We report hyperparameters in appendix D. We fit scaling laws the same way as in section 4.2.

Figure: (a) Priors ($\rho$) of the Bayesian (scoring-wise) method. Probability is abruptly concentrated on HMM 0 past a certain threshold of SFT training, and the threshold increases with model scale.
Refer to caption: x4.png

Figure: (a) NRMSE of all of the laws when varying # of DPO examples. With enough DPO, the ICL curve collapses and is poorly fit by all of the scaling laws. Notably, the laws have very similar NRMSEs.
Refer to caption: x6.png

##### 4.3.1 DPO (eventually) breaks the ICL curve

We show some key results in Figure 5. Unlike SFT, DPO can use very little training data to successfully suppress the disfavoured HMMs beyond the ability of ICL to recover. However, with enough DPO training, the probability of the preferred output (HMM 0) also declines and the ICL curve eventually collapses. As a result, none of the scaling laws model the ICL curve well after some amount of DPO training.We do observe that larger models require slightly more DPO training to suppress the negative distribution, but not as starkly as for SFT.

The collapse of the positive distribution is a known failure mode of DPO, which occurs because it maximises the relative difference between the probabilities of the positive and negative distributions (Pal et al., ; Feng et al., ; D’Oosterlinck et al., ). Overall, DPO impacts more of the model’s knowledge about tasks than SFT.

### 5 Experiments on real-world LLMs and datasets

We extensively studied the application of Bayesian scaling laws on a synthetic testbed (GINC) for pretrained and SFT/DPO models that we trained from scratch. Still, it is unclear to what extent GINC accurately models real-world ICL. Beyond a theoretical proof that ICL on GINC is possible (Xie et al., ), we have no guarantees that findings on our toy model transfer to the real world.
Therefore, we evaluate the actual ICL behaviour of real-world LLMs trained on natural language and fit all the scaling laws at our disposal, using the same methodology as in section 4.1.

#### 5.1 Experiment 4: Bayesian scaling laws are competitive on real-world LLMs

Table: Table 2: Real-world LLMs: Comparison of scaling laws at fitting ICL behaviour on real-world LLMs at a variety of tasks. Bold indicates lowest NRMSE or statistical insignificance when comparing to the lowest. See appendix F for more.

Figure: (a) Raw probabilities of the Llama 3.1 8B Base and Instruct. Instruct has overall better ICL, except on harmbench. Base suffers from degradation with greater numbers of shots.
Refer to caption: x8.png

###### Data.

Our datasets include both capabilities and safety evaluations, including 2 multiple-choice reasoning benchmarks, 3 binary-choice personality evaluations from Perez et al. (), and a new many-shot jailbreaking dataset that we created using HarmBench (Mazeika et al., ). More details are in section E.2.

###### Method.

We experiment on 7 instruction-tuned LLMs from the Gemma and Llama families, with parameter counts spanning from 1B to 405B parameters; see section E.1 for details. For each dataset and model pair, we construct $50$ many-shot prompts adhering to each model’s chat template. We use as many shots as possible, filling the context window. We run the LLM on each of these many-shot prompts and, for each shot, store the next-token prediction probability of the relevant portion of the response. We find that many LLMs suffer degradation near the end of their context window, so we only use the data from the starting $90\%$ of the context window.

###### Results.

As before, we fit each of the scaling laws to the ICL curves and evaluate the quality of the fits by comparing the NRMSE of the predictions. We report overall results across all models in Table 2; we find that most comparisons between the scaling laws are not statistically significant, so again the Bayesian laws are not worse than alternatives.

#### 5.2 Experiment 5: Comparing Llama 3.1 8B Base and Instruct

In our final experiment, we compare the parameters of the Bayesian (scoring-wise) law on Llama 3.1 8B Base and Instruct on all of the real-world tasks. The Base model was not used in the previous experiment. We report raw probabilities as well as the posterior probabilities for the task computed by the scaling law in Figure 6. We find that the instruction-tuning of this model does reduce the prior probability of unsafe behaviours (harmbench and the 3 persona evals) but fails to prevent many-shot jailbreaking.

Our scaling law shows that the posterior eventually saturates even if instruction-tuning reduces the prior. Along with our synthetic experiments with SFT and DPO in a low-data setting, this is additional evidence for the claim that real-world instruction-tuning merely modifies the prior over tasks and not task knowledge. This may be because the compute allocated to instruction-tuning is is still too small compared to that for pretraining.

### 6 Discussion

In-context learning, like most of the noteworthy properties of large language models, is something that we don’t quite understand. This paper emerged from our attempt to reconcile the existing literature that attempts to ascribe a Bayesian basis for the emergence of ICL with the empirical science of scaling laws. We did find that Bayesian scaling laws are competitive with non-theoretical (and relatively unconstrained) scaling laws at modelling ICL behaviour in both toy and real settings.

###### Real-world applications.

The Bayesian approach seems to perform better at extrapolating model behaviour from a few shots. This can be useful for predicting multi-turn safety failures before they happen or whether additional inference-time computation will deliver worthwhile gains.

###### Interpretability.

An additional advantage of our approach is that the parameters of the scaling laws mean something and so can shed light on the internal workings of LLMs without needing to fully open the black box. E.g. studying both the prior over tasks and how ICL affects their posterior is valuable for interpreting the effects of alignment on real-world LLMs. Future work could also mechanistically interpret how Bayesian ICL is performed (e.g. localise the prior in activation space).

###### Are LLMs Bayesian?

In this work we attempt to elucidate model behaviour without reference to model internals. We believe that our results strengthen the claim that LLMs perform Bayesian inference, but do not necessarily prove it. We note that previous works claiming that LLMs are theoretically Bayesian prove their claims on toy models that vastly simplify the complexity of natural language and web-scale pretraining data;(^2^22See e.g. Hahn & Goyal (, sec. 1.4) on the limitations of toy models that assign priors to a fixed non-compositional set of tasks like Xie et al. (), the basis of our toy experiments.) it’s possible that actual web-scale Bayesian reasoning is beyond the capacity of current LLMs, but they still may behave approximately Bayesian, explaining the success of our scaling law.

### 7 Conclusion

In this paper, we combined two questions to make progress at understanding ICL: (1) what scaling law best describes ICL, and (2) is ICL Bayesian? We showed that Bayesian assumptions naturally lead to a scaling law for ICL, and that Bayesian scaling laws are a great fit for both ICL behaviour by small LMs trained on controlled synthetic data, as well as LLMs trained on natural language. Using a Bayesian formulation gave us interpretable parameters for the prior, learning efficiency, and task-conditional probabilities, which can help us understand how model behaviour changes under alignment. We use these to show how ICL ability varies at different model scales, understand how finetuning harms knowledge of disfavoured distributions, and compare base and instruction-tuned LLMs. We are confident that further progress on understanding ICL is possible through the empirical science of scaling laws.

### Acknowledgements

We thank Rohan Pandey, Karel D’Oosterlinck, Ken Liu, Harshit Joshi, Nikil Roashan Selvam, Chenglei Si, Jiuding Sun, Simon Guo, Julie Kallini, Yanzhe ‘Sanju’ Zhang, and Leonard Tang for enlightening discussion about the project and feedback on the manuscript. We also thank members of CoCoLab and Jurafsky Lab for their helpful feedback on early presentations about the project.

We would especially like to thank Rylan Schaeffer for finding and helping us correct a mistake in our proof.

This work is supported in part by a grant from the Laboratory Directed Research and Development program at Sandia National Laboratories. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc. for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

## Appendix

### Appendix A Deriving a law for in-context learning

See 1

See 1

###### Proof.

Consider a particular sequence $D\in\Sigma^{n}$. To compute the posterior probabilities of of the $M$ distributions after the Bayesian learner has processed this sequence, we can use Bayes’ theorem.

$$ $\displaystyle p(T_{j}\mid D)$ $\displaystyle=\frac{p(D\mid T_{j})p(T_{j})}{p(D)}$ (Bayes’ theorem) (10) $\displaystyle=\frac{p(D\mid T_{j})p(T_{j})}{\sum_{m=1}^{M}p(D\mid T_{m})p(T_{m })}$ (expand denominator) (11) $\displaystyle=\frac{p(T_{j})\prod_{i=1}^{n}{p(D_{i}\mid T_{j})}}{\sum_{m=1}^{M }p(T_{m})\prod_{i=1}^{n}{p(D_{i}\mid T_{m})}}$ ( $D$ is an i.i.d. sequence of symbols) (12) $$

We can now marginalise the probability of the next symbol $\sigma$ over these $M$ distributions:

$$ $\displaystyle p(\sigma\mid D)$ $\displaystyle=\sum_{m=1}^{M}{p(\sigma\mid T_{m})p(T_{m}\mid D)}$ (expand) (13) $\displaystyle=\frac{\sum_{m=1}^{M}{p(\sigma\mid T_{m})p(T_{m})\prod_{i=1}^{n}p (D_{i}\mid T_{m})}}{\sum_{m=1}^{M}{p(T_{m})\prod_{i=1}^{n}p(D_{i}\mid P_{m})}}$ (substitute eq. 12 ) (14) $$

What we actually care about though is the expectation of $p(\sigma\mid D)$ over the whole distribution of documents. Since our documents are sequences of symbols sampled i.i.d. from $\lambda$, we can exploit the independence of the symbols to decompose the whole-document probability into a product of symbol probabilities.

Every expectation below is computed over $\sigma\sim\lambda$. For notational simplicity, we do not explicitly indicate this.

$$ $\displaystyle\mathbb{E}\left[p(\sigma\mid D)\right]$ $\displaystyle=\mathbb{E}\left[\frac{\sum_{m=1}^{M}{p(\sigma\mid T_{m})p(T_{m}) \prod_{i=1}^{n}p(D_{i}\mid T_{m})}}{\sum_{m=1}^{M}{p(T_{m})\prod_{i=1}^{n}p(D_ {i}\mid T_{m})}}\right]$ (16) $$

Recall that we enforce that $\lambda$ is a one-hot distribution, i.e. all of its probability mass is allocated to a single symbol. This enables removing the expectation, since each of the $D_{i}$ are now identical and only one outcome of $D$ is possible.

$$ $\displaystyle\mathbb{E}\left[p(\sigma\mid D)\right]$ $\displaystyle=\frac{\sum_{m=1}^{M}{p(\sigma\mid T_{m})p(T_{m})\prod_{i=1}^{n}p (D_{i}\mid T_{m})}}{\sum_{m=1}^{M}{p(T_{m})\prod_{i=1}^{n}p(D_{i}\mid T_{m})}}$ (remove expectation) (18) $\displaystyle=\frac{\sum_{m=1}^{M}{\mathbb{E}_{\sigma\sim\lambda}\left[p( \sigma\mid T_{m})\right]^{n+1}p(T_{m})}}{\sum_{m=1}^{M}{\mathbb{E}_{\sigma\sim \lambda}\left[p(\sigma\mid T_{m})\right]^{n}p(T_{m})}}$ (identical) (19) $$

∎

###### Proof.

### Appendix B Our choices for baselines

Our inclusion of the power law and the bounded power law stem from their use in Anil et al. (). We note that their justification for fitting a power law to the ICL curve is predicated on (1) the ubiquity of power laws in describing language model behaviour in general, particularly during training;(^3^33See section 2 for some works which equate ICL with gradient descent, which would further solidify this reasoning.) and (2) a few toy derivations which show how the attention mechanism could implement ICL in a way that results in a power law shape for the ICL curve.(^4^44Appendix G of Anil et al. ().)

As for the bounded power law, Anil et al. () propose it in Appendix H.1 of the paper, but do not provide theoretical justification for it as they did for the power law. The key advantage of the bounded power law, they point out, is that “it asymptotes to constant values for both limits $n\to 0$ and $n\to\infty$” (where $n$ is the number of ICL examples).

When reading this justification, we couldn’t help but recall the canonical example of a function the asymptotes in both directions: the logistic function. If we apply a log transform to the input variable, the logistic asymptotes to constant values for $n\to 0$ and $n\to\infty$, just like the bounded power law.

We also note that since laws that asymptote towards both limits (such as the bounded power law, our log-logistic baseline, and our Bayesian scaling laws) are empirically better fits for ICL behaviour on real-world LLMs, the toy model of ICL that Anil et al. () propose must not capture the real mechanism underlying ICL, since it only predicts power law fits (which assymptote only as $n\to\infty$).

#### B.1 Our formulation of the logistic baseline

Interestingly, we found that if we define a function $\mathrm{logistic}(\ln{x})$, we get something almost identical to the bounded power law. Starting with the standard logistic function

$$ $\displaystyle f(x)$ $\displaystyle=\frac{L}{1+e^{-k(x-x_{0})}}+C$ (20) $$

we replace $x:=\log{n}$ and $x_{0}:=\log{n_{0}}$.

$$ $\displaystyle f(x)$ $\displaystyle=\frac{L}{1+e^{-k(\log{n}-\log{n_{0}})}}+C=\frac{L}{1+e^{-k\log{n /n_{0}}}}+C$ (21) $\displaystyle=\frac{L}{1+\left(\frac{n}{n_{0}}\right)^{-k}}+C$ (22) $$

The only difference from the bounded power law is that the $1$ added in the denominator is outside the parentheses for the exponentiation.

### Appendix C Implementation of the scaling laws

Oddly, not all scaling laws papers document how they fit their functional forms. We referred to Hoffmann et al. (); Besiroglu et al. (); Borgeaud () to figure out how to fit ours, which we describe in this section.

We implement our scaling laws and their optimisation routines in PyTorch (Paszke et al., ).

#### C.1 Deriving numerically stable expressions

Our first goal is to use parameterisations that maintain numerical stability. A major (and sometimes only) source of instability is exponentiation, which leads to very large or very small numbers that can exceed the precision of our floating-point representations. We can get rid of exponentiations by computing as much as possible in log space.

In the case of the three non-Bayesian laws, we use the following forms:

$$ $\displaystyle\text{NLL}_{\text{power}}(n)$ $\displaystyle=\exp(C^{*}-\alpha^{+}\ln{n})+K$ (23) $\displaystyle\text{NLL}_{\text{bounded}}(n)$ $\displaystyle=\exp\left(C^{*}-\alpha^{+}\mathrm{LSE}\left(0,{\ln{n}-\ln{n_{c}^ {+}}}\right)\right)+\exp{K^{*}}$ (24) $\displaystyle\text{NLL}_{\text{logistic}}(n)$ $\displaystyle=\exp\left(L^{*}-\mathrm{LSE}\left(0,K^{+}({\ln{n}-\ln{x_{0}^{+}} })\right)\right)+\exp{C^{*}}$ (25) $$

In the notation above, $x^{*}=\ln{x}$ indicates that we store the parameter in log space, and $\text{softplus}(x^{+})=x$ indicates that we apply the softplus activation function to put the parameter in the range $[0,\infty)$.(^5^55Other scaling laws work, such as Hoffmann et al. (), uses $\exp{x^{+}}$ to constrain parameters to be positive, but we found this is less numerically stable for our purposes, particularly for fitting the logistic function.) $\mathrm{LSE}$ indicates torch.logsumexp, which uses a numerically-stable algorithm to compute $\ln\sum_{x}\exp{x}$.(^6^66If we weren’t storing these values in log space, we could have used torch.log1p instead. Unfortunately, storing in log space seems necessary for stability.)

Finally, we still have some failures to converge when fitting; we trace these to some parameter values blowing up, so we use torch.clamp to constrain the log-space parameters to the range $(-20,20)$ and add some checks to ensure no intermediate computations become infinite.

For the Bayesian scaling laws, we derived a numerically stable expression for the negative log-likelihood:

$$ $\displaystyle p_{\text{bayesian}}(n,\lambda)$ $\displaystyle=\frac{\sum_{m=1}^{M}{(P_{\lambda,m})^{Kn+1}\rho_{m}}}{\sum_{m=1} ^{M}{(P_{\lambda,m})^{Kn}\rho_{m}}}$ (26) $\displaystyle\text{NLL}_{\text{bayesian}}(n,\lambda)$ $\displaystyle=-\log{\sum_{m=1}^{M}{(P_{\lambda,m})^{Kn+1}\rho_{m}}}+\log{\sum_ {m=1}^{M}{(P_{\lambda,m})^{Kn}\rho_{m}}}$ (27) $\displaystyle=-\text{LSE}_{m}(P^{*}_{\lambda,m}(Kn+1)+\rho^{*}_{m})+\text{LSE} _{m}(P^{*}_{\lambda,m}Kn+\rho^{*}_{m})$ (28) $$

This not only converges well but also turns out to achieve lower error rates than our original naïve implementation. We store the symbol probabilities $P_{i,j}$ in log-spaced with enforcement to be in the range $(-\infty,0]$ using the softplus activation. For the sampling-wise and scoring-wise variants, we find it appropriate to ensure $\gamma_{i}>\beta_{i}$, so to compute $\beta_{i}$ we sum its underlying parameter with the underlying parameters for $\gamma_{i}$, forcing it to always be smaller. This slightly harms performance but leads to more interpretable fits.

#### C.2 Optimisation

At first, we used Adam (Kingma & Ba, ) with early stopping to optimise our scaling law fits, but this led to noisy results and obviously sub-par scores for some of the scaling laws (particularly the logistic).

We thus followed previous work and switched to the L-BFGS optimiser.(^7^77[https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html](https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html)) We use a history_size of $100$ and $100$ max_iter. We run each optimisation step on the whole dataset for $100$ epochs, and use the strong_wolfe as the line search function. Our loss function is sum of the squared error over the dataset, which we minimise.(^8^88We did consider using the [Huber loss](https://pytorch.org/docs/stable/generated/torch.nn.HuberLoss.html) as in Hoffmann et al. (), but didn’t achieve any noticeable gain.)

We store the state of the model at each optimisation step and, at the end of optimisation, load the parameters that achieved the lowest average loss.

### Appendix D GINC hyperparameters

For the GINC experiments, we report model architecture details in Table 3(a), GINC dataset parameters in Table 3(b), and training hyperparameters for both pretraining and SFT in Table 3(c). We ran each of our GINC experiments on a single NVIDIA RTX 6000 Ada Generation.

**(a) Model config for our gpt2 models.**
| Hyperparameter | Setting |
| --- | --- |
| hidden_size | $768$ |
| max_position_embeddings | $1024$ |
| num_hidden_layers | $[4,8,12]$ |
| num_attention_heads | $12$ |
| vocab_size | $50$ |
| intermediate_size | $3072$ |
| tie_word_embeddings | True |

### Appendix E Real-world LLM details

#### E.1 Models

We experiment on the following models. Unless otherwise indicated, we ran our experiments on locally-hosted models on a single NVIDIA A100 80GB.

**Table 4: LLMs used in this work. $\dagger$: Served through the inference provider [Together AI](https://www.together.ai).**
| Family | Model | Precision | Ctx. |
| --- | --- | --- | --- |
| Gemma | google/gemma-2b-it | bf16 | 4000 |
|  | google/gemma-1.1-2b-it | bf16 | 4000 |
|  | google/gemma-7b-it | bf16 | 4000 |
| Llama 3 | meta-llama/Llama-3.2-1B-Instruct | bf16 | 8000 |
|  | meta-llama/Llama-3.2-3B-Instruct | bf16 | 8000 |
|  | meta-llama/Llama-3.1-8B-Instruct | bf16 | 8000 |
|  | meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo^† | fp8 | 8192 |

#### E.2 Datasets

We compute ICL curves on the following datasets:

- •
CREAK (Onoe et al., ) tests commonsense reasoning using entity knowledge. The model must respond with “true” or “false” given a statement.
- •
Persona (Perez et al., ) evals test whether a model adopts a particular persona by asking personality quiz-type questions with “yes” and “no” responses. We evaluate on the machiavellianism, narcissism, and psychopathy subsets. We jointly model positive and negative personality traits when fitting our Bayesian law, which gives us a prior over both e.g. psychopathic and non-psychopathic responses.
- •
HarmBench (ours) is a new many-shot jailbreaking dataset that we generated, which follows the recipe from Anil et al. (). We take prompts from HarmBench (Mazeika et al., ) which safety-tuned models are supposed to refuse, and generate non-refusal responses to them using WizardLM-13B-Uncensored (Hartford, ), an instruction-following LLM that has not been safety-tuned. We force-decode the prefix of the uncensored response with ‘Here is how’. Instead of using an expensive refusal classifier model, we simply compute the probability of this prefix for our ICL curves.
- •
LogiQA (Liu et al., ) is a multiple-choice machine reading comprehension dataset which requires logical reasoning to answer. We use the prompt format from the EleutherAI Eval Harness (Gao et al., ).

#### E.3 Dataset examples

We present examples from each of the datasets used for real-world ICL evaluations below. Some offensive text is censored for the OpenMSJ dataset.

### Appendix F Detailed evaluation results

Table: Table 5: Pretraining, Interpolation: NRMSE of each scaling law when trained on a full ICL curve, for various pretrained models from our GINC experiments. Bold values indicate minimum NRMSE in that row, without controlling for statistical significance.

Table: Table 6: Pretraining, Extrapolation: NRMSE of each scaling law when extrapolating from the first $n\%$ of the ICL curve (evaluated only on the remainder of the curve), for various pretrained models from our GINC experiments. Bold values indicate minimum NRMSE in that row, without controlling for statistical significance.

Table: Table 7: SFT, Interpolation: NRMSE of each scaling law when trained on a full ICL curve, for various amounts of SFT on various models from our GINC experiments. Bold values indicate minimum NRMSE in that row, without controlling for statistical significance.

Table: Table 8: DPO, Interpolation: NRMSE of each scaling law when trained on a full ICL curve, for various amounts of DPO fine-tuning on various models from our GINC experiments. Bold values indicate minimum NRMSE in that row, without controlling for statistical significance.

Table: Table 9: Real-world LLMs, Interpolation: NRMSE of each scaling law when trained on a full ICL curve, for various datasets and real-world LLMs. Bold values indicate minimum NRMSE in that row, without controlling for statistical significance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: I found the observation that ICL efficiency (K) roughly increases with model depth to be interesting. Do the authors have any insights about the role of model width?
soundness: 3
strengths: 1. This work investigates an interesting problem. Based on some assumptions, it proposes a new theory and validates this theory with experiments. The work also does a good job comparing the proposed scaling law to prior works. 
2. I appreciate that the experiments include both toy settings as well as real world LLMs.
summary: This paper makes progress towards understanding (1) scaling laws for ICL and (2) understanding whether ICL is bayesian. In particular, it presents a scaling law for ICL via Bayesian assumptions, and validates these scaling laws using ICL behavior for small LMs trained on controlled synthetic data as well as LLMs trained on natural language.
weaknesses: The paper primarily focuses on transformers. However, it would be interesting to see some experiments for e.g., state space models, and whether ICL is bayesian in these models as well.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: See my comments on weaknesses
soundness: 2
strengths: Overall, this is a very interesting paper, as the Bayesian scaling law offers insights into phenomena such as many-shot prompt jailbreaking. If the conclusions of this work hold, they could strengthen the argument that ICL operates in a Bayesian manner.
summary: This paper proposes a Bayesian scaling law for in-context learning (ICL). Building on prior work that suggests ICL can be understood through a Bayesian framework, the authors relate the number of in-context examples to the probability of predicting the next example. Essentially, the paper iteratively applies Bayes' theorem to model how ICL updates the task prior as new examples are encountered, ultimately deriving a functional form for the posterior distribution. Experimental results are reported on safety alignment tasks, where many-shot prompts are used to demonstrate the predictive ability of the Bayesian scaling law, including cases where prompts can bypass safeguards. The paper also compares the Bayesian scaling law to a baseline power law model, evaluating performance based on interpolation and extrapolation error, and demonstrates the superior predictive accuracy of the Bayesian approach.
weaknesses: My primary concern lies with the functional form of the posterior expectation in Equation (2). Specifically, in Equation (2), over which random variables is the expectation taken? The notation used here is somewhat unclear to me. Additionally, in Equation (17) of the appendix, the authors appear to apply the linearity of expectation, which seems to assume that E(A/B)=E(A)/E(B). Am I missing something here?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: Questions:

Q1: Line 121/122: should the prior be $\rho$ instead of $p$? In general I see both $p$ and $\rho$ for prior throughout the paper. Consistency would help, if possible.

Q2: Why is the prior not included in $\mathcal{M}$ in Theorem 1?

Q3: In section 3.1 are the words “task” and “distribution” used interchangeably? Here also consistency would help, if possible.

Q4: Why is NRMSE a better choice than KL divergence from the ground truth probabilities? What would you expect if the calculations were done with KL divergence?

Q5: I guess Figure 2b is for interpolation? It would be better to write this down explicitly.

Q6: (Optional) Would you be able to adjust figure 1’s colors so that it's colorblind friendly?

Q7: It would be good to have a plot showing the scaling law fit itself. (See W2)
soundness: 3
strengths: Overall, the paper is well motivated, well written and clear. The experiments are all asking very important questions. Here is a list of strengths:

Contributions:

C1: The result clearly separating SFT’s effect on the prior vs in each distribution is interesting and useful to understand how ICL abilities are affected by SFT. 

C2: SFT being increasingly superficial (prior change only) on larger models is interesting, and suggests an important phenomena to study as we scale models.

C3: In general, developing interpretable scaling law from first principles is a good and useful idea.

C4: Showing that DPO destroys Bayesian ICL behavior unlike SFT clearly demonstrates a qualitative change depending on the choices at the fine-tuning stage.


Presentation:

P1: The paper is very well written, the motivation of the work, the development of the theory and the experiments are very clearly described.

P2: The paper’s narrative makes important points about why this work is needed, e.g. “Interpretable scaling laws”
summary: This paper sets up “Bayesian scaling laws” which are a form of scaling law derived from the assumption that in-context learning is performed as Bayesian posterior calculation. The paper is building on recent works understanding ICL with a Bayesian framework, especially Xie et al, 2022. The paper first sets up the Bayesian posterior given by a prior and likelihood over the different tasks the model will be trained on. Parameter tying and efficiency coefficients are introduced to build up to the final scaling law. First, this scaling law is fit to trained transformers on GINC, the authors claim superiority of this scaling law over 3 baselines. Next, the paper explores the effect of SFT, and finds that 1) SFT only modifies the task prior and 2) SFT is more superficial with model scaling. DPO is explored similarly, and the authors find that DPO breaks the Bayesian ICL behavior. Finally, 6 ICL tasks are tested on real language models and the scaling law fits are analyzed.
weaknesses: Even though the paper is great at a conceptual level, unfortunately, many claims are weakly justified. I think the paper has enough qualitative contributions and does not require a justification of competence by scaling law “benchmarking”. In fact, I don’t think the paper’s contribution is degraded at all even if the fits went worse, as the importance of this paper, for me, is that the authors developed a framework to fit a scaling law and interpret its terms, allowing qualitative insights of how Bayesian ICL is, not that NRMSE got 0.02 better. I truly believe this paper has great potential, but here are the main weaknesses:

W1: Even though Table 1 is the basis to the claim “our scaling laws exceed or match existing scaling laws”, it isn’t clear how robust this claim is. The best method alternates between the different Bayesian methods, and the gap with the best base method is very thin. The weight tying scheme introduced gives too many degrees of freedom to set up a Bayesian method (I’m not saying too many model parameters, but many ways to set up different methods, e.g. low rank P matrix) and it is unclear how much this choice will affect the results. Also about significance: Will the DPO fit result stay insignificant even if the number of sample is 10x?


W2: The scaling curves and the fits are not shown as plots for the GINC experiments, preventing the reader from assessing the goodness of fit by eye. Adding these plots will also contribute to making the better performance claim more robust.


W3: I do not understand how Figure 5 supports “DPO can use very little training data to successfully suppress the disfavoured HMMs beyond the ability of ICL to recover.” Is the critical part that the disfavoured classes fall to 0.0 and not something close to 0.1 ? If so, I don’t get the “very little training data” part. If it turns out that this is my misunderstanding, I will reflect this to the final evaluation.


W4: I am not sure if I agree that these results support that LLMs behave in a Bayesian way, from the goodness of fit I see in Figure 6, I would rather argue that they behave in a very non-Bayesian way, especially the instruct models. Even if it outperformed the base scaling laws for these experiments, I do not see why this supports that LLMs do Bayesian ICL since the goodness of fit doesn’t seem persuasive.

W5: The parameter tying and introduction of the efficiency coefficient K is not well justified. Could the intuition behind the tying scheme be clarified?

W6: The paper makes multiple claims about model scaling. However, are these models trained for the same number of steps or (theoretical) FLOPs? Given pre-training scaling laws and the general observation that bigger models train faster (when measured in gradient steps), is it possible that bigger models are just effectively better trained? I see this as a major concern, as real LLMs are trained near the compute optimal frontier, and if small models here are undertrained, the results might be confounding model scale and optimization. At the very least, I believe the equal number of FLOPs should be given for the small models instead of equal gradient steps.

Most weaknesses seems to be addressable, I am willing to modify my score if most of these are addressed, since as mentioned above, the paper makes great points.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please address the problems raised in the weaknesses part.
soundness: 1
strengths: It is reasonable to interpret context relationships by Bayesian inference. Besides, whether a scaling law holds is crucial to the extendibility of a large model (LM). Hence, it is a good motivation to establish Bayesian-type scaling laws for ICL.
summary: This paper tries to establish several Bayesian-type scaling laws for In-context learning (ICL).
weaknesses: 1.The biggest problem of this paper is the lack of theoretical support, deduction, illustration, and explanation of the scaling law Eq. 3 (Final scaling law). The Bayesian formula Eq. 2 is a normal one and the deduction of Eq. 2 (Appendix A) is a common technique in Bayesian inference. However, this paper lacks evidence in deducing Eq. 3 by Eq. 2, though Eq. 3 is somewhat heuristic. Eq. 3 seems to appear suddenly without sufficient explanations.

2.As for particular implementation, the authors take logarithms for non-Bayesian cases and Bayesian scaling laws in different places, respectively. For example, the logarithm is taken outside the whole likelihood, which leads to a negative log-likelihood in the Bayesian scaling laws. However, the logarithm is taken inside the likelihood, and a 1 is added in the denominator, in the non-Bayesian cases. Moreover, the torch.clamp is also used to constrain the log-space parameters to the range (−20, 20), which may significantly affect the experimental results. 

3.This work lacks analysis of how the error be scaled. Given the scaling law Eq. 3, how would the error be scaled and amplified by the current scheme? It can be seen that the errors of the parameters could be amplified along with K (exponentially), n (exponentially), and M. Even if a logarithm is taken, the error may still be of considerable magnitude. Is there any treatment to suppress or control the error along with such a scaling law?

Considering the above theoretical and experimental reasons, it is difficult to confirm that the proposed Bayesian-type scaling laws really hold in ICL.


**Minor comments**:

There are some incorrect English expressions that should be fixed. For example, “This may be because the compute allocated to...” in Line 497, Page 10.

Meta-review: This paper proposes a Bayesian scaling law for in-context learning. The method is motivated by theory, and the paper conducts empirical evaluations to verify the fitting ability of the proposed scaling law. The experiments are conducted by firstly suppressing certain capability of an LLM through SFT and then using ICL to bring this capability back.

The reviewers in general praised the paper in terms of attempting to derive the proposed scaling law from theory, the insights and interpretations offered by the proposed method, and its potential to strengthen the argument that ICL behaves in a Bayesian way.

However, the reviewers also expressed some important concerns about the proposed scaling law. One major concern is whether the proposed Bayesian scaling law is a good fit to the empirical observations, and whether it is better than the non-Bayesian baselines. This is pointed out by both Reviewer UJig and Reviewer GBRU. Reviewer UJig also had extensive discussions with the authors during rebuttal phase regarding whether the scaling law for different architectures should be equal-steps or equal-FLOPs. After the discussions, the concern of whether the proposed scaling law serves as a good fit still remains.

Another concern is the lack of rigorous theoretical support as pointed out by Reviewer GBRU. This concern is also shared by Reviewer Xv3b, who pointed out an error in the original theoretical derivation. The authors proposed a method to resolve this error by imposing an additional assumption. However, in my opinion, this additional assumption may be too restrictive, because it implies that "each ICL example is identical and thus interchangeable". This may also limit the practicality of the theoretical derivations and hence make the proposed scaling law more heuristic in nature.

Given these concerns, rejection is recommended. The paper has taken an interesting approach and we do see the potential of this work, so we encourage the authors to take into account the reviewer comments to further improve the paper.

Decision: Reject