Learning Shrinks the Hard Tail: Training‑Dependent Inference Scaling in a Solvable Linear Model
Authors: Noam Itzhak Levi
Venue: ICLR 2026 Poster (2026)
Keywords: Scaling Laws
Inference scaling
test time compute
linear models
fine tuning
Primary area: learning theory
TL;DR: Our solvable model reveals that inference scaling is training-dependent, directly linking a model's generalization error to its pass@k performance.

Abstract: We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty. In our Latent Instance Difficulty (LID) model, each input's target variance is governed by a latent ''precision'' drawn from a heavy-tailed distribution. While generalization loss recovers standard scaling laws, our main contribution connects this to inference. The pass@$k$ failure rate exhibits a power-law decay, $k^{-\beta_\mathrm{eff}}$, but the observed exponent $\beta_\mathrm{eff}$ is training-dependent. It grows with sample size $N$ before saturating at an intrinsic limit $\beta$ set by the difficulty distribution's tail. This coupling reveals that learning shrinks the ''hard tail'' of the error distribution: improvements in the model's generalization error steepen the pass@$k$ curve until irreducible target variance dominates. The LID model yields testable, closed-form predictions for this behavior, including a compute-allocation rule that favors training before saturation and inference attempts after. 
We validate these predictions in simulations and in two real‑data proxies: CIFAR‑10H (human‑label variance) and a maths teacher–student distillation task.

Published as a conference paper at ICLR 2026
LEARNING SHRINKS TRAINING-DEPENDEII INFERENCE SCALING IN A SOLVABLE LINEAR MODEL
THE
HARD TAIL:
Noam Levi
École Polytechnique Fédérale de Lausanne (EPFL)
Lausanne, Switzerland noam. wwwiieeevi.cc..c.
ABSTRACT
We analyze neural scaling laws in solvable model of last-layer fine-tuning where targets have intrinsic, instance-heteroogeneuus difficulty. our Latent Instance Dif- ficulty (LID) model, each input's 's target variance is governed by latent "precision" drawn from a heavy-tailed distribution. While generalization loss recovers standard scaling laws, rate exhibits a power-law decay, & 9eir, but the observed exponent Ber is training- intrinsic limit set
main contribution connects this to inference. The pass @ k failure
dependent. It grows with sample size N before saturating by the difficulty distribution's 's tail. This coupling reveals that learning shrinks the "hard tail" of the error distribution: improvements in the model's generalization error steepen the pass@k k curve until irreducible target variance dominates, The LID model yields testable, closed-form predictions for this behavior, including a conpute-allocation rule that favors training before saturation and inference attempts after. We validate these predictions in simulations and in two real-data proxies: CIFAR-101 (human-label variance) and maths teacher-student distillation task.
INTRODUCTION
The remarkable success of large-scale machine learning models tightly linked to empirically observed and theoretically understood scaling laws, which characterize performance improvements with increasing data, model size, or training time (Kaplan et al. |020 Hestness al.) 2017] Rosenfeld et al. 2019 Bahri et al. 2024 Maloney al. [2022] Bordelon et al. (022)). These laws have been crucial for predicting learning curves and optimizing resource allocation in the large dataset size N and number of parameters P regime. Most classical analyses focus the training side, relating generalization loss Lyen to (and sometimes P) through properties of the data distribution such as spectral decay (Bartlett et al.
2020, Hastie et al. 2020 [Brddloo al., 2020) tractable model
classes (Maloney et al. 2022
Tay el al. 2022).
A complementary paradigm emphasizes inference-time compute: repeated attempts, best-of-N, and search can produce substantial gains on difficult reasoning tasks even without further training, typically evaluated with pass & under a verifier (Snell et al., 2024; Brown et al., 2024). While practical methods for exploiting inference-time compute are rapidly evolving (see Sec. [2). and some explanations for their success have been given [Levi] 2024 Schaetfer et al. 2025a), basic theoretical question remains: how does training progress shape performance scaling at inference time?
In many real-world settings the onput-output relationship not deterministie; some instances are intrinsically more variable than others (Arpit et al.) 2017 Northcutt et al.) (021). Such instance-level heterogeneity affects both stages: during training we observe only a single noisy realization per input, and during inference we may compare a model prediction to multiple fresh realizations under pass @ k. This motivates a minimal model that explicitly ties training and inference through instance difficulty.
Setting. We adopt deliberately simple but analyzable framework that mirrors common practice in fine-tuning: last-layer (linear) regression on fixed features in high dimension, with intrinsically stochastic targets. Each instance x carries a latent precision Tx (its "difficulty"), drawn from heavy-tailes distribution, which controls the variance of its target around the mean x 0". Training

---

Published as a conference paper at ICLR 2026
observes realization y ~ Y& per input and fits linear head (ridge/OLS). At inference, evaluate pass @ l with a perfect verifier by drawing k fresh realizations per test input and asking whether at least one lies within fixed tolerance of the model prediction.
Overview and contributions.
A solvable LID model for linear fine-tuning. We formalize the Latent Instance Difficulty (LID) model in the high-dimensional linear setting. Training with single realization per input reduces to ridge/OLS and recovers established generalization scaling with respect to sample size N, dimension d, and spectral exponent (including the L/N tail in the classical regime when the average target variance is finite, i.c., 3 2).
Training-infereet coupling via a two-tail law. We show that the distribution of single- trial success probabilities under pass @ k acquires two regularly varying components: an intrinsic tail determined by the latent difficulty distribution (exponent () and a finite-N tail governed by the model's error relative to the mean target (exponent (NN)
Lgn Averaging over trials yields a mixture pass wass@k k law from which the effective inference exponent Bett (N)
min J3, {(N) emerges. Hence the observed pass @k slope increases with N and salurates at the intrinsic difficulty index J3.
Predictions and implications. The theory predicts (i) crossover surface in (N, k) separat- ing a finite-N (las-dominated) region from an intrinsic-tail region; (ii) a saturating 3 (N) curve; and (iii) continued prefactor improvements with N even after the slope has plateaued. These lead to a simple compute-allocation rule: invest in training until Pa Par(N) (N) near B, then prioritize inference attempts.
Evidence in simulation and real-data proxy. Controlled simulations confirm the 1/N training tail (with the correct intercept), the steepening of pass@k k with N, and saturating (NV. Two real-data proxies exhibit analogous behavior: CIFAR-IOH (human-labe) variance) and a mathematical reasoning teacher-student distillation setting using GSMSK.
Taken together, these results provide clean, testable baseline that unifies training and inference scaling setting that captures widely used fine-tuning regime. The model makes explicit when test-time compute should help, how far its benefits can go, and how those benefits depend on training. In the LID linear fine-tuning setting, standard 1 -type generalization scaling with N coexists with a heavy-tailed distribution of instance difficulty. As N grows, the model's 's average squared bias shrinks and the error mass concentrates on a shrinking set intrinsically hard examples. On the inference side, this shows up as a pass @ k failure curve Lir (k; N) P(N) ( whose slope Ben(N) (N) increases with N and eventually plateaus at the intrinsic difficulty index B. Equivalently, training progressively "shrinks" the hard tail of the error distribution until irreducible instance stochasticity dominates and the test time scaling exponent saturates.
RELATED WORK
Here, we provide a focused review of the prior research most central to our contribution. For an additional detailed literature survey, refer the reader to App. A and references therein.
Generalization Scaling Laws. A large body of work has established that the generalization loss of deep networks often scales as predictable power law with resources like dataset size N or model parameters (Hestness et al.] [017]] Kaplan al. 2020 Hoffmann and et al., 2022). Theoretical frameworks seek to explain these laws by appealing to properties of the data, such as its spectral decay, or the model architecture (Bahri et al. 2021 Maloney et al., (022). Our work builds on the standard scaling of generalization loss with N. which serves as the "training" component of our unified model.
Inference- TiferenceTTime Scaling. A parallel line of work has shown that performance on difficult reasoning tasks can be dramatically improved by increasing compute at inference time, even without further training (Snell et al., 2024; Brown et al. 2024). The dominant methods involve generating multiple candidate solutions and selecting the best one, often evaluated with the pass@1. k metric. While some theoretical models for this phenomenon have been proposed (Levi 2024 Schaeffer et al. 2025a).
they typically analyze inference in isolation.
2

---

Published as a conference paper at ICLR 2026
To our knowledge, simple, solvable model that analytically connects the progress of training (i.e., the decrease in generalization error) to the scaling of inference performance still missing. Our work aims to provide exactly this unified view.
The rest of the paper is organized as follows: In Sec. 3 present the LID model. We provide our main results for training and inference scaling laws in Sec. In Sec. dec.55 we present two examples of linear fine-tuning performed inherently stochastically labeled dataset, showing that the LID is reasonable proxy for some real-world tasks. We conclude in Sec.
THE LATENT INSTANCE DIFFICULTY SETTING
Real-world datasets exhibit significant instance-level heterogeneity: some image labels are ambiguous, incurring higher annotator disagreement (Peterson et al.] 2219 Northcutt et al.) (02T), and some reasoning problems are intrinsically harder than others, leading to more variable outputs. Standard homogeneous-nokke noise assumptions overlook this complexity. Addressing this, and connecting it to the distinct scaling behaviors observed during training versus inference (especially when multiple inference attempts can be verified against correct solution; ef. Section [.. motivates setting. Intuitively, factors that increase training difficulty (harder to learn the mean) also shape inference reliability (harder to match fresh realization).
Last-layer fine-tuning view. We instantiate Latent Instance Difficulty (LID) within the common fine-tuning regime: a frozen representation produces features x E I2d and we learn a linear head xTO. 0. Each instance carries latent precision (its "easiness") Tx that controls the variance of its target around the mean x' 0". Training observes one realization per at inference, pass @ k compares the model's prediction & fresh realizations from the same instance-specifie target distribution.
Definition 3.1 (Latent Instance Difficulty (LID) Model). The data generation process for an observa- tion (x, y) is:
1. Features. An input feature vector E R is drawn from distribution p(x) with zero mean E xx] = 0 and covariance Σ = E [xx " We assume eigenvalues of exhibit power-law decay of j1++a) for 1,...,d. d, with > 0.
2. Latent difficulty. Associated with x is a latent difficulty precision Tx E (0, oo), drawn independently from
Tx Gamma(shape 3/22, rate 1),
where 3 controls the near-zero tail. Smaller increases the mass at very low precision (high intrinsic variance), corresponding to "hard" instances.
3. Stochastic target. Conditional on (x, Tx), the instance target is Gaussian around the mean relationship f*(x) = x 0* with variance inversely proportional to Tx:
Y ~mean = 0" variance o /²//Tx) where o >> is global scale.
4. Training labels. The observed label is single realization
Y equivalently
y = x 0" + n, n ~~ Tx).
Comments. (i) The pawer-law spectrum in Item ] abstracts benign-feature regimes observed in practice and used in linear scaling analyses (Maloney et al. 2022; Levi and Oz 2023 2024). (ii) The Gamma family in Eq. (I) is chosen for analytic convenience; all results that govern inference scaling depend only on the near-zero tail Pr(tx & t) = 23/2, so other distributions with the same tail index yield the same exponents (Levi 2024). (iii) Independence Tx and x simplifies exposition; allowing correlation is natural extension and would primarily affect constants and crossover locations rather than exponents. (iv) We will denote the model's bias relative to the mean target on a fresh test feature by Eyen(x (x) := x T 0 - x xTo 0"; its distribution is governed by the training procedure and sample size N (see Sec. 4
Corollary 3.2. The average variance of the target around its mean is
1
B - xT = Exx (Va((Y x)) o
(4)
Tx
3

---

Published as a conference paper at ICLR 2026
For Tx Gamma(3/2,1) B[1/Tx]
T(0/2-1) T(3/2) B-2 when B > 2.
Assumption 3.3 (Finite average target variance). We assume B[ - xx < co, B 2.
Assumption 3.3 enables standard high-dimensional ridge/OLS analyses of Cgen (train- ing/generalization scaling). Importantly, our inference-time results (pass @k scaling) rely only on the small-r tail and continue hold in the sense of exponents even when S (though constants and the onset of asymptotics may change), provided the learned predictor is consistent.
3.1 TRAINING SETUP
We consider learning the mean relationship f"(x) = x TT 0" from a dataset DN = {x,, B:) where each y; single realization sampled according to Def. 3.1 In the fine-tuning view, x; frozen features from a pretrained backbone and we train only a linear head. The learner observes (Xe, ys) and estimates 0 by minimizing a ridge objective against the realized labels while evaluation always against the mean target:
Irain (
Σ u
0) + 2110
(5)
N
where i E Rd (last-layer parameters) and À 2 0 is small regularizer.
The minimizer admits the standard closed form
0* arg min Lirain (0) (N- -
0
where E Xxd stacks the features and y RV the realized labels from equation We will take the ridgeless limit X ~ when well-posed; in the overparameterized case N < d we use the standard
scaling = X/N to recover the minimum-nowe interpolator (see Sec. 4nd App
Bias relative to the mean target. Throughout, we evaluate generalization against the mean signal 0", and denote the model's instancewise deviation by
Egen (x) x O₂ xT0x x O
The test (generalization) loss is (N = Ex,DN (Egen (x)²]. Under Ass. 3.3 the effective 20
training noise has finite average variance moise - o B[1/Tx]
3-2 so standard high-dimensional
ridge/OLS tools apply. In Sec. 4 we analyze how Cgen scales with N, d. and the feature spectrum exponent and how the resulting distribution of Exen(x) controls the finite-N inference behavior (pass@1 k) and the effective inference exponent Bes (N).
4 THEORETICAL ANALYSIS OF SCALING LAWS
We analyze coupled laws: (i) the dependence of the generalization loss Leen on sample size N, and (ii) the pass@k & inference failure Lir( N) on the number of trials k. In our fine-tuning view (last-layer regression on frozen features), Lgen controls the distribution of the instancewise deviation
(x) T T
which turn governs the finite-N behavior (k; .N.. As N grows, (N) shrinks and the Egen Exen(x) (x)-induced penalty in Lin((k; N) recedes; the observed inference slope transitions from finite- N regime to the latent-difficllyy asymptote -B. We quantify this transition empirically via Bee Bee (N) (Fig. right).
4.1 TRAINING SCALING LAW COEN VS. N
We evaluate generalization against the mean target, so the test loss is
Cgen (N, A) Ex-p(x) Dx (x 0r))
(8)
In high dimensions, Egen depends critically on the N ratio and on the spectrum of Σ; we follow the standard decomposition into regimes (Belkin et al. 2019 Hastie al. 2020). Under Assumption 3.3
4

---

Published as a conference paper at ICLR 2026
Training
Inference (vs
Exponent
10
10-3
4a (9) ret X
2.5
104
Cu0::00
10
3 10
1823
10
1.0
d-22 platesu
10a
X-1101
10a3
N1-11157
X=-70013
10.
DEC Am10
10
10P
10
10
10
104
Figure 1: Training and inference-time scaling laws in the LID setting. Left: Generalization error Ligen N, showing double descent and the classical regimes. Center: Pass@k k inference failure rate Lint (k; N) vs. k for several > d, with asymptotic slope -B (dashed black) once the mean well learned. Right: The effective inference exponent Bert (N) extracted from the local log-log slope in a fixed k-window. The dotted line marks the asymptote 3; the solid curve is the empirical fit Bed (N) = 3 A/(1 + c²) We use X 10 and on 10-3
the effective label noise has finite average variance muiss o B[1/Tx], and classical ridge/OLS results apply (see Aor a concise derivation in notation).
for
overparameterized regime (N < d). With ridgeless (or lightly regularized) interpolation. the error is governed by the minimum-nonne norm bias and the data spectrum exponent a (Bartlett et al. 2020 Mci
O
and Montanari 2020 Hastie et al. 2020 Maloney al. 2022). For power-law spectra 0f x j
one oblains
(N) x N~9 <
where Pv is a benign prefactor encapsulating spectrum and teacher alignment. The symbol x here means that Leen (N) (N-o) remains bounded away from zero and infinity as N - oo along this regime (ef. Section B).
Underparameterizee regime (N d). When samples exceed parameters, the variance term dominates and decays at the parametric rate,
d
Lgen (N)
E
N> d,
(10)
N
with the usual 1/N slope and a noise- coise-controlled constant reflecting that the mean target is learned from single-shot labels. The precise constant can be expressed in terms of the inverse Marchenko- Pastur law when are Gaussian with covariance Σ (App.
Transition (N x d). Near interpolation the estimator is ill-conditioned and Eeen exhibits peak ("double descent"), whose height and width depend on À and on the spectrum (Belkin et al. 2019))
In the overparameterized regime, performance limited by how quickly the model can resolve low-variance spectral modes of the data, leading to the N- tail. Once the model has more samples than parameters, the effective noise from single-shot labels dominates and egen becomes variance- limited, decaying as d/N. This classical behavior provides the backdrop against which we interpret the mhining-dependee
changes in inference scaling.
The left panel of Fig. confirms these scalings in our LID setting and provides the (N) baselines used to interpret the finite-N inference behavior discussed next.
4.2 INFERENCE SCALING LAW LINE VS.
We analyze inference through the pass@ k metric (Snell et al., 2024 Brown et al. 2024), i.e., the probability that at least one of ls independent trials matches the target within a fixed tolerance, In the LID setting, for a test instance x with latent precision Tx,
Y
A(x 0-
T//Tx),
and the model outputs x 0x. On trial we draw y3 Yx, and the error is
(x 0x x =T0) 1 Exe(()) 185
B ~ N(0 /T/xx)
ej = -ー Uj
5

---

Published as a conference paper at ICLR 2026
A trial succeeds if |eyl I ö for a fixed tolerance 6 0. Assumption 4.1 (Perfect verification). We assume perfect verifier that declares overall success for a given iff at least one of the trials satisfies |eş] S
Let p(x, Tx) denote the single-trial failure probability
pxx; Tx) F Pnn- (0--0(0.tt///m/(((c((())) 77) 6). Assuming conditional independence across trials, the pass@ @k k failure probability is
Cinr(k; N) = Ex, xx Dx p(x, Tx)
pass k.
(11)
Remark (model-draw target-draw pass @ k). In practical pass k for Large Language Models (LLMs) one draws model outputs and verifies them against a fixed target; here we draw k target realizations and compare them to fixed predictor ğ x 0x. Under a small tolerance window and smooth local densities, the single-try success probability is proportional to local density evaluated at the prediction gap (x) = Ege((),, Whether the randomness is in the model or in the target then affects only constants, not the asymptotic exponent of the pass@ curve; we formalize this equivalence in App. D.2
Bias-free asymptotic tail. We first consider the asymptotic scaling when the mean is learned well enough that the instancewise bias Bx (x) is negligible compared to o,// VTx for the small-Tx instances that dominate the failures. In that regime, small window expansion in 8 yields
p(x, Tx) c8 v Txx + o(1))
5 0.0,
for constant cs 26/(V2T0,)) on): see App. D for details. A Tauberian analysis of the
Laplace-Stieltjes transform under the Gamma prior Tx Gamma (3/2, then gives the following formal statement.
Proposition 4.2 (Bias-free pass @k scaling). Suppose Assumption 3.3 and the regular-variation and smoothness conditions in App. E.2 hold, and consider the limiting bias-free predictor (or a sequence of predictors for which Bx (x) O in distribution). Then, as k oc,
2T(3)
(k)(k)) = Po. k- 11-(11 o(1),
P8... = r(B/2)
(12)
where cs = 25/ m.))
Thus the asymptotic slope -B is governed entirely by the small-Tx tail of the LID prior and is independent of d/N.
Finite-N correction and training-dependent exponent.
For finite N, the instancewise bias (x) Egen is non-negligible and suppresses the per-trial success probability at moderate Tx- For small 8, Gaussian CDF expansion yields
/2 8 V on Vx exp(
Bx(x) Tx
1 p(x, Tx)
o()))
(13)
20
uniformly for Tx in compact subsets of (0, 20) as 0; see App. E.2 generalization, (x) is approximately Gaussian with E B[Br((x) = and
Under standard linear
[xr[Bx(x)) (Cgon(LVV)
Using equation 13 and Tauberian arguments for Saplace--Stiltjee transforms (detailed in App- E we obtain a two-tail mixture law for (k; N).
Theorem 4.3 (Two-tail mixture law for mass@k). Let the assumptions in App. E. hold, and suppose Bx is centered sub-Gaussian with Var[Bx(x)] x (N) and independent of Tx. Then there exist positive constants P and Px (N) and a function x(N) O such that, k +
(ny(k; N) Pk-8 + PN(NN) k- (NN) (1+
(14)
with
>(N) of
=0( Eyen (N)) )
(15)
Var By (x)

---

Published as a conference paper at ICLR 2026
Empirical (i))te-3.aMM)
Constant-8 baseline (3-2.)
104
10
4
3
logao Cus
CyN
logio
2
optimum analytec opetmom
analytoc approx
10a
10
10a
10
10"
10
10'
itraining-inferrnnce weight)
I2 (trainning:nferrnnee weight)
Figure 2: Compute allocation with training-dependent inference exponent. Left: empirical finite- N case. We plot 1og10 frot (N, k; Rt) with R the training:inference weight ratio and C = Nex + kcg-
Black: grid optimum N,; white dashed: analytic approximation from equation 21 using Bes(N) (N) (local log-log slope Lir) and its discrete derivative. Right: constant-3 baseline calibrated from the same data; white dashed is the closed-fon analytic optimum. Contours clearly shift toward larger N the finite-N panel, consistent with the - (N) log(-) correction.
The first term in equation 14 is the intrinsic LID tail governed by 3; the second term a finite-N correction arising from large bias (x)) at moderate Tx.
Corollary 4,4 (Training-dependenn)) effective exponent). Fix k-window (Ma, kg] such that for each N either the k or the k k-())) term in equation 14 dominates over that window. Then, Kı 0o,
Bell (N; k2)) min (,,((N)) + o(1),
(16)
where Boit N; k2)) is defined in equation 23 In particular, as N increases and Eeen de-
creases, (NN increases and the effective inference exponent (N) monotonically approaches the intrinsic LID exponent B.
The two exponents 3 and ((N) play distinct roles. The intrinsic exponent 3 comes solely from the heavy tail of the instance difficulty distribution and cannot be improved by additional training once the mean target has been learned. By contrast, (NN encodes finite-N penalty: when the model's mean error is still large, many test points behave as if they were "effectively hard" even if their latent Tx moderate. As training drives down Lgen (N), this artificial hard tail disappears, and the observed pass@ @ & curves steepen until Bet((N) is limited only by B.
In practice we summarize this monotone saturation by the empirical fit
A
BesF (N) = B
(17)
C9 N
where (A, C3, v) depend weakly on the k-window and on the estimator. The center and right panels of Fig. illlstrate: for fixed N d, Lint(k; N) exhibits slope that steepens with N; (ii) Ber (N)
increases with N and plateaus at B.
4.3 COMPUTE ALLOCATION TRADEOFF WITH TRAINING-DEPENDERR (es((N
Having characterized the finite-N inference scaling, we can combine training and inference laws to study the optimal budget allocation between the two.
We consider a fixed compute budget C' that must be split between training samples (cost each)
and inference trials (cost C₁ each) Nex + kck, s.t. k weighted objective
C- N = Ncx. We minimize
G4
Liot (Ñ) = R Egen (N) + Lial C-N ; .N).
training/inferenee weight R > 0. (18)
In the classical (under-pp parameterized) regime N > we have (N) x PxN with y (Sec. 4.1 more generally may take 7 (1, a depending on d/N. For inference, Theorem 4.3
7

---

simplifies to Published as a conference paper at ICLR 2026
suggests modeling Lint N) over the practical w-window by
(nr(k; N) =e P(N) (B
(e f as N oe,
(19)
where P(N) is slowly varying prefactor capturing residual bias effects. Here ">/" is to be understood in the sense of an empirical fit over fixed k-window, not as an asymptotic statement; the asymptotic behavior k + oo is given by Theorem 4.3
With Ñ = -Now and = (C - (N)////c. Ck. the budget- objective becomes
Ltor (N) = RPN ex N + P(N) of (M)(N) (C- -
(20)
Proposition 4.5 (Optimal allocation with training-dependent Bott (Na)) Assume (g) x PNN and that over the practical k-window the pass @ failure satisfies (k; N) R P(N)k--aa (at with
Bes(N) differentiable in N. Let N = - /eN. Any interior optimum N. E (0, C*) of Fror (N) satisfies
R. PN c3y 7 N-+++ P(N)) Gk Baal()) (C (C-NN)-BaaNN
(ar(N)
[ar(N) In
C-N
(21)
C-N
CN
Ch
If P(N) is slowly varying and (N)) is small over the relevant range, equation 21 the quasi-static balance
PN CN y N BC-
(22)
where P is an N- N-independent constant and the approximation is understood in the sense that the two sides match up to slowly varying multiplicative factors.
Compared to the constant-3 condition, equation 21 includes a new logarithmic term proportional to -P (N) - c)/e)). When the budget is such that = (C N)/ck lies below the 0), this term increases the marginal benefit training. shifting the optimum towards larger N. Once Bell (N) has saturated (so (N) 2 0), equation 21 reduces the constant-exponent balance in equation 22 (recovering the
LID-dominated window (so (NV) 0 and In((C N)/c²)
classical tradeoff with / replaced by Bert (N)).
In particular:
Under-parameterizee near saturation. With 7 = 1, Fn (N) ~ and slowly varying P(N), equation 22 gives an accurate allocation rule: allocate training until the marginal N-gain x Ñ matches the marginal l--gain x (C (N) - (Pa
Finite-N, sub-asymptotic When Ber (N) is still increasing. the -h log) term makes additional training strictly more valuable than the quasi-static approximation predicts; optimal policies invest more in N until the effective slope stabilizes.
This shift toward larger N in the finite-N N regime is evident Fig.
5 EMPIRICAL RESULTS ON REAL WORLD APPLICATIONS
To bridge the LID model with realistic settings, we perform two experiments: we train linear classi- fier on features extracted from CIFAR- and perform a deacher-student distillation experiment on reasoning tasks using auto-regressive LLMs. We discuss the results of these experiments in the following sections, with additional details provided in Apps. Fand H
5.1 TEST CASE I: LID IN CIFAR-1OH WITH PRE-TRAINES FEATURES
Here, we validate the LID predictions on CIFAR-10 Krizhevsky 2012) paired with human label distributions from the annotated CIFAR- 10H dataset Peterson et al. (211)). freeze pretrained ResNet-18 (He et al.) 2015) and fine-tune only a linear head on top of the backbone features, treating human disagreement
destance-dependee noise (high variance = low precision), matching the LID setting. Full protocol and implementation details are in App.
Fig. (top) shows Egen vs. N and Lin((; N) vs. A The generalization curve exhibits the familiar transition near N d and the 1/N decay for N >
consistent with our linear-ridge analysis. For
Herre 32 and 0.05 following the notations in App.
8

---

Published as a conference paper at ICLR 2026
Traning
CIFAR-
Expreent
1.00
9.2
30
2X1P
0.24
-
0.22
0.20
127
12"
True Class:
Mcan:3 45, Var=16.522
Histogram 2000 Samples
0.0
Probaltty
0.
16
EAD
00
Humaan Labd Var 3
Sample Labe Value
Figure 3: Training and inference scaling on CIFAR-10H (frozen backbone, linear head). Top row: Left: Generalization loss [Ne with the N tail in the classical regime. Center: Pass@k ls failure (k: N) for several N on CIFAR-IOFE dashed shows - reference anchored on the largest-N curve. Right: Effective inference slope Bett (N) estimated from the local log-log slope; the saturating fit Bes (N) = B A/ + cgN²) is overlaid, approaching the intrinsic tail index B. Bottom row: CIFAR-I0 image, its human label distribution, and the Gaussian PDF used to sample training/inferrecce labels, illustrating instance difficulty via label variance.
inference, each fixed-N curve follows an approximate power law k (N)) we estimate 3en (er(N) as the local log-log slope in a central is window. As predicted by Sec. 4.2 Ben (N) increases with N and saturates, reflecting the crossover from a bias-limited window (finiteN mean error) to the intrinsic LID-dominated tail.
The bottom row illustrates how human disagreement induces instance difficulty: broad label his- tograms (large instance noise variance) produce wider Gaussians in equation 48 making pass @ success rarer at small k. This matches the mechanism behind the & law in the synthetic LID model [Sec.[4.2) with the caveat that the empirical difficulty distribution is not exactly Gamma; nevertheless, the slope behavior is robust and governed by the prevalence of high-variance instances.
Takeaways. (i) The finearheaad fine-tuning setting reproduces the Lgen scaling predicted by the ridge analysis. (ii) The pass @ l: failure curves exhibit log-log linearity with slope Ben (N) that grows with training data and plateaus, aligning with the finite-N theory and supporting the compute-allocation results in Sec. 4.3 (iii) Treating human uncertainty instance-dependen noise provides a realistic testbed for LID: the qualitative scaling and the training-dependen inference exponent persist despite deviations from the idealized Gamma prior or independence assumptions between Tx and x.
5.2 TEST CASE II: GSM8K TEACHER-STUDEN DISTILLATION
We complement our CIFAR- 10H experiment with a small GSM8K (Cobbe et al. 2021 distillation task designed to probe the LID training-iiffrrncee
inference coupling. A F Flan-T5-22 XL (Chung al. 2022)
teacher produces solution traces on GSM8K-train; a F -e---a--mmlllll- student is fine-tuned with LoRA (Hu et al.) 2022) (r=8) on subsets of sizes N E [10, 6309] (one teacher sample per question).
We evaluate on GSMSK-test using two metrics: (i) a strict greedy proxy for training loss that decodes single student answer, parses only explicit "final answer" numerics, and reports valid-only MSE (with 1% Winsorization) at tolerance = 10 33 and (ii) the pass @+ k failure Lint (k; N) together with the effective slope Berr (N) estimated from central winddwww.
Fig. 4 shows that the strict greedy loss decreases modestly with A while (ir NN steepens and B increases then saturates, which is precisely the two-tail behavior predicted by the LID model
App. E. Full setup and evaluation details appear in App. H
While the test loss is unstable due to large numerical mismatches (some final answers are simply larger numbers than others), the overall trend decreases and the inference metrics stable.
9

---

Published as a conference paper at ICLR 2026
Training
Inference (pass@k)
Exponent
0.400
1077
0.375
1044
066
C.
0.325
0.300
4x10-1
N=110
0.275
107
+
Egen Ivalid MSSE)
N--51
0.250
Pur000
30 10
10P
101
10'
10'
Figure 4: GSM8K distillation -studentt-tudettt) Left: strict greedy valid-only MSE vs. N (1% Winsorized), with a N-1 reference. Center: Lin(kk; N) vs. & for three N values. Right: effective inference slope een (N), which increases with N and plateaus, consistent with the LID two-tail law.
6 DISCUSSION AND CONCLUSION
This work introduced the Latent Instance Difficulty (LID) model, a simple, solvable framework for last-layer fine-tuning that unifies the scaling laws of training and inference. We modeled tasks with intrinsic, hetrrggeeots difficulty and showed that while the generalization loss, Zyen= follows established scaling with sample size N and data spectrum a, its improvement has direct and non-trioe coiitrivial impact on inference performance.
Our central contribution is the derivation of a training-dependenn inference exponent. The pass @ failure rate, (Anr(k), decays power law, but its exponent, Bett (NV), is not fixed. It begins small for poorly-traing models and grows with the number of training samples .N, eventually saturating at an intrinsic limit, 3, determined by the tail of the task's 's true difficulty distribution. This mechanism reveals how reducing the model's 's error relative to the mean target makes inference-ting compute more effective, up point of diminishing returns set by the data's 's irreducible stochasticity.
This unified view yields actionable insights. It predicts clear crossover in the optimal resource allocation strategy: when the model is undertrained and Be (N) < |3, the marginal benefit of acquiring more training data is high. Once the model well-trained and the inference exponent has saturated, further gains are best sought by investing in more inference- compute.
The LID model, while simple, provides a valuable theoretical baseline. It cleanly separates the roles of data structure (ai and data heterogeneity (3) while also explaining how they are coupled through the training process. By providing a glosed-form. testable theory for when and how much test-time compute should help, offers a first step toward a more principled understanding of resource allocation in modern machine learning.
Although we instantiate LID in last-layer fine-tuning with scalar regression, the core mechanism: pass@ & failure mixes an intrinsic difficulty tail (index () with a truining- wruning-dependent tail controlled by fgen( (N), (N), is model-agnostic. Beyond fine-tuning, training from scratch can be analyzed by allowing the representation to evolve, effectively changing the feature spectrum (c.g., the decay exponent a) and thus the rate at which the finite-N tail collapses; the saturation level may then be centteccuur-deeendeee
For classification, instance difficulty can be identified with label ambigu- ity/noise, and pass @ k is the event that at least one of & draws hits the correct class, yielding the same two-tail mixture and saturating 3art (N). ). In reinforcement learning, difficulty corresponds to return variance, so pass @ k is the chance that le rollouts contain a near-optimal return; again the LID predictions carry over. For LLM reasoning, increasing pre- pre-training tokens reduces mean eтror and empirically steepens the pass@A & slope; we conjecture Benl (N) grows with scale and saturates an intrinsic difficulty index, with the same observable signature: as training improves, pass @ k curves steepen and then plateau.
Limitations. We analyze linear last-layer model for clarity; extending the theory to multi-layer, non-linear architectures is an important next step, with random-feature kernel models natural bridge. We also treat the intrinsic difficulty distribution as task-fixed; for complex LLM reasoning, stronger models may both reduce bias and effectively "simplify" the task, shifting the apparent tail index 3. Thus our results should be read as describing, for a fixed architecture, how Ben (N) rises toward ceehittcturrddependee open. Finally, we study single-output regression; carrying the LID perspective on ceininng-ifferncee coupling to fully structured, auto-regressive outputs (as in L.Ms)) is key challenge for future work.
plateau; quantifying how that plateau varies across models remains
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENTS
We thank Yohai Bar-Sinai, Alon Beck, Francesco Cagnetta, Joshua Kazdan, Nadav Outmezguine and Zohar Ringel for fruitful discussions. NL is supported by the EPFL AlAscience/Al Center program.
REFERENCES
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 1712.00471 2017.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. constructive predicctoon of the generalization enror across scales, 2019. URL
12673
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy Sciences, 121(27). June 2024. ISSN 1091-6490. doi: 10. 10.1073/pnas.231 1878121. URL http: doi www.doo....oo.e..eoooeeeeeeoeeee 23118 1511878121
Alexander Maloney, Daniel Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210. 2022.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws, 2024. URL
org/ abs 22..2242.00000 01092
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117738))300633300000 3000303000700, April 2020. ISSN 1091-6490. doi: 10.1073/pnas.
1907378117. (9.1077//nas.19077778111 URL http: /ttp:://dxx.e dww. wtt.:/odooorrg/0.000000000
pnas. 190737817
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-
dimensional ridgeless least squares interpolation, 2020. URL httttt/:///aaa/ httttp///aaxi..oo/////aaa
1903. 08560]
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning. pages 1024-1034, PMLR, 2020.
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws model architectures: How does inductive bias influence scaling?, 2022. URL https:t :ttppis////aiv...///./ org wtwggb///////////// 2207 10551
Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar, Scaling 1lm test compute optimally can be more effective than scaling model parameters, 2024. httl::tttp/::xx.... org abs/
22488.08 03314
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL hoLhttps: http::://axii..rrrrrrr
/2007 217877
Noam Levi. A simple model of inference scaling laws, 2024. URL 2410.16377
Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo. How do large language monkeys get their power (laws)?, 2025a. URL URLphttps. //tp://aaxvvvvv/ www.awwwww...aaaaaaaaaa abs /2502. 17578
11

---

Published as a conference paper at ICLR 2026
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks, 2017. URL http:p///axxxv..oo////o 1706. 05394
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks, 2021. httpa:/ org/ abgggabb/213...../ 2173.14749
Jordan Hoffmann and et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint urXiv:2102.06701, 2021.
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust, 2019. URL httts::///xxxii. wttp:/i.wwg///b/////// 07086
Noam Levi and Yaron Oz. The universal statistical structure and scaling laws of chaos and turbulence, 2023.
/ 01358
Noam Levi and Yaron Oz. The underlying scaling laws and universal statistical structure of complex datasets, 2024. httpsstps://ariiivooo 2235.////3////3.33../////
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, (16632):15549-1555) (Rt.::tp://bb..d...rr//...00 doi 1tt.0////.1111//////1/
July 2019. ISSN 1091-6490. doi: 10.1073/pnas.. 1903070111
1073/ pnas. 903070116
Song Mei and Andrea Montanari. The generalization error of random features regression: Pre- cise asymptotics and double descent curve, 2020. URL htttps://axxxy http://axxxw..../aaaaaa
05355
Alex Krizhevsky.
Learning multiple layers of features from tiny images. Uni versity of Toronto, 05 2012.
URL https::/ww..cs toronto. edu/ -kriz/
features learning -eeature-eettres Settuuess-000--0 9809--RR- TR. pd
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogmition, 2015.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. CRL/attps:// /tattpps//axxi..... arxiv. org/
abs/2110..1 211...111..111
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny language models, 2022. URL
Zhou, Quoc V. Le, and Jason Wei. Scaling
https://arxiln org/ abs 2210.111111 11416
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamín Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020. URL https://arxtt wwwp:/wwwwwww/////////oooos 14701
12

---

Published as a conference paper at ICLR 2026
Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9);1-34, 34, 2022. URL http:// wwt.:operg/pp/./....
20- w1w1.11t..mmm.......
S Spigler, M Geiger, d' Ascoli, Sagun, G Biroli, and M Wyart. A jamming transition from under- to aver-parametriization affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical, (2(44) October 2019. ISSN 1751-8121. doi: (0.1888/151-88 1511-781111///bb33 URL http: do. .org www.rr/g1..1111//111 1008/108111551 81/1////3311112
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, 2023.
URL wtt.thtss///aiii.ccooo
14891
Prectum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Llya Sutskever, Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12): (12):124000, 124003, 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206. /4486, 2022.
Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't it be lazy: Completep enables compute-cfficient deep transformers, 2025. URL https::/aaxxx..cet
01618
Danny Hernandez. Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URLttts::
org/ aas//ab///// wtwp////.............ee.eae.aeee 01293
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law, 2024. URL https:: arwiv. org/ abs 240..240...02222 02314
Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christic- Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp. Rewarding chatbots for real-world engagement with millions of users, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li. Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL httpp:://axxi.../rtrs/e/ abs 2501.19399
Joshua Kazdan, Rylan Schaeffer, Youssef Allouah, Colin Sullivan, Kyssen Yu, Noam Levi, and Sanmi Koyejo. Efficient prediction of @ k scaling in large language models, 2025. URL
Rylan Schaeffer, Noam Levi, Brando Miranda, and Sanmi Koyejo. Pretraining scaling laws for generative evaluations of language models, 2025b. URL
24012
Feng Chen, Allan Raventos, Nan Cheng, Surya Ganguli, and Shaul Druckmann. Rethinking fine- tuning when scaling test-time compute: Limiting confidence improves mathematical reasoning, 2025. /ttt:///tpps//aaaa....oo//////b/
Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling, 2025. URL httts:::: hnttps://arxiv.. //tpx//aaxvvo///////oo abs 220025511111 1125011111 116.1116111
Kuang-Hon Huei Lee, Ian Fischer, Yueh-Huza Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. Evolving deeper Ilm thinking, 2025. URL
abs/
2501 2501.0989 09891
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. Stream search (sos): Learning to search in language, 2024. URL BRLhttps
rxiiv wwwii.oorg/bbs/22400 03683
13

---

Published as a conference paper at ICLR 2026
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024. URL httts:://aaxiit/ org/ abs abs/2408.... 00724]
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don't throw away your value model! generating more preferable text with value- guided monte-carlo tree search decoding, 2024. URL httts:://axiii wttp:/ivwwrgg/ba/oa//oooooo 15028
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. Planning with large language models for code generation, 2023. /
abs/2303 05510
Andy Zhou, Kai Yan, Michal Shlapentokh-Rottnng Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024. URL https:
//axxiv.rgg/ab/// /2300 04406
Sehyun Choi, Tianqing Fang, Zhaowei Wang, and Yangqiu Song. Kets: Knowledge-constrainee tree search decoding with token-level hallucination detection, 2023. URL htLlhtttps///axlv.orr
abs/ 09044
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min- Yen Kan, Junxian He, and Qizhe Xic. Self-evaluation guided beam search for reasoning, 2023. URL httpss://aax..../g/// org//bs/
Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee. Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
pettps://arivy org/ abs 2305. 20050
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations, 2024a.
URL https::/aaxii..oot
abs/ 22t..2212... 08935
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024b. URL hnttps: /ttpss///axxvv...o
wttp:oo///a///o/oeee http:////aax.o.o////////000 08673
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in Ilms through large-scale synthetic data, 2024. /ttl:sttts///axxiv.oo http:://waaiiv.ooraa////////////
14333
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, and Prithviraj Ammanabrolu. Critique-out-ooud reward models, 2024. URL htttps://arxiv..rrrr.. abs/ 0ttpb///4..1199999 11791
Nikolaos Tsilivis, Eran Malach, Karen Ullrich, and Julia Kempe. How www.forcenneettteeeeeeee learning after next-token prediction facilitates learning, 2025. URL /tRl:ttcss///axi..rr/////////
11495
Jack W Silverstein and Z. D. Bai. On the empirical distribution of eigenvalues of class of large dimensional random matrices. Journal of Multivariate Analysis, 54(2):175-199, 1995.
Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning. Cambridge University Press, 2022. doi: 10. (0.1017/9981109918899.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Generalization Scaling Laws Neural scaling laws (NSL) have been shown accurately describe how the generalization loss of deep neural networks improves with scale (model/dataset size, com- pute) (Hestness al.; 2017 Kaplan et al. 2020 Hoffmann and et al. 2022 Language model cross-entropy loss, for instance, often exhibits power-law scaling over orders of magnitude (Kaplan ct al. 2020 Henighan et al., 2020), suggesting quantifiable performance gains with increased re- sources. Theoretical frameworks aim to explain these observations by identifying distinct scaling regimes (e.g.. limite,, resolution-limited) (Bahri et al.) 202T] [Sharma and Kaplan 2022] SSii- gler et al. 2019), sometimes linking them to data manifold intrinsic dimension (Sharma and Kaplan 2022] Bahri et al. (021). More nuanced models address dynamical scaling evolution with training time [Bordelon et al. (2024) and "broken" neural scaling laws (BNSL) that capture transitions between power-law regimes (Caballero et al., 2023 Nakkiran et al. (021). Such breaks highlight complexities in extrapolation and suggest plateaus might stem from suboptimal strategies rather than fundamental limits (Sorscher et al. 2022; Dey et al. 2025). Beyond pre-training, scaling laws also govern fine- tuning performance re relative to pre-trained model size and fine-tuning data volume (Hernandez et al., 2021 Lin et al. 2024). These studies show pre-training effectively augments fine-tuning data, with transfer benefits following predictable, though task-dependent, patterns (Hernandez et al., (02TT) shifting focus towards leveraging pre-training for diverse downstream applications. In this work, we mainly focus on the vanilla NSL setting where the scaling law pertains generalization loss improvement with increased number of training samples.
Inference Time Scaling
Methods for scaling inference-ting compute in deep learning often involve generating multiple solution candidates and then selecting the best one based on specific criteria. These criteria include choosing the most frequent response for majority voting or the best response based on external
reward for Best-of-N [Brwwn et al. 2024
Irvine et al./ 2023 Levi 2024; Muennighoff et al. 2025
Schaeffer et al. 2225a Kazdan et al. 2025 Schaeffer et al. 2025b Chen et al. 2025). Unlike repeated
sampling. previous sequential scaling methods let the model generate solution attempts sequentially.
building upon previous attempts and allowing it refine each attempt based on prior outcomes (Snell
cet al. 2024; Hou et al. 2025 Lee et al. (025). Tree-based search methods (Gandhi et al. 2024
Wu et 2024) offer hybrid approach between sequential and parallel scaling. Examples include
Monte-Carlo Tree Search (MCTS) (Liu et al. 20244] /Zang et al., 2023] /hoou et al. 2024 Choi et al.
2023) and guided beam search (Xic et al. 2023). REBASE (Wu al. 2022) employs a process reward model to balance exploitation and pruning during its tree search. Empirically, REBASE has
been shown to outperform sampling- based methods and MCTS (Wu et al. 2024). Reward models
play key role in these inference-time scaling methods (Lightman et al. [023] Wang et al. [2024a] by
Wu et al. 2024 Gandhi et al.] 2024 Liu et al. [2224] /hang et al. 2023] Zhou et al. | [2224] Choi et al. 2023 Xie et al. 2023 Xin et al. 2024 Ankner et al. 2024). They generally come in two variants:
outcome reward models and process reward models. Outcome reward models (Xin et al. 2024
Ankner et al., 2024) assign score to complete solutions and are particularly useful in oest-of-N selection. In contrast, process reward models (Lightman et al., 2023 Wang et al. 2024a Wu ef al. 2024) assess individual reasoning steps and are effective in guiding
based search methods. The
effect of reinforcement learning has recently been analyzed by Tsilivis et al. (202). Other approaches
also explore simple test-time scaling techniques (Muennighoff et al. 2025
B NOTATION AND ASYMPTOTIC CONVENTIONS
We briefly summarize the main notation and asymptotic conventions used here and in the main body of the paper.
Metrics. The generalization (test) loss against the mean target is
Egen (N, A) Dx (xx 0₂ x x
@k failure probability is The @
(in N) == Pl at least & trials all fail under the pass@ k: protocol).
so that pass @ k- (inr(k; N).
15

---

Published as a conference paper at ICLR 2026
Asymptotic notation. For sequences or functions and depending variable t (typically t k or tr N), we write:
f(t) ~ g(t) as t + oo if lim f(t)/g(t) = 1.
f(t) = O(g(t) as 0o if there exists C > and to such that [f(t) - Clg(t)) for all
t2 to-
f(t) (g(()) as if f(t) O(g(t)) and g(t) O(f(t).
f(t) x g(t) if there exist constants 0 < c S < co such that S f(t)/((t) - c2 for all
in the regime under consideration.
We use o(1 for quantities that tend to in the relevant limit (which will always be explicitly indicated, e.g. h + or + 0). When we write statements such as
f(k) = C' C
(k 20);
the dependence of the o(1) term on other parameters (c.g. N, 8) will be clear from context or specified explicitly.
We reserve the symbol for asymptotic equality in the sense above and do not use the informal symbol "z"" in our main statements. Informal approximations are always tied either to an explicit limit (e.g. k + co, 5 0) or to a pointer to precise asymptotic statement in the appendices.
Local log-log slopes. For fixed k-window [k,, k2] and fixed N, define the empirical effective inference exponent as the local log-log slope
log Lint (k22; N) log Line (k1; N1;N)) N)
Bes N; [K1, ka))=
(23)
log 12 log k1
In experiments we take [k1, Aa]] to be a central window where the log-log curve for Lid N) is approximately linear; the precise window does not affect the qualitative behavior of 3 (NV.
C HIGH DIMENSIONAL RIDGE REGRESSION WITH NOISY LABELS
In this appendix, re-derive and analyze the generalization error for high- high-dimensional Ridge linear regression with additive label noise used in the main text. While these results are well-established in the literature (see, e.g., Maloney al. (2022) Hastie et al. (2020) and references therein), present a concise derivation to highlight the connection to our Latent Instance Difficulty (LID) model and to set the stage for understanding the training scaling laws discussed in Section 4.1
C.1 MODEL SETUP
We consider a linear model where the learner aims to estimate true underlying parameter vector 0r RRd from N training samples (x;, 3/). The input features x; E Rd are drawn i.i.d. from distribution with zero mean and covariance matrix Σ E E[xx generated according to
T. The observed labels yı are
y: x +7,-
(24)
where is the label noise for sample i. In the context of our LID model (Definition 3.1 is
realization from (0, o (xx) ). For the present derivation, we assume т are i.i.. with E6 [m]]
and E[n]] = 2 < oc. we were directly applying LID,, o would be replaced by E[a [x] = o [1/x]]
The learner estimates 0" using Ridge regression by minimizing the loss
N
L( = 2N
Σ (ys - 0)2² 1110
(25)
where À 2 0 the regularization parameter. Let X E Nxd xd be the matrix of training features (rows x.) and y E TRA the vector of observed labels. The solution is
1 0x = ( x x + -La)
N X
(26)
16

---

Published as a conference paper at ICLR 2026
C.2 GENERALIZATION ERROR
The generalization error (or test loss) measures the expected squared prediction unseen test data Xtest with true mean target xes! 0":
geaa (N, 2) DDN Jees (x Ō2
X 0")
(27)
Let A0 0₂ 0r. Then Lgen E Σ00), where the expectation is over the training data DN- Substituting y X0 + (where is the vector of noise realizations 17,) into equation 26
we obtain
θ^ (N N x + ALa) -1 X (X0" +
N
( +
(
1 N x ( +
-1
Thus the error vector is
A 02 0r
N x -NXTX+ ALd)
-1 (1 1T. X0* + N xTTT 7/ 0"
NN
- ALa)
X Tx 20
(28)
N
Assuming 0" is fixed (not random) and E/n
0, EEl E|pr
mosse IN.
E || = - - x + AI.)
20
and
Cov(A0) (
E (
x 11) ( +
=
02
nnise
X X
X X ALa) Aaa)
(29)
N2
Hence
Lyen = E|A0]
EE A0 + Tr E Cor(A
(S+AL xT
Σ E + Mll
+ 2 Th
E + ALa) - 'ESS + AIa) 11a) 1)
(30)
where = X X is the empirical covariance.
C.2.1 UNDERPARAMETTETTAA REGIME (N >> d)
In the underparameterrized limit N >> d with À + 0, the bias term vanishes and obtain
Lgen
0 noise Tr (EÉE-))
(31)
In the setting discussed in the main text, the population covariance can be written as Σ A, where A is diagonal with decaying power-law spectrum as in equation [ As stated in (Silverstein and Bai
(1955): Couillet and Liao (2022), the empirical covariance matrix can be expressed as Σ = AW where W Wishart matrix with aspect ratio A = d/N. Then
Egen 2 Tr Tr(W-').
(32)
The eigenvalues of W follow the inverse Paacchenko-aastte law (Couillet and Liao 2022 which implies
E Tr(W-1))
for N > 1
(33)
d
17

---

Published as a conference paper at ICLR 2026
Thus, in the high-sample limit N/d oc,
d
ggen (N)
noise N -N 00, N >> d),
(34)
which recovers the d/N tail used in equation 10 up constant factor.
Near the interpolation threshold N x d, one obtains the familiar peak in Lgon that forms side of "double descent" (Belkin al. 2019).
C.2.2 OVERPARAMETERIZZE REGIME (N <
In the overparameterized regime, the interpolation solution is obtained by taking vanishing ridge parameter with the scaling À X/N (Bahri al. 2021). Under this scaling, the test loss decomposes as
X2 T
L
f
+
11 +
}1,1) 0r - - (11))
N2
2 noise
Tr
La)
(35)
The first term (the bias) can be analyzed via replica random matrix methods, leading self- consistent equations of the form
X2
B
(36)
N2
2
(=1
where {o?} are the eigenvalues of E: see Bordelon al. (0200) for detailed derivation. For power-law spectra of i1(+)), solving equation 36 yields the scaling
(go x -
(N < d),
up multiplicative constants depending on the spectrum and on teacher alignment (Bartlett et al. 2020 Hastic et al. 2020). The noise term in the overparameterized regime behaves similarly to the underparameterizzed case with the roles of d and N exchanged, and contributes primarily near the interpolation threshold, completing the double-descent picture (Maloney et al. 2022).
In the context of the LID model, o is replaced by [1/x] Thus, Assumption 3.3 (B > 2) is crucial for these scalings to hold with finite prefactors: if 8 < 2, B[1/Tx] diverges and the standard Ridge regression analysis must modified.
D INFERENCE SCALING AT FIXED N
We provide the detailed steps for the asymptotic evaluation of the pass @ @ failure probability integral given in Eq II for large k
(k) = E Gaammaaamm3///2111 (3/2.11
[[p(x, [x]]*] Rs BE Gammaaaaaa/ -eekk
(37)
Here, = 226((V/TT,, and p(x, Tx) ss C5 for small Tx. The PDF of Gammaa833 (3/2, is ((T) 1(3/2) 8/2-1e - The expectation integral is
(in((k) = ( Ko
3/2-1e-
(38)
T(3/2)
For large k the factor ke decays extremely rapidly for any 7 0, forcing the dominant contribution to the integral to come from the region near = 0. In this region, the term e- in the Gamma PDF is approximately Thus, we approximate the integral as
(k) ~
T(3/2) /
-kea Akca/F_911 -ko// dr.
(39)
18

---

Published as a conference paper at ICLR 2026
We perform change of variable: Let kcsvT. Then v u/cks,, which implies (u/ (kcs))² = u² [kcc)) The differential is dr equation 39
2u (Mes)a
du. Substituting these into the integral
J e- u1
8/2-1
du { e (kca))
2u
2u du (kcs) 8-2 2 (kes))
ub-2
(kec))
jo
2
du
(
3-
of
2
u e du.
(kcs) )a
The remaining integral the definition of the Gamma function, ( which converges for B >
u 8 e
du -T)
(40)
Substituting this back into the expression for Cknr(k)
2T(3)
(3) (c3/2)(cs))
(k) R T(3/2) (kcB)
k-
(41)
This confirms the asymptotic scaling Lint k-8 for large k.
D.1 ASSUMPTIONS FOR THE TAUBERIAN STEP
We use standard Laplace-Stieltjes Tauberian arguments under the following mild conditions:
1. Regularly varying difficulty near zero. The latent precision satisfies Pr(t= S t) L()) J 0, with L slowly varying and bounded on (0, to).
2. Uniform sinllwwindow expansion. For C5 [To, nui C co),
12/ and any compact
s(B,T) : Pr (IB 1
26 = cs V exp
B²g
(1 (1))
20
uniformly in r [To, Tui 4 O. with ca 22 8/0, and rj N(O,00//T)
3. Model error. (x) = x (Ø)- 8*) is centered Lub-Gaussian with Var[Bx] = Eggen (N) and is independent of Ta (or weakly dependent so that conditioning on T₂ preserves sub- Gaussian tails).
4. Independent trials and perfect verification. Conditional on (z, Tx) and the training set, the & comparisons are i.i.d., and success is declared if any trial lies within the tolerance.
Under (1)-(4) Karamata-type Tauberian theorems yield the mixture law in equation 14 and the -ー8 tail in equation 12 with constants as stated.
D.2 EQUIVALENCE MODEL-DRAW PASS k.
Consider the alternative protocol that draws is i.i.d. model proposals ú, = - + E with proposal noise & having bounded, smooth density fe independent of r., and verifies against fixed target y*. Writing (x m(x), the per-trial success probability
J (x( - u) du = 26 (x(()x)) (1+)(1)) ()). 14/58
(42)
In our target-draw protocol the corresponding quantity is 28 f. (BN(T);T)) with fn(:TT))
(0 o./TT2). Thus both protocols reduce to (1 (1-2) p)* with 28 times a local density BN (x); since the only heavy tail in LID comes from (- T2) TTx Ta 0, the small-success behavior, and hence the - tail, is unchanged by swapping model target sampling (up to prefactors). This equivalence holds provided fe is bounded and does not itself introduce a -dependeent tail, and trials conditionally independent.
19

---

Published as a conference paper at ICLR 2026
E
TRAINING-DEPENDENT INFERENCE SCALING IN THE LID LINEAR MODEL
We provide a more detailed derivation of the two-tail mixture law (Theorem 4.3) and the training- dependent effective exponent (Corollary 4.4).
E.1
SETUP
Recall the LID setting:
Features E Rd are drawn from (0, Σ) with eigenvalues of x j j-(1o))
The teacher is linear: m(x) T
Each instance has latent precision 7x (13/2,1), independent of
The stochastic target is Yg (m((x)), oT//))
Training observes i.i.d. pairs (x), with Yx, and fits ridge/OLS to obtain (x from N samples.
At test time we evaluate
the training loss ( [(x 02 - x 0-)3].
the inference loss (in k; N) under perfect verifier with tolerance > 0,
where for each test x we draw ls i.i.d. ~ Y and declare success minjge xx TØ UU S
E.2 ASSUMPTIONS FOR THE TAUBERIAN ANALYSIS
We standard Laplace-Stiellies Tauberian arguments under the following conditions: Assumption E.1 (Difficulty tail and small-window success expansion).
1. Near-zero tail of difficulty. The latent precision has regularly varying CDF
Pr(tx < t) = L() as 1 0,
with L slowly varying and 3 > 0. (For (B322 one has L(t) 1/T(3/2+ 1).)
2. Small-window success form. For tolerance and any compact [To, nul C (0, oc). there exists
2 8
C5
V
on
such that
= =B 7/ 5 Tx) = cx expl
(1+0(1))
(43)
20
as J. O, uniformly in Tx E [No. Au]. where ~ N'O o2 Txx.
Assumption E.2 (Model error distribution). The prediction bias B (x) = x T [Ō2 - -0") satisfies:
[x) 0,
(x) is sub-Gaussian with Var[Bx(x) x Egen (N),
(x) independent of Tx (or weakly dependent so that conditioning on Tx preserves sub-Gaussiaa tails).
Assumption E. is mild regular- variation regular-variation and smoothness condition. Assumption E.2 holds for the
linear Ridge setting with Gaussian features and power-law law covariance (App. C. where By (x) is approximately Gaussian with variance scaling Lgen (N)-
20

---

Published as a conference paper at ICLR 2026
E.3 TWO-TAIL LAW FOR SINGLE-TRY SUCCESS PROBABILITIES
Define the single-try success probability
(x, Tx) == p(x, P (x)) 17| S 5 Tx),
and let Sy denote the random variable obtained by sampling (x, Tx) from the test distribution. We first characterize the small-Sx tail of distribution.
Proposition E.3 (Two-tail law for Sx). Under Assumptions E. E.I-2.2
E.2
there exist constants A >
((N) > 0 and function (NN > O such that, as $ O,
P(SN S = As + B(N) 8 + o(1))
(44)
with
>(N Of Var BN (x)
(45)
geen (N)
and
2 8
8 == T(3/2+11 C6"
C8
π on
Sketch. For small Tx, Bx is negligible in equation 43 and SN c Since Pr(tx S t9/2/T + 1) a 4 0, we have
P(SN <S) ] Px ~ r(3/2+ 1) ()
cB )
giving the As term.
For moderate Tx (1), small Sy arises from large BB (x)). Inverting equation 43 for fixed Tx shows that Sx < corresponds to [Bx(x)] 2 (20 log(es////)/ Since Bx(x) is Sab-Gaussian with variance Var (x)), standard tail bounds imply
PSS < Tx) x ((xx)/
up to slowly varying factors, where c(Tx) is bounded away from zero and infinity on Tx E [No, nul Integrating over Tx in this moderate range yields the second term B(N)s>(N) with 7(N) x (x)). Combining the contributions gives equation 44 see, c.g., Tauberian theorems for Saplac--Stieltje transforms in regular-variation settings.
E.4 MIXTURE LAW FOR @
Recall that, under our independence assumptions,
(k N) = E[(1 - [x]1-Nx)]]] $.)
This is Saplace-Stieltjes transform evaluated at k. Applying standard Tauberian theory for such transforms with tail form equation 44 yields:
Corollary E.4 (Mixture law for Cint N)). Under the assumptions of Proposition E.3 there exist
constants C₁ > O and C2(N) > O such that, as lo + 0o,
(ing(;; N) = C1 - C2(N) k ((N) + +o()),
(46)
with (NN as in equation 45 and
2T(3)
2 8
C
T(3/2)
C/
C$
π on
Corollary E.4 is the more detailed version of Theorem
4.3 in the main text.
21

---

Published as a conference paper at ICLR 2026
E.5 EFFECTIVE LOG-LOG SLOPE
Let (N; [K1, k2l) be the local log-log slope defined in equation 23
Substituting equation 46 and laking ki, 42 + oo with k2 / fixed, one sees that whichever term in equation 46 dominates over [ki, Bey determines the slope. This yields:
effective exponent, precise form). Under the conditions of Corollary E.4 fix a window [ki, ki) such that for each N either the k-9 term the k-(()) term dominates over this window. Then, as k1 + OC,
Corollary E.5 CTrining-dependee
(en [k1, ( min [,,)) o(1).
In particular. if (N) decreases monotonically with N, then 7(N) increases and Bell (N) is monotonically non-decreasing in N and converges to 3.
Corollary E.5 is the formal counterpart of Corollary 4.4 in the main text and justifies the empirical
saturation equation 17
E.6 SUMMARY
Summary: finite-N correction and training-depende exponent
Single-trial success (small window). For tolerance 8 and bias (x
1 p(x, Tx) = Tx exp (xxx)² Tx (1 ((1),
cs = / 6
20
on
Two-tail mixture. As + oc.
Linr(k; Cik CCNN)
((N) θ ZpalaN)
with C1 given explicitly above.
Effective slope. any fixed w-window where single tail dominates,
Aet (N) min (in) + o(1) ku oc,
and hence 3.r(N is an increasing function of N that saturates at 3 as the model becomes well trained.
22

---

Published as a conference paper at ICLR 2026
F CIFAR-10H 10H LID: SETUP AND EVALUATION DETAILS
F.I EXPERIMENTAL. SETUP
Feature extraction. We pass each image through ResNet-18 pretrained on ImageNet and extract the penultimate-laye vector z; E R with d = 512. These frozen features play the role of x in our LID analysis, and we fit only the linear head.
Stochastic Targets from Human Labels. The CIFAR-I0H dataset provides, for each image, the distribution of labels assigned by multiple human annotators. For each image x; (mapped to feature
2,), we calculate the mean m, Ee C- P(label c|x;) and variance V; = =E=o P(label = c|x;) of these human label distributions. The key connection to our LID model is made by treating the target for our linear regressor intrinsically stochastic. In order to control the signal to noise ratio, we introduce scaling factor to account for the magnitude of the noise variance, which controllable in the LID setting. The learning problem is then defined on rescaled quantities
m,,².
Rescaled features: z scaled = 2,/s,
Rescaled mean target: m
sealed = m, /s.
orig
(47)
During training, for each rescaled feature vector z, distribution centered at its rescaled mean, using the human variance
zicaled the target label is sampled from Gaussian
yi ~ A (mean- = m sealed variance = 1/ + C.).
(48)
where mi scaled is the rescaled mean human label for the instance, vs the variance of the human labels, and is small constant (e.g., 10-6) to ensure non-zero variance. Here, plays role analogous to 1/ lxx, in our synthetic L.D model, with high human label variance corresponding a "difficult" instance (low effective precision).
Model and Training, We perform fine tuning by training a linear regression model g = scaled,T to predict scalar value from the rescaled features z scaled extracted from the pretrained ResNet. Given the quadratic loss and linear model, we compute the optimal weights Ô analytically using the Ridge regression solution with an effectively scaled regularization parameter Act = A/s²
0^ -N-'ZZ
Z scaled Zwatted Amn -
N 'ZT N
scaledy;
(49)
where Zecated is the matrix of rescaled training features, and is the vector of sampled noisy labels (from Eq. (48)).
Evaluation metrics. Generalization loss Cyen- On held-out set we compute the MSE between Dusl = 2 scaled and the rescaled human mean micalldd 5².)
scaled, (To compare across different multiply by
Inference failure Lun(k; For each validation point we draw h ii.d. realizations Y) N N(maaled Uval 6n) and declare success if Vval yj ōaef for any j, with dott 5/s. We average the all-fail indicator over the set. This implements the perfect-veriffcatioon assumption from Sec. 4.2
23

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EXAMPLES FROM CIFAR-10H
Here, provide additional examples from the CIFAR-IOH dataset, to illustrate the type of stochastic labels inherent in the data.
Index 132
Human Probs
ship
Mean=8.02
Sampling PDF N/8.02, 0.02)
Histogram Samples
1.0
0.8
Probabiity
Density
Dereity
Sampled Label Value
Sampled Label Value
Index: 7652
Human Probs Mean=0.02. Var=0.017
True Class: plane
Sampling PDF N/0.02, 0.02)
Histogram of 2000 Samples
Probabiity
0.6
Demsity
Deneity M
horee
Sampled Label Value
Sampled Label Value
Index: 762
Human Probs Mean=5.80, Var=0.520
True Class: frog
Sampling PDF N(5.80, 0.52)
Histogram of 2000 Samples
Mean
0.6
Probabiity
0.6
Dansty
Densty
0.4
0.4
Sampled
Index:
Human Mean=5.08, Var=15.994
True Class: car
Sampling PDF N(5.06, 15.99)
Histogram 2000 Samples
1.0
0.10
Vene
0.08
Probabiltyy
Derelty
0.06
0.4
0.04
0.2
0.02
0.0
lnok
10
0.00
10 Sampled Labal Value
Sampled Label Value
Index: 2594
Human Probs Maan=3. 3.98, V9r=19.353
True Class: plane
Sampling PDF N(3.98, 19.35)
Histogram 2000 Samples
1.0
0.10
0.8
0.08
0.08
Probabiity
0.6
Denstty
0.06
Denaly
0.06
0.4
0.04
0.02
torge
0010
Inoo
0.00
0.00
Sampled Label Value
Sampled Label Value
Figure 5: Examples for low and high variance images from CIFAR-1OH. Top to bottom: low variance samples are easier to predict (more localized near the average prediction) while high variance samples are difficult.
24

---

Published as a conference paper at ICLR 2026
H GSM8K TEACHER-STUDEE DISTILLATION: SETUP AND EVALUATION DETAILS
Data and models. Teacher: Flan T5-XL. Student: T5-n-T5-malll with LoRA (rank r-8, a=16,, dropout 0.05) trained for one epoch, batch size learning rate 5 10-5. We sample one teacher solution per training question and fine-tune on subsets N E {10,... 6309) (log-spaced). Evaluation uses the standard GSM8K test split.
Strict greedy proxy for training loss. For each test question we decode a single student output and parse a numeric only when it appears in an explicit "Final answer:" slot. Let VN be the set of test items with valid parse. With tolerance 8 10-3 we report
(greedy) (N)
Σ (i(z) WEVN
VV
Winsorized by trimming the top 1% of I y"l to reduce the influence rare formatting/scale outliers. (Coverage /Vvl/ttst] is tracked but not plotted in the main text.)
Inference metrics. We compute Liar(k; N) numeric verifier with the same tolerance 8. The effective slope 3e (NN is the local log-log slope of fmr(;; N) over a fixed central ls window (cf. $4.2p.
pass @ using saved student samples and perfect
Interpretation and link to LID. Distillation uses one noisy teacher sample per training input, so improvements can first appear as increased mass on good answers rather than large shift in the point estimate. Consequently, (i; N) is more sensitive than a strict pass @ I-style greedy metric. As N grows, the finite-N bias component shrinks, the (inr(k; N) curves steepen, and Bon (N) approaches the intrinsic difficulty index 3 (Thcorem) 4.3)
Limitations. (i) The greedy metric intentionally strict (explicit finals only), which undercounts improvements not emitted in canonical form; therefore compute valid-only MSE. (ii) Using one teacher sample per training item does not directly target the teacher's 's mean; training-side MSE gains modest but consistent with the theory's emphasis on inference-time behavior. (iii) Prefactors and saturation N depend on LoRA capacity and teacher/student choice; the qualitative coupling of training and inference exponents is robust.
25


Reviews:
[{"review_id": "c7cxaExogj", "reviewer": "Reviewer_UZyf", "summary": "The paper derives a training-dependent inference exponent and shows that the pass@k failure rate, decays as a power law, with an exponent that is small for poorly-trained models and grows with the number of training samples N, eventually saturating at an intrinsic limit determined by the tail of the task’s true difficulty distribution. This implies that when the model is undertrained, the marginal benefit of\nacquiring more training data is high. Once the model is well-trained and the inference exponent has saturated, further gains are best sought by investing in more inference-time compute.", "questions": "- Can you add experiments with LLMs to confirm the conclusions? I saw that you mentioned \"For complex reasoning, a more powerful model might not only learn the mean better but also fundamentally simplify the problem, an effect our current model does not capture\". Does it mean that you expect the conclusions to not hold in practice for LLMs? or you still believe the conclusions would hold? Confirming this in practice would add a big value to the paper. \n\nps: while experiments with LLMs are not cheap, fine-tuning smaller models is still doable in most academic labs.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Investigating the effect of the training data and its size on pass@k is an important and timely topic\n- The conclusions are backed up with theoretical analysis\n- The analysis yield concrete empirical guidance for training vs test-time compute", "weaknesses": "- The discussion of the analysis is deferred to the last page of the paper (and very briefly in page 7), this makes it hard to follow the derivatives and the conclusions. I suggest the authors discuss the conclusions after every derivation to clarify the messages throughout the paper. Besides, I strongly suggest the authors to revise the introduction and discuss the conclusions as it is done in the last page. This is very important for conveying the message to the readers, currently all the important messages are hidden in notations and equations. As someone who feels comfortable reading math, I had a hard time to understand the \"importance\" of the conclusions even after reading the intro for a couple of times.\n\n- I liked the paper and its message and I have no problem with analysis in the simplified linear setting. However, to confirm the validity of and importance the conclusions in practice (after all, pass@k is mainly used for LLMs), why don't the authors add experiments with LLMs? This is probably the largest weakness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qMme2RGMoK", "reviewer": "Reviewer_QTqr", "summary": "This paper investigates the inference-time neural scaling laws specifically for the pass@$k$ metric. The analysis is grounded in a simple theoretical framework called the Latent Instance Difficulty (LID) model. Using this framework, the authors show that pass@$k$ follows a power law, where the exponent is shown to depend on both the generalization error of the trained model and the difficulty of the instance.", "questions": "* Could the authors elaborate on how these findings can be stated in a more rigorous manner? (See 'Weaknesses')\n* Could the authors discuss on how the LID model might be applied in broader scenarios?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* This work provides a valuable theoretical analysis of inference-time neural scaling laws. A key contribution is establishing a novel connection, demonstrating how the scaling exponent is influenced by training-time dynamics.\n* The paper introduces the Latent Instance Difficulty (LID) model, a simple yet valuable theoretical framework. This model is a notable contribution in its own right and shows potential for broader applicability beyond the immediate scope of this work.\n* The theoretical contributions are well-supported by empirical validation on the Real-World CIFAR-10H dataset. This effectively bridges the gap between the proposed LID framework and real-world phenomena, demonstrating the practical relevance of the findings.", "weaknesses": "My main concern regards the presentation and rigor of the theoretical results. The current draft often discusses findings in a narrative, line-by-line fashion rather than consolidating them into complete, formal theorem statements. This fragmented presentation makes the paper's theoretical contributions difficult to follow and hard to assess.\n\nIn addition, the frequent usage of the approximation symbol ($\\approx$) throughout the theoretical components, without a formal definition of the approximation level (e.g., in terms of asymptotic notation or explicit error bounds), makes the theoretical results feel incomplete and lack rigor.\n\nConsolidating the key results (e.g., bounds, scaling exponents) into clearly stated theorems or propositions, along with rigorous definitions for all approximations used, would significantly improve the paper's clarity and trustworthiness.\n\nI am open to increase my score if the presentation of this work can be improved during rebuttal period.\n\n### Minors\n* Please use LaTeX-style double quotes `` '' instead of \" \" (e.g., line 15, 20, 89).\n* The metric 'pass@k' should be formatted in math mode pass@$k$ (e.g., line 16, 21).\n* In line 134,137, 243, the authors use $Y_x^* \\sim N(...)$ but $Y_x^* = N(...)$ would be more precise.\n* It appears that every block equation has an equation number, even when not referenced. Please consider removing unnecessary equation numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKG2mBO9SJ", "reviewer": "Reviewer_KXZD", "summary": "This paper proposes the LID model, a simple analytical framework linking training-time and inference-time scaling laws. It shows that as training data scales, inference performance (pass@k) follows a steeper power-law w.r.t. k, with the effective exponent increasing and saturating at a limit—capturing how learning “shrinks the hard tail” of errors. The authors further derive a compute-allocation rule balancing training and inference effort and validate the theory through experiments on CIFAR-10H experiments.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces the LID model — a simple yet analytically tractable setup that connects training-time generalization scaling and inference-time scaling laws. This formalization is elegant and fills a clear theoretical gap between two rapidly developing empirical phenomena: training scaling laws and inference-time compute scaling.\n\n- The derived theory provides clear insights into how the scale of training data affects inference scaling by effectively shrinking the hard tails of the error distribution. These are valuable conceptual contributions.\n\n- The compute-allocation analysis (Section 4.3) offers interpretable results on when to invest in training versus inference compute, providing a practical takeaway.\n\n- The paper also presents solid empirical results on CIFAR-10H that support the theoretical conclusions.\n\n- The framework and analysis open up promising directions for future work, offering valuable foundations for studying the interaction between training-time scaling and test-time scaling.", "weaknesses": "The experiments are conducted on CIFAR-10, while inference-time scaling is a more relevant and currently critical topic for language models. It would be nice if some experiments could be conducted in that setting to better connect the work to practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)