Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Authors: Dongwoo Lee
Dong Bok Lee
Steven Adriaensen
Juho Lee
Sung Ju Hwang
Frank Hutter
Seon Joo Kim
Hae Beom Lee
Venue: ICML 2025 poster (2025)
Keywords: Neural Scaling Laws
Bayesian Inference
Prior-data Fitted Networks
Primary area: deep_learning->foundation_models
TL;DR: We propose a novel Bayesian neural scaling law extrapolation method based on Prior-data Fitted Networks.

Abstract: Scaling has been a major driver of recent advancements in deep learning. Numerous empirical studies have found that scaling laws often follow the power-law and proposed several variants of power-law functions to predict the scaling behavior at larger scales. However, existing methods mostly rely on point estimation and do not quantify uncertainty, which is crucial for real-world applications involving decision-making problems such as determining the expected performance improvements achievable by investing additional computational resources. In this work, we explore a Bayesian framework based on Prior-data Fitted Networks (PFNs) for neural scaling law extrapolation. Specifically, we design a prior distribution that enables the sampling of infinitely many synthetic functions resembling real-world neural scaling laws, allowing our PFN to meta-learn the extrapolation. We validate the effectiveness of our approach on real-world neural scaling laws, comparing it against both the existing point estimation methods and Bayesian approaches. Our method demonstrates superior performance, particularly in data-limited scenarios such as Bayesian active learning, underscoring its potential for reliable, uncertainty-aware extrapolation in practical applications.

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Dongwoo Lee
Dong Bok Lee" 22 Steven Adriaensen 3 Juho Lee? Sung Ju Hwang?* 24 Seon Joo Kim
Frank Hutter?
Hae Beom Lee²
Abstract
Dubey et al., 2024) have largely been driven by scaling. Numerous studies have rigorously explored neural scaling laws (Hestness et al., 2017; Johnson et al., 2018; Rosenfeld et al., 2019; Kaplan al., 2020; Rosenfeld, 2021; Ghor- bani et al., 2021; Bansal al., 2022; Hoffmann et al., 2022; Chung et al., 2024), finding that performance metrics, such as error rate or loss, frequently follow the power law func- tion, as key factors such as size of data and model or training duration increase. The problem is that scaling up those key factors for modern neural architectures requires high costs (e.g,, collecting more labeled data or consuming more train- ing resources for larger models), and it would be beneficial to foresee the behavior of neural scaling laws from observa- tions at small-scale, before scaling up.
Scaling has been a major driver of recent ad- vancements in deep learning. Numerous empirical studies have found that scaling laws often follow the power-law and proposed several variants of power-law functions to predict the scaling behav- ior at larger scales. However, existing methods mostly rely on point estimation and do not quan- tify uncertainty, which is crucial for real-world applications involving bacision-making problems such as determining the expected performance improvements achievable by investing additional computational resources. In this work, we ex- plore a Bayesian framework based on Prior-data Fitted Networks (PFNs) for neural scaling law extrapolation. Specifically, we design prior dis- tribution that enables the sampling of infinitely many synthetic functions resembling real-world neural scaling laws, allowing our PFN to meta- learn the extrapolation. We validate the effec- tiveness of our approach real-world neural scaling laws, comparing it against both the ex- isting point estimation methods and Bayesian ap- proaches. Our method demonstrates superior per- formance, particularly in data-limited scenarios such as Bayesian active learning, underscoring its potential for reliable, uncertainty-awer trapolation in practical applications. The code and models are available at https: :/tp:s::::/iiie //github. com/DongMoolee- Eli/nslpfn.
Fortunately, recent studies (Hoffmann et al., 2022; Bansal et al., 2022; Alabdulmohsin et al., 2022; Caballero et al., 2022) have shown that the neural scaling laws can be pre- dicted empirically in modern neural architectures with care- fully crafted functional forms. As typically done in regres- sion, after predefining reasonable functional form, the parameters of the function can be estimated with maximum likelihood given the partial observations in a neural scaling law. After the estimation, extrapolation is done to predict the future behavior of cach curve. Such neural scaling law extrapolation not only provides the understanding of mod- ern neural architectures (Abnar al., 2021; Bansal et al., 2022; Bahri et al., 2024), but is also applicable to range of fields; such as determining the proper amount of samples in data-scarce domains, e.g., medicine (Mukherjee et al., 2003; Figueroa et al., 2012; Beleites et al., 2013; 3; Cho et al., 2015), enabling the carly-stopping of the hyperparameter optimiza- tion (Domhan al., 2015: Hestness al., 2017; Johnson et al., 2018), or neural architecture search (Hestness et al., 2017; Elsken et al., 2019; Klein et al., 2022).
ex-
1. Introduction
Recent advancements in both the vision (Tan, 2019; Alexey, 2020; Kolesnikov et al., 2020; Dai et al., 2021; Zhai al., 2022) and natural language processing (NLP) (Devlin, 2018: Brown, 2020; Achiam et al., 2023; Touvron al., 2023a;b;
However, despite the recent success on the neural scaling law extrapolation (Hoffmann et al., 2022; Bansal et al., 2022; Alabdulmohsin et al., 2022; Caballero et al., 2022), the ex- isting works are inherently limited as they do not consider estimating uncertainty. Considering that the true potential of neural scaling law extrapolation comes from utilizing to make important decisions such whether to spend compu- tational costs further or not, naively relying on simple point estimation is too risky, as depicted in Fig. 1. Therefore, in
Equal contribution 'Yonsei University "KAIST Univerrsity of Freiburg 'DeepAuto Korea University. Correspondence to: Hae Beom Lee <hacbeomlee @ korc
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
this paper, we stress the need for a Bayesian approach to capture uncertainty and quantify the reliability of neural scal- ing law extrapolation. However, typical Bayesian methods, like directly applying Markov Chain Monte Carlo (MCMC) on existing functional forms, struggle to handle chaotic behaviors- as non-monotonicity-oo
context
target
BNSL
Te-tt .6
NSL (ours)
are prevalent in real-world neural scaling laws (Nakkiran et al., 2021). This is because not only are these optimization landscapes non-convex multi-modal (Caballero et al., 2022) but also setting a practical prior distribution that can cover these chaotic behaviors is challenging.
50
100
125
Model Width
Figure 1: Extrapolation results of each method on neural scaling law from the double descent dataset (Nakkiran et al., 2021). Here, the context and target denote the observations and the target points we want to correctly estimate, respectively.
To overcome this challenge, we investigate the potentials of Prior-data Fitted Networks (PFNs; Müller et al., 2021) for neural scaling law extrapolation. PFNs are an in-context approximate Bayesian inference method, which is meta- learned with the training data sampled from the prior distri- bution. PFNs provide several benefits over MCMC. Specif- ically, it is very flexible in defining any complex prior dis- tributions as long as we can efficiently sample from them. Further, the speed of inference is faster by multiple orders of magnitudes since PFN is an in-context inference method. We empirically validate the efficacy and efficiency of our approach, which we name Neural Scaling Law PFN (NSL-PEN), on an extensive set of datasets by comparing it not only with the recent point estimation methods but also their MCMC variants. We also show that our NSL-PEN reli- ably predicts chaotic behaviors of real-world neural scaling laws even with a few observations at a small scale.
2021; Ghorbani et al., 2021; Bansal al., 2022; Hoffmann et al., 2022; Chung et al., 2024). Let y denote the perfor- mance such as prediction error or cross-entropy, and the quantity that is being scaled such as the size of dataset or the number of parameters in the neural network. The power law function in its simplest form is given by:
M1 y=
(1)
M2 y y=oz +c
(2)
where a, b, c 2 0 are the coefficients we want to estimate. The only difference between M₁ and M2 is whether there is bias cor not for capturing saturating performance (Mukher- jee et al., 2003; Figueroa et al., 2012: Domhan et al., 2015; Hestness et al., 2017: Johnson al., 2018; Rosenfeld et al., 2019; Gordon et al., 2021). For neural machine translation, Bansal et al. (2022) propose slightly more expressive form by adding a shifting term d 2 0 to M1:
We summarize the contribution of our paper as follows:
To our knowledge, we introduce Bayesian method for extrapolating neural scaling laws for the first time.
Ms y a(x + +d)
(3)
We propose novel prior distribution for PFN which is specifically tailored to neural scaling law extrapolation.
Note that M3 reduces to Mi when d = 0.
We empirically validate that our method provides better point estimate fits with the ability to automatically infer the best functional form and the number of breaks.
M4. However, those MI, M2, and Ma functions do not have any inflection points, which frequently appear in many scaling law examples-initiall the scaling law curve is flat, i.e., close to the initial performance you then gradually starts following power law function. Alabdulmohsin et al. (2022) propose the following function family to express such inflection points:
We empirically show that our method provides better un- certainty than the non-specialized PFNs and other MCMC baselines, on many real-world neural scaling law exam- ples as well as Bayesian active learning settings.
M4 y g (x), where
2. Background and Related Work
+
(4)
I = g(y)
y
2.1. Point Estimation of Neural Scaling Laws
a(yo - y)
M1, and Mg. Most existing methods for estimat- ing neural scaling laws rely on point estimation. The power law function is the most representative function used for extrapolation and is known to work well for a wide set of neural scaling laws, including many vision and natural lan- guage processing tasks (Hestness et al., 2017; Johnson et al., 2018; Rosenfeld et al., 2019; Kaplan al., 2020; Rosenfeld,
where a, b, a 2 0 and yo 2 0. Similarly to Mss this M1 function reduces to M2 when a = 0 (since we have y = ax -b + c). Therefore, the role of a is to control the degree to which the scaling law initially sticks to yo-
Broken Neural Scaling Laws (BNSL). While M4 is more flexible than M1, M2, and M3. it still has a limita- tion that the function defined as an inverse the function
2

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
should be bijective. However, some real-world scaling law examples exhibit more complicated behaviors such dou-
posterior predictive distribution (PPD). Learning such a transformer is done with meta-learning framework. The meta-training data is generated from a prior distribution as synthetic data, allowing for a potentially infinite supply of examples. The data is then split into a context (training) and a target (test) set, and the transformer model is trained to maximize the likelihood of the target given the context:
ble descent (Nakkiran et al., 2021)-the curve initially de- creases, then suddenly increases, and then decreases again eventually following a power law function. Thus, Caballero et al. (2022) propose the following function family:
n BNSL: y c+ or II
1/8,
-b
1
(5)
f ~ p()); = (D (DIS),
(6)
(7)
Eq. (5) is nothing but the smooth piece-wise concatena- tion of power law functions, where n is the number of breaks, making the corresponding curve consists of n + segments smoothly concatenated sequentially. a, c 2 0 are the power law function coefficients at the initial seg- ment, Ab....,Abb, E IR are the changes in slope in the log-y space, , en > O are the location of the breaks to be estimated, and $1....,5p Sp > control the smooth- ness of the transitions between two consecutive segments. Note that the j-th segment is increasing function when b + E Ab; O. The number of breaks is selected with cross-validation since it is non-differentiable.
max Ec,T~((D) log (X"I',,))))
(8)
where f is a function, p(f) and p(P].) are the functional prior and the likelihood for generating the synthetic data D = {( 9:) M respectively. D is then randomly split into the context C = (X, = ={) and the target T = YY) = {( y:) amortized inference machine such as a Transformer param- eterized by 0. Note that Eq. (8) equivalent to minimizing the following expected KL divergence:
M+N M+1 go is the in-context
mjn Epce, KL/)("' (X)
(9)
Limitation of the point estimation methods. Although M4 (Alabdulmohsin et al., 2022) and BNSL (Caballero et al., 2022) show reasonable prediction performances on a wide set of neural scaling laws, their study also implies that those extrapolations are inherently uncertain. Let the con- text denote the observations. For instance, when the context part of the curve is flat, it can either imply that the curve is already at the saturating stage of the power law function, or before the inflection point and will sharply decrease fur- ther, following the intuition of M4. Also, when the context part of the curve is given with its last segment increasing, the BNSL function family cannot reflect the empirical fact that in most cases the curve will eventually start decreasing at some uncertain future point, but will predict that the curve will monotonically increase indefinitely (Fig. 1). Therefore, the natural question is how we can incorporate those uncer- tainties into the neural scaling law extrapolation to obtain better prediction performance as well as reasonable uncer- tainties that could be used later for many decision- problems, e.g., to trade off the cost of collecting and training on more data and the believed associated performance gains.
where pt p(Y*IX"; = Blcrccl p ,(Y*IX*; is the true posterior predictive distribution (PPD) marginalized over the true functional posterior pl p(f(C). In this way, the learning objective of PFN in Eq. (8) minimizes the discrepancy be- tween qo and the true PPD, in expectation over the sampling distribution p(C, X*) Consequently, after meta-training, given any context C, qo can learn to recover the true PPD at any target points x E X', as long as a sufficiently large architecture and enough training time are given.
Flexible prior. The main strength of PFNs is that we only need to sample the synthetic data from p(D). This implies that the functional prior p(f) does not need to be expressed in a tractable form at all since we do not need to measure its density at both the training and evaluation phases. This property different from the conventional Bayesian inference methods such Monte-Carlo Markov Chain (MCMC) and variational inference, allowing us to remove any restrictions p(f) as long we can efficiently sample from it.
PFNs for learning curve extrapolation. Adriaensen et al. (2023) use PFNs for extrapolating learning curves. Here, p(f) is functional prior implicitly defined over the parame- ters for describing the learning curves, such as the parame- ters of the power law function and the coefficients for lin- early combining set of different function families. While their motivation for using PFNs is similar to ours, their prior is not tailored to extrapolating the neural scaling law, making it suboptimal for our purposes, as demonstrated empirically in $4. In the next section, we introduce our novel way of
2.2. Prior-data Fitted Networks
In this paper, we propose using Prior-data Fitted Net- works (PFNs; Müller et al., 2021), recently introduced in-context Bayesian prediction method based on the Trans- former architecture (Vaswani, 2017). Unlike the conven- tional Bayesian inference methods that attempt to approx- imate the posterior distribution of some latent (e.g., the network parameters or the function itself), PFNs directly learn a single transformer that maps a context set to the
3

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Table 1: The template distributions for each scaling law segment.
Name
Direction
Function
Prior
M3
Down ())
y a(x= d)s
log ~ (V
log(-b) ~ ()
logd ~ ((0,
M4
Down ())
a s(v) = a11-9)77
log ~ 0.)) log(-b) b) ~ V0
log ~ (00,0.5)
BetaCDF
Up (1)
y BetaCD2(xr; B.
B ~ ((01,,1))) log ~ (100011
Norm
y + V(Nmux Musin) + Ymmi
~ ((.222 1.2)
Mmin ~ 24(, Yuux)
Noise
y 4
log (-4,11
1.0
1.0
1.0
0.8
0.8
0.6
0.4
0.2
0.0
0.8
100
(a) Down.
(b) Down-Down.
(c) Down-Up-Down.
Figure 2: Visualization of 20 neural scaling laws sampled from each function family before applying the Noise function in Table 1.
defining the functional prior suitable for neural scaling law extrapolation.
D (()) We define Ms. Mr, BeE Norm, and Noise in Table I; each be applied to every segment. For the downward segments, we choose only M3 and M4, as Mi and M2 are subsets of them. We use BetaCDD as upward segments because its S-shape fits well to our purpose. We use Norm to re-normalize each segment using the sampled maximum and minimum values. We use Noise function to apply the observational noise in the likelihood function. We set the parameters and yo in Eq. (4) 0 and for M1, respectively, the sampled curves are re-normalized using Norm
We defer more discussion on the related work to Ap- pendix A due to the space constraint.
3. Approach
3.1. Neural Scaling Law Prior
Criteria for the functional prior. Our functional prior should give high density on the reasonable scaling law be- haviors such as power law functions, but also should be flexible enough to cover all the functional forms in the real- world examples. Specifically, we propose the following criteria for the functional prior over the neural scaling laws:
Based on those basic template distributions, we define the following function families:
1. Down:
This function family considers simple down- ward trends without any breaks, using either M3 or M4, Specifically, we randomly choose between M3 and M1, sample the parameters, re-normalize each curve with Norm, and add noise with Noise.
We make use of the previously developed functional forms such as M3 or M4, but for flexibility we sample from both of the function families.
2. Based on the observation of BNSL (Caballero et al., 2022), we allow breaks to randomly occur in each scal- ing law example with some probability, and randomly sample the corresponding segments from the diverse functional families. We highlight that this allows the trained PFN to adapt to various scaling curves with arbi- trary breaks, and automatically infer the appropriate number of breaks without any validation process.
2. Down-Down: This function family also represents down- ward trends but more complex behaviors with breaks. We first randomly choose the number of breaks in {1, 2,3) (resulting in 2, 3, or 4 segments), and randomly sample the locations of those breaks along the r-axis. We then randomly select either M3 or M1 for each segment, sample the parameters, re-normalize each segment with Norm, and add noise with Noise Each segment is translated along the y-axis to align with the last y value of the previous segment.
In order to express chaotic behaviors such as double de- scent (Nakkiran et al., 2021), we let some segments exhibit upward trends with some probability, with BetaCDF, i.e., the cumulative distribution function of the beta distribution.
3. Down-Up-Downy
This function family is to ex- press more complicated behaviors such as double de- scent (Nakkiran et al., 2021). Here, we fix the number of breaks (resulting in 3 segments) and randomly sample their locations along the x-axis. We use either M3 M, for the first and third segments, while the
Defining the function families. We now introduce the template distributions for sampling a complete scaling curve

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
upward function Bet&CD is used for the second seg- ment. After sampling the parameters, we re-normalize each segment with Norm, add noise with Noise, and translate the segments along the y-axis to match the last y value of each preceding segment.
where the second term in the expectation is the auto- regressive objective. We find that adding it improves the curve fit around the cutoff position, both in terms of accu- racy and the quality of uncertainty, as shown in Table 6.
Interpolation loss. Note that Eq. (10) is primarily for ex- trapolation (i.e., T always follows C) and, therefore, lacks the ability to interpolate within each curve. On the other hand, in real-world applications such as Bayesian active learning (Gal et al., 2017) (in $4.2), improving the qual- ity of prediction by collecting additional data is often the main interest, but they require the model to be equipped with a good interpolation mechanism. To this end, we also consider training variant of NSL.-PEN that learns both interpolation and extrapolation at the same time. Simply, we first sample the context C and target T following $3.1, and then randomly sample a subset of T to add them to C. We then use the same objective in Eq. (10) for training.
Fig. 2 visualizes the curves sampled from the three function families, before applying the Noise function. We reject any curves including negative y values or NaN.
Defining the cutoff distribution. After sampling com- plete scaling curve DD we need to decide a cutoff position M to split D into the context C = ((,,, 3 and the target T = {x ) for training. However, in defining the cutoff position, we need to think carefully about what kinds of extrapolation problems our model can somehow solve in reliable manner, and what other types of problems we need to consider as inherently difficult.
Suppose the model is given the context part of the curve, i.e. up to just before the cutoff, and the model needs to predict the rest, i.e,, after the cutoff. We can consider the two cases. 1. If the last part of the curve is downward, we had better assume that a similar tendency will be maintained in the future, e.g., the same power law function. This assumption comes from the intuition (and our empirical observation) that in this case, assuming unexpected future breaks are in- herently too difficult to predict due to the lack of information (because almost anything could happen), leading to exces- sive uncertainties with poor predictive performances. On the other hand, 2. if the last part of the curve is upward, we already know empirically that the curve will eventually turn downward at some future points, e.g., as we spend more computational resources. Therefore, in this case, we can say that such a downward trend gives the model some useful information about how to predict the future.
4. Experiments
Datasets. Following the previous work (Alabdulmohsin et al., 2022: Caballero et al., 2022), we first validate our NSL-PEN on the popular benchmark datasets (Alabdul- mohsin et al., 2022) consisting of scaling curves evalu- ated on various tasks in both image and natural language domain. The image classification (IC) dataset includes 72 scaling curves, each of which evaluates few-shot pre- diction performances of various neural network architec- tures w.r.t. the number of training datapoints. Specifically, many popular architectures such as BiT (Kolesnikov et al., 2020), MiX (Tolstikhin et al., 2021), and ViT (Alexcy, 2020) are evaluated on the various image classification datasets such as ImageNet (Russakovsky al., 2015), CI- FAR 100 (Krizhevsky et al., 2009), Birds (Welinder et al., 2010), and Caltech 101 (Fei-Fei et al., 2004). The natu- ral language processing (NLP) dataset contains 20 scaling curves, each of which evaluates the performance of vari- ous Transformer architectures (Bansal et al., 2022: Thop- pilan et al., 2022) w.r.t. the number of training datapoints, on neural machine translation (NMT), language modeling (LM), and Big-Bench (BB; bench authors, 2023). We fur- ther consider the nanoGPT-Bench dataset (Nano; Kadra et al., 2023) consisting of 24 scaling curves obtained by training nanoGPT models with varying model sizes on the Open WebText dataset (Gokaslan et al., 2019). We also use ColPret, a recently released huge dataset containing more than 1000 curves (Choshen et al., 2024), each of which eval- uates the performance various LLMs w.r.t. the number of training datapoints. We subsample it into 192 curves such that the number of points in each curve lies within [10, 1000). Lastly, we consider double descent (DD) dataset (Nakkiran et al., 2021) consisting of 16 curves exhibiting double de- scent behavior, to test each model's ability to predict more complex scaling behaviors. See Appendix B for details.
Based on those intuitions, we restrict the cutoff position for each function family follows: 1. Down: The cutoff position can be at any point. 2. Down-Down: The cutoff position can only be within the last segment. 3. Down-Up- Down: The cutoff position can only be at the second or third (last) segments. Note that such restriction does not violate the PFN framework, as it simply specifies C, T P(D) in Eq. (8), which defines the procedure of randomly splitting the context and target points. For instance, I.C- PFN (Adriaensen et al., 2023) restricts the target points T to be positioned after the context points C, similarly to ours.
3.2. Training Objective
Context regression loss. We use similar objective func- tion to Eq. (8) to train our PFN but add an auto-regressive objective on the context points as follows:
max Rc,T<M(D) [ (oog (Y* ,C) + log (Y
8
(10)
5

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Table 2: Results on image domain tasks (IC). We report mean and standard deviation (std) over 3 runs for the MCMC baselines and our method. For the other methods, we do not report std, as the point estimation methods always produce the same results, and for LC-PFN, we the single model released by the authors. The best results are in hold, and the second-best results are underlined.
ImugeNet
CIFAR100
Binds
Caltech101
Average
Method
Bayes.
RMSLE (
(1
RMSLE())
(t
RMSLE(
(T)
RMSLE(
(T)
(MSLE(() LL(
M;
X
0,0838
0,0803
0.0749
0,1793
0.1046
M2
0.0830
0.0626
0.0729
0.1
0,0823
M1
X
0.0470
0.0436
0,0555
0,1090
0.0615
M4
0.0158
0,0315
0,0302
0,0886
0.0415
BNSL
X
0,0121
0,0351
0,0242
0,0911
0.0406
MCMC (MM)
0.0794
1:93
0.08 =
1.67
0.0724
19993
0.1802
1.957suc:
0.1047
1.930
MCMC (M22)
1m974
0.0618
2.344g100
2.06622m0ss
2.
0. 0.100
2.582gnnes
0.0776ammess
2.2872m00
MCMC (MMa)
0.0e21
2.655
0.0424:auxs
2.663
0.0572
2.348:aus
0,0978
2.627zmee
0.0595
2.568 21004
MCMC (Ma))
0.0160gaaes
3
0.0320gaxxs
2.969
0.0316
3.039
0.881
goven
2.788
0.0422gmrew
g.024
MCMC (BNSL)
0.0109
3.562
0.0412
2.649
0.0277 20000
3.149 2002
0.1782 saurss
2.524
0.0645
20003
I.C.PT PFN
0.0096
4,146
0.0373
1.644
0.0233
2
0.1009
1,065
0 0.0428
2.429
NSL-PEN (ours)
0.0092 k
3.8 g008
0.031 gaawes
2.8 2.00
0.0202
3.464
0.0507 gaomn
3.21 22
0.0280 given
3.830
Table 3: Results on language domain tasks (NIP and Nano)
Method
Bayes.
NMT
LM
BR
Nano
Average
RMSLE (
(t)
(M)LE(
(L
(M)LE
(MSLE())
(L(
RMSLE (1)
(L
M.
X
0.2217
0,0140
0.0148
0.0534
0.0593
M2
X
0.0541
0,0010
0.0146
0.0549
0.0367
Ma
X
0,0471
0,0022
0.0087
0,0534
0.0367
M4
X
0.0208
0.0009
0.0123
0.0360
0.0249
BNSL
0,0184
0,0016
0,0164
0,0299
0,0223
MCMC (M4))
0.2247 sases
0.036
0.0155
2.746
0,0158:no00
2.958 w024
1.05
1.232
0,0569
1.732 m0003
MCMC (Ma)
g.0333
2.791
0.0023
3.695
0.0143
2.9
gooms
0.352
0.0365
1.636
MCMC (Ma))
2.0471sa000
2.592 130
0.0023
3.072
0.0086 -ooee
3.065
0.0537 soumse
0.411
0.0368 1000
1.716
MCMC (.Ma)
0.0144
3,090
0.0035
3,33
0.0083
3.098
0,087425umo
0.0235 1.474
2.21 62rons
MCMC (BNSL)
0.0438-aaam
2.842-aam
0.0022
3,969 1M44
0.0201 E0000s
2.747 -u25
0.0887 suues:
2.076
0.0582 cwoss
2.531 mooe
LC-PFN
0.0471
-0.354
0,1151
2,805
0.0191
1,019
0
1.849
0,0398
1,519
NSL.PEN (ours)
0.0248
2.798gn006
3,00
3.740
0.0202g000s
3.32
0.0260.aaes
2.3399.0R8
0.0194:nmea
2.773
Table 4: Results on ColPret and double descent (DD) dataset. i indicates that few trials failed due to overflow error, which were excluded from the calculation.
monotonically increasing functions. we horizontally flip the model output, i.e., ÿ < 1 ý. See Appendix C for fur- ther details on the baselines, including prior settings for the MCMC variants and other hyperparameter configurations used with EMCEE.
ColPret
DD
Method RM)LE(
(L(
RMSLE (1)
(t)
M
0,0732
0.1140
AM2
0.0545
0.1140
M33
0,0837
0.1333
Mi
0,0410
0.0910
Evaluation metric. We report the root mean squared log error (RMSLE; Alabdulmohsin et al., 2022; Caballero et al., 2022), i.e., / 1 Ey-er(log 9 log y² where T is the target set, á the prediction', and y" the correspond- ing label. For the Bayesian methods, we also evaluate the quality of predictive uncertainties by reporting the average log-likelihook (LL), i.e., . E(e"y") log g(v> ,C), where g is the predictive distribution of each method, and and T are the context and target set, respectively. LL and RMSLE are averaged across all the curves in each dataset.
BNSL
0.0998
0.0468
MCMC (M,)
0.0769
1.334
0.1206
MCMC (Ma))
0.0533-auct
1.855
0.1199
-10.842
MCMC (.Ms)
0.0004
1.798
0.1222 poaa0
127782
MCMC (Mc)
0.0417=40002
1.787
0.0925 saaas
=0.255
MCMC (BNSL)
0.0690-nouz7
2.727
0.0494-aau1
1,250=a34
LC-PFN
0.0289
2.804
0.0706
1.321
NSL-PEN (ours)
0.0271:s100s
2.794
0.0335
2.565
Baselines. We first consider point estimation meth- ods, including M1-4 and BNSL., as described in $2.1. For Bayesian baselines, we compare against a MCMC Boethod-specifeen
et al., the setup of Domhan et al. (2015), We evaluate five variants of EMCEE, each corresponding to the
EMCEE (Foreman-Macee
Implementation. We mostly follow Müller et al. (2021). We use Transformer (Vaswani, 2017) and treat each con- text pair (x. and target x² as individual tokens while omitting positional encoding. We discretize the output dis- tribution into a finite number of bins (1,000). The hyperpa- rameters of architecture are set as follows: nlayers=12, nheads=4, and nhidden=512. We train our model on 1.6M synthetic examples sampled from our prior for 100K iterations. See Appendix C for more details.
MCMC version of one of the function families: M1-4 and
BNSL. The likelihood function is defined as as NNy: f(x), o o²). where x² is parameter capturing observational noise and f one of the models from M1-4 or BNSL. The number of MCMC samples (nsample ess is set to 150 (except in Fig. 6) to ensure that the inference time is comparable to that of NSL-PEN. We also compare against LC-PFN (Adriaensen et al., 2023), another Bayesian method originally developed for extrapolating learning curves. Since LC-PFN assumes
We use mean for the MCMC haselines, and the median for both LC-PFN and NSI. following Adriaensen et al. (2023).
6

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
MCMMC (Ms)
MCMC (Mc)
(MS)
(Me)
(BNSL)
LCC.PEN
(ours)
IC
NLP, Nano
ColPret
DD
3.5
3.5
3.0
2.5
2.0
2.0
2.0
1-
1.5
Log
1.5
1.5
0-
1.0
0.5
-2
2006
40%
6.% Cutoff
B0%
100%
20%
40%5
80%
80%
100%
20%
40%
60%
100%
100%
20%
40%
6.0% Cutorf
807
100%
CLtoff
Outoff
Figure 3: Extrapolation performance (LL) with varying cutoffs.
40%
Cutoff=60%
Cutoff=80%
Cutofi=100%
Target
Cutef
MCNC (M))
1.00
200
300
400
500
100
200
300
400
560
1.00
200
300
400
500
100
200
360
400
500
Thrpet
Corl
MONC (th
1.00
200
300
400
500
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Trrpet
Cutori
LC P7M
1.00
200
360
400
500
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Lone
Conter
3-
Coco
NC
1.00
200
300
400
500
100
200
300
400
500
1.00
200
300
400
500
100
200
300
400
500
Embodcing Dimension
Embedding Dimenssoo
Embeoding Dimension
Embedding Dimonston
Figure 4: Visualization of extrapolation on double descent (DD) (Nakkiran al., 2021) with varying cutoffs. We visualize the mean
or median as solid lines show the 90% confidence intervals with shaded areas. the visualizations are presented in Appendix
4.1. Main Experimental Results
and the DD dataset in Table 4. The results clearly show that our NSL-PEN performs better than all the baselines in terms of both metries, especially more significantly on DD, demonstrating the strong ability of NSL-PFN to predict complex scaling behaviors in a reliable manner.
Quantitative analysis. We first evaluate our NSL-PFN
on the popular benchmark datasets on both image (IC, in Table 2) and language domain (NLP and Nano, in Table 3). On both domains, our NSI.-PEN consistently outperforms the baselines in general, achieving the best RMSLE and LL on average (on the rightmost column for each table). Whereas NSL-PFN significantly performs better than the baselines on almost all the cases, it performs slightly worse than MCMC (M4) and M4 on NMT and LM, respectively. We conjecture that this is because the scaling laws on those cases are rather too simple (see Fig. 12 and Fig. for the visualization) such that considering complicated cases such as double descent in prior design may act as distractor and thus lead to worse predictive performance. Except for such few cases, overall, the results clearly show the superiority of our NSL-PEN in terms of both predictive accuracy and the quality of uncertainty. We also test NSL. PFN on the recently released huge dataset called ColPret,
Effect of context set size. In Fig. 3, we further test the robustness in prediction against varying context set size, i.e., the number of observations in each curve, or the cut- off. We consider only the Bayesian methods since it is Bayesian that aims to make models more robust against fewer observations, such as how well the predictive uncer- tainty (coming from the limited observations) covers the unseen target points. We thus use LL to evaluate both the predictive accuracy and quality of uncertainty at the same time. We can see that our NSL--PFN shows higher LL on all the datasets and various cutoff percentages, clearly showing its robustness in prediction over the varying sizes of context sets. The gap especially prominent on DD as expected, primarily because our functional prior already covers di-

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
MCMC (ML)
MCMC (A1g)
MCMC (Mss
MCMC
(BNSL
NSL-PFN (ours)
IC
NLP
ColPret
D0
Aep. Lop.op-- Lkelhhod
0-
-2-
-5
-7
-1
10-
-12
5
7 Observations
5
6
6
7
Obseevatioos
Observations
Observations
Figure 5: Results of Bayesian active learning experiments.
verse forms of double descent behavior with varying cutoffs, with the resultant posterior predictive distribution (PPD) being more robust against such variation and data scarcity.
Table 5: Ablation study on the prior distribution.
Prior
Break
Up
RMISLE
KC
LTT
RMSLE
NLP. Nano
1..
RMSLE
ColPret
TI.
RMSLE
DD
Ma
0.64
2.54
1.57
1.94
0.131
.171
Ma
x
x
0.050
2.90
0.027
2.83
0.075
2.24
0.058
2,01
Qualitative analysis. In Fig. 4, we visualize the extrapo- lation on DD to better understand how each model actually predicts with varying cutoffs. MCMC (M4) struggles when- ever the context part includes the upward segment (e.g., the upper row), because its base function family M4 cannot ex- press such non-monotonic behavior by definition. MCMC (BNSL) is able to express such upward trends but strug- gles to generalize whenever the last segment of the context part is increasing, predicting that the curve will increase indefinitely (e.g., the last two panels). On the other hand, NSL-PEN shows reasonable predictions even in such cases, predicting that the curve will eventually decrease at some uncertain future points, as intended. The leftmost panel can be seen as the only failure case. As explained in $3. 1, we consider such case as inherently difficult to correctly predict, assuming that there is no way to infer from the single decreasing segment that the curve will go upward. Lastly, LC-PFN shows suboptimal performance than NSL- PFN because its functional prior is unaware of such complex behavior. See Appendix E for other visualizations on DD.
Maa
x
0018
0.022
3.24
0015
0.015
2.82
292
0.032
0.027
2.74
0051
0.71
2.12
1.20
Ma.
0.028
3.33
0,019
277
0.027
2.79
0,033
2,57
Table 6: Ablation study on the context regression loss.
IC
NLP. Nano
ColPret
DD
Contest
RMSLE
LL
RMSLE
LL
RMSLE
LL
RMSLE
LL
0,0309
3.236
0.0205
2.630
0,0263
2.769
0,0322
2.577
0.0280
3,330
0.0194
2,773
0.0271
2,794
0.0335
2.565
lations between context points, explained in $3.2. Note that LC-PFN cannot be used in this experiment since it is not learned to interpolate within each curve.
Fig. 5 shows the results on all the datasets except for Nano, since each curve in Nano contains only 7 points in total, which is too few for conducting this experiment. We can see that the overall performance of NSL-PFN is much better than the basclines, and also consistently improves as more observations are collected. The results clearly demonstrate the superiority of our model in terms of the quality of un- certainty, positioning NSL.-PEN as valuable tool that can help make many important decisions in the real world.
4.3. Other Analysis
4.2. Bayesian Active Learning
Ablation study. We validate criteria used for defining our functional prior explained in $3.1. In Table 5, see that us- ing both M3 and Ma is better than using only one of them in general (Ist and 2nd row vs. 3rd row). The assumption of random breaks also improves the performance (3rd vs. 4th row). Lastly, including the upward trend improves the performance slightly on IC and ColPret, largely on DD as expected, but not on NLP and Nano (4th vs. last row). This is because, as mentioned in $4.1, the scaling laws in the NLP dataset are too simple and do not require training with such unexpected behavior. We further examine in Table 6 the effectiveness of the context regression loss in Eq. (10). In general, including it improves both metrics, except for a slight decrease of RMSLE on the ColPret and DD datasets.
Considering that scaling curves are often collected from training large neural networks with huge amounts of data, collecting each observation point can be very costly. There- fore, we must carefully decide where observe to minimize the cost, especially in real-world scenarios involving im- portant decision maainnnn-annggggo problems such as how much cost should be spent to achieve the desired performance. We frame this as a Bayesian active learning (BAL; Houlsby et al., 2011: Gal et al., 2017) problem-incremeee incrementally ob- serving the unseen point that is expected to maximize the average likelihood on all the other unseen points, based the uncertainty information provided by the given model.
In this experiment, starting from four observations, we iter- atively select the next unseen point with a specific criterion. We use variation ratio, i.e., max,, [0xx|_1mmx,,,,- max, ([y|z, C)] (Free- man, 1965) as our acquisition function, which prioritizes the unseen area where the given model is least confident. For NSL.-PN, we use the model trained to predict interpo-
Efficiency. We highlight the efficiency of amortized in- ference compared to non-amortized baselines such as point estimation and MCMC. As shown in Table 7, the inference time for both point estimation and MCMC are significantly
8

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Table 7: Avg. inference time (8) per curve (nsamples=150).
Table 8: Results of BLR and DKGP. & indicates that some
MM
BNSL
MCMC
MCMC (BNSL)
LC-PFN
NSL-PEN (ours)
RMSLEs could not be calculated due to negative y y-values.
15.65
98.79
154.64
280.55
0.02
0,02
Method
RMSLE
IC
1.1
RMSLE
NLP Nano
11.
RMSLE
ColPret
11.
RMSLE
DD
1.1
MCMC
MCMC (AG)
MCMC (BNSL)
NSL-PFN (ours]
(with
0.307
-0.3
-3.8
0.507
2.73
0.603
-9597
MCM: (MM))
MCMC (AGe)
LC-PIN
DKGP
0.197
-1.86
0.067
0.62
0.101
0.10
0.145
-3.29
NSL-PN (ours)
0.028
333
0,019
2.77
0,027
2.79
0,033
2.57
Log- log-likelihood MCMC plmp
Log-llkellhood vs. Inference
Table 9: Results of calibration using MSCE (U).
Log 1eleehooo
2.5
Aug. Log- Beallaadd
2.5
Method
IC
NLP. Nano
ColPret
DD
2.0
2.0
MCMC (Mi)
0.1422-ne.me
0.2328 a
0.1349, womes
0.1676
MCMC (Mg)
0.0912:010
0.1922
0.2394200s o.
1.5
MCMC (Ma)
0.0538gereh
0.2048gaam ans
0. 0.1010
0,264 1.2641 goo0
MCMC (Ma)
0.0381 eesss
0.1630
0.1169 wwoes?
0.1943 1000ss
1.0
1.0
MCMC (BNSL)
0.0773
D.1238 bwa
0.2072 _0094_.0rs
150
3000
750
1500
3000
10a
LC-PEN
0,1809
0.2674
0.1071
0,1500
MCMC samples
Inference Time (secondss
NSL-PEN (ours)
0.0512
0.1145 A
0,0615 Jounss
0.0826 re0012
Figure 6: MCMC with an increased number of samples (nsamples). Convergence failures were observed for some MCMC (BNSL) runs, and these runs are excluded.
bases, are available in Appendix C.
To additionally assess the quality of predic- tive uncertainties, we incorporate the mean square calibra- tion error (MSCE; Kuleshov et al., 2018), i.e., J Σラー Wi-- (p) -- where m is the number of bins, P/ represents the predicted confidence level, P3 is the empirical frequency of the true outcome falling within that confidence level, and Wy are weights (typically set to 1). This metric evaluates how well the predicted confidence levels align with the ac- tual observed frequencies. As shown in Table 9, NSL-PFN demonstrates notably better calibration performance than baseline models, particularly on ColPret and DD. These results show the robustness of NSL-PEN in handling realis- tic scaling behaviors and the importance of well- well-calibrated uncertainty for reliable extrapolation in such regimes.
Calibration.
higher than those of amortized approaches like LC-PFN and our NSL-PEN. This efficiency arises because, once meta- trained, PFN-based methods require only a single forward pass per curve to make a prediction. BNSL and its MCMC variant are particularly slow due to an additional validation process for selecting the number of breakpoints.
Furthermore, we investigate the effect of increasing the number of MCMC samples (nsamplee les 300, 750, 1500, 3000) on performance. As illustrated in Fig. 6, these exper- iments show that increasing the number of samples yields only marginal performance gains, which are not commen- surate with the additional computational cost. In fact, for MCMC (M1) and MCMC (BNSL), performance slightly degrades with longer sampling chains. Crucially, all base- lines still underperform compared to NSL-PEN, showing both its effectiveness and efficiency.
Prior hyperparameter tuning. For our prior design, we manually adjust the parameters of our functional prior to visually match the shapes of the actual curves collected from various domains. To assess whether NSL-PFN can further improve prior hyperparameter tuning, we performed Bayesian optimization on the prior parameters (e.g.. a, b, d for M3 and a, for Ma). The optimization process successfully identified configurations that yielded slight per- formance improvements over our default settings with just 60 BO steps. See Appendix D for details.
BLR and DKGP. Beyond MCMC-based approaches, we also explore other Bayesian inference baselines for exten- sive comparison. Specifically, we include Bayesian Linear Regression (BLR; Bishop, 2006) with neural network (NN) basis functions and Deep Kernel Gaussian Process (DKGP: Wilson et al., 2016). For BLR with NN, parameters of the neural network serving as basis functions, along with other model parameters, are tuned by maximizing the log marginal likelihood. While these methods-BLa BLR with ap- propriate basis functions offering interpretability and DKGP providing flexible -orpaaamettric modeling- are standard for ID regression, they often encounter difficulties in ef- fectively capturing the distinct inductive biases inherent in neural scaling laws, as demonstrated in Table 8. This chal- lenge in embedding specific prior knowledge about scaling behaviors into their respective basis or kernel functions can render them less competitive for extrapolating neural scal- ing laws when compared to models explicitly designed for this task. Further details regarding the implementation of BLR and DKGP, as well as a discussion of other BLR vari- ants utilizing polynomial, RBF, Fourier, sigmoid, and spline
5. Conclusion
In this work, we introduced Bayesian approach using Prior- data Fitted Networks (PFNs) to improve neural scaling law extrapolation, addressing limitations of traditional point es- timation by incorporating uncertainty quantification. By de- signing a prior distribution that enables PFNs to meta-learn through synthetic functions resembling real-world scaling laws, our method effectively extrapolates scaling behavior in diverse scenarios. Evaluation on real-world scaling laws shows superior performance, particularly in data-limited settings like Bayesian active learning, demonstrating its use- applications.
fulness for practical and

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Acknowledgements
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Ex- plaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):22311878122 2024.
This work was partially supported by Institute of Informa- tion & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-00457882, AI Research Hub Project), ITP grant funded by MSIT (No. RS-2020-11200153, Penetration Secu- rity Testing of ML Model Vulnerabilittie and Defense), IITP grant funded by MSIT (No. RS-2019-1110005, Artificial In- telligence Graduate School Program (KAIST)), ITTP grant funded by MSIT 200.RS-2022-11220113 Meta-learning Applicable to Real-world Problems), Samsung Electronics (I0201214-0811) 241-241440441011 145-01), and National Research Foundation of Korea (NRF) grant funded by MSIT (No. RS-2023- 00256259).
Bansal, Y., Ghorbani, B., Garg, A., Zhang, B., Cherry, C.,, Neyshabur, B., and Firat, O. Data scaling laws in nmt: The effect of noise and architecture. In International Con- ference on Machine Learning, pp. 1466-1482. PMLR, 2022.
Beleites, C., Neugebauer, U., Bocklitz, T., Krafft, C., and Popp, J. Sample size planning for classification models. Analytica chimica acta, 760:25-33, 2013.
bench authors, B. Beyond the imitation game: Quantify- ing and extrapolating the capabilities of language mod- els. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: /opp/reven.e.eeeeeeeee net / forum? conrummid=uyyle 1atammiiiuuyTTLSsS111
Impact Statement
This paper introduces a Bayesian approach to neural scal- ing law extrapolation using Prior-data Fitted Networks (PFNs), emphasizing resource-efficien decision-making. By designing a tailored prior for PFNs, our method en- ables uncertainty-aware predictions, improving over exist- ing point estimation and Bayesian techniques. This en- hances the predictability of neural scaling laws, leading to more efficient model training and optimized resource allo- cation. Reliable extrapolations with quantified uncertainty support better decision decision-making in large-scale model train- ing, reducing unnecessary computational costs. While our approach improves efficiency, practitioners should comple- ment it with empirical validation to ensure responsible and informed scaling decisions.
Bishop, C. M. Pattern Recognition and Machine Learning (Informatinn Science and Statisties). Springer- Vereee,, Berlin, Heidelberg, 2006. ISBN 0387310738.
Bojar, O., Buck, C.,, Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint- Amand, H., et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pp. 12-58, 2014.
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. 2020.
Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. arXiv preprint arXhv:2210.14891, 2022.
References
Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training. arXiv preprint 1rXiv:2/10002000 10.02095, 2021.
Cettolo, M., Girardi, C., and Federico, M. Wit3: Web inven- tory of transcribed and translated talks. In Proceedings of the Conference of European Association for Machine Translation (EAMT), pp. 261-268, 2012.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Chandrashekaran, A. and Lane, I. R. Speeding up hyper- parameter optimization by extrapolation of learning curves using previous builds. Machine Learning and Knowledge Discovery in Databases: European Confer- ence, ECML PKDD 2017, Skopje, Macedonia, Septem- ber 18-22, 22, 2017, Proceedings, Part I 10, pp. 477-492. Springer, 2017.
Adriaensen, S., Rakotoarison, H., Müller, S., and Hutter, F. Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Informa- tion Processing Systems, 36, 2023.
Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X. Revisit- ing neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022.
Cho, J., Lee, K., Shin, E.,, Choy, G., and Do, S. How much data is needed to train medical image deep learning sys- tem to achieve necessary high accuracy? arXiv preprint 2015.
WrXiv:1511.06
Alexey, D. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010. / 1929, 2020.
Choshen, L., Zhang, Y., and Andreas, J. A hitchhiker's guide to scaling law estimation, 2024. URL https: / www.ii.org/aas///..eooa......aaa 11840..
10

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuue of Machine Learning Research, 15570):1-53, 2024.
Freeman, L. G. Elementary applied statistics. 1965.
language models. Journal
Frey, L. J. and Fisher, D. H. Modeling decision tree per- formance with the power law. In Seventh international workshop on artificial intelligence and statistics. PMLR, 1999.
Cortes, C., Jackel, L. D., Solla, S., Vapnik, V., and Denker, Learning curves: Asymptotic values and rate con- vergence. Advances in neural information processing systems, 6, 1993.
Gal, Y., Islam, R., and Ghahramani, Z. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183-1192. PMLR, 2017.
Dai, Z., Liu, H., Le, Q. V., and Tan, M. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems, 34:3965-3977, 2021.
Gargiani, M., Klein, A., Falkner, S., and Hutter, F. Proba- bilistic rollouts for learning curve extrapolation across hy- perparameter settings. arXiv preprini arXiv: 1910.04521 2019.
Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 2018.
Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation. arXiv preprint arXi::2109.077400 2021.
Domhan, T., Springenberg, J. T., and Hutter, F. Speed- ing up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Open- webtext corpus, 2019.
Gordon, M. A., Duh, K., and Kaplan, J. Data and parameter scaling laws for neural machine translation. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5915-5922, 2021.
Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z.,, Chang, B., Sun, X., Xu, J., and Sui, Z. A survey in-context learning. arXiv preprint ar arXiv:2301.002::: 2022.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Dooley, S., Khurana, G. S., Mohapatra, C., Naidu, S. V., and White, C. Forecastpfn: Synthetically trainnnd zero-shot forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling predictable, empirically, arXiv preprint arXiv:1712.00 004099 2017.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788, 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbi, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Elsken, T., Metzen, H., and Hutter, F. Neural architecture search: survey, Journal of Machine Learning Research, 20(55):1-21, 2019.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178-178, IEEE, 2004.
Hollmann, N., Müller, S., Eggensperger, K., and Hut- ter, Tabpfn: transformer that solves small tabu- lar classification problems in second. arXiv preprint arXiv:2207.01848 2022.
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 12.5745, 2011.
Figueroa, R. L., Geng-Treitles Q., Kandula, S., and Ngo, L. H. Predicting sample size required for classification performance. BMC medical informatics and decision making, 12:1-10, 2012.
Johnson, M., Anderson, P., Dras, M., and Steedman, M. Pre- dicting accuracy on large datasets from smaller pilot data. 56th Anmual Meeting of the Association for Computa- tional Linguistics, ACL 2018, pp. 450-455. Association for Computational Linguisties (ACL), 2018.
Mackey, D., Hogg, D. W., Lang, D., and Goodman, emcee: the MCMC hammer. Publications of the Astronomical Society of the Pacific, 125(925):306, 2013.

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Kadra, A., Janowski, M., Wistuba, M., and Grabocka, J. Scaling laws for hyperparameter optimization. Advances in Neural Information Processing Systems, 36, 2023.
Müller, S., Feurer, M., Hollmann, N., and Hutter, F. Pfns4bo: In-context learning for bayesian optimization. In Inter- national Conference Machine Learning, pp. 25444- 25470. PMLR, 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechan- ics: Theory and Experiment, 2021 2021.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Klein, A., Falkner, S., Springenberg, J. T., and Hutter, F. Learning curve prediction with bayesian neural networks. In International conference on learning representations, 2022.
Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E., and Hutter, F. In-context freeze-thaw bayesian optimization for hyperparameter optimization. arXiv preprint arXiv:2404. 16795, 2024.
Kolachina, P., Cancedda, N., Dymetman, M., and Venkatap- athy, S. Prediction of learning curves in machine trans- lation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 22 30, 2012.
Rosenfeld, J. S. Scaling laws for deep learning. arXiv preprint arXiv:2108.0668: 2021.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. arXiv preprini arXiv: 1909. / 12673, 2019.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General vi- sual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part V 16. pp. 491-507. Springer, 2020.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. doi: 10.1007/s11263-011 010-063---511011----
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Sharma, U. and Kaplan, J. Scaling laws from the data mani- fold dimension. Journal of Machine Learning Research, 23(9):1-34, 2022.
Kuleshov, V., Fenner, N., and Ermon, S. Accurate uncer- tainties for deep learning using calibrated regression. In International conference on machine learning. pp. 2796- 2804. PMLR, 2018.
Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unrcasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Lee, B., Zhang, A. S., Kim, B., Park, J., Lee, J., Hwang, J. and Lee, H. B. Cost-sensitive multi- qulti-fidelity bayesian optimization with transfer of learning curve extrapolation. arXiv preprint ar 2rXh:22405.17918. 2024.
Swersky, K., Snoek, J., and Adams, R. P. Freeze-thaw bayesian optimization. arXiv preprint arXiv: 14063:44 2014.
Tan, M. Efficientnet: Rethinking model scaling for convolu- tional neural networks. arXiv preprint arXiv: 1946, 2019.
Mohr, F. and van Rijn, J. N. Learning curves for decision making in supervised machine learning-a survey, arXiv preprint XxXx:222111115500 2022.
Thoppilan, R., De Freitas, D.,, Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog appli- cations. arXiv preprint 2tth::22010083333 2022.
Mukherjee, S., Tamayo, P., Rogers, S., Rifkin, R., Engle, A., Campbell, C., Golub, T. R., and Mesirov, J. P. Estimating dataset size requirements for classifying dna microarray data. Journal of computational biology, 10(2): 19-142, 2003.
Tolstikhin, O.,, Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung. J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all- architec- ture for vision. Advances in neural information process- ing systems, 11:22261-24277 2021.
Müller, S., Hollmann, N., Arango, S. P., Grabocka, J., and
Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2 12. 105/0, 2021.
12

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13397/ 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 002288 2023b.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Verdenius, S., Zerio, A., and Wang, R. L. Lat-pfn: A joint embedding predictive architecture for in-context time-series forecasting, arXiv preprint 30X09 2024.
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F,, Belongie, S., and Perona, P. Caltech-ucsd birds 200. 2010.
Wilson, A. G., Hu, Z.,, Salakhutdinov, R., and Xing, E. Deep kernel learning. Artificial intelligence and statis- tics, pp. 370-378. PMLR, 2016.
Wistuba, M., Kadra, A., and Grabocka, J. Supervising the multi-fidelity race of hyperparameter configurations. Advances in Neural Information Processing Systems, 35: 13470-13484, 2022.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23104-1211. 2022.
13

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
A. More Discussion on Related Work
Neural scaling laws There are many observations that neural scaling laws are empirically predictable (Hestness et al., 2017; Johnson et al., 2018: Rosenfeld al.. 2019; Kaplan et al., 2020; Rosenfeld, 2021: Ghorbani et al., 2021; Bansal et al., 2022; Hoffmann et al., 2022; Chung et al., 2024), just name a few. We only discuss recent studies that have explored parameterizations for neural scaling law extrapolation. The simplest model M1 assumes power law function (e.g.. // = 9x throughout the domain of interest (e.g., data, model size, or training time), where it has been used in various domains, such as healthcase (Cho et al., 2015), neural machine translation (NMT; Gordon et al., 2021), and language models (Kaplan et al., 2020), among others (Hestness et al., 2017; Johnson et al., 2018; Sharma & Kaplan, 2022). The most widely used model in the literature is M2, which adds a bias term to the power law function, to capture saturating performance for large x (Mukherjee et al., 2003; Figueroa et al., 2012; Domhan et al., 2015; Hestness et al., 2017; Johnson al., 2018; Rosenfeld al., 2019; Abnar et al., 2021; Gordon al., 2021; Rosenfeld, 2021). A different parameterization M3 has been recently used in NMT (Bansal et al., 2022) to guarantee that the power law function converges to a finite constant when the domain of interest approaches to infinity. Alabdulmohsin et al. (2022) have proposed M2 to handle non-power law behaviour in partial scaling law. More recently, Caballero et al. (2022) have proposed broken neural scaling law (BNSL) by assuming that real-world neural scaling laws have different power law for each segment (Nakkiran et al., 2021).
Prior-data Fitted Networks (PFNs) In-Context Learning (ICL) represents a paradigm shift from traditional learning approaches by enabling rapid adaptation to new tasks and data without extensive retraining, through conditioning on input prompts or context (Dong et al., 2022). The rise of ICI, parallels the development of Traasfoomer-baseed architectures, particularly large language models (LLMs; Radford et al., 2019; Brown, 2020; Achiam et al., 2023; Touvron et al., 2023a;b; Dubey et al., 2024). Prior-data Fitted Networks (PFNs) are Transformer models designed for in-context Bayesian prediction, facilitating efficient inference in single forward pass without extensive retraining or hyperparameter tuning. PFNs have demonstrated e effectiveness in various applications, such as Bayesian learning curve extrapolation (Adriaensen et al., 2023), in-context tabular classification (Hollmann et al., 2022), black-box hyperparameter optimization (HPO; Müller et al., 2023), freeze-thaw Bayesian optimization (Rakotoarison et al., 2024), and time-series forecasting (Dooley et al., 2024; Verdenius et al., 2024). Building upon this foundation, our work applies PFNs to infer neural scaling laws.
Learning curve extrapolation Learning curves have long been an active area of research, as surveyed by Mohr & van Rijn (2022). Early works (Cortes et al., 1993; Frey & Fisher, 1999; Kolachina et al., 2012) primarily modeled learning curves (LCss) based on dataset size, providing "point estimates" of performance without a probabilistic nature, let alone a Bayesian framework for uncertainty quantification. Additionally, these models often do not address deep neural network (DNN) training, where the x-axis typically represents training time, measured in parameter updates or epochs. Non- -By-saasia probabilistic models for DNN training curves were later explored (Chandrashekaran & Lane, 2017: Gargiani et al., 2019) 1). In contrast, the first Bayesian approach for DNN training curve extrapolation was introduced by Domhan et al. (2015) using Markov Chain Monte Carlo (MCMC) methods. More recently, Adriaensen et al. (2023) introduced a method employing Prior-data Fitted Networks (PFNs) to predict LCs based on training time. While this method bears a resemblance to our approach, it is constrained in its applicability to certain scaling behaviors, such as power-law scaling and double descent, seen when the z-axis reflects model scale or dataset size. Furthermore, it lacks the flexibility to model variable scaling behaviors across different regions of the r-axis, motivating our development of tailored prior capable of accommodating multiple breaks and facilitating Bayesian inference with user-defined configurations. Moreover, while extrapolation has been investigated in AutoML contexts (Swersky et al., 2014; Klein et al., 2022; Wistuba et al., 2022; Kadra et al., 2023; Rakotoarison et al., 2024; Lee et al., 2024), and future applications of our work may include enabling AutoMI. for large-scale deep learning, we currently prioritize the enhancement of neural scaling law extrapolation via Bayesian methods, rather than fully automating scaling up the hyperparameter optimization process.
B. Datasets
IC. We use the scaling laws benchmark (Alabdulmohsin et al., 2022), which covers various scaling laws in both vision and NLP domains. In the vision domain of the benchmark, the downstream evaluation focuses on 5-shot, 10-shot, and 25-shot image classification (IC) tasks. It includes two sizes of big-transfer residual neural networks (BiT; Kolesnikov et al., 2020), MLP mixers (MiX; Tolstikhin et al., 2021), and vision transformers (VIT; Alexey, 2020). These models are pretrained on the JFT-300M (Sun et al., 2017) dataset and tested on downstream tasks: ImageNet (Russakovsky et al.,
14

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
2015), CIFAR-100 (Krizhevsky et al., 2009), Birds (Welinder et al., 2010), and 1a (Fei-Fei et al., 2004). For each downstream task, the benchmark provides 18 scaling law variations (2 model sizes 3 model types x few-shot settings), resulting in total of 72 scaling laws. The x-axis of the scaling law corresponds to the number of examples used for training, while the y-axis represents the error rate in image classification.
NLP. In the NLP domain of the benchmark, scaling laws are provided for downstream tasks including neural machine translation (NMT), language modeling (LM), and Big- Big-Bench (BB; bench authors, 2023). For the NMT task, scaling laws are derived from five encoder-decoder configuration combinations trained following the methodology outlined in (Bansal et al., 2022). In this case, the x-axis represents the number of observed examples, while the y-axis corresponds to the log perplexity. For the LM task, the performance of LaMDA architecture (Thoppilan et al., 2022) is evaluated on next-token prediction. Scaling laws are provided for five different parameter sizes, with the x-axis indicating the number of observed examples and they y-axis showing the validation loss, rescaled to the range [0, 1J. For the BB task, the one-shot and two-shot performance of a 262M- -0aaamarreet decoder-only transformer is assessed across five sub-tasks from (bench authors, 2023). The x-axis denotes the number of observed examples, while the y-axis reports task-speeific metrics such as multiple choice grad or exact string match.
Nano. In addition to the scaling law benchmark, we incorporate data from nanoGPT-Bench and double descent (Nakkiran et al., 2021) to enhance the analysis of scaling behavior in the NLP domain, focusing on factors beyond the number of observed examples. nanoGPT-Bench (Nano) is benchmark introduced in (Kadra et al., 2023), which evaluates the performance of nanoGPT trained on the OpenWebTert WebText dataset (Gokaslan et al., 2019) using 12 different hyperparameter configurations. Both the final and best performance are reported, resulting in total of 24 scaling laws. In this benchmark, the x-axis corresponds to the model embedding size, while the y-axis represents either the final or best validation loss.
ColPret. Furthermore, include the latest large dataset (ColPret; Choshen et al., 2024). The dataset aggregates scaling law data from many language models where the largest model is over 3 billion parameters and provides over 1000 estimated scaling laws. It consists of 485 LLMs, including GPT-3 (Brown, 2020), Mamba (Gu & Dao, 2023). Llama (Touvron et al., 2023b), etc., and covers more than 40 distinct model families. They report the number of epochs and computational cost (FLOPs) additionally, but we focus on the number of seen tokens, leading to 591 scaling laws. We filter scaling laws based on their length, keeping only those between 10 and 1000, resulting in 194 scaling laws. Additionally, we further exclude two scaling laws: pythia- aythia-I9m-deaae deduped due to excessive noise and 1 1=24.h=8-0.25 because its loss saturates. Since remove two more scaling laws, the final dataset consists of 192 scaling laws. The x-axis in this dataset represents the number of seen tokens, while the y-axis corresponds to either the loss or perplexity.
DD. The study by Nakkiran et al. (2021) introduces scaling laws that exhibit a double descent pattern, challenging the conventional assumption of unidirectional performance trends. The researchers trained and evaluated CNNs, ResNets, and Transformers on datasets such as CIFAR-1O, CIFAR-I00 (Krizhevsky et al., 2009), IWSLT' 14 de-en (Cettolo et al., 2012), and WMT' 14 en-fr (Bojar et al., 2014), exploring various combinations of optimizer settings, augmentations, and noise levels. In this work, we incorporate 16 publicly available scaling laws from (Nakkiran et al., 2021) that illustrate double descent behavior (DD). In these scaling laws, the x-axis represents either the model embedding size or the number of observed examples, while the y-axis reflects the test error or cross-entropy test loss.
C. Implementation Details
M1-4- For experiments involving M1-4, we refer to the codebase provided by Alabdulmohsin al. (2022) We fit Mi-4 directly to the original scaling laws without normalization.
TThe implementation for M1 M1-4 is available at: https: petpa://githae http//tttttuutut/// com htt.:///oooggeeeeeeeeeee//oooeee
cree/masterrrreiist wtw.iiiinnnnnanannneeeeee neural @oural_scall laaaling_lwsss
15

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Broken Neural Scaling Laws (BNSL). We obtained the code from the BNSL (Caballero al., 2022) authors. The implementation of the code has minor adjustments ensure stable and efficient training, as detailed below:
BNSL: - exp log (chma(() + +ea) Ymmean
where [bna( = W3 (W2 (oftplus(W13 + b1) + b2) + bz.
3 log(r)
Mr
Ox
(11) The provided code lacked information on determining the number of breaks, deciding whether to crop or selecting cropping points. Despite our follow-up requests for clarification, we did not receive a response. Consequently, we designed and tested various scenarios based on the descriptions in the paper.
Ne mean (log(Traiii))
ox std (Train)
To determine the optimal number of breaks, we validate using either the last 10% or the final point of the context, selecting the number of breaks that produced the best performance. When the optimal number of breaks is greater than zero, further explore cropping. Specifically, we identify five evenly spaced points within the context as potential crop locations and apply the crop that yielded the best validation performance. We then apply the crop that yields the best validation performance to our inference process as well.
Table 10: BNSL RMSLE across implementations.
indicates that some trials failed, and these values were excluded.
Validation split
Crop
IC
NLP, Nano
DD
Last 10%
x
0.0406
0.0223
0.0468
Final point
X
0.0432
0.0186
0.0379
Last 10%
V
0.0576
0.0231
0.0477
Final point
0,0571
0.0228
0.0987
We summarize the results of four experiments in Table 10, each with variations in the validation set configuration and cropping application. Similar to M111, 11, no normalization is applied during the training of BNSL. Due to the large size of ColPret, we focus on IC, NLP, and DD. While the combination of validation using the final point and no cropping yielded the best performance in the NLP and DD tasks, some predictions in the NLP task failed. Consequently, the main text, we report results based on validation using the last 10% of the context without cropping.
Table 11: MCMC prior distribution.
Name
Function
Prior
Mj
y ux
((-1,0.5) ~ (10 1.5) log Nogb~N(-21) 22-2,1) c~U(0,11)
N((2,,1
M2
g ax + c
Mg
y alx + d)b
~ ((, logb Nobb~N( N(12,1) logd 4(0,1
-
M4
x s(y) =
loga ~ loga~
~011 1000) bggh~N N(-1,) loga~~ 6~0,0055
a(1-v)=
y exp (log (efmal()) + ed + Mmean)
BNSL
W -~0,(0 a N00,5
where Saoa (ir) W3 - (W2- softplus
(W5 (Wnplluus(WW.....
6.) (2) 6.s
Markoy Chain Monte Carlo (MCMC). We implement MCMC using EMCEE (Foreman-Mackey et al., 2013). The parameters for EMCEE are configured as follows: the number of walkers in the ensemble (nwalkers) is set to 30, the number of samples (nsample les) set to 150 (except for Fig. 6), the part of the chain omitted to account for mixing (burn_in) is set to 50, and the sub-sampling frequency (thin) is 1. Appropriate prior ranges are defined for each functional form, detailed in Table 11. The number of breaks for MCMC (BNSL) is determined through validation, following the same approach as used for point estimation.
During MCMC training, the x-values are normalized by dividing them by their maximum value, scaling the maximum to 1. However, for MCMC (M4), where smaller x-values frequently lead to failures, the x- x-values are instead normalized to a maximum of 1000. For the DD and Nano datasets, the y-values are normalized by dividing them by the maximum context value.
16

---

Bayesian Neural Scaling Law Extrapolation with Frior-Data Fitted Networks
Table 12: BLR performance with non-NN basis functions
IC AVG
NLP AVG
ColPret
DD
Basis
RMSLE (U)
(t)
RMSLE ())
LL (t)
RMSLE ())
LL (1)
RMSLE ())
LL
Polynomial
3,979
-6.71
1.311
-57.65
2.573
-174.43
2.420
-408.05
RBF
2.118
-34.82
0.253
-38.54
0.633
-34.82
1.085
-11.68
Fourier
1.362
-56.52
0.283
-160.41
0.529
-52.46
1.801
-34.99
Sigmoid
8.819
-3.37
0.187
-70.81
1.324
-454
1.419
-6.51
Spline
0.551
-48.56
0.159
-53.09
0.241
-64.49
0.238
-69.89
Bayesian L Linear Regression (BLR) and Deep Kernel Gaussian Process (DKGP). We conducted additional experiments with the following baselines: BLR (Bishop, 2006) with neural network basis functions, BLR with polynomial basis functions, BLR with RBF basis functions, BLR with Fourier basis functions, BLR with sigmoid basis functions, BLR with spline basis functions, and DKGP (Wilson et al., 2016). Hyperparameters, including those of the neural network used as the basis function for BLR, are tuned via marginal log-likelihood. We implement BLR using Bayes anRi anRidge and DKGP using GPyTorch. The feature dimension fixed to 5 for all additional baselines, and all target values y are scaled using RobustScaler: fitted on the context points (Vy C). The neural networks used for BLR with neural network basis functions and DKGP are trained for 1000 full-batch epochs using the Adam optimizer (Kingma, 2014) with a learning rate of 0.01. As shown in Table 12, BLR models with predefined basis functions exhibit significant performance degradation.
LC-PEN For our experiments, we utilize the LCCPPN as described in Adriaensen et al. (2023)². Since the range of x for the prior was sampled from [0, 100] during LC-PEN training, we normalize x-values to the same range during inference. LC-PFN includes its own normalization method for y-values, enabling it to predict learning curves across various ranges and directions. However, when evaluating LCCPFN on scaling laws, we found that its normalization method failed for certain scaling laws in the LM task. To address this, we apply the NSL-PEN normalization method, followed by horizontal flip, i.e., y +-1 1 - y. This adjustment reduced the RMSLE for the LM task from 0.5254 to 0.1151. We report LC-PFN performance using this updated methodology.
NSL-PEN
We employ a Transformer (Vaswani, 2017), representing each context pair (z, y) and the target x as individual tokens while excluding positional encoding. These tokens are encoded with a simple linear layer. The target tokens are assumed be conditionally independent of each other, i.e., we apply masking to the attention matrix so that each token only attends the context tokens.
Following Müller al. (2021), discretize the output distribution into finite number of bins. The size (width) of each bin is set so that, under the prior, has an equal probability of falling in each bin. The number of bins is set to 1,000. For RMSLE, we use the median of output distribution.
NSL-PFN inherits the hyperparameters of architecture from PFNs, such as the number of layers (1layers=12), number of heads (theads=44, and the size of each hidden layer (nhidde:=512). We train NSL-PEN on 1.6M scaling laws sampled from the prior for 100K iterations, i.e., mini-batch size is set to 16, with Adam optimizer (Kingma, 2014). The learning rate is set to 0,00002 with cosine annealing, and the warm-up phase spans the first 25K iterations. To determine the size of D (i.e,, M + N). we uniformly sample from the log space within the range [50, 500]. The training time is roughly 2.6 hours on NVIDIA -000-XXM4-0GGB.
Since NSL-PFN is trained with priors input ranges of [0.01, 1] for x and [0. 1] for y, we preprocess the evaluation data accordingly. Specifically, we normalize x- x-values to [0.01, 1]. For y-values, no additional preprocessing required for the scaling laws benchmark datasets (IC, NMT, LM, and BB), as they are already scaled to [0. 1]. However, for the Nano and DD datasets, where y-values exceed 1, normalize all y-values by dividing them by the largest value in their respective contexts. This ensures that the context part is normalized to the [0, I] range, although values in the target part may still exceed 1.
'The L.C-PN implementation available at: https: /t/gtttttttttt com/ aut aut.omm/lccf...
17

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
D. Hyperparameter Tuning for Prior Distributions
Table 13: Hyperparameter search space for Bayesian optimization. For each "Log-Paree LLog-Parameter" of given "Model" (e.g., log for M3). its normal prior distribution (p, o²) was refined. This was done by optimizing the "Prior Statistie"sseccifeeeee the prior's mean (/) or standard deviation (or). The "Search Range" column defines the optimization bounds for these statistics. Default prior values are in Table 1.
Model
Log- Parameeee
Prior Statistic
Search Range
Mean (n))
Real(-2.0, 0.0)
log
Std. Dev. (a)
1ea 1.5)
M3
Mean (f4)
Real(-3.0, -1.0)
log(-b)
Std. Dev. (a)
1ea 1.5)
Mean (μ1)
Real(-0.5,, 0.5)
log d
Std. Dev. (0)
Real(0.1, 1.5)
Mean (μ²)
Real(-0.55, 0.5)
log
Std. Dev. (a)
Real(0.1, 1.5)
M4
Mean (μ))
Real(-2.0, 0.0)
log( b)
Std. Dev. (0)
Real(0.1, 1.5)
Mean (j))
Real(-0.5, 0.5)
log
Std. Dev. (T)
Real(0.1, 1.5)
HPO Error per Trial
Cumulative Best Error over Trials
0.034
Trial RMSLE
Cumulative Best RMSLE
Default RMSLE
Detault RMSLE
Best Thal: 59 (Emor: 0.0251)
0.032
Best RMSLE So Far
0.025
RMSLE
0.030
0.0256
0.0252
0.024
10
20
Trial Number
Trial Number
Figure 7: Results of Bayesian Optimization for prior distribution hyperparameters over 60 trials. Top: RMSLE per trial, with the default RMSLE shown as dashed red line and the best trial highlighted. Bottom: Cumulative best RMSLE found over trials, compared against the default RMSLE.
Our proposed methodology utilizes several functional forms (e.g., Ma. M4) for modeling scaling law segments. The prior distributions for the parameters of these functions are detailed in the main text in Table 1. We conduct simple Bayesian optimization (BO) on the prior parameters to minimize the average RMSLE over 60 BO steps. The HPO targeted the location (mean) and seale (standard deviation) parameters of the Normal prior distributions for the logarithmic parameters of M3 and M4. The specific search ranges for these hyperparameters are detailed Table 13. The results of the Bayesian Optimization illustrated in Fig. The BO process successfully identified prior parameter configurations that yielded slight improvements in performance over our default settings. This outcome suggests that our initial procedure for selecting prior parameters was not overly aggressive and that there remains some scope for further marginal enhancements through systematic optimization of these prior hyperparameters.
E. Visualization
From the next page, we present additional extrapolation visualizations.
18

---

Bayesian Neural Scaling Law Extrapolation with Prior-Date Data Fitted Networks
1.0
Task: ImageNet, S-shot Model: BIT/30133
1.0
Task: imagellet, S-shot Model: BIT/5001
1.0
Task: im ageflet, 5-shot Model: Mix/8/L6
0.8-
0.8-
0.8
Test Erron Rate
Test Error Rate
Test Erro Rate
0.6-
0.6
0.6
0.4
0.4
0.2
0.0
Training Data Size
Training Data Size
Training Data Size
Task: ImageNer, S-shet | Modal: MIXL/16
Task: ImaguNet, S-shot] Model: VITI8/16
Task: ImageNet, S-shoz] Model: VIT/S/1é
1.0
0.8
Test Rate
Test Erro Rate
Test Error Ritt
0.0
10
10"
10
10'
10"
10"
Training Data Size
Training Data Size
Training Data Size
807
10-shot Model: B/T/S0/1
1.0
Modell MiX/8/16
Rate
Test Eiror Rate
Test Error Rate
0.6-
0.2
0.2
107
10a
10a
10'
10a
10*
107
10e
10
Training Data Size
Training Data Size
Training Data Size
ImageNet, shot
1.0
ImageNet. Medel:
1.0
ImaeNeet shot
0.8
0.8
Test Rate
Test Error Rate
Text Error Rabe
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
107
10"
107
0.0
10'
10a
10"
0.0
10"
10P
Trainin Data Size
Training Data
Training Data Size
Task: Imagettet, 25-ahat Model: 01
Task: Imagettet, 25-4hot Model: BIT/50/1
ImageNett 25-shot Model: MIX/8/16
1.0
1.0
1.0
0.8
0.8
0.8
Te4t Error Rate
Test Error Rate
Text Error Rote
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
107
10a
0.0
10
10a
10a
0.0
10*
Training
Task ImageNet, 25 shet Model: MIVU16
Task: ImageNet, 25 shot Model: VITIB/L6
Tosk:: ImageNet: 25 shoth Model: VIT/S/16
1.0
1.0
1.0
0.8
0.8
0.8
Test Error Rate
Test Error Pate
Test Error Rate
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10a
10a
0.0
107
102
0.0
10P
10'
Training Data Size
Training Data
Training Data Size
Figure 8: Visualization of extrapolation results ImageNet.
19

---

Bayesian Neural Scaling Law Extrapolation with Prior-Date Data Fitted Networks
1.0
Task: CIFAR-100, 5-shot] Model: DIT/L01/3
1.0
Task: CINAR-100,5-500 Model: BIT/5011
1.0
Task: CIWA--00, 5-shot Model: MIX/W116
0.8-
0.8
0.8
Test Error Rate
Test Error Rate
Test Erto Rate
0.6-
0.6
0.6
0.4-
0.4
0.4
0.2
0.2
16a
0.0
10'
169
0.0
1.2
Training Data Size
Training Data Size
Training Data Size
Task: CIRAN-100, S-shot| Model: MXX1116
Task: OFAR-100, S-shet Model: VIT//16
Task: CIRAR-100, 5-shot| Model: VIT/5/16
1.0
1.0
0.8
0.8
Test Error Rate
Test Error Rate
Test Error Rate
0.6
0.4
0.2
10
10"
10
10
0.0
1d
10"
Training Data Size
Training Data Size
Training Data Size
Tisl: CIFAR-100, BIT/101/3
Task: CIFAR-100, 10-shot Model: BIT/50/1
1.0
CIFA8-100. 10-shet Model: MIX8/16
0.8
0.8
Test Rate
Test Eiror Rate
Test Error Rate
0.6-
0.4
0.2
0.2
107
10
10a
16
10
0.0
107
10f
10
Training Data Size
Training Data Size
Training Data Size
1.0
shot Modell
1.0
100,
Model: VIT/8/26
1.0
Task: CIFAR 100, Model: VIT/S/16
0.8
0.8
0.8
Test Error Rate
Test Error Rate
Test Error Rabe
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10'
10
10a
0.0
10'
10
10p
0.0
10a
10"
Training Data Size
Training Data
Training Data Size
Tosk: CIFAR-100, 25-shot| Model: BIT/101/3
CIFAR-100, 25-shot Model: RIT/50/1
Task: CIFAR-100. 25-4hot Models MiX8/16
1.0
1.0
1.0
0.8
0.8
0.8
Te-tt Error Rate
Test Error Rate
Text Error Rste
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
10
16a
107
10a
10
10'
10'
10*
Training Data
Training
Training Data
Task: CIAA-100,25 25-hhot Model: MIVU16
Task: CIFAR 100. 25 shet | Model: VIT/B/26
Task: CIFAR 100, 25-shot | Nodel:
1.0
1.0
1.0
0.8
0.8
0.8
Test Ertor Rate
Test Error Pate
Test Error Rate
0.6-
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
107
10a
10a
10a
0.0
10P
Training Data Size
Training Data Size
Training Data Size
Figure 9: Visualization of extrapolation results CIFAR-100.
20

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
1.0
Task: Blinds 200, 3-shot Model: BID/101
1.0
Task: Birds 200, 5-shot | Model: BIT/50/1
1.0
Thsk: Binds 200, 5-shot | Model MIMB8118
0.8
0.8
0.8
Test Erron Rate
Test Error Rate
Test Erro Rate
0.6-
0.6
0.6
0.4
0.4
0.4
0.2
0.0
Training Data Size
Training Dota Size
Training Data Size
Task: Birds 200, S-shot| Model MIXI/26
Task: Birds 200, S-shot Model: VIT/8/16
Task: Birds 200, S-shot Model:
1.0
0.8
0.8
Test Error Rate
Test Erro Rate
Test Error Rite
0.6
0.4
10'
10
10"
0.0
10'
10
Training Data Size
Training Data Size
Training Data Size
Task: Birds 200, Model: BIT/101//
Task: Birds 10-shet Model: BIT/S0/1
1.0
Birds 10-shat Model:
0.8
Test Rate
Test Eiror Rate
Test Error Rate
0.6-
0.4
0.2
0.2
10'
10a
16
10'
10a
10
107
10a
10"
Training Data Size
Training Data Size
Training Data Size
shotl
1.0
Task:
Nodel:
1.0
0.8
0.8
0.8
Test Rate
Test Error Rate
Test Error Rabe
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10'
10a
100
10'
10"
10a
0.0
10
107
Training Data Size
Training Data
Traini Data Sibe
1.0
Bints 200, 25.shot BIT/101/8
1.0
Tisk: 200, 25-hot Model: BIT/50/1
1.0
Task:
25-shat Model: MIX/8/1
0.8
0.8
0.8
Te-t Error Rate
Test Error Rate
Text Error Rote
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10a
10
0.0
107
10
10a
0.0
10a
10"
Training Data
Training
Training
Task: Birds 200, shot] Model: MIXU16
Task: Birds 200, 25 25-shct|
Model: viTB/16
Task: Birds 200. shot Model VITI5/16
1.0
1.0
1.0
0.8
0.8
0.8
Test Ertor Rate
Test Error Pate
Test Error Rate
0.6-
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10a
10P
0.90
0.0
10a
Training Data Size
Training Data
Training Data Size
Figure 10: Visualization of extrapolation results Birds 200.
21

---

Bayesian Neural Scaling Law Extrapolation with Prior-Date Data Fitted Networks
1.0
Trsk: Caltech1o1, S-shot Model: BIT/101/3
1.0
Tesk: Caltech1U1, 3-shot Model: B/7/5011
1.0
Task: Caltech2U), 3-shot Model: M00/I1
0.8-
0.8
0.8
Erron Rate
Test Error Rate
Test Erto Rate
0.6-
0.6
0.6
0.4
0.4
0.2
0.0
Training Data Size
Training Data Size
Training Data Size
Task: Caltach101, S-shot] Model: MXX1/16
Task: Caltech101, S-hhet Model: VIT/8/16
Task: Caltech101, S-shot Model: VITS/16
1.0
0.8
Test Eroor Rate
Test Erro Rate
Test Error Rato
10'
10
10
10
10
Training Data Size
Training Data Size
Training Data Size
1usc Caltech101, 10-shot Model: BIT/101/3
1.0
10-shot Model: BIT/S0/1
1.0
Rate
Test Eiror Rate
Test Error Rate
0.2
0.2
107
10a
107
16a
107
10
Training Data Size
Training Data Size
Traininng Data Size
Task:
MXXL116
1.0
Tasic
1.0
0.8
0.8
Test Rate
Test Error Rate
Text Error Rabe
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10'
10F
0.0
10'
10"
0.0
10a
Training
Training Data
Training Data Sibe
1.0
Caltech101, 25-ehot Model: BIT/101/3
1.0
Caltech101, Model: BIT/Sq11
1.0
Task: Caltech101, 25-hot (Model: MIX8/16
0.8
0.8
0.8
Te4t Rate
Test Error Rate
Text Error Rote
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
10'
10
10a
Training
Task: Caltoch101. 25-shot Model: MXXA/1
Tasic Caltoch101, 25 shot Model Vittar16
Task: Caltech101, 25- 25-chot Model: VIT/5/16
1.0
1.0
1.0
0.8
0.8
0.8
Test Error Rate
Test Error Rate
Test Error Rate
0.6-
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10F
0.0
100
0.0
Training Data Size
Training Data
Training Data Size
Figure 11: Visualization of extrapolation results Caltech101.
22

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Model: Enc, Dec
Model: Enc, 28 Dec
Model: Enc, Dec
1.0
1.0
1.0
0.8
Cross-Enrron Enrropy
0.8
test Cross-Encropy
0.8
Test Cross-Entrop
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
10
107
10
0.0
106
107
10
0.0
10
10
10a
Training Data
Training Data
Training Data
Model: Dec-only
Model: TEnc-LSTM
1.0
1.0
Test Cross-Entropy
0.8
Test Cross-Entropi Entropy
0.8
0.6
0.6-
0.4
0.4
0.0
10
107
10"
0.0
10"
10?
10
Training Data Size
Training Data Size
Figure 12: Visualization of extrapolation results neural machine translation (NMT).
Model: 1.07e+09
Model: 4.53e+08
Model: 2.62e+088
1.0
1.0
1.0
Test Cross-Entropo
0.8
Test Cross-Entropo
0.8
Test Cross-Entropy
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10
10
10
10
0.0
1010
10
Training
Training
Training
Model: 1.34e+08
Model: 1.68e+07
1.0
1.0
Test Cross-Entropy
0.8
Test Cross-Entropy Entropy
0.8
0.6
0.6-
0.4-
0.4
0.2
0.0
10
10at
0.0
1030
10
Training Data Size
Training Data Size
Figure 13: Visualization of extrapolation results language modeling (LM).
23

---

Bayesian Neural Scaling Law Extrapolation with Prior-Date Data Fitted Networks
understandino.
understanding,
linguistic mappings,
1.0
1.0
0.8
0.8
Test Erro Rate
Test Errer Rate
0.6
Test
0.2
10th
10tt
10.00
100
0.0
10
101
Training Data Size
Training Data Size
Training Data Size
linguistic mappings, 2-shot
Task: mult data wrang ling, 1-shot
Task: mult data wrangling, 2-shot
1.0
1.0
1.0
0.8
0.8
0.8
Test Rate
Test Error Rate
Test Error Rate
0.6-
0.6-
0.6-
0.4-
0.4
0.4
0.2
0.2
10
10
1036
10
10
10
Training Data Size
Training Data Size
Training Data Size
Task: qa wikidata, 1-shot
Task: qa wikidala, 2-shot
Task: unil conversion, 1-shol
1.0
1.0
1.0
0.8
0.8
0.8
Text Error Rate
Test Error Rate
Test Errer Robe
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10
10
0.0
10:0
10u
0.0
10
1031
Training Size
Training Data
Training Data Sire
Task: conversnn,,,,on, 2-shot
1.0
0.8
Test Error Rate
0.6
0.4
0.2
0.0
10
10
Training
Figure 14: Visualization of extrapolation results Big-Bench (BB).
24

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
rinva hypo
minval hyp1
minwal hyp2
minwal typ3
11.0
11.0
11.0
11.0
10.3-
10.5
10.5
Un Melton
20.5
9.3
9.5
9.5
9.5
121
10
131
12'
301
30'
10
30'
wilth
With
widci
minos] hyp4
minuel hyp5
minwel hyp5
minsel hyp7
11:0
11.0
11.0
11.0
10.5
10.3
10.5
20.5
3.5
5.0
107
0:5
12
107
30
107
minwal hype
mivval hyp9
minval hyp
mivval hyp
11:
11
30
Wideron Lock
9
105
302
207
102
20P
finaival hypo
finalval hyp
finatval hyp2
finalval hyp3
Widdrbo Loos
131
10
Modul
hyp4
bypf
hyp?
La3a
Valkubb Lo55
Vitrraton
12
13a
Modul
finalval
finalval
finalval
finaival
valrgg on Los
Mooel
Modell Witth
Model Witth
Figure 15; Visualization of extrapolation results NanoGPT-Bench (Nano).
25

---

Bayesian Neural Scaling Law Extrapolation with Frior-Data Fitted Networks
clfar1D-mcon-nnouugggd
Cutoff=40%
Cuboff=00%
Cutoff=80%%
Cutoff=100%
Contistt
Nest Urrer
6
6
-6
Targe:
(M4
4
.4
20
100
120
60
80
100
120
20
40
80
80
100
120
20
00
100
120
-7
Conteat
Test Ereer
Cutom
Tarpet
15
-5
MINC JSSs
-3
-3
20
100
120
60
80
100
120
20
40
80
120
20
100
120
Contect
Test Urror
8-
6
6
Terpe:
LC-MFN
-4-
4-
-4
160
120
8b
20
40
20
100
120
Censoct
Test treer
5.
Cutoff
Target
50
.5
MSLPFFN
-3
33
-3
20
100
120
60
80
100
120
20
40
60
100
120
Model Width
Model Width
Model Width
Model Width
cifar10-mcnn-p100e
Cutoff=40%
Cutoff=60%
Cutoff=80%
Cutoff=100%
Conteat
Test Uiror
Target
1.
.5-
(M4)
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
Content
Test Lrvor
Cutof!
Tarpet
MIVC JISU
-4
20
100
120
80
100
120
20
40
100
120
Contest
Test Urrer
Terpe:
Cuteft
.5
LC-11N
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
Correct
Test Crorr
Cutoli
Tarpet
-4.
4
20
100
120
80 Width
100
40
60
100
120
Model Width
Model
Model Width
Model Witthh
Figure 16: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of 8.
26

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
cifar11--mcnnp22--sd
Cutoff=40%
Cutofi-60%
Cutoff-80%
Cutofi-100%
Conteat
Text Crrer
Tarpe:
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
.7
Content
Target
Test trror
Cutoff
MOVE (1N51)
5-
-5
20
100
120
80
120
20
40
100
120
Terget
Test Urror
Dutoff
LC-PTN
20
100
120
60
80
100
120
20
40
60
80
100
120
20
00
100
120
.7
Comrect
Terpet
Test trrer
Cueol
HSLPP1
-5
20
100
120
60
80
100
120
20
40
120
100
120
Model Width
Model Width
Model Width
Model Width
cifar10-menn-p22-sg--ree
Cutoff-40%
Cutoff-60%
Cutoff 100%
.7
Contraa
Test Ereor
(Mte
.5
20
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
Contect
Tarpet
Test treor
Cutoff
MCVC S5SU
20
100
120
80
120
40
100
120
.7
Test Uiror
6
.6
Targe:
Gutoff
1C-PTN
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
Terpet
Test Uroor
Cueoff
NSL-91
20
100
120
60
80
100
120
20
40
100
120
Model Width
Model Width
Model Width
Model Width
Figure 17: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of 8.
27

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
cifrr10-meem--22-sgi-ree
Cutoff=40%
Cutofi-60%
Cutoff-80%
Cutoff-100%
Conteat
Test trrar
Tarpe:
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
.7
Content
Target
Test treor
Cutoff
MONC (1N51)
20
100
120
80
120
20
40
100
120
Terget
Test Urror
Dutoff
20
100
120
60
80
100
120
20
40
60
80
100
120
20
00
100
120
.7
Comrec
Terpet
Test Errer
Cueol
HSLPP1
20
100
120
60
80
100
120
20
40
120
100
120
Model Width
Model Width
Model Width
Model Width
cifar10-mennnp20-so--re
Cutoff-40%
Cutoff-60%
-B0
Cutoff 100%
Contraa
Test Error
6-
.6-
(Mte
12
5-
-5-
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
80
100
120
77
Contect
Test troo
Cutoff
Tarpet
MCVC JNSU
20
100
120
80
40
100
120
Test trror
Targe:
Gutoff
1C-PTN
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
.7
Terpet
Test Urror
Cueoff
NSL-91
20
100
120
60
80
100
120
20
40
100
120
Model Width
Model Width
Model Width
Model Width
Figure 18: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of 8.
28

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
cifrr1--meen--22-sgg-reep
Cutoff=40%
Cutofi-60%
Cutoff-80%
Cutofi-100%
Conteat
Test trrer
Tarpe:
20
40
80
100
120
40
60
80
100
120
20
40
60
80
100
120
20
40
80
100
120
Content
Target
Test treor
Cutoff
HOVE (1N51)
--
-5-
20
100
120
80
120
20
40
100
120
2
-7
Terget
Test Urrer
Dutoff
20
100
120
60
80
100
120
20
40
60
80
100
120
20
00
100
120
Comrect
Terpet
Test trrer
Cutol
HSLPP1
-5
20
100
120
60
80
100
120
20
40
120
100
120
Model Width
Model Width
Model Width
Model Width
cffar10-resnet1kk550kkk
Cutoff-40%
Cutoff-60%
Cu0off-B0%
Cutoff 100%
Contrat
Test Ereor
(ste)
13
-3
10
20
30
50
60
10
30
40
50
60
10
20
30
40
50
60
10
20
30
50
60
Contect
Test treor
Cutoff
Tarpet
MCVC J9SU
20
20
60
30
50
60
30
10
50
-5
.5
Test Liror
Target
Gutoff
IC-PTN
20
20
30
50
60
10
30
40
50
60
10
20
30
40
50
60
10
20
30
50
60
Terpet
Test Iroor
Cueoff
NSL-FN
20
20
60
30
00
50
60
20
30
40
10
50
60
Model Width
Model Width
Model Width
Model Width
Figure 19: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of
29

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
chfa1100-renne118----aaaam
Cutoff=40%
Cutofi-60%
Cutoff-80%
Cutofi-100%
Conteat
Text Ereer
Tarper
6
55
-5-
10
20
30
40
50
60
10
30
40
50
60
10
20
30
40
50
60
10
20
30
50
60
.8
Content
Target
Test Erter
Cutoff
MDVC (IN51)
10
60
50
20
30
10
Test Uirer
Dutoff
5n
-5
10
20
50
60
30
40
50
60
10
20
30
40
10
20
50
60
50
.8
Conrect
Terpet
Test Eneer
.7
Cutoft
HSL-P1N
10
60
30
40
50
60
20
30
40
10
20
50
60
Model Width
Model Width
Model Widthh
Model Width
dd-transatton-smmpe----------
40%
Cutaff-60%
Cutoff-80%
Cutoff-1005
Contect
12
Test 0005
20-
MCMC (ot))
7-
7-
25000 50000 75000 100000125000
25000 58000 75000 100000125000
25000 50000 75000 100000125000
25000 50000 75800 100000125000
Cemmect
12
12
12
Test Loos
10-
10
10-
10-
NCMC (BNSL)
25000 75000 100000120000
25000 50000 75000 100000125000
25000 50000 75000 100000125000
25000 75000 100000125000
12
Tirget
Test Lo3s
Curaff
10
10:
IC-PFM
7-
7-
25000 50000 75000 100000123000
25000 50000 75000 100000125000
25000 50000 75000 100000125000
25000 50000 75000 100000125000
Context
12.
12.
12.
12
Test Loos
Cunet
10.
10
10
NSL-FTM
25000 50000 75000 000000150000
25000 50000
75000 Sample
1000001220000
25000 50000 75000 100000125000
25000 50000 75000 100000125000
Sample
Sample Size
Sample
Figure 20: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of 8.
30

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
ddtranaationsaamgessuggg///ssss
Cutoff=40%%
60%off-60%
Cutoff-80%
Cuteff-100%
Contect
Test Loos
Tarpet
Lake)
8-
25000 50000 75000 100000250000
25000 50000 75000 100000250000
25000 50000
75000 100000120000
25000 50000 75000 100000150000
12
12-
Carroot
Test Loos
10
10
10
10-
Ouesf
MCMC (DNS))
25000 50000 75000 105000120000
25000 50000 75000 100000125000
25000 50000 75000 10000125000
25000 50000 75000 100000125000
12
Target
Test Loos
6-
25000 50000 75000 100000120000
25000 50000 75000 100000125000
25000 50000 75000 100000125000
25000 50000 75000 100000120000
12
12
12
Contect
Test Loss
10
10
10
10
Cutett
NSL-FFN
25000 S0000 75000 100000 100000125000
25000 50000
75000 100000125000
25000 50000
75000 120000125000
25000 50000
75000 100000125000
Sample
Sample Size
Sample Size
Sample Size
df18k
40%
Cutaff-60%
Outoff-80%5
Cutoff-1000%
Contect
"arge:
Test Locs
12
(Mte)
10
50
100
150
200
50
100
150
200
100
150
200
50
100
150
200
Conkeett
14
24
34
14
Test Lo55
Outoff
12
22
12.
12
WCNC JANSU
10
10
10
8
50
200
150
200
100
150
50
100
Gartext
12-
12-
Sarpet Catol! LC-PPM
Test
50
200
150
200
50
300
150
200
50
100
130
200
50
100
150
200
Context
1.4
24
24
14:
Test Loos
12
12
12
12
10
20
20
10
50
300
15.0
200
50
100
150
200
100
150
50
100
150
200
Embedding Dimension
Embedding Dimension
Embedding Dimensson
Embedding Dimersion
Figure 21: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of 8.
31

---

Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
df4k
4u0off=40%
Cuteff-60%
Outeff=8015
Cutoff-100%
20
Contect
Tarpec
Test Loos
15-
15-
15
100
10
10
10
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
20
20
20
20
Conbentt
"arrge:
Test Loos
Cuterf
15.
15
15-
MCNC (TMSS)
50
200
150
200
100
150
50
100
20
1est Uo3s
15
100
20
10
10
50
200
200
50
150
200
100
150
200
50
100
150
200
20
20
20
Contect
Test Loss
Culatt
15
35
15
15
NSL-119
50
100
150
200
50
100
150
200
50
100
150
50
100
150
200
Embedding Dimension
Embedding Dimension
Embedding Dimension
Embedding Dimension
nip-model-dd
40%5
Cutoff-60%
-utoff-80%
Cutoff-100%
7
Contrat
Rest Loos
6
(ste)
4
1.00
200
300
400
500
300
200
300
400
500
100
200
300
400
500
1.00
200
300
400
500
Contect
Tarpet
Test Loss
Cutoff
MCVC SNSU
5-
100
300
500
300
200
400
200
300
400
500
1000
300
500
Target
Test Loos
Gureff
LG-PIN
1.00
200
300
400
500
300
200
300
400
500
100
200
300
400
500
1.00
200
300
400
500
Terpet
Test Loss
Cueoff
NSL-91N
5-
5
100
300
400
500
300
200
400
100
200
300
400
500
100
300
500
Embedding Dimension
Embedding Dimension
Embedding
Dimension
Embedding Dimension
Figure 22: Visualization extrapolation results on double descent (DD; Nakkiran et al., 2021), Part of 8.
32

---

Bayesian Neural Scaling Law Extrapolation with Prior-Date Data Fitted Networks
pcchcffarl00-mmm------gg--
Cutoff=40%
Cutoff=60%
Cutoff=80%
1u0%
Contant
Yest PestCrrr
Cutor!
-7
MCMC (sse)
10
20
50
60
10
30
40
50
10
20
30
40
60
10
20
50
60
165t Enror
8
-5
-5
10
20
60
30
50
30
10
50
Contect
Test Crrer
Cutoft
Tarpe:
-7
(C: (C-FFN
-6-
100
20
40
50
60
10
20
30
40
50
60
100
20
30
40
50
660
100
20
30
50
60
1est Enor
0
.8
.5
.5
10
60
30 Model
40
30
10
50
Model Width
Model Width
Figure 23: Visualization of extrapolation results on double descent (DD; Nakkiran et al., 2021), Part 8 of 8.
33


Reviews:
[{"review_id": "v0O0GGTdWC", "reviewer": "Reviewer_1HYA", "summary": "In this paper, the authors investigate the use of Prior-Data Fitted Networks (PFNs), a Bayesian framework for estimating neural scaling law curves. Existing methods typically produce point estimates, which fail to capture the true scaling law dynamics across different scenarios (e.g., double descent, flat cut-offs) and do not provide uncertainty estimates. These uncertainties are crucial for designing experiments based on early observations of the scaling curve. To address this, the authors introduce a family of priors that effectively represent scaling laws in large model training, synthesizing new samples from these priors to train PFNs, which are based on a standard transformer architecture. The authors demonstrate that their approach, referred to as NSL-PFNs, not only quantifies Bayesian uncertainty but also yields strong performance in terms of RMSLE and likelihood metrics. Extensive evaluations across image and text domain benchmarks, along with ablation studies, highlight the contributions of each component in their framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments. We address each of your comments below. If you have any further questions or comments, we would be happy to discuss them during the author-reviewer interaction phase.\n\n\n---\n\n>**[Q1]** Main novelty of this work compared to LC-PFN at both high- and low-level?\n\nThank you for the question. The main novelty of our work compared to LC-PFN [1] lies in both the high-level approach and low-level modeling choices:\n\nHigh-level differences:\n- Our work is the first to apply a **Bayesian** framework to the problem of **neural scaling law extrapolation**, whose functional distribution and characteristics are much different from those of learning curves the LC-PFN focused on.\n- We further leverage uncertainty estimates to actively guide data acquisition, demonstrating the **practical utility of uncertainty in reducing the cost of collecting high-quality scaling data** — something not explored in LC-PFN.\n\nLow-level differences:\n- As you mentioned, we carefully design **a novel functional prior tailored to neural scaling laws**, informed by limitations of previous empirical approaches. Our prior can capture complex and chaotic behaviors such as double descent, enabling accurate extrapolation and reliable uncertainty estimates even under challenging real-world scenarios. In contrast, the prior of LC-PFN does not cover such complex patterns.\n- We further propose a simple **context-regressive learning objective** that not only improves extrapolation performance but also enables Bayesian active learning — something that LC-PFN lacks.\n\nWe will make these distinctions more explicit in the revision.\n\n---\n\n>**[Q2]** In Table 1, can the authors provide a reference that demonstrates the priors used for the coefficients? How do these choices impact the overall performance? \n\n- Thank you for pointing it out. For the design of our functional prior, we followed the same procedure used in LC-PFN [1]. Specifically, we collected some of the well-known scaling law curves from various domains, and tried to match by the eye the shape of our functional prior to the actual curves by manually controlling the parameters of the prior.\n- Importantly, when controlling the prior parameters, we have assumed that the actual values of the scaling law curves are **not** accessible. Also, for the ColPret dataset (whose size exceeds the sum of all the other datasets), we did not even access the shape of the scaling law curves, but just test our model on the dataset directly. Nevertheless, the model still achieved strong performance on ColPret, indicating that our prior generalizes well across tasks.\n- Further, during the rebuttal period, we conducted a simple Bayesian optimization (BO) on the prior parameters to minimize the average RMSLE over 60 BO steps. As shown in [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/blob/rebuttal/rebuttal/hpo_error.pdf), the BO led to slight improvements on the performance, which indicates that our procedure of tuning prior parameters was not aggressive and there remain some rooms for further improvements of our prior.\n- We will discuss the above in the revision.\n\n---\n\n>**[Q3]** While the authors follow Muller et al. for implementing the PFN, can they clarify why positional encoding has been omitted? \n\n- The original PFN paper (Muller et al. [2]) also omits the positional encoding for the following two reasons. \n- In PFNs, when encoding the context points $\\mathcal{C}=\\lbrace(x_i,y_i)\\rbrace_{i=1}^M$, the input feature $x_i$ itself already contains the \"positional information\" in the input space.\n- Further, PFNs approximate the posterior predictive distribution $p(y^*,x^*|\\mathcal{C})$, which is permutation-invariant w.r.t. the ordering of the context points $\\mathcal{C}=\\lbrace(x_i,y_i)\\rbrace_{i=1}^M$, by definition. The same applies to our model as well as LC-PFN [1].\n- We will clarify this in the revision.\n\n---\n\n>**[Q4]** Update Figure 4 to better highlight the uncertainties for each method?\n\n- Thank you for the suggestion. To improve clarity, we have updated those figures by separating the uncertainty plots for each method. [This external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/tree/rebuttal/rebuttal/DD_visualization) includes all those visualizations. We will include them in the appendix of the revision.\n\n---\n\n**References**\n\n[1] Adriaensen et al., *Efficient Learning Curve Extrapolation via Prior-Data Fitted Networks, NeurIPS* 2023.\n\n[2] Müller et al., *Transformers Can Do Bayesian Inference, ICLR* 2022", "claims_and_evidence": "Yes, the claims are supported by relevant evidence.", "methods_and_evaluation_criteria": "Yes, the authors have considered relevant baselines and benchmarks datasets.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the experimental setup and design is sound.", "relation_to_broader_scientific_literature": "In my humble opinion, the contributions are significant within the broader context of the scientific literature. However, there are some questions I have raised that require further clarification.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "In my view, the primary contribution of this paper lies in the development of a novel family of priors that enhance the training of PFNs, which constitutes the core strength of the work. Other aspects of the paper, as cited by the authors, have been explored in prior research. The emphasis on Bayesian uncertainty in the context of scaling law estimation, given only a set of observations, is particularly valuable for practitioners. It can eventually provide credible intervals and practical insights for conducting experiments at varying scales. Additionally, the paper is exceptionally well-written, clear, and almost free of typos.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can the authors clarify the main novelty of this work compared to Adriaensen et al.? Is it primarily the definition of the functional priors? It would be beneficial to explicitly highlight both high- and low-level differences.\nIn Table 1, can the authors provide a reference that demonstrates the priors used for the coefficients (e.g., $\\log a \\sim \\mathcal{N}(-1, 0.6)$)? How do these choices impact the overall performance?\nWhile the authors follow Muller et al. for implementing the PFN, can they clarify why positional encoding has been omitted?\nCan the authors update Figure 4 to better highlight the uncertainties for each method in the extrapolation regime? Currently, it is difficult to interpret and extract insights from the figure.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d5LceCZdkv", "reviewer": "Reviewer_ogXf", "summary": "This paper proposes the use of Bayesian Neural Networks (BNNs) to model deep learning scaling laws.\nScaling laws in deep learning have gained a considerable amount of interest recently, as they provide quantitative ways to estimate, for example, the amount of computational resources, model size, or dataset size needed to achieve certain performance.\nTraditional approaches estimate these neural scaling laws using simple parametric assumptions (e.g., power law, exponential), and by carrying out \"point estimation\", that is without accounting for any uncertainty in predictions/extrapolations.\nBNNs allow one to tackle these two limitations by employing flexible over-parameterized neural network models and by treating these in a Bayesian way.\nThe added benefit of this approach is the possibility to carry out active learning. \nOne of the main contributions is the use of Prior Fitted Networks (PFNs) to establish sensible neural scaling laws that can be generated from the model prior to conditioning on any data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments. We address each of your comments below. If you have any further questions or comments, we would be happy to discuss them during the author-reviewer interaction phase.\n\n\n---\n\n>**[Q1]** Additional experiments - calibration of the predictive uncertainties.\n\n- Thank you for the suggestion. In the following, we use **mean square calibration error (MSCE)**, the calibration metric used in [1] to assess predictive uncertainty. As shown in [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/blob/rebuttal/rebuttal/main_table_with_calibration_error.md), **our method demonstrates notably better calibration performances than the baselines**. \n- The gap is more prominent especially on the Double Descent (DD) which exhibits **highly chaotic behavior**. In such regimes, well-calibrated uncertainty is particularly important for reliable extrapolation. Furthermore, our method performs especially well on ColPret, the most recent and largest collection of **realistic neural scaling laws**, including models like GPT-3. \n- These results show our model’s strength in handling challenging and real-world scaling behaviors. We will include those findings in the revision.\n\n---\n\n>**[Q2]** Additional baselines - Bayesian linear regression with basis functions.\n\n- Thank you for the suggestion. We agree that Bayesian linear regression (BLR) [2] with appropriate basis functions is a natural and lightweight approach for 1D regression and extrapolation. \n- To evaluate, we conducted additional experiments with the following baselines: 1) BLR with neural network basis functions, 2) BLR with polynomial basis, 3) BLR with RBF basis, 4) BLR with Fourier basis, 5) BLR with sigmoid basis, 6) BLR with spline basis, and 7) Deep Kernel GP (DKGP) [3]. We use log marginal likelihood for tuning the parameters, such as the parameters of the NN as a BLR basis function. We provide qualitative examples of the BLR with NN baseline and DKGP at [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/tree/rebuttal/rebuttal/BLR_visualization) and [this externel link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/tree/rebuttal/rebuttal/DKGP_visualization), respectively.\n- As shown in [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/blob/rebuttal/rebuttal/table_with_dkgp_and_blr.md), **those baselines are not competitive** in the context of neural scaling laws. This is because, while they offer interpretability and closed-form inference, to our knowledge **it is not straightforward to encode a set of inductive biases about neural scaling laws into the basis or kernel functions**. This is why we originally did not consider those baselines.\n- In contrast, our PFN-based method provides a flexible framework to encode our prior belief on the shape of the functions. We can use any functional prior as long as we can efficiently sample from it, which is one of the biggest advantages of using PFNs. This is precisely why we propose to use PFNs for the problem of extrapolating neural scaling laws.\n- We appreciate your further suggestions as to how to effectively encode the prior belief on the scaling law behaviors into the corresponding basis or kernel functions. As long as time allows, we will try our best to do the additional experiments following your suggestions.\n\n---\n\n>**[Q3]** I would have liked to see a much more comprehensive validation.\n\n- We emphasize that, to the best of our knowledge, we have already collected **all publicly available datasets of neural scaling laws**, including recent and diverse settings such as **ColPret** (large-scale LLMs), **DD** (chaotic double descent behavior), and other benchmarks from $\\mathcal{M}_4$ and BNSL. These datasets span a wide range of domains and scaling patterns, including smooth, non-smooth, and chaotic regimes.\n- Across all of these settings, our method consistently shows **strong performance**, demonstrating its general applicability.\n- Also, during the rebuttal period, we conducted the two additional experiments following your suggestion. 1) We additionally consider Bayesian linear regression and deep kernel GP baselines and find that they largely underperform our method. 2) We demonstrate that our method shows superior calibration performance than the baselines. \n- Our paper also includes Bayesian active learning experiments to demonstrate the usefulness of uncertainty, which has not been discussed in the LC-PFN paper. If you find it insufficient, please feel free to suggest any additional experiments in detail, and we would be happy to try our best to do the experiments as long as time allows.\n\n---\n\n**References**\n\n[1] Kuleshov et al., *Accurate Uncertainties for Deep Learning Using Calibrated Regression. ICML* 2018.\n\n[2] Bishop et al., *Pattern Recognition and Machine Learning*, 2006.\n\n[3] Wilson et al., *Deep Kernel Learning, AISTATS* 2016", "claims_and_evidence": "The claims are that the proposed approach gives flexible representation for a prior over neural scaling laws and that the proposed model and Bayesian treatment yield accurate extrapolations.\nAnother claim is that uncertainty in scaling laws can be used effectively to carry out active learning leading to computational savings.\n\nThese claims are generally supported by the experiments provided in the paper.\nHowever, given that this paper is mostly experimental in nature, I would have liked to see a much more comprehensive validation, including more active learning scenarios and more competitors.\nAlso, I think it would have been useful to study the calibration of the predictive uncertainties; at the moment, it is not clear how well the uncertainties produced by the model are calibrated.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally ok, but again I think it would be important to evaluate calibration, which is currently missing.", "theoretical_claims": "There are no theoretical developments in this paper.", "experimental_designs_or_analyses": "As far as I can tell, the experimental design and analyses in the paper are conducted properly.", "relation_to_broader_scientific_literature": "I think that the literature review on neural scaling laws is nicely captured in the paper, although I am not an expert on this.\nMaybe I would have put a bit more emphasis on alternative approaches on BNNs beyond PFNs.", "essential_references_not_discussed": "I don't think that there are other essential references to be added.", "other_strengths_and_weaknesses": "The main weakness is the formulation of a rather detailed PFN for a task that is essentially 1D regression/extrapolation.\nMy immediate reaction after reading this paper was that 1D regression can be done in closed form with Bayesian linear regression, and it can be made quite flexible through the use of basis functions.\nThis would give access to the marginal likelihood for model selection, and it would enable all the nice things that this paper promises to do quite cheaply.\nI believe that this would be a good alternative to the approach proposed in this paper, and I would encourage the Authors to think about this type of competitor. \nSection 3.1 provides a long text specifying the assumptions on the functions that BNNs can represent, whereas there is no effort in trying to encode this information in some basis functions which might ultimately yield competitive performance.", "other_comments_or_suggestions": "The paper is generally well written, so I don't have any particular comments on this.", "questions_for_authors": "Overall I think this paper has a good aim.\nI find the realization overly complicated and I think that the paper is missing some simple Bayesian competitors.\nDue to the paper being experimental in nature, I would encourage the Authors to focus their efforts on a wider experimental campaign, testing as many competitors as possible, and reporting calibration.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMr18rbM40", "reviewer": "Reviewer_zhqD", "summary": "The authors propose a method to infer scaling laws using a Bayesian approach which allows them to model predictive uncertainty. \nTo accomplish this, they rely on prior-data fitted networks (PFNs) introduced by Müller et al., (2022) and their ability to meta-learn from large amounts of synthetic data. \n\nThe method is evaluated both quantitatively and qualitatively on a range of data sets showing competitive and improved performances against several baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments. We address each of your comments below. If you have any further questions or comments, we would be happy to discuss them during the author-reviewer interaction phase.\n\n---\n>**[Q1]** Missing related work - Neural Processes.\n\n- Thank you for the suggestion. We agree that the Neural Process (NP) family, originally introduced by Garnelo et al. [1], is closely related to our approach, as both aim to meta-learn an amortized posterior inference machine over a distribution of tasks. We will include the following discussion in the revision:\n- *Neural Processes (NPs) [1] typically learn a latent variable model where context data are summarized into a global latent representation, and target predictions are made by conditioning on this latent variable. This structure allows for uncertainty modeling and task generalization. The Conditional Neural Process (CNP) [2] simplifies this by removing the latent variable and directly conditioning target predictions on the aggregated context set. The Attentive Neural Process (ANP) [3] improves upon this by incorporating attention mechanisms to better capture context-target relationships. These models have been widely applied to few-shot learning, time series modeling, spatial inference, and so on. Further extensions include Transformer Neural Processes (TNPs) [4], which replace context aggregation with attention-based sequence modeling using Transformer architectures. While conceptually related, PFNs differ from NPs in an important way: PFNs do not require real dataset to train the inference machine. Instead, PFNs directly meta-learn the posterior predictive distribution with synthetic data generated from a manually-defined prior distribution, which allows Bayesian inference instead of maximum likelihood estimation. On the other hand, NPs are basically MLE and can be seen as implicitly learning the data-driven prior.*\n\n---\n\n>**[Q2]** The reference section needs to be updated, as several references point to arxiv preprints despite the respective papers being published work.\n\n- Thank you for the suggestion. We will clarify the publication status of all references and replace arXiv citations with their corresponding published versions wherever applicable. We appreciate your attention to detail and will ensure the revision reflects the accurate and up-to-date citations as follows:\n\n- *Exploring the Limits of Large Scale Pre-Training* → ICLR 2022  \n- *An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale* → ICLR 2021  \n- *Language Models Are Few-Shot Learners* → NeurIPS 2020  \n- *Broken Neural Scaling Laws* → ICLR 2023  \n- *BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding* → NAACL 2019  \n- *A Survey on In-Context Learning* → EMNLP 2024  \n- *Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings* → ICML Workshop 2019  \n- *Scaling Laws for Neural Machine Translation* → ICLR 2022  \n- *Mamba: Linear-Time Sequence Modeling with Selective State Spaces* → COLM 2024  \n- *Training Compute-Optimal Large Language Models* → NeurIPS 2022  \n- *TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second* → ICLR 2023  \n- *Bayesian Active Learning for Classification and Preference Learning* → CoRR 2011  \n- *Adam: A Method for Stochastic Optimization* → ICLR 2015  \n- *Transformers Can Do Bayesian Inference* → ICLR 2022  \n- *In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization* → ICML 2024  \n- ...\n\n---\n\n>**[Q3]** The PFN has been trained with a rather large set of synthetic examples (1.6 million). How sensitive is the performance of the proposed approach to this size?\n\n- Thank you for the question. First of all, it is well-known that PFNs benefit from training on a large amount of synthetic data, as shown in LC-PFN and other related works. Since PFNs are trained to approximate the posterior predictive over a distribution of tasks, a large number of sampled functions is typically required for effective generalization.\n- In [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/blob/rebuttal/rebuttal/learning_curves.pdf), we provide the training losses and test performances over training iterations. We find that they keep converging as the model spends more synthetic data. Note that for each iteration, we randomly sample a batch of synthetic functions (which can be generated cheaply on-the-fly from our functional prior), hence the number of synthetic training data used is proportional to the training iterations. \n\n---\n\n**Reference**\n\n[1] Garnelo et al. *Neural Processes. ICML workshop,* 2018.\n\n[2] Garnelo et al. *Conditional Neural Processes, ICML* 2018.\n\n[3] Kim et al. *Attentive Neural Processes, ICML* 2019.\n\n[4] Nguyen et al. *Transformer Neural Processes: Uncertainty-Aware Meta Learning via Sequence Modeling. ICML* 2022.", "claims_and_evidence": "All claims (080 to 092) are provided with clear evidence throughout the paper.", "methods_and_evaluation_criteria": "The proposed method makes sense for the proposed task. PFNs and other meta-learning methods that learn from a large amount of synthetic data are an obvious fit for tasks such as neural scaling law inference, due to the ease of generating such synthetic data while at the same time having very limited real-world data.\n\nThe evaluation criteria are also broad and valid, both quantitatively, as well as qualitative visualizations.", "theoretical_claims": "There are no theoretical claims that require proof.", "experimental_designs_or_analyses": "The experimental design is well-specified and well-motivated. The analysis is sound.", "relation_to_broader_scientific_literature": "The relation to the broader scientific literature is in the application of PFNs for the task of scaling law inference. \nThe proposal does not contribute anything to the theory of PFNs but takes it as a tool to improve current neural scaling law inference techniques.", "essential_references_not_discussed": "All essential references are discussed properly.", "other_strengths_and_weaknesses": "### Strengths\n- The paper is very well written and can easily be followed.\n- The motivation is clear and justified. The evaluation is convincing.\n\n\n### Weaknesses\n- One omission from the related work discussion in Appendix A is the model family of neural processes originally introduced by Garnelo et al. in 2018. \nThese follow a similar structure and motivation to PFNs and have been adopted in a wide area of applications.", "other_comments_or_suggestions": "- The reference section needs to be updated, as several references point to arxiv preprints despite the respective papers being published work. An example is Müller et al. (2021) which has been published at ICLR 2022.", "questions_for_authors": "- Q1: The PFN has been trained with a rather large set of synthetic examples (1.6 million). How sensitive is the performance of the proposed approach to this size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udfXIP1Sjh", "reviewer": "Reviewer_uVCN", "summary": "The paper proposes using a taylored prior-fitted network to extrapolate neural scaling laws. A new prior over the set of potential curves is designed, capturing both simple power law behavior and more complex double-descent curves. The method is evaluated extensively on several data sets and shows improved performance over existing baselines.\n\n## update after rebuttal\n\nThe rebuttal resolved some issues that I had and I raised my score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful comments. We address each of your comments below. If you have further questions, we’d be happy to discuss them during the author-reviewer interaction phase.\n\n---\n\n> **[Q1]** The MCMC models appear to be specified without noise... This makes the Bayesian inference problem extremely misspecified.\n\n- Thank you for pointing this out, and sorry for the confusion. Actually, we **did account for the observational noise** in the MCMC baselines. Otherwise, we would not be able to compute the log-likelihood.\n- For example, $\\mathcal{M}_1: y=ax^{-b}$ model has two parameters $a$ and $b$, and we define the likelihood as $\\mathcal{N}(y; ax^{-b}, \\sigma^2)$, where $\\sigma^2$ is the observational noise. \n- MCMC is performed over the joint posterior: $p(a, b, \\sigma \\mid \\lbrace x_i, y_i\\rbrace_{i=1}^M) \\propto  \\prod_{i=1}^M \\mathcal{N}(y_i; ax_i^{-b}, \\sigma^2)  \\cdot p(a) \\cdot p(b) \\cdot p(\\sigma)$, where $p(a)$, $p(b)$, and $p(\\sigma)$ are predefined priors. \n- We will clarify this in the revision to avoid further confusion. Thank you again for raising this point.\n\n---\n\n> **[Q2]** The number of samples drawn (150) is tiny. Typical numbers for running MCMC chains are in the thousands.\n\n- Thank you for pointing this out. We acknowledge that 150 MCMC samples are small compared to the typical choice. We chose such a small number because there are too many scaling curves to evaluate and each MCMC inference takes too much time. For instance, applying 3000 MCMC samples to all those curves takes around **4 weeks** in total!\n- In [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/blob/rebuttal/rebuttal/LL_nsamples_and_inference_time.pdf), we compare our method against MCMC baselines with varying numbers of samples: 150, 300, 750, 1500, and 3000. To make the experiments feasible, we evaluated only on a subset of 30 scaling laws randomly sampled from the collection of all neural scaling law datasets we considered.\n- As shown in the results, increasing the number of MCMC samples sometimes leads to slightly improved performance as expected, but the differences are marginal (actually, we had already verified it before submission). Also, the increased performances still fall short of our method, while the inference time increases substantially. This supports one of our key claims: **NSL-PFN achieves strong performance with significantly better efficiency**. After meta-training, PFN-based methods provide **near-instantaneous inference**, which is the power of amortized inference.\n- We will discuss this further and provide the full evaluations in the revision.\n\n---\n\n> **[Q3]** Eq (10): The expectation is over T which doesn't appear in the expression at all. Do you mean that (Y*, X*) is one instance from T? I have no idea what (X,Y) and \"the autoregressive objective\" are.\n\n- As defined in L123–124 (the right column) in the main paper, the context and target set are denoted as $\\mathcal{C}=(X,Y)$ and $\\mathcal{T} = (X^*, Y^*)$, respectively. \n- In Eq. (10), $\\mathcal{C}, \\mathcal{T}\\sim p(\\mathcal{D})$ is thus equivalent to $(X,Y),(X^*, Y^*)\\sim p(\\mathcal{D})$. We call the training objective $\\mathbb{E}_{\\mathcal{C}, \\mathcal{T} \\sim p(\\mathcal{D})}  [\\log q (Y \\mid X, \\mathcal{C})]$ \"autoregressive\" because it is regressing $\\mathcal{C}=(X,Y)$ conditioned on the same context $\\mathcal{C}$. \n- To improve clarity, we will revise the terminology to **\"context regression objective\"**, or any terminology you have in mind. Thank you for bringing this to our attention.\n\n---\n\n> **[Q4]** The priors in Table 1 seem a bit arbitrary. How were they chosen? How sensitive are the results to their choice?\n\n- Thank you for pointing it out. For the design of our functional prior, we followed the same procedure used in LC-PFN. Specifically, we collected some of the well-known scaling law curves from various domains, and tried to match by eye the shape of our functional prior to the actual curves by manually controlling the parameters of the prior.\n- Importantly, when controlling the prior parameters, we assumed that the actual values of the scaling law curves are **not** accessible. Also, for the ColPret dataset (whose size exceeds the sum of all the other datasets), we did not access the shape of the scaling law curves, but just tested our model on the dataset directly. Nevertheless, the model still achieved strong performance on ColPret, indicating that our prior generalizes well across tasks.\n- Further, during the rebuttal period, we conducted a simple Bayesian optimization (BO) on the prior parameters to minimize the average RMSLE over 60 BO steps. As shown in [this external link](https://github.com/nslpfn-anonymous/nslpfn-anonymous/blob/rebuttal/rebuttal/hpo_error.pdf), the BO led to slight improvements on the performance, indicating that our prior tuning was not aggressive and that there remains room for further improvements.\n- We will discuss the above in the revision.", "claims_and_evidence": "The authors evaluate their new model extensively against relevant competitors and provide some ablation studies. However, I was left somewhat unconvinced. The main issue is that the researchers have so many degrees of freedom in building their model (and even in implementing some competitor methods) that it should be easy to craft a method that works better on an already available benchmark suite. Since the benchmark suite is relatively comprehensive and realistic, this may still be valuable.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "I have some issues with how the MCM alternatives were implemented.\n* The models $\\Mcal_j$, BNSL appear to be specified without noise. That is, we expect the observed curves to follow *exactly* the idealized curves. This makes the Bayesian inference problem extremely misspecified, so it's no wonder it doesn't perform well. A more reasonable approach would be to do MCMC inference on the model $y = f(x) + \\epsilon(x)$ where $\\epsilon(x)$ is, e.g., a mean-zero Gaussian process.\n* The number of samples drawn (150) is tiny. Typical numbers for running MCMC chains are in the thousands.", "relation_to_broader_scientific_literature": "The new model improves on several prior works for estimating the parameters in neural scaling laws. PFNs were already previously used for this task by [1]. The main innovation is to design priors tailored explicitly to the task and train a PFN accordingly.\n\n[1] Adriaensen, S., Rakotoarison, H., Müller, S., & Hutter, F. (2023). Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Information Processing Systems, 36, 19858-19886.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The paper is really well written.", "other_comments_or_suggestions": "* Eq (10): \n  * The expectation is over $\\mathcal T$ which doesn't appear in the expression at all. Do you mean that $(Y^*, X^*)$ is one instance from $\\mathcal T$? \n  * I have no idea what $(Y, X)$ and \"the autoregressive objective\" are.", "questions_for_authors": "To convince me about the scientific progress made in the papers, I strongly suggest the authors extend their empirical evaluation.\n1. The MCMC baselines should be implemented in a more reasonable way, esp. regarding noise model and length of the chains; see my comments above.\n2. The priors in Table 1 seem a bit arbitrary. How were they chosen? How sensitive are the results to their choice?\n\nIf these points are addressed convincingly I am inclined to increase my score. (Update: score raised to 3.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper concerns fitting the learning curves from scaling-up neural network experiments as a way to explain the neural scaling law. The authors proposed a Bayesian approach, where contributions include (1) design of the candidate curve families and principles for function prior construction, (2) prior-fitted networks which fits on data generated from the function prior in (1) using a (variational Bayesian) meta-learning style training objective, and (3) the trained PFN will then allow uncertainty estimation for the extrapolation region when tested on its fitness regarding scaling curves from real-word neural network scaling experiments. Experiments on benchmark scaling law curve fitting datasets show the method performs better regarding extrapolation when compared with deterministic methods and simple MCMC baselines over the parameters of pre-specified curve families.

Reviewers found the work to be a novel application of Bayesian methods + meta-learning approaches to the analysis of neural scaling law curves. However, reviewers also raised questions regarding the choice of the fitting model (prior-fitted network vs Bayesian linear regression with non-linear 1D features) as well as the uncertainty quality. Author rebuttal has address some but not most of the questions.

I briefly read the paper. 
- From the point of view of designing the algorithm for fitting the PFN as well as uncertainty quantification, I personally think these approaches are fairly straightforward to people coming from Bayesian neural networks and meta-learning/neural process fields. 
- However I think another strength of the paper is the careful design of the functional prior, which demonstrates a "Bayesian" spirit very well, in the sense that it recognises existing approaches and take inspirations from them as prior knowledge, and the authors also made careful design choices regarding different parts of the scaling law curve. 
- Still I would like to see an MCMC baseline where the functional prior is the designed prior in section 3.1, rather than the parametric prior over each of the function class (say M1 or M2). This can ablate the need of a prior-fitted network. The authors might argue this prior is difficult to perform exact posterior inference (via MCMC). However, I feel this is a hierarchical prior but still parametric, so perhaps methods like a combination of Gibbs and Langevin samplers can be useful.

Overall, I think this paper would be more meaningful to present to neural scaling law community, in a way to show an alternative approach to "guess" how neural networks scale :)
Decision: Accept (poster)