Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks
Authors: Steven Adriaensen
Herilalaina Rakotoarison
Samuel Müller
Frank Hutter
Venue: NeurIPS 2023 poster (2023)
Keywords: learning curve extrapolation
prior-data fitted networks
transformers
Bayesian inference
uncertainty estimation
model selection
TL;DR: We show that Prior-data Fitted Networks (PFNs) are a dramatically cheaper and higher-quality alternative to MCMC for right-censored learning curve inference, enabling practical applications, e.g., early stopping, at almost no overhead.

Abstract: Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance in earlier epochs.
In this work, we argue that, while the inherent uncertainty in the extrapolation of learning curves warrants a Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. We describe the first application of prior-data fitted neural networks (PFNs) in this context. A PFN is a transformer, pre-trained on data generated from a prior, to perform approximate Bayesian inference in a single forward pass. We propose LC-PFN, a PFN trained to extrapolate 10 million artificial right-censored learning curves generated from a parametric prior proposed in prior art using MCMC. We demonstrate that LC-PFN can approximate the posterior predictive distribution more accurately than MCMC, while being over 10 000 times faster. We also show that the same LC-PFN achieves competitive performance extrapolating a total of 20 000 real learning curves from four learning curve benchmarks (LCBench, NAS-Bench-201, Taskset, and PD1) that stem from training a wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 - 6x speed-ups on 45 of these datasets, at virtually no overhead.

Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks
Steven Adriaensen" Machine Learning Lab University of Freiburg
Herilalaina Rakotoarison"
Machine Learning Lab University of Freiburg
adriaens@cs uni ami-free meri-feibburg.cm
rakotoahêcs - uni-freiburge de
Samuel Müller
Frank Hutter
Machine Learning Lab
Machine Learning Lab
University of Freiburg
University of Freiburg
muellesa@ cs. uni-freiburg de
fhåcs. uni-freiburg.de
Abstract
Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance earlier epochs. In this work, we gue that, while the inherent uncertainty in the extrapolation of learning curves warrants Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. We describe the first application of prior-data fitted neural networks (PFNs) in this context. A PFN is transformer, pre-trained data generated from prior, to perform approximate Bayesian inference in a single forward pass. We propose LC-PFN, a PFN trained extrapolate artificial right-censored learning curves generated from parametric prior proposed in prior art using MCMC. We demonstrate that LC-PFN can approximate the posterior predictive distribution over learning curves more accurately than MCMC, while being over 10000 times faster. We also show that the same LFN achieves competitive performance extrapolating total of 000 real learning curves from four learning curve benchmarks (LCBench, TAS--ench-2211 Taskset, and PD1) that stem from training wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 -66 speed-ups on 45 of these datasets, at virtually overhead.
1 Introduction
Learning curve extrapolation [Mohr and van Rijn 2022] aims to predict how much a machine learning model will improve with more training, e.g., determine how much more training data
to collect [Cortes et al. 1993 Frey and Fisher 1999 Leite and Brazdil 2004 Kolachina et al.
2012), or define an early stopping criterion online learning Yao al. 20077] Learning curve
extrapolation has recently been widely studied to speed up automated machine learning (AutoML) and hyperparameter optimization (HPO) of deep neural networks, by discarding non-promising
configurations early [Swersky et al. 2014 Domhan et al. 2015 Klein et al. 2017 Baker et al. 2017
Chandrashekaran and Lane 2017 Gargiani et al. 2019
Wistuba al. 2022
Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite these efforts, learning curve extrapolation is not yet widely adopted in practice, e.g., state-of- the-art multi- culti--ideeityy hyperparameter optimization techniques, such as BOHB [Falkner et al., 2018). still rely on successive halving et 2017]. i.e., the crude heuristic that learning curves mostly do not cross each other.
One reason for this is that, while many learning curves are well-behaved, some exhibit chaotic behavior and are intrinsically difficult to predict accurately [Choi al. 2018). In this setting, Bayesian
approaches (Swersky et al., 2014 Domhan et al. 2015 Klein et al. 2017 Wistuba et al.] 2022].
which also quantify the reliability of their extrapolation show great potential. However, existing methods for Bayesian inference either (i) put strong restrictions on the prior, and are incapable of modeling the variable nature of learning curves, or (ii) are too computationally expensive, limiting their practical applicabilityy. Furthermore, most of this related work focuses on demonstrating the potential that learning curve extrapolation has to accelerate downstream AutOMI tasks, yet fails to fully investigate the quality of the extrapolations themselves, and to quantify the approach's s ability to handle the heterogeneity of real-world learning curves, e.g., varying performance metrics, curve shapes, divergence, heteroscedastic noise, etc.
In this work, we investigate the potential of learning curve extrapolation using prior-data fitted networks (PFNs), a meta-learned approximate Bayesian inference method recently proposed by Müller et al. 2022]. PFNs combine great flexibility with efficient and accurate approximation of the posterior predictive distribution (PPD) in single forward pass of a transformer [Vaswani et al.) [017]] trained on artificial data from the prior only. As PFNs are promising alternative Markov Chain Monte Carlo (MCMC) for approximating Bayesian inference, we compare our approach (ICCPFN) to the MCMC approach for learning curve extrapolation of Domhan et al. (2015), taking into account both the quality and the cost of PPD approximation.
In summary, our contributions are as follow:
We are the first to apply PFNs to an extrapolation task, introducing LC-PEN, the first PFN for learning curve extrapolation.
We demonstrate that LC- CFN can be more than 10000x faster than MCMC while still yielding better probabilistic extrapolations.
We show that LC-PFN does not only yield better probabilistic extrapolations prior samples, but also on real learning curves of a wide range of architectures (MLPs, CNNs, RNNs, Transformers) varying input modalities (tabular, image, text and protein data).
We demonstrate the practical usefulness of IC-PFN construct early stopping criterion that achieves 2 - -6x speedups over baselines.
To facilitate reproducibility and allow others to build on our work, we open-source all code. data, and models used in our experiments at htttps://ittubb.. wwt.tuuw.oooeeoeeeeeeeeeeeeeeeee aut http://pttmmmm//nnn
2
Related work
Learning curves and how to use them for decision makiing has been an active research area, as recently surveyed by MMhrr and van Rijn| [2022]. Most related work considers point estimates of the curve or specific property thereof [Crtes et al., 1993 Frey and Fisher| 1999 Kolachina et al., 2012, Baker et al., 2017, Kaplan et al. 2020), follows non-Bayesian approach to quantify uncertainty (Chandrashekaran and Lane 2017 Gargiani et al. 2019).
Only a few works have explored Bayesian learning curve extrapolation. For example, the Freeze- Thaw Bayesian optimization method Swersky et al. [2014) used a Gaussian process (GP) as a joint model of learning curves and hyperparameters to decide what learning run to continue for a few epochs (or whether to start a new one). The model then dynamically updated to fit the partial learning curve data. Training data grows quickly since each performance observation of the curve treated as a datapoint, making exact GPs intractable, and thus the work relies on approximate GPs. Furthermore, their approach makes strong (prior) assumptions. On top of the standard GP assumptions, they used a specialized kernel assuming exponential growth to improve extrapolation. Domhan et al. [2015) proposed a less restrictive parametric prior (see Section 3.2 for more details) and used the gradient-free MCMC method from Foreman-Mackey et al. (2013) as approximate inference method. While MCMC is very general approach, it can be sensitive to its hyperparameters (c.g..
2

---

burn-in period, chain length, etc.) and, as we will show in Section generating sufficient samples to reliably approximate the PPD may impose significant overhead. Klein al. [2017] extended this parametric prior to also capture the effect of hyperparameter settings. In particular, they used a Bayesian neural network with a specialized learning curve layer, and trained this network using gradient-based MCMC on learning curve data from previously tested hyperparameter settings. While this approach is able to predict learning curves of previously unseen configurations, conditioning on the current partial learning curve requires retraining the Bayesian neural network online, which is costly, Recently, DyHPO [Wistuba et al. [022] followed a similar dynamic HPO setup as Swwrssy et al. [014], but used deep GPs [Damianou and Lawrence 2013 While deep GPs relax some of the standard GP assumptions, extrapolation abilities were not thoroughly analyzed. and DyHPO only predicts one epoch into the future. Finally, it is worth noting that, except for Domhan et al. [2015). all the aforementioned probabilistic approaches [Swersky et al. 2014], Klein et al. 2017 Chandrashekaran and Lane 2017 Gargiani et al. [219 |Wistbba al. 2022] utilize meta-learning across the learner's hyperparameter settings. While this is an interesting line of work, it limits applicability, and introduces confounding factors. We will therefore consider simpler and more general setting in this work (see Section 3.1.. Indeed, akin to the approach of [Domhan et al. (2015), we operate without the assumption of access to data from previous runs employing different hyperparameter settings, nor do we assume the ability to generalize across these settings. Our results highlight that prior-data fitted networks (PFNs) offer a significantly more efficient and practical alternative to Markov Chain Monte Carlo (MCMC) methods. As categorized by Mohr and van Rijn 2022 , Domhan et al. [2015] the only comparable prior work within this category. This underlines the novelty and importance of our approach in the context of advancing current methodologies.
While we are the first apply PFNs [Müller al. 2022 to learning curve extrapolation, PFNs have previously been applied in different settings: Hollmann et al.] (2023) used them to meta-learn a classifier for tabular data: Müller et [2023] as a surrogate model for Bayesian optimization; and most recently concurrent work by Dooley et al. [023 as zero-shot time series forecaster.
3 Methods
3.1 Bayesian learning curve extrapolation
Let y [0.1] represent the model performance (e.g., validation accuracy) at training step t t f1....,,m m). The problem we consider in this paper can be formulated as follows: Given a partial learning curve 1/1. V....... Vr up to some cutoff T, and a prior distribution p(y) over learning curves, approximate the posterior predictive distribution (PPD) q( .i.....yri for < S m. We will further assume that can calculate the relative probability density of p(y). reguirement for MCMC, and that we can generate samples from p(y), a requirement for PFNs. Figure] provides an illustration of Bayesian learning curve extrapolation, showcasing the posterior predictive distributions (PPDs) of the extrapolated curves generated by LC-PFN and MCMC, along with a few representative curves sampled from the prior distribution p(y).
1.0
0.8
0.8
0.4
Sarget
data
LCPPAN
10
30
40
Epochs
Figure 1: (Left) Visualization of Bayesian learning curve extrapolation. The plot shows the median and the 90% confidence interval of the PPDs inferred using MCMC and LC-PFN, given two partial empirical learning curves of 10 and 20 epochs, respectively, and the prior described in Section 3.2 (Right) Example of learning curves sampled from the prior p(y).
3

---

While the fixed ranges for and are well-suited for modeling particular learning curves (e.g.. accu- racy over epochs), they also are restrictive. In Appendix[A] we discuss the invertible normalization procedure we apply to support extrapolating, possibly diverging, iteration-base learning curves across a broad range of performance metrics (e.g., log loss).
3.2 Learning curve prior
Following Domhan et al. [2015], we model y as a linear combination of K hasis growth curves Sk- each parameterized by Og. and iid. additive Gaussian noise with variance o², ie..
K
Yt ~ ([(ffombbt())) o²)
with Scomb =
Σ wk fx(t)0s).
where we assume our model parameters
(W1,,... WK 01 .,.... 02,0²)
to be random variables with prior p(E). Here, Domhan al. [2015) assumed an uninformative prior (i.e.. p(E) x 1), with the exception of some hard constrannts. We adopt strictly more informative prior, because (i) the original prior puts almost all probability mass on parameterizations yielding invalid learning curves, e.g., & [0, 1J: and (ii) we cannot practically sample from this prior having unbounded support, a requirement for PFNs Specifically, to mimic realistic learning curves we use bounded uniform weight priors W* 4 (0.T), low-noise prior log(o²) ~ (~(88, 2), only allow curves with values in 00, I] and, like Domhan et al. [2015), only accept curves whose last value higher than its first. Putting all of these together, our prior distribution thus takes the form:
K p(E) x I[((w>) p((p)))))))))))))))))) p(o²) 1 (fcomb (1)mmb()))))))))))))))))))) < (mmmm(mm))))
II 1 [fcommb (f14) E [0, (0.
Finally, we limit ourselves to three parametric families of learning curves (K 3, see Table D These basis curves were chosen to capture a variety of growth trends and convergence behavior. We show examples of curves sampled from this prior in Figure (right).
Table I: Formulas of the three parametric basis curves and priors over their parameters.
Reference name
Formula (t)
Prior p(O,)
pow3
e~ U(00, 1.25) a a~1((-00,,0.)) log(a) ~ (00,
Janoschek
(a Bre-ne
a U(0, 1) B 2(0,2) log(x) 1 log(d) ((0,0.25)
iloga
c~ U(0,1) a~ ((-0...5)5
(1g(++1)
3.3 Prior-data fitted networks (PFNs)
In this paper, we propose to use prior-data fitted networks (PFNs, Müller et al. 2022) instead of MCMC for learning curve extrapolation. PFNs are neural networks trained to perform approximate Bayesian prediction for supervised learning settings. That is, PFNs are trained to predict some output y E It, conditioned on an input and a training set Dtrain of given input-output examples. The PFN is trained for this task with samples obtained from prior over datasets p(D). The loss function for train- ing a PFN go with parameters 0 is the cross entropy la = ((t. of llog qo (y) f, Devain)) ) for predicting the hold-out example's 's label y, given t and Dtrain- Müller et al. [2022] proved that minimizing this loss over many sampled tasks (t. y) U Dirain directly coincides with minimizing the
KL divergence between the PFN's predictions and the true PPD. In essence, the PFN meta- meta-learns to perform approximate posterior inference on (meta-train) synthetic tasks sampled from the prior, and at inference time also does so for (meta-test) real task.
Note that the probability density of p(y:) is well-defined, requirement for MCMC, but not for PFNs. We found that adding additional curves did not improve predictive performance significantly, and made the MCMC baseline even more expensive and less stable. Klein et al. 2017 also limited themselves to five basis curves for similar reasons.

---

(a)
dl44 D)
v D)
(b)
2C183 D)
epoch:
epoch: 2
opoch:
epoch: 4
epoch: 5
10%
26.6%
acc: 51.4%
005
50%
100%
D
Accuracy
Figure 2: A visualization of our LC-PEN model on the task of predicting accuracy epochs curve. D represents the epoch accuracies up to epoch 3 (= T".. Attention between test and training positions is shown using red and blue arrows, respectively. Plots based on Müller et al. [2022].
3.4 PFNs for Learning Curve Extrapolation (LC-PFNs)
To apply PFNs learning curve extrapolation, we train them on learning curves sampled from given prior over learning curves (in our case, the prior defined in Section 3.2). Specifically, the training set we condition on is the available partial learning curve up some varying cutoff point {T' +1......... ........... m) to predict for, and the desired output y is the value of the learning curve at epoch t. During training, we randomly sample the cutoff points T (0, m 1) for every batch in order to learn to predict for initial learning curves of varying sizes. Figure illustrates the information flow in a LC-PFN during learning curve extrapolation.
TV, i... Dtrain {(t', ye }rr= the input we condition on is the epoch
LC-PFN architecture and hyperparameters We use the PFN architecture proposed by Müller et al. 2022] and visualized T (a). That is, we use a sequence Transformer (Vaswani et al. 2017 and treat each pair (4, (for train) and /. (for test) as a separate position/token. We encode these using a simple linear layer. We do not use positional encoding such that we are permutation invariant. Furthermore, the attention matrix is masked such that every position only attends to the training positions. This way training examples can attend to each other, but the test examples do not influence each other's predictions. Note that the output of the PFN with parameters 0 is distribution go (ylt,, D. Following Müller et al. [2022), we discretize qo in a finite number bins whose probability mass is predicted by the PFN, as is shown in Figure (b). The size of each bin is set such that, under the prior, y/2 is equally likely to fall in each bin. The number of bins is a hyperparameter that we set to 1000 The LC-PFN model further inherits hyperparameters from the Transformer, including the number of layers (nl ayers), number of heads (nheads), embedding size (ems i ze), and hidden size (nh dden). We four heads, a hidden size of 1 0244, and conduct thorough ablation study to investigate the effects of the number of layers and embedding size on the final performance, exploring grid of values (see Table [ We use standard training procedure for all experiments, employing the Adam optimizer (Kingma and Ba (2015) (learning rate 0.0001, batch size 100) with cosine annealing [Loshchilov and Hutter 2017 with linear warmup over the first 25% epochs of the training. Finally, we set m = 100, implying LC-PFN is trained for extrapolating sequences of up to 100 training steps (e.g., epochs). We found that most curves are shorter in practice, and when longer sequences are encountered, we subsample them as described in Appendix
4 Experiments
Our experiments aim to test the hypothesis that PFNs present a practical Bayesian approach to learning curve extrapolation. To this end, we first compare our LC-PFN approach against the MCMC approach of Domhan et al. (2015), using the same prior on samples generated from (Section 4.1.
Then, we extend the comparison to four real-world learning curve benchmarks (Section 4.2). Finally, we look beyond the quality of individual extrapolations and evaluate the potential of LC-PFN in the context of predictive carly stopping to accelerate model selection (Section
5

---

4.1 Extrapolating samples of the prior
The goal of this first experiment is to assess the ability of LC-PFNs and MCMC to approximate the true posterior predictive distribution (PPD). To avoid artifacts due to out-of-distribution data, in this experiment, we use curves sampled from the prior (defined Section B.2). Furthermore, we modified the original implementation of MCMC [Domhan et al. 2015], to use the curve prior we proposed in Section (.2 In the following, we refer to this MCMC variant as MCMC-PP and denote the one using the original prior [Domhan et al. 2015) as MCMC-OP (used in Section 4.2). Since the LC-PFN and MCMC-PP methods are both (approximate) Bayesian inference methods, using the same prior, they aim to approximate the same true target PPD, given partial learning curve.
As a performance metrie, we evaluate the log-likelihood (LL) of the unseen data (right-censored curve) under the inferred PPD. We use this metric, also known as logarithmic score, to assess a model's ability to infer the remaining part of the curve based on the initial observed values. A benefit of this metric is that it measures the quality of the PPD as a whole (rather than merely focusing on the error associated to a specific PPD statistic) and, assuming data is generated by the prior, the exact PPD maximizes this metric.
Importantly, we vary the cutoff, i.e., the percentage of the observed curve used as input, to better assess the model's performance across different amounts of available information. Furthermore, to allow a more comprehensive comparison, we vary the hyperparam- eters of both LC-PFN and MCMC-PP. For LC-PFN, we vary the embedding size (emsize ze), the number of layers (nlayers), and the total number of learn- ing curves used during training (nb_data). MCMC-PP, we vary the number of chains generated by the emcee (Foreman-Mackey et al.) 2013] ensem- ble sampler (nwalkers), the length of each chain (burn-in + naamples), the part of the chain omitted to account for mixing (burn- in), and the sub-sample frequency (thin). The considered values for each hyperparameter are summarized in T
Table 2: Grid of hyperparameter values evaluated for MCMC-PP and LC-PFN.
Hyperparameters
L.- 1.. MCSC FP
osamples E [100, 250. 500, 1000, 2000, 4000 nwalkers [26, 50, 100]
burn-in E [0. 50, 100, 500]
thin E [1, 10, 100)
For
ab data è [100k, 1M, 10M] emsize [128, 256, 512) alayers [3,6, 6, 12)
We conducted the comparison on 10 1000 sampled curves. Figure I shows a few curve examples, as well as inferences using LC-PFN and MCMC-PP given the data of the first 10- 20 epochs (cutoff). We observe that both predicted median and uncertainties are indeed similar. More inference examples, with different cutoffs can be found in Appendix C.4
Results Figure displays the average likelikel across MCMC-PP / LC-PFN inferences for varying hyperparameters and 10% cutoff. The log- lilkikiehhood is shown w.r.t runtime, which is measured as the average wall-clock time for a single inference on a single Intel(R) Xcon(R) Gold 6242 EPU.Note that this inference time includes both the fit and prediction times for MCMC variants. Table 3 provides results on higher cutoffs (20%, 40% 80%) and corresponding runtimes for three variants of each method (M1-3, P1-3), labeled in Figure Generally, the LC-PFN variants (left side of figure 3) are significantly faster than MCMC-PP vanants (right side). LC-PFN always ran in less than 0.1 seconds while the best MCMC PP (M3) took over 100 seconds. Figure 3 also offers insights into the importance of LC-PFN and MCMC-PP PP hyperparameters. For MCMC-PP. PP, both the cost and quality of inference increase with longer chain lengths and higher cutoffs. For LC-PFN, the inference cost increases with the model complexity which is closely related to the number of trainable parameters. Inference quality positively correlates with model size ("larger better"), and the number of data PFN was trained on. Among the hyperparameter grid we examined (Table 2 except for the smallest model (P1), all LC- PFN variants that were trained on 10M samples produce higher ligelikelihood than the best MCMC-PP variant (M3). In particular, an LC-PFN (P 2) with 3 layers, embedding size 256, trained on 10M samples achieved better performance (log-likelihood of PPD) than the best MCMC-PP, but more than 15 000 times faster. We also find that while the runtime of the best MCMC-PP PP can be reduced (with minor loss of quality) by using thinning (M2), the better LC-PFN is still approximately 7 000 times faster. Finally, it is important to note that training the largest LC-PFN (P3, 10M samples with 26M parameters) on the prior took approximately eight hours (single CPU, single RTX2080 GPU), but this cost is incurred only once for all of our experiments.

---

LC-PFN emrii
L28
250
512
PRN
#0 1000
1cok
TM
10M
LC-TT-
3.5-
1.44
1.3-
12-
NCNE-
100 550
2020
250
1000
4030
1.2
Time Isecondio
Figure 3: Runtime (lower is better) vs log-liielihodd of the true curve under the PPD (higher is better), with 10% of the curve observed. See FigurePinn Appendix C.T higher cutoffs. Blue and red markers correspond respectively to LC-PFN and MCMC-PP with varying hyperparameters values. The M1-3/P1-1 labels refer to the PFN /CMMC variants listed in Table The horizontal dashed line indicates the performance of the best MCMC variant.
Table 3: Comparison of three LC-PFN and MCMC-PP variants prior curves in terms of log- likelihood (higher is better) at 10%, 20%, 40%, and 80% cutoff'.. Here, M1 corresponds to the configuration used in Domhan et al. [2015). Please refer to Table 5 for more comprehensive results,
Label
Method
Parameters
10%
20%
40%
80%
Avg. Runtime (s)
M1
MCMC
msamples=2000, nwalkers=100, burn Sum--in-500. thin=1
1.628
1.939
2.265
2.469
54,401
M2
MCMC
nsamples=4000, pwalkers=100, barn-in=100, thin=100
1.641
1.958
2.277
2,477
45,160
M3
MCMC
mamplee==4000, nwalkers=100, burn-in=500., thin=1
1.642
1.956
2.285
2.486
103.151
PFN
nh_ data- 10M, nlayerse3, emsizes 128
1,58
1.99
2.28
2.43
0.004
F2
PFN
nh. data== nlayers=3, emsize==25
1.65
2.04
2.35
2.49
0.006
PFN
nb_ data- 10M, nlayerse12 12, emsizee5112
1.76
2.13
2.40
2.52
0.050
4.2 Extrapolating real-world learning curves
While evaluation on data from the prior gives us a controlled setting to analyse quality and cost of the PPD approximation, performance on real-world learning curves is essential for practical usefulness. This second experiment aims extend the previous comparison of MCMC and LC-PFN to real-world learning curve benchmarks.
We consider the best sessteerfrrmigg variants of LC-PEN and MCMC-PP according to the average log- likelihood they obtained in the first experiment. For LC-PFN, the optimal variant (P3) features an embedding size of 512 and 12 layers, resulting in total of 26M trainable parameters, and is trained on 10 million prior curves. For MCMC-PP, the optimal configuration (M3) involves a chain length of 45000 100 walkers, 500 burn-in samples, without thinning. As additional baseline, we include MCMC-0P, the original MCMC variant proposed by Domhan et al. (2015), which uses the original hyperparameters and curve prior (11 basis curves and uninformative prior over the curve parameters).
Benchmarks To evaluate the generalization capabilities of our model, we consider a diverse set of real-world curves. Our dataset comprises 20000 learning curves, sourced from four dis- tinct benchmarks: LCRench [Zimmer et al. [021], NAS TAss-Bench-201 (Dong and Yangl 2020). Taskset [Metz et al., 2020) and PD1 [Wang et al. 2022). J each contributing 5 000 curves, randomly selected from specific subtasks. These benchmarks and subtasks were chosen to span broad spec- trum of supervised deep learning problems, training MLP (LCBench), CNN (NAS- Baus--Becch 201),, RNN (Taskset), and Transformer (PD1) architectures on input modalities ranging from tab- ular data (LCBench), text (Taskset and PD1), protein sequence (PD1), to vision problems (NAS-Bench-201), From LCBench and MAs-Bench-201 we use validation accuracy curves whereas from Taskset and PD1 the log loss validation curves. terms curve length, LCBench and Taskset cover 50 epochs, Ses-BBnnh-222 contains up to 200 epochs, and PD1 curves have varying lengths (22 1414). Further details, including sample curves, on these benchmarks are provided in Appendix B
7

---

LC-PFN
MCHC
RCMC prlor)
LCBench
MAS-Bench-201
Taskset
PD1
3.0
Aug. cank Lop-Bellhodd
2.5
2.5
2.0
2.0
1.5
1.5
1.5
1.5
1.0
1.0
1.0
1.0
60 90 Pencentage the curve observed
10
20 30
40
70
10
60 90
Percentage the curve observed
Parcentage of the curve observed
Percentage of the curve observed
(a) Average rank of log-likelihood values (lower better) Ps cutoffs
LCBench
NAS- MAS-Bench-201
Taskse:
PD1
3.0
3.0
3.0
3.0
Aeg mit MSE
1.0
1.0
1.0
1.0
10 20 50
00 50 60
20 30
40 50 60
20 30
40
50
00
50 00
Percentege of the curve observed
Percentage of the curve observed
Perceritage of the curve observed
Percentage of the curve observed
(b) Average rank of mean squared error (MSE) values (lower is better) cutoffs
Figure 4: Comparison of LC-PFN with two MCMC variants on three real-data benchmarks.
Our focus lies on the relative performance of MCMC and LC-PFN, as absolute perfor- mance significantly influenced by the choice of prior. We consider the log-likelihood and the mean squared error (MSE) of the predictions as metrics. Here, the MSE is calculated w.r.t. the median of the PPD.
Metries
For each benchmark, we report the average rank of these metrics to aggregate results from different curves, supposed to the average values. While the latter better captures performance differences, it is very sensitive to outliers and scale-dependent. When computed in normalized space, would strongly depend on our choice of normalization parameters (see Appendix A
Results For each benchmark, Figures 4a and display the average rank obtained by each method in terms of log-likelihooe and MSE, respectively, where ranks are averaged across all 5 000 curves. We do not include error bars as the standard errors are neglectable (less than 0.02). In summary, we observe similar trends for both metrics all benchmarks. LC-PFN is never truly outperformed by -P. On LCBench both methods rank similarly, with LC-PEN being slightly worse high cutoffs. LC-PEN ranks better PD1, B.AS-Bench- 201, and Taskset. MCMC-OP performs clearly worse on LCBench, Taskset, and PD1. On NAS nAS-Bench--2 20henchh2C1,, MCMC OP performs best for lower cutoffs, outperforming MCMC-PP, suggesting that NAS-Bench-201 curves are better captured by the original prior. Figure [I and Figure 12 in Appendix C.2 show the log- lig-liielshoods and MSEs, respectively, for all three methods, for each curve and cutoff, per benchmark, providing more detailed perspective.
4.3 Application: Extrapolation-based early stopping in model selection
Thus far, we have shown that L.CPFN produces extrapolations of similar or better quality to MCMC, at small fraction of the cost. However, these extrapolations are not perfect. In many cases, the practical relevance of any errors only be assessed in the context of specific application.
In this final experiment, we thus consider model selection setting where after every epoch of training we have the choice between (i) continuing the current training, or (ii) stopping early (T < m) and starting new training run using different training pipeline (e.g., model architecture, hyperparame- ters, etc.). Here, we assume that runs cannot be resumed once stopped and that the order in which training pipelines are be considered is given. Our objective is to obtain high-quality models as quickly possible, by stopping suboptimal training runs as early as possible. This setting is also known as vertical model selection [Mohr and van Rijn 2022] and was also considered by Domhan al. 2015).
8

---

rcolop
seterse
palanaat)
pelomnat20
pabanaat0
LCHench
NAS- B21--B
Iikset
1x30
2X32
Cerrace
Ganncce
2x35-
200
050
1000
1000 1503 abde 2500 1200 2000 1000
200
402
600
3000
Sour Epoctss Turned
Mai a Epeche Thams
Th.al of Epacts Thane
PDL Languuge Modiaing (lm)lo)
701 Translatior (WKT)
9D1 -P PPotainn Sequence RafSS)
10
30"1-
217
100
235
302 420 302 6CD 700
5000 10003 13000
20033
23000
100
2001
302
400
Epoces
Figure 5: Comparison of our LC-PFN based early stopping mechanism to naive baselines no-stop (no stopping, train for the full budget m) and Patience (k) (stop training after epochs without improvement) for vertical model selection where runs are considered in a fixed order and cannot be resumed. Shown is the anytime regret (lower is better) different approaches achieve after a total number of training epochs, averaged per benchmark and across 40 orderings per task.
We consider the extrapolation- based termination criterion proposed by Domhan et al. [2015], but use LC-PFN instead of MCMC to predict the likelihood Pr(ye Ybestt V1; . .9) that the current run will at epoch 1' obtain model better than the best obtained by any run thus far (Wbest); for T t' < m and decide to stop the current run if that probability does not exceed some fixed threshold (at any point). In our experiments, we use confidence level 1 et al. 2 Note that this criterion can be applied at every step or only at specific cutoffs. To simulate the effect of varying granularity, we consider a coarse- coarse-grained grained variant with 4 cutoffs T {[0.1m.] [0.2m] [0.4m], [0.8m]] ]. and fine ine-grained variant with T E {T
0.95 as Domhan
T m).
We investigate alternative choices for cutoffs and confidence levels in Appendix C.3.2
Following Domhan et al. 2015 we compare against black box approach no-stop that does not implement early stopping. We further compare against criterion Patience (k) that implements the popular heuristic to terminate a training run when model performance did not improve for & epochs. For evaluation, we consider the same benchmarks as in Section 4.2 (for details, see Appendix [B. We chose the total budget for model selection to correspond to 20 full runs (20m). For each task, we consider the training runs in 40 different random orderings. This totals 120 model selection experiments per method, spread across the 53 different tasks.
Results Figure shows the anytime performance of all methods in our comparison, on each of the benchmarks, in terms of regret. Here, regret the absolute difference in performance between the best model obtained thus far and the best model attained by any run on the task. Results are averaged over the different run orderings. For LCBench, NAS NAS-Rench-200 and Ta Taskset results are further averaged across all tasks (results for individual tasks can be found in Appendix C.3. The shaded area corresponds + 1 standard error. On all benchmarks, except for NAS-Bench-201 we observe that LC-PFN based termination criteria clearly perform best. Also, the fine-grained variant performs better on average, suggesting that errors inference are compensated for by the time saved by stopping runs earlier. In terms of expected speed-up. up. this LC-PFN variant obtains an expected regret lower than that obtained by no-stop, approximately 3.3x faster on LCBench and ses. Looking at individual tasks, we note 2 6x speed-ups on all 3 PD1 tasks, all 12 Taskset tasks, and of the 35 LCBench tasks (we obtain 1-2 x speed-ups on the remaining 5). we find that all termination criteria considered, including standard Patience heuristics, fail on all 3 tasks; this likely related to the particular shape of learning curves on this benchmark, having inflection point (see Figure g and Figure 19) not resembling any in the prior.
On

---

Finally, in terms of overhead, the computational costs of the LC-PFN inferences per model selection experiment range from seconds (coarse- -gerseeg up minute -imnnegai and are negligible compared to the cost of the 20 full training runs of deep neural networks.
5 Summary, limitations, and future research
We presented the first work using prior-data fitted networks (PFNs) for Bayesian learning curve extrapolation. We show that our LC-PFN obtains qualitatively similar extrapolations, for wide variety of learning curves, more than 10 000 x faster than the MCMC method proposed by Domhan now fast enough (under 100 milliseconds CPU, and even less
et al. 2015]. These inferences on GPU), to be used in the context of online learning, at virtually no overhead. This opens up wide variety of possible applications, e.g., speed up automated model selection in AutoML and HPO by discarding poor configurations early. It would be interesting to integrate LC- PPN as new termination criterion in existing deep learning libraries.
Often, we also have more data available than a single partial learning curve, e.g., other curves on the same task, their hyperparameters, and/or curves of the same method on different task, and meta-features. Previous work [Swersky et al.) [0144] Klein et al.) 2017 Wistuba et al. [0022] Ruhkopf al. 2022] has already exploited this, and we could explore the potential of using PFNs for Tew-shot in-context meta-learning, by feeding the model multiple curves and hyperparameters as input.
2015 reusing the original code, our prior (Section 3.2) closely resembled the one of Domhan et al. 2015), future work could improve upon this prior and overcome its limitations (e.g.. on the NAS Bench cieench-200 201 tasks) by modelling divergence, slow start, double-descent, correlated heteroscedastic noise, etc.
While for a fair comparison to Domhan et al.
Finally, PFNs, unlike other Bayesian methods, must learn the prior from data, which implies that the prior must be generative. Also, suggests that high entropy priors may present challenges. Future research should investigate these limitations and how to overcome them.
6 Acknowledgments and disclosure of funding
We acknowledge funding by the European Union (via ERC Consolidator Grant Deep Learning 2.0, grant no. 101045765), TAILOR, project funded by EU Horizon 2020 research and innovation programme under GA No 952215, the state of Baden- Württemberg through bwHPC and the German Research Foundation (DFG) through grant numbers INST 39/963-1 FUGG and 417962828. Views and opinions expressed are however those the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Funded by the European Union
10

---

References
B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. arXiv:1611.02161 [cs. 2017.
A. Chandrashekaran and I. R. Lane. Speeding up hyper-parameter optimization by extrapolation of learning curves using previous builds. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 477 492. Springer, 2017.
D. Choi, H. Cho, W. Rhee. the difficulty of DNN hyperparameter optimization using learning curve prediction. In 2018 IEEE Region 10 Conference (TENCON 2018), pages 651-656. IEEE, 2018.
C. Cortes, I.. D. Jackel, Solla, Vapnik, and Denker. Learning curves: Asymptotic values and rate of convergence. Advances in neural information processing systems, 6, 1993.
A. Damianou and N. D. Lawrence. Deep gaussian processes. Artificial intelligence and statistics, pages 207-215. PMLR, 2013.
Domhan, J. Springenberg, and F. Hutter. Speeding up automatic Hyperparameter Optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI'55) 15), pages 3460-3468, 2015.
X. Dong and Y. Yang. NAS-B B-S--Bncc-200 2011: Extending the scope of reproducible Neural Architecture Search. In Proceedings of the International Conference Learning Representations (ICLR'20) 2020.
S. Dooley, G. S. Khurana, C. Mohapatra, S. Naidu, and C. White. ForecastPFN: Synthetically-raane trained zero- shot forecasting. In Proceedingy of the 37th International Conference on Advances Neural Information Processing Systems (NeurlPS'23), 23), 2023.
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In Proceedings of 35th International Conference on Machine Learning (ICML'18), volume 80, pages 1437-1446. PMLR, 2018.
D. Foreman-Mackey D. Hogg, D. Lang, Goodman. emcee: MCMC Hammer. Publications of the Astronomical Society of the Pucific, 125(925):306, 2013.
L. Frey and D. H. Fisher. Modeling decision tree performance with the power law, Seventh International Workshop on Artificial Intelligence and Statistics. PMLR, 1999.
M. Gargiani, A. Klein, S. Falkner, and F. Hutter. Probabilistic rollouts for learning curve extrapolation across hyperparameter settings. In ICMI. workshop on Automated Machine Learning (AutoMI. workshop 20/9), 2019.
P. Gijsbers, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. An open source automl benchmark. In ICML workshop Automated Machine Learning (AutoML workshop 2019), 2019.
N. Hollmann, S. Müller, K. Eggensperger. and F. Hutter. TabPFN: A transformer that solves small tabular classification problems in second. In The Eleventh International Conference on Learning Representations, 2023.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, Wu, and D. Amodei. Scaling laws neural language models. arXiv:2001.0836. [cs.LG], 2020.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR'I5) 2015.
A. Klein, S. Falkner, J. Springenberg, and Hutter. Learning curve prediction with Bayesian neural networks.
In Proceedings of the International Conference on Learning Representations (ICL.' /7), 2017.
Kolachina, Cancedda, M. Dymetman, and Venkatapathy. Prediction of learning curves in machine translation. In Proceedings of the 50th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 22-30, 2012.
R. Leite and P. Brazdil. Improving progressive sampling via meta-learning on learning curves. In European conference on machine learning, pages 250-261. Springer, 2004.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: Bandit- based configuration evaluation for Hyperparameter Optimization. In Proceedings of the International Conference on Learning Representations (ICLR'17) 17), 2017.

---

1. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Leurning Representations (ICLR'I7), 2017.
A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng. and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 150, 2011.
L. Metz, N. Maheswaranathan, R. Sun, C. D. Freeman, B. Poole, and J. Sohl-Dickstein, Using thousand optimization tasks learn hyperparameter search strategies. arXiv:2002.11887 [cs.LG), 2020.
Mohr and Rijn. Learning curves for decision making in supervised machine learning-a survey. arXiv:2201.121500 [cs.LG], 2022.
Müller, N. Hollmann, S. Arango, Grabocka, and Hutter. Transformers do bayesian inference. In Proceedings of the International Conference on Learning Representations (ICLR'2) 22), 2022.
Müller, M. Feurer, N. Hollmann, and F. Hutter. PFNs4BO: In-context learning for bayesian optimization. In ICML 23, 2023.
Ruhkopf, A. Mohan, D. Deng, A. Tornede, Hutter, and M. Lindauer. Masif: Meta-lcarned algorithm selection using implicit fidelity information. Transactions on Machine Learning Research, 2022.
D. Salinas, M. Seeger, A. Klein, V. Perrone, M. Wistuba, and Archambeau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. International Conference on Automated Machine Learning, AutoMI. 2022, 2022.
K. Swersky, J. Snoek,
Adams. Freeze-thaw Bayesian optimization. arXiv:1406.3880 [stats.MLJ. 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. Gomez, L.. Kaiser, and I. Polosukhin. Attention all you need. Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurlPS' 17), 2017.
Wang, G. E. Dahl, Swersky, C. Lee, Z. Mariet, Nado, J. Gilmer, J. Snoek, and Z. Ghahramani. Pre-training helps bayesian optimization too, 2022.
M. Wistuha, Kadra, and Grabocka. Supervising the multi-fidelity race hyperparameter configurations. Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems 22)),, 2022.
Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approxi- mation, 26(2):289-155, 2007.
L. Zimmer, M. Lindauer, and F. Hutter. Auto-Py PyTorch Tabular: Multi-Fidelity MctaLcarning for Efficient and Robust AL, IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:3079-3090, 2021.
12

---

A Learning curve normalization
Not all learning curves of interest may resemble those of the prior described in Section 3.2 For example, when minimizing the negative log-likelihood loss (log loss), learning curves will mostly decrease (vs. increase), be convex (vs. concave), and could take values that exceed 1.0 (there is no hard upper bound for log loss).
One approach would be to define prior and train specialized PFN for every performance measure. In this work, we use more general approach: We apply a normalization procedure that allows us to accurately extrapolate wide variety of learning curves using single PFN, without retraining or fine-tuning. We consistently apply this procedure to all inferences involving real learning curves (i.e., the experiments described in Section 4.2 and Section 4.3) even if observations are naturally constrained to [0, 1J. e.g., accuracy curves.
Step 1: Normalize the partial learning curve: Let yf be the 7t performance observation. We start by normalizing using a generalized logistic sigmoid transform yi 92 (v,), that re-parametrized by the following five parameters (A):
min? A Boolean specifying that we expect learning minimize the performance measure. E.g., this would be true for error rate and log loss, and false for accuracy.
hard, Uhard These are possibly infinite hard lower upper bounds for model performance. This would, e.g., be 0/1 1 for accuracy and error rate; and 0/ +oc for log loss.
balts Usoft These are finite soft lower/upper bounds for model performance.
They specify the range in distinguish between) to lie in practice. For accuracy and error rate, one could set these equal to the hard bounds, whereas for log loss one could, e.g., use estimate of the loss for the untrained network (i.e., epoch 0) as Uson and laoft hard, or if available, choose loot to be an optimistic estimate of performance at convergence (e.g., state-of-the-art), Narrower ranges will result in more accurate inferences.
which expect performance values (that
Specifically, we perform a linear transformation, followed by a logistic sigmoid, another linear transform, and minimization conditional reflection around 2: i.e.
(x(x)) = cro.5
+ aax=- + +
where
[1 y if min? yy
2
Usoft
soft
cTu.s(V)
b
min?
Usott loof
Usott
Igoft
1 + e- aumm + e -llam + at than 26)
aahha b) a( b)
1 e a(haad bb
Note that for A (False, -00, 11,1,,,,,,00 this reduces to the canonical logistic sigmoid y; 1te -+e-*
This transform will be approximately linear (shape preserving) for y? E [looa UsoR]. and the tails of the sigmoid will gradually squish values outside this range to [0, 1], supporting unbounded performance measures. Figure 6 visualizes this projection in general, and provides examples for accuracy with = (False, 0, 0, 1, 1) and log loss with À = (True, 0, 0, log(10), oo).
Step 2: Infer using normalized data: Next, we perform Bayesian learning curve extrapolation on the normalized partial curve y, with our usual prior ply), resulting an approximation of the PPD in the transformed space.
Step 3: Inverse transform the PPD: Finally, can obtain the property of interest of the PPD in the original space by applying the inverse transform, given by
log(
( (cro.s
b
(vl))
We use order-based statistics (e.g., median or other percentiles) in our experiments, to which we can simply apply 95 directly since it is monotonic transform. For other statistics (e.g., mean, variance) may need to resort to Monte Carlo estimation, applying 9A to samples of the PPD.
13

---

14
04
0.0
The generic logistic sigmoid transformation described in Appendix A and used normalize learning curves to support wide range of possibly unbounded performance metrics. middle/right: An example of the normalization for curves maximizing accuracy / minimizing log loss, using the transformation.
Figure 6: left:
24
34-
02-
20
100
apocts
Figure 7: Sample of 10 curves taken i..dd from the prior (Section 3.2 (Right) The same sample after applying the inverse transformation 9X obtain samples from the log loss prior as described in Appendix A using the transform shown in Figure (right).
By applying given normalization, effectively transform our prior, i.e., p(y") 9% (p(Y)). To get some intuition of what that prior looks like, we can sample from p(y") by applying 9 to samples of our prior p(g), which is useful for finettuiin the parameters of this transformation. Figure (right) shows samples of the log loss prior for X (True, 0, O, (og(1)),
B Detailed benchmark description
To evaluate the generalization capabilities of LC-PFN, we consider diverse set of real-world curves. The reduce computational cost and improve reproducibility, we source these from four existing benchmarks that provide a collection of learning curves for wide variety of supervised learning tasks. In what follows, we describe each of these benchmarks in more detail, the subset of tasks and curves we considered, and any preprocessing we did. Table provides an overview of the characteristics of each benchmark and Figure 8 (a) visualizes the curves selected from each benchmark.
LCBench provides learning curve data for 2 200 configurations of AutoPytorch Zimmer et al. 2021] trained for epochs on 35 tabular classification datasets from the AutoML benchmark [Gijs bers et al. 2019). All of these configurations use momentum SGD to train an MLP. but with varying number of layers and units per layer, and varying optimization hyperparameters (batch size, learning rate, momentum, I.2 regularization, dropout rate). In Section 4.2] we consider subset of 5 000 curves selected uniformly at random from all 70 000 validation accuracy curves in the benchmark. In Section 4.3 we consider all 2 000 validation accuracy curves, for every task, in 40 different orderings.
NAS MAS-Bench-201 is a benchmark Neural Architecture Search methods [Dong and Yang 2020 It provides learning curve data for training 15 625 different architectures for 200 epochs on three different image classification datasets (CIFAR-10, CIFAR-100, and ImageNet16 6- 120) for three different random seeds. Other aspects of the training pipeline (c.g.. optimizer, hyperparameters)
14

---

Table 4: Overview of the characteristics of the real-wort learning curve benchmarks we used in Section 4.2 and Section 4.3 For each benchmark, the last columns list the normalization parameters used (see Appendix|e A), where yoo corresponds to the performance of the untrained model.
benchmark
sublasks
architeccurr
model
(datasels)
Input modulity
(# epochs)
length
metric
(split)
min?
hard
lun
Must
Waand
LCRencc
MI.P
Taholar Classification (OpenMI.)
accuracy
False
NAS Benca-201
3
CNN
Cifar- 100, ImageNet16- 120)
Image Classification (Cifar-10, 10,
200
ernor rule
(val)
True
0
I
Taskset
12
RNN
(Seatincet Analysis IMDB)
Tex: Classificttinn
log (val)
Truc
9b
+0U
201
Transformer
Text (Language Modeling lmilb)
log loss (val) log (val)
Trae
3.46
9b
Transformer
Tex: (Trunslation WMT)
1414
Trae
1.68
50
4:00
Transformer
Protein sequences (UniRef50))
log loss
True
50
are fixed. We use the validation error rate curves provided through the Syne Tune [Salinas et al.) 2022] interface to this benchmark. As discussed in Section 3.4 the LC-PFN we consider is trained to extrapolate curves up to length 100 (m)) To handle the 200 epoch curves from NAS- Ben-h-nhhhh- we chose to subsample them, feeding only every second observation into the LC-PFN. In Section 4.2 we consider a subset of 000 curves selected uniformly at random from all 625 error rate curves in the benchmark. In Section 4.3 we consider all 46 875 error rate curves, for each of the three datasets, in 40 different orderings.
Taskset Metz et al., 2020] provides total of roughly 29 million learning curves for over 1 162 different deep learning tasks. Here, a task defined as optimizing a given neural network architecture for given supervised learning problem, and the curves correspond to using 5 different optimizers, using 000 different hyperparameter settings, and different random sceds. Following Wiistbb et al. only use the validation log loss curves of small subset of 12 tasks, that consider training different RNN architectures, with varying architectural parameters, for 50 epochs on the IMDB sentiment analysis dataset [Maas et al., (011). binary text classification task. As objective was to evaluate the robustness of our approach, we avoided excluding ill-behaved (e.g., diverging) curves. However, upon inspecting the data, found that some of these tasks up to 90% of the curves fail to significantly improve upon the initial model performance, some of which diverge the
2022
almost instantly. In combination with the normalization procedure described in Appendix majority of curves are effectively mapped onto the same narrow range, producing a constant trend (at O in case of divergence). While this is reasonable in practice, and both LC-PFN and MCMC variants predict this constant trend with very high confidence, it does create a bias in our evaluation towards methods excelling at extrapolating poor curves. To eliminate and investigate this bias, we selected the 5 000 curves used in Section 4.2 such that 2 500 are taken from the top 10% bes besf] curves per task, and 2 500 from the 90% others. In Figure 4 we compared methods on the "good" curves only, an evaluation on the 2 500 "bad" curves is presented in Figure X (b). In Section 4.3 we do not make this distinction and consider all 25 000 curves per task, in 40 different random orderings.
PD1 is recent benchmark that Wang et al. 2022 describe as collecting "a large multi-task hyperparameter tuning dataset by training tens thousands of configurations of near-state-of-the -hhth-- models on popular image and text datasets, as well as protein sequence dataset". We access this benchmark through the synetune [Salinas et al., 2022) library interface, which provides learning curve data for 23 tasks, where the number of curves provided, as well as the curve length, varies per task. Here, we limit our selection to log loss curves for the three tasks that consider training Transformer architectures for 22 epochs on language modeling (Imlb), 1414 epochs on translation (WMT), and 38 epochs on protein sequences (UniRef50). To handle the very long learning curves on the translation task, we subsample these aggressively, feeding only every 14th observation into the LC-PFN. In Section 4.2, we select 5 000 curves from these three tasks uniformly at random. In the
Section 4.3 we consider all curves per task, in 40 different random orderings. Finally, optimal loss achieved by any of the runs as lyon > O in our normalization. While unknown in practice, we do not expect qualitative differences in evaluation if a rough estimate were to be used instead.
"This was our first choice m and we retained throughout the project. We not anticipate serious issues when scaling m up to 000, except for increased training times. "Curves are ranked based on the lowest validation loss obtained at any point during training.
15

---

LcBencc
Bcrch 201
04
0.1
0.2
02
8.2
0.3
25
45
83
390
(a) Learning curves sourced from each benchmark
LC-PFN
MCNC(orignal)
MCMC[jur prior)
Bad
Bad
3.0
3.0
Average rank Log htppllllshooodddooooo values
3.0
3.0
Average MSE values
2.5
2.5
2.5
2.5
NSE
2.0
2.0
2.0
2.0
15
1.5
1.5
1.5
1.0
1.0
100
30 35
40
25
35
15 20
Percentage the curve observed
Percentage of the curve observed
Percentage the curve observed
Percentage of the curve observed
(b) Quality of curves quality of extrapolations Taskset (average rank, lower is better)
Figure 8: (a) Illustration showing the 5 000 curves used for each of the four benchmarks in our experiments in Section 4.2 All curves shown after normalization and subsampling, and random subset of individual curves is highlighted. For Task "good" "bad" curves (top 10% / bottom 90% in their task) are shown green red, respectively. (b) Average rank of Log- log-likelihood and MSE values on Taskset curves for the three methods (LC-PFN, MCMC-PP PP, and MCMC-OPP OP). The two leftmost plots show results on the 000 sampled Taskset curves. The rightmost plots provide the same comparison, but specifically for the 2500 "bad" curves. Here, we observe that MCMC-FP performs better on the "bad" curves, which we believe can be attributed the discretization of the PPD in 1000 bins, limiting the maximal confidence / accuracy of LC-PFN, since the majority of these curves are quasi-constant (fall the same bin) after normalization.
C Additional analyses
C.1 Extrapolating samples the prior
C.1.1 Detailed results of section 4.1
Figure 9 provides visual representation of the log-likelihood values prior curves for higher cutoffs, similar to Figure 3 As expected, the difference log-likelihood values between JC-PFM and MCMC-PP decreases
more points of the curve observed. The detailed results can be found in Table 5) Both Figure | and cablel5]clearly demonstrate that certain variants of LC-PFN, particularly those trained with IOM examples, consistently outperform the best variant of MCMC-PP across all considered cutoffs.
16

---

JC.PN ri,dana
LCPPN ceatze
LC-97-
MCMC- Fsorcles
100k
10M
322
226
522
100
250
500
1000
2000
4000
Comparison of LCC-PF and MCMC-PP with varying hyperparameters values on prior curves in terms of average runtime (lower is better) and average log-likelihood (higher is better) for different cutoff values (20%, 40%, 80%)
Figure 9:
Table 5: Comparison of the 25 best CCN and MCMC-PP PP variants on prior curves in terms of log-likelihood (higher is better) at 10%, 20%, 40%, and 80% cutoffs. Values in brackets correspond to one standard error.
Method
Purameters
10%
20%
40%
80%
Avg. Runtime
MCMC- Pp
msamples-2000 nealkers 100, huns-in-500, thin-10
(0:01)
1.939 (0:011)
(0:01)
(0.009)
29:944 (7.6--0)
XCXC-PP
msmples=2000. mwalkers 100, bums-nn500,, thin=1
(6.2.(0.01)
1.999 (0.011)
2.265 (0.01)
2,469 (0.009)
54.401 (3.1E-01)
msmples=1000. mwalker=50 bur in=0, thin==1
1.299 (0.01)
1992 (0.011)
2.266 (0.009)
2.467 (0.009)
28.783 (6.2E-01)
MCXC-FF
msmples=:0000 pwalkm-500 bum--in=0, [hin=l
1629(00.0)
1,913 (0.011)
2.266 (0.009)
2.167 (0.009)
53 29.2 (1.36-01)
MCSC-PP
mampless=4000 mwalken=51, bum-inn:5), thin=100
6.29
1.943 (0.011)
2.366
(7,36-0)
MCVC-PP
namplec-4111) nealker-50, hum-in- 100. thine
(0.01)
1.943 (0:011)
2.267 (0.009)
(0) (0.099)
26-99 (7.4-01))
XCXC-PP
mamples=/000, pwalken=50, bum-in=50, thin=10
1631 (0.01)
1.914 (0.011)
2.267 (0.009)
2.168 (0.009)
29.109 (6.3E-01)
MCXC-P
msmples=4000, nwalken=51, bumt-in=511, thin=1
1.63 (0.01)
1.944 (0.011)
2.267 (0.009)
2.469 (0.009)
51 601 (4. 26-01)
MCVC-PP
msamlec--(11) nealken-50, bum--n- 1000 thin=10 10
1,32 (0.01)
1.943 (0.011)
2.268 (0.01)
2.47 (0:D09)
20. 233 (6-45-01)
MCMC pp
msamplet-4111) mealkers-50. hum-in- 100.
thin-1
1,32
1.943 (0.011)
2.268
2.47 (0:009)
(4-20-01
XCCC
msmples-44000
nealker--50, bum-in-500,, thin- 100
(.632(0.01)
1.94 (C.012)
(0:01)
2.477 (0.009)
(8.28-01)
8CXC-PP
mamples=0000.
pwalkers=50, bum-in-500, thin=10
(..2((0.0))
1.942 (0.011)
2:275 (0.01)
2.473 (0.009)
32 (7.16-
MCMC-
msample--0001,
nealkes--i) 50, hum-in-500),
thin-1
1,622 (0.01)
1.942 (0.011)
2.275 (0.01)
2.477 (0.009)
56.526 (3-4F-01)
HCMC hp usamples-4000, mwalkes-1000 hums-in-0, thin-100
1,337 (0.01)
1 1.954 (0.011)
2.274 (0.01)
2.474 (0.009)
44 076 (1 4B 00))
XCMC-PP
mamples-4000. pwalkers- 100, hum num--nn-0, thin=10 10
1.39 (0.01)
1,956 (0.011)
2.276
2.476
947 (1.2E+00)
XCSC-PP
mamples=0000.
mealken==100 burn
thin=1
1.956 (0.011)
(0.01)
2.476
(9 9E
MCXC-FF
msmple)=0000, pwalken==000 100, bum-in=500, thin= 100
16399(0.01)
1,956 (0.011)
2.276 (0.01)
2.476 (0.009)
61. (E+0000)
MCXC-PP
msimples=0000, mwalken=1000 100) bum-in=50), 1hin=10
164 (0.01)
1. 1.95 (0.011)
2.277 (0.01)
2.477 (0.009)
49 475 (1 2E-00)
MP
(samplec-4/0)) nealkes-100 100, hum--n-50), thin-1
1.64 (0.01)
.95 (0.01T)
2.277 (0.01)
2.477 (0.009)
548 (1 40))
XCXC-FF
msmples==0000 nealkm-=100, bum- 100,
thin=100
1661 (0.01)
1,958 (0.011)
2.277 (0.01)
2.173 (0.
45.16 (1 1E000
MC3C-PP
msmples=4000 mealkun=100, bum-in=:100, 100), 1h
2.278
MCWC-PP
namplec-4111) nealken=100, hum-in-100))
thin=1
(0.01)
1.957 (0:011)
(0:01)
2.47%
98 (1 1F400)
MCMC np
msample--0001, nealkes-1000 1000 hum-in-500), thin-100
1.642(0.01)
1.955 (0.011)
2.284 (0.01)
2.435 (0.009)
49.5 (1.6200))
PP
4000)
mwaless-1000 hum- it-000 thin-10
1.642
1 .9566 (0.011)
2.285 (0.01)
2.436 (0.009)
54 328 (1 (0)
MCMC-PP
mamples=5000, mwalker==100, burn in=500, thin=1
(..0((
1,956 (0.011)
2.255 (0.01)
2.486 (0.009)
103.151 (L.E)00)
LC-FFN
dals= mlayenss,,cmsiiue
1.242 (0.001)
(0.001)
(0.001)
1,709
(1.8--00
LC-39H
dott-100kk, nlayers-12, emsizee-125 128
1.254 (0:001)
(0:001)
(0.001)
1.73 (0:002)
0.012 (R. 3E-04)
10-2FH
Nb data=100k, wlayers-a, cmssiee22666
1.267 (0.002)
1.597 (0.001)
1.764 (0.001)
L.843 (0.002)
0 (8. 2. 04)
1C-2FN
Nb data=
1.354 (0.001)
1.661 (0.001)
1,833 (0.001)
1.9 (0.002)
0.021 (1 (L.EE.0 03)
LC-FFN
Nb dala:= 100ka nlw.ersyyyss33mmmmm
5m
1.392 (0.001)
1.679 (0.001)
1.571 (0.001)
1,959 (0.00)
0.016 (1 (E-0))
LC-PFN
mlayersm,, 1
1.499
1.745
1.547
1.C-99N
data- 100k, mlajers- em1
(0.001)
1.817 (9:001)
2,033 (0.001)
2.001
LC-2FN
Nb data: IM. nlayem=33, emssso==11 128
1.528 (0.001)
1,894 (0.001)
2.159 (0.001)
2.279 (0.002)
0.001 (2.8E-00)) 00)
LC-
Nb dala: MM.
emsiic-1111 128
1.581 (0.001)
1.949 (0.001)
2 20202 (0.001)
2. 3.308 (0.002)
0.007 (4 (4.6E-04)
1.C-99N
Nb data- IM. nlayeree3,
1,583 (0.001)
1. 96997 (0.001)
2.230 (0.001)
2.357 (0.102)
0.007 (5 (F-0F-4)
1C-2FN
Nb data- 10M, olayers-3, 122
1,585 (0.001)
1.989 (0.001)
2.282
2.431
(2, (2
1C-3FH
dota- nlayess-1, -22
(0.001)
1.982 (0.001)
2.223 (0.001)
1C-2FN
Nb data IM, playessa3, 4msino=512 512
1.624 (0.001)
2.011 (0.001)
2.291 (0.001)
2.417 (0.002)
(1 03)
1C-2FN
Nb dala: IM, nlayem=st.cmsioo2223.
1.627 (0.001)
2.007 (0.001)
2.269 (0.001)
2.38] (0.002)
0.012 (8.7E-04)
1C-29H
Nb datt-1 10M, nlayess-3,
256
1.654 (0.001)
2.044 (0.001)
2.347 (0.002)
2.492 (0.003)
0.006 7E 04)
1C-2FN
Nb data- IM, playess=12, 100_:sss222 425
1667 (0.001)
2.005 (0.001)
2.294 (0.001)
2.406 (0.002)
(1.5E
LC-EFN
data= www.yaaas.s.ccmmmmmmmmm cmmiio05111
1.672
2.051 (0.001)
(0.001)
2.443
LC-2FN
Nb dalag= 10M, mlayersa6, emsia==12 128
1,673 (0.001)
2.058 (0.001)
2,352 (0.001)
2.478 (0.003)
0.007 (4.6E-04)
LC-2FN
Nb data: IM, nlayers1222
1,99 (0.001)
2.072 (0.001)
2. 2.444 (0.001)
2.46 (D.002)
0.051 (4.6F-03)
1.C-9N
Nb dada-- 10M, nww.es-.,.emmt cmniie--1
1.507 (0.001)
2.083 (0.001)
2.37% (0.062)
2.517 (0.065)
0.017 (1 (1.4F-03)
1C-3FH
Nh dodd-- n0M,,ssa-s--22ee e2.iziz
1.517 (0.001)
2.094 (0.001)
2.377 (0.001)
2.498
0.022 (8. (8.1-04)
LC-PFN
Modd::10MM playersm6,,.mmmmm amsin
1,721 (0.001)
(0.001)
2.514
(8 (K)2E-04)
16-99N
Matta110.. alayers-6, emsize-11
1.744 (0.001)
2.125 (0.001)
2,404 (0.002)
2.527 (0.063)
0.026 (2.3-03)
Nb dota- 10M, mlayess-12. emsizze2266
1.746 (0.001)
2.124 (0.001)
2.403 (0.00))
2.522 (0.003)
0.02 (1.58-03)
10-35H
Nb data- nlayess-12. emsize-555
1.758 (0.001)
2.133 (0.001)
2.403 (0.002)
2.519 (0.003)
0.06 (4.5E-00)
17

---

LC-PFN INb data-10M, mlayers-12 emsi
177-
inttalzatioon
Log 10g-itshhodd
3.5
MAP
Dzfout
of Samp
1000
14-
4000
10000
1.3
100000
10
10F
10
10
Figure 10: runtime vs log-likeli lill-iikeihood values of MCMC-PP, using thinning 100, with different initial- ization strategies and sample sizes (per walker), averaged across 000 curves generated i.idd from the prior. The blue point represents IC-PFN, with the dotted line a reference for its LL value. The shaded areas correspond to = standard error.
C.1.2 Effect of MCMC chain length and initialization
In our experiments in Section 4.1 observed that the quality of MCMC inference increases with chain length. In particular, our best MCMC-PP variant (M3 in Figure 3). uses the same hyperparameter settings as Domhan al. [015] (M1), but with a double as long chain (discounting the burn-in).
In this section, we investigate whether the chains considered are too short and whether increasing chain lengths further will eventually cause MCMC-PP
outperform LC-PFN. To study this, we will run MCMC-PP to collect up to 100000 samples per walker (50x more than Domhan et al. (2015). To reduce the computational cost of this experiment, we only extrapolate 000 (instead of 10000)
prior curves, on cutoff 10%, and use MCMC-PP with a 100-thinning strategy (M2). This variant is more compute-efficient and the minor negative effect of thinning expected further decrease with increasing chain length. Concurrently, we study the effect of initialization. There are different ways of initializing MCMC chains and the chosen strategy can strongly affect performance chains are too short. Domhan et al. (2015) initializes the chain by setting the parameters for each of the K basis curves to their Squared Estimates (LSE). Weights are initialized to be K If this initial point violates the constraints imposed by the prior, a default starting point is used instead. As an ablation, we compare it to an initialization always starting at the default (de (aeluult), As another ablation, we compare to a strategy using the maximum posteriori estimate (MAP) starting point instead, which unlike LSE takes the likelihood under the prior into account.
Figure I0 shows the quality and cost of inferences for each of the initialization strategies when collecting up to 100000 samples (per walker). While these results clearly show that increasing chain length continues to improve performance. The trends we observe, suggest that PP could eventually attain or even overtake the best LC-PFN. That being said, the best MCMC we considered has 20 x00 longer runtimes than that LC-PFN requires, yet it does not quite reach the same performance, so outperforming it would require impractically long chains. When optimistically extending the trends, we estimate to need at least 10x longer chains, and inference times of multiple hours. terms of initialization, we observe that MCMC performance for shorter chains indeed more strongly depends on the initialization strategy, where the LSE strategy by Domhan et al. (2015) does best for short chains, followed by MAP and defaul 1 t. When increasing chain length, we find that differences get smaller and order reverses such that the fixed dafault initialization is best, suggesting that greedy initialization (LSE, MAP) may hurt performance on some curves.
C.2 Extrapolating real-world learning curves
we also perform pairwise comparisons of the log-likelihood and mean squared error values among the different methods (LC-PEN and MCMC variants). This approach allows for more detailed analysis, as it compares the absolute values of the metrics. Unlike the rank plots, these pair and curvewise plots provide insights into outliers which thus further enhance understanding of the results.
In addition to the average rank plots presented in Section 4.2
18

---

Percentage
LCBench
NAS-Bench-201
Taskset
PD1
10
LC-PP%
0-
-20
MCNCCoour prior)
MCMOlour prior)
McMCiour prior)
McChClour prior)
(a) PFN vs MCMC with our prior (MCMC-PP) -P))
LCBench
NAS Bench-201
Tasksat
LC-PN
0-
-20
-10
10
MCHCConginali
MCMCloriggnal
MCMCCoolggnaa)
MCNClorignall
(b) PFN MCMC as Domhan et al. [2015 (MCMC-OP)
Figure 11: Pairwise comparison of log- ligl-iiklihhodd (higher is better) between PFN and MCMC.
Figure presents pairwise comparison of the log-likelihood values between our PFN method and the two variants of MCMC, per curve, considering different cutoff values on the four benchmark datasets. Figure 12 presents the same comparison based on mean squared eтrоr (MSE).
We observe that LC-PFN compares favorably to MCMC-PP PD1 and MASS-Bnnch-20- tasks. Both methods demonstrate comparable performance on the Taskset dataset. However, is worth noting that for some curves of the LCBench dataset, MCMC-PP obtains very high likelihoooo and low MSE scores, while LC-PFN falls short in this aspect. We believe this discrepancy can be attributed the fact that the LCBench dataset contains significant number of constant curves (see Figure (h) that fall within the same bin of the discretized PPD, effectively limiting the the maximal confidence / accuracy of LC-PFN. Looking at MSE specifically, we find that for curves, at high cutoffs, LC-PFN obtains very high errors, while MCMC-B -P does not. Upon closer inspection, we found that these curves quickly converge to a value close the optimal accuracy, and while LC-PFN captures this trend for lower cutoffs, it suddenly fails for larger cutoffs (see 5th example Figure 18). This can likely be explained by the fact that these curves are not adequately captured by the prior proposed in Section 3.2 and this seems to be one of the few cases where LC-PEN generalizes poorly out of distribution.
LCBench
When comparing with MCMC-OP, we observe that MCMC-OP has relatively few outliers terms of log-likelihood, both in the positive and negative sense. This can likely be explained by the prior of MCMC-02, which is more flexible (i.e. has higher entropy) than ours and making MCMC-OP thus less confident about its predictions. However, MCMC-OP performs clearly worse terms of MSE, suggesting that the median of the high entropy PPDs produced by this method do not provide good point estimate.
19

---

Percentage curve atserved
80%
40%
20%
10%
LGBench
NAS-Bench-201
Taskset
PD1
10
LCWW
10
10"50
10r32
10-11
10-30
10-1
10
10-1
10-11 10-33
10-1
10t4
10-1
10-15
10-11
30
10-4
10th
10
10-1
10t7
10
10
MCMCClour prior)
MCMMClou prior)
McMEleur
MCMClour
(a) PFN vs MCMC with our prior (MCMC-PP)
LCBench
NAS Rench-201
10a1
LCPP
10-1-
10"14
10-13
10-20
10-)
10-1
10-1
10-12
10-11
10-1
10th
10-1
10-44
10-1
10-4
10
10.4
10-44
10-7
10-1
10
MCMCCorginal)
MCNCZonginal)
MCMCZonginal)
MMMcConginn))
(b) PFN MCMC as Domhan et al.
2015
(MCMC-OP)
Figure 12: Pairwise comparison of MSE values (lower is better) between PFN and MCMC.
C.3 Application: Extrapolation-baat early stopping in model selection
In what follows, we further analyze and discuss our experiments described in Section 4.3
C.3.1 Results on individual tasks
While Figure 5 shows the results of our early stopping experiments for each PD1 task, results for the three remaining benchmarks summarized by averaging them hide task-dependent variation and outliers, we have a closer look at the performance on individual tasks. Figure 13 shows the results for each of 35 LCBench tasks. Here, we observe that relative performances vary per task. The aggressive Pat Paice (k) (i.e., low k) baselines perform best on some tasks, but fail on others. The LC-PFN based termination criteria consistently perform well on each task, obtaining 2-6x speed-up (w.r.t. the final performance of no-stop) on 30 of the 35 tasks, and no significant slow-down on the remaining five. Figure [ shosows the results for each of the 3 NAS-Bers cB--200- tasks. Here, we observe failure on all 3 tasks, where the degree of failure seems correlated with the complexity of the image classification task. Figure shows the results for each of the 12 Taskset tasks. Here, the LC-PEN based termination criteria consistently perform well, obtaining 2-6x speed-ups on 12 tasks. We conclude that F accurrtely reflects relative performances on all four benchmarks.
all tasks. Since this may
20

---

301
150
46
15.
U'hern Jomber
Ldb
12feen vibble
LCbech
Figure 13: Average anytime regret (lower is better) obtained by all early stopping criteria for 40 run orderings for each of the 35 LCBench tasks.
21

---

patiencat00
NAS M45- cFar30
NAS-Bench-201 -ofari00
N65-Benchh00 troaeteetle..1
Caneeee Ragee
2211
12-4
2000
2305
2000
4000
322
1000
:300
2020
2202
2000
Figure 14: Average anytime regret (lower better) obtained by all early stopping criteria 40 run 201 tasks.
orderings for each of the 3
teatip
patinnal
2atarce
petancep2)
IC
Todcue patch178 28, muy,
Tesket - patcht2
FRUM weg
Twicset parrh1 /STM178 In178
6x:0-
6X
6x30
2x:0
10000
Kamsga
1x
200
420
132
200
1000
700
422
600
500
1000
220
433
EED
802
1020
Spods
Epocts Takred
Spodtt
8x
4.30
Jerrey Aager
Jrsrage Sogee:
20200
2x:0
2x12*
24:011
3000
220
3skset patch128 LSTM128 emw.d126..ccc b564
Teskstt patch32 JRNNE4 relu Jest 36128
bskset patch32, VEMM54 rdu, 3vg bc128
2xx10
Kanogy
2x30
2x12*
7xt0
10
a Tpochs
Freens
Tports
Trannd
Thakset pwwch522 GRU128 bs128
Thrcset paten32 C0ne022_LSTTTL1 LSTM1 20 F128 px120
Tho-set petch12 VAN..6 6C ww___nn_wwggggattt 20122
EX40
6'.12
6.00
4.10-
farry bogar
3.10
3x
2
242011
200
410
620
200
1000
200
401
600
500
1000
21.0
400
6C0
602
1010
Spochs
Epects
Spachs
Figure 15: Average anytime regret (lower is better) obtained by all carly stopping criteria for 40 run orderings for each of the 12 Taskset tasks.
22

---

C.3.2 Parameter sensitivity analysis
In our comparison in Figure we considered two variants of the LC-PFN based termination criterion, varying the frequency at which it is applied, and found the fine-grained strategy, applying every epoch, to perform best, However, many variants of this scheme exist, and in what follows we investigate the impact of some of our other choices on performance.
In our experiments, we terminate training run if we are confident it will not improve upon the best model seen so far. Following prior art [Domhan et al. 2013]. we adopted a confidence threshold of 0.95. Figure 16 (a) shows the anytime performance of variants using lower (0.90) and bigher (0.99, 0.999) confidence levels. We observe that all of these perform relatively well, suggesting some degree of robustness. The speed-ups obtained using higher confidence levels are mostly lower (up to 2x), but more consistent. In particular, we only observe minimal slow-down on confidence level 0.999, and higher confidences are also beneficial for the PD1 protein sequence (UniRef50) task.
Confidence level:
NAS-Bench-22 20 201
While apply our predictive termination criterion every epoch, we do not apply "as of the first epoch", but only after two observations are available (T 2 2). Since LC-PFN's predictions conditioned on the partial curve only, it will predict the prior for T = 0. Beyond not allowing us to discriminate between runs, it introduces a potential failure mode for easy tasks (many training runs obtain models close to the theoretical optimum) and LC-PFN (unaware of task hardness) will terminate runs before they even began because is very unlikely to obtain better model under the prior. Figure (6) (b) shows the anytime performance of variants that apply the criterion as as one, three, four, or five observations are made. We find that a choice of one (instead of two) only negatively impacts performance on LCBench (containing some very easy tasks). A minimal cutoff of two works well on all benchmarks, with minor slow-downs for the higher minimal cutoffs.
Minimal cutoff:
C.4 Qualitative plots of learning curve extrapolation
To complement the quantitative evaluation in the main paper, we provide examples of extrapolations, at the four different cutoffs, for seven curves from the prior (Figure 17p. LCBench (Figure T8). NAS-Bench-201 (Figure 19). Taskset (Figure 20), and PD1 (Figure 21). The plots show the median and two-sided 90% confidence interval of the PPDs inferred using the LC-PFN and MCMC-PP P? variant considered in Section 4.2 We observe that while the quality of the extrapolations produced by LC- PPN varies strongly, they are mostly logical, given the data observed and the prior used, and find that the inferences of MCMC-PP are rarely preferable.
D Breakdown computational cost
Overall, reproducing all our experiments in the main paper requires approximately 163 CPU days and 60 GPU hours on our systems (GPU: NVIDIA (R) GeForce (R) RTX 2080, CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz). These costs break down as follows:
entails 80 CPU days to run all MCMC-PP variants on the 10000 sampled curves. The overall prediction time of LC-PFN negligible (<1 hr for each variant). However, for LC-PFN, there is an initial training cost, which amounts to approximately 60 GPU hours to train all 27 LC-PFN variants.
Section 4.1
Section 4.2 requires 80 CPU days to run the MCMC algorithm on the considered real curve benchmarks.
Section 4.3 involves a maximum of CPU days to replicate the early stopping results on all the benchmarks for both LC-PFN and the baselines.
23

---

Cllenc
6x110
22
522
100e
1966
2003
2558
2283
1900
0330
631
1300
1D1 Larguage Modiling (1m1))
PDL Were
PDL Protoon Secperie (UmBeeo)
127-
3,107
2:10
382
230
300
500
D00
700
5003
10C.9
15300
20003
25000
130
200
300
400
(a) Varying confidence levels (1 5)
300
UC (nsoee))
LC Eap
LCBench
6x10
2x10
2003
FD1 Larggusee Modeling (mmbb)
POL Tendatten
P3) Rictenn Sequerce LetBa
12"
nage Ingrer
1011
8-10 2
12007
352
993
16C38 1930 28033
2540
201
(b) Varying minimal cutoff (< T)
Figure 16: Parameter sensitivity analysis of our fine-grained LC- IC-PFN based termination criterion in terms of the average regret (lower is better).
24

---

Alle Wi
hart bo fostt
woth 4o
Figure 17: Extrapolations of different curves from the prior at 10, 20, 40, and 80 cutoff
25

---

Figure 18: Extrapolations of 7 different curves from LCBench at 10%, 20%, 40%, and 80% cutoff
26

---

Figure 19: Extrapolations of different curves from NAS- Benchh2h1 10%, 20%, 40%, and 80% cutoff
27

---

Figure 20: Extrapolations of 7 different curves from Taskset at 10%, 20%, 40%, and 80% cutoff
28

---

19
180
340 JDCUP
10
20
Figure 21: Extrapolations of 7 different curves from PD1 at 10%, 20% 40%, and 80% cutoff
29


Reviews:
[{"review_id": "58nQg2CxUS", "reviewer": "Reviewer_KctK", "summary": "The authors used prior fitted networks to perform learning curve extrapolation. Crucially, they demonstrate that their method vastly outperforms approximate Bayesian inference via MCMC (both in terms of inference time and predictive log likelihood). Moreover, they demonstrate that the proposed approach outperforms heuristics for ending unpromising training runs commonly used in hyper-parameter optimization.", "questions": "Below I will list some comments\n1. In figure 1 there is a black dashed line but I can't find in the text what it represents. Without prior knowledge, I thought that maybe it was the true log posterior predictive distribution, which would make it seem like MCMC is outperforming the proposed method. Also, error bars here would be great.\n2. For all results, it would help if it was made clear if higher values are better or the converse.\n3. For Fig. 4, errors bar would be very informative, especially given the range of values. I suggest plotting 20% and 80% quantiles.\n4. In Fig 5., it isn't clear what the error bars are, i.e. standard error, quantiles, etc.", "limitations": "The authors did a good job explaining the limitation of the methods.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper prevents a super practical method that not only performs well *but* also runs fast, allowing it to be used in many real-time applications. The practicality of the method is extremely bolstered by the impressive suite of empirical experiments performed. Lastly, the paper was written well and was pretty easy to follow (though I have some comments).", "weaknesses": "The biggest weakness to me is that I don't think the authors spent time on the potential difficulty of using MCMC for this problem. Specifically, the prior, and the corresponding posterior, is constrained on some non-standard subset that prevents standard MCMC algorithms to be used. The geometry of this will heavily affect the performance of MCMC algorithms as samples that are proposed outside of the set will always be rejected requiring substantial tuning. That being said, I am a little concerned with the number of samples the chain was run: 4,000 samples seems too low for this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding the weaknesses you raised:\n\n**The biggest weakness to me is that I don't think the authors spent time on the potential difficulty of using MCMC for this problem. Specifically, the prior, and the corresponding posterior, is constrained on some non-standard subset that prevents standard MCMC algorithms to be used ….** \n\nWe agree that Bayesian LCE is a non-trivial problem for MCMC. In fact, this was an important motivation of LC-PFN. For our MCMC baseline, we relied on previous work (Domhan et al., 2015) to make MCMC work reasonably in this challenging setting. While we were forced to make minor changes to the setup, in particular to the prior, we made considerable efforts to retain the competitiveness of the original implementation. Furthermore, we also included the original MCMC implementation as a baseline in our experiments in Section 4.2. To further address your concern, we conducted extensive experiments the results of which can be found in the PDF attached to our global response. First, it is important to note that our original results in Figure 3 in the paper are averages across 10,000 curves. When comparing performances on individual curves, shown in Figure 2 (left, red dots) in the attached PDF, we observe that for the majority of the curves, the best LC-PFN and MCMC perform similarly (i.e., are near the diagonal), and inferior average performance can be largely attributed to ‘failures’ of MCMC on a small fraction of the curves, the ones that are in the lower-right region of Figure 2 (from the rebuttal PDF). After analyzing these failure cases of MCMC, we noticed behavior inconsistent with what was described by Domhan et al. (2015) and fixed the bug we thereby identified as discussed in the global response. Doing this and running MCMC much longer, as you suggested (more below), reduces these failure cases considerably (see Figure 2 (left to right, red vs green dots)).\n\n**I am a little concerned with the number of samples the chain was run: 4,000 samples seems too low for this problem.**\n\nWe understand your concern. It is worth noting that Domhan et al. (2015) only used 2,000 samples, which motivated our choice of 4,000 samples as an upper limit. Also, please note that Domhan’s implementation (which we use) uses an ensemble sampler (EMCEE) with 100 workers, and the number of 4,000 samples is actually for each worker, which results in 400,000 samples overall. The trend we observed in Figure 3 of our paper is that while increasing chain length indeed improves performance, the slow rate of improvement suggests that impractically long chains would be required to attain the performance of even the smallest PFN. However, after fixing the bug mentioned above in Domhan’s original implementation, improvement no longer stagnates, and results running MCMC for up to 100,000 samples per worker (i.e., a total of 10M samples) suggest that the gap in performance will close with even more samples (see Figure 1 in the rebuttal PDF, red line). That being said, MCMC still does not reach the average performance of the best LC-PFN, while taking approximately 25,000 times as long.\n\n### Regarding your questions / comments:\n\n**In figure 1 there is a black dashed line but I can't find in the text what it represents. Without prior knowledge, I thought that maybe it was the true log posterior predictive distribution, which would make it seem like MCMC is outperforming the proposed method. Also, error bars here would be great.**\n\nUnless we are mistaken, there is no black dashed line in Figure 1 of the paper. The red/blue dashed lines in Figure 1 represent the 5 and 95 percentiles of the PPD (so the shaded area ~ 90% CI) as inferred using MCMC/LC-PFN, respectively. We will update the legend of Figure 1 to clarify this. We assume you meant Figure 3 instead. The black dashed line in Figure 3 represents the highest log score attained by any MCMC variant (higher is better). Thus, this does not imply that MCMC is outperforming the proposed method but the opposite. We will update the legend / caption of Figure 3 to clarify this.\n\n**For all results, it would help if it was made clear if higher values are better or the converse.**\n\nDefinitely, thank you for the suggestion, FYI:\n- Figure 3 / Table 3: \n    - Higher is better for log score (average LL)\n    - Lower is better for average runtime\n- Figure 4: Lower is better (average rank)\n- Figure 5: Lower is better (average regret)\nWe will clarify this in the captions.\n\n**For Fig. 4, errors bar would be very informative, especially given the range of values. I suggest plotting 20% and 80% quantiles.**\n\nAgreed. However, as Figure 4 shows averages of ranks amongst 3 methods, we feel quantiles (which will be one of the integers 1, 2, or 3) would not be very informative. We propose to use +-1 standard error (SE) instead.\n\n**In Fig 5., it isn't clear what the error bars are, i.e. standard error, quantiles, etc.**\n\nThe shaded area corresponds to +- 1 standard error. We will update the caption to clarify this, thanks!\n"}, {"review_id": "uuhcjMuy8p", "reviewer": "Reviewer_tPbz", "summary": "The authors propose applying prior-data fitted NNs to learning curve extrapolation.\n\nThe authors demonstrate that this approach outperforms MCMC inference and is substantially faster.\n", "questions": "Is poor performance of MCMC attributed to slow mixing or initializing at a poor point? What about MCMC inference initialized at MAP?\n\nIf MCMC has substantially more computational time will it outperform PFN? At the end of the day exact inference should outperform approximate inference.\n\n", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a novel idea of applying approximate inference via meta-learning learning curves.\n\nThe method appears to work in experimental evaluation, but more experiments would make the paper stronger.\n\nThe manuscript (all but experimental section) is clear and relatively easy to follow.\n\nThe authors provide the code.\n\n", "weaknesses": "The paper contains only two experiments. One is based on synthetic data.\n\nTable 1 has no error bars. Table 1 indicates that while the performance of MCMC is stable across the hyperparameters, PFN score is dependent on the network size indicating it might not generalize well to other problems.\n\nWhile the difference in time is clearly significant, the differences in LL are rather small.\n\nWhile the idea of applying the described method in the considered context is novel, much of this work is a combination of previous work by Domhan 2015 and Muller 2022.\n\nThe experimental section could have been written in a more clear way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding the weaknesses you raised\n\n**The paper contains only two experiments. One is based on synthetic data.**\n\nWe evaluated LC-PFN in three different experimental setups (~ Section 4.1, 4.2, and 4.3). In Section 4.1, we evaluate LC-PFN (and MCMC) on samples of the prior. From a learning curve perspective, this is indeed ‘synthetic’ data. However, from a Bayesian perspective, this is arguably the best way of evaluating and comparing the ability of both methods to perform approximate inference without confounding factors. In Sections 4.2 and 4.3 we use real learning curve data from four different benchmark suites, training a wide variety of different architectures (e.g., MLP, CNN, RNN, Transformer) on a variety of different datasets (e.g., tabular, images, text, protein data) using a variety of different hyperparameter settings (e.g., optimizer, learning rate).\n\n**Table 1 has no error bars.**\n\nTable 1 in the paper does not present experimental results. We assume you mean Table 3? Our apologies, please find the updated table including the standard error in the PDF attached to the global rebuttal (as Table 1).\n\n\n**Table 1 indicates that while the performance of MCMC is stable across the hyperparameters, PFN score is dependent on the network size indicating it might not generalize well to other problems.**\n\nTable 3 only shows three configurations for each method. In total, we evaluated 27 PFN variants and 216 MCMC variants (see Table 2), and the results for all of these are shown in Figure 3 of the paper. Here, you can see that there is also variability amongst the MCMC configurations (red dots). For PFNs, variability is mostly determined by nb_data (how many curves the PFN was trained on) and to a lesser extent its size (``nlayers`` and ``emsize``). In fact, the figure shows that each of the 9 PFNs considered can be trained to outperform any of the MCMC configurations. While larger models do better (as also observed in other studies using transformers, e.g., Kaplan et al., 2020), this is generally viewed positively (e.g., the method can be scaled up with the availability of compute).\n\n**While the difference in time is clearly significant, the differences in LL are rather small.**\n\nAgreed, our main claim is that LC-PFN is faster without loss of quality. \n\n**While the idea of applying the described method in the considered context is novel, much of this work is a combination of previous work by Domhan 2015 and Muller 2022.**\n\nAgreed, our work essentially applies the method of Muller 2022 (PFNs) to the problem of Domhan 2015 (Bayesian LCE). However, given the novelty of the method, the relevance of the problem, and the magnitude of improvement (in terms of time complexity / practicality), we believe this to be a very significant contribution nonetheless; reviewer 3hos also seems to share this perspective. Also, we made several smaller, yet relevant contributions. In particular, we introduce a novel normalization procedure (see Appendix A), allowing both LC-PFN, and Domhan et al.’s (2015) MCMC to be applied to learning curves using a wide variety of different, possibly unbounded, performance metrics (not just maximizing a metric in [0,1], e.g., accuracy), including the popular “minimizing log loss”.\n\n**The experimental section could have been written in a more clear way.**\n\nThank you for your feedback, we will improve the writing of this section for the final version but any concrete suggestions you have to further improve this section would be much appreciated.\n\n### Regarding your questions (in reverse order):\n\n**If MCMC has substantially more computational time will it outperform PFN? At the end of the day exact inference should outperform approximate inference.**\n\nYou are right, at least in theory and in the limit this should be the case (MCMC with finite chains is also approximate inference). However, the trends observed in Figure 3, in Section 4.1, seem to suggest only marginal improvements can be made by further increasing chain length. To further address your concern, we conducted extensive experiments the results of which can be found in the PDF attached to our global response. In summary, we discovered / fixed what we presume to be a bug in the original implementation (see global response for details), and ran experiments considering up to 50x longer chain lengths. The trends we observe (see Figure 1 in the rebuttal PDF), suggest that MCMC could eventually attain or even overtake the best LC-PFN. That being said, the best MCMC has approx. 25.000x longer runtimes than LC-PFN requires, MCMC yet not quite reach its performance, so outperforming it would require impractically long chains.\n\n**Is poor performance of MCMC attributed to slow mixing or initializing at a poor point? What about MCMC inference initialized at MAP?**\n\nThis is a very good question. First, a few words on how we initialize the chain (also see Domhan et al., 2015, Section 3.1, page 4). Following Domhan et al. (2015), parameters for each of the basis curves are Least-Squared Estimates (LSE) obtained using the BFGS optimizer (Scipy implementation). Weights are initialized to be 1/K, where K(=3) is the number of basis curves. If this initial point violates the constraints imposed by the prior, a default starting point is used instead. Following your recommendation, we investigate the behavior of MCMC with different initializations including starting from the MAP estimate and the default value of the parameters. Figure 1 in the rebuttal PDF shows that the MAP does not improve over the original LSE implementation of Domhan et al. (2015). The fixed default initialization is worst for short chains, but best for very long chains (>20,000), suggesting that greedy initialization (LSE, MAP) may hurt performance on some curves.\n"}, {"review_id": "YLLjSawIOA", "reviewer": "Reviewer_7zKX", "summary": "The authors in this submission applied the prior-data fitted neural networks (PFNs) in the learning curve extrapolation task, for which the main goal is to predicit the performance of a machine learning model in later epochs, based on the information from earlier epochs. The target is modeled as a linear combination of basis growth curves, and the proposed LC-PFN algorithm aims to minimize the cross entropy loss as in PFN. The authors tested the proposed method across a few datasets and showed that PFN (i) achieves better prediction and (ii) is computationally efficient, when compared with its competitors.", "questions": "1. In Table 1, is the support for x integers, as it corresponds to $f_k (t | \\theta_k)$ in the model?\n\n2. For the equation just below L132, what does $<$ mean? \n\n3. Do you need to assume $t > T^{'}$ in your LC-PFN model during training? ", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of applying PFNs in the learning curve extrapolation task sounds reasonable, and is intuitively more efficient when compared with MCMC.\n\n2. The application of LC-PFN in early stopping could be valuable, as it can be helpful for the model selection step. The experiments in Section 4.3 also provide some promising results. ", "weaknesses": "1. In my opinion the selection for the 3 parametric basis curves seems a bit ad-hoc. It's not convinving whether they are sufficient to fit different learning curves. Furthermore, selection of hyperparemters in priors there also lacks details.\n\n2. The results could be more convincing if the authors can compare aganist more previous methods, instead of MCMC only. For instance, how about the one by Klein et al., 2017, which used Bayesian neural networks for learning curve prediction?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding the weaknesses you raised:\n\n**In my opinion the selection for the 3 parametric basis curves seems a bit ad-hoc. It's not convincing whether they are sufficient to fit different learning curves. Furthermore, selection of hyperparameters in priors there also lacks details.**\n\nThanks for the question. We would like to add some further context on how we got to this prior: For a fair comparison, we chose this prior to mimic the setting in Domhan et al. (2015), which is the most closely related to the prior art. Sadly, using the exact same prior was not possible (we cannot generate samples from a uniform distribution with unbounded support, see Section 3.2). Hence, we decided to ‘peak’ the prior / limit parameter ranges to cover a wide spectrum of possible curves, while still guaranteeing the competitiveness of our baseline. Despite these efforts, we agree that the resulting prior is far from perfect (see also Section 5, l326). Note that we settled for 3 curves, as using all 11 curves would have only partially fixed this issue, made this ‘peaking the prior’ exercise much more complicated, and in relative comparisons would likely negatively impact our MCMC baseline. In Section 4.2, we also compare to the original implementation using all 11 basis curves, to establish that LC-PFN is competitive on real learning curves, despite using only 3 basis curves.\n\n**The results could be more convincing if the authors can compare against more previous methods, instead of MCMC only. For instance, how about the one by Klein et al., 2017, which used Bayesian neural networks for learning curve prediction?**\n\nTo the best of our knowledge, and based on a recent literature survey (Mohr and van Rijn 2022), Domhan et al. (2015) is the only work that previously considered Bayesian LCE requiring ONLY a single partial learning curve as input. LCNet (Klein et al., 2017), see also our discussion in the last paragraph of Section 2 (l94), in addition, requires the training hyperparameter settings as input. In fact, the Bayesian neural network used by Klein et al. (2017) only models the dependency of the prior on these training hyperparameters. This dependency must be meta-learned across multiple training runs. The only way we could compare to LCNet, in the current scope, would have been to remove this dependency (e.g., fix the input of the BNN to 0), but this would essentially reduce the BNN to a (gradient-based) MCMC method.\n\nIf you are aware of any other prior art that should be included in our comparison, please let us know, and we would gladly add it.\n\n\n### Regarding your questions / comments:\n\n**In Table 1, is the support for x integers, as it corresponds to $f_k(t | \\theta_k)$  in the model?**\n\nYes. To avoid confusion, we will update the table to use t instead of x here.\n\n**For the equation just below L132, what does < mean?**\n\nIt simply means “less than”, comparing two real values (initial and final model performance), constraining the final performance to be better than the initial one (following Domhan et al., 2015).\n\n**Do you need to assume $t’>T$ in your LC-PFN model during training?**\n\nNo, one could easily train an LC-PFN by censoring arbitrary points in the curves (vs right censoring only). The benefit is that the resulting model would also be able to do interpolation at the cost of requiring a slightly larger model/training time. Since our work is about extrapolation, we decided to train LC-PFN for this task specifically.\n"}, {"review_id": "Nr1VobmbU7", "reviewer": "Reviewer_3hos", "summary": "The authors consider the task of learning curve extrapolation, i.e., the aim is to predict the performance of a given model wrt. e.g., accuracy/log-likelihood over time, given current observations.\nTheir proposal primarily relies on Prior-data fitted networks (Müller et al., 2022) trained on samples from a prior set of curves adapted from their sole comparison method by Dunham et al. (2015). \nCompared to this prior MCMC-based method (Dunham et al., 2015), the authors can show greatly improved performance in most setups as well as very strong improvements in runtime.\n", "questions": "- In l216 the authors mention that the best MCMC-PP method took over 80 seconds. In Table 3 the 80s method does, however, not not improve upon the 30-second version. Can the authors clarify this statement?\n", "limitations": "The authors discuss limitations of their method, but no potential negative societal impact of their work. The second is of lesser importance in this work anyway.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper focuses on an important task in the AutoML literature that has so far not received a lot of attention.\n- The authors evaluate their approach extensively on a varied set of experiments properly evaluating all their claims.\n- The paper is overall well written with two minor deficiencies (see next section). \n- The authors provide an extensive implementation of the model. However, it is also necessary as replicability would be difficult given the coarse level of detail in the written part.\n- With respect to originality. The contribution is primarily in the application. The model itself relies on the PFN model by Mueller and the prior is an adaptation of Dunham et al.\nThis can be seen as a weakness, yet given that the application of PFNs to this field is novel and the application is an important one with clear results, I see this only as a very minor weakness if at all.\n- Concerning significance, I lack a deep understanding of the AutoML literature to properly judge the significance. But from what I know of the field, the results look very promising and should be of great interest to many readers.", "weaknesses": "- It has two deficiencies in the writing/structure of the paper. \n   1.  Its backbone is a prior-data fitted network. However, that model itself is only briefly introduced in a single paragraph with a figure that is barely understandable without reading the original paper.\nThe paper requires a proper discussion of this approach in either the main text or the appendix. \n   2. Training details and hyperparameters are barely discussed with many essentials missing, e.g., the reader has to guess the meaning of `nb_data`, `emsize`, `nlayers`. It is an easy guess, but a guess nevertheless. Adding the abbreviations to the paragraph in l157 would quickly fix this.\n- As stated above, the novelty is almost solely in the application which could be considered a weakness. But in my opinion this is only a very minor weakness if at all.\n- Table 3 lacks error bars.\n\n## Minor\n- Figure 2 is missing details on the colors of the arrows \n- The caption of Table two should be above\n- Figure 3 contains an unexplained horizontal line\n- l319-320 the sentence is broken (interesting appears twice)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding the weaknesses you raised:\n\n**Its backbone is a prior-data fitted network. However, that model itself is only briefly introduced in a single paragraph with a figure that is barely understandable without reading the original paper. The paper requires a proper discussion of this approach in either the main text or the appendix.**\nWe agree that our description thereof in the current manuscript is rather high-level. We will add a section elaborating on some PFN-specific implementation details and design choices in the appendix. (We also would like to mention that the code for both PFN and LC-PFN is open source, at least specifying all details in the code. But that’s of course not an excuse for the method to be only explained very briefly in the paper, and we’ll fix that.)\n\n**Training details and hyperparameters are barely discussed with many essentials missing, e.g., the reader has to guess the meaning of ``nb_data``, ``emsize``, ``nlayers``. It is an easy guess, but a guess nevertheless. Adding the abbreviations to the paragraph in l157 would quickly fix this.**\n\nWe will modify this paragraph accordingly, thank you for pointing out this issue and your suggestion.\n\n**As stated above, the novelty is almost solely in the application which could be considered a weakness. But in my opinion this is only a very minor weakness if at all.**\n\nAgreed, thank you for this perspective.\n\n**Table 3 lacks error bars.**\nOur apologies, please find the updated table including the standard error in the PDF attached to the global rebuttal (as Table 1).\n\n**Minor comments**\nWe will make the necessary changes, thank you for pointing out these minor issues.\n\n\n### Regarding your questions / comments:\n\n**In l216 the authors mention that the best MCMC-PP method took over 80 seconds. In Table 3 the 80s method does, however, not not improve upon the 30-second version. Can the authors clarify this statement?**\n\nThis is an unfortunate error in the discussion. Our apologies and thank you for spotting it.\nIt should say  “the best MCMC-PP method took over 30 seconds.”\n\nContext: In our preliminary experiment, conducted with a smaller sample size, the mentioned MCMC configuration appeared to show a slight advantage. However, this difference did not hold up in the final, large-scale experiment. Unfortunately, we overlooked updating this information in our discussion, and we apologize for any confusion this may have caused. In the final experiment, in Section 4.1, we found that no single MCMC variant consistently outperformed others across different cutoffs. Given the subtle differences among top-performing configurations and the varying computational efficiency, we chose the more efficient variant for future comparisons. Note that this ‘cheaper’ MCMC variant still requires multiple orders of magnitude time than the cheapest LC-PFN variant (to obtain a worse average LL).\n"}]

Author rebuttal:
In this global response, we would like to thank all reviewers for their constructive feedback. We are glad that our work was generally well-received and we address specific concerns / questions raised by the reviewers in our individual responses. Multiple reviewers expressed some concerns about our MCMC baseline. To support our response, we conducted extensive experiments that we discuss in the following; accompanying figures are found in the attached PDF.

**Improving baseline implementation:** We analyzed cases where MCMC performed poorly in order to determine any potential causes of its non-convergence to the optimum. In these analyses, we noticed behavior inconsistent with what was described by Domhan et al. (2015). Specifically, the hard constraints (monotonicity and range) were checked for every basis curve ($f_k$), rather than on the combination ($f_{comb}$), as described in their paper (see Domhan et al., 2015, Equation 5). While we assume this was intended as an optimization in their code, it is not equivalent (i.e., a combination can satisfy these constraints, while individual curves do not) and this is exactly what was happening for the curves MCMC performed worst on. After fixing this (which we would call a bug) in their original implementation, we re-ran the fixed version of the controlled experiment (Section 3). This yielded results much more in line with what one would expect from MCMC for higher sample sizes: while it takes a long time, it finds solutions as good as those by LC-PFN (see Figure 1 in the rebuttal PDF). It is crucial to emphasize that despite these improved results, our main claim about LC-PFN obtaining many orders of speedup without loss of quality w.r.t. MCMC remains valid.


**Ablating MCMC w.r.t the initialization and the number of samples:** We ran experiments to assess the effect of the initialization strategies, including the original least-square estimate (LSE), the Maximum a posteriori (MAP) estimate, and a fixed default value of the parameters. Furthermore, we ran experiments on considerably larger sample sizes (up to 100,000 samples). As shown in Figure 1 in the attached PDF, LC-PFN still consistently outperforms MCMC in this ablation.


Meta-review: This paper applies prior fitted networks (PFNs) to learning curve extrapolation. Predicting learning curve for early stopping is an important research topic in AutoML. This work applies PFNs to improve the inference of the Bayesian model proposed in Domhan et al. (2015). It results in better prediction and significantly higher computation efficiency. 

The main idea is an application of PFNs to the model of Domhan et al. (2015) with some modification on the prior although some reviewers still consider this novel application in learning curve extrapolation as a good contribution.

Reviewers consider it as a highly practical method with good performance. It is also evaluated on a variety of real-data experiments.

Some reviewers have concerns in the experiments, including (1) the Bayesian model with only 3 basis curves maybe too simple to capture all learning curves (2) comparison with additional existing works. The authors' rebuttal provided explanation on the model choice and why the additional work is not applicable in the scope of the paper.
Decision: Accept (poster)