Title: Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning curve extrapolation

prior-data fitted networks

transformers

Bayesian inference

uncertainty estimation

model selection
Abstract: Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance in earlier epochs.
In this work, we argue that, while the inherent uncertainty in the extrapolation of learning curves warrants a Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. We describe the first application of prior-data fitted neural networks (PFNs) in this context. A PFN is a transformer, pre-trained on data generated from a prior, to perform approximate Bayesian inference in a single forward pass. We propose LC-PFN, a PFN trained to extrapolate 10 million artificial right-censored learning curves generated from a parametric prior proposed in prior art using MCMC. We demonstrate that LC-PFN can approximate the posterior predictive distribution more accurately than MCMC, while being over 10 000 times faster. We also show that the same LC-PFN achieves competitive performance extrapolating a total of 20 000 real learning curves from four learning curve benchmarks (LCBench, NAS-Bench-201, Taskset, and PD1) that stem from training a wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 - 6x speed-ups on 45 of these datasets, at virtually no overhead.

Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks

Abstract

Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance in earlier epochs. In this work, we argue that, while the inherent uncertainty in the extrapolation of learning curves warrants a Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. We describe the first application of prior-data fitted neural networks (PFNs) in this context. A PFN is a transformer, pre-trained on data generated from a prior, to perform approximate Bayesian inference in a single forward pass. We propose LC-PFN , a PFN trained to extrapolate artificial right-censored learning curves generated from a parametric prior proposed in prior art using MCMC. We demonstrate that LC-PFN can approximate the posterior predictive distribution over learning curves more accurately than MCMC, while being over 10 000 times faster. We also show that the same LC-PFN achieves competitive performance extrapolating a total of 20 000 real learning curves from four learning curve benchmarks ( LCBench , NAS-Bench-201 , Taskset , and PD1 ) that stem from training a wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 - 6 × speed-ups on 45 of these datasets, at virtually no overhead.

1 Introduction

Learning curve extrapolation [Mohr and van Rijn, 2022] aims to predict how much a machine learning model will improve with more training, e.g., to determine how much more training data to collect [Cortes et al., 1993, Frey and Fisher, 1999, Leite and Brazdil, 2004, Kolachina et al., 2012], or to define an early stopping criterion in online learning [Yao et al., 2007]. Learning curve extrapolation has recently been widely studied to speed up automated machine learning (AutoML) and hyperparameter optimization (HPO) of deep neural networks, by discarding non-promising configurations early [Swersky et al., 2014, Domhan et al., 2015, Klein et al., 2017, Baker et al., 2017, Chandrashekaran and Lane, 2017, Gargiani et al., 2019, Wistuba et al., 2022].

Herilalaina Rakotoarison ∗

Frank Hutter

Despite these efforts, learning curve extrapolation is not yet widely adopted in practice, e.g., state-ofthe-art multi-fidelity hyperparameter optimization techniques, such as BOHB [Falkner et al., 2018], still rely on successive halving [Li et al., 2017], i.e., the crude heuristic that learning curves mostly do not cross each other.

One reason for this is that, while many learning curves are well-behaved, some exhibit chaotic behavior and are intrinsically difficult to predict accurately [Choi et al., 2018]. In this setting, Bayesian approaches [Swersky et al., 2014, Domhan et al., 2015, Klein et al., 2017, Wistuba et al., 2022], which also quantify the reliability of their extrapolation, show great potential. However, existing methods for Bayesian inference either (i) put strong restrictions on the prior, and are incapable of modeling the variable nature of learning curves, or (ii) are too computationally expensive, limiting their practical applicability. Furthermore, most of this related work focuses on demonstrating the potential that learning curve extrapolation has to accelerate downstream AutoML/HPO tasks, yet fails to fully investigate the quality of the extrapolations themselves, and to quantify the approach's ability to handle the heterogeneity of real-world learning curves, e.g., varying performance metrics, curve shapes, divergence, heteroscedastic noise, etc.

In this work, we investigate the potential of learning curve extrapolation using prior-data fitted networks (PFNs), a meta-learned approximate Bayesian inference method recently proposed by Müller et al. [2022]. PFNs combine great flexibility with efficient and accurate approximation of the posterior predictive distribution (PPD) in a single forward pass of a transformer [Vaswani et al., 2017] trained on artificial data from the prior only. As PFNs are a promising alternative to Markov Chain Monte Carlo (MCMC) for approximating Bayesian inference, we compare our approach ( LC-PFN ) to the MCMC approach for learning curve extrapolation of Domhan et al. [2015], taking into account both the quality and the cost of PPD approximation.

In summary, our contributions are as follow:

We are the first to apply PFNs to an extrapolation task, introducing LC-PFN , the first PFN for learning curve extrapolation.

We demonstrate that LC-PFN can be more than 10 000 × faster than MCMC while still yielding better probabilistic extrapolations.

We show that LC-PFN does not only yield better probabilistic extrapolations on prior samples, but also on real learning curves of a wide range of architectures (MLPs, CNNs, RNNs, Transformers) on varying input modalities (tabular, image, text and protein data).

We demonstrate the practical usefulness of LC-PFN to construct an early stopping criterion that achieves 2 - 6 × speedups over baselines.

To facilitate reproducibility and allow others to build on our work, we open-source all code, data, and models used in our experiments at https://github.com/automl/lcpfn .

2 Related work

Learning curves and how to use them for decision-making has been an active research area, as recently surveyed by Mohr and van Rijn [2022]. Most related work considers point estimates of the curve or a specific property thereof [Cortes et al., 1993, Frey and Fisher, 1999, Kolachina et al., 2012, Baker et al., 2017, Kaplan et al., 2020], or follows a non-Bayesian approach to quantify uncertainty [Chandrashekaran and Lane, 2017, Gargiani et al., 2019].

Only a few works have explored Bayesian learning curve extrapolation. For example, the FreezeThaw Bayesian optimization method Swersky et al. [2014] used a Gaussian process (GP) as a joint model of learning curves and hyperparameters to decide what learning run to continue for a few epochs (or whether to start a new one). The model is then dynamically updated to fit the partial learning curve data. Training data grows quickly since each performance observation of the curve is treated as a datapoint, making exact GPs intractable, and thus the work relies on approximate GPs. Furthermore, their approach makes strong (prior) assumptions. On top of the standard GP assumptions, they used a specialized kernel assuming exponential growth to improve extrapolation. Domhan et al. [2015] proposed a less restrictive parametric prior (see Section 3.2 for more details) and used the gradient-free MCMC method from Foreman-Mackey et al. [2013] as approximate inference method. While MCMC is a very general approach, it can be sensitive to its hyperparameters (e.g.,

burn-in period, chain length, etc.) and, as we will show in Section 4, generating sufficient samples to reliably approximate the PPD may impose significant overhead. Klein et al. [2017] extended this parametric prior to also capture the effect of hyperparameter settings. In particular, they used a Bayesian neural network with a specialized learning curve layer, and trained this network using gradient-based MCMC on learning curve data from previously tested hyperparameter settings. While this approach is able to predict learning curves of previously unseen configurations, conditioning on the current partial learning curve requires retraining the Bayesian neural network online, which is costly. Recently, DyHPO [Wistuba et al., 2022] followed a similar dynamic HPO setup as Swersky et al. [2014], but used deep GPs [Damianou and Lawrence, 2013]. While deep GPs relax some of the standard GP assumptions, extrapolation abilities were not thoroughly analyzed, and DyHPO only predicts one epoch into the future. Finally, it is worth noting that, except for Domhan et al. [2015], all the aforementioned probabilistic approaches [Swersky et al., 2014, Klein et al., 2017, Chandrashekaran and Lane, 2017, Gargiani et al., 2019, Wistuba et al., 2022] utilize meta-learning across the learner's hyperparameter settings. While this is an interesting line of work, it limits applicability, and introduces confounding factors. We will therefore consider a simpler and more general setting in this work (see Section 3.1). Indeed, akin to the approach of Domhan et al. [2015], we operate without the assumption of access to data from previous runs employing different hyperparameter settings, nor do we assume the ability to generalize across these settings. Our results highlight that prior-data fitted networks (PFNs) offer a significantly more efficient and practical alternative to Markov Chain Monte Carlo (MCMC) methods. As categorized by Mohr and van Rijn [2022], Domhan et al. [2015] is the only comparable prior work within this category. This underlines the novelty and importance of our approach in the context of advancing current methodologies.

While we are the first to apply PFNs [Müller et al., 2022] to learning curve extrapolation, PFNs have previously been applied in different settings: Hollmann et al. [2023] used them to meta-learn a classifier for tabular data; Müller et al. [2023] as a surrogate model for Bayesian optimization; and most recently concurrent work by Dooley et al. [2023] as a zero-shot time series forecaster.

3 Methods

3.1 Bayesian learning curve extrapolation

Let y t ∈ [0 , 1] represent the model performance (e.g., validation accuracy) at training step t ∈ { 1 , . . . , m } . The problem we consider in this paper can be formulated as follows: Given a partial learning curve y 1 , . . . , y T up to some cutoff T , and a prior distribution p ( y ) over learning curves, approximate the posterior predictive distribution (PPD) q ( y t ′ | y 1 , . . . , y T ) for T < t ′ ≤ m . We will further assume that we can calculate the relative probability density of p ( y ) , a requirement for MCMC, and that we can generate samples from p ( y ) , a requirement for PFNs. Figure 1 provides an illustration of Bayesian learning curve extrapolation, showcasing the posterior predictive distributions (PPDs) of the extrapolated curves generated by LC-PFN and MCMC, along with a few representative curves sampled from the prior distribution p ( y ) .

1.0

1.0

0.8

0.8

0.6

0.6

WMWWWM

0.4

target

0.4

0.2

data

LC-PFN

MCMC

0.2

0.0

0

10

20

30

40

50

MW

Epochs

0.0

0

20

40

60

80

100

t

While the fixed ranges for y t and t are well-suited for modeling particular learning curves (e.g., accuracy over epochs), they also are restrictive. In Appendix A, we discuss the invertible normalization procedure we apply to support extrapolating, possibly diverging, iteration-based learning curves across a broad range of performance metrics (e.g., log loss).

3.2 Learning curve prior

Following Domhan et al. [2015], we model y as a linear combination of K basis growth curves f k , each parameterized by θ k , and i.i.d. additive Gaussian noise with variance σ 2 , i.e.,

where we assume our model parameters

to be random variables with prior p ( ξ ) . Here, Domhan et al. [2015] assumed an uninformative prior (i.e., p ( ξ ) ∝ 1 ), with the exception of some hard constraints. We adopt a strictly more informative prior, because (i) the original prior puts almost all probability mass on parameterizations yielding invalid learning curves, e.g., y t / ∈ [0 , 1] ; and (ii) we cannot practically sample from this prior having unbounded support, a requirement for PFNs. 1 Specifically, to mimic realistic learning curves we use bounded uniform weight priors w k ∼ U (0 , 1) , a low-noise prior log( σ 2 ) ∼ N ( -8 , 2) , only allow curves with values in [0 , 1] and, like Domhan et al. [2015], only accept curves whose last value is higher than its first. Putting all of these together, our prior distribution thus takes the form:

Finally, we limit ourselves to three parametric families of learning curves ( K = 3 , see Table 1). 2 These basis curves were chosen to capture a variety of growth trends and convergence behavior. We show examples of curves sampled from this prior in Figure 1 (right).

3.3 Prior-data fitted networks (PFNs)

In this paper, we propose to use prior-data fitted networks (PFNs, Müller et al., 2022) instead of MCMCfor learning curve extrapolation. PFNs are neural networks trained to perform approximate Bayesian prediction for supervised learning settings. That is, PFNs are trained to predict some output y ∈ R , conditioned on an input t and a training set D train of given input-output examples. The PFN is trained for this task with samples obtained from a prior over datasets p ( D ) . The loss function for training a PFN q θ with parameters θ is the cross entropy ℓ θ = E ( t,y ) ∪ D train ∼ p ( D ) [ glyph[axisshort] log q θ ( y | t, D train )] for predicting the hold-out example's label y , given t and D train . Müller et al. [2022] proved that minimizing this loss over many sampled tasks ( t, y ) ∪ D train directly coincides with minimizing the KL divergence between the PFN's predictions and the true PPD. In essence, the PFN meta-learns to perform approximate posterior inference on (meta-train) synthetic tasks sampled from the prior, and at inference time also does so for a (meta-test) real task.

(a)

epoch: 1

acc: 10%

epoch: 2

acc: 26.6%

epoch: 3

acc: 51.4%

D

q

(

·|

5

, D

)

q

(

·|

4

, D

)

epoch: 4

epoch: 5

(b)

0%

50%

100%

q

(

·|

5

, D

)

3.4 PFNs for Learning Curve Extrapolation (LC-PFNs)

To apply PFNs to learning curve extrapolation, we train them on learning curves sampled from a given prior over learning curves (in our case, the prior defined in Section 3.2). Specifically, the training set we condition on is the available partial learning curve up to some varying cutoff point T ′ , i.e., D train = { ( t ′ , y t ′ ) } T ′ t ′ =1 , the input we condition on is the epoch t ∈ { T ′ + 1 , . . . , m } to predict for, and the desired output y is the value of the learning curve at epoch t . During training, we randomly sample the cutoff points T ′ ∼ U (0 , m glyph[axisshort] 1) for every batch in order to learn to predict for initial learning curves of varying sizes. Figure 2 illustrates the information flow in a LC-PFN during learning curve extrapolation.

LC-PFN architecture and hyperparameters We use the PFN architecture proposed by Müller et al. [2022] and visualized in Figure 2 (a). That is, we use a sequence Transformer [Vaswani et al., 2017] and treat each pair ( t, y ) (for train) and t (for test) as a separate position/token. We encode these using a simple linear layer. We do not use positional encoding such that we are permutation invariant. Furthermore, the attention matrix is masked such that every position only attends to the training positions. This way training examples can attend to each other, but the test examples do not influence each other's predictions. Note that the output of the PFN with parameters θ is a distribution q θ ( y | t, D train ) . Following Müller et al. [2022], we discretize q θ in a finite number of bins whose probability mass is predicted by the PFN, as is shown in Figure 2 (b). The size of each bin is set such that, under the prior, y t is equally likely to fall in each bin. The number of bins is a hyperparameter that we set to 1 000. The LC-PFN model further inherits hyperparameters from the Transformer, including the number of layers ( nlayers ), number of heads ( nheads ), embedding size ( emsize ), and hidden size ( nhidden ). We use four heads, a hidden size of 1 024, and conduct a thorough ablation study to investigate the effects of the number of layers and embedding size on the final performance, exploring a grid of values (see Table 2). We use a standard training procedure for all experiments, employing the Adam optimizer [Kingma and Ba, 2015] (learning rate 0 . 0001 , batch size 100 ) with cosine annealing [Loshchilov and Hutter, 2017] with a linear warmup over the first 25% epochs of the training. Finally, we set m = 100 , implying LC-PFN is trained for extrapolating sequences of up to 100 training steps (e.g., epochs). We found that most curves are shorter in practice, and when longer sequences are encountered, we subsample them as described in Appendix B.

4 Experiments

Our experiments aim to test the hypothesis that PFNs present a practical Bayesian approach to learning curve extrapolation. To this end, we first compare our LC-PFN approach against the MCMC approach of Domhan et al. [2015], using the same prior on samples generated from it (Section 4.1). Then, we extend the comparison to four real-world learning curve benchmarks (Section 4.2). Finally, we look beyond the quality of individual extrapolations and evaluate the potential of LC-PFN in the context of predictive early stopping to accelerate model selection (Section 4.3).

4.1 Extrapolating samples of the prior

The goal of this first experiment is to assess the ability of LC-PFN s and MCMC to approximate the true posterior predictive distribution (PPD). To avoid artifacts due to out-of-distribution data, in this experiment, we use curves sampled from the prior (defined in Section 3.2). Furthermore, we modified the original implementation of MCMC [Domhan et al., 2015], to use the curve prior we proposed in Section 3.2. In the following, we refer to this MCMC variant as MCMC-PP and denote the one using the original prior [Domhan et al., 2015] as MCMC-OP (used in Section 4.2). Since the LC-PFN and MCMC-PP methods are both (approximate) Bayesian inference methods, using the same prior, they aim to approximate the same true target PPD, given a partial learning curve.

As a performance metric, we evaluate the log-likelihood (LL) of the unseen data (right-censored curve) under the inferred PPD. We use this metric, also known as logarithmic score , to assess a model's ability to infer the remaining part of the curve based on the initial observed values. A benefit of this metric is that it measures the quality of the PPD as a whole (rather than merely focusing on the error associated to a specific PPD statistic) and, assuming data is generated by the prior, the exact PPD maximizes this metric.

Importantly, we vary the cutoff, i.e., the percentage of the observed curve used as input, to better assess the model's performance across different amounts of available information. Furthermore, to allow a more comprehensive comparison, we vary the hyperparameters of both LC-PFN and MCMC-PP . For LC-PFN , we vary the embedding size ( emsize ), the number of layers ( nlayers ), and the total number of learning curves used during training ( nb_data ). For MCMC-PP , we vary the number of chains generated by the emcee [Foreman-Mackey et al., 2013] ensemble sampler ( nwalkers ), the length of each chain

Hyperparameters

MCMC-PP

∈

[100

,

250

,

500

,

1000

,

2000

,

4000]

nwalkers

∈

[26

,

50

,

100]

burn-in

∈

[0

,

50

,

100

,

500]

thin

∈

[1

,

10

,

100]

LC-PFN

nb_data

∈

[

100

k,

1

M,

10

M]

emsize

∈

[128

,

256

,

512]

nlayers

∈

[3

,

6

,

12]

( burn-in + nsamples ), the part of the chain omitted to account for mixing ( burn-in ), and the sub-sample frequency ( thin ). The considered values for each hyperparameter are summarized in Table 2.

We conducted the comparison on 10 000 sampled curves. Figure 1 shows a few curve examples, as well as inferences using LC-PFN and MCMC-PP given the data of the first 10 - 20 epochs (cutoff). We observe that both predicted median and uncertainties are indeed similar. More inference examples, with different cutoffs can be found in Appendix C.4.

Results Figure 3 displays the average log-likelihood across MCMC-PP / LC-PFN inferences for varying hyperparameters and a 10% cutoff. The log-likelihood is shown w.r.t. runtime, which is measured as the average wall-clock time for a single inference on a single Intel(R) Xeon(R) Gold 6242 CPU. Note that this inference time includes both the fit and prediction times for MCMC variants. Table 3 provides results on higher cutoffs ( 20% , 40% , 80% ) and corresponding runtimes for three variants of each method ( M1-3 , P1-3 ), labeled in Figure 3. Generally, the LC-PFN variants (left side of figure 3) are significantly faster than MCMC-PP variants (right side). LC-PFN always ran in less than 0 . 1 seconds while the best MCMC-PP ( M3 ) took over 100 seconds. Figure 3 also offers insights into the importance of LC-PFN and MCMC-PP hyperparameters. For MCMC-PP , both the cost and quality of inference increase with longer chain lengths and higher cutoffs. For LC-PFN , the inference cost increases with the model complexity which is closely related to the number of trainable parameters. Inference quality positively correlates with model size ('larger is better'), and the number of data LC-PFN was trained on. Among the hyperparameter grid we examined (Table 2), except for the smallest model ( P1 ), all LC-PFN variants that were trained on 10M samples produce higher log-likelihood than the best MCMC-PP variant ( M3 ). In particular, an LC-PFN ( P2 ) with 3 layers, embedding size 256, trained on 10M samples achieved better performance (log-likelihood of PPD) than the best MCMC-PP , but more than 15 000 times faster. We also find that while the runtime of the best MCMC-PP can be reduced (with minor loss of quality) by using thinning ( M2 ), the better LC-PFN is still approximately 7 000 times faster. Finally, it is important to note that training the largest LC-PFN ( P3 , 10M samples with 26M parameters) on the prior took approximately eight hours (single CPU, single RTX2080 GPU), but this cost is incurred only once for all of our experiments.

1.8

LC-PFN - emsize

128

256

512

1.7

LC-PFN - nb_data

P2

M2

M3

口

100k

D

1M

■

10M

1.6

P1

LC-PFN - nlayers

3

6

12

Log-likelihood

1.5

1.4

1.3

1.2

MCMC - nsamples

100

500

2000

250

1000

4000

1.1

10-2

10-1

100

101

102

Time (seconds)

4.2 Extrapolating real-world learning curves

While evaluation on data from the prior gives us a controlled setting to analyse quality and cost of the PPD approximation, performance on real-world learning curves is essential for practical usefulness. This second experiment aims to extend the previous comparison of MCMC and LC-PFN to real-world learning curve benchmarks.

We consider the best-performing variants of LC-PFN and MCMC-PP according to the average loglikelihood they obtained in the first experiment. For LC-PFN , the optimal variant ( P3 ) features an embedding size of 512 and 12 layers, resulting in a total of 26M trainable parameters, and is trained on 10 million prior curves. For MCMC-PP , the optimal configuration ( M3 ) involves a chain length of 4500, 100 walkers, 500 burn-in samples, without thinning. As an additional baseline, we include MCMC-OP , the original MCMC variant proposed by Domhan et al. [2015], which uses the original hyperparameters and curve prior (11 basis curves and uninformative prior over the curve parameters).

Benchmarks To evaluate the generalization capabilities of our model, we consider a diverse set of real-world curves. Our dataset comprises 20 000 learning curves, sourced from four distinct benchmarks: LCBench [Zimmer et al., 2021], NAS-Bench-201 [Dong and Yang, 2020], Taskset [Metz et al., 2020] and PD1 [Wang et al., 2022], each contributing 5 000 curves, randomly selected from specific subtasks. These benchmarks and subtasks were chosen to span a broad spectrum of supervised deep learning problems, training MLP ( LCBench ), CNN ( NAS-Bench-201 ), RNN ( Taskset ), and Transformer ( PD1 ) architectures on input modalities ranging from tabular data ( LCBench ), text ( Taskset and PD1 ), protein sequence ( PD1 ), to vision problems ( NAS-Bench-201 ). From LCBench and NAS-Bench-201 we use validation accuracy curves whereas from Taskset and PD1 the log loss validation curves. In terms of curve length, LCBench and Taskset cover 50 epochs, NAS-Bench-201 contains up to 200 epochs, and PD1 curves have varying lengths (22 - 1414). Further details, including sample curves, on these benchmarks are provided in Appendix B.

(a) Average rank of log-likelihood values (lower is better) vs cutoffs

LC-PFN

MCMC(original)

MCMC(our prior)

LCBench

NAS-Bench-201

Taskset

PD1

3.0

3.0

3.0

3.0

2.5

2.5

2.5

2.0

2.0

2.0

1.5

1.5

1.5

1.0

1.0

1.0

1.0

102030405060708090

102030405060708090

102030405060708090

102030405060708090

Percentageof the curveobserved

Percentage of the curve observed

Percentageofthecurveobserved

Percentage of the curve observed

LCBench

NAS-Bench-201

Taskset

PD1

3.0

3.0

3.0

3.0

Avg rank MSE

2.5

2.5

2.5

2.5

2.0

2.0

2.0

2.0

1.5

1.5

1.5

1.5

1.0

1.0

1.0

1.0

102030405060708090

102030405060708090

102030405060708090

102030405060708090

Percentage of thecurveobserved

Percentage of thecurveobserved

Percentageof thecurveobserved

Percentage of thecurveobserved

(b) Average rank of mean squared error (MSE) values (lower is better) vs cutoffs

Metrics Our focus lies on the relative performance of MCMC and LC-PFN , as absolute performance is significantly influenced by the choice of prior. We consider the log-likelihood and the mean squared error (MSE) of the predictions as metrics. Here, the MSE is calculated w.r.t. the median of the PPD.

For each benchmark, we report the average rank of these metrics to aggregate results from different curves, as supposed to the average values. While the latter better captures performance differences, it is very sensitive to outliers and scale-dependent. When computed in normalized space, it would strongly depend on our choice of normalization parameters (see Appendix A).

Results For each benchmark, Figures 4a and 4b display the average rank obtained by each method in terms of log-likelihood and MSE, respectively, where ranks are averaged across all 5 000 curves. We do not include error bars as the standard errors are neglectable (less than 0 . 02 ). In summary, we observe similar trends for both metrics on all benchmarks. LC-PFN is never truly outperformed by MCMC-PP . On LCBench both methods rank similarly, with LC-PFN being slightly worse at high cutoffs. LC-PFN ranks better on PD1 , NAS-Bench-201 , and Taskset . MCMC-OP performs clearly worse on LCBench , Taskset , and PD1 . On NAS-Bench-201 , MCMC-OP performs best for lower cutoffs, outperforming MCMC-PP , suggesting that NAS-Bench-201 curves are better captured by the original prior. Figure 11 and Figure 12 in Appendix C.2 show the log-likelihoods and MSEs, respectively, for all three methods, for each curve and cutoff, per benchmark, providing a more detailed perspective.

4.3 Application: Extrapolation-based early stopping in model selection

Thus far, we have shown that LC-PFN produces extrapolations of similar or better quality to MCMC , at a small fraction of the cost. However, these extrapolations are not perfect. In many cases, the practical relevance of any errors can only be assessed in the context of a specific application.

In this final experiment, we thus consider a model selection setting where after every epoch of training we have the choice between (i) continuing the current training, or (ii) stopping early ( T < m ) and starting a new training run using a different training pipeline (e.g., model architecture, hyperparameters, etc.). Here, we assume that runs cannot be resumed once stopped and that the order in which training pipelines are to be considered is given. Our objective is to obtain high-quality models as quickly as possible, by stopping suboptimal training runs as early as possible. This setting is also known as vertical model selection [Mohr and van Rijn, 2022] and was also considered by Domhan et al. [2015].

no-stop

patience(0)

patience(1)

patience(5)

patience(10)

patience(20)

patience(50)

LC-PFN(0.95,coarse)

LC-PFN(0.95,fine)

LCBench

NAS-Bench-201

Taskset

6 ×10-1

4 × 10-1

Average Regret

Average Regret

Regret

3 × 10-1

Average

2 ×10-1

200

400

600

800

1000

0

0000 0000 0000 0002 0002 0000 0000 009

0

200

400

600

800

1000

PD1 - Language Modeling (lm1b)

PD1 - Translation (WMT)

100

Average Regret

Average Regret

100

Average Regret

10-1,

6 ×10-2

10-1

10-1

4 ×10-2

3×10-2

0

100

200300400500600

700

0

5000

10000150002000025000

0

100

200

0OE

400

We consider the extrapolation-based termination criterion proposed by Domhan et al. [2015], but use LC-PFN instead of MCMC to predict the likelihood Pr( y t ′ > y best | y 1 , . . . , y T ) that the current run will at epoch t ′ obtain a model better than the best obtained by any run thus far ( y best ), for T < t ′ ≤ m and decide to stop the current run if that probability does not exceed some fixed threshold δ (at any point). In our experiments, we use confidence level 1 -δ = 0 . 95 as Domhan et al. [2015]. Note that this criterion can be applied at every step or only at specific cutoffs. To simulate the effect of varying granularity, we consider a coarse-grained variant with 4 cutoffs T ∈ {⌈ 0 . 1 m ⌉ , ⌈ 0 . 2 m ⌉ , ⌈ 0 . 4 m ⌉ , ⌈ 0 . 8 m ⌉} , and a fine-grained variant with T ∈ { T | 1 < T < m } . We investigate alternative choices for cutoffs and confidence levels in Appendix C.3.2.

Following Domhan et al. [2015], we compare against a black box approach no-stop that does not implement early stopping. We further compare against a criterion Patience(k) that implements the popular heuristic to terminate a training run when model performance did not improve for k epochs. For evaluation, we consider the same benchmarks as in Section 4.2 (for details, see Appendix B). We chose the total budget for model selection to correspond to 20 full runs (20 m ). For each task, we consider the training runs in 40 different random orderings. This totals 2 120 model selection experiments per method, spread across the 53 different tasks.

Results Figure 5 shows the anytime performance of all methods in our comparison, on each of the benchmarks, in terms of regret. Here, regret is the absolute difference in performance between the best model obtained thus far and the best model attained by any run on the task. Results are averaged over the different run orderings. For LCBench , NAS-Bench-201 , and Taskset results are further averaged across all tasks (results for individual tasks can be found in Appendix C.3.1). The shaded area corresponds to ± 1 standard error. On all benchmarks, except for NAS-Bench-201 , we observe that LC-PFN based termination criteria clearly perform best. Also, the fine-grained variant performs better on average, suggesting that errors in inference are compensated for by the time saved by stopping runs earlier. In terms of expected speed-up, this LC-PFN variant obtains an expected regret lower than that obtained by no-stop , approximately 3.3 × faster on LCBench and Taskset . Looking at individual tasks, we note 2 - 6 × speed-ups on all 3 PD1 tasks, all 12 Taskset tasks, and 30 of the 35 LCBench tasks (we obtain 1 - 2 × speed-ups on the remaining 5). On NAS-Bench-201 , we find that all termination criteria considered, including standard Patience heuristics, fail on all 3 tasks; this is likely related to the particular shape of learning curves on this benchmark, having an inflection point (see Figure 8 and Figure 19), not resembling any in the prior.

Finally, in terms of overhead, the computational costs of the LC-PFN inferences per model selection experiment range from 3 seconds (coarse-grained) up to 1 minute (fine-grained), and are negligible compared to the cost of the 20 full training runs of deep neural networks.

5 Summary, limitations, and future research

We presented the first work using prior-data fitted networks (PFNs) for Bayesian learning curve extrapolation. We show that our LC-PFN obtains qualitatively similar extrapolations, for a wide variety of learning curves, more than 10 000 × faster than the MCMC method proposed by Domhan et al. [2015]. These inferences are now fast enough (under 100 milliseconds on CPU, and even less on GPU), to be used in the context of online learning, at virtually no overhead. This opens up a wide variety of possible applications, e.g., to speed up automated model selection in AutoML and HPO by discarding poor configurations early. It would be interesting to integrate LC-PFN as a new termination criterion in existing deep learning libraries.

Often, we also have more data available than a single partial learning curve, e.g., other curves on the same task, their hyperparameters, and/or curves of the same method on a different task, and meta-features. Previous work [Swersky et al., 2014, Klein et al., 2017, Wistuba et al., 2022, Ruhkopf et al., 2022] has already exploited this, and we could explore the potential of using PFNs for few-shot in-context meta-learning, by feeding the model multiple curves and hyperparameters as input.

While for a fair comparison to Domhan et al. [2015], reusing the original code, our prior (Section 3.2) closely resembled the one of Domhan et al. [2015], future work could improve upon this prior and overcome its limitations (e.g., on the NAS-Bench-201 tasks) by modelling divergence, slow start, double-descent, correlated heteroscedastic noise, etc.

Finally, PFNs, unlike other Bayesian methods, must learn the prior from data, which implies that the prior must be generative. Also, it suggests that high entropy priors may present challenges. Future research should investigate these limitations and how to overcome them.

6 Acknowledgments and disclosure of funding

We acknowledge funding by the European Union (via ERC Consolidator Grant Deep Learning 2.0, grant no. 101045765), TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant numbers INST 39/963-1 FUGG and 417962828. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

References

Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. arXiv:1611.02167 [cs.LG] , 2017.

Chandrashekaran and I. R. Lane. Speeding up hyper-parameter optimization by extrapolation of learning curves using previous builds. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 477-492. Springer, 2017.

Choi, H. Cho, and W. Rhee. On the difficulty of DNN hyperparameter optimization using learning curve prediction. In 2018 IEEE Region 10 Conference (TENCON'2018) , pages 651-656. IEEE, 2018.

Cortes, L. D. Jackel, S. Solla, V. Vapnik, and J. Denker. Learning curves: Asymptotic values and rate of convergence. Advances in neural information processing systems , 6, 1993.

Damianou and N. D. Lawrence. Deep gaussian processes. In Artificial intelligence and statistics , pages 207-215. PMLR, 2013.

Domhan, J. Springenberg, and F. Hutter. Speeding up automatic Hyperparameter Optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI'15) , pages 3460-3468, 2015.

Dong and Y. Yang. NAS-Bench-201: Extending the scope of reproducible Neural Architecture Search. In Proceedings of the International Conference on Learning Representations (ICLR'20) , 2020.

Dooley, G. S. Khurana, C. Mohapatra, S. Naidu, and C. White. ForecastPFN: Synthetically-trained zeroshot forecasting. In Proceedings of the 37th International Conference on Advances in Neural Information Processing Systems (NeurIPS'23) , 2023.

Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In Proceedings of the 35th International Conference on Machine Learning (ICML'18) , volume 80, pages 1437-1446. PMLR, 2018.

Foreman-Mackey, D. Hogg, D. Lang, and J. Goodman. emcee : The MCMC Hammer. Publications of the Astronomical Society of the Pacific , 125(925):306, 2013.

J. Frey and D. H. Fisher. Modeling decision tree performance with the power law. In Seventh International Workshop on Artificial Intelligence and Statistics . PMLR, 1999.

Gargiani, A. Klein, S. Falkner, and F. Hutter. Probabilistic rollouts for learning curve extrapolation across hyperparameter settings. In ICML workshop on Automated Machine Learning (AutoML workshop 2019) , 2019.

Gijsbers, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. An open source automl benchmark. In ICML workshop on Automated Machine Learning (AutoML workshop 2019) , 2019.

Hollmann, S. Müller, K. Eggensperger, and F. Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations , 2023.

Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv:2001.08361 [cs.LG] , 2020.

Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR'15) , 2015.

Klein, S. Falkner, J. Springenberg, and F. Hutter. Learning curve prediction with Bayesian neural networks. In Proceedings of the International Conference on Learning Representations (ICLR'17) , 2017.

Kolachina, N. Cancedda, M. Dymetman, and S. Venkatapathy. Prediction of learning curves in machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 22-30, 2012.

Leite and P. Brazdil. Improving progressive sampling via meta-learning on learning curves. In European conference on machine learning , pages 250-261. Springer, 2004.

Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: Bandit-based configuration evaluation for Hyperparameter Optimization. In Proceedings of the International Conference on Learning Representations (ICLR'17) , 2017.

Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR'17) , 2017.

Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages 142-150, 2011.

Metz, N. Maheswaranathan, R. Sun, C. D. Freeman, B. Poole, and J. Sohl-Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies. arXiv:2002.11887 [cs.LG] , 2020.

Mohr and J. van Rijn. Learning curves for decision making in supervised machine learning-a survey. arXiv:2201.12150v1 [cs.LG] , 2022.

Müller, N. Hollmann, S. Arango, J. Grabocka, and F. Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference on Learning Representations (ICLR'22) , 2022.

Müller, M. Feurer, N. Hollmann, and F. Hutter. PFNs4BO: In-context learning for bayesian optimization. In ICML 23 , 2023.

Ruhkopf, A. Mohan, D. Deng, A. Tornede, F. Hutter, and M. Lindauer. Masif: Meta-learned algorithm selection using implicit fidelity information. Transactions on Machine Learning Research , 2022.

Salinas, M. Seeger, A. Klein, V. Perrone, M. Wistuba, and C. Archambeau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In International Conference on Automated Machine Learning, AutoML 2022 , 2022.

Swersky, J. Snoek, and R. Adams. Freeze-thaw Bayesian optimization. arXiv:1406.3896 [stats.ML] , 2014.

Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS'17) , 2017.

Wang, G. E. Dahl, K. Swersky, C. Lee, Z. Mariet, Z. Nado, J. Gilmer, J. Snoek, and Z. Ghahramani. Pre-training helps bayesian optimization too, 2022.

Wistuba, A. Kadra, and J. Grabocka. Supervising the multi-fidelity race of hyperparameter configurations. In Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22) , 2022.

Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approximation , 26(2):289-315, 2007.

Zimmer, M. Lindauer, and F. Hutter. Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43:3079-3090, 2021.

A Learning curve normalization

Not all learning curves of interest may resemble those of the prior we described in Section 3.2. For example, when minimizing the negative log-likelihood loss (log loss), learning curves will mostly decrease (vs. increase), be convex (vs. concave), and could take values that exceed 1.0 (there is no hard upper bound for log loss).

One approach would be to define a prior and train a specialized PFN for every performance measure. In this work, we use a more general approach: We apply a normalization procedure that allows us to accurately extrapolate a wide variety of learning curves using a single PFN, without retraining or fine-tuning. We consistently apply this procedure to all inferences involving real learning curves (i.e., the experiments described in Section 4.2 and Section 4.3), even if observations are naturally constrained to [0 , 1] , e.g., accuracy curves.

Step 1: Normalize the partial learning curve: Let y o i be the i th performance observation. We start by normalizing it using a generalized logistic sigmoid transform y i = g λ ( y o i ) , that is re-parametrized by the following five parameters ( λ ):

min? A Boolean specifying that we expect learning to minimize the performance measure. E.g., this would be true for error rate and log loss, and false for accuracy.

lhard , uhard These are possibly infinite hard lower / upper bounds for model performance. This would, e.g., be 0 / 1 for accuracy and error rate; and 0 / + ∞ for log loss.

l soft , usoft These are finite soft lower/upper bounds for model performance. They specify the range in which we expect performance values (that we care to distinguish between) to lie in practice. For accuracy and error rate, one could set these equal to the hard bounds, whereas for log loss one could, e.g., use an estimate of the loss for the untrained network (i.e., at epoch 0) as usoft and lsoft = lhard, or if available, choose lsoft to be an optimistic estimate of performance at convergence (e.g., state-of-the-art). Narrower ranges will result in more accurate inferences.

Specifically, we perform a linear transformation, followed by a logistic sigmoid, another linear transform, and a minimization conditional reflection around 1 2 , i.e.,

where

1 .

Note that for λ = ( False , glyph[axisshort] ∞ , -1 , 1 , ∞ ) this reduces to the canonical logistic sigmoid y i = 1+ e -y o i This transform will be approximately linear (shape preserving) for y o i ∈ [ l soft , usoft ] , and the tails of the sigmoid will gradually squish values outside this range to [0 , 1] , supporting unbounded performance measures. Figure 6 visualizes this projection in general, and provides examples for accuracy with λ = ( False , 0 , 0 , 1 , 1) and log loss with λ = ( True , 0 , 0 , log(10) , ∞ ) .

Step 2: Infer using normalized data: Next, we perform Bayesian learning curve extrapolation on the normalized partial curve y , with our usual prior p ( y ) , resulting in an approximation of the PPD in the transformed space.

Step 3: Inverse transform the PPD: Finally, we can obtain the property of interest of the PPD in the original space by applying the inverse transform, given by

We use order-based statistics (e.g., median or other percentiles) in our experiments, to which we can simply apply g glyph[axisshort] λ directly since it is a monotonic transform. For other statistics (e.g., mean, variance) we may need to resort to Monte Carlo estimation, applying g glyph[axisshort] λ to samples of the PPD.

1.2

1.2

1.0

1.0

inference space

0.8

8'0

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.0

soft

original space

Unard

0.2

0.2

0.0

0.2

0.4

accuracy

0.6

0.8

1.0

1.2

0.2

4

log loss

6

8

10

1.0

b

0.8

5

0.6

loss

m

3

0.4

2

WWM

0.2

/

0.0

0

0

20

40

60

80

100

0

20

40

60

80

100

t

epochs

By applying a given normalization, we effectively transform our prior, i.e., p ( y o ) = g glyph[axisshort] λ ( p ( y )) . To get some intuition of what that prior looks like, we can sample from p ( y o ) by applying g glyph[axisshort] λ to samples of our prior p ( y ) , which is useful for fine-tuning the parameters of this transformation. Figure 7 (right) shows samples of the log loss prior for λ = ( True , 0 , 0 , log(10) , ∞ ) .

B Detailed benchmark description

To evaluate the generalization capabilities of LC-PFN , we consider a diverse set of real-world curves. The reduce computational cost and improve reproducibility, we source these from four existing benchmarks that provide a collection of learning curves for a wide variety of supervised learning tasks. In what follows, we describe each of these benchmarks in more detail, the subset of tasks and curves we considered, and any preprocessing we did. Table 4 provides an overview of the characteristics of each benchmark and Figure 8 (a) visualizes the curves selected from each benchmark.

LCBench provides learning curve data for 2 000 configurations of AutoPytorch [Zimmer et al., 2021] trained for 50 epochs on 35 tabular classification datasets from the AutoML benchmark [Gijsbers et al., 2019]. All of these configurations use momentum SGD to train an MLP, but with varying number of layers and units per layer, and varying optimization hyperparameters (batch size, learning rate, momentum, L2 regularization, dropout rate). In Section 4.2, we consider a subset of 5 000 curves selected uniformly at random from all 70 000 validation accuracy curves in the benchmark. In Section 4.3, we consider all 2 000 validation accuracy curves, for every task, in 40 different orderings.

NAS-Bench-201 is a benchmark for Neural Architecture Search methods [Dong and Yang, 2020]. It provides learning curve data for training 15 625 different architectures for 200 epochs on three different image classification datasets (CIFAR-10, CIFAR-100, and ImageNet16-120) for three different random seeds. Other aspects of the training pipeline (e.g., optimizer, hyperparameters)

are fixed. We use the validation error rate curves provided through the Syne Tune [Salinas et al., 2022] interface to this benchmark. As discussed in Section 3.4, the LC-PFN we consider is trained to extrapolate curves up to length 100 ( m ). 3 To handle the 200 epoch curves from NAS-Bench-201 , we chose to subsample them, feeding only every second observation into the LC-PFN . In Section 4.2, we consider a subset of 5 000 curves selected uniformly at random from all 140 625 error rate curves in the benchmark. In Section 4.3, we consider all 46 875 error rate curves, for each of the three datasets, in 40 different orderings.

Taskset [Metz et al., 2020] provides a total of roughly 29 million learning curves for over 1 162 different deep learning tasks. Here, a task is defined as optimizing a given neural network architecture for a given supervised learning problem, and the curves correspond to using 5 different optimizers, using 1 000 different hyperparameter settings, and 5 different random seeds. Following Wistuba et al. [2022], we only use the validation log loss curves of a small subset of 12 tasks, that consider training different RNN architectures, with varying architectural parameters, for 50 epochs on the IMDB sentiment analysis dataset [Maas et al., 2011], a binary text classification task. As our objective was to evaluate the robustness of our approach, we avoided excluding ill-behaved (e.g., diverging) curves. However, upon inspecting the data, we found that on some of these tasks up to 90% of the curves fail to significantly improve upon the initial model performance, some of which diverge almost instantly. In combination with the normalization procedure described in Appendix A, the majority of curves are effectively mapped onto the same narrow range, producing a constant trend (at 0 in case of divergence). While this is reasonable in practice, and both LC-PFN and MCMC variants predict this constant trend with very high confidence, it does create a bias in our evaluation towards methods excelling at extrapolating poor curves. To eliminate and investigate this bias, we selected the 5 000 curves used in Section 4.2, such that 2 500 are taken from the top 10% best 4 curves per task, and 2 500 from the 90% others. In Figure 4, we compared methods on the 'good' curves only, an evaluation on the 2 500 'bad' curves is presented in Figure 8 (b). In Section 4.3, we do not make this distinction and consider all 25 000 curves per task, in 40 different random orderings.

PD1 is a recent benchmark that Wang et al. [2022] describe as collecting 'a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset'. We access this benchmark through the synetune [Salinas et al., 2022] library interface, which provides learning curve data for 23 tasks, where the number of curves provided, as well as the curve length, varies per task. Here, we limit our selection to log loss curves for the three tasks that consider training Transformer architectures for 22 epochs on language modeling (lm1b), 1414 epochs on translation (WMT), and 38 epochs on protein sequences (UniRef50). To handle the very long learning curves on the translation task, we subsample these aggressively, feeding only every 14th observation into the LC-PFN . In Section 4.2, we select 5 000 curves from these three tasks uniformly at random. In Section 4.3, we consider all curves per task, in 40 different random orderings. Finally, we use the optimal loss achieved by any of the runs as lsoft > 0 in our normalization. While unknown in practice, we do not expect qualitative differences in evaluation if a rough estimate were to be used instead.

LCBench

NAS-Bench-201

Taskset

PD1

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

M

0.0

10

20

30

40

50

0.0

20

40

60

80

100

0.0

10

20

30

40

50

0.0

20

40

60

80

100

t

t

LC-PFN

MCMC(original)

MCMC(our prior)

Allcurves

Allcurves

Badcurves

Badcurves

Average rank Log-likelihood values

3.0

3.0

s

3.0

3.0

Average rank MSE values

Average rank MSE values

2.5

25

20

15

15

15

15

10

10

10

10

51015202530354045

51015202530354045

51015202530354045

51015202530354045

Percentageofthecurveobserved

Percentageof thecurve observed

Percentageofthecurveobserved

Percentageofthecurveobserved

Figure 8: (a) Illustration showing the 5 000 curves used for each of the four benchmarks in our experiments in Section 4.2. All curves are shown after normalization and subsampling, and a random subset of individual curves is highlighted. For Taskset , 'good' / 'bad' curves (top 10% / bottom 90% in their task) are shown in green / red, respectively. (b) Average rank of Log-likelihood and MSE values on Taskset curves for the three methods ( LC-PFN , MCMC-PP , and MCMC-OP ). The two leftmost plots show results on the 5 000 sampled Taskset curves. The two rightmost plots provide the same comparison, but specifically for the 2500 'bad' curves. Here, we observe that MCMC-PP performs better on the 'bad' curves, which we believe can be attributed to the discretization of the PPD in 1000 bins, limiting the maximal confidence / accuracy of LC-PFN , since the majority of these curves are quasi-constant (fall in the same bin) after normalization.

C Additional analyses

C.1 Extrapolating samples of the prior

C.1.1 Detailed results of section 4.1

Figure 9 provides a visual representation of the log-likelihood values on prior curves for higher cutoffs, similar to Figure 3. As expected, the difference in log-likelihood values between LC-PFN and MCMC-PP decreases as more points of the curve are observed. The detailed results can be found in Table 5. Both Figure 9 and Table 5 clearly demonstrate that certain variants of LC-PFN , particularly those trained with 10M examples, consistently outperform the best variant of MCMC-PP across all considered cutoffs.

t

LC-PFN - nb_data

LC-PFN - emsize

LC-PFN - nlayers

MCMC-nsamples

口

100k

D

1M

■

10M

128

256

512

6

12

100

250

500

1000

2000

4000

20%ofthecurveobserved

40%of the curve observed

80%ofthecurveobserved

2.8

2.6

2.4

2.2

2.0

1.8

1.6

1.4

102

10-1

100

101

102

102

10-1

100

101

102

102

10-1

100

101

102

Time (seconds)

Time (seconds)

Time (seconds)

1.8

emsize=512]

1.7

MCMC initialization

Log-likelihood

1.6

LSE

MAP

Default

1.5

NumberofSamples

1.4

1000

4000

10000

1.3

40000

★

100000

10-1

100

101

102

103

Time (seconds)

C.1.2 Effect of MCMC chain length and initialization

In our experiments in Section 4.1 we observed that the quality of MCMC inference increases with chain length. In particular, our best MCMC-PP variant ( M3 in Figure 3), uses the same hyperparameter settings as Domhan et al. [2015] ( M1 ), but with a double as long chain (discounting the burn-in). In this section, we investigate whether the chains considered are too short and whether increasing chain lengths further will eventually cause MCMC-PP to outperform LC-PFN . To study this, we will run MCMC-PP to collect up to 100 000 samples per walker (50 × more than Domhan et al. [2015]). To reduce the computational cost of this experiment, we only extrapolate 1 000 (instead of 10 000) prior curves, on cutoff 10%, and use MCMC-PP with a 100-thinning strategy ( M2 ). This variant is more compute-efficient and the minor negative effect of thinning is expected to further decrease with increasing chain length. Concurrently, we study the effect of initialization. There are different ways of initializing MCMC chains and the chosen strategy can strongly affect performance if chains are too short. Domhan et al. [2015] initializes the chain by setting the parameters for each of the K basis curves to their Least-Squared Estimates ( LSE ). Weights are initialized to be 1 K . If this initial point violates the constraints imposed by the prior, a default starting point is used instead. As an ablation, we compare it to an initialization always starting at the default ( default ). As another ablation, we compare to a strategy using the maximum a posteriori estimate ( MAP ) as a starting point instead, which unlike LSE takes the likelihood under the prior into account.

Figure 10 shows the quality and cost of inferences for each of the initialization strategies when collecting up to 100 000 samples (per walker). While these results clearly show that increasing chain length continues to improve performance. The trends we observe, suggest that MCMC-PP could eventually attain or even overtake the best LC-PFN . That being said, the best MCMC we considered has 20 000 × longer runtimes than that LC-PFN requires, yet it does not quite reach the same performance, so outperforming it would require impractically long chains. When optimistically extending the trends, we estimate to need at least 10 × longer chains, and inference times of multiple hours. In terms of initialization, we observe that MCMC performance for shorter chains indeed more strongly depends on the initialization strategy, where the LSE strategy by Domhan et al. [2015] does best for short chains, followed by MAP and default . When increasing chain length, we find that differences get smaller and order reverses such that the fixed default initialization is best, suggesting that greedy initialization ( LSE , MAP ) may hurt performance on some curves.

C.2 Extrapolating real-world learning curves

In addition to the average rank plots presented in Section 4.2, we also perform pairwise comparisons of the log-likelihood and mean squared error values among the different methods ( LC-PFN and MCMC variants). This approach allows for a more detailed analysis, as it compares the absolute values of the metrics. Unlike the rank plots, these pair and curvewise plots provide insights into outliers which thus further enhance our understanding of the results.

(a) PFN

vs

MCMCwith our prior (

MCMC-PP

)

Percentageofthecurveobserved

80%

40%

20%

10%

LCBench

NAS-Bench-201

Taskset

PD1

20

10

LC-PFN

0-

-10 -

-20

-20

-10

0

10

20-20

-10

0

10

20-20

-10

0

10

20-20

-10

0

10

20

MCMC(our prior)

MCMC(ourprior)

MCMC(ourprior)

MCMC(our prior)

LCBench

NAS-Bench-201

Taskset

PD1

20

10

LC-PFN

0-

-10

-20-

-20

-10

0

10

20-20

-10

0

10

20-20

-10

0

10

20-20

-10

10

20

MCMC(original)

MCMC(original)

MCMC(original)

MCMC(original)

(b) PFN

vs

MCMCas in Domhan et al. [2015] (

MCMC-OP

)

Figure 11 presents a pairwise comparison of the log-likelihood values between our PFN method and the two variants of MCMC, per curve, considering different cutoff values on the four benchmark datasets. Figure 12 presents the same comparison based on mean squared error (MSE).

We observe that LC-PFN compares favorably to MCMC-PP on PD1 and NAS-Bench-201 tasks. Both methods demonstrate comparable performance on the Taskset dataset. However, it is worth noting that for some curves of the LCBench dataset, MCMC-PP obtains very high log-likelihood and low MSE scores, while LC-PFN falls short in this aspect. We believe this discrepancy can be attributed to the fact that the LCBench dataset contains a significant number of constant curves (see Figure 8a) that fall within the same bin of the discretized PPD, effectively limiting the the maximal confidence / accuracy of LC-PFN . Looking at MSE specifically, we find that for some LCBench curves, at high cutoffs, LC-PFN obtains very high errors, while MCMC-PP does not. Upon closer inspection, we found that these curves quickly converge to a value close to the optimal accuracy, and while LC-PFN captures this trend for lower cutoffs, it suddenly fails for larger cutoffs (see 5th example in Figure 18). This can likely be explained by the fact that these curves are not adequately captured by the prior proposed in Section 3.2 and this seems to be one of the few cases where LC-PFN generalizes poorly out of distribution.

When comparing with MCMC-OP , we observe that MCMC-OP has relatively few outliers in terms of log-likelihood, both in the positive and negative sense. This can likely be explained by the prior of MCMC-OP , which is more flexible (i.e. has higher entropy) than ours and making MCMC-OP thus less confident about its predictions. However, MCMC-OP performs clearly worse in terms of MSE, suggesting that the median of the high entropy PPDs produced by this method do not provide a good point estimate.

(a) PFN

vs

MCMCwith our prior (

MCMC-PP

)

Percentageofthecurveobserved

80%

40%

20%

10%

LCBench

NAS-Bench-201

Taskset

PD1

10-1

10-4

LC-PFN

10-7.

10-10

10-13

10-13 10-1010-7 10-4 10-1

10-1310-1010-710-410-1

10-13 10-1010-7 10-410-1

10-1310-1010-710-41

10-1

MCMC(our prior)

MCMC(our prior)

MCMC(our prior)

MCMC(our prior)

LCBench

NAS-Bench-201

Taskset

PD1

10-1

10-4

LC-PFN

10-7

10-10

10-13

10-1310-1010-710-4

10-1

10-1310-1010-71

10-4

10-1

10-13 10-1010-71

10-4

10-1

10-1310-1010-710-4

10-1

MCMC(original)

MCMC(original)

MCMC(original)

MCMC(original)

(b) PFN

vs

MCMCas in Domhan et al. [2015] (

MCMC-OP

)

C.3 Application: Extrapolation-based early stopping in model selection

In what follows, we further analyze and discuss our experiments described in Section 4.3.

C.3.1 Results on individual tasks

While Figure 5 shows the results of our early stopping experiments for each PD1 task, results for the three remaining benchmarks are summarized by averaging them across all tasks. Since this may hide task-dependent variation and outliers, we have a closer look at the performance on individual tasks. Figure 13 shows the results for each of 35 LCBench tasks. Here, we observe that relative performances vary per task. The aggressive Patience(k) (i.e., low k ) baselines perform best on some tasks, but fail on others. The LC-PFN based termination criteria consistently perform well on each task, obtaining a 2-6 × speed-up (w.r.t. the final performance of no-stop ) on 30 of the 35 tasks, and no significant slow-down on the remaining five. Figure 14 shows the results for each of the 3 NAS-Bench-201 tasks. Here, we observe failure on all 3 tasks, where the degree of failure seems correlated with the complexity of the image classification task. Figure 15 shows the results for each of the 12 Taskset tasks. Here, the LC-PFN based termination criteria consistently perform well, obtaining 2-6 × speed-ups on all 12 tasks. We conclude that Figure 5 accurately reflects relative performances on all four benchmarks.

no-stop

patience(1)

patience(5)

ience(10)

patience(20)

LC-PFN(0.95,c0

LC-PFN(0.95,fine)

LCBench - APSFailure

LCBench - bank-marketing

LCBench - fabert

LCBench - nomao

10

naubany

10

10-2

200

400

600

800

1000

200

400

600

800

1000

0

200

400

600

800

1000

0

200

600

800

1000

LCBench - Amazon_employee_access

LCBench - blood-transfusion-service-center

LCBench - helena

LCBench - numerai28.6

auBan

101

10

103

102

200

400

600

800

1000

200

400

600

800

1000

200

400

600

800

1000

0

200

400

800

1000

LCBench - Australian

LCBench - car

LCBench -higgs

LCBench-phoneme

10“

10-2

200

800

1000

200

400

600

800

1000

0

200

400600

800

1000

200

400

800

1000

LCBench - Fashion-MNIST

LCBench - christine

LCBench - jannis

LCBench - segment

abag sbeany

10-

10

10-2

200

400

009

800

1000

200

400

800

1000

0

200

400

600

800

1000

200

400

800

1000

LCBench - KDDCup09_appetency

LCBench - cnae-9

LCBench - jasmine

LCBench - shuttle

106

qaubay

10-

10-

aAt

10

10-2

10-2

200

400

800

1000

200

400

600

800

1000

0

200

400

800

1000

200

400

600

800

1000

LCBench - MiniBooNE

LCBench - connect-4

LCBench-jungle_chess_pcs_raw_endgame_complete

LCBench - sylvine

4 ×10-1

3×10-1

4 × 10-1

3×10-1

2× 10-1

10

10-1

102

6 × 10-2

6 × 10-2

200

800

1000

200

800

1000

200

400

800

1000

0

200

800

1000

LCBench - adult

LCBench -covertype

LCBench -kc1

LCBench -vehicle

10-1

10-

-ot

10-2

200

400600

800

1000

200

400600

800

1000

200

400600

800

1000

0

200

400

800

1000

LCBench -airlines

LCBench - credit-g

LCBench - kr-vs-kp

LCBench-vokert

10-1

aubay

4×10

3 × 10-2

2 × 10-2

102

200

800

1000

200

800

1000

200

400

600

800

1000

200

800

1000

LCBench - albert

LCBench - dionis

LCBench - mfeat-factors

10°

jajfagy afieuary

aub

10-2

0

200

400

600

800

1000

0

200

600

800

1000

0

200

400

800

1000

no-stop

patience(0)

patience(1)

patience(5)

patience(10)

patience(20)

patience(50)

LC-PFN(0.95,coarse)

LC-PFN(0.95,fine)

NAS-Bench-201-cifar10

NAS-Bench-201-cifar100

NAS-Bench-201-ImageNet16-120

Average Regret

Average Regret

Average Regret

101

10-1

101

10-2

0

5001000150020002500300035004000

0

0

5001000150020002500300035004000

no-stop

patience(0)

patience(1)

patience(5)

patience(10)

patience(20)

patience(50)

LC-PFN(0.95,coarse)

LC-PFN(0.95,fine)

Taskset-patch128_LSTM128_avg_bs64

Taskset-patch32_GRU64_avg_bs128

Taskset-patch32_LSTM128_bs128

6 × 10-1

6 ×10-1

6 × 10-1

4 ×10-1

4 ×10-1

3 × 10-1

Average Regret

4 × 10-1

Average Regr

3 ×10-1

Average

2 ×10-1

3 ×10-1

2×10-1

2 ×10-1

10-1

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Taskset-patch128_LSTM128_bs64

Taskset-patch32_IRNN64_relu_avg_bs128

Taskset-patch32_VRNN128_tanh_bs128

6 × 101

6 × 10-1

6 ×10-1

Average Regret

4 ×10-1

Average Regret

4 × 10-1

Average Regret

4 × 10-1

3×10-1

3 ×10-1

3 ×10-1

2 ×10-1

2 ×10-1

2 × 10-1

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Taskset-patch128_LSTM128_embed128_bs64

Taskset-patch32_IRNN64_relu_last_bs128

Taskset-patch32_VRNN64_relu_avg_bs128

6 × 101

6 ×10-1

6 × 10-1

4 ×10-1

Average Regret

4 ×10-1

Average Regret

4 × 10-1

3 ×10-1

3×10-1.

2 ×10-1

2 ×10-1

10-1

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Taskset-patch32_GRU128_bs128

Taskset-patch32_LSTM128_E128_bs128

Taskset - patch32_VRNN64_tanh_avg_bs128

6×10-1

6 × 10-1

6 ×10-1

4 ×10-1

4 ×10-1

Average Regret

4 × 10-1

Average Regret

3 ×10-1

3×10-1

3×10-1

2×10-1

2 ×10-1

2 ×10-1

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

C.3.2 Parameter sensitivity analysis

In our comparison in Figure 5, we considered two variants of the LC-PFN based termination criterion, varying the frequency at which it is applied, and found the fine-grained strategy, applying it every epoch, to perform best. However, many variants of this scheme exist, and in what follows we investigate the impact of some of our other choices on performance.

Confidence level: In our experiments, we terminate a training run if we are confident it will not improve upon the best model seen so far. Following prior art [Domhan et al., 2015], we adopted a confidence threshold of 0.95. Figure 16 (a) shows the anytime performance of variants using lower (0.90) and higher (0.99, 0.999) confidence levels. We observe that all of these perform relatively well, suggesting some degree of robustness. The speed-ups obtained using higher confidence levels are mostly lower (up to 2 × ), but more consistent. In particular, we only observe minimal slow-down on NAS-Bench-201 at confidence level 0.999, and higher confidences are also beneficial for the PD1 protein sequence (UniRef50) task.

Minimal cutoff: While we apply our predictive termination criterion every epoch, we do not apply it 'as of the first epoch', but only after two observations are available ( T ≥ 2 ). Since LC-PFN 's predictions are conditioned on the partial curve only, it will predict the prior for T = 0 . Beyond not allowing us to discriminate between runs, it introduces a potential failure mode for easy tasks (many training runs obtain models close to the theoretical optimum) and LC-PFN (unaware of task hardness) will terminate runs before they even began because it is very unlikely to obtain a better model under the prior . Figure 16 (b) shows the anytime performance of variants that apply the criterion as soon as one, three, four, or five observations are made. We find that a choice of one (instead of two) only negatively impacts performance on LCBench (containing some very easy tasks). A minimal cutoff of two works well on all benchmarks, with minor slow-downs for the higher minimal cutoffs.

C.4 Qualitative plots of learning curve extrapolation

To complement the quantitative evaluation in the main paper, we provide examples of extrapolations, at the four different cutoffs, for seven curves from the prior (Figure 17), LCBench (Figure 18), NAS-Bench-201 (Figure 19), Taskset (Figure 20), and PD1 (Figure 21). The plots show the median and two-sided 90% confidence interval of the PPDs inferred using the LC-PFN and MCMC-PP variant considered in Section 4.2. We observe that while the quality of the extrapolations produced by LC-PFN varies strongly, they are mostly logical, given the data observed and the prior used, and we find that the inferences of MCMC-PP are rarely preferable.

D Breakdown computational cost

Overall, reproducing all our experiments in the main paper requires approximately 163 CPU days and 60 GPU hours on our systems (GPU: NVIDIA (R) GeForce (R) RTX 2080, CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz). These costs break down as follows:

Section 4.1 entails 80 CPU days to run all MCMC-PP variants on the 10 000 sampled curves. The overall prediction time of LC-PFN is negligible (<1 hr for each variant). However, for LC-PFN , there is an initial training cost, which amounts to approximately 60 GPU hours to train all 27 LC-PFN variants.

Section 4.2 requires 80 CPU days to run the MCMC algorithm on the considered real curve benchmarks.

Section 4.3 involves a maximum of 3 CPU days to replicate the early stopping results on all the benchmarks for both LC-PFN and the baselines.

(a) Varying confidence levels (

1

-

δ

)

(b) Varying minimal cutoff (

T

)

no-stop

LC-PFN(0.90,fine>=2)

LC-PFN(0.95,fine>=2)

LC-PFN(0.99,fine>=2)

LC-PFN(0.999,fine>=2)

LCBench

NAS-Bench-201

Taskset

6 × 101

4 ×10-1

Average Regret

Average Regret

Regret

3 × 10-1

10-1

Average P

2 × 10-1

0

200

400

600

800

1000

0

5001000 1500 2000 2500 3000 3500 4000

0

200

400

600

800

1000

PD1 - Language Modeling (lm1b)

PD1 - Translation (WMT)

100

10-1

Average Regret

Average Regret

100

Average Regret

6 × 10-2

10-1

10-1

4 ×10-2

3 ×10-2

2 × 10-2

0

100

200

300

400

500

600

700

0

5000

10000

15000

20000

25000

0

100

200

300

400

no-stop

LC-PFN(0.95,fine>=1)

LC-PFN(0.95,fine>=2)

LC-PFN(0.95,fine>=3)

LC-PFN(0.95,fine>=4)

LC-PFN(0.95,fine>=5)

LCBench

NAS-Bench-201

Taskset

6 × 10-1

4 ×10-1

Average Regret

Average Regret

Average Regret

10-

3 ×10-1

10-1

2 × 10-1

0

200

400

600

800

1000

0

500

1000150020002500300035004000

0

200

400

600

800

1000

PD1 - Language Modeling (lm1b)

PD1 - Translation (WMT)

PD1-Protein Sequence (UniRef50)

Average Regret

100

Average Regret

100

Average Regret

10-1

6 ×10-2

10-

10-1

4 ×10-2

3 ×10-2

0

100

200

300

400

500

600

700

0

5000

10000

1500020000

25000

0

100

200

0OE

400

L8

LC-PFN

4CMC-PP

trget

LC-PFN

MCMC-PP

LC-PFN

target

MCMC-PP

0.2

LC-PFN

MCMC-PP

LC-PFN

Nd3

Nd1

MCMC-PP

8'0

136ue

0.2

target

LC-PFN

0.2

MCMC-PP

LC-PFN

0.2

trget

MCMC-PP

LC-PFN

0.2

MCMC-PP

LC-PFN

data

1

20

20

LC-PFN

MCMC-PP

1.0

MCMC-PP

LC-PFN

MCMC-PP

LC-PFN

MCMC-PP

8℃

target

0.8

target

data

target

8'0

target

9'0

MCMC-PP

LC-PFN

LC-PFN

data

target

MCMC-PP

LC-PFN

data

LC-PFN

MCMC-PP

arges

MCMC-PP

MCMC-PP

MCMC-PP

LC-PFN

LC-PFN

LC-PFN

LC-PFN

0.8

target

data

0.8

target

data

LC-PFN

LC-PFN

LC-PFN

LC-PFN

data

target

MCMC-PP

LC-PFN

MCMC-PP

LC-PFN

MCMC-PP

LC-PFN

target

target

target

data

data

data

MCMC-PP

LC-PFN

Epochs

20

Epochs

0'

2

Epochs

MCMC-PP

LC-PFN

1.0

LC-PFN

MCMC-PP

target

epep

0.8

target

data

target

data

target

20

80

20

Epochs

30

0

10

20

Epochs

30

40

10

20

Epochs

1.0

MCMC-PP

1.0

MCMC-PP

LC-PFN

LC-PFN

target

target

MCMC-PP

LC-PFN

0.2

MCMC-PP

LC-PFN

target

target

data

20

Epochs

data

10

20

Epochs

30

40

0'0

10

20

Epochs

OE

40

0.0

OT

20

Epochs

1.0

MCMC-PP

1.0

MCMC-PP

LC-PFN

LC-PFN

0.8

186e

data

0.8

target

epep

MCMC-PP

LC-PFN

MCMC-PP

LC-PFN

target

target

10

20

30

40

data

0.0

10

Epochs

30

40

data

0S

0°0

10

20

Epochs

30

0.0

10

20

LC-PFN

target

data

0.2

LC-PFN

MCMC-PP

0.2

targt

LC-PFN

MCMC-PP

0.2

tarset

LC-PFN

MCMC-PP

Epochs

10

20

Epochs

W

10

20

Epochs

30

U

0.0

IV

Epochs

MCMC-PP

1.0

MCMC-PP

1.0

MCMC-PP

1.0

MCMC-PP

LC-PFN

LC-PFN

LC-PFN

LC-PFN

target

data

0.8

target

data

0.8

target

data

0.8

target

data

Z°0

0.2

20

30

0.0

10

20

Epochs

30

40

50

10

20

Epochs

30

40

0.0

10

20

30

MCMC-PP

1.0

MCMC-PP

MCMC-PP

MCMC-PP

·

LC-PFN

target

data

0.8

LC-PFN

target

data

LC-PFN

data

target

data

LC-PFN

target

Epochs

Epochs

Epoct

AW

MVW

WW

0.g

g

0.4

0.2

MCMC-PP

LC-PFN

0.2

MCMC-PP

LC-PFN

0.2

MCMC-PP

LC-PFN

0.2

MCMC-PP

LC-PFN

50

75

125

150

175

200

25

50

75

Epochs

100

125

150

175

200

25

50

75

Epochs

100

125

150

175

200

00

25

p

75

125

150

175

1.0

一

target

data

target

data

二

data

0.8

一

MCMC-Pp

LC-PFN

target

data

23

0.4

0.4

0.4

0.2

0.0

25

50

75

100

Epochs

125

150

175

200

25

50

75

Epochs

100

125

150

175

200

00

25

50

75

Epochs

100

125

150

175

200

0.0

25

50

75

Epochs

100

125

150

175

200

一

target

LC-PFN

MCMC-PP

data

一

MCMC-PP

N

target

data

1.0

—target

二

·

MCMC-PP

data

— target

·

MCMC-PP

LC-PFN

data

0.4

0.2

0.2

50

75

Epochs

100

125

150

175

200

00

25

50

75

125

150

175

200

0.0

25

50

75

Epochs

100

125

150

175

200

25

50

75

125

150

175

1.0

二

二

MCMC-Pp

target

LC-PFN

data

1.0

0.8

MCMC-PP

LC-PFN

target

data

·

data

1.0

0.8

— target

MCMC-PP

LC-PFN

data

AM

0.2

25

50

75

100

125

150

175

50

75

Epochs

100

125

150

175

200

0.0

25

Epochs

100

125

150

175

200

0.0

25

50

75

100

125

150

175

MCMC-PP

target

LC-PFN

data

1.0

MCMC-PP

LC-PFN

target

data

MCMC-PP

LC-PFN

0.8

MCAC-PP

LC-PFN

target

data

M

0.4

MMh

0.6

MMh

0.4

0.2

0.2

0.2

25

Epochs

100

125

150

175

200

0.0

50

75

fpochs

100

125

150

175

00

25

50

75

Epochs

100

125

150

175

200

0.0

Epochs

125

150

175

MCMC-PP

LC-PFN

1.0

MCMC-PP

LCPFN

一

target

data

target

data

MW

J

0.4

0.2

二

MCMC-Pp

LC-PFN

0.2

MCMC-PP

LC-PFN

0.2

·

25

50

75

100

fpochs

125

150

175

200

25

50

75

Epochs

100

125

150

200

25

50

75

Epochs

100

125

150

200

75

Epochs

100

125

150

175

200

1.0

18

1.0

MCMC-Pp

LC-PFN

target

data

0.6

0.4

0.4

0.4

0.0

0.2

一

·

MCMC-PP

LC-PFN

25

50

75

100

Epochs

125

175

200

00

0.2

一

MCMC-PP

LC-PFN

25

50

75

100

Epochs

125

150

175

200

0.2

一

LC-PFN

MCMC-PP

25

50

75

Epochs

100

125

150

200

0.2

00

25

50

75

Epochs

100

125

175

200

00

MCMC-PP

MCMC-PP

MCMC-PP

MCMC-PP

LC-PFN

target

target

LC-PFN

LC-PFN

target

LC-PFN

target

data

data

Sata

data

0.2

0.2

02

0.1

80

S501

0.3

0.2

0.2

0.2

0.2

I°0

LCPFN

MCMC-PP

0.1

LC-PFN

MCMC-PP

0.1

一

LC-PFN

taret

MCMC-PP

0.1

一

target

LC-PFN

10

20

Epochs

30

0.0

10

20

Epochs

30

0.0

10

Epochs

30

10

20

Epochs

0.6

0.6

LC-PFN

data

MCMC-

carget

MCMC-PP

LC-PFN

data

target

0.5

30.3

0.2

0.2

0.2

0.1

targt

LC-PFN

MCMC-PP

0.1

LC-PFN

MCMC-PP

01

0.1

10

20

80

40

10

20

Epochs

30

60

0.0

Epochs

30

10

30

0.6

0.5

0.5

0.5

0.5

0.3

03

0.2

0.2

MCMC-PP

LC-PFN

0.1

MCMC-PP

LC-PFN

0.1

MCMC-PP

taset

LC-PFN

MCMC-PP

N-1

10

20

Epochs

30

40

10

20

Epochs

30

40

0°0

10

Epochs

30

40

0°0

10

20

Epochs

30

LC-PFN

MCMC-PP

MCMC-PP

target

target

0.5

0.5

603

0.2

0.1

MCMC-PP

LC-PFN

0.1

MCMC-PP

LC-PFN

0.1

0.1

10

20

Epochs

40

10

20

Epochs

30

20

Epochs

40

Epochs

30

MCMC-PP

LC-PFN

MCMC-PP

LC-PFN

target

target

target

lata

0.5

S5OT

0.2

0.2

.2

MCMC-PP

t'0

LC-PFN

0.1

0.1

0.1

10

20

Epochs

OE

10

20

Epochs

30

40

20

Epochs

30

40

0.0

10

20

Epochs

30

MCMC-PP

MCMC-PP

MCMC-P

MCMC-PP

LC-PFN

target

0.6

LC-PFN

target

9'0

target

LC-PFN

0.6

target

LC-PFN

data

data

data

data

5507

0.2

0.2

0.1

0.1

0.1

Epoch

Epoch

Epochs

Epoche

4.3

MCMC-PP

MCMC-PP

MCMC-PP

MCMC-PP

LC-PFN

tasgt

LC-PFN

LC-PFN

LC-PFN

target

。

data

3.7

3.7

3.6

3.5

3.5

10

15

Epochs

20

25

30

10

15

25

30

35

10

15

25

30

35

5

10

15

Eno20

Epochs

25

30

SE

MCMC-PP

MCMC-PP

4.6

MCMC-PP

4.4

LC-PFN

data

·

target

data

LC-PFN

4.4

一

target

data

LC-PFN

LC-PFN

MCMC-PP

4.2

4.2

4.0

data

target

4.0

4.0

4.0-

3.8

3.8

3.8

3.8

3.6

3.6

3.6

10

15

25

30

10

15

25

30

35

10

Epochs

20

25

30

SE

10

15

Epochs

25

30

SE

MCMC-PP

LC-PFN

二

MCMC-PP

LC-PFN

MCMC-PP

data

LC-PFN

target

MCMC-PP

LC-PFN

target

data

200

400

600

Epochs

800

1000

1200

1400

200

400

600

Epochs

800

1000

1200

1400

200

400

600

Epochs

800

1000

1200

1400

200

400

600

Epochs

800

1000

1200

1400

MCMC-Pp

MCMC-PP

MCMC-PP

MCMC-PP

LC-PFN

target

tagt

LC-PFN

LC-PFN

target

LC-PFN

target

data

·

data

data

2

2

200

400

600800

1000

1200

1400

200

400

600

C800

1000

1200

1400

200

400

600

800

1000

1200

1400

200

400

600

Einoc800

1000

1200

1400

Epochs

Epochs

Epochs

Epochs

MCMC-PP

MCMC-PP

MCMC-PP

MCMC-PP

LC-PFN

target

data

LC-PFN

target

data

LC-PFN

data

target

LC-PFN

target

data

200

400

600

Epochs

800

1000

1200

1400

200

400

600

Epochs

800

1000

1200

1400

200

400

600

800

1000

1200

1400

200

400

600

Epochs

1000

1200

1400

MCMC-PP

MCMC-P

2.85

2.85

LC-PFN

2.85

一

LC-PFN

data

target

·

data

target

2.80

2.80

2.80

2.80

2.75

2.75

2.75

2.75

501

3

2.70

2.70

2.70

2.70

2.65

LC-PFN

2.65

MCMC-P

LC-PFN

2.65

2.65

00

5.0

7.5

10.0 12.5

Epochs

15.0

17.5 20.0

0.0

2.5

5.0

7.5

10.0  12.5

Epochs

15.0

17.5

20.0

0.0

2.5

5.0

S'L

10.0

Epochs

12.5

15.0

17.5 20.0

0.0

25

5.0

7.5

10.0

Epochs

12.5

15.0

17.5 20.0

2.800

2.800

2.800

2.800

2.775

2.775

2.775

2.775

2.750

2.750

2.750

2.750

2.00

2.700

2.700

2.700

2.675

2.675

2.675

2.675

2.650

MCMC-PP

2.650

MCMC-PP

2.650

MCMC-PP

2.650

MCMC-Pp

2.625

·

一

LC-PFN

2.625

LC-PFN

2.625

LC-PFN

2.625

·

一

LC-PFN

0.0

2.5

5.0

7.5

10.012.5

15.0

17.5 20.0

0.0

2.5

5.0

7.5

10.0 12.5

15.0

17.5

20.0

0.0

2.5

5.0

7.5

10.012.5

15.0

17.520.0

0.0

2.5

5.0

7.5

10.012.5

15.0

17.520.0

Epochs

Epochs

Epochs

Epochs

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the task of learning curve extrapolation, i.e., the aim is to predict the performance of a given model wrt. e.g., accuracy/log-likelihood over time, given current observations.\nTheir proposal primarily relies on Prior-data fitted networks (Müller et al., 2022) trained on samples from a prior set of curves adapted from their sole comparison method by Dunham et al. (2015). \nCompared to this prior MCMC-based method (Dunham et al., 2015), the authors can show greatly improved performance in most setups as well as very strong improvements in runtime.\n\nSTRENGTHS:\n- The paper focuses on an important task in the AutoML literature that has so far not received a lot of attention.\n- The authors evaluate their approach extensively on a varied set of experiments properly evaluating all their claims.\n- The paper is overall well written with two minor deficiencies (see next section). \n- The authors provide an extensive implementation of the model. However, it is also necessary as replicability would be difficult given the coarse level of detail in the written part.\n- With respect to originality. The contribution is primarily in the application. The model itself relies on the PFN model by Mueller and the prior is an adaptation of Dunham et al.\nThis can be seen as a weakness, yet given that the application of PFNs to this field is novel and the application is an important one with clear results, I see this only as a very minor weakness if at all.\n- Concerning significance, I lack a deep understanding of the AutoML literature to properly judge the significance. But from what I know of the field, the results look very promising and should be of great interest to many readers.\n\nWEAKNESSES:\n- It has two deficiencies in the writing/structure of the paper. \n   1.  Its backbone is a prior-data fitted network. However, that model itself is only briefly introduced in a single paragraph with a figure that is barely understandable without reading the original paper.\nThe paper requires a proper discussion of this approach in either the main text or the appendix. \n   2. Training details and hyperparameters are barely discussed with many essentials missing, e.g., the reader has to guess the meaning of `nb_data`, `emsize`, `nlayers`. It is an easy guess, but a guess nevertheless. Adding the abbreviations to the paragraph in l157 would quickly fix this.\n- As stated above, the novelty is almost solely in the application which could be considered a weakness. But in my opinion this is only a very minor weakness if at all.\n- Table 3 lacks error bars.\n\n## Minor\n- Figure 2 is missing details on the colors of the arrows \n- The caption of Table two should be above\n- Figure 3 contains an unexplained horizontal line\n- l319-320 the sentence is broken (interesting appears twice)\n\nQUESTIONS:\n- In l216 the authors mention that the best MCMC-PP method took over 80 seconds. In Table 3 the 80s method does, however, not not improve upon the 30-second version. Can the authors clarify this statement?\n\nLIMITATIONS:\nThe authors discuss limitations of their method, but no potential negative societal impact of their work. The second is of lesser importance in this work anyway.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

comments: SUMMARY:
The authors in this submission applied the prior-data fitted neural networks (PFNs) in the learning curve extrapolation task, for which the main goal is to predicit the performance of a machine learning model in later epochs, based on the information from earlier epochs. The target is modeled as a linear combination of basis growth curves, and the proposed LC-PFN algorithm aims to minimize the cross entropy loss as in PFN. The authors tested the proposed method across a few datasets and showed that PFN (i) achieves better prediction and (ii) is computationally efficient, when compared with its competitors.

STRENGTHS:
1. The idea of applying PFNs in the learning curve extrapolation task sounds reasonable, and is intuitively more efficient when compared with MCMC.

2. The application of LC-PFN in early stopping could be valuable, as it can be helpful for the model selection step. The experiments in Section 4.3 also provide some promising results.

WEAKNESSES:
1. In my opinion the selection for the 3 parametric basis curves seems a bit ad-hoc. It's not convinving whether they are sufficient to fit different learning curves. Furthermore, selection of hyperparemters in priors there also lacks details.

2. The results could be more convincing if the authors can compare aganist more previous methods, instead of MCMC only. For instance, how about the one by Klein et al., 2017, which used Bayesian neural networks for learning curve prediction?

QUESTIONS:
1. In Table 1, is the support for x integers, as it corresponds to $f_k (t | \theta_k)$ in the model?

2. For the equation just below L132, what does $<$ mean? 

3. Do you need to assume $t > T^{'}$ in your LC-PFN model during training?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

comments: SUMMARY:
The authors propose applying prior-data fitted NNs to learning curve extrapolation.

The authors demonstrate that this approach outperforms MCMC inference and is substantially faster.

STRENGTHS:
The paper has a novel idea of applying approximate inference via meta-learning learning curves.

The method appears to work in experimental evaluation, but more experiments would make the paper stronger.

The manuscript (all but experimental section) is clear and relatively easy to follow.

The authors provide the code.

WEAKNESSES:
The paper contains only two experiments. One is based on synthetic data.

Table 1 has no error bars. Table 1 indicates that while the performance of MCMC is stable across the hyperparameters, PFN score is dependent on the network size indicating it might not generalize well to other problems.

While the difference in time is clearly significant, the differences in LL are rather small.

While the idea of applying the described method in the considered context is novel, much of this work is a combination of previous work by Domhan 2015 and Muller 2022.

The experimental section could have been written in a more clear way.

QUESTIONS:
Is poor performance of MCMC attributed to slow mixing or initializing at a poor point? What about MCMC inference initialized at MAP?

If MCMC has substantially more computational time will it outperform PFN? At the end of the day exact inference should outperform approximate inference.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors used prior fitted networks to perform learning curve extrapolation. Crucially, they demonstrate that their method vastly outperforms approximate Bayesian inference via MCMC (both in terms of inference time and predictive log likelihood). Moreover, they demonstrate that the proposed approach outperforms heuristics for ending unpromising training runs commonly used in hyper-parameter optimization.\n\nSTRENGTHS:\nThis paper prevents a super practical method that not only performs well *but* also runs fast, allowing it to be used in many real-time applications. The practicality of the method is extremely bolstered by the impressive suite of empirical experiments performed. Lastly, the paper was written well and was pretty easy to follow (though I have some comments).\n\nWEAKNESSES:\nThe biggest weakness to me is that I don't think the authors spent time on the potential difficulty of using MCMC for this problem. Specifically, the prior, and the corresponding posterior, is constrained on some non-standard subset that prevents standard MCMC algorithms to be used. The geometry of this will heavily affect the performance of MCMC algorithms as samples that are proposed outside of the set will always be rejected requiring substantial tuning. That being said, I am a little concerned with the number of samples the chain was run: 4,000 samples seems too low for this problem.\n\nQUESTIONS:\nBelow I will list some comments\n1. In figure 1 there is a black dashed line but I can't find in the text what it represents. Without prior knowledge, I thought that maybe it was the true log posterior predictive distribution, which would make it seem like MCMC is outperforming the proposed method. Also, error bars here would be great.\n2. For all results, it would help if it was made clear if higher values are better or the converse.\n3. For Fig. 4, errors bar would be very informative, especially given the range of values. I suggest plotting 20% and 80% quantiles.\n4. In Fig 5., it isn't clear what the error bars are, i.e. standard error, quantiles, etc.\n\nLIMITATIONS:\nThe authors did a good job explaining the limitation of the methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}