SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Runtime (lower is better) vs log-likelihood of the true curve under the PPD (higher is better), with  $10\%$  of the curve observed. See Figure 9 in Appendix C.1 for higher cutoffs. Blue and red markers correspond respectively to LC-PFN and MCMC-PP with varying hyperparameters values. The M1-3/P1-3 labels refer to the PFN/MCMC variants listed in Table 3. The horizontal dashed line indicates the performance of the best MCMC variant.
Table 3: Comparison of three LC-PFN and MCMC-PP variants on prior curves in terms of log-likelihood (higher is better) at  $10\%$ ,  $20\%$ ,  $40\%$ , and  $80\%$  cutoffs. Here, M1 corresponds to the configuration used in Domhan et al. [2015]. Please refer to Table 5 for more comprehensive results.
Label	Method	Parameters	10%	20%	40%	80%	Avg. Runtime (s)
M1	MCMC	nsamples=2000, nwalkers=100, burn-in=500, thin=1	1.628	1.939	2.265	2.469	54.401
M2	MCMC	nsamples=4000, nwalkers=100, burn-in=100, thin=100	1.641	1.958	2.277	2.477	45.160
M3	MCMC	nsamples=4000, nwalkers=100, burn-in=500, thin=1	1.642	1.956	2.285	2.486	103.151
P1	PFN	nb_data=10M, nlayers=3, emsize=128	1.58	1.99	2.28	2.43	0.004
P2	PFN	nb_data=10M, nlayers=3, emsize=256	1.65	2.04	2.35	2.49	0.006
P3	PFN	nb_data=10M, nlayers=12, emsize=512	1.76	2.13	2.40	2.52	0.050
4.2 Extrapolating real-world learning curves
While evaluation on data from the prior gives us a controlled setting to analyse quality and cost of the PPD approximation, performance on real-world learning curves is essential for practical usefulness. This second experiment aims to extend the previous comparison of MCMC and LC-PFN to real-world learning curve benchmarks.
We consider the best-performing variants of LC-PFN and MCMC-PP according to the average log-likelihood they obtained in the first experiment. For LC-PFN, the optimal variant (P3) features an embedding size of 512 and 12 layers, resulting in a total of 26M trainable parameters, and is trained on 10 million prior curves. For MCMC-PP, the optimal configuration (M3) involves a chain length of 4500, 100 walkers, 500 burn-in samples, without thinning. As an additional baseline, we include MCMC-OP, the original MCMC variant proposed by Domhan et al. [2015], which uses the original hyperparameters and curve prior (11 basis curves and uninformative prior over the curve parameters).
**Benchmarks** To evaluate the generalization capabilities of our model, we consider a diverse set of real-world curves. Our dataset comprises 20000 learning curves, sourced from four distinct benchmarks: LCBench [Zimmer et al., 2021], NAS-Bench-201 [Dong and Yang, 2020], Taskset [Metz et al., 2020] and PD1 [Wang et al., 2022], each contributing 5000 curves, randomly selected from specific subtasks. These benchmarks and subtasks were chosen to span a broad spectrum of supervised deep learning problems, training MLP (LCBench), CNN (NAS-Bench-201), RNN (Taskset), and Transformer (PD1) architectures on input modalities ranging from tabular data (LCBench), text (Taskset and PD1), protein sequence (PD1), to vision problems (NAS-Bench-201). From LCBench and NAS-Bench-201 we use validation accuracy curves whereas from Taskset and PD1 the log loss validation curves. In terms of curve length, LCBench and Taskset cover 50 epochs, NAS-Bench-201 contains up to 200 epochs, and PD1 curves have varying lengths (22 - 1414). Further details, including sample curves, on these benchmarks are provided in Appendix B.
(a) Average rank of log-likelihood values (lower is better) vs cutoffs
(b) Average rank of mean squared error (MSE) values (lower is better) vs cutoffs
Figure 4: Comparison of LC-PFN with two MCMC variants on three real-data benchmarks.
Metrics Our focus lies on the relative performance of MCMC and LC-PFN, as absolute performance is significantly influenced by the choice of prior. We consider the log-likelihood and the mean squared error (MSE) of the predictions as metrics. Here, the MSE is calculated w.r.t. the median of the PPD.
For each benchmark, we report the average rank of these metrics to aggregate results from different curves, as supposed to the average values. While the latter better captures performance differences, it is very sensitive to outliers and scale-dependent. When computed in normalized space, it would strongly depend on our choice of normalization parameters (see Appendix A).
Results For each benchmark, Figures 4a and 4b display the average rank obtained by each method in terms of log-likelihood and MSE, respectively, where ranks are averaged across all 5000 curves. We do not include error bars as the standard errors are negligible (less than 0.02). In summary, we observe similar trends for both metrics on all benchmarks. LC-PFN is never truly outperformed by MCMC-PP. On LCBench both methods rank similarly, with LC-PFN being slightly worse at high cutoffs. LC-PFN ranks better on PD1, NAS-Bench-201, and Taskset. MCMC-OP performs clearly worse on LCBench, Taskset, and PD1. On NAS-Bench-201, MCMC-OP performs best for lower cutoffs, outperforming MCMC-PP, suggesting that NAS-Bench-201 curves are better captured by the original prior. Figure 11 and Figure 12 in Appendix C.2 show the log-likelihoods and MSEs, respectively, for all three methods, for each curve and cutoff, per benchmark, providing a more detailed perspective.
4.3 Application: Extrapolation-based early stopping in model selection
Thus far, we have shown that LC-PFN produces extrapolations of similar or better quality to MCMC, at a small fraction of the cost. However, these extrapolations are not perfect. In many cases, the practical relevance of any errors can only be assessed in the context of a specific application.
In this final experiment, we thus consider a model selection setting where after every epoch of training we have the choice between (i) continuing the current training, or (ii) stopping early  $(T < m)$  and starting a new training run using a different training pipeline (e.g., model architecture, hyperparameters, etc.). Here, we assume that runs cannot be resumed once stopped and that the order in which training pipelines are to be considered is given. Our objective is to obtain high-quality models as quickly as possible, by stopping suboptimal training runs as early as possible. This setting is also known as vertical model selection [Mohr and van Rijn, 2022] and was also considered by Domhan et al. [2015].
LC-PFN MCMC(original) MCMC(our prior)
Figure 5: Comparison of our LC-PFN based early stopping mechanism to naive baselines no-stop (no stopping, train for the full budget  $m$ ) and Patience(k) (stop training after k epochs without improvement) for vertical model selection where runs are considered in a fixed order and cannot be resumed. Shown is the anytime regret (lower is better) different approaches achieve after a total number of training epochs, averaged per benchmark and across 40 orderings per task.
We consider the extrapolation-based termination criterion proposed by Domhan et al. [2015], but use LC-PFN instead of MCMC to predict the likelihood  $\operatorname{Pr}(y_{t'} > y_{\mathrm{best}} | y_1, \ldots, y_T)$  that the current run will at epoch  $t'$  obtain a model better than the best obtained by any run thus far ( $y_{\mathrm{best}}$ ), for  $T < t' \leq m$  and decide to stop the current run if that probability does not exceed some fixed threshold  $\delta$  (at any point). In our experiments, we use confidence level  $1 - \delta = 0.95$  as Domhan et al. [2015]. Note that this criterion can be applied at every step or only at specific cutoffs. To simulate the effect of varying granularity, we consider a coarse-grained variant with 4 cutoffs  $T \in \{\lceil 0.1m \rceil, \lceil 0.2m \rceil, \lceil 0.4m \rceil, \lceil 0.8m \rceil\}$ , and a fine-grained variant with  $T \in \{T | 1 < T < m\}$ . We investigate alternative choices for cutoffs and confidence levels in Appendix C.3.2.
Following Domhan et al. [2015], we compare against a black box approach no-stop that does not implement early stopping. We further compare against a criterion Patience (k) that implements the popular heuristic to terminate a training run when model performance did not improve for  $k$  epochs. For evaluation, we consider the same benchmarks as in Section 4.2 (for details, see Appendix B). We chose the total budget for model selection to correspond to 20 full runs (20m). For each task, we consider the training runs in 40 different random orderings. This totals 2120 model selection experiments per method, spread across the 53 different tasks.
Results Figure 5 shows the anytime performance of all methods in our comparison, on each of the benchmarks, in terms of regret. Here, regret is the absolute difference in performance between the best model obtained thus far and the best model attained by any run on the task. Results are averaged over the different run orderings. For LCBench, NAS-Bench-201, and Taskset results are further averaged across all tasks (results for individual tasks can be found in Appendix C.3.1). The shaded area corresponds to  $\pm 1$  standard error. On all benchmarks, except for NAS-Bench-201, we observe that LC-PFN based termination criteria clearly perform best. Also, the fine-grained variant performs better on average, suggesting that errors in inference are compensated for by the time saved by stopping runs earlier. In terms of expected speed-up, this LC-PFN variant obtains an expected regret lower than that obtained by no-stop, approximately  $3.3 \times$  faster on LCBench and Taskset. Looking at individual tasks, we note  $2 - 6 \times$  speed-ups on all 3 PD1 tasks, all 12 Taskset tasks, and 30 of the 35 LCBench tasks (we obtain  $1 - 2 \times$  speed-ups on the remaining 5). On NAS-Bench-201, we find that all termination criteria considered, including standard Patience heuristics, fail on all 3 tasks; this is likely related to the particular shape of learning curves on this benchmark, having an inflection point (see Figure 8 and Figure 19), not resembling any in the prior.
Finally, in terms of overhead, the computational costs of the LC-PFN inferences per model selection experiment range from 3 seconds (coarse-grained) up to 1 minute (fine-grained), and are negligible compared to the cost of the 20 full training runs of deep neural networks.
5 Summary, limitations, and future research
We presented the first work using prior-data fitted networks (PFNs) for Bayesian learning curve extrapolation. We show that our LC-PFN obtains qualitatively similar extrapolations, for a wide variety of learning curves, more than  $10000 \times$  faster than the MCMC method proposed by Domhan et al. [2015]. These inferences are now fast enough (under 100 milliseconds on CPU, and even less on GPU), to be used in the context of online learning, at virtually no overhead. This opens up a wide variety of possible applications, e.g., to speed up automated model selection in AutoML and HPO by discarding poor configurations early. It would be interesting to integrate LC-PFN as a new termination criterion in existing deep learning libraries.
Often, we also have more data available than a single partial learning curve, e.g., other curves on the same task, their hyperparameters, and/or curves of the same method on a different task, and meta-features. Previous work [Swersky et al., 2014, Klein et al., 2017, Wistuba et al., 2022, Ruhkopf et al., 2022] has already exploited this, and we could explore the potential of using PFNs for few-shot in-context meta-learning, by feeding the model multiple curves and hyperparameters as input.
While for a fair comparison to Domhan et al. [2015], reusing the original code, our prior (Section 3.2) closely resembled the one of Domhan et al. [2015], future work could improve upon this prior and overcome its limitations (e.g., on the NAS-Bench-201 tasks) by modelling divergence, slow start, double-descent, correlated heteroscedastic noise, etc.
Finally, PFNs, unlike other Bayesian methods, must learn the prior from data, which implies that the prior must be generative. Also, it suggests that high entropy priors may present challenges. Future research should investigate these limitations and how to overcome them.
6 Acknowledgments and disclosure of funding
We acknowledge funding by the European Union (via ERC Consolidator Grant Deep Learning 2.0, grant no. 101045765), TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant numbers INST 39/963-1 FUGG and 417962828. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR'17), 2017.
A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
L. Metz, N. Maheswaranathan, R. Sun, C. D. Freeman, B. Poole, and J. Sohl-Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies. arXiv:2002.11887 [cs.LG], 2020.
F. Mohr and J. van Rijn. Learning curves for decision making in supervised machine learning-a survey. arXiv:2201.12150v1 [cs.LG], 2022.
S. Müller, N. Hollmann, S. Arango, J. Grabocka, and F. Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference on Learning Representations (ICLR'22), 2022.
S. Müller, M. Feurer, N. Hollmann, and F. Hutter. PFNs4BO: In-context learning for bayesian optimization. In ICML 23, 2023.
T. Ruhkopf, A. Mohan, D. Deng, A. Tornede, F. Hutter, and M. Lindauer. Masif: Meta-learned algorithm selection using implicit fidelity information. Transactions on Machine Learning Research, 2022.
D. Salinas, M. Seeger, A. Klein, V. Perrone, M. Wistuba, and C. Archambeau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In International Conference on Automated Machine Learning, AutoML 2022, 2022.
K. Swersky, J. Snoek, and R. Adams. Freeze-thaw Bayesian optimization. arXiv:1406.3896 [stats.ML], 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS'17), 2017.
Z. Wang, G. E. Dahl, K. Swersky, C. Lee, Z. Mariet, Z. Nado, J. Gilmer, J. Snoek, and Z. Ghahramani. Pre-training helps bayesian optimization too, 2022.
M. Wistuba, A. Kadra, and J. Grabocka. Supervising the multi-fidelity race of hyperparameter configurations. In Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS'22), 2022.
Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315, 2007.
L. Zimmer, M. Lindauer, and F. Hutter. Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:3079-3090, 2021.
A Learning curve normalization
Not all learning curves of interest may resemble those of the prior we described in Section 3.2. For example, when minimizing the negative log-likelihood loss (log loss), learning curves will mostly decrease (vs. increase), be convex (vs. concave), and could take values that exceed 1.0 (there is no hard upper bound for log loss).
One approach would be to define a prior and train a specialized PFN for every performance measure. In this work, we use a more general approach: We apply a normalization procedure that allows us to accurately extrapolate a wide variety of learning curves using a single PFN, without retraining or fine-tuning. We consistently apply this procedure to all inferences involving real learning curves (i.e., the experiments described in Section 4.2 and Section 4.3), even if observations are naturally constrained to  $[0,1]$ , e.g., accuracy curves.
Step 1:Normalize the partial learning curve: Let  $y_{i}^{o}$  be the  $i^{\mathrm{th}}$  performance observation. We start by normalizing it using a generalized logistic sigmoid transform  $y_{i} = g_{\lambda}(y_{i}^{o})$  , that is re-parametrized by the following five parameters  $(\lambda)$  ..
min? A Boolean specifying that we expect learning to minimize the performance measure. E.g., this would be true for error rate and log loss, and false for accuracy.
$\mathbf{l}_{\mathrm{hard}}$ ,  $\mathbf{u}_{\mathrm{hard}}$  These are possibly infinite hard lower / upper bounds for model performance. This would, e.g., be  $0 / 1$  for accuracy and error rate; and  $0 / +\infty$  for log loss.
$\mathbf{l}_{\mathrm{soft}}$ ,  $\mathbf{u}_{\mathrm{soft}}$  These are finite soft lower/upper bounds for model performance. They specify the range in which we expect performance values (that we care to distinguish between) to lie in practice. For accuracy and error rate, one could set these equal to the hard bounds, whereas for log loss one could, e.g., use an estimate of the loss for the untrained network (i.e., at epoch 0) as  $\mathbf{u}_{\mathrm{soft}}$  and  $\mathbf{l}_{\mathrm{soft}} = \mathbf{l}_{\mathrm{hard}}$ , or if available, choose  $\mathbf{l}_{\mathrm{soft}}$  to be an optimistic estimate of performance at convergence (e.g., state-of-the-art). Narrower ranges will result in more accurate inferences.
Specifically, we perform a linear transformation, followed by a logistic sigmoid, another linear transform, and a minimization conditional reflection around  $\frac{1}{2}$ , i.e.,
$$
g _ {\boldsymbol {\lambda}} \left(y _ {i} ^ {o}\right) = \operatorname {c r} _ {0. 5} \left(\frac {c}{1 + e ^ {- a (x - b)}} + d\right)
$$
$$
\begin{array}{l} \operatorname {c r} _ {0. 5} (y) = \left\{ \begin{array}{l l} 1 - y & \text {i f m i n ?} \\ y & \text {i f \neg m i n ?} \end{array} \right. \quad a = \frac {2}{\mathrm {u} _ {\text {s o f t}} - \mathrm {l} _ {\text {s o f t}}} \quad b = - \frac {\mathrm {u} _ {\text {s o f t}} + \mathrm {l} _ {\text {s o f t}}}{\mathrm {u} _ {\text {s o f t}} - \mathrm {l} _ {\text {s o f t}}} \\ c = \frac {1 + e ^ {- a \left(\mathrm {u} _ {\text {h a r d}} - b\right)} + e ^ {- a \left(\mathrm {l} _ {\text {h a r d}} - b\right)} + e ^ {- a \left(\mathrm {u} _ {\text {h a r d}} + \mathrm {l} _ {\text {h a r d}} - 2 b\right)}}{e ^ {- a \left(\mathrm {l} _ {\text {h a r d}} - b\right)} - e ^ {- a \left(\mathrm {u} _ {\text {h a r d}} - b\right)}} \quad d = \frac {- c}{1 + e ^ {- a \left(\mathrm {l} _ {\text {h a r d}} - b\right)}} \\ \end{array}
$$
Note that for  $\lambda = (\text{False}, -\infty, -1, 1, \infty)$  this reduces to the canonical logistic sigmoid  $y_{i} = \frac{1}{1 + e^{-y_{i}^{o}}}$ . This transform will be approximately linear (shape preserving) for  $y_{i}^{o} \in [\mathrm{l}_{\text{soft}}, \mathrm{u}_{\text{soft}}]$ , and the tails of the sigmoid will gradually squish values outside this range to [0, 1], supporting unbounded performance measures. Figure 6 visualizes this projection in general, and provides examples for accuracy with  $\lambda = (\text{False}, 0, 0, 1, 1)$  and log loss with  $\lambda = (\text{True}, 0, 0, \log(10), \infty)$ .
Step 2: Infer using normalized data: Next, we perform Bayesian learning curve extrapolation on the normalized partial curve  $\pmb{y}$ , with our usual prior  $p(\pmb{y})$ , resulting in an approximation of the PPD in the transformed space.
Step 3: Inverse transform the PPD: Finally, we can obtain the property of interest of the PPD in the original space by applying the inverse transform, given by
$$
g _ {\bar {\lambda}} (y _ {i} ^ {p}) = \frac {\log (\frac {\mathrm {c r} _ {0 . 5} (y _ {i} ^ {p}) - d}{c - (\mathrm {c r} _ {0 . 5} (y _ {i} ^ {p}) - d)}) - b}{a}
$$
We use order-based statistics (e.g., median or other percentiles) in our experiments, to which we can simply apply  $g_{\lambda}^{-}$  directly since it is a monotonic transform. For other statistics (e.g., mean, variance) we may need to resort to Monte Carlo estimation, applying  $g_{\lambda}^{-}$  to samples of the PPD.
Figure 6: left: The generic logistic sigmoid transformation described in Appendix A. and used to normalize learning curves to support a wide range of possibly unbounded performance metrics. middle/right: An example of the normalization for curves maximizing accuracy / minimizing log loss, using the transformation.
Figure 7: Sample of 10 curves taken i.i.d. from the prior (Section 3.2). (Right) The same sample after applying the inverse transformation  $g_{\lambda}^{-}$  to obtain samples from the log loss prior as described in Appendix A, using the transform shown in Figure 6 (right).
By applying a given normalization, we effectively transform our prior, i.e.,  $p(\pmb{y}^o) = g_{\lambda}^{-}(p(\pmb{y}))$ . To get some intuition of what that prior looks like, we can sample from  $p(\pmb{y}^o)$  by applying  $g_{\lambda}^{-}$  to samples of our prior  $p(\pmb{y})$ , which is useful for fine-tuning the parameters of this transformation. Figure 7 (right) shows samples of the log loss prior for  $\lambda = (\text{True}, 0, 0, \log(10), \infty)$ .
B Detailed benchmark description
To evaluate the generalization capabilities of LC-PFN, we consider a diverse set of real-world curves. The reduce computational cost and improve reproducibility, we source these from four existing benchmarks that provide a collection of learning curves for a wide variety of supervised learning tasks. In what follows, we describe each of these benchmarks in more detail, the subset of tasks and curves we considered, and any preprocessing we did. Table 4 provides an overview of the characteristics of each benchmark and Figure 8 (a) visualizes the curves selected from each benchmark.
LCBench provides learning curve data for 2000 configurations of AutoPytorch [Zimmer et al., 2021] trained for 50 epochs on 35 tabular classification datasets from the AutoML benchmark [Gijsbers et al., 2019]. All of these configurations use momentum SGD to train an MLP, but with varying number of layers and units per layer, and varying optimization hyperparameters (batch size, learning rate, momentum, L2 regularization, dropout rate). In Section 4.2, we consider a subset of 5000 curves selected uniformly at random from all 70000 validation accuracy curves in the benchmark. In Section 4.3, we consider all 2000 validation accuracy curves, for every task, in 40 different orderings.
NAS-Bench-201 is a benchmark for Neural Architecture Search methods [Dong and Yang, 2020]. It provides learning curve data for training 15625 different architectures for 200 epochs on three different image classification datasets (CIFAR-10, CIFAR-100, and ImageNet16-120) for three different random seeds. Other aspects of the training pipeline (e.g., optimizer, hyperparameters)
Table 4: Overview of the characteristics of the real-world learning curve benchmarks we used in Section 4.2 and Section 4.3. For each benchmark, the last columns list the normalization parameters used (see Appendix A), where  $y_0$  corresponds to the performance of the untrained model.
benchmark	# subtasks	model architecture	input modality (datasets)	length (# epochs)	metric (split)	min?	lhard	lsoft	usoft	uhard
LCBench	35	MLP	Tabular Classification (OpenGL)	50	accuracy (val)	False	0	0	1	1
NAS-Bench-201	3	CNN	Image Classification (Cifar-10, Cifar-100, ImageNet16-120)	200	error rate (val)	True	0	0	1	1
Taskset	12	RNN	Text Classification (Sentiment Analysis - IMDB)	50	log loss (val)	True	0	0	y0	+∞
PD1	3	Transformer	Text (Language Modeling - lm1b)	38	log loss (val)	True	0	3.46	y0	+∞
Transformer	Text (Translation - WMT)	1414	log loss (val)	True	0	1.68	y0	+∞
Transformer	Protein sequences (UniRef50)	22	log loss (val)	True	0	2.60	y0	+∞