SlowGuess/NeurIPS_2023_OCR huggingface dataset:
are fixed. We use the validation error rate curves provided through the Syne Tune [Salinas et al., 2022] interface to this benchmark. As discussed in Section 3.4, the LC-PFN we consider is trained to extrapolate curves up to length  $100(m)$ . To handle the 200 epoch curves from NAS-Bench-201, we chose to subsample them, feeding only every second observation into the LC-PFN. In Section 4.2, we consider a subset of 5000 curves selected uniformly at random from all 140625 error rate curves in the benchmark. In Section 4.3, we consider all 46875 error rate curves, for each of the three datasets, in 40 different orderings.
Taskset [Metz et al., 2020] provides a total of roughly 29 million learning curves for over 1162 different deep learning tasks. Here, a task is defined as optimizing a given neural network architecture for a given supervised learning problem, and the curves correspond to using 5 different optimizers, using 1000 different hyperparameter settings, and 5 different random seeds. Following Wistuba et al. [2022], we only use the validation log loss curves of a small subset of 12 tasks, that consider training different RNN architectures, with varying architectural parameters, for 50 epochs on the IMDB sentiment analysis dataset [Maas et al., 2011], a binary text classification task. As our objective was to evaluate the robustness of our approach, we avoided excluding ill-behaved (e.g., diverging) curves. However, upon inspecting the data, we found that on some of these tasks up to  $90\%$  of the curves fail to significantly improve upon the initial model performance, some of which diverge almost instantly. In combination with the normalization procedure described in Appendix A, the majority of curves are effectively mapped onto the same narrow range, producing a constant trend (at 0 in case of divergence). While this is reasonable in practice, and both LC-PFN and MCMC variants predict this constant trend with very high confidence, it does create a bias in our evaluation towards methods excelling at extrapolating poor curves. To eliminate and investigate this bias, we selected the 5000 curves used in Section 4.2, such that 2500 are taken from the top  $10\%$  best<sup>4</sup> curves per task, and 2500 from the  $90\%$  others. In Figure 4, we compared methods on the "good" curves only, an evaluation on the 2500 "bad" curves is presented in Figure 8 (b). In Section 4.3, we do not make this distinction and consider all 25000 curves per task, in 40 different random orderings.
PD1 is a recent benchmark that Wang et al. [2022] describe as collecting "a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset". We access this benchmark through the synetune [Salinas et al., 2022] library interface, which provides learning curve data for 23 tasks, where the number of curves provided, as well as the curve length, varies per task. Here, we limit our selection to log loss curves for the three tasks that consider training Transformer architectures for 22 epochs on language modeling (lm1b), 1414 epochs on translation (WMT), and 38 epochs on protein sequences (UniRef50). To handle the very long learning curves on the translation task, we subsample these aggressively, feeding only every 14th observation into the LC-PFN. In Section 4.2, we select 5000 curves from these three tasks uniformly at random. In Section 4.3, we consider all curves per task, in 40 different random orderings. Finally, we use the optimal loss achieved by any of the runs as  $l_{\mathrm{soft}} > 0$  in our normalization. While unknown in practice, we do not expect qualitative differences in evaluation if a rough estimate were to be used instead.
This was our first choice of  $m$  and we retained it throughout the project. We do not anticipate serious issues when scaling  $m$  up to 1000, except for increased training times.
<sup>4</sup>Curves are ranked based on the lowest validation loss obtained at any point during training.
(a) Learning curves sourced from each benchmark
(b) Quality of curves vs. quality of extrapolations on Taskset (average rank, lower is better)
Figure 8: (a) Illustration showing the 5000 curves used for each of the four benchmarks in our experiments in Section 4.2. All curves are shown after normalization and subsampling, and a random subset of individual curves is highlighted. For Taskset, "good" / "bad" curves (top  $10\%$  / bottom  $90\%$  in their task) are shown in green / red, respectively. (b) Average rank of Log-likelihood and MSE values on Taskset curves for the three methods (LC-PFN, MCMC-PP, and MCMC-OP). The two leftmost plots show results on the 5000 sampled Taskset curves. The two rightmost plots provide the same comparison, but specifically for the 2500 "bad" curves. Here, we observe that MCMC-PP performs better on the "bad" curves, which we believe can be attributed to the discretization of the PPD in 1000 bins, limiting the maximal confidence / accuracy of LC-PFN, since the majority of these curves are quasi-constant (fall in the same bin) after normalization.
C Additional analyses
C.1 Extrapolating samples of the prior
C.1.1 Detailed results of section 4.1
Figure 9 provides a visual representation of the log-likelihood values on prior curves for higher cutoffs, similar to Figure 3. As expected, the difference in log-likelihood values between LC-PFN and MCMC-PP decreases as more points of the curve are observed. The detailed results can be found in Table 5. Both Figure 9 and Table 5 clearly demonstrate that certain variants of LC-PFN, particularly those trained with 10M examples, consistently outperform the best variant of MCMC-PP across all considered cutoffs.
Figure 9: Comparison of LC-PFN and MCMC-PP with varying hyperparameters values on prior curves in terms of average runtime (lower is better) and average log-likelihood (higher is better) for different cutoff values (20%, 40%, 80%)
Figure 10: runtime vs log-likelihood values of MCMC-PP, using thinning 100, with different initialization strategies and sample sizes (per walker), averaged across 1000 curves generated i.i.d. from the prior. The blue point represents LC-PFN, with the dotted line as a reference for its LL value. The shaded areas correspond to  $\pm 1$  standard error.
C.1.2 Effect of MCMC chain length and initialization
In our experiments in Section 4.1 we observed that the quality of MCMC inference increases with chain length. In particular, our best MCMC-PP variant (M3 in Figure 3), uses the same hyperparameter settings as Domhan et al. [2015] (M1), but with a double as long chain (discounting the burn-in). In this section, we investigate whether the chains considered are too short and whether increasing chain lengths further will eventually cause MCMC-PP to outperform LC-PFN. To study this, we will run MCMC-PP to collect up to 100 000 samples per walker ( $50 \times$  more than Domhan et al. [2015]). To reduce the computational cost of this experiment, we only extrapolate 1 000 (instead of 10 000) prior curves, on cutoff  $10\%$ , and use MCMC-PP with a 100-thinning strategy (M2). This variant is more comput-efficient and the minor negative effect of thinning is expected to further decrease with increasing chain length. Concurrently, we study the effect of initialization. There are different ways of initializing MCMC chains and the chosen strategy can strongly affect performance if chains are too short. Domhan et al. [2015] initializes the chain by setting the parameters for each of the  $K$  basis curves to their Least-Squared Estimates (LSE). Weights are initialized to be  $\frac{1}{K}$ . If this initial point violates the constraints imposed by the prior, a default starting point is used instead. As an ablation, we compare it to an initialization always starting at the default (default). As another ablation, we compare to a strategy using the maximum a posteriori estimate (MAP) as a starting point instead, which unlike LSE takes the likelihood under the prior into account.
Figure 10 shows the quality and cost of inferences for each of the initialization strategies when collecting up to 100000 samples (per walker). While these results clearly show that increasing chain length continues to improve performance. The trends we observe, suggest that MCMC-PP could eventually attain or even overtake the best LC-PFN. That being said, the best MCMC we considered has  $20000 \times$  longer runtimes than that LC-PFN requires, yet it does not quite reach the same performance, so outperforming it would require impractically long chains. When optimistically extending the trends, we estimate to need at least  $10 \times$  longer chains, and inference times of multiple hours. In terms of initialization, we observe that MCMC performance for shorter chains indeed more strongly depends on the initialization strategy, where the LSE strategy by Domhan et al. [2015] does best for short chains, followed by MAP and default. When increasing chain length, we find that differences get smaller and order reverses such that the fixed default initialization is best, suggesting that greedy initialization (LSE, MAP) may hurt performance on some curves.
C.2 Extrapolating real-world learning curves
In addition to the average rank plots presented in Section 4.2, we also perform pairwise comparisons of the log-likelihood and mean squared error values among the different methods (LC-PFN and MCMC variants). This approach allows for a more detailed analysis, as it compares the absolute values of the metrics. Unlike the rank plots, these pair and curvewise plots provide insights into outliers which thus further enhance our understanding of the results.
(a) PFN vs MCMC with our prior (MCMC-PP)
(b) PFN vs MCMC as in Domhan et al. [2015] (MCMC-OP)
Figure 11: Pairwise comparison of log-likelihood (higher is better) between PFN and MCMC.
Figure 11 presents a pairwise comparison of the log-likelihood values between our PFN method and the two variants of MCMC, per curve, considering different cutoff values on the four benchmark datasets. Figure 12 presents the same comparison based on mean squared error (MSE).
We observe that LC-PFN compares favorably to MCMC-PP on PD1 and NAS-Bench-201 tasks. Both methods demonstrate comparable performance on the Taskset dataset. However, it is worth noting that for some curves of the LCBench dataset, MCMC-PP obtains very high log-likelihood and low MSE scores, while LC-PFN falls short in this aspect. We believe this discrepancy can be attributed to the fact that the LCBench dataset contains a significant number of constant curves (see Figure 8a) that fall within the same bin of the discretized PPD, effectively limiting the the maximal confidence / accuracy of LC-PFN. Looking at MSE specifically, we find that for some LCBench curves, at high cutoffs, LC-PFN obtains very high errors, while MCMC-PP does not. Upon closer inspection, we found that these curves quickly converge to a value close to the optimal accuracy, and while LC-PFN captures this trend for lower cutoffs, it suddenly fails for larger cutoffs (see 5th example in Figure 18). This can likely be explained by the fact that these curves are not adequately captured by the prior proposed in Section 3.2 and this seems to be one of the few cases where LC-PFN generalizes poorly out of distribution.
When comparing with MCMC-OP, we observe that MCMC-OP has relatively few outliers in terms of log-likelihood, both in the positive and negative sense. This can likely be explained by the prior of MCMC-OP, which is more flexible (i.e. has higher entropy) than ours and making MCMC-OP thus less confident about its predictions. However, MCMC-OP performs clearly worse in terms of MSE, suggesting that the median of the high entropy PPDs produced by this method do not provide a good point estimate.
Figure 12: Pairwise comparison of MSE values (lower is better) between PFN and MCMC.
C.3 Application: Extrapolation-based early stopping in model selection
In what follows, we further analyze and discuss our experiments described in Section 4.3.
C.3.1 Results on individual tasks
While Figure 5 shows the results of our early stopping experiments for each PD1 task, results for the three remaining benchmarks are summarized by averaging them across all tasks. Since this may hide task-dependent variation and outliers, we have a closer look at the performance on individual tasks. Figure 13 shows the results for each of 35 LCBench tasks. Here, we observe that relative performances vary per task. The aggressive Patience (k) (i.e., low  $k$ ) baselines perform best on some tasks, but fail on others. The LC-PFN based termination criteria consistently perform well on each task, obtaining a  $2 - 6 \times$  speed-up (w.r.t. the final performance of no-stop) on 30 of the 35 tasks, and no significant slow-down on the remaining five. Figure 14 shows the results for each of the 3 NAS-Bench-201 tasks. Here, we observe failure on all 3 tasks, where the degree of failure seems correlated with the complexity of the image classification task. Figure 15 shows the results for each of the 12 Taskset tasks. Here, the LC-PFN based termination criteria consistently perform well, obtaining  $2 - 6 \times$  speed-ups on all 12 tasks. We conclude that Figure 5 accurately reflects relative performances on all four benchmarks.
Figure 13: Average anytime regret (lower is better) obtained by all early stopping criteria for 40 run orderings for each of the 35 LCBench tasks.
Figure 14: Average anytime regret (lower is better) obtained by all early stopping criteria for 40 run orderings for each of the 3 NAS-Bench-201 tasks.
Figure 15: Average anytime regret (lower is better) obtained by all early stopping criteria for 40 run orderings for each of the 12 Taskset tasks.
C.3.2 Parameter sensitivity analysis
In our comparison in Figure 5, we considered two variants of the LC-PFN based termination criterion, varying the frequency at which it is applied, and found the fine-grained strategy, applying it every epoch, to perform best. However, many variants of this scheme exist, and in what follows we investigate the impact of some of our other choices on performance.
**Confidence level:** In our experiments, we terminate a training run if we are confident it will not improve upon the best model seen so far. Following prior art [Domhan et al., 2015], we adopted a confidence threshold of 0.95. Figure 16 (a) shows the anytime performance of variants using lower (0.90) and higher (0.99, 0.999) confidence levels. We observe that all of these perform relatively well, suggesting some degree of robustness. The speed-ups obtained using higher confidence levels are mostly lower (up to  $2 \times$ ), but more consistent. In particular, we only observe minimal slow-down on NAS-Bench-201 at confidence level 0.999, and higher confidence are also beneficial for the PD1 protein sequence (UniRef50) task.
Minimal cutoff: While we apply our predictive termination criterion every epoch, we do not apply it "as of the first epoch", but only after two observations are available ( $T \geq 2$ ). Since LC-PFN's predictions are conditioned on the partial curve only, it will predict the prior for  $T = 0$ . Beyond not allowing us to discriminate between runs, it introduces a potential failure mode for easy tasks (many training runs obtain models close to the theoretical optimum) and LC-PFN (unaware of task hardness) will terminate runs before they even began because it is very unlikely to obtain a better model under the prior. Figure 16 (b) shows the anytime performance of variants that apply the criterion as soon as one, three, four, or five observations are made. We find that a choice of one (instead of two) only negatively impacts performance on LCBench (containing some very easy tasks). A minimal cutoff of two works well on all benchmarks, with minor slow-downs for the higher minimal cutoffs.
C.4 Qualitative plots of learning curve extrapolation
To complement the quantitative evaluation in the main paper, we provide examples of extrapolations, at the four different cutoffs, for seven curves from the prior (Figure 17), LCBench (Figure 18), NAS-Bench-201 (Figure 19), Taskset (Figure 20), and PD1 (Figure 21). The plots show the median and two-sided  $90\%$  confidence interval of the PPDs inferred using the LC-PFN and MCMC-PP variant considered in Section 4.2. We observe that while the quality of the extrapolations produced by LC-PFN varies strongly, they are mostly logical, given the data observed and the prior used, and we find that the inferences of MCMC-PP are rarely preferable.
D Breakdown computational cost
Overall, reproducing all our experiments in the main paper requires approximately 163 CPU days and 60 GPU hours on our systems (GPU: NVIDIA (R) GeForce (R) RTX 2080, CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz). These costs break down as follows:
- Section 4.1 entails 80 CPU days to run all MCMC-PP variants on the 10000 sampled curves. The overall prediction time of LC-PFN is negligible (<1 hr for each variant). However, for LC-PFN, there is an initial training cost, which amounts to approximately 60 GPU hours to train all 27 LC-PFN variants.
- Section 4.2 requires 80 CPU days to run the MCMC algorithm on the considered real curve benchmarks.
- Section 4.3 involves a maximum of 3 CPU days to replicate the early stopping results on all the benchmarks for both LC-PFN and the baselines.
(a) Varying confidence levels  $(1 - \delta)$
Figure 16: Parameter sensitivity analysis of our fine-grained LC-PFN based termination criterion in terms of the average regret (lower is better).
Figure 17: Extrapolations of 7 different curves from the prior at 10, 20, 40, and 80 cutoff
Figure 18: Extrapolations of 7 different curves from LCBench at  $10\%$ ,  $20\%$ ,  $40\%$ , and  $80\%$  cutoff
Figure 19: Extrapolations of 7 different curves from NAS-Bench-201 at  $10\%$ ,  $20\%$ ,  $40\%$ , and  $80\%$  cutoff
Figure 20: Extrapolations of 7 different curves from Taskset at  $10\%$ ,  $20\%$ ,  $40\%$ , and  $80\%$  cutoff
Figure 21: Extrapolations of 7 different curves from PD1 at  $10\%$ ,  $20\%$ ,  $40\%$ , and  $80\%$  cutoff