Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
Authors: Dong Bok Lee
Aoxuan Silvia Zhang
Byungjoo Kim
Junhyeon Park
Steven Adriaensen
Juho Lee
Sung Ju Hwang
Hae Beom Lee
Venue: NeurIPS 2025 poster ()
Keywords: Cost-Sensitive
Bayesian Optimization
Multi-Fidelity HPO
PFNs
Transfer Learning
Primary area: deep_learning

Abstract: In this paper, we address the problem of cost-sensitive hyperparameter optimization (HPO) built upon freeze-thaw Bayesian optimization (BO). Specifically, we assume a scenario where users want to early-stop the HPO process when the expected performance improvement is not satisfactory with respect to the additional computational cost. Motivated by this scenario, we introduce \emph{utility} in the freeze-thaw framework, a function describing the trade-off between the cost and performance that can be estimated from the user's preference data. This utility function, combined with our novel acquisition function and stopping criterion, allows us to dynamically continue training the configuration that we expect to maximally improve the utility in the future, and also automatically stop the HPO process around the maximum utility. Further, we improve the sample efficiency of existing freeze-thaw methods with transfer learning to develop a specialized surrogate model for the cost-sensitive HPO problem. We validate our algorithm on established multi-fidelity HPO benchmarks and show that it outperforms all the previous freeze-thaw BO and transfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance.

Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
Dong Bok Lee'
Aoxuan Silvia Zhang
Byungjoo Kim"
Junhyeon Park !
Steven Adriaensen
Juho Lee' Sung Ju Hwang!
Hae Beom Lee
KAIST
Universiiyy of Freiburg DeepAuto.ai 'Korea University
markhi0kaist. @kaistaai ac. kr
haebeomlee@korea. ac.kn
Abstract
In this paper, we address the problem of cost-sensitive hyperparameter optimiza- tion (HPO) built upon freeze-thaw Bayesian optimization (BO). Specifically, we assume a scenario where users want to early-stop the HPO process when the ex- pected performance improvement is not satisfactory with respect to the additional computational cost. Motivated by this scenario, we introduce utility in the freeze- thaw framework, function describing the trade-off between the cost and perfor- mance that can be estimated from the user's s preference data. This utility function, combined with our novel acquisition function and stopping criterion, allows us dynamically continue training the configuration that we expect to maximally im- prove the utility in the future, and also automatically stop the HPO process around the maximum utility. Further, we improve the sample efficiency of existing freeze- thaw methods with transfer learning to develop specialized surrogate model for the cost-sensitive HPO problem. We validate our algorithm on established multi- fidelity HPO benchmarks and show that outperforms all the previous freeze- thaw BO and wrasfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance. Our code is publicly available at https: httr//gittti github..... com/db- LaeeadbLLeeCCBB
Introduction
Hyperparameter optimization [HPO; 22, 8, 48, 16, 34, 12] stands as a crucial challenge in the domain of deep learning, given its importance in achieving optimal empirical performance. Unfor- tunately, the field of HPO for deep learning remains relatively underexplored, with many practition- resorting to simple trial-and-error methods [8, 34]. Moreover, traditional black-box Bayesian optimization (BO) approaches for HPO [9, 48, 12] face limitations when applied to deep neural net- works due to the impracticality of evaluating convergence, each of which may take several days.
number of hyperparameter configurations until
Recently, multi-fidelity HPO [53, 34, 14, 4, 62, 3, 24, 44] has gained increasing attention to improve the sample efficiency of traditional black-box HPO. leverages lower-fide fewer-fidelity information (e.g., validation accuracies at fewer training epochs) to predict and optimize performance higher or full fidelity (e.g., validation accuracies at the last training epoch). Furthermore, unlike black-box HPO, multi-fidelity HPO dynamically selects hyperparameter configurations even before finishing single training run, demonstrating its ability of finding better configurations sample-efficiently
'Equal Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, one critical limitation of the conventional multi-fidelity HPO frameworks is their lack of awareness of the trade-off between the cost and performance. For example. given a limited amount of total credits, customers of cloud computing services (e.g., GCP, AWS, or Azure) can heavily penalize the cost of HPO relative to its performance to conserve credits for other tasks. A similar scenario applies to task manager users such as Slurm, who aim to optimize their allocated time within a computing instance. In such cases, users may prefer that the HPO process focuses on exploiting the current belief' about good hyperparameter configurations rather than exploring new ones, to efficiently consume their limited resources. However, existing methods [53, 34, 14, 4, 62, 3, 24, 44] generally do not consider this scenario, as they typically assume sufficiently large budget (e.g., total credits or allocated time) and aim to achieve the best performance on a validation set.
Utility: trade-off between cost and performance. Therefore, in this paper, we introduce more sophis- ticated notion of cost sensitivity for HPO. Specifi- cally, assume that users have their own prefer-
(iee,, Best val. performance so far)
1.0
0.9
0.8
0.7
Performance of BO
0,6
Maximum utility
ences regarding the trade-off between the cost and performance of HPO. We formalize this trade-off in utility function that represents user preferences and can be estimated from user preference data. It
80 trajectory
0.5
0,4
assigns higher values as costs decrease and perfor-
0.3
mance increases, and vice versa. Some users may
0.2
strongly penalize the budget spent on HPO, while
others may penalize it weakly or not at all, as in the conventional multi-fidelity HPO. We explicitly max- imize this utility by dynamically selecting hyperpa- rameter configurations expected to achieve the great- est improvement in the future and by automatically terminating the BO around the maximum utility instead of halting at an arbitrary target budget.
8
BO steps
Figure 1: A concept of user utilityy
Fig. illustrates the concept of utility: the red line shows the BO trajectory, and the blue asterisk marks the maximum utility (around 0.7) achieved at budget b*. This example heavily penalizes the additional cost, although extending the BO to the full budget B could yield better performance.
Solving this problem requires our method to have the following capabilities. Firstly, it should sup- port freeze-thaw BO [53, 44], advanced form of multi-fidelity BO that dynamically pauses (freezes) and resumes (thaws) hyperparameter configuration trainings based on future performance extrapolated from set of partially observed learning curves (LCs) with various configurations. Such efficient and fine-grained allocation of computational resources aligns well with the goal of achiev- HPO. Secondly, freeze-thaw
ing the best trade-off between cost and performance in BO requires that its surrogate function be capable of LC extrapolation [24, 2, 44]. In our case, this critical for making probabilistic inferences about future utilities, which guide the selection of the best configuration and enable precise early stopping of the HPO. Lastly, since users are assumed to prefer stopping HPO as carly as possible when performance saturates, LC extrapolation must be accurate even in the early stages of HPO. Therefore, it is essential to use transfer learning to maximize the sample efficiency of BO [3] and to prevent premature stopping.
Based on these criteria, we introduce our novel Cost-sensitive Freeze-thaw BO (CFBO), which effectively maximizes utility using the three components mentioned above. We first explain the notation and background on freeze-thaw BO and Prior-Fitted Networks [PFNs; 39, 2, 44] for LC extrapolation ($2). We define the utility function and explain how to estimate it from user pref- erence data² ($3.1). We describe the acquisition function and the stopping criterion specifically developed for our problem setting, explaining how they achieve a good trade-off between cost and performance ($3.2 and $3.3). We show how to train a PFN using existing real-world LC datasets to create a sample-efficient in-context surrogate function for freeze-thaw BO, which effectively cap- tures correlations between different hyperparameter configurations ($3.4). Finally, we empirically demonstrate the superiority of CFBO on diverse set of utility functions, three multi-fidelity HPÒ benchmarks, and real-world object detection LC dataset we collected, showing that it significantly outperforms relevant baselines, including transfer-BO ($4).
We defer the discussion of related work, e.g., early-stopping BO methods, to Appendix 3 'Some users may already have an exact form of their utility function, but for others, we provide method quantify it based on their preference data.
2

---

We summarize our contributions and findings as follows:
We propose new problem formulation, cost-sensitive multi-fidelity HPO, which focuses max- imizing the trade-off between cost and performance (i.e., utility) as defined by users, rather than optimizing asymptotic validation performance.
We introduce novel acquisition function and stopping criterion specifically designed for this problem formulation, incorporating transfer learning to enhance in-context LC extrapolation.
We extensively validate the effectiveness of method across diverse cost-sensitive multi-fidelity HPO scenarios using three popular LC benchmarks.
2 Background and Related Work
Notation. Let X' = (.1,...,, .Ny denote the set of hyperparameter configurations, where Hin Rd and N is the number of configurations [62, 24, 44]. Let E [7] = (1....,T) T) denote the training epochs, with T being the last epoch, and yn,1:T (Un, 1: ......... -Y,.T) T)) E 1]" the measure of model performance to be maximized, e.g., learning curve (LC) of validation accuracies with Tn- We now introduce the notation for multi-fidelity BO. fn < T denotes the last observed epoch of In if the model performance of is partially observed Ii,,,: Let B denote the total budget spent during BO, and yn E [0, 1] the best cumulative performance achieved up to budget b E [B]. respectively. See Tab. 4 in Appendix B for summary of the notation used throughout this paper.
Freeze-thaw BO. Freeze-thaw BO [53] is an advanced form of multi-fidelity BO, which aims to maximize the best cumulative performance ýn by efficiently allocating the limited total budget B. Assuming that allocate one budget unit (ie.. epoch) for each freezee-thr thaw BO step, allows us to dynamically select and evaluate the best hyperparameter configuration Zn- with n" E [N] denoting the corresponding index, while pausing the evaluation of the previously selected configuration. Specifically, given the context C = ((x,t,)) t, v)), which represents a set of partial (or full) LCs collected up to a specific BO step (i.e., history), that maximizes a predefined acquisition function (e.g., the expected improvement [38].. We then evaluate the selected configuration nn- for one additional budget unit (i.e., one epoch) and observe yar ... +I- Next, we update the history C with (In*, In- + 1, 1, -.n=...+++.. The cumulative best validation performance dh is updated accordingly. This process repeated for B steps. Please Alg. for the pseudocode (except for the blue parts).
select one configuration Ina E X
+1
PFNs for LC extrapolation. Freeze-thaw BO usually requires the ability to extrapolate LCs to compute acquisition functions [62, 24, 44). Among many plausible options, in this paper, we use Prior-data Fitted Networks [PFNs; 39] for the I.C extrapolation. PFNs are an in-context Bayesian in- ference method based on Transformer [54] and show good performance in LC extrapolation [2, 44] without computationally expensive online retraining [62, 24J. Specifically, after training, they allow us approximate the posterior predictive distribution (PPD) with single forward pass: po (,l f, C) ~ p(y jr, C, where pe is the approximate PPD parameterized by f. This is achieved by minimizing the cross- cross-entropy for predicting the hold-out example's 's label y, given x, t,, and C:
[(0)= Cxp(() [- log (y (t,,,,,,, C)],
where p(D) is prior from which we can sample infinitely many synthetic training data. We defer the architectural and training details to Appendix E.
Related work. We defer the discussion of related work on (1) multi-fidelity HPO, (2) freeze- thaw BO, (3) LC extrapolation, (4) transfer-BO, (5) cost-sensitive HPO, (6) early stopping BO, (7) BO with user preferences, and (8) neural processes to Appendix A.
3 Method: Cost-sensitive Freeze-thaw Bayesian Optimization (CFBO)
We now introduce our method, Cost-sensitive Freeze-Thaw Bayesian Optimization (CFBO), sum- marized in Alg. where the blue components indicate the parts specific our approach.
3.1 Utility: Trade-off between Cost and Performance
Utility function. A utility function U (b, (b) E [B] [0, 1] + [0. 1] describes the trade-off between the budget b and the best cumulative performance yt. Its value decreases with increasing
3

---

b and increases with increasing Vo. We assume that the utility is given by the users. For example, it can be simply defined as (b, (b) = Uh a ( " where 0 S S 1 is penalty coefficient and c = 1, 2, or 0.5 for linear, quadratic, or square-root utility, respectively. Furthermore, the total budget limit can be modeled by setting (bb, 0b) = oo if b > B otherwise in-
Utility estimation. It is often challenging for users to quantify their preference on the trade-off. We therefore propose to use the Bradley-Terry model [10] for utility estimation:
exp(U(b, 5n) /T)
p(U(b. U(V (6-) exp(U(b, (o) /T) + exp(U(V, (,)/
where U is the user utility we want to estimate, and T is a temperature hyperparameter. Eq. describes the probability that the user prefers (6) to (V, Th-)) in terms of utility U. Following the preference learning literature [6], we can collect user preference data by asking users which point they prefer: (1) sampling a pair of points (b, ūb), (b', kg.), (2) labeling user preference la- bel y> E {0,1] as 1 if (b, (o.) > U(b', otherwise O, and (3) constructing a dataset Dy == {((b, m6), (6', 1.)) y>)}. We then optimize the parameter of the utility function (e.g., the penalty coefficient a) by minimizing the binary cross-entropy loss f(z, v) == -y -ylog x (1 - y) log(1 a):
Σ
(U (b,06) > U U(' (b) リン)
(3)
Duul
(b.56).666. 1. E.U
An example of utility estimation. Fig. 2 illustrates an exam-
Cumulative Best (%)
ple of our utility estimation process. Here, we assume a sim- ulated user who consistently favors outcomes that improve upon
...
the baseline freeze-thaw HPO trajectory, e.g., ifBO [44]. We first
run ifBO for up to 300 budgets and obtain wb E [B], cs. where
9.
Üb denotes the best cumulative performance of BO up to bud- get b. For each b E
0.2
we then sample y/ (up)
320
Uniform((, ün)), and record the preference (b, (up)) > U(b dddww)). We exclude the initial bud- gets b E (1.....50) to prevent the utility model from overfitting to the early steep improvements. This simulates a user who always prefers outcomes above the base- line trajectory. The solid black line in Fig. 2 represents the ifBO trajectory (jn)). the dotted blue lines indicate the estimated utility, and the yellow asterisk marks the maximum utility. We observe that the utility reaches its maximum when the improvement becomes saturated.
Uniformu(ss: 1) and yb
((dww))
Total Budget Spent (b)
Figure 2: An example of utility function estimation.
In practice. We recommend providing a small library of functional forms-e.g. linear (c = 1), quadratic (c = 2), square-root (c 0.5), or staircase- and fitting the utility via preference learning using Eqs. 2 and 3. Assuming the true utility of users follows one of these forms, Appendix C shows that it can recovered with only a few pairwise comparisons (e.g., 30 pairs).
3.2 Acquisition
EI-based acquisition function. We define the acquisition function A(n; ()) for maximizing utility based on the Expected Improvement (EI) method [38]:
Utility
A(n;U) = max Eyn, ~p [ At, Do4 - Upl") AVE(T
G...
(4)
where [x]+ max(x 0). In Eq. 4, we first extrapolate Da.- == (yn,tn+at At E - the remaining part of the learning curve (LC) associated with In, using po. Then, we compute the corresponding cumulative best performances Figure 3: An illustration of A in Eq. 4. (jo-s: Al E [T- - [.]]. According to the definition in $2, jtt is computed by taking the maximum of the previous best performance ÿs and the newly ex- ,.n.t.,77 +T. Based on the updated budget + Al and the corresponding performance j6+Ai, we compute the utility and its expected improvement over
b
6.
B0
steps
trapolated validation performances
4

---

the previous utillity Up. The expectation is evaluated over the distribution of LC extrapolations (pe), using Monte Carlo estimation (detailed in Appendix F). Finally, the acquisition A(n; U') for each configuration index is determined by selecting the hest increment At E [T [..] that maximizes
the expected improvement. We then choose the best index over configurations n' that maximizes A(n; U), i.e., n* arg max, (nn; U). Assuming three configurations x,, and r3, Fig. 3 illus- trates the selection of configuration 23.
Differences from existing EI. The main differences between our acquisition fune- tion in Eq. 4 and the usual El-based acqui- sition are twofold. First, instead of max- imizing the expected improvement of the validation performance y, we maximize the El of utility. Second, rather than fix- ing the target epoch for evaluating the ac- quisition to the last epoch T [24] or using random increment [44], we dynamically select the best target epoch that is expected
Algorithm 1 Cost-Sensitive Freeze-thar BO. Blue parts correspond to specifics of our method.
1:
Input: X hyperparameter configuration space
P6 LC extrapolator with transfer learning
3:
A : acquisition function
4:
B total HPO budget
5:
U utility function
6:
or 3.7: threshold hyperparameters
7: 50 <
C 4- 0, 4,..., + 0,U, +- 0, Rs 0
8: for b ,,..., do
9:
n <- arg maxx A(n;))
Acquisition Eq. b Pps in Eq. 4
to yield the highest improvement in utility.
10:
5, + BetaCDF
These aspects enable our BO framework to carefully select configurations at each suggestion step, aiming to achieve the
11:
R. 8 or d, then break end if
12:
P Stopping criterion
13:
14:
Evaluate +p-++ with Fn.
best trade-off between the cost and per-
15:
C CU (ID0, tu- + 1, u +.++1]3
formance of the HPO process. Specifi-
16:
db f- max((i-11 Ho- Unn.,n +1)+
cally, the acquisition function initially fa- vors configurations that are expected to yield strong asymptotic validation perfor-
17:
bn <- la +1
18:
Up <- Ub, V6)
19:
R. + Unn Tanc-Oaa Unin
Emex Up
D Estimated regret Eq.
mances y. However, as the BO pro-
20:
end for
gresses, the acquisition function gradually
-" 21: Output: The model trained with x" up to th epoch
becomes greedier as the performance sat- urates and the associated cost begins dominate the utility function (empirically
s.t. (x" tt",,",, y²) = arg (t) y
illustrated in Figs. 8a to 8c of $4.3). Consequently, the acquisition function shifts from exploration to prooitatioonppiiiiiiiinn the current configurations over selecting new ones to maximize short- term performance.
Utility is irreversible. Note that Up in Eq. 4, the reference value for EI, is not the greatest utility achieved far (corresponds to fr in the typical EI A(x) E,ff - f)]+). Instead, it is set to the utility value achieved most recently (i.e., Up in line 18 of Alg. 1), as the computational budgets spent are irreversible. This also contrasts with typical El-based BO settings, where all previous evaluations remain meaningful, allowing the reference value can be set to the maximum among them. Consequently, Up can either increase or decrease during the BO process.
3.3 Stopping Criterion
Regret-based criterion. The next question is how to properly stop the HPO around the maximum utility. We propose stopping when the following criterion is satisfied:
Ünix Up > 8.
Rs
(5)
Ünax Umin
In Eq. 5, Up is the utility value at the previous step b 1, Ümmx is defined as the maximum utility value seen up to the previous (6 1) step, and Unin (B, Ø1). The role of Ümax and Ümin is roughly estimate the maximum and minimum utility achievable over the course BO, respectively. îG e [0, can be seen as roughly estimated normalized regret at the current step b. This regret-based criterion terminates the BO process once the estimated regret exceeds a predefined threshold d e J0, 1]. The intuition is pessimistic: if the current utility regret is already high, it is unlikely that further optimization will yield a significant improvement, and thus the BO process is terminated.
Adaptive threshold. We can fix the threshold as a hyperparameter in Eq. 5 (e.g., baselines); however, this approach does not account for possibility of potential improvement in the future. To
5

---

address this, we propose an adaptive threshold based on the probability of improvement [PI: 38]:
db- BetaCDF((6: 3)7, 8,7> 0, E py [1 (U (6 + (o, > Up)
(6)
pb AIE
max
(7)
Here, BetaCDF is the cumulative distribution function (CDF) of the Beta distribution, and 1 is the indicator function. The PI ps in Eq. represents the probability that the selected configuration In- using Eq. 4 improves Up in some future BO step. Intuitively, we aim to defer termination as ps increases and vice versa. This behavior is incorporated into Eq. 6-as as pa increases, the adaptive threshold 6ú also increases because BetaCDF B, B)7 is monotonically increasing function in [0, 1]. Consequently, according to Eq. 5, there is less motivation terminate the BO process when Ps. is high. For our CFBO, we use the adaptive threshold 6, instead of the fixed threshold 8 in Eq. 5.
Role of y and 3. Fig. 4a shows that 7 controls ōb at ps 0,5. For example, if we set 7 = logo_s 0.2, 6, becomes 0.2 when Ps 0.5 regard- less of B. By controlling y, we can set the threshold to proper value when the is uncertain (p& = 0.5).
1.0
1.0
exp110)
BetaCDFloui .y
8 expp3t
a.n
0.8
R-exp121
f-expt31
V-a loges 0.5
P expto)
0.6
3.8
0.6
8-espi-11
8-expt-2 21
ya yallogs 0.6
8-cxpi-31
0.4
y Iegegex
0.4
8-eepi-10)
D.1
Fig. 4b illustrates the effect of B. As + 0, 6, becomes horizontal, fix- ing 6 0.2 regardless of ps- This results in ignoring the PI ps- In contrast, as 3 + co, ôg becomes vertical, causing ò, to take binary values, either 0 or 1, depending on whether ps > 0.5 or not. This corre- sponds to a purely PI-based crite- rion. Thus, 3 provides smooth interpolation that controls the degree to which the PI ps in Eq. 7 is incorporated into the adaptive threshold 8, in Eq. and the stopping criterion in Eq. 5.
6
y 100.502
loges D.1
0.5
0.0
0.0
Pb
(a) Varying Y
(b) Varying /
Figure 4: (a) 8 vs. ps for varying y with 3 exp( 1).
(b) dn vs. pa for varying 3 with y logu.s 0.2.
3.4 Transfer Learning with L.C mixup
Motivation. Since users may want to early-stop the BO process, it crucial to ensure an accurate LC extrapolation to prevent premature stopping during the early stages of BO. To address this, we propose the use of transfer learning to maximize the sample efficiency of LC extrapolators.
As discussed in $2, among various plausible options, we employ PFNs [39] for LC extrapolation. Regarding the network architecture and training objective, we primarily follow ifBO [44], with more details deferred to Appendix E. A significant challenge in using PFNs for our purpose is that PFNs require relatively large Transformer architectures and massive amounts of training examples for strong generalization [2], making it risky to train them on a limited data.
LC Mixup. To overcome these challenges, we propose novel transfer learning approach using the mixup strategy [66]. Assume that we have M different training LC datasets and the correspond-
ing M sets of LCs collected from N hyperparameter configurations. Let f(mm) ( ..... (m)) be a -ddimensional vector representing the validation performances of the m-th dataset and the n-th configuration, forming complete LC of length T. Define the matrix ((m) = [!(=)) :...; ((m))
N as the stack of these LCS. To augment LCC,, we propose two consecutive mixup strategies across datasets and configurations:
Dataset: Sample new LC dataset as L' = ((m) + +11 (1- (1->x))((m)) (m)) where A Uniform (0, 1) and m, m' E [M].
Configuration: Sample new configuration x' = \2 + (1 Xg \2) and its corresponding LC I' X2ln + (1 Aa) where I, and b,, denote the n-th and -h rows of L', respectively, A2 Uniform(0, 1), and n, E [N].
Using this approach, we can sample infinitely many training examples (o', by interpolating between LCS, resulting in a robust L.C. extrapolator with reduced overfitting.

---

4 Experiments
We next empirically validate the proposed method on various cost-sensitive multi- multi-fidelity HPO set- tings. Our code is publicly available at https: /tt/t/gttttb...// com Leee/d
4.1 Experimental Setups
Datasets. We evaluate CFBO on three standard LC benchmarks. LCBench [67]:
Table I: Dataset overview.
Dataset
dz
[x]
T
Train
Test
contains learning curves of MLPs trained on multiple tabular datasets, TaskSet [37]:
LCBench [67]
7 2,000
51
20 datasets
15 datasets
TaskSet [37]
8 1,000
50
tasks
tasks
provides diverse optimization tasks across domains; we focus on 30 NLP tasks (text classification and language modeling), and PD1 [56]: includes learning curves of modern neu- ral architectures, such as Transformers, trained on large-scale datasets (CIFAR-10/100 [31], Ima- geNet [45], and bioinformatics corpora). We split each benchmark into disjoint training and test tasks for transfer learning of LC extrapolators ps. Detailed dataset statisties are summarized in Tab. I, and additional descriptions are provided in Appendix D.
PDI [56]
240
50
16 tasks
tasks
Baselines without transfer learning. We compare CFBO with a wide range of multi-fidelity HPO methods. Random Search [8] sequentially samples configurations at random. We also evaluate two Hyperband [34] variants: BOHB [14], which replaces random sampling with BO, and DEHB [4], which integrates evolutionary strategies for knowledge transfer. Among recent multi-fidelia BO methods, we include DyHPO [62], which combines deep kernel Gaussian Process [GP; 58] with greedy short-horizon LC extrapolation; DPL [24], which fits power-law functions with ensemble modeling; and ifBO [44], PFN-based [39] freeze-thaw BO method using Pl acquisition at random future epochs. For a fair comparison, we use non-transfer (NT) variant, CFBO-NT, with the ICC extrapolator of ifBO: excludes transfer-learning with L.C mixup in $3.4, but retains A( (ef. Eq. in $3.2) and stopping criterion with the adaptive threshold 6, (ef. Eqs. 5-7 in $3.3).
Baselines with transfer learning. Quick-Tume [3] corresponds to the transfer-learning version of DyHPO, trained the same LC datasets as extrapolator. FSBO [60] is a black-box transfer-BO method that uses the same LC datasets to train a deep kernel GP surrogate. The both do not use the LC mixup in $3.4. The key difference from Quick-Tune lies in the prediction target: FSBO predicts the final-epoch performance, whereas Quick- extrapolates the next-epoch performance.
Implementation details for CFBO and the baselines provided in Appendix F.
Utility function. It is possible to manually collect user preference data and estimate the corre- sponding utility function ($3.1). In our experiments, however, we simplify this process by using linear, quadratic or square root function, i.e., (b, 56) D6- ( where c E {1,2, 0.5), with 2-5 2 2,, 22-3, 2 2- Notably, setting a = 0 removes any penalty associated with the budget b spent during HPO, causing the HPO process to continue until the final step B, as in the conventional multi-fidelig HPO setup.
Threshold hyperparameters 5, 7, and G. For the baselines, set the threshold 5 = 0.2 in Eq. 5, as it performs well on the training split. For CFBO, we also use ry = logo.5 0.2, which corresponds to the adaptive threshold & 0.2 when ps = 0.5, to ensure a fair comparison with the baselines. We use 3 = exp(- for all experiments in this paper, except for the ablation study in Fig. 8d.
Evaluation protocol. We set the maximum budget" as B = 300 and report:
Umax - Up
R
E j0, 1], where Umax == max (t, yn, and Unin Ro min U(B In,1), n,t
(8)
Umax
Umin
n
which is similar to Eq. 5, but uses the true Umax and Umin- Here, Up is the utility termination (e.g.. after break in line of Alg. 1). Umix == max,,t (1, yn.e) denotes the maximum utility achievable by single optimal configuration. Since exact computation of Umin is intractable, we approximate it as min, UB, 1n. where yn,1 is the first epoch performance. We report tea of normalized regrets over 5 runs (30 for Random, BOHB, and DEHB) and the average rank across all tasks.
Modiiiidd version without pretrained model selection or wall-time balancing. WWile prior work [62, 44] uses B 1,000, we set B 300 to better show the benefit of transfer learning.
7

---

Random inBO
BOHB CFBO-NT
DEHB Quick-Tune!
DyHPO FSBO
DPL
CFBO
LCBench
Taskset
PD1
Normalized Regret
10-1
10-2
100
200
300
0
100
200
300
100
200
300
Total Epochs Spent
Figure 5: Results on conventional multi-fidelityy HPO (a 0). X and O markers denote non-transfer and
transfer learning methods, respectively.
LCBench Linear (c=1)
Taskset Linear (c=1)
PD1 Linear (c=1)
PD1 Square (c=2)
PD1 Square-Root (c=0.5)
Normalized Regret
10-
10
Rank
2-
2
2-
2
2
2-
2
2
2
2 2
2
2
2
2
aE 9,2
Figure 6: Results on cost-sensitive multi-fidelity HPO (c E (1,2,0.5) and a E (2 12-6,21 2-2.2..5
2-
2 2-2)))
2-
2
2
2
Normalized Regrot
10"
20-1
1.0
200
300
200
100
200
300
Total Epachs Spont
Figure 7: Visualization of the normalized regrets on PD1 benchmark. See Appendix G for the others.
4.2 Main Results
Results conventional multi-fidelity HPO (a 0). We first validate the effectiveness of the proposed CFBO in conventional multi-fidelity HPO setups (a = 0). The results are presented in Fig. 5, where non-transfer learning and transfer learning methods markers, respectively. Notably, although our methods are not specifically designed for conventional setups (a 0), they achieve results comparable the baselines. For example, CFBO-NT, the non-transfer learning variant, performs on par with the best non-transfer baseline, ifBO. Similarly, CFBO achieves comparable performance to transfer-BO baselines, showing better normalized regret on TaskSet and PDI, and slightly higher regret on LCBench compared to FSBO.
denoted by X and O
Results on cost-sensitive multi-fidelity HPO (a > 0). In real-world scenarios, utility functions can take various forms and may be defined differently by different users. To evaluate the effec- tiveness of CFBO under such realistic cost-sensitive settings, we conduct experiments on vari- ous utility functions, including linear (c = 1), square (c = 2), and square-root (c = 0.5), with E (2-6,2-- -,,,, 2-2 2-3 2- 2-2). Fig. 6 presents the normalized regret (first row) and rank (second row) for each method. First, CFBO-NT not only outperforms all non-transfer learning base- lines (denoted with the X markers) but also surpasses strong transfer-learning baseline (i.e., FSBO) in several settings e.g., most PDI cases in terms of rank-excepp for c and a = 2-6 CFBO exceeds all relevant baselines in most settings, except TaskSet with 1 and 2 demonstrating the robustness of our method to variations in the utility function.
8

---

CFBO
CF8O (a=2-1)
30-1
0.8
DyHPO
06
CFBC (a=2-1)
Normalzed Rugnet
DPL
Fres Rajo
CF80 (a=0)
Nomalized
1.9
QuickTTnee
DyHPO la=?" 2-
LCBecch
CFBO DyHPO
la=2
laskSet
Quick Tune" la= 2-1
PD1
10
0.2
Average
Quick-Tune!
1oz Hi
236
250
302
0.0
Tor
150
Total Epochs Spent
Toral Epochs Sport
HP Index Sorted by Freq. Rabio
log
(a) Optimal regret of 2,--
(b) At
(c) Top- distribution
(d) Varying |
Figure 8: (a) Optimal regret of the selected configuration Ji-" at future step b + At. (b) Future At in (a) which the optimal regret of z,- achieved. (c) Distribution of the top-10 hyperparameter configurations selected during BO. (d) Normalized regrets with varying 33 of BetaCDF( 3) in Eq. ( logo.s 0.2).
Visualization. We also visualize the normalized regret during the BO process using task from the PDI benchmark in Fig. 7. Asterisks indicate stopping points, while dotted lines represent the normalized regret achievable without stopping. Although our CFBO struggles to stop near the max- imum utility (minimum regret) during the BO process when the penalty is weak (a E (2-6, 2 2-5)), 00-6,-2
the regrets obtained at those points are still lower than the baselines. In addition, CFBO success- fully stops almost at the optimum when the penalty is stronger (a E (2-1, 022-,,23,22- 2-
Algorithm runtime. In Tab. 2, we report the av- erage wall-clock time per BO step over five runs for DPL [24], ifBO [44], and CFBO. All mcasurements conducted on single A 100 GPU. While ifBO is
Table 2: Wall-clock time (seconds) per BO step on LCBench, TaskSet, and PDI.
Method
LCBench
TaskSet
PD1
the most efficient method, the difference between the
DPL [24]
0.65.om
0.64aom
0.63
ilBO [44]
0.58
0.30
0.082am
wall-clock times of ifBO and CFBO is negligible, as neural network training dominates the total wall-
CFBO (ours)
1.52
0.78
0.23
clock time in HPO (e.g., 90 seconds for ResNet-50 [20] per training epoch in CIFAR-10/100 [31]).
Real-world object detection dataset. We evaluate CFBO on a real-world object detection dataset and observe that it achieves the best performance (in terms of both normalized regret and rank)
among all baselines. Details and additional experimental results are provided in Appendix G.
4.3 Analysis
Ablation study. To evaluate the effectiveness of each component, we conduct ablation studies on the proposed (1) stopping criterion ($3.3), (2) acquisition function ($3.2), and (3) transfer learning with the LC mixup ($3.4) on the PDI benchmark. For (1) the stopping criterion, we compare two approaches: the smoothly mixed criterion with our adaptive threshold (6, v/ and the only -bared criterion with a fixed threshold 5 (6, X), which is used by the baselines. For (2) the acquisition function, we use either our proposed approach (Eq. 4; A v) the acquisition function of ifBO [44, A x]. For (3) transfer learning, we compare our surrogate trained with the proposed mixup strategy (T. V) against the ifBO [44] which is only trained on synthetic prior data (T. X).
In Tab. 3, comparing the first and sec- ond rows, see that transfer learn- ing is relatively more effective in con- ventional settings (a 0) than in cost-sensitive settings (a > 0). Com- paring between the second and third rows shows that our acquisition func- tion enhances performance more in cost-sensitive settings. Lastly, com- paring between the third and last row shows that our adaptive threshold significantly improves per- formance in cost-sensitive settings.
Table 3: Results of ablation studies using the PD1 benchmark. For better readability, we multiply 100 to normalized regrets.
66
T.
2-6
2-5
2-4
2-3
2
x
0.8
2.3
3.7
6.0_ne
9.8gm1
15.2
X
0.2s00
1.7401
3.2=a1
5.9203
9.4
11.7=04
X
0.2
1.5100
2.6
4.5_mm
6.9
8.5
0.2
1.0
1.3
0.9_no
1.1
1.7
Analysis on acquisition function ($3.2). To better understand the sources of improvement, we analyze the configurations selected by each method. Specifically, for each BO step b, we run the configuration In, currently chosen at step b up to its final epoch T and compute two metrics: the minimum ground-truth regret R achievable at some future step b + At (Fig. 8a) and the correspond- ing optimal increment At (Fig. 8b). In Fig. 8a, our method achieves significantly lower minimum regret compared to baselines. This indicates that our acquisition function Eq. 4 performs as in- lended, selecting at each BO step the configuration expected to maximally improve utility in the

---

future. Fig. 8b shows that the configurations selected by our method initially correspond to larger At values (i.e., non-greedy behavior), but progressively shift smaller At values (i.e. greedy be- havior). This transition occurs because, as the BO progresses, the performance improvements begin to saturate, causing the cost of BO to dominate the trade-off. Consequently, At becomes smaller, eventually approaching zero. These behaviors are not prominently observed in the baselines.
Fig. 8c shows the distribution of the top-10 most frequently selected configurations during BO. As expected, our method tends to focus on a smaller subset of configurations, optimizing for short- term performance, particularly under stronger penalties (i.e., larger a). The baselines exhibit exces- sive exploration of configurations, even under the strongest penalty (i.e. a = 2 2)
Analysis on stopping criterion ($3.3). Next, we analyze the effectiveness of our stopping crite- rion with the adaptive threshold ó, defined in Eqs. 5-7. Fig. 8d presents the normalized regret in various values of 3, the mixing coefficient between the two extreme stopping criteria discussed in $3.3. log B - 10 corresponds to the baseline criterion (the fixed threshold ó = 0.2), which relics solely on the estimated normalized regret in Eq. 5, log 3 + -10 corresponds to hard thresholding based exclusively on the PI ps in Eq. 7. The results show that the optimal criterion is achieved with smooth balance between the two (3 = e e demonstrating the importance of incorporating the possibility of further improvement through pb-
To further assess the significance of the stopping criterion in cost-sensitive multi- fidelity HPO, we quantify the improve- ment of normalized regrets obtained by an optimal stopping criterion for each method. The optimal stopping criterion assumes that the BO process is stopped at the point of maximum utility in the BO trajectory for each method. Fig. 9 presents the average improvement on the PD1 benchmark across all a E 92-9,2-9.2-1,2-3,33
Fixed Threshold
Adaplive Threshold
20.00
6 0.2
6.00
0.00
0.00
0.00
fernoor
DyNed
Quick
1560
ure
Figure 9: Improvement with the optimal stopping criterion.
The improvement is smaller when using the adaptive threshold 5, in Eq. 6; ie., our adaptive stopping criterion closely approximates the optimal stopping point, leaving minimal room for improvement and demonstrating its effectiveness.
5 Conclusion
In this paper, we present cost-sensitive freeze-thaw Bayesian optimization to improve the efficiency of hyperparameter optimization (HPO). Assuming that users aim early-stop HPO when the utility saturates, we introduced a novel acquisition function and stopping criterion specifically tailored to this problem setup. Additionally, we proposed novel transfer learning method for training sample-efficient in-context learning curve (LCC) extrapolator. Our empirical evaluation demonstrated the effectiveness of our approach compared to existing multi-fidelity HPO and transfer-BO methods.
Limitations and future work. We identify three main limitations of our study and outline poten- tial directions for future research as follows:
We focus pose-based HPO, i.e., |x|| N, which may not fully reflect real-world applications.
To enable continuous optimization over v,, E Rd,, common approach [48] is to perform grudient ascent on the hyperparameter configuration Fn to maximize the acquisition function A. In this case. A must be differentiable with respect to Ix, Unfortunately, our acquisition function A(:; U) in Eq. 4 does not satisfy this property, since the sampling from py (for MC estimation) is not differentiable. Extending CFBO to this continuous setting remains an interesting direction.
In $3.4, we explore transfer learning by training Frior-Data Fitted Networks [39] on mixed LC datasets. Recently, Tune-Tables [15] have demonstrated strong performance in fine-tuning PFNs on tabular datasets through context optimization. Since transfer learning is not only a key com- ponent of our work but also widely recognized in the literature [60, 3]. integrating the LC mixup with the context optimization approach of Tune-Tablls presents promising direction.
Our work assumes that users care only about the trade-off within a single task. In worldworld scenarios, however, users (e.g., cloud service customers) may also consider future budgets or the costs of multiple tasks running in parallel. Developing a system that accounts for these more complex and realistic settings would be valuable direction.
10

---

Acknowledgement
We express our sincere gratitude to the anonymous reviewers (uqhd, AJQp, Ly5J, and Vxb6) for their valuable feedback and efforts in helping us improve this paper.
Funding. This work was supported by Institute for Information communications Technology
Planning & Exaluationn(ITPP) grant funded by the Korea governmen((MSSI)) (RS-2019-11190075. Artificial Intelligence Graduate School PAISra No.RS-2022-11220713 Meta-learning Applicable to Real-world Problems), IITP with grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collab- orative Research (No. RS-2024-00469458 & RS- 25-2024-00509279), National Research Foundation of Korea (NRF) grant funded by MSIT (No. RS- 20--222000225255 and Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD).
References
[1] Majid Abdolshah, Alistair Shilton, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Cost- aware multi-objective bayesian optimisation. arXiv preprint 1oXiv:1909.03600, 2019.
[2] Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. Advances in Neural Information Processing Systems, 36, 2023.
[3] Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, and Josif Grabocka. Quick-tune: Quickly learning which pretrained model to finetune and how. The Twelfth International Conference Learning Representations, 2023.
[4] Noor Awad, Neeratyoy Mallik, and Frank Hutter. Dehb: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2147- 2153. International Joint Conferences Artificial Intelligence Organization, 2021. doi: 10. 24963/jjcai.2221122 URL https: //doi. org/10 10. 24963/ ijcai. 2021/296. Main Track.
[5] Tianyi Bai, Yang Li, Yu Shen, Xinyi Zhang, Wentao Zhang, and Bin Cui. Transfer learning for bayesian optimization: A survey. arXiv preprint 2302v 2023.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.058: 2022.
[7] Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architec- ture search using performance prediction. arXiv preprint 1rXiv::705..11 2017.
[8] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization Journal of machine learning research, 13(2), 2012.
[9] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.
[10] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4)3324-455, 1952.
[11] Floriana Ciaglia, Francesco Saverio Zuppichini. Paul Guerrie, Mark McQuade, and Jacob So- lawetz. Roboflow 100: A rich, multi-domain object detection benchmark. arXiv preprint
arXiv:2211.13523. 2022.
[12] Alexander I Gowen-Rivers Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. Hebo: Pushing the limits of sample-efficient typee-parameter optimisation. Journal of Artificial In- telligence Research, -111299--34,, 2022.

---

[13] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hy- perparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence, 2015.
[14] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International conference on machine learning, pages 1437-1446.
PMLR, 2018.
[15] Benjamin Feuer, Robin T' Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. Advances in Neural Information Processing Systems, 37:83430- 83464, 2024.
[16] Luca Franceschi, Michele Donini. Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-base based hyperparameter optimization. In International Conference on Machine Learning, pages 1 166--1173 PMLR, 2017.
[17] Matilde Gargiani, Aaron Klein, Stefan Falkner, and Frank Hutter. Probabilistic rollouts for learning curve extrapolation across hyperparameter settings. arXiv preprint urXiv:1910.04522 2019.
[18] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Es- lami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv: 02222, 2018.
[19] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and David Sculley. Google vizier: A service for black-hox optimization. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, pages 1487-1495, 2017.
[20] Kaiming Hc, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pages 770-778, 2016.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 11Xiv:1606.08415, 2016.
[22] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model- based optimiza- tion for general algorithm configuration. In Learning and Intelligent Optimization: 5th In- ternational Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5, pages 507-523. Springer, 2011.
[23] Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, and Luigi Nardi. Api bo: Augmenting acquisition functions with user beliefs for bayesian optimization. In International Conference on Learning Representations, 2021.
[24] Arlind Kadra, Maciej Janowski, Martin Wistuba, and Josif Grabocka. Scaling laws for hy- perparameter optimization. In seieth- Conference on Neural Information Processing Systems, 2023.
[25] Kirthevasan Kandasamy, Gautam Dasarathy, Barnabas Poczos, and Jeff Schneider. The multi- fidelity multi-armed bandit. Advances neural information processing systems, 29, 2016.
[26] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. Multi- fidelity bayesian optimisation with continuous approximations. In International conference on machine learning, pages 1799-1808. PMLR, 2017.
[27] Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In International conference on machine learning, pages 1238-1246. PMLR, 2013.
[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14l..112... 2014.
[29] Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics, pages 528-536. PMLR, 2017.
12

---

[30] Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In International conference on learning representa- lions; 2017.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] Eric Hans Lee. Valerio Perrone, Cedric Archambeau, and Matthias Seeger. bayesian optimization. arXiv preprint arXiv:2003. 2020.
Cost-aware
[33] Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. system for massively parallel hy hyperparameter tuning. Proceedings of Machine Learning and Systems,
2020.
[34] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research,
52, 2018.
[35] Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, An- dreas Krause, Matthias Seeger, and Cedric Archambeau. Automatic termination for hyper- parameter optimization. In International Conference on Automated Machine Learning, pages 7-1. PMLR, 2022.
[36] Neeratyoy Mallik, Edward Bergman, Carl Hvarfner. Danny Stoll, Maciej Janowski, Marius Lindauer, Luigi Nardi, and Frank Hutter. Priorband: Practical hyperparameter optimization in the age of deep learning. Advances in Neural Information Processing Systems, 36, 2024.
[37] Luke Metz, Niru Maheswaranathan. Ruoxi Sun, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Using thousand optimization tasks to learn hyperparameter search strategies. arXiv preprint arXiv:2002.1 1887, 2020.
[38] J Mockus, V Tiesis, and A Zilinskas. The application of bayesian methods for seeking the extremum, vol. 2. L Dixon and G Szego. Toward Global Optimization, 2, 1978.
[39] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Represen- tations, 2021.
[40] Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. Pfns4bo: In-context learning for bayesian optimization. In International Conference on Machine Learning, pages 25444-25470. PMLR, 2023.
[41] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In International Conference on Machine Learning, pages 16569-16594. PMLR, 2022.
[42] Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cédric Archambeau. Scalable hyperparameter transfer learning. Advances in neural information processing systems, 31, 2018.
[43] Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi-information source optimization. Advances in neural information processing systems, 30, 2017.
[44] Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov, Edward Bergman, and Frank Hutter. In-context freeze-thaw bayesian optimization for hyperparam- eter optimization. arXiv preprini arXiv:2404. 16795, 2024.
[45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2211 1-252, 2015. doi: 12611111--1---0016666
13

---

[46] David Salinas, Matthias Seeger, Aaron Klein, Valerio Perrone, Martin Wistuba, and Cedric Archambeau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In International Conference on Automated Machine Learning, AutoMI. 2022, 2022. URL. hol.itttss:/proceeedngg .mw...gssm...................... mml. htt.:/dddinnnn.....r//e/ee///tee Sat///slllnnn inas22a. httml.
[47] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 14104450, 2018.
[48] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of ma- chine learning algorithms. Advances in newral information processing systems, 25, 2012.
[49] Jasper Snoek, Kevin Swersky, Rich Zemel, and Ryan Adams. Input warping for bayesian optimization of non-stationary functions. In International conference on machine learning, pages 1674-1682. PMLR, 2014.
[50] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sun- daram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on muchine learning, pages 2171-2180. PMLR, 2015.
[51] Artur Souza, Luigi Nardi, Leonardo B Oliveira, Kunle Olukotun, Marius Lindauer, and Frank Hutter. Bayesian optimization with a prior for the optimum. In Machine Learning and Knowl- edge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pages 265-296 Springer, 2021.
[52] Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013.
[53] Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint arXiv: 1406:3440.896 2014.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep nigh-resooluton representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(0)); 3364, 2020.
[56] Zi Wang, George E Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snock, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization. arXiv preprint arXiv:2109.08215. 2021.
[57] Ying Wei, Peilin Zhao, and Junzhou Huang. Meta-learning hyperparameter performance prediction with neural processes. In International Conference on Machine Learning, pages 10058-11067. PMLR, 2021.
[58] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016.
[59] James Wilson. Stopping bayesian optimization with probabilistic regret bounds. Advances in Neural Information Processing Systems, 37:98264 98299,, 2024.
[60] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surro- gates. In International Conference on Learning Representations, 2020.
[61] Martin Wistuba and Tejaswini Pedapati. Learning to rank learning curves. In International Conference Machine Learning, pages 10303-10312. PMLR, 2020.
[62] Martin Wistuba, Arlind Kadra, and Josif Grabocka. Supervising the multi-fidelity race of hy- perparameter configurations. Advances in Neural Information Processing Systems, 35:13470- 13484, 2022.
14

---

[63] Jian Wu and Peter I Frazier. Continuous-iiellit bayesian optimization with knowledge gradi- ent. 2018.
[64] Jian Wu, Saul Toscano-Palmerin, Peter Frazier, and Andrew Gordon Wilson. Practical multi- fidelity bayesian optimization for hyperparameter tuning. In Uncertainty in Artificial Intelli- gence, pages 788-798. PMLR, 2020.
[65] Qian Xie, Raul Astudillo, Peter Frazier, Ziv Scully, and Alexander Terenin. Cost-aware bayesian optimization via the pandora's 's box gittins index. Advances in Neural Information Processing Systems, 37:115523-115566 2024.
[66] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL htpps/////ppeeeeieee net/forumidd-r1ne 1Ddp1--b.
[67] Lucas Zimmer, Marius Lindauer, and Frank Hutter. Auto-pytorch: hulti-fidelity metalearning for efficient and robust autodl. IEEE transactions on pattern analysis and machine intelligence, 4309):3079--000, 2021.
15

---

Appendix Overview
This appendix provides supplementary materials to support the main paper as follows:
Related Work (Appendix A): discusses the relevant literature (1) multi-fidelitt HPO, (2) freeze-thaw BO, (3) LC extrapolation, (4) transfer BO, (5) cost-sensitive HPO, (6) early stopping BO, (7) BO with user preferences, and (8) neural processes.
Notation (Appendix B): summarizes an overview of the notation used throughout this paper.
Utility Estimation (Appendix C): details how we can estimate user utility with Bradley-Terry model [10] and provides several examples.
Dataset (Appendix D): provides details on benchmarks (LCBench [67], TaskSet [37], PD1 [55]) and data preprocessing.
Details on LC Extrapolator (Appendix E): contains the architectural and training details of the LC extrapolator ps-
Implementation Details (Appendix F); includes implementation details CFBO and baselines.
Additional Experiments (Appendix G): presents additional experiments, including results on real-world object detection datasets and visualizations of normalized regrets and LC extrapolations during Bayesian optimization process.
A Related Work
Multi-fidelity HPO. Unlike traditional black-box approaches for HPO [8, 22, 9, 48, 50, 49, 12, 40], multi-fidelity (or gray-box) HPO aims to optimize hyperparameters a sample-efficient manner by utilizing low-fidelity information (e.g., validation performances with smaller training datasets) as a proxy for higher or full fidelities [52, 25, 29, 43, 26, 63, 64), dramatically speeding up the HPO process. In this paper, we focus on using performances at fewer training epochs to better predict and optimize performances at longer training epochs. One well-known example is Hyperband [34], a bandit-based method that randomly selects set of hyperparameter configurations and stops poorly performing ones using successive halving [27] before reaching the last training epoch. While Hyper- band shows much better performance than random search [8], its computational or sample efficiency can be further improved by replacing random sampling of configurations with Bayesian optimiza- tion [BOHB; 14], adopting an evolution strategy to promote internal knowledge transfer [DEHB; 4], or making it asynchronously parallel [33].
Freeze-thaw BO [53] dynamically pauses (freezes) and resumes (thaws) config- urations based on the last epoch's ss performances extrapolated from set of partially observed learn- ing curves (LCs) obtained from other configurations, leading to an efficient and sensible allocation of computational resources. DyHPO [62] and its transfer version [3] improve the computational
Freeze-thaw BO.
efficiency of freeze-thaw BO using deep kernel Gaussian processes [GPs; 58], but their acquisition maximizes one-step forward fidelity (i.e., epoch). producing myopic strategy. Other recent variants of freeze-thar thaw BO include DPL [24] and ifBO [44], which are non-myopic. and their acquisitions maximize performance either the last fidelity or random future steps. On the other hand, we maximize the utility specified by each user.
LC extrapolation. Freeze-thaw BO requires the ability to dynamically update predictions on fu- ture performances from partially observed learning curves (LCs), thus heavily relying on LC ex- trapolation [7, 17, 61]. DyHPO [62] and Quick-Tume
[3] propose to extrapolate LCs for only single step forward. Freeze-thaw BO [53] and DPL [24] use non- non-greedy extrapolations but limit the shape of the L.C.. [13] consider a broader set of basis functions but require computationally expen- sive Markov Chain Monte Carlo (MCMC), and also do not consider correlations between different configurations. [30] models interactions between configurations with Bayesian neural networks (BNNs) but suffers from the same computational inefficiency MCMC and online retraining. LC- PFNs [2] are an in-context Bayesian LC extrapolation method without retraining, but they do not consider interactions between configurations. Recently, ifBO [44] further combined LC-PFNs with PFNs4BO [40] to develop in-context surrogate function for freeze-thaw BO, but they train PFNs only with a prior distribution. On the other hand, we use transfer learning with LC mixup ($3.4), i.e., training PFNs with existing LC datasets, to improve the sample efficiency of freeze-thaw BO while successfully encoding the correlations between configurations at the same time.
16

---

Transfer-BO. Transfer learning can be used to improve the sample efficiency of BO [5], and here we list few examples. Some recent work has explored scalable transfer learning with deep neural networks [42, 60]. Additionally, different components of BO can be transferred, such as obser- vations [52], surrogate functions [19, 60], hyperparameter initializations [60], or all of them [57]. However, most of the existing transfer-BO approaches assume traditional black-box BO settings. To the best of our knowledge, Quick- Tune [3] is the only recent work that targets multi-fidelity and transfer-BO BO simultaneously. However, their multi-fidelityy BO formulation is greedy. As described in Figs, Sa and 8b of $4.3, our CFBO can dynamically control the degree of greediness during BO by explicitly taking into account the trade-off between the cost and performance of BO.
Cost-sensitive HPO. Multi-fidelity BO is inherently cost-sensitive since predictions get more ac- curate as the gap between fidelities becomes smaller. However, the performance metric of such vanilla multi -ulti-fidelity BO monotonically increases as spend more budget. In this paper, we aim to find the optimal trade-off between the budget spent and the corresponding intermediate per- formances of BO, thereby automatically early-stopping the BO around the maximal utility. Quick- Tune [3] also suggests cost-sensitive BO in multi-fidelity settings, but unlike our work, their primary focus is the trade-off between performance and the cost of BO associated with pretrained models of various sizes, which can be seen as a generalization of the traditional notion of cost-sensitive BO [48, 1, 32, 65] from black-box to multi multi-fidelity settings.
Early stopping BO.
Makarova et al. [35] propose a principled early stopping BO using:
f - < 2x + y -ob.
(9)
where f denotes the true performance of population, Do the validation performance, * the maximizer, and e E R2 statistical error term. Eq. 9 leads to stopping condition as follows:
jv S e.
(10)
Eq. 10 can be expressed using a utility function:
Db. if Ub 2 5 -,,
(b,, (b)
(11)
1-00, otherwise.
Since 5* and e are unknown, Makarova et al. [35] estimate them using lower confidence bound (LCB), upper confidence bound (UCB), and the coefficient of variation (CV). As these estimators are functions of the budget 6, Eq. 11 can be rewritten in a utility view:
U(b (b) = 9
if is 2 c(b),
(12)
1-20 otherwise.
Interpreted this way, the stopping criterion in Makarova et al. [35] roughly corresponds to the prefer- ence: "Stop once performance exceeds c(b); additional budget has no value thereafter." Therefore, Eq. 12 represents a special case of our utility formulation.
Similarly, Wilson [59] roughly fits into a utility view:
9b: (-cb
if p(ra S c(C) < 8,
U (b, 5b)
(13)
otherwise.
where rb = f fo and G is the cumulative cost. Furthermore, although both Makarova et al. [35] and Wilson [59] provide principled and general approaches for early stopping in BO, they primarily target black-box BO settings. Due to this fundamental difference, we exclude them from our baselines in $4.
BO with user preference. Several works have tried to encode the user beliefs about hyperparame- ter configurations into BO frameworks [51, 23, 36]. On the other hand, our paper suggests encoding user preferences regarding the trade-off between cost and performance. Therefore, the notion of user preference in this paper is largely different from the previous literature.
Neural Process Our training method is more similar to Transformer Neural Processes (TNPs) [41], a Transformer variant of Neural Processes [NPs; 18]. Similarly to PFNs [39, 2, 44], TNPs directly maximize the likelihood of target data given context data with a Transformer archi- tecture, which differs from the typical NP variants that summarize the context into latent variable and perform variational inference on it. Moreover, as with the other NP variants, TNPs meta-learn a model over distribution of tasks to perform sample-efficient inference. In this vein, the whole training pipeline of our ICC extrapolator can be seen as an instance of TNPs-we we also meta-learn a sample-efficient LCC extrapolator over the distribution of L.Cs induced by the mixup strategy.
17

---

B Notation
In this section, we summarize the notation used throughout the paper in Tab.
Table 4: Notation summary.
Notation
Description
Za d
n-th hyperparameterr configuration with dimension of dx
X [IN
Set of hyperparameter configurations
TEN N
Maximum training epochs for each configuration
E [7] = (I....,T}
Training-epoch index
Un.t E [0, 1]
Validation performance œn at epoch
B BEN
Maximum total training epochs (overall BO budget)
b [B]
Budget spent far
Db
(Cumulative) best performance observed up to budget b
C= ((x,6,))
Collected partial or full learning curves
Po Rd
[T] x - 0,
Learning-curve extrapolator
U: [B] |,1]
Utility function
Hyperparameter which decides for functional form of (b, (o) ūe a )
e [0.1]
Penalty coefficient of (b, ū6) 0o od B)
Up E [0,
Utility immediately before current BO step Acquisition function in Eq. 4
A(=U): [N]- - IR
Rs Omee-la E [0,1] Unas Unis
roughly estimated normalized regret Eq.
Omz E [0,
Maximum utility observed so far for computing RG Eq.
Umin E
Approximated minimum utility for computing RG in Eq. 5 the true normalized regret in Eq. 5
R =
Unn UPp E [0 Omo Unit
Una = maxn, U al. E [0, 1]
True maximum utility for computing It in Eq. 8
Umin AS min, U(B (m.1) E [0, 1]
Approximated true minimum utility for computing R in Eq. 8 Fixed stopping threshold for baselines in Eq.
E [0.
6, 8,, [0,
Adaptive stopping threshold instead of in Eq.
E [0
Probability that current configuration will improve Up in Eq.
BetaCDF(;333)
[0, + [0
the CDF of Beta distribution with shape 3
1>0
Hyperparameter controlling interpolation in Eq.
? E Ry0
Hyperparameter setting Pb 0.5 in Eq. 6
18

---

C Utility Estimation
Nermelized Computationa Budge {3
Normaal Computationl Sudget 13
hormllzed Computationl Sudger ()
Norma ized Computations Budgget (p)
polo momun 00
Reformoce (P)
Normalloed Computational Budges
Narmallass Computational Sudget
Normllzed computatioal Budger
Norma leed Computation Budget (p)
Ne 851
Normollzed Computationl Budget +)
Normaalzze Compunatiooal budget 17
Normalized Computational budpet 91
Numalized Computationn Budgett go
Figure 10: Contour plots of true utilities and their approximations. From left to right, the columns show different functional forms of linear (c = 1), quadratic (c 2), square root (c 0.5), and combination of four different functions including a staircase function. From top to bottom, the rows represent 30, 100, and 1000 user preference data pairs.
Functional forms. In real-world scenarios, users can have more complex utility functions. We therefore consider the following functional forms: (1) Linear: Úb a a. (2) Quadratic: Üt a )²) (3) Square-Root: j/b - a( b )0.5, and (4) Staircase: Dh - -,,, E A.) (I()) is an
indicator function, and A; is an i-th interval). We further assume that these utility functions can be linearly combined, e.g., -.,-) w (lineur) (..-) +... + u (stiiiicas))) U (stuircase)
(-,-), where
(lineur) +... + w (Stuircase)
1.
Data collection. We now describe how we roughly estimate user utility based on the user prefer- ence data pairs. First of all, we assume that it is possible for users to decide whether they prefer one point the other one, instead of quantifying their utility, i.e., we can collect user preference data. For simulation, we assume that we are given these preference data generated by true utility function. True utility function U is randomly defined by sampling penalizing coefficient from Uniform(0 (0, 1) and linear combination coefficients from Dirichlet distribution. We randomly select meaningful data pairs from b/ B Uniform(O, 1) and ÿb Uniform(O, 1). Here, "meaningful data pairs" means that one datapoint of each pair is not trivially preferred by user: for example, one has larger per- formance Se with smaller budget than the other. Users then label their preference on these data pairs; for simulation, we label the pairs by using the true utility functions with sampled as and linear combination coefficients ws.
Training details. As explained in Eqs. 2 and 3, we use the binary cross-entropy loss {(x y) = -ylog x - (1 y) log(1 x>) Eq. 3 between the probability of preference described by the BT model and the preference label yı E {0. 1}. We begin by randomly initializing another utility func- tion to approximate a randomly sampled true utility function, setting the ws and as to and 0. 0.0001, respectively. We use gradient-based optimization algorithm (e.g., SGD) with 1000 iterations. The temperature term T in Eq. 2 is set to 0.05.
Estimation results. Fig. 10 demonstrates that not only can single utilities-linee quadratic, and square root-be well approximated using preference data, but even more complex utilities (e.g.. a combination of four different utilities) can also be accurately approximated. Furthermore, we found that the approximation works well even with smaller numbers (e.g., 30, 100) of user preference data pairs for simpler cases (i.e., single utilities).
19

---

D Dataset
LCBench. We use the LCBench benchmark [67], which consists of learning curves for MLPs trained on multiple tabular datasets. Each task contains 2,000 learning curves with 51 training epochs. We summarize the hyperparameter settings of LCBench in Tab. 5.
Table 5: The d, hyperparameters for I.Ceench datasets.
Name
Type
Vaules
Info
batch_ size
integer
12
log
learning rate
continuous
[10 [10-1, 10
log
max max.dropout
continuous
[0. 1]
max_units
integer
12, 2
log
momentus
continuous
0.99
max_layers
integer
[1,5]
weight.decay
continuous
[10 9.10-
The training/test splits are as follows:
Training datasets:
APSFailure, Amazon www.:eeepppll...ooooooooeeeeeeee access, Australian, Fashion-MMIST,
MiniBooNE, adult, airlines, albert, bank-marketing, car, christine, cnae-9, connect-4, covertype, credit-g, dionis, fabert, helena.
Test datasets:
higgs, jamnis, jasmine, sungle.chess. 2pcs raw complett, kci, kr-vs-kp, mfeat-factors, nomao, numerai28.6. 6, phoneme, segment, shuttle, sylvine, vehicle, volkert.
TaskSet. We the TaskSet benchmark [37], which provides learning curves from diverse opti- mization tasks across multiple domains. Each task contains 1,000 learning curves with 50 training epochs. We summarize the hyperparameter settings of TaskSet in Tab.
Table The d. 8 hyperparameters for TaskSet tasks.
Name
Type
Vaules
Info
learning rate
continuous
[10 10*)
log
betal
continuous
[10-4,1
beta2
continuous
[10-3
epsilon
continuous
[10-12 10 10
log
11
continuous
[10 1109,10*
log
12
continuous
[10-9. 10
log
sinear_decay
continuous
10 10-3 10
log
exponential decay
continuous
[10 110-6, 10
log
The training/test splits are as follows:
Training tasks:
3, 46, 47, 59, 6, 66),
word_unn.lngguage htt.guugge.odel..aallleeeeeeeeee 22, 47, 48, 74, 76, 81) char rnn language model family seed 26, 31, 42, 48, 5, 74).
Test tasks:
rnn text classification family seed 8, 82, 89).
word rnn language model family seed 84, 98, 99). char langunng caceee_oeelleee foodel_family_seee 84, 94, 96)
We use the PDI benchmark [56]. where each task contains 240 LCs with 50 training epochs. To facilitate transfer learning, we preprocess LC data of PDI by excluding hyperparameter configurations with their training diverging (e.g., LCs with NaN), and linearly interpolate the LCs to match their length across different tasks. We then obtain the LCs of 50 epochs over the 240 configurations. We summarize the hyperparameter of PDI in Table 7.
PD1.
20

---

Table 7: The hyperparameters for PD1 tasks.
Name
Type
Vaules
Info
1r. initiiiilll...... value
continuous
[10-5, 10"]
log
1r 1r-power
continuous
[10-1.2]
1r 1r_decay seweayssteps..m wtt.ssteeecttto... factor
continuous
[10 110-2 0.99
one momentum
continuous
[10-;,]
log
The training/test splits are as follows:
Training tasks:
uniref50 transformer batch size 128, 1mib transformer batch size 2048,
256, mnist_max.p wtwg.//cnttan.batccl..... (256- 2048),
mnist htt.:eeeccn.aachhsihieeeeeeee
cessttssmmle_mmtt fashion mww.m htt.:/mwwxpoooingcmmmtaaae www.hatthha.cchaaaoaeo.aoo size._s82
www..chc.ssseee..eee. 2048),
fashion mnist max pooling cmn relu batch size {256, 2048), f fashion_mnis http://ssssssmpeeee.mmm.aaaeaaaa
2048),
svhn wtw_nno_eett___aeeeeee wwwdereesnet..acch.siieeeee cifar(10, wide resnet watnnettbatch.cc size 2048.
Test tasks:
imagenet wwnggetetrrennt..... resnet.batch............... mnist max pooling.cooooo con tanh. batch size 256,
translate. wmt xformer www.smeatrannlatteb.. batch size 64,
cnn
fashion mnist max_maxx pax.pnoling cmn tanh. batch size 256, svhn no extra wide resnet batch size 256,
Data Preprocessing We follow the convention LC-PFN [2] for data preprocessing; we consis- tently apply non-linear LC normalization" the LC data of three benchmarks, which only maps either accuracy or log-loss LCs into [0, 1] but also simply makes our optimization as a maix- imization problem. We also the maximum and minimum values for each task in LCBench and PDI benchmark for the LC normalization. In TaskSet, we only use the yo,0 (i.e., the imitial log-loss without taking any gradient steps) for the LC normalization.
E
Details on LC Extrapolator
Construction of context and query points. Following ifBO [44], we can simulate each step of BO; predicting the remaining part of LC in all configurations conditioned on the set C of the collected partial LCs. To do so, we construct a training task by randomly sampling the context and query points from the LC benchmarks after the proposed LC mixup in $3.4 as follows:
1. We choose LC dataset L = .......,yy [N]] E NxT by randomly sampling [M].
2. From ((m), we randomly sample .1...., nc E [N] and (1, .1....4 lg E [T] and construct context
points of ((e) [in s....... I RRxxs T(e) [1/T, ..... [ E RCx1, and y
(0y.t1 Pee E C
3. From the chosen L., we exclude n1...., ne E [N] and l,,.., lc [7] and randomly sample
n/....,,ne n/2 E [N] and "i,...tty t/2 E [T] and construct query points of X = [rn ...., ,.ngg l
RQxd, T(t) = (M/T, SEt/T,..,,tc.. ......//TTTT TT E RQx1, and ( = [Wnn ..... Un Rx11
Architecture for predicting LCs. From now on, we denote any row vectors of the constructed context and query points lowercase, e.g.,
x (()) of ( or y(a) of y(4), We train a Transformer [54] that is probabilistic model of ((Y Tet), (()), X (2)) T(a)) Conditioned on any subsets of LCs (i.e., (() _Te), and (()),, this model predicts mini-batch of the remaining part LCs of existing hyperparameter configurations in given dataset (Le.. y( of X()) and T(4)). For
'The details can be found in Appendix A of PFN [2] and http:::///ithubb com/ http://www..ccooooo/o/o////////a hiin/lcpfn/uutlss
21

---

computational efficiency, we further assumee that the query points are independent of each other, as done in PFNs [39. 44]:
Po 1 T(o), y x(1) T(a))
II
Po(yl tl0), x(e), T(o), v(e)). (14)
z(w),eto),g/0)
Before encoding the input into the Transformer, we first encode the input of x(o)) y(o), x(,), and T(s) using a simple linear layer as follows:
H X(O + TW + Y'O,,
(15)
H = x + Tw,,,,
(16)
where W₂ Idd, W, 1xxd and WG Rixd, Here, we abbreviate the bias term.
Then we concatenate the encoded representations of H(e) and H(a), and feedforward it into the Transformer layer by treating each pair of each row vector of H(()) and H()) as separate posi- tion/token as follows:
H = Transformer((H H(4), Mask)) E (MM+N)xd.
(17)
Y Head H) R(M++N)xd
(18)
where (ransformer()) and Head(.) denote the Transformer layer and multi-layer perceptron (MLP) for the output prediction, respectively. Mask E I2(Ne++N.)((M+NNN) is the transformer mask that allows all tokens to attend context tokens only.
In Eq. 18, the output dimension d, is specified by the output distribution of y E [0, 1]. Following PFNs [39, 2, 44], we discretize the domain of by d, = 1000 and use the categorical distribution. Finally, we only take the output of the last Ng tokens as the output, i.e., ÿ7(0) = rl: No (N₂ + Ng)] E N.x,, (using the indexing operation), since we only need the output of query tokens for modeling p (0) ((9), X(e), T(c), y())).
Training Objective.
Our training objective then defined as follows:
arg min Ep, 6
Σ
log Po (y) (1))) pla), x(e) T(c) rich)
+ APpn
(19)
where p is the empirical distribution of LC data with the LC mixup in $3.4. We additionally mini- mize LPEN with coefficient ApPN, which is the LC extrapolation loss of LC-PFN [2] for each LC. We found APEN = 0.1 works well for most cases.
Training Details. We sample 4 training tasks for each iteration, Le., the size of meta mini-batch set to 4. We uniformly sample the size C of context points from to 300, and the size of query points
2.048. Following LC-PFN [2], the hidden size of each Transformer block d,, the hidden size of feed-forward networks, and the number of layers of Transformer, dropout rate are set 1.024, 2.048, 12, and 0.2, respectively. We use GeLU activation [21]. We train the extrapolator for 100,000 iterations on training split of each benchmark with Adam [28] optimizer. The &g norm of the gradient clipped 1.0. The learning rate is linearly increased 2- 10 -05 for 25,000 iterations (25% of the total iteration). and it is decreased with a cosine scheduling until the end. The whole training process takes roughly 10 hours in a single A100 GPU.
F Implementation Details
O-epoch LC value. We assume access to the 0-epoch LC value (yn.o) which is the performance without taking any gradient steps, i.e., random guessing. This also plausible for realistic scenarios, since in most deep-learning models one evaluation cost is acceptable in comparison to training costs. The O-epoch LC values originally are not provided except for LCBench; we use the log-loss of the first epoch as the O-epoch LCC value for TaskSet, as it is already sufficiently large in our chosen tasks. For PD1, we interpolate the LCs to be the length of 51 training epochs, and we take the first performance as the O-epoch LC value. Furthermore, we take the average over the 0-epoch LC values To - Σ Vn,0 for convenience. The average 0-epoch LC value Zo is always conditioned on our LCC extrapolator po for both the training stage in Appendix and the BO stage in Alg. 1.
22

---

Monte-Carlo (MC) sampling for reducing variance of LCs. We estimate the expectation of the proposed acquisition function A(: U) Eq. 4 with 1,000 MC samples. We found that each LC Va,t.. extrapolated from the LCC extrapolator po (o C) is noisy, due to the assumption that the query points of Vr,f...r are independent of each other in Eq. 14. Since we compute ia by taking the maximum among the last step BO performance (i.e., cumulative max operation), the quality of the estimation degenerates significantly due to the noise. To avoid this, we reduce the variance of the MC samples by taking the average of the sampled LCs. For example, sample 5,000 LC samples from the LC extrapolator po. then divide them into 1,000 groups and take the average among the 5 LC samples in each group. We empirically found that this stabilize the estimation of not only acquisition function A (:: U) and probability of utility improvement pa in Eq. 7.
Implementation details for baselines. We list the implementation details for baselines as follows:
1. Random Search [48]: Instead of randomly selecting hyperparameter configuration for each BO step, we run the selected configuration until the last epoch T'.
2. BOHB [14] and DEHB [4]: We follow the most recent implementation of these algorithms in [3]. We slightly modify the official code (https:// htt///gitubb/ com cot/r//eeeemmiifrrrbu/////////// QuickTune), which is based on SyneTune [46] package.
DPL [24]: We follow the official code (https: // github...b... cachhuu.cooo ccw//rreeeuunnffreirrr///rreee and slightly modify the benchmark implementation to incorporate our experimental setups.
4. ifBO [44]: We use the official code fnttps://githhe cat.ua.cco//automm/a/cac autp///mmmmmm// for the surrogate model (i.e., the LC extrapolator), and incorporate it into our code base to be aligned with our experimental setups.
DyHPO [62] and Quick-Tune [3]: We follow the official code (https://gi /tgp:e/////tttut../ com/ rele veleaumiff frei wereebbrrr//OOO and slightly modify the benchmark implementation to incorporate our experimental sctups. For Quick-Tune with Adam optimizer with batch size of 512. The initial learning rate is set to 10 and decayed with cosine scheduling. To leverage the transfer learning scenario, we use the best configuration among the LC datasets which is used for training the GP an initial guess of BO.
we pretrain the deep kernel GP for 50,000 iterations
6. FSBO [60]: We use the official code (https: //gi thub. com/ com/releaunis weaaunffeebuus freiburg/ hoabbburgg//oo////////////o////s sbo), slightly modify it to incorporate our experimental setups, and use the best configuration among the LC datasets as an initial guess.
G Additional Experiments
Table 8: Results on object detection datasets (c and a E (0,2-8, 6,2-6,277 2-))), We multiply 100 to the
normalized regret for better readability.
a 0
a = 2 6
a = 2
Method
Regret
Rank
Regret
Rank
Regret
Rank
Random
5.0213
6.5
7.1
6.4
13. 13.6
6.5
BOHB
3.2210
5.2
4.8
5.3
10.7
5,4
DEHB
5.0214
6.6
6.6214
6.5
12.4+13
6.6
DyHPO
16.0#25
5.9
17.5
6.0
23.1
6.2
DPL
3,9414
4.6
5,5
4.8
11.4+1.
5.2
ifBO
2.3
4.3
3,9405
4.3
9,8+05
4.4
Quick-Tume
5.3200
4.8
6,9200
4.9
12.6200
5.0
FSBO
2.0
3.9
3.7
3.9
9.62oa
4.2
CFBO (ours)
1.3
3.3
3,6:03
2.9
5.7
1.4
Effectiveness on the real-world HPO. We investigate the effectiveness of CFBO on real-world
object-detection dataset. From the 10 different datasets in RoboFlow 100 [11], we collect 500 LCs of validation performances by training three different network architectures, such as ResNet-50 [20], HRNet [55], MobileNetv2 [47], with 4 different hyperparameters (batch size, learning rate, momentum, and weight_decay _ddccayfactorr) Based on this setting, we construct 30 tasks (=3 network architecturesx 10 datasets) and split them into 20/10 tasks for peeta-traninggmmetttsss re- spectively. In Tab. 8, we observe that CFBO consistently and significantly outperforms all basclines on this real-world dataset.
23

---

Visualization of normalized regret. We provide visualization for the normalized regret of each method LCBench, TaskSet, and PDI throughout Figs. 11 to 16.
Visualization of the LC extrapolation over BO steps. We provide visualization for the LC ex- trapolation over BO steps of CHBO throughout Figs. 17 to 22. Specifically, we plot the LC extrap- olation results of unseen hyperparameter configurations through BO process. Each row shows the results for a different size of the observation (IC) e {0, 10, 50, 300)). and each column shows different size of context points St, where E 20
2, 5, 10, 20, 30}) in each LC.
Random
BOHB
DEHB Quick-Tune T
DyHPO
DPL
ifBO
CFBO-NT
FSBO
CFBO
Normalzeed Regret
10a
k0ml
t--4
Total Epochs Spent
2
2
2
Nomalzzd Regret
200
300
200
300
Total Epochs Spent
a 2-2
2
2
a= 2-2
2
Normalized Regrot
20ml
10n2
200
300
100
200
300
Total Epachs Spont
2-2
2
2
2-
2
Normalized Regrot
10-1
100
200
300
200
200
300
300
100
200
300
Total Epachs Spent
0' 2
a- 2-5
d 2
d-
d 2
Noraalzed Regrot
20m1
100
200
300
300
200
300
100
200
300
100
200
300
100
200
300
Total Epochs Spent
o 2
2-1
a 2
a. 2
Rornalized Regrett
1.0
20-1
20.2
1.00
200
300
300
200
300
1.00
203
300
100
200
300
1.0
200
300
Total Epochs Spent
Figure 11: Visualization of the normalized regret LCBench.

---

Random
BOHB CFBO-NT
DEHB Quick-Tune
DyHPO FSBO
DPL
IIBO
CFBO
Normaltzen Ragret
100
20-1
Epochs 2-1
2
Normaltzed Regrot
20-1
20-2
Total Epachs Spont
2-4
2-
2-4
Normalzzdd Regrot
10-2
20-4
Total Epachs Spent
2
Nommllzzd Regret
10m1
2007
100
200
300
Total Epochs Spent
2-
2
2
a= -
Normalzzd Regret
1.0
10-4
1.00
200
100
1.0
100
200
300
Total Epochs Spent
2-3
2-1
2
2-2
2
Normaizzd Regrot
1.0
200
300
100
200
300
Total Epachs Spont
Figure 12: Visualization of the normalized regret on LCBench.
25

---

Random
BOHB CFBO-NT
DEHB Quick-Tune
DyHPO FSBO
DPL CFBO
IIBO
Normaltzen Bagret
2001
Epochs 2-1
2
2-1
Normaltzed Ragrot
10a
2..1
10-4
Totall Epachs Sport 2-
2-4
2-
2
Normalized Regret
20-1
20-2
Tetal Epochs Spent
Figure 13: Visualization of the normalized regret L.Bench.
26

---

Random
BOHB CFBO-NT
DEHB Quick-Tune
DyHPO FSBO
DPL CFBO
IIBO
Normalzee Ragret
1.0
10-
Epochs 2-1
2
Normatzzed Ragrot
100
20-1
Totall Epachs Spont
2-4
2-
2-
2-3
2
Normalzzed Regrot
20-1
Tetal Epochs Spent
Normatzed Regret
22-2
20-2
300
100
200
300
Total Epochs Spent
2-
2
2
a=
Noralizzdd Regrot
1.0-1
k0-3
1.00
200
100
100
100
200
300
Total Epochs Spent
2-3
2-1
2
2-
2-
Normalized Regrot
20-1
2.
200
300
200
100
200
300
Total Epachs Spont
Figure 14: Visualization of the normalized regret TaskSet.
27

---

Random IIBO
BOHB CFBO-NT
DEHB Quick-Tune
DyHPO FSBO
DPL
CFBO
Normaltzed Ragret
Epochs 2-1
2
2-5
Normalized Ragrot
20m1
2.
Totall Epachs Spont 2-4
2-4
2-
Normaizzed Regrot
2013
Tetal Epachs Spent
Figure 15: Visualization of the normalized regret TaskSet.
28

---

Random
BOHB CFBO-NT
DEHB Quick-TTnee
DyHPO FSBO
DPL
IIBO
CFBO
Normattred Regret
10a
k0n2
RD-4
20
Epochs Spent
2
2
2
Nomaltzed Regrett
1.0
20-1
20-7
2--1
20-4
Totall Epachs
2-
2-1
2
Normalzzed Regrot
20--
10
Total Epachs Spent
Nomalzzd Regret
20-
10.
100
200
300
Total Epochs Spent
2-
2-1
2
a=
a= -
Nomalized Regret
1.00
100
100
200
300
Total Epochs Spent
2-3
2-1
2-4
2-2
2
Normalizdd Regrot
10a
20-1
100
300
200
100
200
300
Total Epachs Spont
Or 2-
- 2-5
d 2-
2
Noraalted Regrat
20-1
100
200
300
300
200
300
100
200
300
100
200
300
100
200
300
Total Epochs Spent
Figure 16: Visualization of the normalized regret PD1.
29

---

Cortext Porba LC:0
Contert Pairts in LC:2
Contert Panta SC.3
Conter: Pants LC
Contect Parta LC: 20
Contest Ponta in Lc: 30
Figure 17: Visualization of LC extrapolation on LCBench.
Contect Poinba LC:0
Contect: Contecc: Points LC
Contecc: in LC:10
Ponts Conceot Panta
CSFFF
Figure 18: Visualization of LC extrapolation LCBench.
30

---

Cortext Poinka LC:0
Pairta in LC:2
Contert: Pants SC:3
Conter: Pontss LC:
Contect Portla in LC: 20
Concestt Po in Lc: 30
Figure 19: Visualization of LC extrapolation on TaskSet.
Contect Poinba LC:
Porta
Contecc Poinaa LC
Contect Pointain LC:10
Ponts Contaot Pantaa
Figure 20: Visualization of LC extrapolation TaskSet.
31

---

Cortext Tainaa LC:0
Contert Pairts in LC:2
Contert Pants in SC:3
Conter: Poinss LC:
Contect Poinla LC: 20
Concestt Ponta in Lc: 30
03
Figure 21: Visualization of LC extrapolation on PD1.
Contest in (C:0
Contect Pairks LC:2
Contect Points LC:
Contect Pointiin LC:10
Contect Pontss LC: 20
e Conceot Ponts LC: 30
Figure 22: Visualization of LC extrapolation on PD1.
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the experimental section, we provided numerous supporting evidences for the claims made in the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide the Limitations paragraph in $5.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include any theoretical claims.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide experimental details including datasets, training details, and base- line implementation throughout Appendices D and F.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We public benchmark datasets and upload anonymous code in the sup- plemental materials.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We not only provide experimental setups in main paper but also detail the setups in depth throughout Appendices and F.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide mean and standard deviation over 5 runs (or 30 runs) for all our experiments.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
33

---

http///awrr..cccollll/ccccctaaaa Justification: We provide the detailed information in the Appendix.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respeet, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We carefully reviewed the NeurIPS Code of Ethics.
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: The topic of our paper about improving the efficiency of hyperparameter optimization, which is irrelevant to societal impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our paper is not relevant such issues and risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We properly put citations and URLs for them throughout the paper.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release any new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve any crowdsourcing nor research with human sub- jects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]-
Justification: Our paper does not involve any crowdsourcing nor research with human sub- jects.
34


Reviews:
[{"review_id": "4Am9gRiyMk", "reviewer": "Reviewer_uqhd", "summary": "This paper considers a Bayesian optimization method that combines freeze-thaw BO, preference learning, and transfer learning. The method uses a variant of expected improvement on the preference function (utility), where the incumbent threshold is set to the most recent utility, rather than the previously evaluated point with max utility. The method maximizes EI over the parameters and target epoch. The method leverages a regret-based stopping criterion for decide when to stop the entire HPO loop. The paper shows how training the PFN via transfer learning (via mixup) improves performance.", "questions": "* The intro discusses the motivation being that budget is typically limited and users may want to allocate more budget to training the model than to exploration. Why not focus the paper on BO algorithms that explicitly account for a known budget and plan accordingly (using lookahead and planning w.r.t the budget)?\n* Is it reasonable for a BO user to pick the parametric form of their utility function? That seem hard for a user to specify\n* How does performance change if parts of the AF are ablated: a) if only final performance is optimized for rather than the optimizing the target epoch, b) if the threshold used in EI is the best incumbent and not the most recent?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The empirical performance is strong\n* Sensitivity analyses show the best choice of HPs for for the Beta CDF in the stopping criterion\n\nWeaknesses\n* The empirical evaluation gives the proposed method a sizable advantage notably since methods are evaluated w.r.t  the utility which the proposed method is aware of (it omnisciently knows the right preference function class) and baselines are not.\n* The contribution is pretty small as the paper combines freeze-thaw BO, transfer learning, and preference learning.\n* The method introduces quite a few hyperparameters (HPs for the Beta CDF, choice of parametric function class, etc)\n* The transfer learning method requires retraining the PFN for each new use case", "was_revised": true, "final_justification": "Thanks for the response and clarifying the ablations. Most of my concerns have been addressed, and I have increased my score. My main criticism is lack of originality, as the paper combines many existing approaches. Nevertheless, the combination is effective and tackles an important problem.", "rebuttal": "We sincerly appreciate you for your time and constructive comments which improve our paper. We respectfully address your concerns as following:\n\n---\n> **[Q1]** Baseline unfairness.\n- Thank you for pointing this out. We already provide a fair comparison under the conventional multi‑fidelity HPO setting with $\\alpha=0$ (`Figure 6`). In this case the utility collapses to pure performance, so CFBO and all baselines optimize the same objective. CFBO still achieves superior results, showing its advantage is **not tied to privileged knowledge**.\n- Furthermore, incorporating our utility into existing baselines is **fundamentally difficult** because $U$ depends on future performance–cost trade‑offs and requires a probabilistic inference over full learning curves. DyHPO [1] predicts only the next step, and FSBO [2] predicts only the final performance, so they **cannot evaluate expected future utility** or apply our adaptive stopping and acquisition rules.\n- For fairness, we tuned a fixed stopping threshold ($\\delta = 0.2$ in `Eq. 4`) for all baselines using learning curve (LC) training dataset. They cannot use the adaptive criterion $\\delta_b$ in `Eq. 5` because it relies on computing the future utility in `Eq. 6`.\n- ifBO [3] is the closest prior work because it models long‑horizon performance, but it lacks transfer learning and the utility‑aware criteria. We therefore built **CFBO‑NT**, which augments ifBO with our acquisition and stopping rules while excluding transfer learning. CFBO‑NT still outperforms all baselines on cost‑sensitive benchmarks, confirming that the observed gains come from **our methodological contributions**.\n- We will discuss the above details in the revision.\n---\n> **[Q2]** Limited novelty.\n- We respectfully disagree with this assessment. We acknowledge that each individual ingredient, freeze–thaw BO, transfer learning across tasks, and preference learning, has appeared in prior work. The novelty of CFBO lies in **how these elements are brought together to tackle a problem** that, to our knowledge, has not been formally addressed before: hyperparameter optimization when **users explicitly trade off model performance against computational cost**.\n- As discussed in `L32‑36`, for example, cloud‑service users often operate under credit budgets. They care about final performance, but they also place a penalty on the HPO cost so that credits remain for other jobs. Existing algorithms either **ignore cost entirely or treat it only as a hard budget constraint**; they do not optimise expected utility that mixes both factors. Empirically, we show (see the answer to **[Q1]**) that traditional methods are ineffective in this setting.\n- CFBO is designed for this realistic scenario. We formulate a utility function encoding user preference, identify the need for probabilistic inference to evaluate the utility, and integrate freeze–thaw BO (for curve extrapolation) with transfer learning (for sample efficiency) to realise the objective. Thus our contribution consists of\n    1. introducing a practical, cost‑aware HPO **problem formulation**,\n    2. defining a **concrete utility‑based objective**,\n    3. analysing why **existing methods fall short**, and\n    4. presenting CFBO, **an effective solution** that—while built from known components—solves the new problem end‑to‑end.\n- The advance therefore also resides in the problem definition and its principled resolution, not merely in assembling familiar techniques.\n- We will clarify the above in the revision.\n---\n> **[Q3]** Hyperparameters.\n- We kindly note that CFBO introduces **only one tunable hyperparameter**, $\\beta$; all other quantities are either fixed for fairness or specified by the task itself.\n- The adaptive stopping rule uses $\\text{BetaCDF}$ with parameters $\\beta$ and $\\gamma$. To align CFBO with baselines, we select a fixed stopping threshold $\\delta = 0.2$ using the learning‑curve training set, then set $\\gamma = \\log_{0.5} 0.2$ so that the adaptive rule reproduces this threshold when $p_b = 0.5$. Since $\\gamma$ is fixed, it does not introduce additional tuning.\n- The coefficients $c$ and $\\alpha$ reflect how users trade off cost and performance. These are **task parameters**, not algorithm hyperparameters, and apply equally to all methods.\n- `Figure 7(d)` shows CFBO’s performance is robust for $\\beta \\ge e^{-1}$, allowing users to select a reasonable default without costly sweeps.\n- We will clarify this distinction and the limited tuning burden in the revision.\n---\n> **[Q4]** PFN retraining required.\n- Thank you for pointing this out. We acknowledge this concern and have already evaluated a **non‑transfer variant**, CFBO‑NT, described in `L276–278`. CFBO‑NT integrates ifBO with our utility‑aware acquisition function (`Eq. 3`) and adaptive stopping criterion (`Eq. 5`) but omits the PFN retraining step.\n- CFBO‑NT still achieves substantial improvements over all baselines that do not use transfer learning, demonstrating that the core ideas of CFBO—utility‑aware acquisition and adaptive stopping—remain effective without a pre‑trained PFN.\n- Practitioners lacking a learning‑curve training dataset can therefore adopt CFBO‑NT to obtain most of CFBO’s benefits without the overhead of PFN retraining.\n---\n> **[Q5]** Why not plan for a fixed budget?\n- We respectfully believe this is a misunderstanding. In our formulation the user **need not** spend the entire budget; they may stop BO early and reserve the remaining credits for downstream training or other jobs. So the choice is whether **the extra benefit from more exploration** is worth the credits that could instead be spent on other tasks.\n- A hard‑budget BO that plans to exhaust the budget can be sub‑optimal in this setting. If the performance gains flatten out well before the cap, it wastes credits that could yield higher overall utility elsewhere. By contrast, CFBO casts the trade‑off as a **continuous utility function** that penalizes additional cost; the algorithm stops automatically once expected gains fall below that penalty.\n- We evaluate CFBO across a wide range of cost‑sensitivity levels ($\\alpha$) and cost‑penalty shapes ($c$) to show that it consistently outperforms methods that ignore costs and use a fixed stopping rule (`Figure 4`). When $\\alpha$ is large—mimicking a tight, known budget—CFBO’s decisions converge to those of a hard‑budget BO (e.g., $\\alpha=2^{-2}$ in `Figure 5`), demonstrating that our framework **subsumes** the fixed‑budget case while remaining effective when early stopping is beneficial.\n- Practitioners who truly face a rigid, must‑spend budget can still apply CFBO by setting $\\alpha=0$ until the must-spend budget and apply $\\alpha>0$ after the budget; the method will naturally plan to use the budget. In all other cases, CFBO offers higher expected utility by adapting spending to the observed learning progress.\n- We will include the above discussion in the revision.\n---\n> **[Q6]** Users may not define a utility function.\n- We sincerly agree with that most users cannot write down an exact analytic utility. \n- To address this, we embed **preference learning** via the Bradley–Terry model [4] (`L133–140`). The system presents pairs of outcomes $(b_1, y_1)$ and $(b_2, y_2)$; the user simply chooses the one they prefer, e.g., $U(b_1, y_1)>U(b_2, y_2)$, and the model fits parameters that explain those choices. \n- This approach assumes only that the user can (1) state a preference between two options and (2) select a rough shape—linear ($c=1$), quadratic ($c=2$), square-root ($c=0.5$) or staircase—from a small predefined library. These shapes cover a broad range of cost‑sensitivity patterns while keeping inference tractable. \n- In `L141–150` we demonstrate a user who prefers any trajectory that outperforms the baseline ifBO trajectory at every point. We simulate preference pairs under this assumption, and `Figure 2` shows that the fitted utility closely follows the true trend.\n- Furthermore, as noted in `L149–150`, `Figure 9` in `Appendix B` shows that with about 30 pairwise comparisons the learned utility already tracks the true curve closely; with 100 comparisons the approximation error is negligible. The interaction burden is therefore modest.\n- Once the utility parameters are estimated, CFBO proceeds unchanged, allowing users who cannot specify a utility function explicitly to benefit from the method with minimal preference feedback.\n---\n> **[Q7]** What if acquisition function is ablated? a) optimize only the final performance, b) use the best incumbent as the EI threshold \n- We kindly note that the requested ablations coincide with CFBO evaluated under the **conventional HPO setting**, where the cost‑penalty weight is set to $\\alpha = 0$.\n- With $\\alpha = 0$ the utility simplifies to $U(b,\\tilde{y}_b) = \\tilde{y}_b$, so CFBO’s acquisition reduces to maximizing predicted **final performance** while ignoring computational cost—exactly ablation (a). \n- Under the same simplification, the threshold $U_p$ in `Eq. 3` becomes the **current best incumbent** because the utility input is the pair $(b, \\tilde{y}_b)$, where $\\tilde{y}_b$ is the running best performance. This reproduces ablation (b).\n- Empirically, CFBO with $\\alpha = 0$ is reported in `Figure 6`; it still outperforms all baselines in this pure‑performance regime, demonstrating that the performance gains are not contingent on the cost‑aware components.\n---\n### References\n\n[1] Wistuba, Martin, Arlind Kadra, and Josif Grabocka. \"Supervising the multi-fidelity race of hyperparameter configurations.\" NeurIPS. 2022.\n\n[2] Wistuba, Martin, and Josif Grabocka. \"Few-shot Bayesian optimization with deep kernel surrogates.\" arXiv. 2021.\n\n[3] Rakotoarison, Herilalaina, et al. \"In-context freeze-thaw bayesian optimization for hyperparameter optimization.\" ICML. 2024.\n\n[4] Bradley, Ralph Allan, and Milton E. Terry. \"Rank analysis of incomplete block designs: I. the method of paired comparisons.\" Biometrika. 1952."}, {"review_id": "CH9PCdVAwD", "reviewer": "Reviewer_AJQp", "summary": "The authors propose a cost-sensitive freeze-thaw BO approach that not only thaws the most promising configuration at each step, but also is aware of the cost of evaluating the thawed candidate. The awareness is realized by a utility function describing the trade-off between expected performance improvement and computational cost of the evaluation. An experimental evaluation shows that the proposed approach is competitive with other existing multi-fidelity approaches.", "questions": "see weaknesses", "limitations": "The authors do not discuss the limitations of their approach. This should be added (at least in the Appendix). Also, there is no related work section in the paper (however, a \"Background\" section). Still, discussing some related work, such as FSBO and QuickTune, would be beneficial (can be done in the Appendix).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The work considers an important problem in BO\n- The paper is written clearly\n- The proposed approach is presented well, and the empirical evaluation is appropriate for the proposed method\n\n**Weaknesses**\n\n- The authors do not provide any theoretical considerations on their proposed utility score. Though not being essential, an analysis of the convergence of multi-fidelity BO using the proposed utility score would be interesting\n- line 147-149: It is not clear to me what the authors mean by \"selecting one point above\" in this context.\n- Choice of hyperparameters of utility function: I can imagine that, depending on the hyperparameter choice of the utility function, the BO might yield solutions that differ quite much in terms of their evaluation score. Can the authors provide best practices to set these parameters?\n- Fig. 6: I'm wondering why QuickTune underperforms here. Since QuickTune comes with transfer capabilities, it would be nice if the authors could provide more details on that.\n- Results on cost-sensitive MF-HPO: I'd expect FSBO to be better than CFBO because FSBO is not considering any trade-offs but merely performs for maximum performance. Therefore, I found it surprising to see that FSBO performs worse than CFBO throughout all tasks. Maybe the authors could comment on that.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerly appreciate you for your time and constructive comments which improve our paper. We respectfully address your concerns as following:\n\n---\n\n>**[Q1]** The authors do not provide any theoretical considerations on their proposed utility score. Though not being essential, an analysis of the convergence of multi-fidelity BO using the proposed utility score would be interesting\n\n- We sincerely agree that theoretical analysis based on the utility function would further strengthen our work. While a full convergence analysis is beyond the scope of this paper, we note that our acquisition function is built on the classical **Expected Improvement (EI)** principle (`Eq. 3`), which is well-studied and underpins many convergence results in Bayesian optimization.\n- Our formulation generalizes EI by replacing the target (maximum performance) with a **utility function** that accounts for both performance and cost. This preserves the core idea of improvement in expectation while aligning optimization with user preferences.\n- Empirically, we show across diverse benchmarks and utility functions that CFBO leads to consistent and stable improvements over baselines (e.g., `Figure 4`, `Figure 6`), suggesting that the algorithm behaves robustly.\n- We view formal convergence guarantees under utility-based multi-fidelity BO as an important direction for future work and will mention this in the revision.\n\n---\n\n>**[Q2]** line 147-149: It is not clear to me what the authors mean by \"selecting one point above\" in this context.\n\n- We sincerely apologize for the confusion. In `L141–150`, our intention is to illustrate how the user utility function can be approximated using the Bradley–Terry model with simulated preference data.\n- To fit the model, we construct preference pairs $(b, y_1)$ and $(b, y_2)$ and assign preference labels (e.g., $U(b, y_1) > U(b, y_2)$) based on a simulated user's behavior.\n- Specifically, we assume a user who prefers **any outcome that improves upon the baseline ifBO trajectory** in the black line of `Figure 2`. \"selecting one point above\" is simulated to reflect this assumption as follows:\n- For each budget $b$, let $y_b^{(\\text{BO})}$ denote the performance of ifBO. We then sample $y_1 \\sim \\text{Uniform}(y_b^{(\\text{BO})}, 1)$ and $y_2 \\sim \\text{Uniform}(0, y_b^{(\\text{BO})})$, and record a preference $U(b, y_1) > U(b, y_2)$. This simulates **a user who consistently favors improvements over the ifBO trajectory at each budget level**.\n- We will clarify this explanation and notation in the revision.\n\n---\n\n>**[Q3]** Choice of hyperparameters of utility function: I can imagine that, depending on the hyperparameter choice of the utility function, the BO might yield solutions that differ quite much in terms of their evaluation score. Can the authors provide best practices to set these parameters?\n- Thank you for pointing this out. We respectfully note that $c$ and $\\alpha$ are not hyperparameters of the algorithm but **define the problem setup**: they encode the user's preference over the trade-off between model performance and computational cost.\n- In practice, users may not be able to specify this utility analytically. To address this, one can provide a small library of functional forms—e.g., linear ($c=1$), quadratic ($c=2$), square-root ($c=0.5$), or staircase—and then fit the utility using **preference learning** via the Bradley–Terry model. This is our recommended practice for estimating $c$ and $\\alpha$ from a few preference comparisons.\n- Utility functions also allow for more tailored behavior. For example, if users have a hard minimum budget $B'$ they want to fully use, but prefer to stop early unless improvements are significant, their utility could be:\n$$\nU(b,\\tilde{y}_b)=\\begin{cases}\\tilde{y}_b &\\text{if }b \\leq B',\\\\\\ \\tilde{y}_b - \\alpha B'/B &\\text{otherwise},\\end{cases} \\quad \\text{where} \\quad \\alpha \\rightarrow 1\n$$\n- This flexibility enables CFBO to reflect a wide range of practical user criteria. Moreover, experiments in `Figure 4` show CFBO performs well across diverse settings ($c \\in \\{1,2,0.5\\}$ and $\\alpha \\in \\{2^{-6},\\ldots,2^{-2}\\}$), demonstrating its robustness to different utility choices.\n- We will clarify the above in the revision.\n\n---\n\n>**[Q4]** Fig. 6: I'm wondering why QuickTune underperforms here. Since QuickTune comes with transfer capabilities, it would be nice if the authors could provide more details on that.\n- Thank you for your insightful feedback. QuickTune$^\\dagger$ (a transfer variant of DyHPO) performs comparably to non-transfer baselines, except on TaskSet. We attribute this to three key limitations: (1) a **simple deep kernel GP** surrogate [1] with limited expressivity, (2) its **greedy acquisition strategy**, and (3) the **lack of data augmentation**.\n- In contrast, CFBO uses a **Transformer-based PFN** [3] that scales better with data and supports richer transfer. As discussed in `Section 2.3`, we also apply a **mixup-style augmentation** to improve generalization across tasks.\n- Together with a **non-greedy acquisition** (when $\\alpha=0$), these design choices allow CFBO to more effectively leverage transfer learning and outperform QuickTune on standard multi-fidelity HPO benchmarks.\n- Notably, FSBO—despite using the same limited architecture (deep kernel GP)—significantly performs better than QuickTune because it learns to predict only the final performance, avoiding greedy behavior.\n- We will include the above discussion in the revision.\n\n\n---\n\n>**[Q5]** Results on cost-sensitive MF-HPO: I'd expect FSBO to be better than CFBO because FSBO is not considering any trade-offs but merely performs for maximum performance. Therefore, I found it surprising to see that FSBO performs worse than CFBO throughout all tasks. Maybe the authors could comment on that.\n\n- We kindly note that this is because **performance is measured based on user utility**, which accounts for both accuracy and cost.\n- As you noted, when utility considers only final performance (i.e., the conventional setting with $\\alpha=0$), FSBO performs similarly to CFBO (`Figure 6`). \n- However, in the cost-sensitive setting ($\\alpha>0$), FSBO ignores cost entirely and continues exploration even when the marginal utility gain is low. As a result, it achieves **lower overall utility** than CFBO, which actively trades off performance and cost.\n- We will clarify this distinction in the revision.\n\n---\n\n### References\n\n[1] Bradley, Ralph Allan, and Milton E. Terry. \"Rank analysis of incomplete block designs: I. the method of paired comparisons.\" Biometrika. 1952.\n\n[2] Wilson, Andrew Gordon, et al. \"Deep kernel learning.\" PMLR, 2016.\n\n[3] Rakotoarison, Herilalaina, et al. \"In-context freeze-thaw bayesian optimization for hyperparameter optimization.\" ICML. 2024."}, {"review_id": "DyKFnvyKC3", "reviewer": "Reviewer_Ly5J", "summary": "The paper introduces a cost-sensitive freeze-thaw BO method for HPO (CFBO) that focuses on performance vs. computational cost. The core idea is optimizing a utility function defined on performance and cost.\nThis utility function in general can be user-defined.\nCFBO is built upon ideas from freeze-thaw BO, however, the acquisition function is based on expected utility improvement instead of performance to (dynamically) determine the training length of configurations with high expected utility (performance vs. computational cost trade-off).\nThe authors further propose a regret-based stopping rule defined on the utility to terminate optimization when further improvements become inefficient.\nThe learning curve extrapolation within the freeze-thaw part makes use of PFNs and builds on existing works but implements a mixup-based transfer learning strategy.\nIn their experiments, the authors compare CFBO against baselines and competitors on three multi-fidelity HPO benchmarks and show that CFBO outperforms all baselines in cost-sensitive settings when looking at the regret in utility.\nIn conventional HPO with no cost penalty, CFBO performs competitively.", "questions": "I already reviewed the paper this year when it was submitted to ICML 2025.\nAs far as I can tell, only minor changes were made prior to re-submitting this version.\nI therefore follow up with similar questions I already had when first reviewing:\n\nHow did you select the maximum budget for each benchmark task?\n\nThe expected improvement of utility in 2.2 has the implicit assumption that past performance (and utility) observations are noise-free, correct?\nI.e., in (3), we subtract from the utility function at a future time $b + \\Delta t$ with cumulative best performance $\\tilde{y}_{b + \\Delta t}$ the most recent utility value $U_p$.\nHowever, the expectation only applies to the utility part of the future time and is derived from the predicted learning curve and $U_p$ is treated as deterministic.\n\nIn the checklist regarding limitations you state:\n`Justification: We provide the Limitations paragraph in §4.`\nHowever, this simply references the Conclusion section were limitations are not discussed.\nCan you discuss this in the context of HPO where the objective function is per se not deterministic (i.e., in contrast to the tabular benchmarks with pre-collected data used in the experiments, real-world HPO is stochastic and noisy as we are optimizing an estimate of the generalization performance obtained via model fitting and evaluation based on a resampling)?\n\nI believe it could be beneficial to add a notation table in the appendix.", "limitations": "Limitations and potential negative societal impact are addressed adequately.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nCFBO makes use of solid building blocks to tackle the problem of cost-sensitive multi-fidelity HPO (e.g., freeze-thaw BO, transfer learning for learning curve extrapolation via PFNs).\nBenchmarks are reasonably selected and the set of competitors and baselines is sufficiently large.\nThe authors investigate normalized regret (on utility) and rank for each benchmark.\nThe paper is written well and clarity is high.\nOriginality is given to some extent and the problem CFBO tackles is a relevant one.\nCFBO ties in with the related literature of multi-fidelity HPO, focusing on freeze-thaw BO.\nThe authors correctly identify that existing methods generally do not focus on performance vs. computational cost trade-offs but instead assume a fixed budget.\n\n\n\n### Weaknesses\n\nNovelty is limited given the reuse of existing building blocks (EI, freeze-thaw BO, learning curve extrapolation via PFNs).\nOne notable caveat is that all other baselines and competitors are not naturally designed to consider the trade-off between performance and computational cost so the metric of normalized regret on utility can be considered slightly biased w.r.t CFBO.\nComputational efficiency is not analyzed in detail and detailed wall-clock time comparisons are missing.\nOverall, the method is strongly heuristically motivated.\n\nThe authors do not address recent literature concerned with terminating BO ([1], [2]).\nEspecially [1] is directly tailored to the problem of HPO and stopping is performed if an upper bound on the simple regret (difference between current best validation score and estimated global minimum based on the surrogate model) is less or equal to the estimated variance of the validation error.\nThis stopping criterion is less heuristically motivated than the one proposed by the authors and further does take uncertainty in the objective function in HPO into account.\nIn l738 in the appendix, the authors briefly address [1] but I do not think that this is sufficient.\nMoreover, the paper does not make a strong connection to the general literature of cost-aware BO [3].\n\n[1] Makarova et al. (2021) https://arxiv.org/abs/2104.08166\n\n[2] Wilson (2024) https://arxiv.org/abs/2402.16811\n\n[3] Xie et al. (2024) https://arxiv.org/abs/2406.20062", "was_revised": true, "final_justification": "I do maintain a somewhat reserved stance but can acknowledge several improvements in my understanding of the paper's contributions.\nThe authors have successfully clarified the practical motivation for cost-sensitive HPO (e.g. in cloud computing settings) and addressed runtime concerns by pointing to negligible overhead compared to model training time.\n\nHowever, for me some concerns remain.\nWhile the problem formulation is interesting, the technical approach still remains largely incremental, primarily combining existing methods in a building block fashion, although with some implementation challenges or minor technical challenges that needed to be solved.\nMoreover, the majority of evaluation setting in the paper give CFBO an inherent advantage as the only method naturally operating in the utility-aware setting.\n\nFor me, of concern is also that previously raised issues about connections to early stopping literature were not addressed in this submission, despite the paper being previously reviewed (ICML) with similar feedback.\nWhile the authors have now promised to emphasize this related work more strongly in the appendix, I believe this literature is important enough to be included in the main paper.\nThe authors showed some progress during the rebuttal by implementing FSBO+[8], but the results were (somewhat unexpectedly) poor.\nThis might be due to structural mismatches (the method not operating in the multi-fidelity setting like CFBO, or because the benchmarks only provide simple validation holdout splits, making it challenging to apply early stopping techniques that rely on variance estimates of generalization error without further adjustments).\n\nTherefore, while I have slightly increased my score, I cannot actively push for acceptance.\nOverall, after reading the other reviews and the author responses, I do not have a strong opinion about rejection either because the paper does have its pros and therefore I maintain this borderline rating.", "rebuttal": "We sincerly appreciate you for your time and constructive comments which improve our paper. We respectfully address your concerns as following:\n\n---\n>**[Q1-1]** Limited novelty.\n- We respectfully disagree with this assessment. While freeze–thaw BO, transfer learning, and preference learning have appeared in prior work, CFBO is novel in **how it combines these elements** to address a **new problem**: hyperparameter optimization when users **explicitly trade off model performance against computational cost**.\n- As discussed in `L32–36`, cloud‑service users often work under credit budgets—they care about performance but also penalize HPO cost to conserve resources. Existing methods **either ignore cost or treat it as a hard constraint**, rather than optimizing expected utility. We show that traditional methods perform poorly under such trade-offs.\n- CFBO is designed for this realistic setting. We (1) formulate a **different notion of cost‑aware HPO problem**, (2) define a **utility‑based objective**, (3) explain why **prior methods fall short**, and (4) integrate known tools into **an effective solution**.\n- The contribution lies not just in technical components but in defining and solving this practically motivated problem. We will clarify this in the revision.\n---\n>**[Q1-2]** Baselines not designed for trade-off.\n- Thank you for pointing this out. We already provide a fair comparison under the conventional multi‑fidelity HPO setting with $\\alpha=0$ (`Figure 6`), where utility reduces to pure performance. CFBO still outperforms all baselines, showing its advantage is **not due to privileged knowledge**.\n- Incorporating our utility into existing baselines is **fundamentally difficult**. $U$ depends on future performance–cost trade‑offs and requires full learning curve inference, which DyHPO [1] and FSBO [2] cannot support as they predict only the next or final step.\n- For fairness, we tuned a fixed stopping threshold ($\\delta=0.2$) for all baselines. They cannot use our adaptive threshold $\\delta_b$ since it relies on computing future utility (`Eq. 6`).\n- We also implemente **CFBO‑NT** by augmenting ifBO [3] with our stopping and acquisition rules. It outperforms all baselines without transfer learning, confirming that gains stem from **our methodological contributions**.\n- We will include this in the revision.\n---\n>**[Q1‑3]** Missing runtime analysis.\n- We kindly note that in `L897‑907` of `Appendix E`, we report wall‑clock times for running a full BO process (300 epochs) with DPL, IFBO, and CFBO on LCBench, TaskSet, and PD1.\n- While IFBO is the most efficient, the additional cost of CFBO is **negligible** because model training dominates total runtime.\n- In real‑world HPO experiments (`Appendix E`), we train ResNet‑50 [4], HRNet [5], and MobileNetV2 [6] on 10 RoboFlow100 [7] datasets to collect 500 curves.\n- Across these runs, CFBO adds only 44.4 s to 281.6 s relative to IFBO—trivial compared to training modern networks for 300 epochs.\n---\n>**[Q1-4]** Heuristic motivation.\n- Thank you for pointing this out. While some components (e.g., $\\text{BetaCDF}$ interpolation and adaptive stopping) are heuristic in form, they are derived from an **explicit utility‑maximization objective** and guided by uncertainty estimates over learning curves. `Table 1` provides ablations that isolate each component and confirm its quantitative impact.\n- We also evaluate CFBO under diverse settings, from conventional HPO ($\\alpha=0$) to three utility functions with different penalties ($c\\in\\\\{1,2,0.5\\\\}$ and $\\alpha\\in\\\\{2^{-6},\\ldots,2^{-2}\\\\}$), validating the heuristics at well‑understood extremes. Across four benchmarks (including the experiment in `Appendix F`), CFBO consistently outperforms baselines, demonstrating that these heuristics translate into reliable real‑world gains.\n---\n> **[Q2‑1]** Missing discussion of BO stopping works [8,9].\n- Thank you for this concern. **We respectfully explain below how the stopping rule of Makarova et al. [8] roughly fits within our utility‑based framework**.\n- Proposition 1 in [8] states that, for BO step $b$,$$f^* - f_b \\le 2\\epsilon + \\tilde{y}^* - \\tilde{y}_b,$$\n- where $f$ denotes tbe true population performance, $\\tilde{y}_b$ the validation performance, $*$ a maximizer, and $\\epsilon$ statistical error. The above leads to the **stopping condition** $\\tilde{y}^\\* -\\tilde{y}_b\\le\\epsilon$.  \n- This can be expressed through a utility:$$U(b,\\tilde{y}_b)=\\begin{cases}\\tilde{y}_b&\\text{if }\\tilde{y}_b \\ge\\tilde{y}^* -\\epsilon,\\\\\\ -\\infty &\\text{otherwise}.\\end{cases}$$ \n- Since $\\tilde{y}^\\*$ and $\\epsilon$ are unknown, [8] estimates them via LCB/UCB and CV variance. This estimator is a function of $b$, the above can be rewritten as:$$U(b,\\tilde{y}_b)=\\begin{cases}\\tilde{y}_b&\\text{if }\\tilde{y}_b \\ge c(b),\\\\\\ -\\infty &\\text{otherwise}.\\end{cases}$$\n- Interpreted this way, the preference in [8] is **\"stop once performance exceeds $c(b)$; additional budget has no value thereafter.\"** This represents a special case of our utility formulation.\n- Wilson [9] similarly fits into a utility view:$$U(b,\\tilde{y}_b)=\\begin{cases}\\tilde{y}_b &\\text{if } Pr[r_b\\le \\epsilon|\\mathcal{C}]<1-\\delta,\\\\\\ - c_b &\\text{otherwise},\\end{cases}$$where $r_b=f^*-f_b$ and $c_b$ is cumulative cost.  \n- We will revise `Appendix A` to incorporate these links.\n---\n>**[Q2-2]** Link to cost‑aware BO [10].\n- Xie et al. [10] also pursue a cost‑sensitive objective, but their Gittins‑index policy is designed for single‑fidelity black‑box BO, whereas our method predicts full learning curves and uses an adaptive utility‑based stopping rule in the multi‑fidelity HPO setting. We will add this clarification to `Appendix A`.\n---\n>**[Q3]** Choice of maximum budget.\n- In the multi‑fidelity BO literature there is no fixed rule for the maximum budget, though many studies adopt 1000 epochs/evaluations.\n- We set the cap to 300 to reflect cloud‑credit limits and highlight early stopping. CFBO‑NT still outperforms baselines, showing gains are not solely from transfer learning.\n---\n>**[Q4]** EI assumes noise‑free past utility $U_p$.\n- Yes, we follow the classical Expected Improvement assumption: the best utility observed so far, $U_p$, is treated as a fixed constant, and uncertainty is modeled only for the future term $U(b+\\Delta t,\\tilde{y}_{b+\\Delta t})$.\n- If $U_p$ were noisy, one could adopt noisy‑EI variants that integrate over its posterior [11,12]; we keep the deterministic form for simplicity.\n---\n>**[Q5]** Limitations paragraph missing.\n- We will add a **Limitations** subsection in `§4`. Our PFN‑based extrapolator works on discrete steps and may falter with continuous budgets and noise; observation noise can destabilise utility and stopping. We will explicitly acknowledge these points.\n---\n>**[Q6]** Need notation table.\n- We sincerely agree with this suggestion and will include the provided notation table in `Appendix`.\n\n|Notation|Description|\n|-|-|\n|$x_n\\in\\mathbb{R}^{d_x}$|$n$‑th hyperparameter configuration|\n|$\\mathcal{X}=\\\\{x_1,\\ldots,x_N\\\\}$|Set of hyperparameter configurations|\n|$T\\in\\mathbb{N}$|Maximum training epochs for each configuration|\n|$t\\in[T]:=\\\\{1,\\ldots,T\\\\}$|Training‑epoch index|\n|$y_{n,t}\\in[0,1]$|Performance of $x_n$ at epoch $t$|\n|$B\\in\\mathbb{N}$|Maximum total training epochs (overall BO budget)|\n|$b\\in[B]$|Budget spent so far|\n|$\\tilde{y}_b$|Best performance observed up to budget $b$|\n|$\\mathcal{C}=\\{(x,t,y)\\}$|Collected partial or full learning curves|\n|$q_\\theta:\\mathbb{R}^{d_x}\\times[T]\\times\\mathcal{C}\\to[0,1]$|Learning‑curve extrapolator|\n|$U:[B]\\times[0,1]\\to\\mathbb{R}_{\\ge0}$|Utility function|\n|$U_p\\in\\mathbb{R}_{\\ge0}$|Utility immediately before current BO step|\n|$A(\\cdot;U):[N]\\to\\mathbb{R}$|Acquisition function (`Eq. 3`)|\n|$\\hat{U}_{\\text{max}}$|Maximum utility observed so far|\n|$\\hat{U}_{\\text{min}}:=U(B,\\tilde{y}_1)$|Approximated minimum utility|\n|$\\hat{R}\\_b:=\\frac{\\hat{U}_{\\text{max}} - U_p}{\\hat{U}\\_\\text{max} - \\hat{U}\\_\\text{min}}\\in [0,1]$|Normalized regret|\n|$\\delta\\in[0,1]$|Fixed stopping threshold for baselines|\n|$\\delta_b:=\\text{BetaCDF}(p_b,\\beta,\\beta)^\\gamma\\in[0,1]$|Adaptive stopping threshold|\n|$p_b\\in[0,1]$|Probability that current configuration will improve $U_p$ (Eq. 6)|\n|$\\text{BetaCDF}(\\cdot;\\beta,\\beta):[0,1]\\to[0,1]$|Beta CDF with shape $\\beta$|\n|$\\beta\\in\\mathbb{R}_{>0}$|Hyperparameter controlling interpolation|\n|$\\gamma\\in\\mathbb{R}_{>0}$|Hyperparameter setting $\\delta_b$ at $p_b$|\n---\n### References\n[1] Wistuba, Martin, Arlind Kadra, and Josif Grabocka. \"Supervising the multi-fidelity race of hyperparameter configurations.\" NeurIPS. 2022.\n\n[2] Wistuba, Martin, and Josif Grabocka. \"Few-shot Bayesian optimization with deep kernel surrogates.\" arXiv. 2021.\n\n[3] Rakotoarison, Herilalaina, et al. \"In-context freeze-thaw bayesian optimization for hyperparameter optimization.\" ICML. 2024.\n\n[4] He, Kaiming, et al. \"Deep residual learning for image recognition.\" CVPR. 2016.\n\n[5] Wang, Jingdong, et al. \"Deep high-resolution representation learning for visual recognition.\" IEEE transactions on pattern analysis and machine intelligence. 2020.\n\n[6] Sandler, Mark, et al. \"Mobilenetv2: Inverted residuals and linear bottlenecks.\" CVPR. 2018.\n\n[7] Ciaglia, Floriana, et al. \"Roboflow 100: A rich, multi-domain object detection benchmark.\" arXiv. 2022.\n\n[8] Makarova, Anastasia, et al. \"Automatic termination for hyperparameter optimization.\" International Conference on Automated Machine Learning. PMLR, 2022.\n\n[9] Wilson, James. \"Stopping Bayesian optimization with probabilistic regret bounds.\" NeurIPS. 2024.\n\n[10] Xie, Qian, et al. \"Cost-aware Bayesian optimization via the Pandora's Box Gittins index.\" NeurIPS. 2024.\n\n[11] Picheny, Victor, Tobias Wagner, and David Ginsbourger. \"A benchmark of kriging-based infill criteria for noisy optimization.\" Structural and multidisciplinary optimization. 2023.\n\n[12] Letham, Benjamin, et al. \"Constrained Bayesian optimization with noisy experiments.\" 2019."}, {"review_id": "Po1OZjjmRm", "reviewer": "Reviewer_Vxb6", "summary": "The paper describes a freeze-thaw BO algorithm for hyper-parameter optimization that optimizes the utility, which is a measure of the user's internal tradeoff between cost and performance.", "questions": "See above re my (mis)understanding of equation (2).", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The central idea behind this paper is solid and the paper is well-written for the most part. The paper is well structured - relevant background material is introduced systematically, the algorithm described clearly and the experimental verification appears reasonable.\n\nIt strikes me that clarity has been somewhat compromised to fit within the page limit in some places. For example I am not at all familiar with PFNs, so ideally I would like a more detailed exposition to be presented: however I understand that this may be incompatible with page limits. Also the figures in the results section are basically unreadable, but again I understand that this is almost certainly due to page limits, so I will overlook this.\n\nMore seriously equation (2) is not clear to me. If I understand correctly, $u_1$ and $u_2$ are pairs $(b,\\tilde{y}_n)$, which appears incompatible with exponentials and inversions as per the equation. This seems central to the paper - could you please clarify?", "was_revised": true, "final_justification": "My main point of uncertainty in the initial reviews was Q2. The authors response covers this, so I am happy to increase my recommendation to accept.", "rebuttal": "We sincerly appreciate your constructive comments to improve our paper, and the positive review. We address your concerns in the following:\n\n---\n\n>**[Q1]** It strikes me that clarity has been somewhat compromised to fit within the page limit in some places. For example I am not at all familiar with PFNs, so ideally I would like a more detailed exposition to be presented: however I understand that this may be incompatible with page limits. \n- We sincerely apologize for the insufficient explanation of essential preliminaries. \n- To improve clarity and ensure self-containedness, we include a more detailed exposition of the PFN-based learning curve extrapolation mechanism in `Appendix D`.\n- We will revise the main text to include a pointer:  \n  \"Please refer to Appendix D for the detailed mechanism of learning curve extrapolation.\"\n\n\n>**[Q2]** Also the figures in the results section are basically unreadable, but again I understand that this is almost certainly due to page limits, so I will overlook this.\n- We sincerly apologize you about the unreadbility of experimental results, espeically `Figure 6`.\n- We ensure you that we will try our best to improve the clarity of experimental results in the revision by making room for the figures.\n\n---\n\n>**[Q2]** More seriously equation (2) is not clear to me. If I understand correctly, $u_1$ and $u_2$ are pairs $(b, \\tilde{y}_n)$, which appears incompatible with exponentials and inversions as per the equation. This seems central to the paper - could you please clarify?\n\n- We sincerely apologize for the confusion. `Eq. 2` defines the objective for fitting the utility function using **preference learning** via the Bradley–Terry model [1].\n- In this setup, we are given a pair of outcomes—e.g., $(b_1, y_1)$ and $(b_2, y_2)$, where $b_1, b_2 \\in [B]$ and $y_1, y_2\\in[0,1]$—and a corresponding preference label indicating which is preferred.\n- Let $u_1 := U(b_1, y_1) \\in [0,1]$ and $u_2 := U(b_2, y_2) \\in [0,1]$ denote the utility scores of these two outcomes. These are **scalar** values derived from a parametric utility function.\n- The Bradley–Terry model is then trained, i.e., minimize `Eq. 2` w.r.t the parameters of  $U$ (e.g., $\\alpha$), on such pairs $(u_1, u_2)$ with preference labels ($u_1>u_2$), and the exponential/logistic form in `Eq. 2` applies directly to these scalar utilities—not the raw pairs.\n- We will clarify this in the revision.\n\n---\n\n### References\n\n[1] Bradley, Ralph Allan, and Milton E. Terry. \"Rank analysis of incomplete block designs: I. the method of paired comparisons.\" Biometrika. 1952."}]

Author rebuttal:

Meta-review: This work builds on freeze-thaw Bayesian optimization, developing a cost-sensitive version that early stops non-promising hyperparameter configurations. The authors propose to leverage an utility function that is a function of performance and cost, which they use in conjunction with a dedicated acquisition function and a regret-based stopping criterion. The authors provided solid empirical evidence to support the main claims of the paper.

The authors provided an extensive rebuttal, addressing the questions and concerns raised by the reviewers. The authors also captured the key improvements to incorporate into the final version of the manuscript. The work is timely and the authors addressed runtime concerns in a satisfactory manner by pointing to the negligible overhead compared to model training time.

While the work builds on a combination of existing approaches, it does so in an effective and novel way as acknowledged by most reviewers. One reviewer remained more critical, indicating that the early stopping literature was not well represented in the original submission. The authors provided additional data points during the rebuttal to fill that gap. On balance, I am in agreement with one of the reviewers indicating that the paper has included enough relevant baselines to warrant acceptance.
Decision: Accept (poster)