Cost-Sensitive Multi-Fidelity Bayesian Optimization

## Abstract

Abstract In this paper, we address the problem of cost-sensitive multi-fidelity Bayesian Optimization (BO) for efficient hyperparameter optimization (HPO). Specifically, we assume a scenario where users want to early-stop the BO when the performance improvement is not satisfactory with respect to the required computational cost. Motivated by this scenario, we introduce utility , which is a function predefined by each user and describes the trade-off between cost and performance of BO. This utility function, combined with our novel acquisition function and stopping criterion, allows us to dynamically choose for each BO step the best configuration that we expect to maximally improve the utility in future, and also automatically stop the BO around the maximum utility. Further, we improve the sample efficiency of existing learning curve (LC) extrapolation methods with transfer learning, while successfully capturing the correlations between different configurations to develop a sensible surrogate function for multi-fidelity BO. We validate our algorithm on various LC datasets and found it outperform all the previous multi-fidelity BO and transfer-BO baselines we consider, achieving significantly better trade-off between cost and performance of BO.

## 1 Introduction

Hyperparameter optimization (HPO) [, , , , , , ] stands as a crucial challenge in the domain of deep learning, given its importance of achieving optimal empirical performance. Unfortunately, the field of HPO for deep learning remains relatively underexplored, with many practitioners resorting to simple trial-and-error methods [, ]. Moreover, traditional black-box Bayesian optimization (BO) approaches for HPO [, , ] have limitations when applied to deep neural networks due to the impracticality of evaluating a vast number of hyperparameter configurations until convergence, each of which may take several days.

Recently, multi-fidelity (or gray-box) BO [, , , , , , , ] receives more attention to improve the sample efficiency of traditional black-box BO. Multi-fidelity BO makes use of lower fidelity information (e.g., validation accuracies at fewer training epoches) to predict and optimize the performances at higher or full fidelities (e.g., validation accuracies at the last training epoch). Unlike black-box BO, multi-fidelity BO dynamically select hyperparameter configurations even before finishing a single training run, demonstraing its ability of finding better configurations in a more sample efficient manner than black-box BO.

However, one critical limitation of the conventional multi-fidelity BO frameworks is that they are not aware of the trade-off between the cost and performance of BO. For instance, some users may want to strongly penalize the cost of BO with respect to its performance, and in this case the BO process should focus on exploiting the current belief on good hyperparameter configurations than trying to explore new configurations. Yet, existing multi-fidelity BO methods tend to over-explore because they usually assume a sufficiently large budget for the BO and aim to obtain the best asymptotic performance on a validation set, hence are not able to properly penalize the cost [, ]. One may argue that we could lower the total BO budget and maximize the performance at that specific budget, but in practice it is hard to specify the target budget in advance as it is difficult to accurately predict the future performances of BO. Rakotoarison et al. [] propose to randomize the target budget, but it implicitly assumes that a single optimal configuration can achieve strong performances at any stages of training, which is unrealistic [] (e.g., regularization).

Figure: (a)
Refer to caption: x1.png

Therefore, in this paper we introduce a more sophisticated notion of cost-sensitivity for multi-fidelity BO. Specifically, we assume that it is easier to specify the trade-off between cost and performance of multi-fidelity BO, than to know the proper target budget in advance or resort to the unrealistic assumption mentioned above. We call this trade-off *utility*. It is a function predefined by each user and describes users’ own preferences about the trade-off. It has higher values as cost decreases and performance increases, and vice versa (Fig. 1(a)). Also, some utility functions may strongly penalize the amount of BO budgets spent, while others may weakly penalize or not penalize at all as with the conventional multi-fidelity BO. We explicitly maximize this utility by dynamically selecting hyperparamter configurations which are expected to maximally improve it in future (Fig. 1(b)), and also automatically terminating the BO around the maximum utility (Fig. 1(a)), instead of terminating at an arbitrary target budget.

Solving this problem requires our multi-fidelity BO frameworks to have the following capabilities. Firstly, they should support freeze-thaw BO [, ], an advanced form of multi-fidelity BO in which we can dynamically pause (freeze) and resume (thaw) hyperparameter configurations based on future performances extrapolated from a set of partially observed learning curves (LCs) with various configurations. Such efficient and sensible allocation of computational resources suits well for our purpose of finding the best trade-off between cost and performance of multi-fidelity BO.
Secondly, freeze-thaw BO requires its surrogate function to be equipped with a good LC extrapolation mechanism [, , ]. In our case, it is crucial for making a good probabilistic inference on future utilities with which we dynamically select the best configuration and accurately early-stop the BO. Lastly, since we assume that users want to stop the BO as early as possible, LC extrapolation should be accurate even at the early stages of BO. Therefore, we should make use of transfer learning to maximally improve the sample efficiency of BO [] and to prevent inaccurate early-stopping when there are only few or even no observations in the BO.

Based on those criteria, we introduce our novel Cost-sensitive Multi-fidelity BO (CMBO) that makes use of transfer learning and in-context LC extrapolation. We first introduce the acquisition function and stopping criteria specifically developed for our framework, and explain how to achieve with them a good trade-off between cost and performance of multi-fidelity BO (§3.2).
Also, based on a recently introduced Prior-Fitted Networks (PFNs) [, ] for in-context Bayesian inference, we explain how to train a PFN with the existing LC datasets to develop a sample efficient in-context surrogate function for freeze-thaw BO that can also effectively capture the correlations between different hyperparameter configurations (§3.3). Lastly, we empirically demonstrate the superiority of CMBO on a set of diverse user preferences and three HPO benchmarks, showing that it significantly outperforms all the previous multi-fidelity BO and the transfer-BO baselines we consider (§4).

## 2 Related Work

#### Multi-fidelity HPO.

Unlike traditional black-box approaches for HPO [, , , , , , , ], multi-fidelity (or gray-box) HPO aims to optimize hyperparameters in a sample efficient manner by utilizing low fidelity information (e.g., validation set performances with smaller training dataset) as a proxy for higher or full fidelities [, , , ], dramatically speeding up the HPO. In this paper, we focus on making use of performances at fewer training epochs to better predict/optimize the performances at longer training epochs. One of the well-known examples is Hyperband [], a bandit-based method that randomly selects a set of random hyperparameter configurations, and stops poorly performing ones using successive halving [] even before reaching the last training epoch. While Hyperband shows much better performance than random search [], its computational or sample efficiency can be further improved by replacing random sampling of configurations with Bayesian optimization [], adopting evolution strategy to promote internal knowledge transfer [], or making it asynchronously parallel [].

#### Freeze-thaw BO.

Whereas the form of knowledge transfer of Hyperband (and its variants) from lower to higer fidelity is indirect, freeze-thaw BO [] transfers knowledge more directly by explicitly modeling a GP kernel to jointly model interactions between different training budgets and configurations. It then dynamically pauses (freezes) and resumes (thaws) configurations based on the last epoch performances extrapolated from a set of partially observed LCs obtained from other configurations, leading to an efficient and sensible allocation of computational resources. DyHPO [] and its transfer version [] improve the computational efficiency of freeze-thaw BO [] with deep kernel GP [], but their multi-fidelity version of expected improvement (EI) acquisition extrapolates the LCs only a one-step forward, producing a greedy strategy for dynamically selecting configurations. Other recent variants of freeze-thaw BO include DPL [] and ifBO [] which are not greedy and show competitive performances, but they are either lack of transfer learning [, ], resort to too strong assumptions on the shape of LCs [], or incur the cost of retraining the surrogate function for each BO step []. On the other hand, we maximize the sample efficiency of freeze-thaw BO with transfer learning, while getting rid of any need for such strong assumptions or retraining costs.

#### Learning curve extrapolation.

Freeze-thaw BO requires the ability of dynamically updating predictions on future performances from a growing set of partially observed LCs, thus heavily relies on the ability of LC extrapolation [, , ]. The LC extrapolation used in DyHPO [] and Quick-Tune [] is based on deep kernel GP [], but extrapolates only a single step forward, making it hard to be used for our case - maximizing utilites at any possible future time steps. Freeze-thaw BO [] and DPL [] use non-greedy extrapolations but limit the shape of LCs with exponential decay kernel or power law functions, which is questionable if such strong inductive biases are applicable to more general cases. Domhan et al. [] consider a broader set of basis functions to define a prior on LCs and infer its posterior, but requires computationally expensive MCMC, and also do not consider correlations between different configurations. Klein et al. [] models interactions between configurations with a Bayesian neural network (BNN), but suffers from the same computational inefficiency of MCMC and also the additional cost for online retraining of the BNN. LC-PFNs [] are an in-context Bayesian LC extrapolation method without retraining, based on recently introduced Prior-Fitted Networks (PFNs) []. However, as with Domhan et al. [], LC-PFNs do not consider interactions between configurations and hence is suboptimal to use as a surrogate function for freeze-thaw BO. Recently, Rakotoarison et al. [] further combine LC-PFN with PFN4BO [] to develop an in-context surrogate function for freeze-thaw BO, but they train PFNs only with a prior distribution similarly to the original PFN training scheme. On the other hand, we use transfer learning, i.e., train PFNs with the existing LC datasets, to improve the sample efficiency of freeze-thaw BO while successfully encoding the correlations between configurations at the same time.

#### Transfer BO.

Transfer learning can be used for improving the sample efficiency of BO [], and here we list a few of them. Some of recent works explore scalable transfer learning with deep neural networks [, ]. Also, different components of BO can be transferred such as observations [], surrogate functions [, ], hyperparmater initializations [], or all of them []. However, most of the existing transfer-BO approaches assume the traditional black-box BO settings. To the best of our knowledge, Quick-Tune [] is the only recent work which targets multi-fidelity and transfer BO at the same time. However, as mentioned above, their multi-fidelity BO formulation is greedy, whereas our transfer-BO method can dynamically control the degree of greediness during the BO by explicitly taking into consideration the trade-off between cost and performance of BO.

#### Cost-sensitive BO.

Multi-fidelity BO
is inherently cost-sensitive since predictions get more accurate as the gap between the fidelities becomes closer. However, the performance metric of such vanilla multi-fidelity BO monotonically increases as we spend more budget, whereas in this paper we want to find the optimal trade-off between the amount of budget spent thus far and the corresponding intermediate performances of BO, thereby automatically early-stopping the BO around the maximal utility. Quick-Tune [] also suggests a cost-sensitive BO in multi-fidelity settings, but unlike our work, their primary focus is to trade-off between the performance and the cost of BO associated with pretrained models of various size, which can be seen as a generalization of more traditional notion of cost-sensitive BO [, , ], from black-box to multi-fidelity settings.

## 3 Approach

In this section, we introduce CMBO, a novel framework for cost-sensitive multi-fidelity BO. We first introduce notation and background on freeze-thaw BO in §3.1. We then introduce the overall method and algorithm in §3.2, and explain the transfer learning of surrogate function in §3.3.

### 3.1 Notation and Background on freeze-thaw BO

#### Notation.

Following the convention, we assume that we are given a finite pool of hyperparameter configurations $\mathcal{X}=\{x_{1},\dots,x_{N}\}$, with $N$ the number of configurations. Let $t\in[T]:=\{1,\dots,T\}$ denote the training epochs, $T$ the last epoch, and $y_{n,1},\dots,y_{n,T}$ the validation performances (e.g., validation accuracies) obtained with the configuration $x_{n}$.
We further introduce notations for multi-fidelity BO. Let $b=1,\dots,B$ denote the BO steps, $B$ the last BO step, and $\tilde{y}_{1},\dots,\tilde{y}_{B}$ the BO performances, i.e., each $\tilde{y}_{b}$ is the best validation performance ($y$) obtained until the BO step $b$.

#### Freeze-thaw BO.

Freeze-thaw BO [] is an advanced form of multi-fidelity BO. At each BO step, it allows us to dynamically select and evalute the best hyperparameter configuration $x_{n^{*}}$ with $n^{*}\in[N]$ denoting the corresponding index, while pausing the evaluation on the previous best configuration. Specifically, given $\mathcal{C}=\{(x,t,y)\}$ that represents a set of partial LCs collected up to a specific BO step, we predict for all $x\in\mathcal{X}$ the remaining part of the LCs up to the last training epoch $T$ with a (pretrained) LC extrapolator, compute the acquisition such as the expected improvement [] of validation performance at epoch $T$, and select and evaluate the best configuration $x_{n^{*}}$ that maximizes the acquisition. Note that at any BO step, the partial LCs in $\mathcal{C}$ can have different length across the configurations. Suppose that at BO step $b$ the next training epoch for $x_{n^{*}}$ is $t_{n^{*}}$. We then evaluate $x_{n^{*}}$ a single epoch from the corresponding checkpoint to obtain the validation performance $y_{n^{*},t_{n^{*}}}$ at the next epoch $t_{n^{*}}$, which we use to update the corresponding partial LC in $\mathcal{C}$ and compute the BO performance $\tilde{y}_{b}$. We repeat this process $B$ times until convergence. See Alg. 1 for the pseudocode (except the red parts).

### 3.2 Cost-sensitive Multi-fidelity BO

We next introduce our main method and algorithm based on freeze-thaw BO (§3.1).

#### Utility function.

A utility function $U$(^1^11In this paper we assume that the utility function is predefined by a user. One can instead try to learn it from data, but we leave that as a future work.) describes the trade-off between the BO step $b$ and the BO performance $\tilde{y}_{b}$. Its values $U(b,\tilde{y}_{b})$ negatively correlate with $b$ and positively with $\tilde{y}_{b}$. For instance, we may simply define $U(b,\tilde{y}_{b})=\tilde{y}_{b}-\alpha b$ for some $\alpha>0$, such that the utility gives linear incentives and penalties to the performance and number of BO steps, respectively.

#### Acquisition function.

Let $t_{n}$ be the next training epoch for the configuration $x_{n}$ at a BO step $b$.
Further, suppose we have a LC extrapolator $f(\cdot|x_{n},\mathcal{C})$ that can probabilistically estimate $x_{n}$’s remaining part of LC, $y_{n,t_{n}:T}$, conditioned on $\mathcal{C}$ a set of partial LCs collected up to the step $b$.
Then, based on the expected improvement (EI) [], we define the acquisition function $A$ as follows:

$$ $\displaystyle A(n)=\max_{\Delta t\in\{0,\dots,T-t_{n}\}}\mathbb{E}_{y_{n,t_{n} :T}\sim f(\cdot|x_{n},\mathcal{C})}\left[\max\left(0,U(b+\Delta t,\tilde{y}_{b +\Delta t})-U_{\text{prev}}\right)\right].$ (1) $$

In Eq. (1), we first extrapolate $y_{n,t_{n}:T}$, the remaining part of the LC associated with $x_{n}$, and compute the corresponding predictive BO performances $\{\tilde{y}_{b+\Delta t}\ |\ \Delta t=0,\dots,T-t_{n}\}$.
Note that according to the definition in §3.1, $\tilde{y}_{b+\Delta t}$ is computed by taking the maximum among the last step BO performance $\tilde{y}_{b-1}$ as well as the newly extrapolated validation performances $y_{n,t_{n}},\dots,y_{n,t_{n}+\Delta t}$. Then, based on the increased BO step $b+\Delta t$ and the updated BO performance $\tilde{y}_{b+\Delta t}$, we compute the corresponding utility, and its expected improvement over the previous utility $U_{\text{prev}}$ over the distribution of LC extrapolation with the Monte-Carlo (MC) estimation. The acquisition $A(n)$ for each configuration index $n$ is defined by picking the best increment $\Delta t\in\{0,\dots,T-t_{n}\}$ that maximizes the expected improvement, and we eventually choose the best configuration index $n$ that maximizes $A$ (see Fig. 1(b)).

Figure: Algorithm 1 Cost-sensitive Multi-fidelity BO

The main differences of our acquisition function in Eq. (1) from the EI-based acquisitions used in the previous works are twofold. First, instead of maximizing the expected improvement of validation performance $y$, we maximize the expected improvement of utility. Second, rather than setting the target epoch at which we evaluate the acquisition to the last epoch $T$, we dynamically choose the optimal target epoch that is expected to maximally improve the utility.

Those aspects allow our BO framework to more carefully select configurations for each BO step, seeking the best trade-off between cost and performance of BO.
Specifically, the acqusition function initially prefers configurations that are expected to produce good asymptotic validation performances, but as the BO proceeds it will gradually become greedy as the performance of BO saturates and the associated cost dominates the utility function. As a result, the acquisition function will tend to exploit more than explore – it will try to avoid selecting new configurations but stick to the few current configurations to maximize the short term performances.

Note that $U_{\text{prev}}$ in Eq. (1), the threshold of EI, is *not* the greatest utility achieved so far, but simply set to the utility value achieved most recently (line 11 in Alg. 1).
This is because the cost of BO that has previously been incurred is not reversible.
It differs from the typical EI-based BO settings where we assume all the previous evaluations are freely accessible and set the threshold to the maximum among them.
As a result, $U_{\text{prev}}$ can either increase or decrease during the BO, and we need to stop the BO when $U_{\text{prev}}$ starts decreasing monotonically, i.e., when the performance of BO stops improving meaningfully with respect to the associated cost.

Figure: Figure 2: Eq. (3) with $\gamma=\log_{2}5$ and the various values for $\beta$.
Refer to caption: x3.png

#### Stopping criterion.

The next question is how to properly stop the BO around the maximum utility.
We propose to stop when the following criterion is satisfied at each BO step $b>1$:

$$ $\displaystyle\frac{\hat{U}_{\text{max}}-U_{\text{prev}}}{\hat{U}_{\text{max}}- \hat{U}_{\text{min}}}>\delta_{b}.$ (2) $$

In Eq. (2), $U_{\text{prev}}$ is the utility value at the last step $b-1$, $\hat{U}_{\text{max}}$ is defined as the maximum utility value seen up to the last step, and $\hat{U}_{\text{min}}=U(B,\tilde{y}_{1})$. The role of $\hat{U}_{\text{max}}$ and $\hat{U}_{\text{min}}$ is to roughly estimate the maximum and the minimum utility achievable over the course of BO, respectively. Therefore, the LHS of Eq. (2) can be seen as the *normalized regret* of utility roughly estimated at the current step $b$, and we stop the BO as soon as the current estimation on the regret exceeds some threshold $\delta_{b}$.

To define $\delta_{b}$, let $n^{*}=\operatorname*{arg\,max}_{n}A(n)$ denote the index of the currently chosen best configuration $x_{n^{*}}$ based on Eq. (1), $\operatorname{BetaCDF}$ the cumulative distribution function (CDF) of $\operatorname{Beta}$, and $\mathbbm{1}$ the indicator function. Then, we have:

$$ $\displaystyle\delta_{b}$ $\displaystyle=\operatorname{BetaCDF}(p_{b};\beta,\beta)^{\gamma},\quad\beta, \gamma>0,$ (3) $\displaystyle p_{b}$ $\displaystyle=\max_{\Delta t\in\{1,\ldots,T-t_{n^{*}}\}}\mathbb{E}_{y_{n,t_{n^ {*}}:T}\sim f(\cdot|x_{n^{*}},\mathcal{C})}\left[\mathbbm{1}\left(U\left(b+ \Delta t,\tilde{y}_{b+\Delta t}\right)>U_{\text{prev}}\right)\right].$ (4) $$

$p_{b}$ in Eq. (4) is the probability that the current best configuration $x_{n^{*}}$ improves on $U_{\text{prev}}$ in some future BO step (i.e., probability of improvement, or PI []). Intuitively, we want to defer the termination as $p_{b}$ increases, and vice versa. It is considered in Eq. (3) – as $p_{b}$ increases, the threshold $\delta_{b}$ increases as well because $\operatorname{BetaCDF}(\cdot;\beta,\beta)^{\gamma}$ is a monotonically increasing function in $[0,1]$, so we have less motivation to stop according to Eq. (2).

Fig. 2 plots $\operatorname{BetaCDF}(\cdot;\beta,\beta)^{\gamma}$ in Eq. (3) over the various values of $\beta$ and when $\gamma=\log_{2}5$. We see that the function becomes vertical as $\beta\rightarrow+\infty$ and horizontal as $\beta\rightarrow 0$. In the former case, we terminate the BO when $p_{b}<0.5$ while ignoring the regret in the LHS of Eq. (2), whereas in the latter case we ignore $p_{b}$ and only decide based on the regret, with the threshold $\delta_{b}$ fixed to a specific value specified by $\gamma$ (e.g., $\gamma=\log_{2}5$ corresponds to $\delta_{b}=0.2$ in Fig. 2). Therefore, the role of $\beta$ is to smoothly interpolate between the two extreme stopping criteria, whereas $\gamma$ decides the range of the overall shape of the interpolated criterion.

#### Algorithm.

We summarize the pseudocode of our overall method in Alg. 1, with the red parts corresponding to the specifics of our method.

### 3.3 Transfer Learning of LC
Extrapolation

As explained above, our BO framework is based on freeze-thaw BO (§3.1)
which heavily resorts to accurate LC extrapolation. Since we assume that users may want to early-stop the BO, we should have an even more sample efficient LC extrapolation mechanism for preventing inaccurate early-stopping before collecting a sufficient amount of BO observations. Therefore, we propose to make use of transfer learning to maximally improve the sample efficiency of the LC extrapolator.

#### Transfer learning with LC mixup.

Among many plausible options, in this paper we propose to use Prior Fitted Networks (PFNs) [] for LC extrapolation. PFNs are an in-context Bayesian inference method based on Transformer architectures [], and show good performances on LC extrapolation [, ] without the computationally expensive online retraining []. A major difficulty of using PFNs for our purpose is that they are not trained with the existing datasets, but trained with a prior distribution, which is essential to perform Bayesian inference. Also, PFNs require relatively a large Transformer architecture [] as well as huge amounts of training examples for good generalization performance [], which makes it risky to train PFNs with a finite set of examples.

Here we explain our novel transfer learning method for PFNs that can circumvent those difficulties with the mixup strategy []. Suppose we have $M$ different datasets and the corresponding $M$ sets of LCs collected from $N$ hyperparameter configurations. Define $l_{m,n}=(y^{m}_{n,1},\dots,y^{m}_{n,T})$, the $T$-dimensional row vector of validation performances ($y$’s) collected from the $m$-th dataset and the $n$-th configuration, forming a complete LC of length $T$. Further define the matrix $L_{m}=[l_{m,1}^{\top};\dots;l_{m,N}^{\top}]^{\top}$, the row-wise stack of those LCs. In order to augment training examples, we propose to perform the following two consecutive mixups []:

- 1.
Across datasets: $L^{\prime}=\lambda_{1}L_{m}+(1-\lambda_{1})L_{m^{\prime}}$, with $\lambda_{1}\sim\operatorname{Unif}(0,1),\quad\text{for all }m,m^{\prime}\in[M]$.
- 2.
Across configurations: $(x^{\prime\prime},l^{\prime\prime})=\lambda_{2}(x_{n},l^{\prime}_{n})+(1-
\lambda_{2})(x_{n^{\prime}},l^{\prime}_{n^{\prime}})$
.                          with $l^{\prime}_{n}\text{ the }n\text{-th row of }L^{\prime},\ \ \lambda_{2}\sim
\operatorname{Unif}(0,1),\quad\text{for all }L^{\prime}\text{ and }n,n^{\prime
}\in[N]$.

In this way, we can sample infinitely many training examples $\{(x^{\prime\prime},l^{\prime\prime})\}$ by interpolating between the LCs, leading to a robust LC extrapolator with less overfitting.
Note that in the first step, we do not individually perform the mixup over the configurations but apply the same $\lambda_{1}$ to all the configurations, in order to preserve the correlations between the configurations encoded in the given datasets.

As for the network architecture and the training objective, we mostly follow Rakotoarison et al. []. We use a similar Transformer architecture that takes a set of partial LCs and the corresponding configurations as an input and extrapolates the remaining part of the LCs. The training objective then maximizes the likelihood of those predictions conditioned on the partial LCs. We defer more details on the training to §B. Also, see §D for more discussion about the connection of our transfer learning method with ifBO [] and Transformer Neural Processes (TNPs) [].

## 4 Experiments

We next validate the efficacy of our method on various multi-fidelity HPO settings. We will publicly release our code upon acceptance.

Figure: Figure 3: The results on the conventional multi-fidelity HPO setup ($\alpha=0$). For each benchmark, we report the normalized regret of utility aggregated over all the test datasets.
Refer to caption: extracted/5621347/figures/lcbench.jpg

#### Datasets.

We use the following benchmark datasets for multi-fidelity HPO. LCBench []: A LC dataset that evaluates the performance of 7 different hyperparameters on 35 different tabular datasets. The LCs are collected by training MLPs with 2,000 hyperparameter configurations, each for 51 epochs. We train our LC prediction model on 20 datasets and evaluate on the remaining 15 datasets. TaskSet []: A LC dataset that consists of a diverse set of 1,000 optimization tasks drawn from various domains. We select 30 natural language processing (text classification and language modeling) tasks, train our LC extrapolator on 21 tasks, and evaluate on the remaining 9 tasks.
Each task include 8 different hyperparameters and 1,000 their configurations. Each LC is collected by training models for 50 epochs. PD1 []: A LC benchmark that includes the performance of modern neural architectures (including Transformers) run on large vision datasets such as CIFAR-10, CIFAR-100 [], ImageNet [], as well as statistical modeling corpora and protein sequence datasets from bioinformatics. We select 23 tasks with 4 different hyperparameters based on SyneTune [] package, train our LC extrapolator on 16 tasks, and evaluate on the remaining 7 tasks. For easier transfer learning, we preprocess the data by excluding hyperparameter configurations with their training diverging (e.g., LCs with NaN), and linearly interpolate the LCs to match their length across different tasks. We then obtain the LCs of 50 epochs over the 240 configurations. See §A for more details.

#### Baselines.

We compare our method against Random Search [] that randomly selects hyperparameter configurations sequentially. We next compare against several variants of Hyperband [] such as
ASHA [] the asynchronous parallel version of it, BOHB [] which replaces its random sampling of configurations with BO, and DEHB [] which promotes internal knowledge transfer with evolution strategy. We also compare against more recent multi-fidelity BO methods such as DyHPO [] which uses deep kernel GP [] and a greedy acquisition function with a short-horizon LC extrapolation, and DPL [] which extrapolates LCs with power law functions and model ensemble. Quick-Tune^†, is a modified version of Quick-Tune [] which is originally developed for dynamically selecting both pretrained models and hyperparmater configurations, with the additional cost term penalizing the non-uniform evaluation wall-time associated with each joint configuration. Since our experimental setup does not consider selecting pretrained models nor non-uniform evaluation wall-time, we only leave the transfer learning part of the model, which corresponds to a transfer learning version of DyHPO, i.e., we train its surrogate function with the same LC datasets used for training our LC extrapolator. Lastly, we compare against FSBO [], a black-box transfer-BO method that uses the same LC datasets to train a deep kernel GP surrogate function. The difference of FSBO from Quick-Tune^† is that its surrogate models the validation performances at the last epoch, whereas the surrogate of Quick-Tune^† predicts the performances at the next epoch for multi-fidelity HPO. See §C for more details.

#### Utility function.

While there are many plausible options for the utility function, in this paper we use a linear function for penalizing the cost of multi-fidelity BO, i.e., $U(b,\tilde{y})=\tilde{y}-\alpha b$ where $\tilde{y}$ is the BO performance, $b$ the BO steps, and $\alpha\in\{0,4e\text{-}05,2e\text{-}04\}$. Note that $\alpha=0$ does not penalize the number of BO steps at all, hence the BO does not terminate until the last BO step $B$ as with the conventional multi-fidelity BO setup.

#### Stopping criterion.

For the baselines, we simply use the fixed threshold $\delta_{b}=0.2$ in Eq. (2) as computing the PI in Eq. (4) is not straightforward for them. For our model, we use $\beta=\exp(3)$ and $\gamma=\log_{2}5$
for all the experiments in this paper, except the ablation study in Fig. 5(d).

#### Evaluation metric.

In order to report the average performances over the tasks, we use the normalized regret of utility $(U_{\text{max}}-U_{b^{*}})/(U_{\text{max}}-U_{\text{min}})\in[0,1]$, similarly to Eq. (2).
$U_{b^{*}}$ is the utility obtained right after the BO terminates at step $b^{*}$, and $U_{\text{max}}$ is the maximum achievable by running a single optimal configuration up to its maximum utility. Computing the exact $U_{\text{min}}$ is a difficult combinatorial optimization problem, thus we simply approximate it with $U(B,y^{\text{worst}}_{1})$, where $y^{\text{worst}}_{1}$ is the worst 1-epoch validation performance across the configurations – we simply let $y_{1}^{\text{worst}}$ decay over the maximum BO steps $B$, corresponding to a lower bound of the exact $U_{\text{min}}$. We then average the normalized regret across all the tasks in each benchmark, and report the mean and standard deviation over 5 runs. Lastly, we also report the rank of each method averaged over the tasks.

**Table 1: Results on the cost-sensitive multi-fidelity HPO setups ($\alpha=4e\text{-}05,2e\text{-}04$). For better readability, we multiply 100 to the normalized regret. The transfer learning methods are indicated by blue.**
| Method | LCBench | TaskSet | PD1 |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\alpha=4e\text{-}05$ | $\alpha=2e\text{-}04$ | $\alpha=4e\text{-}05$ | $\alpha=2e\text{-}04$ | $\alpha=4e\text{-}05$ | $\alpha=2e\text{-}04$ |  |  |  |  |  |  |  |
| Regret | Rank | Regret | Rank | Regret | Rank | Regret | Rank | Regret | Rank | Regret | Rank |  |
| Random [] | 14.1$\pm$1.8 | 7.7 | 18.1$\pm$1.7 | 7.5 | 18.9$\pm$4.6 | 7.8 | 22.3$\pm$4.3 | 7.7 | 05.4$\pm$2.3 | 7.1 | 11.0$\pm$5.6 | 7.2 |
| ASHA [] | 08.6$\pm$1.0 | 6.3 | 13.7$\pm$1.1 | 6.8 | 08.4$\pm$4.1 | 6.2 | 14.7$\pm$5.4 | 7.0 | 05.9$\pm$7.3 | 6.5 | 09.7$\pm$7.4 | 6.5 |
| BOHB [] | 06.4$\pm$1.0 | 5.0 | 11.6$\pm$1.0 | 5.5 | 07.4$\pm$1.4 | 6.9 | 11.2$\pm$1.9 | 6.2 | 01.6$\pm$0.2 | 4.7 | 04.9$\pm$0.2 | 4.9 |
| DEHB [] | 06.1$\pm$1.6 | 4.6 | 11.0$\pm$1.4 | 4.9 | 05.8$\pm$2.3 | 6.1 | 10.0$\pm$1.7 | 6.0 | 02.1$\pm$0.1 | 6.1 | 05.4$\pm$0.1 | 6.0 |
| DyHPO [] | 07.2$\pm$1.2 | 5.7 | 12.1$\pm$1.6 | 5.9 | 07.5$\pm$2.1 | 6.4 | 11.1$\pm$2.0 | 6.3 | 02.5$\pm$0.6 | 6.2 | 06.2$\pm$0.9 | 6.7 |
| DPL [] | 03.8$\pm$0.5 | 3.2 | 09.3$\pm$0.5 | 4.4 | 02.6$\pm$0.7 | 3.4 | 07.5$\pm$0.6 | 4.5 | 01.8$\pm$0.3 | 4.5 | 05.1$\pm$0.6 | 4.7 |
| Quick-Tune^† [] | 09.6$\pm$0.0 | 6.9 | 12.7$\pm$0.0 | 6.1 | 03.7$\pm$0.0 | 3.7 | 05.6$\pm$0.0 | 3.1 | 02.4$\pm$0.0 | 5.4 | 05.5$\pm$0.0 | 5.0 |
| FSBO [] | 02.6$\pm$0.0 | 3.0 | 06.4$\pm$0.0 | 2.7 | 02.9$\pm$0.0 | 3.0 | 04.9$\pm$0.0 | 2.6 | 01.3$\pm$0.0 | 2.6 | 04.2$\pm$0.0 | 3.1 |
| CMBO (ours) | 02.3$\pm$0.1 | 2.7 | 03.1$\pm$0.0 | 1.3 | 01.3$\pm$0.0 | 1.5 | 03.1$\pm$1.0 | 1.5 | 00.8$\pm$0.0 | 1.9 | 00.9$\pm$0.0 | 1.0 |

Figure: Figure 4: Visualization of the normalized regret over BO steps. The first and second row correspond to $\alpha=4e\text{-}05$ and $2e\text{-}04$, respectively. Each column shows the results on a cherry-picked task from each benchmark. The asterisks indicate the stopping points, and the dotted lines represent the normalized regret achievable by running each method without stopping. See §E for the results on all the other tasks.
Refer to caption: x4.png

Figure: (a)
Refer to caption: x11.png

### 4.1 Analysis

Figure: (a) Test loss
Refer to caption: x15.png

#### Effectiveness of our transfer learning.

We first demonstrate the effectiveness of our transfer learning method. Fig. 3 shows the results on the conventional multi-fidelity HPO setting where we do not penalize the cost of BO at all ($\alpha=0$). First of all, note that FSBO, a black-box transfer-BO method which switches its configuration only after a single complete training (e.g., 50 epochs), even outperforms all the other multi-fidelity methods that can change the configurations every epoch. The result clearly shows the importance of transfer learning for improving the sample efficiency of HPO. Quick-Tune^†, a transfer version of DyHPO, performs similarly to the other baselines despite of the transfer learning, except on TaskSet benchmark. We attribute this result to its greedy acquisition function, and more importantly its lack of data augmentation. On the other hand, our method is non-greedy (when $\alpha=0$) and can effectively augment the data with our mixup strategy, thereby showing significantly better performances than all the other multi-fidelity methods. Fig. 6 shows the ablation study on our mixup training. Fig. 6(a) shows that we can effectively reduce the risk of overfitting by adding the mixup strategy. As a result, the performance of BO improves significantly (Fig. 6(b)). Lastly, our method significantly outperforms FSBO on TaskSet and slightly on LCBench and PD1, showing the superiority of multi-fidelity BO to black-box BO.

#### Effectiveness of our acquisition function.

Next, Table 1 shows the performance of each method on the cost-sensitive multi-fidelity HPO setup ($\alpha>0$). We see that our method largely outperforms all the methods on all the settings, including the multi-fidelity HPO and the transfer-BO methods, in terms of both normalized regret and average rank. Notice, our method achieves better average rank as the penalty becomes stronger ($\alpha=2e\text{-}04$). Fig. 4 visualizes the normalized regret over the course of BO, where our method achieves significantly lower regret upon termination. Our method tends to achieve the minimum regret earlier than the baselines, demonstrating its sample efficiency in searching good hyperparameter configurations by explicitly considering the utility during the BO.

In order to clearly understand the source of improvements, we next analyze the configurations chosen by each method. Specifically, for each BO step $b$, we run the configuration currently selected at step $b$ up to its last epoch $T$, and compute its minimum ground-truth regret achievable at some future step $b+\Delta t$ (Fig. 5(a)), as well as the corresponding optimal increment $\Delta t$ (Fig. 5(b)). In Fig. 5(a), our method shows much lower minimum regret than the baselines. It means that our acquisition function in Eq. (1) works as intended, trying to select at each BO step the best configuration which is expected to maximally improve the utility in future. Fig. 5(b) shows that the configurations chosen by our method initially correspond to greater $\Delta t$ (i.e., non-greedy), but gradually to the smaller $\Delta t$ (i.e., greedy). It is because as the BO proceeds, the performance improvements of BO saturate, so the cost of BO quickly dominates the trade-off, leading to smaller $\Delta t$ even close to 0 0.
On the other hand, the tendencies of the baselines are very noisy and relatively unclear.
Lastly, Fig. 5(c) shows the distribution of the top-10 most frequently selected configurations during the BO. As expected, our method tend to focus only on a few configurations during the BO to maximize the short-term performances, especially when the penalty is stronger with greater $\alpha$. On the other hand, the baselines tend to overly explore the configurations even when the penalty is the strongest ($\alpha=2e\text{-}04$).

#### Effectiveness of our stopping criterion.

Lastly, we analyze the effectiveness of our stopping criterion discussed in Eq. (2), (3), and (4). Fig. 5(d) shows the normalized regret over the different values of $\beta$, a mixing coefficient between the two extreme stopping criteria, as discussed in §3.2. $\beta\rightarrow 0$ corresponds to the criterion used by the baselines which is only based on the estimated normalized regret, whereas $\beta\rightarrow+\infty$ corresponds to the hard thresholding only based on the PI. We can see that the optimal criterion is achieved by smoothly mixing between the two ($\beta=e^{-1}$), demonstrating the superiority of our stopping criterion to the one used by the baselines ($\beta\rightarrow 0$).

## 5 Conclusion

In this paper, we discussed cost-sensitive multi-fidelity BO, a novel framework for improving the sample efficiency of HPO. Based on the assumption that users want to early-stop the BO when the utility saturates, we explained how to achieve the maximum utility with our novel acquisition function and the stopping criterion specifically tailored to this problem setup, as well as the novel transfer learning method for training a sample efficient in-context LC extraploator. We empirically demonstrated the effectiveness of our method over the previous multi-fidelity HPO and the transfer-BO methods, with the numerous empirical evidences strongly supporting our claim.

#### Limitations.

Although our method sheds light on improving the efficiency of HPO, there remain a few limitations.
First, we assumed that the utility function is given, but instead we could learn it from data provided by each user.
Second, our LC extrapolator is prone to overfitting even with the mixup strategy when the training dataset is small. Thus, we need a theoretically grounded way to incorporate the prior on LCs and infer the corresponding posterior.
Lastly, PFNs assume the conditional independencies between the outputs and thus generate very noisy LCs, which may distort the estimation of future utilites. Solving them can be an interesting extension of our work.

## Appendix A Details on Benchmarks and Data Preprocessing

In this section, we elaborate the details on the LC benchmarks and data preprocessing we have done.

#### LCBench

We use [APSFailure, Amazon_employee_access, Australian, Fashion-MNIST, KDDCup09_appetency, MiniBooNE, adult, airlines, albert, bank-marketing, blood-transfusion-service-center, car, christine, cnae-9, connect-4, covertype, credit-g, dionis, fabert, helena] for training LC extrapolator. We evaluate it on [higgs, jannis, jasmine, jungle_chess_2pcs_raw_endgame_complete, kc1, kr-vs-kp, mfeat-factors, nomao, numerai28.6, phoneme, segment, shuttle, sylvine, vehicle, volkert]. Each task contains 2000 LCs with 51 training epochs. We summarize the hyperparameter of LCBench in Table 2.

**Table 2: The 7 hyperparameters for LCBench tasks.**
| Name | Type | Vaules | Info |
| --- | --- | --- | --- |
| batch_size | integer | $[16,51]$ | log |
| learning_rate | continuous | $[0.0001,0.1]$ | log |
| max_dropout | continuous | $[0.0,1.0]$ |  |
| max_units | integer | $[64,1024]$ | log |
| momentum | continuous | $[0.1,0.99]$ |  |
| max_layers | integer | $[1,5]$ |  |
| weight_decay | continuous | $[1e-05,0.1]$ |  |

#### TaskSet

We use [rnn_text_classification_family_seed{19, 3, 46, 47, 59, 6, 66},
word_rnn_language_model_family_seed{22, 47, 48, 74, 76, 81}, char_rnn_language_model_family_{seed19, 26, 31, 42, 48, 5, 74}] for training LC extrapolator. We evaluate it on [rnn_text_classification_family_seed{8, 82, 89},
word_rnn_language_model_family_seed{84, 98, 99},
char_rnn_language_model_family_seed{84, 94, 96}]. Each task contains 1000 LCs with 50 training epochs. We summarize the hyperparameter of TaskSet in Table 3.

**Table 3: The 8 hyperparameters for Taskset tasks.**
| Name | Type | Vaules | Info |
| --- | --- | --- | --- |
| learning_rate | continuous | $[1e-09,10.0]$ | log |
| beta1 | continuous | $[0.0001,1.0]$ |  |
| beta2 | continuous | $[0.001,1.0]$ |  |
| epsilon | continuous | $[1e-12,1000]$ | log |
| l1 | continuous | $[1e-09,10.0]$ | log |
| l2 | continuous | $[1e-09,10.0]$ | log |
| linear_decay | continuous | $[1e-08,0.0001]$ | log |

#### PD1

We use [
uniref50_transformer_batch_size_128, lm1b_transformer_batch_size_2048,
imagenet_resnet_batch_size_256, mnist_max_pooling_cnn_tanh_batch_size_2048,
mnist_max_pooling_cnn_relu_batch_size_{2048, 256}, mnist_simple_cnn_batch_size_{2048, 256},
fashion_mnist_max_pooling_cnn_tanh_batch_size_2048, fashion_mnist_max_pooling_cnn_relu_batch_size_{2048, 256}, fashion_mnist_simple_cnn_batch_size_{2048, 256}, svhn_no_extra_wide_resnet_batch_size_1024,
cifar{100, 10}_wide_resnet_batch_size_2048] for training LC extrapolator. We evaluate it on
[imagenet_resnet_batch_size_512, translate_wmt_xformer_translate_batch_size_64,
mnist_max_pooling_cnn_tanh_batch_size_256, fashion_mnist_max_pooling_cnn_tanh_batch_size_256,
svhn_no_extra_wide_resnet_batch_size_256, cifar100_wide_resnet_batch_size_256,
cifar10_wide_resnet_batch_size_256]. Each task contains 240 LCs with 50 training epochs. We summarize the hyperparameter of PD1 in Table 4.

**Table 4: The 8 hyperparameters for PD1 tasks.**
| Name | Type | Vaules | Info |
| --- | --- | --- | --- |
| lr_initial_value | continuous | $[1e-05,10.0]$ | log |
| lr_power | continuous | $[0.1,2.0]$ |  |
| lr_decay_steps_factor | continuous | $[0.01,0.99]$ |  |
| one_minus_momentum | continuous | $[1e-05,1.0]$ | log |

#### Data Preprocessing

As will be detailed in the §C, we use the 0-epoch LC value $y_{n,0}$ which is the performance before taking any gradient steps. The 0-epoch LC values originally are not provided except for LCBench; we use the log-loss of the first epoch as the 0-epoch LC value for TaskSet, as it is already sufficiently large in our chosen tasks. For PD1, we interpolate the LCs to be the length of 51 training epochs, and we take the first performance as the 0-epoch LC value. Furthermore, we take the average over the 0-epoch LC values $\bar{y}_{0}$ since it is hard to have different initial values among optimizer hyperparameter configurations in a task, without taking any gradient steps. For transfer learning, we follow the convention of PFN [] for data preprocessing; we consistently apply non-linear LC normalization(^2^22The details can be found in [Appendix A](https://arxiv.org/pdf/2310.20447) of PFN [] and [https://github.com/automl/lcpfn/blob/main/lcpfn/utils.py](https://github.com/automl/lcpfn/blob/main/lcpfn/utils.py).) to the LC data of three benchmarks, which not only maps either accuracy or log-loss LCs into $[0,1]$ but also simply make our optimization as a maiximization problem. To facilitate transfer learning, we use the maximum and minimum values in each task in LCBench and PD1 benchmark for the LC normalization. In TaskSet, we only use the $\bar{y}_{0}$ for the LC normalization.

## Appendix B Details on Architecture and Training of LC Extrapolator

In the section, we elaborate our LC extrapolator model and how to train it on the learning curve dataset.

#### Construction of Context and Query points.

As mentioned earlier in §3.3, the whole training pipeline of our learning curve extrapolator model can be seen an instance of TNPs []. Here we can simulate each step of Bayesian Optimization; predicting the remaining part of LC in all configurations conditioned on the set $\mathcal{C}$ of the collected partial LCs. To do so, we construct a training task by randomly sampling context and query points from LC benchmark after the proposed LC mixup as follows:

- 1.
We choose a LC dataset $L_{m}=[l_{m,1}^{\top};\dots;l_{m,N}^{\top}]^{\top}\in\mathbb{R}^{N\times T}$ by randomly sampling $m\in[M]$.
- 2.
From $L_{m}$, we randomly sample $n_{1},\ldots,n_{C}\in[N]$ and $t_{1},...,t_{C}\in[T]$ and construct context points of $X^{(c)}=[x_{n_{1}}^{\top},\ldots,x_{n_{C}}^{\top}]^{\top}\in\mathbb{R}^{C
\times d_{x}}$, $T^{(c)}=[t_{1}/T,\ldots,t_{C}/T]^{\top}\in\mathbb{R}^{C\times 1}$, and $Y^{(c)}=[y_{n_{1},t_{1}},\ldots,y_{n_{C},t_{C}}]\in\mathbb{R}^{C\times 1}$.
- 3.
From $L_{m}$, we exclude $n_{1},\ldots,n_{C}\in[N]$ and $t_{1},...,t_{C}\in[T]$ and randomly sample $n^{\prime}_{1},\ldots,n^{\prime}_{Q}\in[N]$ and $t^{\prime}_{1},...,t^{\prime}_{Q}\in[T]$ and construct query points of $X^{(q)}=[x_{n^{\prime}_{1}}^{\top},\ldots,x_{n^{\prime}_{Q}}^{\top}]^{\top}\in
\mathbb{R}^{Q\times d_{x}}$, $T^{(q)}=[t^{\prime}_{1}/T,\ldots,t^{\prime}_{C}/T]^{\top}\in\mathbb{R}^{Q
\times 1}$, and $Y^{(q)}=[y_{n^{\prime}_{1},t^{\prime}_{1}},\ldots,y_{n^{\prime}_{Q},t^{\prime}
_{Q}}]\in\mathbb{R}^{Q\times 1}$.

#### Transformer for Predicting Learning Curves.

From now on, we denote each row vector of the constructed context and query points with the lowercase, e.g., $y^{(q)}$ of $Y^{(q)}$. We learn a Transformer-based learning curve extrapolator model which is a probabilistic model of $f(Y^{(q)}|X^{(c)},T^{(c)},Y^{(c)},X^{(q)},T^{(q)})$. Conditioned on any subsets of LCs (i.e., $X^{(c)},T^{(c)}$, and $Y^{(c)}$), this model predicts a mini-batch of the remaining part of LCs of existing hyperparameter configurations in a given dataset (i.e., $Y^{(q)}$ of $X^{(q)}$ and $T^{(q)}$). For the computational efficiency, we further assume that the query points are independent to each other, as done in PFN []:

$$ $f(Y^{(q)}|X^{(c)},T^{(c)},Y^{(c)},X^{(q)},T^{(q)})=\prod_{x^{(q)},t^{(q)},y^{( q)}}f(y^{(q)}|x^{(q)},t^{(q)},X^{(c)},T^{(c)},Y^{(c)}).$ (5) $$

Before encoding the input into the Transformer, we first encode the input of $X^{(c)},T^{(c)},Y^{(c)},X^{(q)}$, and $T^{(q)}$ using simple linear layer as follows:

$$ $\displaystyle H^{(c)}=X^{(c)}W_{x}+T^{(c)}W_{t}+Y^{(c)}W_{y}$ (6) $\displaystyle H^{(q)}=X^{(q)}W_{x}+T^{(q)}W_{t},$ (7) $$

where $W_{x}\ \in\mathbb{R}^{d_{x}\times d_{h}}$, $W_{t}\in\mathbb{R}^{1\times d_{h}}$, and $W_{y}\in\mathbb{R}^{1\times d_{h}}$. Here, we abbreviate the bias term.

Then we concatenate the encoded represnetations of $H^{(c)}$ and $H^{(q)}$, and feedforward it into Transformer layer by treating each pair of each row vector of $H^{(c)}$ and $H^{(q)}$ as a separate position/token as follows:

$$ $\displaystyle H=\mathtt{Transformer}([H^{(c)};H^{(q)},\mathtt{Mask}])\in \mathbb{R}^{(M+N)\times d_{h}}$ (8) $\displaystyle\hat{Y}=\mathtt{Head}(H)\in\mathbb{R}^{(M+N)\times d_{o}},$ (9) $$

where $\mathtt{Transformer}(\cdot)$ and $\mathtt{Head}(\cdot)$ denote the Transformer layer and multi-layer perceptron (MLP) for the output prediction, respectively. $\mathtt{Mask}\in\mathbb{R}^{(N_{c}+N_{q})\times(N_{c}+N_{q})}$ is the mask of transformer that allows all the tokens to attend context tokens only. Here, the output dimension $d_{o}$ is specified by output distribution of $y$. Following PFN [], we discretize the domain of $y$ by $d_{o}=1000$ and use the categorical distribution. Finally, we only take the output of the last $N_{q}$ tokens as output, i.e., $\hat{Y}^{(q)}=\hat{Y}[:,N_{c}:(N_{c}+N_{q})]\in\mathbb{R}^{N_{q}\times d_{h}}$ (PyTorch-style indexing operation), since we only need the outputs of query tokens for modeling $\prod f(y^{(q)}|x^{(q)},t^{(q)},X^{(c)},T^{(c)},Y^{(c)})$.

#### Training Objective.

Our pre-training objective is then defined as follows:

$$ $\operatorname*{arg\,min}_{f}\mathbb{E}_{p}\left[-\sum_{x^{(q)},t^{(q)},y^{(q)} }\log f(y^{(q)}|x^{(q)},t^{(q)},X^{(c)},T^{(c)},Y^{(c)})\right]\\ +\lambda_{\text{PFN}}\mathcal{L}_{\text{PFN}},$ (10) $$

where $\mathcal{D}_{KL}$ is the Kullback–Leibler divergence, and $p$ is the empirical LC data distribution. We additionally minimize $\mathcal{L}_{\text{PFN}}$ with coefficient $\lambda_{\text{PFN}}$, which is the LC extrapolation loss in each LC []. We found $\lambda_{\text{PFN}}=0.1$ works well for most cases. We use the stochastic gradient descent algorithm to solve the above optimization problem.

#### Training Details.

We sample 4 training tasks for each iteration, i.e., the size of meta mini-batch is set to 4. We uniformly sample the size $C$ of context points from 1 to 300, and the size of query points $Q$ is set to 2048. Following PFN [], the hidden size of each Transformer block $d_{h}$, the hidden size of feed-forward networks, the number of layers of Transformer, dropout rate are set 1024, 2048, 12, 0.2. We use GeLU []. We train the extrapolator for 10,000 iterations on training split of each benchmark with Adam [] optimizer. The $\ell_{2}$ norm of meta mini-batch gradient is clipped to 1.0. The learning rate is linearly increased to 2e-05 for 25000 iterations, and it is decreased with a cosine scheduling until the end. The whole training process takes roughly 10 hours in one NVIDIA Tensor Core A100 GPU.

## Appendix C Additional Details on Experimental Setups

In this section, we elaborate additional details on the experimental setups.

#### 0-epoch LC value.

We assume the access of the 0-epoch LC value $\bar{y}_{0}$ in §A which is the model performance before taking gradient steps. This is also plausible for realistic scenarios since in most deep-learning models one evaluation cost is acceptable in comparison to training costs. The 0-epoch LC value $\bar{y}_{0}$ is always conditioned on our LC extrapolator $f$ for both pretraining and BO stage.

#### Monte-Carlo (MC) sampling for reducing variance of LCs.

As mentioned in §3.2, we estimate the expectation of proposed acquisition function $A$ in Eq. (1) with 1000 MC samples. We found that each LC $y_{n,t_{n:T}}$ sampled from LC extrapolator $f(\cdot|x_{n},\mathcal{C})$ is noisy, due to the assumption that query points of $y_{n,t_{n:T}}$ are independent to each other in Eq. (5). We compute $\tilde{y}_{b+\Delta t}$ by taking the maximum among the last step BO performance (i.e., cumulative max operation), therefore, the quality of estimation highly degenerates due to the noise in the small $\Delta t$. To prevent this, we reduce the variance of MC samples by taking the average of the sampled LCs. For example, we sample 5000 LC samples from the LC extrapolator $f$, then we divide them into 1000 groups and take the average among the 5 LC samples in each group. We empirically found that this stabilize the estimation of not only acquisition function $A$ and probability of utility improvement $p_{b}$ in Eq. (4).

#### Inference Time for BO.

The most of time for each BO step in our method is spent during LC extrapolation. In Table 5, we report the wall-clock time spent on LC extrapolation for 100 mini-batches of LCs. The wall-clock times vary depending on the context size. We measure all the wall-clock times in one one NVIDIA Tensor Core A100 GPU.

**Table 5: Wall-clock time for Inference on 100 mini-batches of LCs.**
| Context Size | Inference Time (s) |
| --- | --- |
| 1 | 0.00921010971069336 |
| 10 | 0.01493692398071289 |
| 20 | 0.01413583755493164 |
| 50 | 0.017796993255615234 |
| 100 | 0.01770782470703125 |
| 200 | 0.025087356567382812 |
| 300 | 0.027765989303588867 |

#### Details on Baseline Implementation.

We list the implementation detils for baselines as follows:

- 1.
Random Search. Instead of randomly selecting a hyperparameter configuration for each BO step, we run the selected configuration until the last epoch $T$.
- 2.
ASHA, BOHB, and DEHB. We follow the most recent implementation of these algorithms in Quick-Tune []. We slightly modify the official code(^3^33[https://github.com/releaunifreiburg/QuickTune](https://github.com/releaunifreiburg/QuickTune)), which is heavily based on SyneTune [] package.
- 3.
DyHPO and Quick-Tune^†. We follow the official code(^4^44[https://github.com/releaunifreiburg/DyHPO](https://github.com/releaunifreiburg/DyHPO)) provided the authors of DyHPO [], and slightly modify the benchmark implementation to incorporate our experimental setups. For Quick-Tune^†, we pretrain the deep kernel GP for 50000 iterations with Adam optimizer with mini-batch size of 512. The initial learning rate is set to 1e-03 and decayed with cosine scheduling. To leverage the transfer learning scenario, we use the best configuration among the LC datasets which is used for training the GP as an initial guess of BO.
- 4.
DPL. We follow the official code(^5^55[https://github.com/releaunifreiburg/DPL](https://github.com/releaunifreiburg/DPL)) provided the authors of DPL [], and slightly modify the benchmark implementation to incorporate our experimental setups.
- 5.
FSBO. FSBO does not provide an official code, therefore, we follow an available code in the internet(^6^66[https://github.com/releaunifreiburg/fsbo](https://github.com/releaunifreiburg/fsbo)). We also slightly modify the benchmark implementation, and use the best configuration among the LC datasets as an initial guess.

## Appendix D Connection between our Mixup Strategy with ifBO and TNP

Our mixup strategy is reminiscent of the data generation scheme of ifBO [], a variant of PFNs for in-context freeze-thaw BO.
Similarly to our ancestral sampling, ifBO first samples random weights for a neural network (i.e., a prior distribution) to sample a correlation between configurations (the first mixup step), and then linearly combines a set of basis functions to generate LCs (the second mixup step). Our training method differs from ifBO in that our prior distribution is implicitly defined by LC datasets and the mixup strategy, whereas ifBO resorts to a manually defined distribution.

Indeed, our training method is more similar to Transformer Neural Processes (TNPs) [], a Transformer variant of Neural Processes (NPs) []. Similarly to PFNs, TNPs directly maximize the likelihood of target data given context data with a Transformer architecture, which differs from the typical NP variants that summarize the context into a latent variable and perform variational inference on it.
Moreover, as with the other NP variants, TNPs meta-learn a model over a distribution of tasks to perform sample efficient probabilistic inference. In this vein, the whole training pipeline of our LC extrapolator can be seen as an instance of TNPs – we also meta-learn a sample efficient Transformer-based LC extrapolator over the distribution of LCs induced by the mixup strategy.

## Appendix E Additional Experimental Results

#### Visualizations of the normalized regret over BO steps

for LCBench ($\alpha=4e\text{-}05$), LCBench ($\alpha=2e\text{-}04$), TaskSet ($\alpha=4e\text{-}05$), TaskSet ($\alpha=2e\text{-}04$), PD1 ($\alpha=4e\text{-}05$), and PD1 ($\alpha=2e\text{-}04$) are provided Figure 7, 8, 9, 10, 11, and 12, respectively.

#### Visualizations of the LC extrapolation over BO steps

for LCBench, TaskSet, and PD1 are provided Figure 13, 14, and 15, respectively. Here, we plot the LC extrapolation results of unseen hyperparameter configurations through BO. Each row shows the results for a different size of the observation set ($|\mathcal{C}|=0,10,50$, and $300$), and each column shows a different size of context points in each LC (0, 2, 5, 10, 20, and 30).

Figure: Figure 7: Visualization of the normalized regret over BO steps on LCBench ($\alpha=$4e-05).
Refer to caption: x17.png

Figure: Figure 8: Visualization of the normalized regret over BO steps on LCBench ($\alpha=$2e-04).
Refer to caption: x32.png

Figure: Figure 9: Visualization of the normalized regret over BO steps on TaskSet ($\alpha=$4e-05).
Refer to caption: x47.png

Figure: Figure 10: Visualization of the normalized regret over BO steps on TaskSet ($\alpha=$2e-04).
Refer to caption: x56.png

Figure: Figure 11: Visualization of the normalized regret over BO steps on PD1 ($\alpha=$4e-05).
Refer to caption: x65.png

Figure: Figure 12: Visualization of the normalized regret over BO steps on PD1 ($\alpha=$2e-04).
Refer to caption: x72.png

Figure: Figure 13: Visualization of LC extrapolation over BO steps on LCBench.
Refer to caption: x79.png

Figure: Figure 14: Visualization of LC extrapolation over BO steps on TaskSet.
Refer to caption: x81.png

Figure: Figure 15: Visualization of LC extrapolation over BO steps on PD1.
Refer to caption: x83.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 3
presentation: 1
questions: - *line 164* - The proposed method works with a fixed, finite pool of hyperparameters. Firstly, I would contend with the claim that this is the "convention" in BO, where it is arguable the exception rather than the rule. However, my biggest question is how this pool is populated in the first place? I would guess randomly, which begs the question of how comparisons are carried out against other methods in which this is not a common practice, e.g. BOHB? Furthermore, details are missing as to how many hyperparameter configurations there are in this pool. Fig 7c hints that this is around 10, which seems minute?
  - As you progress through the BO procedure and gain more information about the correlations between hyperparameter configurations, it is not really possible to leverage this knowledge to consider novel hyperparameter settings to evaluate meaning you're simply stuck with your initial pool. Do you directly compare against methods with this limitation imposed or not?
- Please clarify the question regarding "normalized regret of utility" (Eq. 3) raised above
- Please clarify the question regarding the horizontal axis raised above
- The ablation study concerning the use of mix-up is interesting (Fig 6). However, have you carried out an analysis to compare the standalone extrapolation performance of your proposed in-context LC curve prediction set-up (with/without mix-up) against more traditional approaches (that may be both simpler and cheaper)?
- *line 406* "Each column corresponds to the cherry-picked examples from each benchmark" - "cherry-picked" is a pejorative term used to describe the practice of isolating only the results that place your method in a good light and/or your competitor's in a bad light. Are the chosen results actually representative of all the other results you could have shown, or are they in fact cherry-picked? If so, we cannot rely upon these reported results to make an assessment of the proposed method!
- *line 446* - how special is the mix-up augmentation strategy? Could you obtain comparable results by fitting an emulator/surrogate model to be able to interpolate entire learning curves between hyperparameter configurations? This would also give you infinitely many training examples that would more accurately preserve correlations between the configurations; granted, it's relatively more expensive but still cheap in absolute terms.
- *line 183* ("utility" function) - It is unclear to me at what stage and exactly how you would elicit user preference data. By generating many pairs of performance-cost pairs upfront and having the user choose their preferences before proceeding with the optimization procedure I assume?
- *line 350* - "We select 23 tasks with 4 different hyperparameters based on [*sic*] SyneTune (Salinas et al., 2022) package" -- what does it mean to select tasks based on some package? You adopt the same set of tasks that they consider in their experimental benchmarks?
soundness: 2
strengths: ## Significance

This manuscripts seeks to simultaneously improves on various aspects of BO, including multi-task BO (warm-starting BO from related tasks), multi-fidelity BO (leveraging low fidelity performance signals), cost-aware BO, and also optimal stopping for BO, all in a coherent and unified manner. These aspects are timely and important long-standing open problems in BO. In spite of some potential shortcomings (identified below), some of the high-level ideas and concepts could readily translate to other multi-fidelity frameworks of this kind.

## Originality

This work proposes novelties in number of different components, e.g. in the acquisition function by a) generalizing the performance ($y$) to a quantity that is normalized/penalized by budget/cost (which they call a "utility" function), and also b) generalize this quantity to be based on extrapolated performances. Finally, c) to be able to accurately and efficiently carry out this extrapolation in a few-shot manner, they incorporate recent in-context learning approaches based on PFNs, which are trained on priors implicitly specified by benchmark datasets containing learning curve, and adapt augmentation strategies like mix-up to learning curve data to mitigate the possibility of overfitting.
summary: This manuscript seeks to address the problem of hyperparameter optimization particularly for the training of machine learning models which routinely provide low fidelity performance signals that can be leveraged to improve the efficiency of the optimization procedure. More specifically, this paper proposes numerous improvements to multi-fidelity Bayesian optimization (BO), including (i) a generalized notion of performance that accounts for cost and (ii) an acquisition function based on the expected improvement in the full trajectory of extrapolated future performance outcomes (rather than the typical one-step ahead outcome). The generalized performance, which the authors call "utility", can be specified analytically/parametrically, or otherwise learned from a user's preference. The authors further propose a cost-based stopping criterion for the BO procedure according to values of this "utility". Finally, the authors investigate the use of in-context learning frameworks such as prior-fitted networks (PFNs) to accurately extrapolate learning curves in a few-shot manner by transfer learning from curves of hyperparameter configurations from related tasks/datasets.
weaknesses: ## Quality

The technical quality of the proposed methodology could be improved. In particular, I found many aspects of the approach to be arbitrary and not well-motivated. For instance, for the stopping criterion described starting *line 250*, the choice of using the BetaCDF with parameters $\beta, \gamma > 0$ and probability $p$ as the probability of improvement (PI), beyond working fine empirically on the benchmark problems considered, seem highly convoluted and totally arbitrary to me.

A major concern I have in the empirical evaluation of the proposed method is in the "normalized regret of utility" (Eq. 3), which is the primary metric that is reported. Beyond being quite complicated to compute (evidenced by lines 417-420), it is also not obvious to me that this is the "holy grail" metric we should be aiming for in the first place. Does this metric not differ depending on the surrogate/extrapolation model of choice? Furthermore, I am unclear as to how this metric is even defined for other methods such as Random, BOHB, etc. which don't explicitly model the performance $y$, and in which it's unclear how the "utility" can be incorporated? I would be interested in seeing a more conventional plot showing the current best performance (or regret) along the vertical axis. 

Another concern is that the reported empirical results all display the BO iteration along the horizontal axis, which is highly misleading in the context of multi-fidelity BO. It seems to me that the notion of a BO step means totally different things in different frameworks. For instance, in BOHB, a BO step signifies training a model with a particular hyperparameter configuration to full completion, but in most cases they are trained for a fraction, e.g. 27/81 epochs, resulting in a *fractional* BO step (in this example 1/3rd of a BO step). In contrast, under the proposed framework, a BO step is the advancement of a configuration by a single epoch. Therefore, I am doubtful that the results presented show an apples-to-apples comparison.

Furthermore, in multi-fidelity BO just showing BO steps along the horizontal axis (fractional or otherwise) is not entirely informative either, especially when cost is of interest. In addition to fractional BO steps, I would like to see a plot with the wall-clock time along the horizontal axis.

## Clarity

A significant weakness of this paper is its lack of clarity, particularly in many parts of the paper (describing important technical details) which I found cryptic and difficult to parse. Some examples include:
- *lines 215-219* (details on learning curve extrapolation and how its used to compute the "BO performance")
- *lines 417-420* (details on computing bounds on the "utility") - this is indecipherable

More generally, this manuscript could benefit from more careful copy-editing. Some specific examples (non-exhaustive) of where writing quality could be improved are enumerated in the "Miscellaneous Remarks" section.


### Miscellaneous Remarks

- The overloaded use of the term "utility" is confusing as utility functions already plays a central role in Bayesian decision theory (of which Bayesian optimization is an special case [Garnett, 2023]). As such, statements such as "We call this trade-off utility" (*line 67*), "We introduce the concept of utility, ..." (*line 110*), and "We first introduce the detailed notion of utility function" (*lines 95-96*), are likely to raise eyebrows.
- *line 40* - "receives more attention"
- *line 77* - "hyperparamter"
- *line 78* - "improve it in future"
- *lines 84-86* - ?
- *line 98* - "a recently introduced"
- *line 107* - "a reasonable and stable way" -- "reasonable" the reader can probably infer but what makes a "utility" function "stable"?
- *line 77* - "hyperparmater"
- *line 161* - "surrogate function" -> "surrogate functions"
- *line 199-200* - the sign of the second term in the binary cross-entropy loss is wrong
- *line 431* - "despite of the transfer learning"
- *line 351* - "For easier transfer learning"
- *line 364* - "training epochs at future"
- *line 538* - "numerous empirical evidences"

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: -	In Table 1, the results of baseline methods change when alpha changes. Is this because of the different normalized regret that I mentioned in the Weakness section? This is because alpha is a parameter of utility function, which only belongs to the proposed CMBO. Other methods, such as Random Search should not be affected by this parameter. Can the authors provide addition results - the normalized regret of evaluation values f(x) - as other baselines?

-	Are PFNs trained only once, or do we need to retrain PFN during CMBO? How does the training time of PFNs compare to the evaluation of a HPO epoch?

-	Minor: Can the authors explain more about the choice of PFNs? How about using Deep Gaussian Process as FSBO baseline?
soundness: 2
strengths: -	The idea of extrapolating LC to compute the acquisition function (AF) sounds interesting to me.
-	The authors also improve the problem of PFN cannot be trained with dataset, and instead need prior distribution. The mixup strategy fixes this issue.
-	There are many analyses to support the results.
-	The ranking results in Tables 1, 3, 4 seem to be impressive.
summary: The paper proposes CMBO, a cost-sensitive multi-fidelity BO methods, which target hyper-parameter tuning problems. CMBO is built upon freeze-thaw BO, which allows for pausing (freeze) the configuration run at intermediate epoch, and resuming (thaw) the run at with remaining epoch later. Specifically, CMBO introduces the utility function to account for the trade-off between the cost and performance of BO steps, then propose a EI-based acquisition function for this utility. CMBO’s optimization policy is based on maximizing the utility (the expectation of the proposed utility), instead of maximizing the validation performance as previous methods. Moreover, to support these steps, CMBO adopts Prior-fitted Network concept to extrapolate the learning curves (LC), which allows for computing the utility-based acquisition function.
weaknesses: -	It’s strange that the reported results are in terms of normalized regret of the utility U, which seems to be not common in the field. The utility is the term proposed by the authors. FSBO, ifBO and QuickTune seem to use the normalized regret of the function evaluation f(x).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. How to determine the hyperparameters in the framework, such as $\delta_b$ in the stopping criterion?
2. In line 412, why use different $\beta$ for the proposed method only?
3. Why not consider a bigger $\alpha$ and other format of cost independent of the budget number $b$?
soundness: 3
strengths: 1. The motivations are clear. Existing multi-fidelity optimization methods mainly assumed a given budget, little focusing on the active learning strategy with limited cost.
2. The solutions are innovative, including the improved acquisition function and the data augmentation strategy in transfer learning.
3. The experiments is well-organized to show the effectiveness and implications.
summary: This work developed a new framework of multi-fidelity Bayesian optimization in consideration of budget/cost penalty. An improved acquisition function, as a variant of the expected improvement (EI), was proposed to represent the cost-performance utility of decision makers. The surrogate models were built by the learning curve extrapolation method in conjunction with a data augmentation strategy. Additionally, a stopping criterion was introduced to adaptively save the cost, thereby achieving best utility. Finally, the HPO experiments demonstrated the effectiveness and superiority of the framework.
weaknesses: 1. The presentation is not good enough. The utility function is hard to define and learn from noisy preference data. As aligning utility from preference learning was not performed in the experiments, a definition or assumption should be enough rather than trivial content from line 182 to line 201. On the other hand, the basic definition of multi-fidelity optimization problems were missing. It is not mentioned whether it is a maximization or minimization problem. This should be highly related to the design of utility function.
2. The solution, especially the improved acquisition function, may not work in cost-limited problems. This work only introduces the penalty rather than constraints. In other word, if there were cost constraints, this framework may not ensure the cost during search is within the constraints.
3. The experiment settings seem unfair. The metric was related to utility $U$, however, it is not clear whether other baselines considered $U$ as their objectives as well. If not, the comparison may be unfair due to different objectives.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: How does the proposed utility-based acquisition function fundamentally differ from existing multi-fidelity acquisition functions that already balance exploration and exploitation? Could you provide a detailed comparison with specific acquisition functions?

Have you considered comparing your method with non-myopic acquisition functions or RL-based approaches (e.g., work by Hsieh et al. 2021 or Dong et al. 2022) that might address similar concerns about long-term optimization strategy?

Could you provide additional ablation studies that:
- Compare the method without the utility function to isolate its contribution
- Evaluate the performance against other transfer learning HPO methods
- Demonstrate the individual impact of each component (acquisition function, stopping criterion, transfer learning)

The paper assumes users can effectively specify their preferences regarding the cost-performance trade-off. How sensitive is the method to misspecified preferences, and what guidance can be provided to users for setting these preferences effectively?

Could you clarify how the proposed method differs from simply annealing the exploration parameter in traditional acquisition functions? The current presentation makes it difficult to distinguish the novelty of your approach from this simpler alternative.
soundness: 2
strengths: The technical approach is well-developed and carefully constructed. The paper provides detailed explanations of each component, including the mathematical formulations and algorithmic details.

The transfer learning component using PFNs with the proposed mixup strategy shows notable improvement in sample efficiency, which is particularly important for early-stage optimization decisions.

The paper addresses a practical concern in hyperparameter optimization - the need to balance performance gains against computational costs - that is relevant to many real-world applications.
summary: This paper introduces CMBO (Cost-sensitive Multi-fidelity Bayesian Optimization), a novel framework for hyperparameter optimization that explicitly considers the trade-off between computational cost and performance. The key innovation is the introduction of a utility function that captures user preferences regarding this trade-off, which can be learned from preference data. The method combines three main components: (1) a novel acquisition function that maximizes expected utility improvement, (2) an adaptive stopping criterion that determines when to terminate optimization based on utility saturation, and (3) a transfer learning approach using Prior-Fitted Networks (PFNs) with a novel mixup strategy for learning curve extrapolation. The authors evaluate their method on three benchmark datasets (LCBench, TaskSet, PD1) against several baseline methods, demonstrating superior performance especially in scenarios with strong cost penalties.
weaknesses: The paper's fundamental premise regarding the novelty of considering cost-performance trade-offs in multi-fidelity BO is questionable. Many existing multi-fidelity acquisition functions already incorporate such trade-offs implicitly. The authors should better differentiate their approach from existing methods that handle exploration-exploitation balance.

The experimental comparisons may not be entirely fair, as some baselines do not incorporate transfer learning while the proposed method does. A more equitable comparison would include state-of-the-art transfer learning HPO methods.

The paper could benefit from a more thorough ablation study to isolate the contributions of different components, particularly to demonstrate whether the utility function provides benefits beyond what's already captured in traditional multi-fidelity acquisition functions. 

Also as a MFBO paper, the paper did not compare to SOTA MFBO methods like MFKG, CFKG, BOCA, MF-UCB.

The claim that existing multi-fidelity BO methods "tend to over-explore" (Line 53) is not well-substantiated and could be contested. The authors should provide empirical evidence for this claim or revise it.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 2
questions: 1. In Equation 2, the expression of b + t, b represents BO step, and t represents training epoch. How do they add up?

2. In algorithm 1, why is t updated in step 12?

3. In Equation 3, utility starts to decrease as bo progresses, so there should be a time when Umax=Uprev, and regret=0, but in the following experiments, there is no time when Umax=Uprev is equal to 0.

4. In equation 5, p_b > 0.5 stops. However, a larger p_b indicates a higher probability of subsequent utility. Why does this stop at this time?

5. Line 448: Multi-fidelity BOs are better than black-box BOs. However, multi-fidelity is not applicable in real industrial scenarios. Since black boxes are used, different epochs cannot be used to terminate the BOs. This statement is not appropriate.
soundness: 3
strengths: 1. The utility is closer to the actual user's consideration, rather than waiting for training convergence, which will waste a lot of computing resources, and the user only gets limited benefits.

2. The utility's quantification design is clever, allowing users to provide data through 2 out of 1 judgment, and then accurately estimate the quantification function. This design is easy to use and more suitable for users.

3. A large number of experiments are conducted under the utility criterion, and the method shows consistent superiority and stability.
summary: A new optimization objective, utility, is proposed in Bayesian optimization. Under this goal, the corresponding acquisition function, multi-fidelity strategy and stop strategy are reconstructed. Among the highlights, utility proposes that it is more suitable for scenarios where users need to balance precision requirements and computing overheads. When a large amount of resources are invested to improve only a small amount of precision, the optimization process can be terminated. As for large and small quantities of quantification, the author proposes a kind of quantitative definition of utility on the user side by allowing users to make choice feedback. Finally, under this index, the method presented in this paper shows excellent advantages.
weaknesses: 1. Some words are not serious or well-founded, such as One may argue on line 55 and One may argue on line 65, saying that it is difficult for users to obtain the total budget and let users evaluate how to balance benefits and expenses.

2. Figure 2 is called true utility. I think the evaluation here is not true utility. Instead, the author assumes the model and estimates the parameters based on the data generated by the model (user data). However, if the data answered by the user does not match the model set by the author, the fitting effect is much worse.

3. In algorithm 1, if N is given here, how to solve the continuous parameter problem faced by traditional BO? It's supposed to be easy to perfect, but the author didn't do it in this version.

4. This paper explains the ins and outs of utility, but only regret is used as the core index during comparison. If the utility curves, termination points and subsequent trends of different algorithms can be compared, the advantages of the algorithm will be more intuitively understood.

Meta-review: This work proposes a cost-sensitive multi-fidelity optimization algorithm that includes a novel transfer learning model based on learning-curve prior fitted networks (LC-PFNs), a preference-based utility model for understanding decision-maker’s preferences with respect to cost and evaluation performance, and an acquisition function.

Reviewers found the combination of transfer learning, cost sensitivity search, and multi-fidelity modeling to be timely and interesting.  The use of mixup for transfer learning with PFNs also appears to be novel and is surprisingly effective.

There were three main issues raised by the reviewers.  First, the motivation for using preference learning is unclear. In particular, why would a human decision-maker be able to more optimally decide when a task should be terminated, compared with a principled algorithm that is able to e.g., compute the information gain from continuing to evaluate (vs starting a new evaluation?) (Pvea, 8PmF).  Second, the acquisition function is heuristic and design choices are not well motivated (TsBW, 8PmF).  Finally, reviewers found the evaluation criteria, such as the use of “normalized regret”, and reporting results in terms of “utility”, which had been criticized as not being inherently meaningful (8PmF, wRPD), nor something that is well-defined or targeted by “baseline” methods (TsBW).

Reviewers have left detailed comments on gaps in the presentation, along with substantive concerns.   The work has a number of moving parts and novel contributions (e.g., the transfer learning, the preference learning, the acquisition function), however, it is not clear why all of these must be combined as they are,  whether all aspects are necessary, and why certain design choices of hyper parameters were selected  (TsBW). Considering each design choice in more detail, and understanding the contribution of each component (while considering more standard MF formulations, such as linear costs, see e.g., comments by Pvea) could help improve understanding here.

This work has a number of interesting ideas and I look forward to seeing future iterations of this work that take into account the thoughtful feedback provided by the reviewers.

Decision: Reject