Title: Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Cost-Sensitive

Bayesian Optimization

Multi-Fidelity HPO

PFNs

Transfer Learning
Abstract: In this paper, we address the problem of cost-sensitive hyperparameter optimization (HPO) built upon freeze-thaw Bayesian optimization (BO). Specifically, we assume a scenario where users want to early-stop the HPO process when the expected performance improvement is not satisfactory with respect to the additional computational cost. Motivated by this scenario, we introduce \emph{utility} in the freeze-thaw framework, a function describing the trade-off between the cost and performance that can be estimated from the user's preference data. This utility function, combined with our novel acquisition function and stopping criterion, allows us to dynamically continue training the configuration that we expect to maximally improve the utility in the future, and also automatically stop the HPO process around the maximum utility. Further, we improve the sample efficiency of existing freeze-thaw methods with transfer learning to develop a specialized surrogate model for the cost-sensitive HPO problem. We validate our algorithm on established multi-fidelity HPO benchmarks and show that it outperforms all the previous freeze-thaw BO and transfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance.


Peer reviews:
comments: SUMMARY:
The paper describes a freeze-thaw BO algorithm for hyper-parameter optimization that optimizes the utility, which is a measure of the user's internal tradeoff between cost and performance.

STRENGTHS AND WEAKNESSES:
The central idea behind this paper is solid and the paper is well-written for the most part. The paper is well structured - relevant background material is introduced systematically, the algorithm described clearly and the experimental verification appears reasonable.

It strikes me that clarity has been somewhat compromised to fit within the page limit in some places. For example I am not at all familiar with PFNs, so ideally I would like a more detailed exposition to be presented: however I understand that this may be incompatible with page limits. Also the figures in the results section are basically unreadable, but again I understand that this is almost certainly due to page limits, so I will overlook this.

More seriously equation (2) is not clear to me. If I understand correctly, $u_1$ and $u_2$ are pairs $(b,\tilde{y}_n)$, which appears incompatible with exponentials and inversions as per the equation. This seems central to the paper - could you please clarify?

QUESTIONS:
See above re my (mis)understanding of equation (2).

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a cost-sensitive freeze-thaw BO method for HPO (CFBO) that focuses on performance vs. computational cost. The core idea is optimizing a utility function defined on performance and cost.\nThis utility function in general can be user-defined.\nCFBO is built upon ideas from freeze-thaw BO, however, the acquisition function is based on expected utility improvement instead of performance to (dynamically) determine the training length of configurations with high expected utility (performance vs. computational cost trade-off).\nThe authors further propose a regret-based stopping rule defined on the utility to terminate optimization when further improvements become inefficient.\nThe learning curve extrapolation within the freeze-thaw part makes use of PFNs and builds on existing works but implements a mixup-based transfer learning strategy.\nIn their experiments, the authors compare CFBO against baselines and competitors on three multi-fidelity HPO benchmarks and show that CFBO outperforms all baselines in cost-sensitive settings when looking at the regret in utility.\nIn conventional HPO with no cost penalty, CFBO performs competitively.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nCFBO makes use of solid building blocks to tackle the problem of cost-sensitive multi-fidelity HPO (e.g., freeze-thaw BO, transfer learning for learning curve extrapolation via PFNs).\nBenchmarks are reasonably selected and the set of competitors and baselines is sufficiently large.\nThe authors investigate normalized regret (on utility) and rank for each benchmark.\nThe paper is written well and clarity is high.\nOriginality is given to some extent and the problem CFBO tackles is a relevant one.\nCFBO ties in with the related literature of multi-fidelity HPO, focusing on freeze-thaw BO.\nThe authors correctly identify that existing methods generally do not focus on performance vs. computational cost trade-offs but instead assume a fixed budget.\n\n\n\n### Weaknesses\n\nNovelty is limited given the reuse of existing building blocks (EI, freeze-thaw BO, learning curve extrapolation via PFNs).\nOne notable caveat is that all other baselines and competitors are not naturally designed to consider the trade-off between performance and computational cost so the metric of normalized regret on utility can be considered slightly biased w.r.t CFBO.\nComputational efficiency is not analyzed in detail and detailed wall-clock time comparisons are missing.\nOverall, the method is strongly heuristically motivated.\n\nThe authors do not address recent literature concerned with terminating BO ([1], [2]).\nEspecially [1] is directly tailored to the problem of HPO and stopping is performed if an upper bound on the simple regret (difference between current best validation score and estimated global minimum based on the surrogate model) is less or equal to the estimated variance of the validation error.\nThis stopping criterion is less heuristically motivated than the one proposed by the authors and further does take uncertainty in the objective function in HPO into account.\nIn l738 in the appendix, the authors briefly address [1] but I do not think that this is sufficient.\nMoreover, the paper does not make a strong connection to the general literature of cost-aware BO [3].\n\n[1] Makarova et al. (2021) https://arxiv.org/abs/2104.08166\n\n[2] Wilson (2024) https://arxiv.org/abs/2402.16811\n\n[3] Xie et al. (2024) https://arxiv.org/abs/2406.20062\n\nQUESTIONS:\nI already reviewed the paper this year when it was submitted to ICML 2025.\nAs far as I can tell, only minor changes were made prior to re-submitting this version.\nI therefore follow up with similar questions I already had when first reviewing:\n\nHow did you select the maximum budget for each benchmark task?\n\nThe expected improvement of utility in 2.2 has the implicit assumption that past performance (and utility) observations are noise-free, correct?\nI.e., in (3), we subtract from the utility function at a future time $b + \\Delta t$ with cumulative best performance $\\tilde{y}_{b + \\Delta t}$ the most recent utility value $U_p$.\nHowever, the expectation only applies to the utility part of the future time and is derived from the predicted learning curve and $U_p$ is treated as deterministic.\n\nIn the checklist regarding limitations you state:\n`Justification: We provide the Limitations paragraph in §4.`\nHowever, this simply references the Conclusion section were limitations are not discussed.\nCan you discuss this in the context of HPO where the objective function is per se not deterministic (i.e., in contrast to the tabular benchmarks with pre-collected data used in the experiments, real-world HPO is stochastic and noisy as we are optimizing an estimate of the generalization performance obtained via model fitting and evaluation based on a resampling)?\n\nI believe it could be beneficial to add a notation table in the appendix.\n\nLIMITATIONS:\nLimitations and potential negative societal impact are addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a cost-sensitive freeze-thaw BO approach that not only thaws the most promising configuration at each step, but also is aware of the cost of evaluating the thawed candidate. The awareness is realized by a utility function describing the trade-off between expected performance improvement and computational cost of the evaluation. An experimental evaluation shows that the proposed approach is competitive with other existing multi-fidelity approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The work considers an important problem in BO\n- The paper is written clearly\n- The proposed approach is presented well, and the empirical evaluation is appropriate for the proposed method\n\n**Weaknesses**\n\n- The authors do not provide any theoretical considerations on their proposed utility score. Though not being essential, an analysis of the convergence of multi-fidelity BO using the proposed utility score would be interesting\n- line 147-149: It is not clear to me what the authors mean by \"selecting one point above\" in this context.\n- Choice of hyperparameters of utility function: I can imagine that, depending on the hyperparameter choice of the utility function, the BO might yield solutions that differ quite much in terms of their evaluation score. Can the authors provide best practices to set these parameters?\n- Fig. 6: I'm wondering why QuickTune underperforms here. Since QuickTune comes with transfer capabilities, it would be nice if the authors could provide more details on that.\n- Results on cost-sensitive MF-HPO: I'd expect FSBO to be better than CFBO because FSBO is not considering any trade-offs but merely performs for maximum performance. Therefore, I found it surprising to see that FSBO performs worse than CFBO throughout all tasks. Maybe the authors could comment on that.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nThe authors do not discuss the limitations of their approach. This should be added (at least in the Appendix). Also, there is no related work section in the paper (however, a \"Background\" section). Still, discussing some related work, such as FSBO and QuickTune, would be beneficial (can be done in the Appendix).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a Bayesian optimization method that combines freeze-thaw BO, preference learning, and transfer learning. The method uses a variant of expected improvement on the preference function (utility), where the incumbent threshold is set to the most recent utility, rather than the previously evaluated point with max utility. The method maximizes EI over the parameters and target epoch. The method leverages a regret-based stopping criterion for decide when to stop the entire HPO loop. The paper shows how training the PFN via transfer learning (via mixup) improves performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The empirical performance is strong\n* Sensitivity analyses show the best choice of HPs for for the Beta CDF in the stopping criterion\n\nWeaknesses\n* The empirical evaluation gives the proposed method a sizable advantage notably since methods are evaluated w.r.t  the utility which the proposed method is aware of (it omnisciently knows the right preference function class) and baselines are not.\n* The contribution is pretty small as the paper combines freeze-thaw BO, transfer learning, and preference learning.\n* The method introduces quite a few hyperparameters (HPs for the Beta CDF, choice of parametric function class, etc)\n* The transfer learning method requires retraining the PFN for each new use case\n\nQUESTIONS:\n* The intro discusses the motivation being that budget is typically limited and users may want to allocate more budget to training the model than to exploration. Why not focus the paper on BO algorithms that explicitly account for a known budget and plan accordingly (using lookahead and planning w.r.t the budget)?\n* Is it reasonable for a BO user to pick the parametric form of their utility function? That seem hard for a user to specify\n* How does performance change if parts of the AF are ablated: a) if only final performance is optimized for rather than the optimizing the target epoch, b) if the threshold used in EI is the best incumbent and not the most recent?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}