Title: Informed Initialization for Bayesian Optimization and Active Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian Optimization

Gaussian Process

Experimental Design

Bayesian Active Learning
Abstract: Bayesian Optimization (BO) is a widely used method for optimizing expensive black-box functions, relying on probabilistic surrogate models such as Gaussian Processes (GPs). The quality of the surrogate model is crucial for good optimization performance, especially in the few-shot setting where only a small number of batches of points can be evaluated. In this setting, the initialization plays a critical role in shaping the surrogate's predictive quality and guiding subsequent optimization. Despite this, practitioners typically rely on (quasi-)random designs to cover the input space. However, such approaches neglect two key factors: (a) random designs may not be space-filling, and (b) efficient hyperparameter learning during initialization is essential for high-quality prediction, which may conflict with space-filling designs. To address these limitations, we propose Hyperparameter-Informed Predictive Exploration (HIPE), a novel acquisition strategy that balances space-filling exploration with hyperparameter learning using information-theoretic principles. We derive a closed-form expression for HIPE in the GP setting and demonstrate its effectiveness through extensive experiments in active learning and few-shot BO. Our results show that HIPE outperforms standard initialization strategies in terms of predictive accuracy, hyperparameter identification, and optimization performance, particularly in large-batch, few-shot settings relevant to many real-world BO applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose HIPE, a method for “cold start” initialization of the hyperparameters of surrogate models. The method is intended for application to surrogate models that will be used in the context of sequential learning settings like Bayesian optimization or Bayesian active learning. The proposed acquisition function is an additive combination of two existing acquisition functions, EPIG and BALD.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper tackles the important problem of hyperparameter initialization in a principled way. The proposed method is intuitive, and the authors provide thorough empirical evaluation of HIPE against a range of benchmarks and in a variety of settings. Excepting some instances of unclear notation (see Questions below), the paper is clearly written.\n\nThe technical contribution strikes me as somewhat limited: HIPE is a relatively straightforward combination of two existing methods (EPIG and BALD). The paper nevertheless carefully considers and specifies the objectives of the initialization phase, which is important and not always straightforward. I find the proposed hyperparameter-free method for setting $\\beta$ quite interesting and creative.\n\nQUESTIONS:\n- The motivation for HIPE is to balance between hyperparameter learning and space-filling design. Yet the EPIG objective targets reduction in predictive uncertainty, not space-filling design. Under what conditions are these equivalent? If and when they disagree, should the user prefer the design that reduces predictive uncertainty, and why? If not, why not use an acquisition function that optimizes for space-filling design directly?\n- I find the notation for the information theoretic measures a bit confusing. In line 79, $y(\\boldsymbol{x})$ appears to be defined as a given value of the measurements (a scalar or vector). In eqs. 1, 2 and 4-8, is $y(\\boldsymbol{x})$ intended to be a fixed value of the measurements, or the random variable characterizing measurements that arise from a particular value of $\\boldsymbol{x}$? The latter is consistent with standard definitions of the EIG. If the latter, should the expectation in eq. 1 be w.r.t. this random variable? If not, could you please clarify how to interpret $\\mathrm{H}[y(\\boldsymbol{x})]$; what does it mean to take the entropy of a fixed value?\n- How are the batches selected? Especially in the large-batch setting, enumerating over every possible batch of locations is generally computationally intractable. It’s common to approximate the optimal batch by greedily optimizing the acquisition function over candidate locations one-at-a-time. How would this practice affect the performance of HIPE?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

comments: SUMMARY:
The paper introduces a new acquisition function for Bayesian Optimization and Active Learning in the case of hyperparameter uncertainty modeled via fully Bayesian Gaussian Processes. This acquisition function is designed to perform especially well in the large-batch and few shot case. It balances hyperparameter uncertainty reduction, as well as space filling designs for uniform test space distributions.

STRENGTHS AND WEAKNESSES:
- The topic of hyperparameter uncertainty in practical examples is under-adressed in the current GP literature and therefore of great interest
- The paper is well written and gives a good intuition for the new acquisition function
- The experiments are extensive and support the superiority of the new acquisition function against baselines

- I am not sure, whether the authors compare to the current state of the art in their experiments (see questions for suggestions)

QUESTIONS:
- Why don't you take Query-By-Committee and its variant (Riis et al., 2022) as a baseline since it outperforms BALD in the settings used in their paper ?
- Would Statistical distance-based Active Learning from Hvarfner et al., 2023 be another choice as a comparative acquisition function?
- I am wondering whether there are any theoretical guarantees to proof for your acquisition function

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel acquisition function (HIPE) for Bayesian Optimization (BO) that aims for an effective initialization of BO algorithms. HIPE incorporates surrogate model hyperparameters into the selection of new candidates to account for different importances of dimensions while aiming to be space-filling (i.e., achieving a good coverage of candidate configurations). They empirically evaluate the proposed acquisition function on BO tasks, showing HIPE's effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The work tackles the important problem of making batch BO more sample-efficient by better initialization/selection of query points\n- mostly clearly written\n- The proposed approach is simple and easy to adopt\n- extensive empirical evaluation is provided \n\n**Weaknesses**\n\n- as far as I understood, the novel acquisition function proposed is a linear combination of EPIG and BALD. Therefore, it is not necessary to discuss NIPV in Sec. 2\n- Fig. 1: it is not clear what the authors consider to be \"optimal\" in this example. It is not easy to see why HIPE selects better points than NIPV since both have broad coverage.\n- The authors do not explain why strategies like Sobol are not able to accomplish a space-filling selection (see Fig. 1 caption).\n- Prop. 1 introduces a connection between EPIG, BALD, and HIPE. However, it is not clear to me what this connection means for the proposed method. For example, does this influence the convergence properties of the BO? It would be good if the authors could comment on this.\n- A theoretical consideration of HIPE in terms of convergence would be good in general.\n- in line 209, the authors claim that \"entropy in Eq. 10 is Gaussian\". Can the authors provide proof or a reference showing that this is the case?\n- in the experimental section, it would be great if the authors could provide an ablation varying the number of batches and the batch size to see if HIPE performs well for different settings as well (esp. for Sec. 5.2, 5.3, 5.4).\n- to further strengthen the evaluation, considering other HPO-benchmarks like [1] and [2] could be beneficial.\n\n**References**\n\n[1] HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML. Arango et al. 2021.\n\n[2] HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO. Eggensperger et. al. 2021.\n\nQUESTIONS:\n- in Fig. 2, the rank of BALD and NIPV tremendously changes when considering RMSE instead of NLL. Can the authors explain why this is happening?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a new initialisation scheme for sequential experimental design methods, specifically for active learning and Bayesian Optimisation. The authors argue that previously proposed initialisation schemes may not be sufficiently space-filling and that they often do not sufficiently consider how to learn hyperparameters of the surrogate model. Based on this the authors propose hyperparameter-informed predictive exploration (HIPE) which essential combines initialisation/sampling strategies that either optimise for predictive information gain or hyperparameter learning. The initial formulation of HIPE contains a hyperparameter that should balance either optimising predictive information gain or hyperparameter learning which can be cumbersome to tune at run time. However, the authors show that this hyperparameter can be found optimally and precomputed without traditional hyperparameter tuning. The authors then illustrate the efficacy of the proposed method via a number of experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper reads clearly and is motivated well.\n- There are essentially _no_ hyperparameters in the method which is a big plus for the sequential experimental design setting.\n- Good initialisation schemes for sequential experimental design setting is an often overlooked problem that in practice can translate to big differences in performance - this paper contributes significantly to this issue.\n\nWeaknesses\n- Missing run time costs of experiments which can be important for these types of applications.\n- The method is presented only for GPs, which may be a shortcoming as BNNs and NNs are becoming more used surrogates in the sequential experimental design setting.\n- The experimental setup and evaluation is somewhat strict and specific in some senses. I elaborate below.\n\nQUESTIONS:\nI firstly list larger questions here, and below more optional questions.\n\nMy main questions in relation to aforementioned weaknesses and clarifications, which if most of were addressed/discussed would cause me to raise my score:\n1. The authors discuss how the presented method may be too expensive for some applications - would the authors consider including compute times in the work to make it clear how expensive the method is?\n2. Am I understanding correct that using the proposed method requires a candidate set of initial data points?\n3. I think some ablation studies and some more insights on experimental results would be useful. I elaborate here:\n    - Do the authors have any insights on why the init scheme seems to be relatively problem sensitive? In Figure 2, 3 and 4 there are some problem types where HIPE perform a lot worse than in other problems. Some discussion on why this might be the case would be very useful for practitioners and further development.\n    - Is there any reasons the authors only show the Sobol results in Figure 5 (right) and not all the other methods?\n    - Did the authors look at performance of their proposed method in the BO setting for iterations > 2?\n4. Do the authors have any insights on how the proposed init scheme would work for other surrogate models such as BNNs or Random Forests?\n5. I think the title is a little confusing (although this is an opinion) whilst I think the abstract is overclaiming. To me the title indicates something where the init scheme is prior knowledge dependant, but HIPE is actually \"fully automized\" requiring no prior from the user - perhaps including Hypeparameter in the title as in the method would be helpful. In the abstract, the authors write that HIPE outperforms other standard init schemes, which is not always the case - I believe this should be amended.\n\nMinor comments or questions:\n1. There is a citation missing in relation to BO surrogate quality and calibration in lines 34 and lines 136-139, see [1].\n2. The authors write \"at the core of BO is an acquisition function\", but I would argue that the surrogate is equally important.\n3. In line 162 the authors write that Scrambled Sobol sequences fail to achieve the desired coverage, but I do not see any citation providing evidence for this.\n4. In lines 163-166 the authors repeat much of what was already said in the paragraph running from lines 149-158 if I am not mistaken.\n5. Based on Figure 1, I have a hard time seeing qualitative differences between NIPV and HIPE. Maybe the authors could discuss this more in detail.\n6. In line 306 there appears to be a grammatical error in the sentence beginning with \"its HIPE\".\n\n[1]: Jonathan Foldager, Mikkel Jordahn, Lars Kai Hansen, and Michael Riis Andersen. On the role of\nmodel uncertainties in Bayesian optimization. In Uncertainty in Artificial Intelligence (UAI), 2023.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hyperparameter-Informed Predictive Exploration (HIPE), a principled initialization strategy for Bayesian Optimization and active learning that jointly balances space-filling exploration with efficient hyperparameter learning using information-theoretic objectives. The method is proposed for the large-batch and few-shot setting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The paper explores an interesting direction direction about the initialization for Bayesian Optimization and Active Learning in few-shot, large-batch scenarios.\n\n2. The experimental evaluation spans multiple settings, deterministic and noisy test functions and hyperparameter tuning, including LCBench and SVM.\n\n**Weaknesses**:\n\n1. The core idea behind HIPE offers limited novelty, essentially combining two well-established acquisition strategies, and introduces a hyperparameter $\\beta$ that may be difficult to tune. Moreover, the fast implementation of the objective builds heavily on existing techniques.\n\n2. While the authors introduce NIPV, it is not empirically evaluated in combination with BALD, leaving its practical benefits underexplored. The authors are encouraged to also test this combination.\n\n3. Aside from Proposition 1, which establishes an information-gain equivalence, the paper lacks theoretical guarantees (e.g., regret analysis) to support the proposed method’s performance.\n\n4. The current experiment presentation lacks sufficient details to enable reproducibility, particularly regarding the choice and tuning of the hyperparameter $\\beta$, which is not explicitly specified.\n\nQUESTIONS:\n1. Could the authors justify why setting $\\beta = $EIG$(y(x_*); \\theta)$ achieves a balance between the two competing objectives?\n\n2. What do the red and blue boxes represent in the right subplot of Figure 5? A legend or explanation would improve clarity.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}