Title: Exploring and Exploiting Model Uncertainty in Bayesian Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gaussian Process

Bayesian Optimization

Uncertainty Quantification
Abstract: In this work, we consider the problem of Bayesian Optimization (BO) under reward model uncertainty—that is, when the underlying distribution type of the reward is unknown and potentially intractable to specify. 
This challenge is particularly evident in many modern applications, where the reward distribution is highly ill-behaved, often non-stationary, multi-modal, or heavy-tailed.
In such settings, classical Gaussian Process (GP)-based BO methods often fail due to their strong modeling assumptions. 
To address this challenge, we propose a novel surrogate model, the infinity-Gaussian Process ($\infty$-GP), which represents a sequential spatial Dirichlet Process mixture with a GP baseline. The $\infty$-GP quantifies both value uncertainty and model uncertainty, enabling more flexible modeling of complex reward structures. Combined with Thompson Sampling, the $\infty$-GP facilitates principled exploration and exploitation in the distributional space of reward models. Theoretically, we prove that the $\infty$-GP surrogate model can approximate a broad class of reward distributions by effectively exploring the distribution space, achieving near-minimax-optimal posterior contraction rates. Empirically, our method outperforms state-of-the-art approaches in various challenging scenarios, including highly non-stationary and heavy-tailed reward settings where classical GP-based BO often fails.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers Bayesian optimization where there is not only uncertainty in the reward model parameters but also in the reward model itself. They propose a new GP-based BO method to facilitate Bayesian optimization under this relaxed modeling assumption. The authors prove theoretical approximation guarantees and empirical results on synthetic experiments and LLM experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- the paper seems to tackle an important problem, which in my view is well-motivated by the discussion of model uncertainty in the introduction\n- the paper includes both theoretical and empirical results, and the theoretical results match the minimax optimal rate, which is a strength \n- it seems that the problem of model uncertainty has not directly been studied in the research previously \n\nWeaknesses\n- as someone who does not have a Bayesian optimization background (but still an ml/optimization theory background), I found the paper quite difficult to follow; this could be because I am not familiar with the landscape of the literature or because the writing lacks clarity. to be more specific, it was a bit difficult for me to understand how this paper sits in the related work, and I think that section in particular could be explained in a way that better explains the topic to a nonexpert in this specific subfield. \n- on many of the experiments, it seems that confidence bars between the new method overlap. \n- it is not clear to me how much the extension to capture model uncertainty constitutes as a\n\nQUESTIONS:\n1. Could the authors elaborate more on the extension to high-dimensional BO problems, which were not included in the empirical valuation? \n2. Could the authors describe in more detail how their setting differs from mis-specified BO?\n3. Does equation (5) just expand the model class, and hence, allow one to avoid specifying a specific GP prior? To what extent is this broader class able to capture arbitrary reward models? (Apologies, I am not very familiar with SDP in this context.)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Traditional Gaussian Process (GP) surrogate-based Bayesian Optimisation (BO) struggles with non-stationary, multi-modal, and heavy-tailed objective function distributions $f(x)$ due to strong structural assumptions. Most of the standard GPs capture value uncertainty but not model uncertainty, leading to poor performance when the underlying Gaussian structural assumptions are violated.\n\n- This paper introduces a $\\infty-$Gaussian Process ($\\infty-$GP), a novel surrogate that constructs the reward function or the objective function as an infinite mixture of GP surfaces via a spatial Dirichlet Process (SDP). The proposed $\\infty-$GP is paired with the Thompson Sampling (TS) strategy that samples a surrogate function from the posterior and maximises it to select the next candidate for function evaluation. Additionally, they also balance between exploring the unknown search surfaces and exploiting the known search surfaces to find the optima. \n\n- The empirical results provided outperform GP-based BO methods on both synthetic and real-world tasks, especially in non-stationary and heavy-tailed noise settings. Authors have also demonstrated the applicability of their proposed method in complex tasks like LLM prompt tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n\n- In this paper authors bring in the notion of _Model Uncertainty_ aspects of the BO Process, in contrast to the _value uncertainty_ generally considered in the Bayesian optimisation literature. \n\n- Inspired by the prior literature on infinite mixture of GPs, authors have devised a novel approach $\\infty-$GP, that models the objective function as an infinite mixture of GP surfaces via a spatial Dirichlet Process (SDP). \n\n- The proposed $\\infty-$GP accounts for uncertainty in the choice of surrogate model itself (model uncertainty), not just the output values (value uncertainty), by allowing each input location to probabilistically mix over infinite GP surfaces.\n\n- Authors have combined $\\infty-$GP surrogate model with Thompson Sampling (TS) acquisition function to effectively balance between the exploration of the unknown regions and exploitation of the known regions/solutions. \n\n- In contrast to exploration-exploitation in the traditional TS-based BO methods, the proposed approach discusses the exploration-exploitation tradeoff in the distributional space wherein the exploitation refers to probabilistic selection of the existing surfaces and exploration refers to introducing a new GP surface for realising the candidate points.\n\n\n**Weaknesses** \n\n- The authors have performed posterior inference using a Gibbs sampling strategy under a fully Bayesian model. In general, Gibbs sampling over infinite (or many) GP surfaces and posterior inference can be computationally intensive, especially for large datasets. Authors have also mentioned that the number of mixture components is set to a finite value $L$ (Line 571 in B.2 of Appendix), contradicting the motivation to use an infinite mixture of GP surfaces. Further, the performance may degrade if $L$ is not chosen appropriately. \n\n- The success of the proposed sampling strategy and thus the overall BO procedure relies heavily on the prior specification of the hyperparameters (Equations 24 to 27 in Appendix B.1), including kernel width, noise variance, and the Dirichlet process concentration parameter. These priors are often highly domain-specific and task-sensitive, which may limit the method’s generalizability without some tuning efforts.\n\n- In the ablation results, the authors concluded that the reward space can be effectively modelled as a mixture of multiple latent GP surfaces. However, the paper does not discuss the existing BO literature that leverages the additive structure of the objective function. Incorporating a discussion of such approaches could provide some context and help understand the advantages of the proposed method.\n\n- Although the empirical analyses demonstrate improved performance over classical GP-based BO methods, it does not include comparisons against state-of-the-art approaches specifically designed to address non-stationarity [1,2] or exploit additive structure [3] in the objective function space, including such baselines would provide a comprehensive evaluation of the proposed method’s strengths and limitations. \n\n- In the manuscript, the authors do not discuss the implications of the sampling strategy on the regret bounds of the overall Bayesian optimisation process. Discussing the regret bounds of the proposed approach could be highly beneficial to compare with other state-of-the-art methods that generally discuss the space/time complexities and the regret bounds.\n\n**References** \n\n[1] Ruben Martinez-Cantin. Funneled Bayesian optimisation for design, tuning and control of autonomous systems. IEEE Transactions on Cybernetics, 49(4):1489– 1500, 2019. \n\n[2] Remes, Sami, Markus Heinonen, and Samuel Kaski. \"Non-stationary spectral kernels.\" Advances in neural information processing systems 30 (2017).\n\n[3] Gardner, Jacob, Chuan Guo, Kilian Weinberger, Roman Garnett, and Roger Grosse. \"Discovering and exploiting additive structure for Bayesian optimisation.\" In Artificial Intelligence and Statistics, pp. 1311-1319. PMLR, 2017.\n\nQUESTIONS:\n- How is the scalar precision parameter $\\nu$ determined? While its prior distribution is mentioned as a Gamma distribution in Appendix B.1 (Equation 27), the specific values or settings for the hyperparameters $a_\\nu$ and $b_\\nu$ are not clearly stated. Clarification on these parameters would be helpful.\n\n- How sensitive are the hyperparameters priors used in the posterior inference procedure (Gibbs Sampling)?\n\n- Is there a specific statistic/metric that quantifies the reuse of latent surfaces across iterations to support the claims about exploitation behaviour of the proposed approach (discussed in Lines 221–223)? \n\n- What is the theoretical order of the regret for the proposed method? Does it maintain sub-linear regret, similar to standard Thompson Sampling-based BO? If so, what are the key theoretical differences due to operating in the distributional space, and would they affect regret bound?\n\n- How does the proposed approach compare with existing BO methods that exploit additive structures of the objective function [1]? Given that the $\\infty-$GP uses multiple latent GP surfaces, does it implicitly capture the additive nature of the objective function, or are there fundamental differences?\n\n- While the authors address non-stationarity via input warping, several BO methods use explicitly non-stationary kernels or model heteroskedastic noise directly. Why were such approaches not included in the comparisons? A direct evaluation against BO methods with non-stationary kernels would provide a more thorough benchmark and further strengthen the benefits of the proposed approach.\n\n\n\n - **Minor Questions**\n \t- What does the y-axis represent in Figure 2? Would normalisation or standardisation of the reward values impact the performance or stability of the proposed method?\n   - What is the number of input dimensions used in the synthetic function benchmarks? Including this detail would help assess the scalability and complexity of the evaluation tasks.\n\nLIMITATIONS:\nAlthough the authors acknowledge the issue of scalability of the proposed approach to high dimensions, there are a few other important limitations that are not mentioned in the manuscript. \n\n- The method relies on Gibbs sampling for posterior inference over multiple GP surfaces, which is computationally intensive, especially for large datasets.\n\n- The performance of the $\\infty-$GP model is highly dependent on the careful selection of hyperparameter priors and these priors are often domain-specific and may not generalise well across different tasks.\n\n- The paper does not provide formal regret bounds for the proposed BO strategy, thereby limiting its theoretical interpretability and comparability to standard BO methods (backed by known sub-linear regret guarantees).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel surrogate model for Bayesian Optimization (BO), the infinity-Gaussian Process ($\\infty$-GP), designed to address challenges where the reward distribution is complex, non-stationary, or heavy-tailed. The $\\infty$-GP is formulated as a sequential spatial Dirichlet Process mixture with a Gaussian Process (GP) baseline. This approach allows the model to quantify not just value uncertainty but also *model uncertainty*—uncertainty about the underlying distributional form of the reward. The authors provide a theoretical analysis demonstrating that the $\\infty$-GP posterior can approximate a broad class of reward distributions with a near-minimax optimal posterior contraction rate. Empirically, the proposed $\\infty$-GP combined with Thompson Sampling ($\\infty$-GP-TS) is shown to outperform standard GP-based BO methods on various challenging synthetic and real-world benchmarks, including a prompt optimization task.\n\nSTRENGTHS AND WEAKNESSES:\n#### **Strengths**\n\n- **Novelty and Significance:** The paper proposes a novel and principled approach ($\\infty$-GP) to handle model misspecification in BO, which is a significant and practical problem. Standard GP-based methods often fail when their strong modeling assumptions are violated, and the $\\infty$-GP offers a flexible alternative by creating an infinite mixture of GP surfaces.\n- **Theoretical Justification:** A solid theoretical contribution supports the work. Theorem 1 establishes posterior convergence guarantees for the $\\infty$-GP model, demonstrating its ability to approximate a broad class of reward distributions at a near-minimax-optimal rate. This provides confidence in the model's robustness, extending beyond its empirical performance.\n- **Empirical Validation:** The experimental evaluation is comprehensive, covering synthetic benchmarks with non-stationary and heavy-tailed properties, real-world tasks like portfolio optimization and neural architecture search, and a modern application in LLM prompt optimization. The results consistently show the superiority of $\\infty$-GP-TS over standard GP baselines in these challenging settings.\n\n\n**Weaknesses**\n\n1.  **Limited Literature Review on Model Uncertainty:** The concept of \"model uncertainty\" is framed as a key motivator, yet the literature review on this topic feels incomplete. This issue has been previously addressed under the umbrella of hyperparameter learning for surrogate models. A relevant and missing work is \"Self-Correcting Bayesian Optimization through Bayesian Active Learning\" by Hvarfner et al. (2024), which also actively manages model uncertainty. The current related work section could be strengthened by positioning the $\\infty$-GP in the context of these alternative approaches.\n\n2.  **Missing State-of-the-Art Baselines:** The paper compares $\\infty$-GP-TS against four standard academic baselines: GP-TS, GP-UCB, GP-EI, and GP-KG. While these are foundational, the comparison lacks more recent, powerful, and industrially relevant baselines. For instance, HEBO, which explicitly handles heteroscedastic and non-stationary rewards via input and output warping, has demonstrated state-of-the-art performance. The recent Vizier report (2024) also highlights robust methods for these scenarios. Including a comparison to at least one such advanced baseline would provide a more convincing measure of the proposed method's practical utility.\n\n3.  **Lack of Regret Analysis:** While the paper provides a posterior convergence guarantee (Theorem 1), it does not offer a theoretical analysis of the cumulative regret for the proposed $\\infty$-GP-TS algorithm. Regret bounds are a standard way to characterize the sample efficiency of a BO algorithm. Without this analysis, the theoretical understanding of the algorithm's exploration-exploitation trade-off remains incomplete.\n\n4. **Insufficient Experimental Details:** Although the empirical studies are broad, they lack key details necessary for full reproducibility.\n    * For **HPOBench**, the paper states it uses the \"MLP task\" but does not specify the task ID or the number of training epochs, making it difficult to replicate the results.\n    * The **prompt optimization** section is missing comparisons to domain-specific BO baselines. The setup could also be clarified further regarding the construction of the UMAP search space.\n    * The hyperparameter choices for each experiment are not explicitly discussed.\n\n\nReference:\n- Hvarfner, Carl, Erik Hellsten, Frank Hutter, and Luigi Nardi. \"Self-correcting bayesian optimization through bayesian active learning.\" Advances in Neural Information Processing Systems 36 (2023): 79173-79199.\n\n- Cowen-Rivers, Alexander I., Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval et al. \"Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation.\" Journal of Artificial Intelligence Research 74 (2022): 1269-1349.\n\n- Song, Xingyou, Qiuyi Zhang, Chansoo Lee, Emily Fertig, Tzu-Kuo Huang, Lior Belenki, Greg Kochanski et al. \"The vizier gaussian process bandit algorithm.\" arXiv preprint arXiv:2408.11527 (2024).\n\nQUESTIONS:\n1.  What is the conceptual relationship between the proposed $\\infty$-GP model and the **spectral mixture kernel**? Both are designed to model complex, non-stationary functions. Does $\\infty$-GP offer any distinct advantages over using a highly flexible kernel within a standard GP framework?\n\n2.  Could the authors further elaborate on the statement in lines 143-145 regarding the role of the noise term $\\epsilon_i$? The authors state it is introduced for \"modeling purposes to mix with the discrete-distributed $\\xi(x)$ and produce a continuous reward distribution\".\n\n3.  Could you provide more specific details for the empirical studies to improve reproducibility? Specifically:\n    * **HPOBench:** What is the specific task ID for the MLP experiment from the benchmark suite?\n    * **Prompt Engineering:** How was the UMAP projection configured (e.g., number of components, neighbors)? Were other dimensionality reduction techniques considered?\n    * **General:** How many MCMC iterations (B) and mixture components (L) were used in the Gibbs sampler (Algorithm 2) for the experiments?\n\nLIMITATIONS:\nNot discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. This paper tackles the challenge of Bayesian Optimization when the reward distribution is unknown or complex (e.g., multi-modal or heavy-tailed). \n\n2. the authors propose a novel surrogate model $\\infty$-GP, which explicitly handles both uncertainty about the reward's value and uncertainty about its underlying distribution type. \n\n3. The authors provide theoretical proof that their method can approximate a wide class of reward distributions and show empirically that it outperforms SOTA in difficult optimization scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written. It clearly presents the problem domain, thoroughly reviews related work, and precisely outlines its core contributions.\n\n2. The proposed $\\infty$-GP model is rigorously tested across a wide range of synthetic and real-world experiments. The results consistently show that the method outperforms state-of-the-art baselines, particularly in challenging scenarios.\n\n3. A significant strength of this work is the inclusion of theoretical guarantees for the $\\infty$-GP model. These guarantees are established under mild, reasonable assumptions, providing a solid theoretical foundation for the method's strong empirical performance.\n\nWeakness:\n\n1. To address the limitations of Gaussian Processes (GPs) in modeling complex reward functions, the paper introduces a Spatial Dirichlet Process (SDP) prior, $\\operatorname{SDP}(\\nu G_0)$, placed on the GP to improve model uncertainty estimation. However, the resulting uncertainty estimates are sensitive to the choice of hyperparameters $G_0$ and $\\nu$. The paper does not provide a theoretical or empirical analysis of how the model uncertainty depends on these hyperparameters, nor does it assess the sensitivity of the results to their variation. This lack of discussion raises concerns about the robustness and reproducibility of the proposed method, particularly in settings where appropriate values for $G_0$ and $\\nu$ are difficult to determine a priori.\n\n2. In Thompson Sampling, Gibbs sampling is required to draw samples from the posterior. However, Gibbs sampling can become computationally expensive as the dimensionality of the parameter space increases. The paper does not provide any discussion or analysis of the time complexity or scalability of the method. I am concerned about whether the proposed approach can be implemented efficiently and remain practical in high-dimensional settings.\n\n3. [Minor]. While the appendix provides experimental details, the main paper could be improved by more explicitly defining the performance metrics used in the plots and their corresponding discussions. Clearly stating what each axis and value represents directly in the captions or main text would enhance readability.\n\nQUESTIONS:\nOverall, this is a strong paper with compelling contributions. The idea of quantifying model uncertainty by placing a prior over Gaussian Processes (GPs) is natural and well-motivated, and the proposed approach represents a meaningful extension of mixture-of-experts (MOE) GPs. I think this work is a nicely-executed work. I have a few questions and comments below:\n\nQ1: The paper would be significantly strengthened by including a time complexity analysis. A formal analysis comparing the proposed method to the other baselines would be invaluable. Furthermore, reporting empirical runtime comparisons in the experiments would provide practical insight into the method's efficiency.\n\nQ2: Can the authors provide theoretical guarantees or further discussion on how the model uncertainty depends on the Spatial Dirichlet Process prior, $\\operatorname{SDP}(\\nu G_0)$? Specifically, I am curious about the sensitivity of the resulting uncertainty estimates to the choice of hyperparameters $\\nu$ and $G_0$, and how strongly this prior influences the posterior. How robust is the method to different human-specified priors?\n\nQ3: Regarding your main theoretical contribution (Theorem 1), could you provide more discussion on how your results compare to existing work? While you briefly mention that your assumptions are weaker than those in [Kanagawa et al., 2018], it would be helpful to elaborate on this point. Specifically, could you clarify what aspects of the assumptions are relaxed, and how your theoretical results—such as convergence rates—compare quantitatively or qualitatively with prior work?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}