Title: Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Black Box Optimization

Bayesian Optimization

Energy-Based Model

Reinforcement Learning
Abstract: Existing Bayesian Optimization (BO) methods typically balance exploration and exploitation to optimize costly objective functions. However, these methods often suffer from a significant one-step bias, which may lead to convergence towards local optima and poor performance in complex or high-dimensional tasks. Recently, Black-Box Optimization (BBO) has achieved success across various scientific and engineering domains, particularly when function evaluations are costly and gradients are unavailable. Motivated by this, we propose the Reinforced Energy-Based Model for Bayesian Optimization (REBMBO), which integrates Gaussian Processes (GP) for local guidance with an Energy-Based Model (EBM) to capture global structural information. Notably, we define each Bayesian Optimization iteration as a Markov Decision Process (MDP) and use Proximal Policy Optimization (PPO) for adaptive multi-step lookahead, dynamically adjusting the depth and direction of exploration to effectively overcome the limitations of traditional BO methods. We conduct extensive experiments on synthetic and real-world benchmarks, confirming the superior performance of REBMBO. Additional analyses across various GP configurations further highlight its adaptability and robustness. Our code is publicly available at: https://github.com/ruiyaoMiao0809/Black-Box-Bayesian-Optimization-with-Energy-Based-Model-and-Reinforcement-Learning

Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning

Abstract

Existing Bayesian Optimization (BO) methods typically balance exploration and exploitation to optimize costly objective functions. However, these methods often suffer from a significant one-step bias, which may lead to convergence towards local optima and poor performance in complex or high-dimensional tasks. Recently, Black-Box Optimization (BBO) has achieved success across various scientific and engineering domains, particularly when function evaluations are costly and gradients are unavailable. Motivated by this, we propose the Reinforced EnergyBased Model for Bayesian Optimization (REBMBO), which integrates Gaussian Processes (GP) for local guidance with an Energy-Based Model (EBM) to capture global structural information. Notably, we define each Bayesian Optimization iteration as a Markov Decision Process (MDP) and use Proximal Policy Optimization (PPO) for adaptive multi-step lookahead, dynamically adjusting the depth and direction of exploration to effectively overcome the limitations of traditional BO methods. We conduct extensive experiments on synthetic and real-world benchmarks, confirming the superior performance of REBMBO. Additional analyses across various GP configurations further highlight its adaptability and robustness. Our code is publicly available at:https://github.com/ruiyaoMiao0809/Black-BoxBayesian-Optimization-with-Energy-Based-Model-and-Reinforcement-Learning

1 Introduction

Black-box optimization (BBO) is crucial for solving complex scientific and engineering problems when gradient information is unavailable or function evaluations are expensive [1]. In practice, BBO approaches are widely applied to hyper-parameter tuning in machine learning, materials discovery, drug formulation, and industrial process optimization, where each evaluation often incurs costly simulations or physical trials. Bayesian Optimization (BO) is a prominent BBO technique that builds a probabilistic surrogate (e.g., a Gaussian Process [2]) and an acquisition function to guide new sample queries, thereby balancing exploration and exploitation in a principled manner. However, standard GP-based BO can suffer from 'one-step myopia,' focusing on short-term predicted gains at the expense of more thorough exploration, a limitation that becomes especially pronounced in high-dimensional or multi-modal environments.

Existing Bayesian Optimization (BO) methods primarily aim at efficiently locating optimal solutions by carefully balancing exploration and exploitation [3]. Common strategies for handling complex

optimization problems include dimensionality reduction methods like REMBO [4], or local partitioning techniques such as TuRBO [5]. While effective in relatively simple settings, these approaches tend to converge prematurely to local optima when applied to complex, high-dimensional problems. [6, 5]. To solve this limitation, recent techniques incorporate resource-intensive multi-step look-ahead schemes, including 2-step Expected Improvement (EI) [7], Knowledge Gradient (KG) [8], and reinforcement learning-driven methods like EARL-BO [9]. However, such methods typically demand significant computational resources yet still fail to achieve effective global exploration in challenging environments.

In this study, we introduce the Reinforced Energy-Based Bayesian Optimization Model (REBMBO), depicted in Figure 1, which addresses traditional shortfalls by combining an Energy-Based Model (EBM) with multi-step Reinforcement Learning (RL). Our novel EBM-UCB acquisition function integrates Gaussian Process local uncertainty estimates with global signals derived from a neural network-based energy landscape learned via short-run MCMC, thereby guiding exploration away from less promising regions. In addition, we treat each Bayesian Optimization iteration as a Markov Decision Process (MDP) and employ Proximal Policy Optimization (PPO) for adaptive multi-step lookahead, thus dynamically adjusting exploration depth and direction to enhance robustness. To capture both local and global exploration objectives, we propose a theoretically justified LandscapeAware Regret (LAR) metric that incorporates global exploration penalties, offering a more holistic assessment of performance in complex optimization scenarios.

(1) We incorporate Energy-Based signals into a UCB-style acquisition function in the GP surrogate to capture diverse behaviors. This synergy between global exploration and precise local modeling addresses the limitations of single-step acquisition approaches.

(2) Bayesian Optimization is modeled as a Markov Decision Process (MDP) with Proximal Policy Optimization (PPO) . This technique addresses the one-step myopia of typical GP-based strategies by adaptively balancing exploration and exploitation via multi-step lookahead.

(3) We introduce a theoretically justified Landscape-Aware Regret (LAR) metric that extends standard regret with an energy-informed global term. This metric provides a fair and balanced evaluation by jointly reflecting local exploitation and global exploration efficiency in complex optimization landscapes.

Experimental results, summarized in Figure 2, indicate that REBMBO reduces final LandscapeAware Regret (LAR) and improves overall performance scores compared to state-of-the-art methods, consistently outperforming both single-step and short-horizon lookahead approaches even under challenging high-dimensional conditions. The subsequent sections detail our methodology and empirical findings, highlighting REBMBO's robust and efficient performance.

2 Related Work

2.1 BO Background and Shortcomings

Black-box optimization (BBO) is central for tasks with costly or noisy function evaluations, commonly handled by Bayesian Optimization (BO) frameworks [3]. However, high-dimensional or discrete domains often overwhelm classical Gaussian Process (GP) surrogates, prompting techniques such as ARD-based variable selection [10], REMBO [4], additive models [11], or local partitioning in TuRBO [5]. Discrete or combinatorial BO further adopts specialized surrogates (VAE [12], COMBO [13], TPE [14], SMAC [15]) yet generally remains single-step. Likewise, robust or constrained variants [16, 17, 18, 19, 20], multi-objective [21, 22, 23], transfer/multi-fidelity [24, 25, 26, 27], and parallel [28, 29, 30] approaches usually retain one-step acquisitions. TruVaR (Truncated Variance Reduction) [8] unifies BO and level-set estimation with strong guarantees under pointwise costs or heteroscedastic noise, while look-ahead or rollout-based schemes [7 ? ] often incur high computational overhead. Recent work leverages MLMC for nested integrals [31] or formulates BO as an MDP under transition constraints [32], and methods like GLASSES [33] approximate multi-step losses via forward simulation. Yet, many remain domain-specific or lack synergy with short-run MCMC. Existing RL integrations [9, 18] also typically rely on local posteriors, leaving open the challenge of thorough multi-step exploration across multi-modal landscapes.

2.2 Baseline Targeting Global Optima and Limitations

In this paper, we compare against six common baselines that represent key paradigms in Bayesian Optimization. Classic BO [3] is a canonical single-step GP-based approach. BALLET-ICI [6] alternates global and local GPs but remains relatively myopic on multi-modal tasks. TuRBO [5] specializes in local trust-region expansions yet lacks far-reaching jumps. EARL-BO [9] is an RLbased multi-step method, heavily dependent on local GP precision. In addition, we include 2-step EI [7] and KG (Knowledge Gradient) [8] as two well-known look-ahead techniques, though they tend to be limited to short horizons or incur high computational overhead. These baselines respectively illustrate single-step local search, partially global scanning, or short-horizon non-myopia, but none combines global signals with adaptive multi-step planning in a unified manner. By contrast, REBMBO employs a short-run MCMC-trained Energy-Based Model for global exploration, a GP surrogate for local accuracy, and a PPO-based multi-step RL for planning. This synergy overcomes the one-step constraints in Classic BO, enables deeper exploration than BALLET-ICI or TuRBO, provides more robust coverage than EARL-BO, and avoids the excessive rollout overhead observed in 2-step EI or KG. As detailed in Section 2.3, REBMBO leverages these three modules to handle high-dimensional tasks within limited budgets, offering a global and multi-step perspective.

2.3 RL in BO and Energy-Driven Multi-Step Planning

Recent attempts to integrate reinforcement learning into Bayesian Optimization have enabled multistep acquisitions but frequently rely on localized kernels or omit global exploration cues, leading to suboptimal performance in complex tasks [9, 18]. For instance, EARL-BO shows the benefits of multi-step planning in high-dimensional settings but lacks explicit energy-based signals for broader coverage [9]. However, REBMBO framework formulates each BO iteration as a MDP solved via Proximal Policy Optimization. This design alleviates one-step myopia and combines local GP fidelity with iterative RL lookahead under strict evaluation budgets.

3 Preliminaries

Online Black-Box Optimization (BBO). We consider a continuous function f ( x ) defined over x ∈ X ⊂ R d , with the objective:

under a strict evaluation budget. Each evaluation of f ( x ) can be computationally or financially expensive [34], thus data efficiency is crucial. Unlike offline methods that rely on fixed sampling designs, online BO adaptively selects x t based on previously observed data, facilitating faster discovery of optimal regions.

Bayesian Optimization (BO) and Gaussian Processes (GP). BO maximizes max x ∈X f ( x ) by employing a Gaussian Process (GP) prior: f ( x ) ∼ GP ( m ( x ) , k ( x , x ′ ) ) , where typically m ( x ) = 0 , and the kernel function k ( x , x ′ ) (e.g., RBF or Matérn) encodes assumptions about the function's smoothness [35]. Assuming noisy observations y i = f ( x i ) + ε i , with ε i ∼ N (0 , σ 2 n ) , the GP posterior is given by:

with observed data D k = { ( x i , y i ) } k i =1 . Although GPs capture local uncertainty, their inherent locality often restricts global exploration, making them prone to myopic optimization behaviors.

Energy-Based Models (EBMs). EBMs specify an unnormalized probability density:

where the energy function E θ ( x ) is typically parameterized as a neural network through short-run MCMC-based Maximum Likelihood Estimation [36, 37]. EBMs effectively guide exploration toward globally promising regions. Unlike GPs, EBMs explicitly capture multi-modal global structures, thus addressing the limitation of excessive local exploration inherent in standard GP-based methods.

Reinforcement Learning (RL) and Proximal Policy Optimization (PPO). RL formalizes the optimization process as a sequential decision-making task, wherein a policy π ϕ ppo , parameterized

by neural network weights ϕ ppo , maps states s t to actions a t . PPO [38] stabilizes the training by limiting policy changes through a clipped probability ratio:

thereby preventing erratic changes in the parameters. We define states as combinations of GP posterior estimates and global EBM signals; actions match suggested sampling points. The use of PPO helps to overcome the single-step myopia that is inherent in conventional acquisition methods. This is accomplished through the use of multi-step reasoning.

4 REBMBO Model & Algorithmic Details

After updating the GP model (which focuses on local predictions) and the EBM model (which looks at global patterns), REBMBO uses the PPO technique to choose the next sample point by combining both local and global information. The GP posterior, EBM signals, and PPO's multi-step planning help REBMBO find good solutions in complex spaces with many dimensions or peaks. PPO's planning horizon mitigates the short-sightedness of single-step approaches. Overall, REBMBO provides a unified solution for balancing global exploration and sequential decision-making in challenging black-box optimization settings in Figure 1.

4.1 Module A: Gaussian Process Variants

As new data comes into the REBMBO framework, it first goes to Module A (the Gaussian Process surrogate, shown in Figure 1) to improve the estimate of the objective function f ( x ) . In particular, the GP uses the input-output data it has seen to improve its predictions about the average outcome and the level of uncertainty in specific areas, ensuring precise local modeling with each update. Local modeling is essential for black-box optimization because it lets us acquire solid insights from a few observations before sampling the full domain.

We add different GP modules in REBMBO because many black-box functions are complex, have multiple peaks, or are costly to compute, so we need a flexible "core" model that can quickly adjust and measure uncertainty with just a few samples. GPs provide estimates of the average and variability for f ( x ) , making them useful for identifying significant local trends, particularly when limited data is available at the start. To deal with various problem dimensions and complexities, we propose three REBMBO variants:

(1) REBMBO-C (Classic GP) [39]. Employs exact O ( n 3 ) GP inference, which is practical for moderate n . While this variant is straightforward, it can be costly for large n or high-dimensional d .

(2) REBMBO-S (Sparse GP) [40]. Adopts a sparse approximation to alleviate the O ( n 3 ) bottleneck in higher dimensions. It introduces m ≪ n inducing points { z j } and approximates K x , x ≈ K x , z K -1 z , z K z , x , lowering the update cost to O ( nm 2 ) . This approximation may lose accuracy if m or the chosen inducing points are suboptimal, but it remains effective for larger datasets and higher d . In our EBM-driven acquisition, the approximate mean ˜ µ f,t ( x ) and variance ˜ σ 2 f,t ( x ) replace the exact GP posteriors.

Posterior Mean: µ

t

(

x

)

Posterior      Std:

σ

t

(x)

Action =

EBM for Global Converge

Short Run MCMC for

Global Energy.

𝜋ϕ

← optimize via RL

objective

GP Surrogate

State & Action Agent

Energy Based Model

Black-Box Oracle

Reward & Policy Agent

Dynamic Data Loader

Update GP

Initial Data D_

θ

Train EBM

Next Iteration(Update D _ t)

RL Action A1

Measure f(x

）

Evaluate f(x)

Reward & New Policy

Data Processor

Module A

Module B

Module C

Update Reward

Initial Iteration(Update

D_

θ

)

Clip

𝑟 𝑡

(𝑠 𝑡

, 𝑎

𝑡

) =  𝑛𝑓 𝑎 𝑡

- 𝜆𝐸𝜃(𝑎𝑡)

(3) REBMBO-D (Deep GP) [41]. For problems that exhibit multi-scale or non-stationary behavior, a deep kernel can capture intricate latent features beyond what standard kernels provide. We use a deep network Θ to map inputs x into latent features ϕ GP ( x ) , then compute GP-like statistics:

where η ( x ) is a (potentially learned) mean function, and K is a smaller covariance matrix in the latent space. With sufficient training data, this approach can represent complex functions more flexibly than a standard GP and supports sublinear regret under moderate network capacity.

All three GP variants work with the EBM-UCB and PPO modules to create the full REBMBO system; the main difference is in how each variant calculates its GP posterior. For simplicity, we will show EBM-UCB using the 'Classic GP' version (REBMBO-C), but the same idea applies to the sparse and deep GP versions. More information about differences of GP variants, including their complexity and how to implement them, can be found in Appendices F, J and K.

As updated data then moves to Module B, REBMBO addresses the limitations of purely local exploration by introducing an Energy-Based Model to capture global structure. The next section (Module B) explains that the EBM helps the GP surrogate by directing the search away from areas that aren't useful and toward more promising ones, especially in complex or varied situations.

4.2 Module B: EBM-Driven Global Exploration

To overcome the limitations of purely local GP-based search, we introduce an Energy-Based Model (EBM) defined as E θ ( x ) . After Module A updates the GP posterior with newly collected data (see Figure 1), Module B uses these updates to train the EBM, which captures a global 'energy' landscape. While the GP's uncertainty σ f,t ( x ) pinpoints locally undersampled regions, the EBM reveals which basins in X are more likely to contain near-optimal solutions. Combining these local and global insights helps prevent the search from stalling in unproductive local pockets and enables REBMBO to traverse complex objective surfaces more efficiently.

EBM Training Mechanism. We parameterize E θ ( x ) as a neural network trained under short-run MCMC-based Maximum Likelihood Estimation (MLE). At each iteration, we alternate:

Positive Phase: Lower E θ ( x i ) for real data points x i , guiding the model to 'observed' regions.

Negative Phase: Draw a small number ( K ) of Langevin samples from p θ ( u ) ∝ exp[ -E θ ( u )] , then push these model-generated samples to higher energy unless they reflect data-like features. This short-run MCMC procedure (e.g. Stochastic Gradient Langevin Dynamics, detailed in Appendix G) ensures that low-energy regions correspond to promising global basins.

EBMParameterization Details. We specifically train E θ ( x ) via short-run MCMC-based MLE as follows. Suppose we have data { x i } n i =1 from an unknown distribution p data . We fit θ by minimizing

which is equivalent to maximizing ∑ n i =1 log p θ ( x i ) . Let L ( θ ) = ∑ n i =1 log p θ ( x i ) , where p θ ( x i ) ∝ exp[ -E θ ( x i )] .Then, we have

Dividing by n yields the well-known positive-minus-negative decomposition:

which balances data alignment against model-drawn samples [42, 43]. Since short-run MCMC approximates p θ ( u ) sufficiently well, it lets us implement a Robbins-Monro-style gradient update [44]. Iteratively alternating positive and negative phases steers E θ ( x ) to be low in data-like basins and high elsewhere, thus revealing globally promising regions for exploration.

EBM-UCBAcquisition Function. Once the EBM is trained, we embed its negative energy -E θ ( x ) into a standard GP-UCB scheme. Let µ f,t ( x ) and σ f,t ( x ) be the GP posterior mean and standard deviation at iteration t . A typical UCB function is α UCB ( x ) = µ f,t ( x ) + β σ f,t ( x ) , where β > 0 controls exploitation vs. exploration. To incorporate the EBM's global guidance, we define

where γ > 0 specifies how strongly -E θ ( x ) biases the search toward underexplored basins. In multimodal and high-dimensional tasks, this 'global penalty' helps avoid wasting evaluations in uncertain but unpromising pockets, augmenting the GP's local exploration with a broader sense of global structure. As further discussed in Section 5, this synergy accelerates convergence on challenging landscapes and reduces the need for purely local or manually specified look-ahead heuristics.

Theoretical Contributions and Landscape-Aware Regret (LAR). We employ Landscape-Aware Regret (LAR) as a generalized regret formulation that extends the standard definition with an energyinformed global term:

where α ≥ 0 controls the relative influence of the global energy term. For non-energy-based baselines, we set α =0 to recover standard regret and ensure fair comparison, while for energy-aware methods such as REBMBO, α> 0 provides a holistic measure that captures missed global opportunities in the learned landscape. Under mild alignment and regularity assumptions (Appendix E), our EBMUCB retains the GP-UCB-type sublinear rate, so incorporating E θ ( x ) preserves the same optimality guarantees as standard regret [42, 43].

Mixture kernel for the GP posterior (rationale + form). The GP posterior is computed by inverting an n × n kernel matrix built from a mixture of Radial Basis Function (RBF) and Matérn covariances:

with k RBF ( x , x ′ ) = exp ( -1 2 ( x -x ′ ) ⊤ Λ -1 ( x -x ′ ) ) and, for ν =2 . 5 and r = ∥ x -x ′ ∥ , k Matern ( x , x ′ ) = ( 1 + √ 5 r ℓ + 5 r 2 3 ℓ 2 ) exp( -√ 5 r ℓ ) . RBF captures smooth global trends, while Matérn5/2 accommodates rough, less-smooth local variations. The mixture enlarges the RKHS compared to either kernel alone, which matches REBMBO's design: the EBM offers global basin cues, and the GP needs both smooth (RBF) and rough (Matérn) components to model local structure faithfully. The mixture weights { w RBF , w Matern } are learned by type-II marginal likelihood (evidence maximization), avoiding per-task manual tuning.

By unifying the global signal E θ ( x ) with these locally expressive GP statistics (via the mixture kernel), REBMBO couples principled global exploration with precise local modeling; Module C then employs PPO-based multi-step planning to mitigate one-step myopia and fully exploit this synergy.

4.3 Module C: Multi-Step Planning via PPO

While α EBM -UCB ( x ) enhances global exploration over local approaches, a single-step acquisition can still cause local myopia. The algorithm prioritizes instant rewards above long-term queries. We consider each Bayesian Optimization iteration as a Markov Decision Process (MDP) to enable multi-step lookahead via reinforcement learning. Although Proximal Policy Optimization (PPO) [38] is well-known, our work combines it with the GP surrogate and the EBM's global energy signal. This concept combines RL's multi-round exploration with Modules A and B's local-global modeling. Figure 1 illustrates how Module B updates the EBM, and Module C guides PPO-based policy adjustments based on local uncertainty and global energy cues.

MDPFormulation: States, Actions, and Rewards. At iteration t , we define the state

where µ f,t ( x ) , σ f,t ( x ) come from the current GP posterior, and E θ ( x ) denotes the learned global energy map. The action is the proposed query point a t ∈ X ⊂ R d ; evaluating f ( a t ) updates the GP and EBM for the next state s t +1 .

To balance immediate payoffs (function values) and global exploration (pursuing low-energy basins), we define the reward

where λ > 0 governs how strongly -E θ ( a t ) influences exploration. A higher λ promotes thorough global searching, while a lower λ emphasizes direct improvement in f ( a t ) . By embedding E θ in the reward, we ensure that REBMBO actively targets regions the EBM deems globally promising.

PPO Training Process. We employ a stochastic policy π ϕ ppo ( a t | s t ) to maximize the cumulative reward over T steps. Though PPO is an established RL algorithm [38], our adaptation ensures it co-evolves with both the GP posterior and the EBM distribution, rather than being a standalone module. Concretely, we define r t ( ϕ ppo ) = π ϕppo ( a t | s t ) π ϕ old ppo ( a t | s t ) , which measures how much the new policy

π ϕ ppo deviates from the previous one π ϕ old ppo . The clipped objective to be maximized is

where ̂ A t is an advantage estimate derived from r t ( s t , a t ) minus a learned baseline. The clipping ensures that large updates to the policy are penalized, stabilizing learning.

4.4 Overall Methodology and Synergy of GP, EBM, and PPO

After each query a t is evaluated, REBMBO synchronously updates three components: 1) GP Posterior Update: Incorporate ( a t , f ( a t )) to refine µ f,t +1 and σ f,t +1 , preserving reliable local predictions. 2) EBM Retraining: Run short-run MCMC with the expanded dataset to improve E θ ( x ) (Section 4.2), thereby maintaining a coherent global energy landscape. 3) PPO Policy Optimization: Use the new reward r t = f ( a t ) -λE θ ( a t ) and the transition ( s t , a t , s t +1 ) to update π ϕ ppo via the clipped objective L CLIP ( ϕ ppo ) . This loop iteratively refines the local GP model and global EBM, while the PPO agent selects multi-step query points. Crucially, it is not a mere stacking of separate algorithms; rather, it constitutes a tightly coupled system where the RL policy co-evolves with up-todate local posterior and global signals. The EBM term -E θ ( x ) augments UCB-based sampling with long-range structure, and PPO transforms this single-step acquisition into an MDP-based multi-round planner, thereby mitigating the near-sightedness of conventional BO.

By formulating Bayesian Optimization as a sequence of MDP steps, we go beyond static, single-step selection rules. Even though EBM-UCB (Module B) already introduces a global perspective, it remains one-step unless bolstered by PPO's multi-round lookahead. The reward function r t = f ( a t ) -λE θ ( a t ) drives the policy toward robust global basins, balancing immediate gains and exploratory push. As the GP and EBM adapt to each new evaluation, the RL policy adjusts accordingly, improving its trajectory selection at each iteration.

Putting It All Together. Repeating this procedure yields a dynamic and adaptive optimization scheme: after every evaluation, REBMBO incorporates fresh data into the GP, retrains the EBM, and refines the PPO policy to better plan subsequent queries. Section 5 presents empirical results showing how this synergy enables REBMBO to tackle high-dimensional, multi-modal functions more effectively than single-step or purely local methods, while our theoretical analysis (Appendix E) ensures sublinear Landscape-Aware Regret (LAR) under mild assumptions. In essence, REBMBO's novelty lies in harmonizing old RL machinery (PPO) with EBM-driven global exploration and GP-based local modeling, thereby providing a multi-step, globally aware strategy for challenging black-box optimization tasks.

5 Experiments

5.1 Experiment Setups

In this study, we evaluate REBMBO (variants C, S, D) against leading Bayesian Optimization (BO) methods across multiple synthetic tasks, including Branin in 2D, Ackley in 5D, Rosenbrock in 8D, and high-dimensional BO (HDBO) in 200D, and real-world tasks such as Nanophotonic in 3D and Rosetta in 86D, shown in Figure 2. Baselines include BALLET-ICI, TuRBO, EARL-BO, and Classic BO, representing varied modeling strategies from local Gaussian Processes (TuRBO) to single-step

Algorithm 1 REBMBO

Require: GP config ({Classic, Sparse, Deep}), EBM config ( E θ , MCMCsteps), PPO config ( π ϕ ppo , clip ϵ , mini-batch size), and an initial dataset D 0 of size n 0 .

1: Train GP on D 0 to obtain ( µ 0 , σ 0 ) .

2: Initialize EBM E θ ( x ) and PPO policy π ϕ ppo .

3: for t = 1 to T do

4: (A) Update the GP with D t -1 , yielding ( µ t , σ t ) .

5: (B) Retrain or partially train the EBM using data in D t -1 (via short-run MCMC).

6: (C) Form the RL state: s t ← [ µ t ( · ) , σ t ( · ) , E θ ( · )] .

7: (D) Select action: x t ← π ϕ ppo ( s t ) .

8: (E) Evaluate: y t ← f ( x t ) (expensive black-box call).

9: (F) Compute reward: r t ← y t -λE θ ( x t ) ; update π ϕ ppo with ( s t , x t , r t ) via PPO.

10: (G) Augment dataset: D t ←D t -1 ∪ { ( x t , y t ) } .

11: end for

12: Return the best sampled point x ∗ ∈ D T in terms of f ( x ∗ ) .

Branin 2D (mean ± std, 5 runs)

Ackley 5D (mean ± std, 5 runs)

Rosenbrock 8D (mean ± std, 5 runs)

Landscape-Aware Regret (LAR)

ClassicBO

EARL-BO

TuRBO

BALLET-ICI

REBMBO-C

REBMBO-S

REBMBO-D

Landscape-Aware Regret (LAR)

12.5

20.0

17.5

15.0 -

REBMBO-C

REBMBO-S

REBMBO-D

BALLETICI

EARL-BO

TuR80

ClassicBO

)

50000

40000

00009

EARL-BO

REBMB0-S

REBMB0-D

BALLETICI

TuR8O

ClassicBO

REBMBO-C

10.0

00000

7.5

5.0

20000

2.5

ue"

10000

0°0

F0

0

5

10

15

20

25

30

0

5

10

15

20

25

30

0

5

10

15

20

25

30

Iteration

Iteration

Iteration

HDBO 200D (mean ± std, 5 runs)

Rosetta 86D (mean ± std, 5 runs)

REBMB0-C

REBMB0-C

Landscape-Aware Regret (LAR)

100

120-

Landscape-Aware Regret (LAR)

5

REBMBO-S

REBMBO-D

EARL-BO

TUuRBO

ClassicBO

BALLET-ICI

6

5

EARL-B0

REBM80-5

REBMB0-D

BALLET-ICI

TURBO

ClassicBO

40-

REBMBO-C

20 -

EARL-BO

REBMBO-S

REBMBO-D

BALLET-ICI

1

o

ClassiBO

TuR8O

0

10

20

30

40

50

0

10

20

30

40

50

0

10

20

30

40

50

Iteration

Iteration

Iteration

RL (EARL-BO) and iterative confidence intervals (BALLET-ICI). We extend this analysis in Table 1 and Table 2 with two additional real-world tasks (NATS-Bench in 20D, Robot Trajectory in 40D) and two more baselines (Two-step EI, KG), covering a wider set of approximate lookahead methods and practical optimization cases. All algorithms are quantitatively compared using a Landscape-Aware Regret (LAR) metric: R e t = [ f ( x ∗ ) -f ( x t )] + α [ E θ ( x ∗ ) -E θ ( x t )] , which jointly assesses local suboptimality and global exploration; all reported values reflect the mean ± standard deviation over 5 independent runs. Further details on these baselines and benchmarks are provided in Appendix A, including the rationale for selecting tasks and the chosen hyperparameter settings (such as 10-20 short-run MCMC steps per iteration in the EBM, a 2-layer policy network with 64-256 hidden units for PPO, and Matérn-RBF kernel mixtures in the GP). Notably, REBMBO-D (see Section 3.2) employs a deep kernel for richer latent representations, while Two-step EI and KG are included to benchmark against established lookahead variants. The final scores in the tables reflect the average Landscape-Aware Regret (LAR) (or normalized objective) under predefined iteration budgets ( T = 30 , 50 for Branin, Ackley, Rosenbrock, and T = 50 , 100 for HDBO); each entry in Table 1 and Table 2 includes both the mean outcome and its standard deviation. Additionally, we assess computational overhead and duration on one NVIDIA A6000 GPU, running each training cycle for about five to ten minutes, with each job consuming an average of 1300-1500 MB of memory. The Appendix C, D provides experimental information for supplemental experiments and parameter ranges, in addition to a brief summary of baselines.

5.2 Main Results

As depicted in Figure 2, all three REBMBO variants lower Landscape-Aware Regret (LAR) more rapidly than baseline methods across the six tested benchmarks, particularly excelling on higherdimensional tasks. Table 1 (synthetic) and Table 2 (real-world) further quantify these findings for iteration budgets T = 30 and T = 50 . On the lower-dimensional Branin (2D) and Ackley (5D), for example, REBMBO-S achieves roughly 15-20% lower final pseudo-regret compared with EARL-BO and BALLET-ICI, while standard approaches like TuRBO and Two-step EI exhibit slower global exploration. Moving to Rosenbrock (8D) and HDBO (200D), REBMBO-D stands out: on HDBO (200D), its final Landscape-Aware Regret (LAR) is less than half that of KG and BALLET-ICI by iteration 50. Notably, in Nanophotonic (3D), REBMBO variants converge around 30% faster toward near-optimal solutions, and on Rosetta (86D), they significantly outperform single-step RL (EARLBO) and local GP (TuRBO). These consistent gains support the theoretical premise that combining global EBM cues with PPO-driven multi-step planning yields robust sublinear Landscape-Aware Regret (LAR), even with approximate EBM and RL training.

5.3 Supplementary Experiment

In addition to our primary benchmarks, we conducted several supplementary experiments (see Appendix C) to further validate REBMBO's theoretical guarantees and empirical robustness under diverse conditions.

5.3.1 Design and Modeling Choices

An ablation study (Appendix Table 4 and Table 5) isolates the roles of EBM, multi-step PPO, and short-run MCMC by incrementally removing or modifying these components, and performance drops whenever a core element is omitted, which confirms that global energy-based exploration, local GP modeling, and reinforcement learning each contribute critically to REBMBO. We further test kernel choice in the GP surrogate and find that a learned RBF+Matérn mixture performs best on Branin 2D, Ackley 5D, and HDBO 200D (Appendix C.10, Table 7), which supports the sum RKHS view and tighter regret guarantees. A one-at-a-time hyperparameter study and a dedicated sweep for λ

identify a broad safe band λ ∈ [0 . 2 , 0 . 5] and show that the default configuration is near optimal (Appendix C.11, Tables 8 and 9), which supports the theory that a balanced reward f ( x ) -λE θ ( x ) keeps information gain controlled and simplifies tuning.

5.3.2 Robustness and Reliability

Weillustrate REBMBO's behavior on 1D/2D multi-modal functions (Appendix Figures 3-4), showing how EBM-UCB avoids local optima and uses broader structural information, and trajectory comparisons (Appendix Figures 5-6) show less unnecessary exploration than GP-UCB and GLASSES with more direct convergence to global maxima. Robustness tests cover EBM convergence and removal and also scale mismatch between f and E θ ; REBMBO-C degrades gracefully under failed EBM and remains competitive without it, and normalization plus adaptive λ recovers most losses under severe scale gaps while keeping PPO stable in most runs (Appendix C.12, Tables 10 and 11).

5.3.3 Practicality and Fair Evaluation

We quantify compute overhead and observe a small constant-factor increase relative to TuRBO that matches polynomial scaling and parallelizes well on GPU, which is negligible when function evaluations dominate time (Appendix C.13, Table 12). Finally, we report standard regret in addition to pseudo-regret and REBMBO-C achieves the best values on all three tasks, which shows that improvements are not tied to one metric and that dual reporting reflects both exploration quality and final solution quality (Appendix C.14, Table 13); taken together with a benefit and overhead analysis (Appendix Figure 7), detailed comparisons (Appendix Table 6), performance heatmaps (Appendix Figure 8), and statistical significance checks (Appendix Figures 9-10), these results corroborate the premise of robust sublinear Landscape-Aware Regret when global EBM signals and multi-step RL are integrated and they reinforce the practical value of REBMBO for challenging BBO tasks.

6 Conclusion

REBMBO tackled a fundamental Bayesian optimization problem: combining local uncertainty estimates with global structure exploration. Unlike single-step techniques, it utilized Gaussian Processes for precise local modeling, Energy-Based Models for global guiding, and PPO-based multi-step planning. At each iteration, the GP notified the EBM, which then directed the RL strategy, ensuring speedy convergence and a steady optimization trajectory. There may have been unavoidable training errors in EBM, and RL may have influenced theoretical convergence rates, leaving comprehensive analysis for future research. Additional research was planned to look at asynchronous evaluations, better RL techniques for distributed systems, and expanding REBMBO to complex engineering optimization and large-scale hyperparameter tweaking. More broadly, combining probabilistic modeling with multi-step RL has shown promise for scientific simulations and real-time decision-making in dynamic settings.

Acknowledgments

This research was partially supported by the following sources: Y. W. is partially supported by NSF DMS-2415226, DARPA W912CG25CA007, and research gift funds from Amazon and Qualcomm. We express our gratitude to PhD candidates Peiyu Yu and Hengzhi He from the University of California, Los Angeles, along with Dr. Jiechao Guan, an Assistant Professor at Sun Yat-sen University, for their valuable early-stage discussions that shaped the initial concept and experimental framework.

References

Stéphane Alarie, Charles Audet, Aïmen E Gheribi, Michael Kokkolaras, and Sébastien Le Digabel. Two decades of blackbox optimization applications. EURO Journal on Computational Optimization , 9:100011, 2021.

Xilu Wang, Yaochu Jin, Sebastian Schmitt, and Markus Olhofer. Recent advances in bayesian optimization. ACM Computing Surveys , 55(13s):1-36, 2023.

Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In NeurIPS 2020 Competition and Demonstration Track , pages 3-26. PMLR, 2021.

Z Wang, M Zoghi, F Hutter, D Matheson, and ND Freitas. Bayesian optimization in a billion dimensions via random em-beddings. In Proceedings of the Twenty-Third international joint conference on Artificial Intelligence. , 2013.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems , 32, 2019.

Fengxue Zhang, Jialin Song, James C Bowden, Alexander Ladd, Yisong Yue, Thomas Desautels, and Yuxin Chen. Learning regions of interest for bayesian optimization with adaptive level-set estimation. In International Conference on Machine Learning , pages 41579-41595. PMLR, 2023.

Eric Lee, David Eriksson, David Bindel, Bolong Cheng, and Mike Mccourt. Efficient rollout strategies for bayesian optimization. In Conference on Uncertainty in Artificial Intelligence , pages 260-269. PMLR, 2020.

Ilija Bogunovic, Jonathan Scarlett, Andreas Krause, and Volkan Cevher. Truncated variance reduction: A unified approach to bayesian optimization and level-set estimation. Advances in neural information processing systems , 29, 2016.

Mujin Cheon, Jay H Lee, Dong-Yeun Koh, and Calvin Tsay. EARL-BO: Reinforcement learning for multi-step lookahead, high-dimensional bayesian optimization. In Forty-second International Conference on Machine Learning , 2025.

Sebastian E Ament and Carla P Gomes. Scalable first-order bayesian optimization via structured automatic differentiation. In International Conference on Machine Learning , pages 500-516. PMLR, 2022.

Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. High dimensional bayesian optimisation and bandits via additive models. In International conference on machine learning , pages 295-304. PMLR, 2015.

Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science , 4(2):268-276, 2018.

Changyong Oh, J Tomczak, Efstratios Gavves, and Max Welling. Combo: Combinatorial bayesian optimization using graph representations. In ICML Workshop on Learning and Reasoning with Graph-Structured Data , 2019.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in neural information processing systems , 24, 2011.

Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5 , pages 507-523. Springer, 2011.

Zi Wang, Chengtao Li, Stefanie Jegelka, and Pushmeet Kohli. Batched high-dimensional bayesian optimization via structural kernel learning. In International conference on machine learning , pages 3656-3664. PMLR, 2017.

Jian Wu and Peter Frazier. The parallel knowledge gradient method for batch bayesian optimization. Advances in neural information processing systems , 29, 2016.

Ilija Bogunovic, Jonathan Scarlett, Stefanie Jegelka, and Volkan Cevher. Adversarially robust optimization with gaussian processes. Advances in neural information processing systems , 31, 2018.

Robert B Gramacy. lagp: large-scale spatial modeling via local approximate gaussian processes in r. Journal of Statistical Software , 72:1-46, 2016.

Michael A Gelbart, Jasper Snoek, and Ryan P Adams. Bayesian optimization with unknown constraints. arXiv preprint arXiv:1403.5607 , 2014.

Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE transactions on evolutionary computation , 10(1):50-66, 2006.

Wolfgang Ponweiser, Tobias Wagner, Dirk Biermann, and Markus Vincze. Multiobjective optimization on a limited budget of evaluations using model-assisted-metric selection. In International conference on parallel problem solving from nature , pages 784-794. Springer, 2008.

Daniel Hernández-Lobato, Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective bayesian optimization. In International conference on machine learning , pages 1492-1501. PMLR, 2016.

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Practical automated machine learning for the automl challenge 2018. In International workshop on automatic machine learning at ICML , volume 12, 2018.

Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Hyperparameter search space pruning-a new component for sequential model-based hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part II 15 , pages 104-119. Springer, 2015.

Marc C Kennedy and Anthony O'Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika , 87(1):1-13, 2000.

Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. Multi-fidelity bayesian optimisation with continuous approximations. In International conference on machine learning , pages 1799-1808. PMLR, 2017.

David Ginsbourger, Janis Janusevskis, and Rodolphe Le Riche. Dealing with asynchronicity in parallel Gaussian process based global optimization . PhD thesis, Mines Saint-Etienne, 2011.

Thomas Desautels, Andreas Krause, and Joel W Burdick. Parallelizing exploration-exploitation tradeoffs in gaussian process bandit optimization. J. Mach. Learn. Res. , 15(1):3873-3923, 2014.

Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabás Póczos. Parallelised bayesian optimisation via thompson sampling. In International conference on artificial intelligence and statistics , pages 133-142. PMLR, 2018.

Shangda Yang, Vitaly Zankin, Maximilian Balandat, Stefan Scherer, Kevin Carlberg, Neil Walton, and Kody JH Law. Accelerating look-ahead in bayesian optimization: Multilevel monte carlo is all you need. arXiv preprint arXiv:2402.02111 , 2024.

Jose Pablo Folch, Calvin Tsay, Robert Lee, Behrang Shafei, Weronika Ormaniec, Andreas Krause, Mark van der Wilk, Ruth Misener, and Mojmír Mutny. Transition constrained bayesian optimization via markov decision processes. Advances in Neural Information Processing Systems , 37:88194-88235, 2024.

Javier González, Michael Osborne, and Neil Lawrence. Glasses: Relieving the myopia of bayesian optimisation. In Artificial Intelligence and Statistics , pages 790-799. PMLR, 2016.

Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

David JC MacKay et al. Introduction to gaussian processes. NATO ASI series F computer and systems sciences , 168:133-166, 1998.

Tetiana Parshakova, Jean-Marc Andreoli, and Marc Dymetman. Distributional reinforcement learning for energy-based sequential models. arXiv preprint arXiv:1912.08517 , 2019.

Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 2021.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Miguel González-Duque, Richard Michael, Simon Bartels, Yevgen Zainchkovskyy, Søren Hauberg, and Wouter Boomsma. A survey and benchmark of high-dimensional bayesian optimization of discrete sequences. arXiv preprint arXiv:2406.04739 , 2024.

Mitchell McIntire, Daniel Ratner, and Stefano Ermon. Sparse gaussian processes for bayesian optimization. In UAI , volume 3, page 4, 2016.

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics , pages 370-378. PMLR, 2016.

Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5272-5280, 2020.

Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems , 33:2199422008, 2020.

H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics , 1951.

Jialin Song, Yury S Tokpanov, Yuxin Chen, Dagny Fleischman, Kate T Fountaine, Harry A Atwater, and Yisong Yue. Optimizing photonic nanostructures via multi-fidelity gaussian processes. arXiv preprint arXiv:1811.07707 , 2018.

Anqing Jiang, Osamu Yoshie, and LiangYao Chen. A new multilayer optical film optimal method based on deep q-learning. arXiv preprint arXiv:1812.02873 , 2018.

Tian Hua Wang, Hai Ping Shao, Bing Qiang Zhao, and Hong Lin Zhai. Molecular insights into the variability in infection and immune evasion capabilities of sars-cov-2 variants: A sequence and structural investigation of the rbd domain. Journal of Chemical Information and Modeling , 64(8):3503-3523, 2024.

Thomas Desautels, Adam Zemla, Edmond Lau, Magdalena Franco, and Daniel Faissol. Rapid in silico design of antibodies targeting sars-cov-2 using machine learning and supercomputing. BioRxiv , pages 2020-04, 2020.

Rhiju Das and David Baker. Macromolecular modeling with rosetta. Annu. Rev. Biochem. , 77(1):363-382, 2008.

Kyle A Barlow, Shane O Conchuir, Samuel Thompson, Pooja Suresh, James E Lucas, Markus Heinonen, and Tanja Kortemme. Flex ddg: Rosetta ensemble-based estimation of changes in protein-protein binding affinity upon mutation. The Journal of Physical Chemistry B , 122(21):5389-5399, 2018.

Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence , 44(7):3634-3646, 2021.

A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning , pages 561-591. PMLR, 2018.

Jian Cao, Myeongjong Kang, Felix Jimenez, Huiyan Sang, Florian Tobias Schaefer, and Matthias Katzfuss. Variational sparse inverse cholesky approximation for latent gaussian processes via double kullback-leibler minimization. In International Conference on Machine Learning , pages 3559-3576. PMLR, 2023.

Shion Takeno, Masahiro Nomura, and Masayuki Karasuyama. Towards practical preferential bayesian optimization with skew gaussian processes. In International Conference on Machine Learning , pages 33516-33533. PMLR, 2023.

Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning, ser. adaptive computation and machine learning. Cambridge, MA, UsA: MIT Press , 38:715-719, 2006.

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning , pages 3732-3747. PMLR, 2020.

Masaaki Nagata, Katsuki Chousa, and Masaaki Nishino. A supervised word alignment method based on cross-language span prediction using multilingual bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 555-565, 2020.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

Yakoub Bazi, Mohamad M Al Rahhal, Haikel Alhichri, and Naif Alajlan. Simple yet effective fine-tuning of deep cnns using an auxiliary classification loss for remote sensing scene classification. Remote Sensing , 11(24):2908, 2019.

Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning , pages 1064-1071, 2008.

Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book , 2018.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Wenjie Xu, Wenbin Wang, Yuning Jiang, Bratislav Svetozarevic, and Colin N Jones. Principled preferential bayesian optimization. arXiv preprint arXiv:2402.05367 , 2024.

Nathaël Da Costa, Marvin Pförtner, Lancelot Da Costa, and Philipp Hennig. Sample path regularity of gaussian processes from the covariance kernel. arXiv preprint arXiv:2312.14886 , 2023.

Mónika Farsang and Luca Szegletes. Decaying clipping range in proximal policy optimization. In 2021 IEEE 15th International Symposium on Applied Computational Intelligence and Informatics (SACI) , pages 000521-000526. IEEE, 2021.

A Benchmark Details

A.1 Branin Toy Dataset (2D)

We employ the standard Branin objective function for a two-dimensional input vector x = ( x 1 , x 2 ) . The function is given by:

In our experiments, each x is drawn from a bounded domain (e.g., [ -5 , 10] × [0 , 15] ) and the output f ( x ) serves as a classical test for global optimization. Within the REBMBO framework, the input consists of the current guess x ∈ R 2 , and the output is the Branin objective value. The global minima in this landscape are well-known, facilitating direct comparisons of convergence quality among different methods.

A.2 Ackley Function (5D)

Although the Ackley function is often defined in two dimensions, we employ a five-dimensional variant to increase the complexity of local minima. The Ackley function typically has the form:

where d = 5 in our case, and standard constants ( a = 20 , b = 0 . 2 , c = 2 π ) are chosen. The domain can be set to [ -32 . 768 , 32 . 768] 5 . Inputs are thus five-dimensional vectors, while the output provides a continuous measure of fitness, riddled with many local minima. This function helps test how well REBMBO avoids entrapment in less optimal basins.

A.3 Rosenbrock Function (8D)

The Rosenbrock function, often referred to as the Banana function, appears here in an eightdimensional variant:

Although it is uni-modal, the valley leading to the global minimum is curved and narrow, making convergence notoriously difficult for local methods. We set an input domain such as [ -2 , 2] 8 , and let each x ∈ R 8 map to a scalar output f ( x ) . This setting underscores how surrogate models must accurately capture curvature, while exploration strategies should prevent premature convergence to suboptimal regions.

A.4 Nanophotonic Structure Design (3D)

In this dataset, each input x ∈ R 3 represents physical parameters (for instance, thickness or refractive indices) that define a layered optical filter. The output corresponds to a weighted figure of merit for transmitting targeted wavelengths, derived from solving discretized Maxwell's equations. Given the computational intensity of this solver, strategies such as deep Q-learning can be employed to optimize the design efficiency of multi-layer films [45]. This approach converges to the global optimum of the optical thin film structure, addressing the limitations of traditional numerical algorithms that often converge to local optima [46]. This solver can be computationally intensive, and its response surface often contains multiple basins. Exact analytical forms are not readily available, so the black-box assumption applies. In REBMBO, the agent queries the simulator with a proposed x , and obtains the numeric figure of merit indicating how well the design meets hyperspectral criteria.

A.5 HDBO-200D

To investigate high-dimensional performance, we construct a synthetic function in 200 dimensions. Each component x i is initially drawn from a standard normal distribution, and the objective is:

Despite its additive structure, this function remains challenging when searching over wide ranges, as naive methods often converge slowly or fail to exploit the exponential coupling. We select a suitable domain constraint (e.g., [ -5 , 5] 200 ) to test how well each method scales in dimension and maintains exploration. Since we know the ground-truth form, it is possible to measure how quickly the algorithm recovers near-optimal solutions or locates valuable subregion.

A.6 Rosetta Protein Design (86D)

This data set represents a realistic antibody engineering task, where each input x ∈ R 86 encodes structural modifications relative to a reference antibody that binds to the SARS-CoV-2 spike protein. The implications of minimizing changes in binding free energy ( ∆∆ G ) are significant for the efficacy and safety of antibodies engineered to target the SARS-CoV-2 spike protein. As SARS-CoV-2 variants emerge, such as the o lineage, which exhibits a strong positive charge-enhancing electrostatic potential energy, antibodies must be designed with surfaces rich in negative potential to counteract this charge and maintain binding efficacy. Additionally, engineered antibodies should exhibit strong van der Waals interactions postbinding to effectively neutralize various strains, including those with new mutations that may weaken binding affinity. [47, 48] The simulator uses Rosetta Flex [49, 50] to estimate changes in binding free energy ( ∆∆ G ), requiring extensive CPU time per query. The objective is to minimize ∆∆ G (or equivalently maximize -∆∆ G ), though no simple functional form exists. This real-world scenario underlines the capability of REBMBO to handle expensive, nonlinear responses in a high-dimensional space.

A.7 Additional Benchmark: NATS-Bench (20D)

This benchmark is drawn from a unified framework for neural architecture search (NAS) designed to evaluate both architecture topology and size [51]. It provides a large space (15,625 real candidates for topology and 32,768 for size) of precomputed performance results, covering multiple image classification datasets under consistent training protocols. In our setup, each configuration x ∈ R 20 (or a discrete embedding) represents a distinct candidate architecture in the NATS-Bench search space, where the goal is to maximize validation or test accuracy of the trained model. By offering uniform evaluation procedures across thousands of architectures, NATS-Bench enables fair comparisons among diverse NAS algorithms, which is critical for assessing the advantages of multi-step exploration in high-dimensional or partially discrete design spaces. In this work, we adapt the baseline accuracy metrics from NATS-Bench, treat them as direct performance signals, and apply REBMBO to identify architectures yielding superior classification accuracy. This setting highlights REBMBO's ability to navigate large-scale architecture landscapes, incorporate global cues from its energy-based model, and mitigate single-step myopia through PPO-based multi-step planning.

A.8 Additional Benchmark: Robot Trajectory (40D)

This benchmark is drawn from real-world robotic tasks introduced by Mahmood et al. [52], featuring multiple commercially available robots with varying degrees of difficulty and repeatability. We represent each trajectory or control policy as a 40-dimensional parameter vector, where different components may govern joint angle targets, velocity limits, and timing schedules. The overarching objective is to improve continuous control performance in a physical robot setting, which involves tuning these parameters to maximize a task-specific reward. Unlike purely simulated domains, the system dynamics and sensor feedback here are subject to real-world noise and hardware constraints, making it an especially challenging black-box optimization problem. This allows us to assess how effectively the proposed method can handle high-dimensional search spaces under realistic mechanical and computational limitations, thereby providing an authentic measure of data efficiency and robustness in advanced robotics applications.

B Baseline Characteristics and Ranking

Table 3 provides a side-by-side comparison of the baseline algorithms and our proposed REBMBO variants, listing each method's modeling strategy (local, single-step RL, short-horizon lookahead, or multi-step RL), the underlying surrogate (e.g. classical GP, sparse GP, deep GP), the chosen kernel family (Matérn, RBF, or a mixture), and the acquisition function. The final column shows

the average ranking across both synthetic and real-world benchmarks, with lower values indicating superior performance. Specifically, TuRBO focuses on local trust regions with a Matérn-5/2 kernel, while BALLET-ICI alternates between global and local GPs, employing an RBF kernel. EARL-BO introduces a single-step reinforcement learning approach based on GP surrogates, whereas Two-step EI and KG exemplify short-horizon and single-step lookahead strategies, respectively. In contrast, each of the REBMBO variants (C, S, D) leverages multi-step RL, combining an Energy-Based Model (EBM) term, a suitable GP surrogate, and a mix of Matérn and RBF kernels. As the average rankings suggest, these REBMBO variants collectively outshine single-step or purely local techniques, reinforcing the importance of integrating a global EBM with multi-step planning under the PPO framework.

C Supplementary Experiment

C.1 Abalation study

Numerical Analysis. Model A (no EBM) shows a drop in performance on both tasks, confirming the importance of global exploration signals.

Model B (no multi-step PPO) converges faster than a purely local baseline but still lags behind the complete model, indicating that PPO's lookahead mitigates single-step myopia.

Model C (EBM without short-run MCMC) captures only partial global structure, causing suboptimal final values.

Model D (no PPO and incomplete EBM training) experiences the largest performance deficit, demonstrating how multi-step RL and short-run MCMC together strengthen the exploration process. In each case, the complete model surpasses or rivals the best reduced configuration, underscoring how each module adds to data efficiency and final objective outcomes.

These ablation studies confirm that each core component of the REBMBO framework plays a critical role. Short-run MCMC training expands the EBM's capacity to capture complex global

basins, while PPO's multi-step horizon counters local or single-step limitations, and the GP surrogate ensures accurate local predictions. Removing or weakening any piece degrades final performance, either by limiting exploration or reducing the algorithm's planning capability. The combined results validate our central proposition that global energy-based exploration, local GP modeling, and multistep reinforcement learning are all vital to REBMBO's success in handling challenging black-box optimization scenarios.

C.2 1D Multi-modal Function Optimization

This experiment demonstrates how REBMBO leverages its Energy-Based Model (EBM) to navigate multiple local maxima in a one-dimensional domain, with a small set of initial observations (red points) spanning -1 . 0 ≤ x ≤ 1 . 0 . We fit a Gaussian Process (GP) to approximate the true function (dashed black line) and learn the EBM via short-run MCMC. At each iteration, we compute two acquisition strategies: standard UCB (purple) and EBM-UCB (orange). In Figure 3, the top panel contrasts the GP mean (solid blue) and uncertainty (shaded area) with the true function, highlighting how EBM-UCB (orange triangles) focuses exploration around the global optimum (vertical green dashed line at x ≈ 0 . 25 ) more effectively than standard UCB (purple triangles). The middle panel plots -E θ ( x ) , revealing where the EBM predicts promising global regions, and the bottom panel compares acquisition values from both strategies. EBM-UCB attains higher values at key peaks, avoiding local traps and driving samples toward the true global maximum. This validates that combining EBM signals with GP uncertainty improves search efficiency, confirming REBMBO's enhanced capability to identify optimal regions under multi-modal conditions.

1D Multi-modal Function Optimization

True Function

2

GP Mean

GP Uncertainty

Observations

UCB Selection

EBM-UCBSelection

0

-2

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

X

Negative Energy (-E(x)

Negative Energy

01

-1

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

X

Acquisition Value

4

5

UCB

EBM-UCB

3

-1

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

X

C.3 2D Multi-modal Function Component Analysis

This experiment decomposes REBMBO in a two-dimensional multi-modal setting to highlight how its GP surrogate, Energy-Based Model (EBM), and acquisition mechanism jointly guide sampling. In Figure 4, the top-left panel shows the true function, where two distinct basins of high value

emerge, along with initial observations (various markers). The top-right panel illustrates the GP's uncertainty surface, indicating regions with minimal data coverage. The bottom-left panel plots the learned negative energy -E θ ( x ) , revealing how EBM emphasizes broader structural cues rather than solely relying on local uncertainty. Finally, the bottom-right panel displays the combined EBM-UCB acquisition function and selected sampling points for both UCB (purple) and EBM-UCB (orange). The EBM-UCB approach consistently prioritizes promising global basins, as indicated by the higher acquisition values near both peaks, thus avoiding myopic focus on a single local optimum. By comparing these panels, we see that the global signal from -E θ ( x ) helps REBMBO balance local exploration (informed by GP uncertainty) with global navigation (guided by the EBM), validating its capacity to locate and refine multiple maxima in complex search spaces.

1.00

1.00

GP Uncertainty (a(x))

1.125

0.96

0.75

1.000

0.75

0.84

0.50

0.875

0.50

0.72

0.25

0.750

0.25

0.60

x0.00

0.625

0.00-

0.48

0.500

0.25

0.25

0.36

0.375

0.50 -

0.50 -

0.24

0.250

0.75 -

0.125

0.75

0.12

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0.000

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0.00

X1

X1

EBM Negative Energy (-E(x))

EBM-UCB Acquisition Function with Selected Points

1.00

0.00

1.00

2.00

0.75

0.12

0.75

1.75

0.50

0.24

0.50

1.50

0.25

0.36

0.25

1.25

0.00

0.48

x0.00-

1.00

0.25

0.60

0.25

0.75

0.72

0.50 -

0.50

0.50

0.84

0.75

0.75 -

0.25

UCB selection

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0.96

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

EBM-UCB selection

0.75

1.00

0.00

X1

X1

C.4 2D Multi-modal Optimization Trajectory Comparison

To illustrate REBMBO's sampling efficiency relative to GP-UCB and GLASSES, we optimize a two-peak function in the 2D plane and plot each method's trajectory over ten iterations. In Figure 5, the top-left panel overlays all three trajectories on the true function contours, while the other panels present each method individually with iteration labels (e.g., 0 to 10). Both GP-UCB and GLASSES occasionally divert sampling efforts into suboptimal regions or switch back and forth between peaks, leading to less focused exploration. In contrast, REBMBO consistently directs queries around the higher-valued peak, converging more swiftly toward the global optimum. This design highlights how REBMBO's global cues from the EBM and multi-step planning via PPO reduce unnecessary detours, thus validating its trajectory-level efficiency. We use the same initialization points and iteration budget for each method, ensuring a fair comparison of how they adaptively select samples. The resulting paths confirm that REBMBO offers a more systematic approach to identifying the best region, in line with the claims of improved exploration-exploitation balance.

1.00

True Function with All Trajectories

1.00

GP-UCB Trajectory

1.125

0.75

0.75

1.000

0.50

0.875

0.50

0.25

0.750

0.25

000

0.625

0.00-

0.500

0.25

0.25

0.375

0S'0-

0.50 -

0.250

0.75

GLASSES

GP-UCB

0.125

0.75

REBMBO

1.00

1.00

0.75

0.50

0.25

000

0.25

0.50

0.75

1.00

0.000

1.00

1.00

0.75

0.50

0.25

000

0.25

0.50

0.75

1.00

X1

X1

1.00

GLASSES Trajectory

1.00

REBMBOTrajectory

1.125

0.75

0.75 -

1.000

0S'0

0.50 -

0.875

0.25

0.25

0.750

0.625

00°0

0.00-

0.500

0.25

0.25

0.375

0.50 -

0.50 -

0.250

0.75

0.75

0.125

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

-1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0.000

X1

X1

C.5 Optimization Trajectory Visualization

In Figure 6, we compare the sampling paths of GP-UCB, GLASSES, and REBMBO on a challenging 2D objective with multiple local basins. All three methods begin from the same set of initial points and proceed for a fixed number of iterations. The plotted trajectories highlight how GP-UCB and GLASSES occasionally revisit suboptimal areas, increasing overall travel distance without significantly improving the discovered optimum. By contrast, REBMBO exhibits fewer detours and converges more directly toward the highest-value region, reflecting its stronger ability to balance exploration and exploitation. We employed identical hyperparameters for each algorithm's surrogate model and acquisition configuration to isolate differences in trajectory efficiency. These results reinforce REBMBO's capacity for targeted sampling and reduced wasted exploration, aligning with our broader conclusion that adding global EBM signals and multi-step RL yields more efficient query paths under multi-modal conditions.

C.6 REBMBO Benefit vs. Computational Overhead

In Figure 7, we illustrate how REBMBO's performance gains (blue and orange bars, indicating regret improvement and convergence speedup) compare against its additional computational cost (green bars) under varying problem complexities. Each bar above zero signifies a positive contribution (e.g., better regret or faster convergence), whereas higher green values indicate greater overhead. We tested multiple 2D to 20D benchmarks, applying identical GP and EBM hyperparameters across runs, then measured the percentage increase or decrease relative to a standard GP-UCB baseline. Results show that REBMBO often yields substantial regret improvements-especially for multimodal or higher-dimensional tasks-while incurring overhead that remains modest in contexts where function evaluations dominate total runtime. Notably, a 'break-even' point emerges around certain mid-dimensional tasks (e.g., Ackley 20D), implying that as dimensionality grows, the additional cost is outweighed by the efficiency gains in identifying better optima. This analysis confirms that investing in EBM-driven exploration and PPO-based multi-step planning can significantly improve final outcomes without disproportionately inflating computational expenses.

REBMBO-CvsGP-UCB:Performance Metrics

Sphere(2D)-

-8.5

0.9

3.2

40

Branin(2D) -

6.5

1.1

3.1

-30

Function

Hartmann(6D) -

22.4

1.5

2.8

20

Test

Six-hump Camel(2D)-

16.7

1.2

3.2

-10

Rosenbrock(10D)-

41.3

1.9

2.5

Ackley(20D) -

46.7

2.2

2.2

Regret Improv. (%)

ConvergenceSpeedup

ComputationOverhead

REBMBOBenefit vs.Overhead byProblem Complexity

Regret Improvement(%)

ComputationOverhead

40

30

Value

20

10

Break-even

-10

Sphere

Six-hump Camel

Hartmann

Rosenbrock

Ackley

(2D)

(2D)

(2D)

(6D)

(10D)

(20D)

Test Function (Dimension)

C.7 Detailed Comparison of REBMBO vs. GP-UCB

In Table 6, we provide a head-to-head comparison of REBMBO-C versus GP-UCB on multiple benchmarks, listing each algorithm's final regret, the percentage improvement

, convergence speedup (the ratio of required evaluations or wall-clock time to reach a near-optimal solution), computation overhead (the extra training cost from short-run MCMC and PPO updates, normalized by GP-UCB's overhead), and an overall 'Efficiency Score' reflecting the net trade-off between performance gains and overhead. Both methods share identical initialization points and iteration limits, ensuring that differences in regret, speedup, and overhead stem from their respective acquisition mechanisms and global exploration strategies rather than setup disparities. Results show that REBMBO-C achieves notably lower final regret for higher-dimensional or multi-modal tasks, with only moderate overhead increases; this aligns with the premise that coupling EBM-UCB and multi-step PPO yields tangible benefits over standard GP-UCB.

C.8 Performance Metrics Heatmap

In Figure 8, we present a color-coded matrix depicting how REBMBO's regret improvement, convergence speedup, and computational overhead vary across multiple benchmark functions. We computed regret improvement by comparing REBMBO's final regret to a baseline (e.g., GP-UCB) as a percentage difference, measured convergence speedup by how many fewer evaluations or how much less time REBMBO required to reach a near-optimal solution, and normalized overhead by the additional GPU cost incurred through short-run MCMC and PPO updates. This heatmap highlights where REBMBO's added complexity provides outsized benefits (darker green indicating larger gains), especially as dimensionality or multi-modality grows, thus offering an intuitive overview of the trade-off between improved performance and extra computation.

Final Regret (lower is better)

Sphere(2D)-

0.047

0.042

0.053

0.051

0.055

0.057

- 0.5

Branin(2D) -

0.062

0.057

0.061

0.058

0.054

0.051

0.4

Function

Hartmann(6D) -

0.183

0.175

0.162

0.142

0.138

0.133

0.3

Six-hump Camel(2D)-

0.078

0.072

0.069

0.065

0.063

0.060

0.2

Rosenbrock(10D)-

0.421

0.408

0.362

0.247

0.232

0.215

0.1

Ackley(20D)-

0.537

0.522

0.418

0.286

0.263

0.241

GP-UCB

GLASSES

REBMBO-C

REBMBO-S

REBMBO-D

Function Evaluations to 10% Regret (lower is better)

Sphere(2D)-

15

14

16

17

18

19

80

Branin(2D) -

22

19

21

20

19

18

70

Function

Hartmann(6D) -

47

43

38

31

29

27

60

50

Test

Six-hump Camel(2D)-

25

22

21

21

20

19

40

Rosenbrock(10D)-

73

68

60

36

30

20

Ackley(20D) -

89

8

72

41

38

35

80n-d9

目

GLASSES

REBMBO-C

REBMBO-S

REBMBO-D

Method

C.9 Statistical Significance Analysis of REBMBO vs. Baselines

To further validate REBMBO's reliability, we performed a detailed statistical significance study on every algorithm's performance across multiple benchmark tasks. In total, each method was run five times independently. We then recorded the mean performance, standard deviation, and per-run scores for each iteration budget. Figure 9 shows how raw performance data are collected and verified, ensuring each method truly has five distinct runs; the middle panels illustrate mean and standard deviation checks against target reference values, while the bottom panel provides an example of how per-run results are nested and processed. This verification step ensures that our reported aggregated

metrics (including those shown in Table 1 and Table 2) accurately reflect the inherent variability among runs, which is vital for reliable statistical testing.

Next, as depicted in Figure 10, we visualize the overall distribution of performance scores via boxplots and bar charts, then examine pairwise significance using Welch's t-tests. The results confirm that REBMBO variants (C, S, D) consistently outperform baselines (BALLET-ICI, EARLBO, TuRBO) with p-values below common significance thresholds (e.g., p < 0 . 01 in many cases). Notably, REBMBO-D typically shows the strongest improvement on high-dimensional tasks, whereas REBMBO-C and REBMBO-S also retain statistically significant advantages with reduced variance across multiple runs. Together, these findings validate REBMBO's stable performance gains and reinforce the main experimental outcomes reported in Section 5.

talysisResults rows

8

2

1

3

4

5

6

7

Branin2D

Benchmark

2D

2D

2D

2D

2D

Iteration

33

30

30

REBMBO-S

REBMBO-S

REBMBO-S

REBMBO-D

REBMBO-D

REBMBO-D

REBMBO-D

REBMBO-S

REBMBO-S

Method

Run

123

34

5

2

4

Performance

91.946555

89.348711

88.981679

91.476739

87.204878

92.388742

95.795938

87.437993

6

30

REBMBO-D

5

93.848118

RunCountVerification

Run count checkfor each configuration:

Benchmark

Iteration

Method

Ackley5D

30

５５５５５

EARL-BO

REBMBO-C

REBMBO-D

REBMBO-S

:55555

Rosetta 86D100

EARL-B0

REBMBO-C

REBMBO-D

REBMBO-S

TuRBO

Length: 72,dtype: int64

MeanValueVerification

Mean check:

2

3

Benchmark

Ackley

Ackley

Ackley

Ackley

Ackley

5D

5D

5D

5D

5D

Iteration

BALLET-ICI

REBMBO-D

REBMBO-C

REBMBO-S

EARL-BO

Method

Performance

82.84

Target _Mean

89.93

80.46

85.25

86.85

82.84

-1.421085e-14

0.000000e+00

1.421085e-14

0.000000e+00

Difference

5

6

7

8

9

Ackley

Ackley

Ackley

5D

5D

5D

5D

5D

BALLET-ICI

REBMBO-C

REBMBO-D

EARL-BO

TuRBO

87.22

94.46

91.53

87.78

87.22

94.46

91.53

90.84

87.78

-1.421085e-14

-2.842171e-14

0.000000e+00

00+00000

0.000000e+00

StandardDeviationVerification

Standard deviation check:

1

2

3

4

Ackley

Ackley

Benchmark

Ackley

Ackley

5D

5D

5D

5D

5D

Iteration

BALLET-ICI

REBMBO-C

REBMBO-D

REBMBO-S

EARL-BO

Method

Performance

1.039772

1.375182

1.397542

1.654690

1.118034

Target_Std

0.93

1.48

1.00

Difference

0.109772

0.147542

0.145182

0.174690

0.118034

5

6

7

8

9

Ackley

Ackley

Ackley

5D

5D

5D

5D

5D

BALLET-ICI

REBMBO-C

REBMBO-D

EARL-BO

TuRBO

2.392593

2.034822

1.173936

2.303150

1.732953

1.82

1.05

2.06

2.14

1.55

0.182953

0.252593

0.214822

0.123936

0.243150

IndividualRunResultsExample

Statistical significance levels:

Nested dictionary format example:

*p<0.05:Significantdifference

Branin 2D, T=30,REBMB0-S,5 run results:{0:88.98167906054202,1: 87.43799280696837,2:89*p<0.01:Highly significantdifference461, 4:87.20487837019436}

***p<0.001:Extremely significantdifference

T-tests between REBMBO variants and baselines confirm

the statistical significance of performance improvements.

Thisanalysisconfirmsthateachalgorithmwasrun5timesindependently,withmeansmatchingexactly

thereportedvaluesinTable2,andstandarddeviationsshowingappropriatevariability.

C.10 Kernel Choice: RBF+Matérn vs. Single Kernels

We ablate the GP surrogate kernel in REBMBO across four choices: RBF, Matérn-5/2, Rational Quadratic, and a learned RBF+Matérn mixture k ( x, x ′ ) = w RBF k RBF ( x, x ′ ) + w Mat´ ern k Mat´ ern -5 / 2 ( x, x ′ ) with nonnegative weights fit by marginal likelihood at each BO update. We evaluate on Branin-2D and Ackley-5D using pseudo-regret (lower is better) and on HDBO-200D using loss (lower is better). Each cell in Table 7 reports mean ± std over five independent runs under the same evaluation budget and seeds.

StatisticalSignificanceAnalysisofREBMBOVariantsvs.Baselines

Based on 5 independent runs for each algorithm on each benchmark

DistributionofPerformanceScoresacrossMultipleRuns

MeanPerformancewithStandardDeviation

95.0

100

92.5

08

Mean Performance

90.0

Performance

60

87.5

85.0

82.5

20

80.0

REBMBO-S

REBMBO-D

REBMBO-C

BALLET-ICI

EARL-BO

TuRBO

Method

StatisticalSignificance(p-values)betweenMethods

Method

REBMBO-S

0.175

Statistical Analysis Methodology:

0.1933

0.0528

0.0091

• Each algorithm was executed 5 times independently

0.150

• We performed Welch's t-test between REBMBO variants

and baseline methods

REBMBOVariants

• Significance levels: * p<0.05, ** p<0.01, *** p<0.001

0.125

REBMBO-D

Key Findings:

0.0021

0.0061

0.0100

0.100

• REBMBO variants consistently outperform baseline

methods with statistical significance (p<0.05)

0.075

• REBMBO-D shows the most significant advantage in

high-dimensional tasks (p<0.01)

REBMBO-C

0.050

• REBMBO variants demonstrate lower variance across

multiple runs, indicating greater stability

0.0270

0.0494

0.0026

0.025

• The performance gains are most pronounced in complex

tasks such as HDBO 118D and Rosetta 86D

BALLET-ICI

EARL-BO

TuRBO

The RBF+Matérn mixture attains the best score on all three benchmarks (Table 7). Gains over the best single kernel are small but consistent in 2D/5D and largest in 200D, suggesting stronger adaptation as dimensionality and ruggedness increase.

The mixture induces a sum RKHS, H mix = H RBF ⊕H Mat´ ern , so the surrogate can capture smooth global trends (RBF) and rough local variations (Matérn-5/2) at the same time, reducing kernel misspecification. Its information gain satisfies γ mix ( T ) ≤ γ RBF ( T ) + γ Mat´ ern ( T ) , which tightens regret bounds relative to a single kernel. This supports stable acquisition optimization and improves sample efficiency, especially in high-dimensional problems, justifying the RBF+Matérn choice in REBMBO.

C.11 Hyperparameter Sensitivity and Simple Tuning

We perform a one-at-a-time hyperparameter study for α, β, γ, λ on three benchmarks: Branin2D (smooth), Nanophotonic-3D (moderately rugged), and HDBO-200D (highly multi-modal, high-dimensional). For each parameter, we vary its value while fixing the others to α =0 . 30 , β =2 . 00 , γ =0 . 10 , λ =0 . 35 , using identical evaluation budgets and ten independent runs per setting; results are summarized in Table 8. In addition, we conduct a dedicated sweep of λ ∈ { 0 . 05 , 0 . 10 , 0 . 20 , 0 . 50 , 1 . 00 , 2 . 00 } with five runs per value on the same three tasks (Table 9). Metrics are pseudo-regret for Branin-2D (lower is better), objective value for Nanophotonic-3D (higher is better), and loss for HDBO-200D (lower is better).

Across tasks, the default configuration is already close to optimal: the largest gap from the best per-parameter choice is ≤ 9 . 1% (Table 8). The λ sweep shows a broad, flat optimum on [0 . 2 , 0 . 5] and clear degradation outside this range (Table 9). Sensitivity ranks as λ ≈ β > γ ≥ α , indicating that performance is most affected by the exploration/energy balance ( λ ) and uncertainty calibration ( β ), while the remaining weights have smaller effects.

These observations support the theoretical picture that (i) moderate λ yields a balanced reward r t = f ( x t ) -λE θ ( x t ) , keeping the contribution from the energy term within the bounded-informationgain regime assumed by our regret analysis, and (ii) tuning β adjusts the effective confidence in GP uncertainty, matching noise levels and stabilizing acquisition decisions. The flat optimum around λ ∈ [0 . 2 , 0 . 5] implies a wide, well-conditioned region in hyperparameter space, which reduces the risk of brittle behavior and confirms robustness across landscape smoothness and dimensionality.

Practically, these results justify a simple tuning rule: keep α =0 . 30 and γ =0 . 10 fixed; tune only λ and β . Use λ ≈ 0 . 20 for smoother objectives and λ ≈ 0 . 50 for highly multi-modal ones; set β ∈ [1 . 5 , 2 . 5] to match noise. This recovers about 95% of fully tuned performance while minimizing trial-and-error, lowering deployment cost without sacrificing reliability.

C.12 Robustness: EBM Convergence and Scale Mismatch

We study two failure modes. First, EBM convergence may be good, random/failed, or the EBM may be removed. Second, f ( x ) and E θ ( x ) may live on different numeric scales; we stress this by multiplying E θ while keeping f fixed and by enabling adaptive λ .

When EBM functions normally, REBMBO-C is best. With random/failed EBM, performance drops by ∼ 7% yet still exceeds GP-UCB and Random Search. With EBM removed, GP+PPO remains competitive (Table 10). Under severe scale gaps ( × 100 ), performance degrades; normalization plus adaptive λ recovers most of the loss and maintains high PPO stability (Table 11).

These results show graceful degradation. Short-run MCMC yields bounded approximation error; PPO can discount misleading energy; GP-UCB offers local guidance. Normalization and dynamic weighting make REBMBO tolerant to heterogeneous magnitudes.

C.13 Compute Overhead and Complexity

We measure per-iteration wall-clock time on an NVIDIA A6000 GPU over five BO benchmarks, excluding objective evaluations to isolate algorithmic overhead. Methods: TuRBO, BALLET-ICI, EARL-BO, and REBMBO-C. Timing covers all internal steps; for REBMBO-C this includes GP

updates, EBM training, and a small PPO update. Each entry in Table 12 is mean ± sd over five runs with identical budgets.

REBMBO-C incurs a 2 . 1 -2 . 5 × overhead vs. TuRBO across tasks; on HDBO-200D it is 28 . 3 ± 1 . 48 s vs. 12 . 8 ± 0 . 72 s (Table 12). The same constant-factor gap holds in lower dimensions, i.e., slower than GP-only pipelines but within a modest multiple.

Observed scaling matches expected polynomial costs: GP O ( n 3 ) , EBM O ( KBLdh ) (MCMC steps K , batch B , depth L , width h , input dim d ), PPO O ( ML π h π ) (epochs M , policy size L π , h π ). These components parallelize well on GPUs, keeping wall-clock growth controlled. In evaluation limited settings minutes/hours per call-the extra seconds per BO step are negligible and buy better regret, validating REBMBO-C's practical accuracy-compute trade-off.

C.14 Fairness: Standard Regret vs. Pseudo-Regret

We evaluate standard regret to complement pseudo regret and ensure a fair comparison. Using the conventional definition R T = min t ≤ T ( f ( x ⋆ ) -f ( x t ) ) with lower being better, we measure Branin 2D, Ackley 5D, and HDBO 200D under identical evaluation budgets and identical seeds for all methods. The pseudo regret weight in REBMBO's reward is fixed to α =0 . 3 and is chosen once by a small grid search, then kept constant. Table 13 reports mean ± sd over repeated runs.

Across all three benchmarks, REBMBO C attains the lowest standard regret and the best overall mean in Table 13. The largest margin appears on HDBO 200D, which indicates that the benefit persists in high dimensions. TuRBO, BALLET ICI, and EARL BO trail by a consistent margin.

These results support two claims. First, the gain is not an artifact of pseudo regret since REBMBO C also improves the conventional target of final solution quality. Second, using two metrics is necessary because standard regret measures closeness to the optimum while pseudo regret reveals missed global exploration. Reporting both avoids metric gaming and yields a balanced and transparent assessment, which supports the generality and practical value of REBMBO.

D Parameter Explanation and Computational Complexity

This appendix clarifies the primary parameters used in our method and illustrates how the per-iteration computational cost of REBMBO compares to simpler single-step Bayesian optimization approaches.

D.1 Parameter Sets in the REBMBO Framework

The table below outlines the main symbols and their roles in the overall algorithm. Each parameter is linked to a particular module, allowing readers to track how surrogates, energy models, and multi-step policies interact.

D.2 Hyperparameters for the EBM

The next table details the architecture and training choices for our energy-based model, including network depth, optimizers, and MCMC sampling procedures.

D.3 Policy Network and PPO Settings

We adopt a multi-step strategy through PPO, and the following table lists key network dimensions, reward formulation, and update parameters.

D.4 General Complexity Comparison

To clarify per-iteration computational demands, we compare the asymptotic complexities for standard single-step GP-based methods and our REBMBO approach in Table 17. While both require O ( n 3 ) to update the GP, REBMBO additionally integrates EBM training and PPO updates, though these can be parallelized on modern hardware.

D.5 Concrete Example of Per-Iteration Operation Counts

Table 18 offers an approximate breakdown of operations in a small-scale setting, showing that the EBM step typically dominates extra costs but can be mitigated by parallel computation, especially where function evaluations are expensive.

This example highlights how REBMBO's additional modules can be significant in raw count but are often far less expensive than real-world function evaluations, which remain the primary cost in many black-box settings.

E Landscape-Aware Regret (LAR): Definition and Rationale

E.1 Background and Motivation

In conventional Bayesian optimization or black-box optimization settings, the most common definition of instantaneous regret is:

where x ∗ is a global optimizer ( arg max x ∈X f ( x ) ) and x t is the point selected in the t -th iteration. This quantity captures how far x t is from the global optimum in terms of objective function value. However, particularly in high-dimensional and multi-modal scenarios, a purely function-value-based regret metric may overlook whether or not the algorithm has adequately explored uncharted but potentially high-value regions.

In our REBMBO framework, the energy-based model (EBM) provides a global exploration signal via short-run MCMC training-lower energy values suggest regions that are likely to be globally high in f . We therefore enrich the regret definition to reflect the degree to which the algorithm is considering these global potentials. Specifically, we propose the following 'Landscape-Aware Regret (LAR),' denoted R LAR t :

where α > 0 is a hyperparameter, and E θ ( · ) denotes the EBM's learned energy function. If E θ ( x t ) is much higher than E θ ( x ∗ ) , the term α [ E θ ( x ∗ ) -E θ ( x t ) ] penalizes the algorithm for not taking advantage of globally promising regions flagged by the EBM. Thus, R LAR t incorporates both local function-value suboptimality and disregard for the EBM's global signal.

E.2 Mathematical Form and Key Properties

Let x ∗ = arg max x ∈X f ( x ) , and assume the EBM function E θ ( x ) is defined and differentiable over the domain X . We define:

Notes: n is the number of sampled data points (for GP training), d is input dimension, L, h are EBM network layers or hidden units, L π , h π are PPO policy network dimensions, K is MCMC steps, B is mini-batch size, and M is PPO epochs.

with α > 0 . If α = 0 , R LAR t reduces to the standard regret R t . If α > 0 and E θ aligns well with f (i.e., points of high function value have correspondingly low energy), R LAR t distinguishes whether x t is good in terms of both objective value and EBM-indicated global potential.

E.2.1 Relation to Classical Regret

Special Case ( α = 0 ):

which matches the conventional regret measure.

Full Global Exploration Term : With α > 0 , the difference [ E θ ( x ∗ ) -E θ ( x t ) ] adds a global exploration penalty: ignoring a low-energy (high-potential) region leads to an increased regret. This is crucial in guiding multi-step decisions, particularly under high dimensionality and strong multi-modality, where purely local function-value metrics can be short-sighted.

Hence:

In many Bayesian optimization analyses, it is well-known that

where g ( T ) might be √ T or log T , depending on the kernel, dimension, and assumptions. Consequently,

showing sublinear growth in the new Landscape-Aware Regret (LAR) and hence preserving overall convergence properties. While the exact constant C and function g ( T ) may vary, this demonstrates that our Landscape-Aware Regret (LAR) does not compromise sublinear convergence guarantees; rather, it offers a refined perspective by incorporating global EBM signals.

E.4 Addressing Potential Concerns

1. Why not stick with standard regret?

Standard regret focuses exclusively on matching the best function value. In complex, highdimensional tasks, algorithms may become trapped in local optima yet still show decreasing f ( x ∗ ) -f ( x t ) . Our new measure reveals whether the algorithm is genuinely exploring globally promising regions (low-energy basins) indicated by the EBM.

E.3 Theoretical Justification

E.3.1 Smoothness and Boundedness Assumptions

We typically assume: f -Lipschitz: The objective f is L -Lipschitz continuous:

E θ -Lipschitz: The EBM function E θ is L E -Lipschitz:

In practice, E θ is learned in such a way that lower energy typically correlates with higher values of f .

E.3.2 Sublinear Regret Bounds (Sketch)

Consider the cumulative Landscape-Aware Regret (LAR) over T iterations:

If E θ is positively correlated with f -for instance, E θ ( x ) ≈ C f ( x ) in high-value neighborhoodsthen:

Does this Landscape-Aware Regret (LAR) conflict with standard BO theory? Not necessarily. Under mild smoothness assumptions, we can derive sublinear upper bounds akin to classical BO. The energy term reweighs the local vs. global potential. When E θ aligns with f , the sublinear property is preserved.

Choosing α . In LAR, α weights the global-opportunity term α · ( E θ ( x ∗ ) -E θ ( x t ) ) rather than serving as a task-specific tuning knob. We fix a single default α = 0 . 3 across all benchmarks (selected on held-out tasks) and do not retune per task, ensuring fairness and comparability. This design reflects LAR's motivation-penalizing missed globally promising basins-while preserving interpretability: α → 0 recovers standard regret; moderate values in [0 . 2 , 0 . 5] are robust in sensitivity studies (App. E). Practically, use smaller α on smoother/unimodal landscapes and nearer 0 . 5 for highly multi-modal ones. Empirical online min-max scaling of f and E θ keeps α numerically stable; overly large α may overemphasize the energy term and is discouraged.

By incorporating both function value and energy signals into R LAR t , our Landscape-Aware Regret (LAR) drives the algorithm to account for the broader search landscape of potential optima-an especially important factor when dealing with multi-step lookahead and high-dimensional complexity. The theoretical analysis suggests that well-known sublinear regret behavior can remain intact, provided the EBM faithfully reflects the global structure of f . Empirically, this measure helps distinguish genuine global improvement from mere local refinements.

F Classic (Exact) Gaussian Process: Full Derivations

F.1 Posterior Distribution Proof for Noisy Observations

Consider the dataset D n = { ( x i , y i ) } n i =1 where each observation is modeled by

Assume f ( · ) has a Gaussian Process prior:

Let X ∈ R n × d denote the collection of inputs, and y ∈ R n the corresponding outputs. We define:

where m ( X ) = [ m ( x 1 ) , . . . , m ( x n ) ] ⊤ and

Given observation noise ε i ∼ N (0 , σ 2 n ) , the likelihood is:

Hence, the marginal distribution of y is

Posterior at a New Point. Let x ∗ ∈ R d be a test input. Define

The joint distribution of f ( X ) is which implies

From properties of multivariate Gaussians, we have

Conditioning on y yields the posterior distribution:

where the posterior mean and variance are given by:

This completes the proof that the posterior is Gaussian and that the conditional distributions adhere to the expressions above.

F.2 Derivation of Statistics: Variance, CDF of f ( x ) , and Probability of Duel

This appendix provides a deduction of the mathematical derivations, posterior distributions, and proof related to Classic, Sparse, and Deep Gaussian Processes (GPs). We expand posterior mean and variance, detail the derivations of common statistics such as cumulative distribution functions (CDFs), and illustrate how 'Probability of Duel' or more exotic acquisition-related quantities can be computed within the GP framework. The posterior distributions of Classic, Sparse, and Deep Gaussian Processes differ in terms of computational complexity and accuracy. Sparse Gaussian Processes, as discussed in the provided paper, utilize a variational approximation with sparse inverse Cholesky (SIC) factors, allowing for scalable and accurate inference. This method achieves highly accurate prior and posterior approximations with a computational complexity that can be handled via stochastic gradient descent in polylogarithmic time per iteration [53].

Posterior Variance. Recall from the standard GP posterior derivation (see Eqs. (1) and (2) in the main text or Appendix) that if

then the variance σ 2 ( x ∗ ) is given by

where

This posterior variance (also often denoted σ 2 f ∗ ) captures the GP's uncertainty at x ∗ . Many acquisition functions in Bayesian Optimization rely explicitly on σ ( x ∗ ) := √ σ 2 ( x ∗ ) to steer exploration. A canonical example is the Upper Confidence Bound (UCB):

where β > 0 is a user-chosen parameter balancing exploitation (the posterior mean) and exploration (the posterior std. dev.).

Derivation Outline for Posterior Variance

Joint Gaussian Setup. From the GP prior and noisy observations, we have

For a new test point x ∗ , the joint distribution of y and f ( x ∗ ) is again Gaussian.

Conditional Gaussian Formula. Conditioning on y follows the standard multivariate normal conditioning rule:

Extracting Covariance. The variance σ 2 ( x ∗ ) comes from the Schur complement of the block matrix

This completes the derivation of Eq. (2).

yielding

CDF of f ( x ∗ ) . Because

Its mean is

it follows that the random variable f ( x ∗ ) is distributed normally with mean µ ( x ∗ ) and variance σ 2 ( x ∗ ) . The cumulative distribution function (CDF) at a threshold z ∈ R is given by

where Φ( · ) is the standard Normal CDF. This CDF can be used in 'probabilistic improvement'-type acquisitions, such as:

where f + is the current best objective value and ξ is a small positive constant (the 'improvement' margin).

Detailed derivation for CDF.

From the posterior formula, f ( x ∗ ) is normally distributed with known mean and variance.

Probability of Duel. Sometimes one wishes to compare two inputs x i and x j under the GP posterior and compute P ( f ( x i ) > f ( x j ) ) . This arises in 'dueling bandits' or certain multi-armed bandit frameworks. Under GP modeling, ( f ( x i ) , f ( x j )) is a jointly Gaussian vector:

where µ ij = ( µ ( x i ) , µ ( x j ) ) ⊤ , and Σ ij depends on the posterior variances σ 2 ( x i ) , σ 2 ( x j ) and the posterior covariance Cov ( f ( x i ) , f ( x j ) ) . For instance, in the simpler scenario of comparing a new candidate x to a known best x + , the probability that x outperforms x + is

Below, we outline how to derive this formula from the bivariate normal distribution.

Probability of Duel Derivation.

Step 1: Joint Posterior of ( f ( x ) , f ( x + )) . By the GP's properties,

Step 2: Probability P ( f ( x ) > f ( x + ) ) . Define g = f ( x ) -f ( x + ) . Then g is also a Gaussian random variable because any linear combination of jointly Gaussian variables remains Gaussian. Concretely,

and its variance is

Hence,

Step 3: Probability Computation via Normal CDF.

where Φ( · ) is the standard Normal CDF. This completes the derivation of the 'Probability of Duel' formula.

We have shown how the posterior variance arises directly from the conditional Gaussian equations, how the CDF of f ( x ∗ ) follows from standardizing a normal variable, and how to compute the Probability of Duel by considering the bivariate normal distribution over ( f ( x ) , f ( x + )) . These derivations hold whenever ( f ( x ) , f ( x + )) is jointly Gaussian, which is guaranteed under the GP prior for any finite collection of points. [54]

Implications in Bayesian Optimization. The posterior variance σ 2 ( x ∗ ) drives exploration-based acquisitions like UCB or ϵ -greedy strategies [55]. The CDF of f ( x ∗ ) allows one to formulate Probability of Improvement (PI) or Expected Improvement (EI) style criteria. The Probability of Duel P ( f ( x ) > f ( x + ) ) helps in advanced multi-armed or pairwise preference settings, ensuring that the decision to pick x over x + is grounded in the bivariate normal property of the GP posterior.

G Short-Run MCMC and Energy Bounds for EBMs

This appendix provides a detailed mathematical exposition of how short-run MCMC is used to approximate sampling from an Energy-Based Model (EBM) and how energy bounds can be established under limited MCMC steps. Our derivation is inspired by prior works on learning latent-space EBMs and training EBMs via short-run MCMC [43, 56].

G.1 Background and Setup

Let { x i } n i =1 be i.i.d. samples drawn from an unknown data distribution p data ( x ) . We consider an EBM of the form

where E θ ( x ) is the energy function , parameterized by θ ∈ Θ , and Z θ is the partition function (intractable for high-dimensional x ). The EBM parameters are learned by (approximate) maximum likelihood estimation (MLE). In the ideal MLE scenario, one would seek to solve

However, because Z θ is typically intractable, direct gradient computations of log p θ ( x ) require approximations of the underlying distribution p θ ( x ) . Short-run MCMC addresses this by running a limited number of MCMC steps to sample from p θ ( x ) in an approximate manner.

or equivalently,

G.2 Maximum Likelihood and Its Gradient

G.2.1 Exact Log-Likelihood Gradient

By definition, the EBM log-likelihood for a single data point x is

The full-data negative log-likelihood for a sample set { x i } n i =1 is

Taking the gradient with respect to θ yields

Combining (6) and (7) gives

After factoring out n , the average gradient can be written as

where L ( θ ) denotes the average negative log-likelihood (or equivalently, -1 n ∑ n i =1 log p θ ( x i ) ).

G.2.2 Challenges and the Need for MCMC Sampling

The second term in (9) requires sampling from p θ ( u ) . Classic MCMC methods (e.g. Metropolis-Hastings, Hamiltonian Monte Carlo) can in theory generate samples from p θ , but in highdimensional settings or when E θ is complicated, running sufficiently long chains is computationally expensive. This motivates the short-run MCMC approximation ˜ p θ .

G.3 Short-Run MCMC Approximation

G.3.1 Definition of Short-Run MCMC

Let ˜ p θ ( u ) be the distribution of a K -step Markov chain initialized from a simple or random prior p 0 ( u ) (e.g. Gaussian). Concretely, short-run MCMC often uses K ≪ (chain length needed for full convergence). A popular choice is the Langevin dynamics update :

where η > 0 is a step size. After K short iterations, we obtain u (mcmc) approximately drawn from ˜ p θ ( u ) . This approach is known as short-run or persistent short-run MCMC [43].

G.3.2 Approximate Gradient with Short-Run MCMC

Replacing E u ∼ p θ in (9) by E u ∼ ˜ p θ gives:

One may interpret ˜ p θ ( u ) as a short-run approximation to p θ ( u ) . The objective function thus becomes

where the second term reflects the mismatch between ˜ p θ and p θ . A smaller D KL indicates better short-run approximation.

G.4 Energy Bounds and Convergence Analysis

G.4.1 KL Divergence Between ˜ p θ and p θ

When the chain length K is insufficient for exact sampling, we have ˜ p θ ( u ) = p θ ( u ) . The quality of the short-run approximation can be measured by the KL divergence

If K is very large and η is sufficiently small, ˜ p θ ≈ p θ in principle. However, large K introduces high computational cost. Various theoretical works [57] suggest that if ηK remains moderate, we can keep D KL ( ˜ p θ ∥ p θ ) bounded by a constant multiple of ηK .

G.4.2 Bounding the Error Term

A typical bound states

for some constant C > 0 that depends on the Lipschitz properties of ∇ x E θ ( x ) and the dimension of x . Intuitively: Smaller η (step size) reduces the discrepancy but slows mixing. Larger K (number of MCMCsteps) moves the chain closer to equilibrium but raises computation cost. Hence, short-run MCMC is a practical trade-off: we accept a bounded deviation D KL in exchange for significantly faster per-iteration updates.

G.5 Contrastive Divergence and Other Corrective Methods

To mitigate the approximation error introduced by short-run sampling, some methods incorporate a contrastive divergence term [58]:

Minimizing CD encourages p θ to reduce mismatch with both p data and ˜ p θ . Additional refinements include: Adding an auxiliary network to learn corrections between ˜ p θ and p θ [59], further closing the gap induced by short-run sampling.

Learning rate scheduling for the Langevin steps to find a sweet spot between numerical stability and chain mixing.

Persistent chains, where the final states of short-run MCMC at iteration t become initial states for iteration t +1 , improving chain continuity over time [60].

Key Takeaways. Short-run MCMC approximates the model distribution p θ with fewer sampling steps K , making large-scale or high-dimensional EBM training more tractable. The MLE gradient is modified by E ˜ p θ in place of E p θ , introducing an error controlled by D KL (˜ p θ ∥ p θ ) . The total error often scales with ηK , where η is the Langevin step size, yielding a bounded but non-negligible gap.

Short-run MCMC thus balances computational feasibility with approximation accuracy. In the main text of this paper, we rely on short-run MCMC to train E θ ( x ) for identifying globally promising or underexplored basins in REBMBO, without requiring fully normalized densities at every iteration.

H MDPFormulation and PPO Details in REBMBO

This appendix provides a more detailed exposition of how Reinforcement Learning (RL), pecifically, Proximal Policy Optimization (PPO), is formulated and analyzed within the REBMBO framework. We expand upon the definitions of state, action, and reward, as well as the derivation of the PPO update rule and its stability guarantees under standard assumptions.

H.1 A Markov Decision Process (MDP) for Black-Box Optimization

MDPSetup. In REBMBO, each iteration of black-box optimization is cast as one time step of an MDP, M = ( S , A , P, R, γ ) , where:

̸

State space S : A state s t ∈ S comprises:

where ( µ f,t , σ f,t ) is the current GP posterior, and E θ ( · ) encodes the global energy landscape from the EBM.

Action space A : An action a t = x t is a point in the domain X ⊆ R d to be evaluated by the (expensive) black-box function f ( x ) .

State transition P ( s t +1 | s t , a t ) : Once x t is evaluated, we observe y t = f ( x t ) and update:

The GP posterior and the EBM parameters are then retrained on D t , resulting in a new posterior ( µ f,t +1 , σ f,t +1 ) and a possibly updated energy function E θ ( · ) . These updates define s t +1 .

Reward function R ( s t , a t ) : We design the reward to guide multi-step exploration and exploitation. A common choice is:

where λ ≥ 0 controls the trade-off between immediate function value f ( x t ) and global exploration via -E θ ( x t ) .

Discount factor γ : Often set to γ = 1 for finite-horizon tasks, since we may only have a limited evaluation budget in black-box optimization.

At each iteration (time step) t , the agent (the PPO policy) observes s t , chooses an action x t ∈ X , obtains reward r t , and transitions to s t +1 . This process continues for T steps until the budget of function evaluations is exhausted.

Objective. We wish to find a policy π ϕ that maximizes the expected return,

subject to the black-box constraint that f ( x ) can only be observed by actual function queries. Unlike single-step acquisition functions, an MDP formulation allows us to consider the cumulative effect of each action on future states and rewards.

H.2 Policy Gradient and PPO Derivation

Policy Gradient. A standard approach in RL is to parametrize a stochastic policy π ϕ ( a t | s t ) and update ϕ via gradient ascent on the expected return. The policy gradient theorem [61] states:

where ̂ A t is an estimator of the advantage function , typically computed as

with Q π and V π denoting the state-action and state value functions, respectively.

PPO with Clipped Objectives. Proximal Policy Optimization [38] stabilizes policy gradients by clipping the probability ratio

so that the new policy π ϕ does not deviate too drastically from the old policy π ϕ old . The PPO objective is defined as

where ε is a small hyperparameter (e.g. 0.1 or 0.2) controlling how far r t ( ϕ ) may deviate from 1. By taking gradient steps on L CLIP ( ϕ ) , PPO ensures that policy updates remain proximal to π ϕ old , thus preventing large destructive leaps in parameter space.

Proposition H.1 (Bounded Rewards and PPO Convergence) . If the reward r t is uniformly bounded, i.e. | r t | ≤ R max , and the value function estimator is sufficiently accurate, then under mild Lipschitz assumptions on ∇ ϕ log π ϕ ( a t | s t ) , the PPO updates converge to a stable policy that locally maximizes the expected cumulative reward. See [38, 62] for rigorous proofs.

Collect transitions : For iteration t = 1 , . . . , T , we observe s t , pick x t = a t ∼ π ϕ ( a t | s t ) , evaluate f ( x t ) , and compute reward r t = f ( x t ) -λE θ ( x t ) .

Estimate advantage : We use an estimator ̂ A t based on a learned value function V ψ ( s t ) or a truncated GAE [63].

Policy update : Apply gradient ascent to maximize L CLIP ( ϕ ) , restricting the probability ratio r t ( ϕ ) to the [1 -ε, 1 + ε ] interval.

Update environment state : Augment D t with ( x t , f ( x t )) ; retrain both the GP surrogate and the EBM for iteration t +1 . Over time, π ϕ evolves from an initial random or weak policy to one that strategically balances local exploitation (through the GP posterior) and global exploration (through the negative energy term -E θ ( x ) ).

H.3 REBMBO's Multi-Step Advantage

Unlike single-step Bayesian Optimization strategies that select a point x solely by an acquisition function α ( x ) at each iteration, REBMBO acknowledges that the agent's current choice influences future states (through the updated GP and EBM). By incorporating a Markov Decision Process view:

the PPO agent plans multiple steps ahead, mitigating the one-step bias that can plague conventional BO acquisitions. The synergy arises from: 1) GPPosterior : Provides local uncertainty estimates for f ( x ) , guiding short-term exploitation. 2) EBM Global Signal : Encourages sampling in underexplored or globally significant basins ( -E θ ( x ) ). 3) PPO Multi-Step : Dynamically balances short-run exploitation and long-run exploration via advantage-based updates.

Hence, REBMBO avoids pure greediness at each iteration and can systematically reduce cumulative regret over T steps, even in high-dimensional or multi-modal settings.

I Detailed Proofs and Derivations for Landscape-Aware Regret (LAR) Computation

I.1 Finite-Set Case and the Proof of Theorem B.1

Theorem I.1. Let { x t } T t =1 ⊂ D be any sequence of query points selected by the GP-UCB algorithm with confidence parameter

and suppose D is a finite set. Then for a Gaussian Process prior with mean function zero and covariance k ( x, x ′ ) , and for noise variance σ 2 , the cumulative regret up to time T satisfies

where

and I ( y A ; f A ) denotes the mutual information between the GP function values at A and the observations at A .

Proof of Theorem I.1. We prove Theorem I.1 by combining the results of several lemmas, each bounding different contributions to the regret [55]. Below is an outline of the argument:

Lemma 1 shows how to bound the deviation of the true function value from the GP-posterior mean by a scaled version of the posterior variance at each point in our domain D .

Lemma 2 then uses this fact to bound the instantaneous regret r t .

Lemma 3 writes an expression for the information gain I ( y 1: t ; f 1: t ) in terms of the estimated variance and observation noise σ 2 .

Lemma 4 combines the bounds established in Lemmas 2 and 3 to control the sum of squared regrets, ∑ T t =1 r 2 t , via γ T .

Finally, we sum up instantaneous regrets over t = 1 , . . . , T and invoke a Cauchy-Schwarz argument to relate ∑ T t =1 r t to √ ∑ T t =1 r 2 t . This yields the advertised high-probability bound on R T .

Full details appear in the lemmas below.

Lemma I.2 (Deviation Bound) . Pick δ ∈ (0 , 1) , and set

for t ≥ 1 . Then, for any x t ∈ D chosen at time t , the following holds with probability at least 1 -δ (over all t ):

Proof. The result follows the standard GP concentration argument. By construction of the GP posterior,

The union bound and a tail bound on Gaussian random variables (applied for all t ) give the desired statement. The precise choice β t = 2log( t π δ ) ensures that the event

holds for all t with probability at least 1 -δ .

Lemma I.3 (Bound on Instantaneous Regret) . Using the same β t as in Lemma I.2, the instantaneous regret at each t satisfies

with high probability.

Proof. By definition of x t , we pick x t to maximize the UCB,

Since x ∗ maximizes the true function f , we compare f ( x ∗ ) -f ( x t ) by bounding f ( x ∗ ) -µ t -1 ( x ∗ ) and µ t -1 ( x t ) -f ( x t ) via Lemma I.2. A short calculation then yields

Lemma I.4 (Information Gain Expression) . Let I ( y 1: t ; f 1: t ) be the mutual information between the observations y 1 , . . . , y t and the function values f ( x 1 ) , . . . , f ( x t ) under the GP prior with noise variance σ 2 . Then

Outline. Recall that y s = f ( x s ) + ϵ s , with ϵ s ∼ N (0 , σ 2 ) . The joint distribution of { f ( x s ) } t s =1 under the GP is Gaussian, and so is that of { y s } t s =1 . The standard formula for the log-determinant of a covariance matrix in a GP regression problem gives exactly

See standard references on GP mutual information bounds.

Lemma I.5 (Sum of Squared Regrets via Cauchy-Schwarz) . Under the same setup and notation as above,

where C 1 = 8 / log(1 + σ -2 ) and C 2 = σ -2 / log(1 + σ -2 ) are constants. Consequently,

Proof. One first bounds r 2 t ≤ 4 β T σ 2 t -1 ( x t ) (using Lemma I.3). Summing in t gives

for a suitable constant C 2 , and use Lemma I.4 to relate the final logarithmic sum to γ T . A CauchySchwarz argument yields

Proof of Theorem I.1. Combining Lemmas I.2-I.5 shows that with probability at least 1 -δ ,

A minor refinement yields the extra additive constant π 2 / 6 when summing over time (arising from finer bounding of the union events or the explicit 1 /t 2 terms if present), giving

Then observe that as claimed.

I.2 Generalization to Continuous and Convex D ⊂ R d

We now extend Theorem I.1 to arbitrary compact and convex domains D ⊂ R d . The statement of the result is as follows. [64]

Theorem I.6. Let D ⊂ [0 , 1] d be compact and convex, d ∈ N , r > 0 . Suppose the kernel k ( · , · ) of our GP prior satisfies a high-probability bound on the derivatives of (sample) paths: for some a, b > 0 ,

Pick δ ∈ (0 , 1) and define

Then, running the GP-UCB algorithm with β t on D , for a GP f with mean zero and covariance k ( · , · ) , the cumulative regret obeys

Precisely, with C 1 = 8 log(1+ σ -2 ) , one has

where γ T = max A ⊂ D, | A | = T I ( y A ; f A ) is the maximum information gain at the end of T rounds.

Remarks. The assumption on kernel derivatives (weaker than a global Lipschitz condition) imposes that the slope of the GP sample path at any point is large only with low probability [65]. Many standard kernels (e.g. RBF) satisfy this type of condition in practice.

Proof Strategy for Theorem I.6. Compared to the finite-domain case of Theorem I.6, we can no longer set β t in terms of | D | . Instead, we use a sequence of lemmas that replace discrete enumeration of D with a carefully chosen discretization D t . The main additional steps are:

Lemma 5 extends the finite-domain confidence bound ( cf. Lemma I.2) to the continuous domain by slightly redefining β t and using a union bound argument in time. We discretion D on a grid D t so that any x ∈ D is within a small distance of some [ x ] ∈ D t . This appears in Lemma 7 , allowing us to relate f ( x ) to f ([ x ]) . Combining these discretization bounds with the same style of argument as in Lemmas I.3-I.5 yields a regret bound of similar order, plus an extra ∑ ∞ t =1 1 /t 2 term that converges to a constant ( π 2 / 6 ).

Lemma I.7 (Continuous Confidence Bound) . Pick δ ∈ (0 , 1) , and set

Then for all t ≥ 1 , with probability at least 1 -δ ,

Proof. The proof is exactly as in Lemma I.2 except β t is defined independently of | D | (since D is now infinite). We apply a Gaussian tail bound plus a union bound over t , ensuring

Lemma I.8 (Grid Discretization) . Consider a discretization D t with mesh size τ d t such that for every x ∈ D , there exists some [ x ] ∈ D t with

This ensures that D can be covered by small hypercubes of side length r d/τ t . We will choose τ t so that ∑ t ≥ 1 τ -1 t = 1 (or a similarly convergent series) and thereby control the union over t .

Lemma I.9 (Bounding f ( x ∗ ) -µ t -1 ( x ∗ ) t ) . Pick δ ∈ (0 , 1) , set

and let [ x ∗ ] t be the closest point to x ∗ in the discretization D t from Lemma I.8. Then with probability at least 1 -δ (over all t ),

Sketch. The derivative bound on the GP sample paths implies

with probability at least 1 -ae -( L/b ) 2 , for L suitably large. Using the discretization property ∥ x ∗ -[ x ∗ ] t ∥ 1 ≤ r d/τ t (from Lemma I.8), one obtains

by choosing τ t to shrink quickly enough in t . The remainder of the argument parallels Lemma I.2, showing that µ t -1 ([ x ∗ ] t ) is close to f ([ x ∗ ] t ) by β 1 / 2 t σ t -1 ([ x ∗ ] t ) . Combining these two pieces completes the proof.

Lemma I.10 (Final Regret Bound in the Continuous Case) . Pick δ ∈ (0 , 1) , and set

with ∑ t ≥ 1 τ -1 t = 1 and τ t > 0 . Then the instantaneous regret at time t satisfies

with probability at least 1 -δ . Consequently,

with probability at least 1 -δ , establishing Theorem I.6.

Sketch. By definition of the GP-UCB strategy,

Arguing as in Lemma I.3, one obtains

Using the grid argument (Lemma I.9) to control | f ( x ∗ ) -µ t -1 ([ x ∗ ] t ) | and then bounding µ t -1 ( x ∗ ) -µ t -1 ( x t ) similarly gives

Summing over t = 1 to T yields

As ∑ ∞ t =1 1 /t 2 = π 2 / 6 , this second term is bounded independently of T . The first term is handled by the same technique as in the proof of Theorem ?? (Lemma I.5), giving the factor √ C 1 T β T γ T . Hence,

J Sparse Gaussian Process: Detailed Variational Derivations

J.1 Inducing Points and Joint Distribution

We introduce m ≪ n inducing points Z = { z j } j m =1 and define the function values at these points as u = f ( Z ) ∈ R m . Under the GP prior:

where f = f ( X ) ∈ R n . For clarity:

J.2 Variational Approximation for Sparse GP

We define a variational distribution

where q ( u ) is free to be any Gaussian N ( µ u , Σ u ) . The posterior p ( f , u | X , y ) is approximated by q ( f , u ) . We optimize the Evidence Lower BOund (ELBO):

Substituting q ( f , u ) = p ( f | u ) q ( u ) , we can rewrite the bound in terms of q ( u ) alone, thus reducing the cost from O ( n 3 ) to roughly O ( nm 2 ) or O ( m 3 ) , depending on the particular scheme (e.g., FITC, VFE, etc.).

J.3 Sparse Posterior for New Points

After learning q ( u ) = N ( µ u , Σ u ) , the posterior predictive at a test location x ∗ is:

where p ( f ( x ∗ ) | u ) can be computed from the conditional Gaussian rule. One obtains a Gaussian with mean

and variance

(a typical formula in variational sparse GPs; some specifics differ depending on the chosen approximate method). These steps confirm that the predictive posterior remains a Gaussian with a distinct mean-variance form from exact GPs, yet still compatible with BO or RL-based exploration.

K Deep Gaussian Processes: Extended Proofs

K.1 Two-Layer Deep GP Setup

We demonstrate a two-layer DGP as the simplest hierarchical example. Let h (1) ( x ) ∈ R D 1 be the first latent layer, and h (2) ( h (1) ( x )) ∈ R D 2 be the second layer, eventually producing a scalar f ( x ) . For clarity, suppose D 1 = D 2 = 1 , so each layer is one-dimensional. Then:

where u = h (1) ( x ) , u ′ = h (1) ( x ′ ) .

K.2 Approximate Variational Inference

Since there is no closed-form expression for p ( h (1) , h (2) | X , y ) , we introduce variational distributions at each layer with inducing points or random Fourier features. For instance, define:

We then maximize a suitable ELBO with respect to q ( u 1 , u 2 ) . Once optimized, the posterior predictive for f ( x ∗ ) = h (2) ( h (1) ( x ∗ )) is approximated by integrating out the latent variables in each layer. The final result is typically a mixture or an integral of Gaussians, but many DGP frameworks simplify to produce an effectively Gaussian output with approximated mean ˜ µ ( x ∗ ) and variance ˜ σ 2 ( x ∗ ) . Although more complex than single-layer GPs, the procedure can model multi-scale or nonstationary phenomena.

K.3 Deep GP Posterior Probability Computations

Once we have the approximate distribution q ( f ( x ∗ ) ) from a DGP, we can in principle compute or approximate:

Variance. ˜ σ 2 ( x ∗ ) emerges from the nested GP integrals. In practice, a sample-based approach can be used:

CDF and Probability of Duel. If the final output remains approximated by a Gaussian, we can directly apply the same reasoning as in the Classic GP. If the distribution is not a simple Gaussian, one may resort to numerical quadrature or sampling-based estimates of P ( f ( x ) > f ( x ′ ) ) .

L Additional Details on Deep GP (REBMBO-D)

L.1 Approximation Theory and Regret Bounds

In this section, we elaborate on the theoretical considerations for REBMBO-D using a more formal mathematical argument, avoiding bullet points and instead employing direct deductive reasoning [41]. While the use of deep kernels can provide substantial flexibility for modeling highly non-stationary or multi-scale objectives, it also introduces new challenges in establishing sub-linear regret guarantees that parallel those of simpler Gaussian Process (GP) kernels (e.g., RBF or Matérn).

Formal Conditions on the Latent Mapping ϕ . Let X ⊂ R d be the original input domain, and let ϕ : X → R D be the embedding function parameterized by Θ . We first assume that ϕ is sufficiently smooth, in the sense that there exists a constant L > 0 such that

for all x , x ′ ∈ X . In addition, we posit that ϕ is invertible (or approximately invertible) on the regions of interest, so there exists a function ϕ -1 : ϕ ( X ) → X such that ϕ -1 ( ϕ ( x )) ≈ x for all x in the subdomain where data are collected. The invertibility condition prevents pathological distortions in the latent space, which ensures that distances in X are consistently reflected in ϕ ( X ) . Under these conditions, if f : X → R is the black-box function of interest, then the composition f ◦ ϕ -1 inherits properties similar to functions typically modeled by kernels in standard Bayesian Optimization.

Implications for the GP Surrogate. Define a deep-kernel function k Θ ( x , x ′ ) = k ( ϕ ( x ) , ϕ ( x ′ )) , where k is a positive-definite function on R D . Because ∥ ϕ ( x ) -ϕ ( x ′ ) ∥ ≤ L ∥ x -x ′ ∥ , one can show that the kernel k Θ remains Lipschitz in each argument up to a constant factor dependent on L . Classical results on GP-based Bayesian Optimization (BO) often rely on bounding the maximum information gain γ T after T observations:

where I ( · ) denotes the mutual information. In typical kernel-based BO analyses, γ T = O (log T ) or other sublinear forms in T , provided D is not too large or the kernel has controlled smoothness. Hence, under smoothness and boundedness assumptions on ϕ , one can adapt standard coveringnumber arguments from reproducing kernel Hilbert spaces (RKHS) to show that γ T remains small or grows sublinearly in T .

Deduction of Sublinear Regret. Let x ∗ be the global maximizer of f in X , and let x 1 , . . . , x T be the points sampled by REBMBO-D over T iterations. We define the cumulative regret

Because REBMBO-D employs a Gaussian-process-like surrogate in the latent space ϕ ( X ) , the usual BO proofs (for example, those from GP-UCB or GP-EI) can be transferred under suitable transformations. Specifically, if one can establish that the posterior variance σ 2 t ( x t ) in the deep kernel scenario shrinks at a rate governed by γ T , then one obtains an upper bound of the form

for a constant C that depends on hyperparameters of the kernel, the Lipschitz constant L , and the amplitude of noise. Since γ T grows at most sublinearly in T under the aforementioned conditions on ϕ and k Θ , this implies sublinear growth in R ( T ) . Formally, if γ T = O (log p T ) for some p ≥ 1 , then R ( T ) = O ( √ T log p T ) , which remains sublinear in T .

REBMBO-D inherits the potential for sub-linear regret from classical GP-based BO methods by casting the black-box function f into a latent space via ϕ , applying standard kernel-based BO proofs under smoothness and boundedness conditions, and leveraging EBM-driven exploration and PPObased control to avoid premature convergence. More formally, given that ∥ ϕ ( x ) -ϕ ( x ′ ) ∥ ≤ L ∥ x -x ′ ∥ and ϕ -1 exists in relevant regions, one can show that the maximum information gain γ T remains manageable, leading to regret bounds R ( T ) = O ( √ T γ T ) . If γ T grows sublinearly in T , then R ( T ) is itself sublinear. Therefore, under conditions of smooth embedding, stable EBM exploration, and bounded PPO updates, REBMBO-D can achieve the claimed sublinear regret growth.

M Detailed Implementation on Nanophotonic Structure

This appendix provides an expanded discussion of how REBMBO can be applied to the design of nanophotonic structures, using a nanosphere simulation as an illustrative example. Each iteration refines multiple components: a Gaussian Process (GP) for local uncertainty modeling, an EnergyBased Model (EBM) for global exploration, and a Proximal Policy Optimization (PPO) agent for adaptive multi-step decision-making. The detailed steps and associated formulas are presented below.

M.1 Problem Setup and Parameterization

Consider a nanosphere configuration described by physical parameters such as layer thickness, doping level, refractive index, or particle radius. Collect these parameters into an input vector

where each component x i lies within a feasible range determined by physical constraints (for instance, layer thickness within [ 0 , 500 ] nm, refractive index in [ 1 . 2 , 2 . 5 ] , or doping concentration in [ 0 , 10 20 cm -3 ] ). A forward simulator, such as a finite-difference time-domain (FDTD) solver or a rigorous coupled-wave analysis (RCWA) tool, evaluates the optical response of this nanosphere. Define

where F represents the black-box nanosphere simulation that outputs a scalar performance metric (for example, transmittance, reflectance, or absorption). Each call to F is typically expensive, motivating a data-efficient optimization approach.

M.2 Initialization and Data Collection

Initial Dataset. At the start, gather n 0 initial samples,

by either drawing randomly from X or using a space-filling design (e.g., Latin hypercube sampling). This initial dataset seeds both the GP and the EBM with basic knowledge of the input-output relationship.

Dynamic Dataloader. In subsequent iterations, the framework proposes new points x t . If x t was evaluated previously, retrieve the stored outcome to avoid redundant simulation. Otherwise, run the nanosphere simulator:

and append the pair ( x t , y t ) to the dataset, now denoted D t . This online loading procedure is essential for large-scale or expensive problems, as it triggers computationally heavy simulations only when the algorithm deems a candidate worthwhile.

M.3 Gaussian Process Surrogate

After the initial dataset D 0 is collected, train a GP to estimate the posterior mean µ 0 ( x ) and variance σ 2 , 0 ( x ) . At iteration t , the GP is updated with the most recent dataset D t -1 , yielding

The explicit forms for µ t ( x ) and σ 2 t ( x ) follow from standard GP regression:

where X t -1 = [ x 1 , . . . , x n t -1 ] , y t -1 = [ f ( x 1 ) , . . . , f ( x n t -1 ) ] T , K is the kernel matrix ( K ij = k ( x i , x j )) , and σ 2 n is a noise variance. Common kernel choices (e.g., RBF or Matérn) may be combined or extended based on domain expertise in nanophotonics. The GP posterior helps identify regions of high predicted performance or substantial uncertainty, both relevant for exploration.

M.4 Energy-Based Model (EBM) Training

In parallel with GP updates, the EBM parameters θ are retrained or partially trained at each iteration to ensure global coverage of the search space. Formally, the EBM density is given by

Since Z θ is typically intractable, the algorithm employs approximate methods such as short-run MCMCor contrastive divergence. In short-run MCMC, one might initialize a set of particles { x ′ j } near the dataset D t -1 , then evolve them briefly under gradient-based moves

where z j ∼ N (0 , I ) . A few such iterations suffice to update θ based on the mismatch between synthesized samples and real data. The resulting energy function E θ ( x ) tends to give lower values (higher probabilities) to regions that have shown promise or remain unexplored. This global signal complements the GP's local insights.

M.5 Reinforcement Learning via PPO

Let s t be the RL state that aggregates information from the GP and EBM, such as µ t ( x ) , σ t ( x ) , E θ ( x ) , or other relevant features (e.g., iteration count or GP hyperparameters). A policy π ϕ then chooses a new configuration x t = π ϕ ( s t ) . Proximal Policy Optimization (PPO) [38] updates ϕ by constraining large changes in the probability ratio

thereby promoting stable learning. In the nanophotonic setting, a t ≡ x t . After simulating y t = f ( x t ) , define a reward function that balances direct performance and EBM-driven exploration. A common choice is

where R penalizes lower performance (e.g., by taking the negative of a target error or the negative of -f ) and γ EBM scales the global exploration term. The PPO objective,

where A t is the advantage function, is then maximized to refine π ϕ . This process endows the sampling policy with the capacity to move beyond local maxima and systematically explore new regions of the parameter space.

M.6 Iterative Algorithm and Convergence

At each iteration t , the GP and EBM are updated to reflect the newly acquired observations { ( x t , y t ) } . The GP posterior µ t , σ t captures refined local estimates of f ( x ) , while the EBM modifies E θ ( x ) for broader coverage. The RL agent receives an updated state s t +1 and modifies its policy π ϕ according to the reward signals. The next point x t +1 emerges from this policy, balancing local exploitation with global exploration. Over multiple iterations, the collected samples cluster near high-performance configurations, effectively converging on near-optimal nanosphere designs with fewer evaluations than naive or single-step methods.

Practical Considerations. The nanosphere simulator must be callable multiple times under varying parameter inputs, and the MCMC-based EBM training typically requires choosing a small number of gradient steps per iteration. PPO's hyperparameters (e.g., learning rate, minibatch size, and clipping threshold ϵ ) can be selected via standard tuning heuristics [66]. When applied to more general nanophotonic designs, the same approach applies as long as the underlying simulator remains differentiable or partially differentiable if gradient-based EBM updates are desired, although purely sampling-based EBM training can also succeed. Finally, switching the GP variant (classic, sparse, or deep) depends on data scale and computational feasibility, making the method adaptable to a range of problem sizes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the abstract and introduction sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: See our limitation discussion in the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The paper includes each theoretical result accompanied by a full set of assumptions and complete, correct proofs. These proofs, along with the necessary theorems and lemmas, are clearly numbered, cross-referenced, and thoroughly detailed either within the main text or the supplemental materials.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: The paper provides all necessary information required to reproduce the main experimental results, thereby supporting its main claims and conclusions effectively. The information can be found in the Experiment section and Appendix C, D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: The paper ensures open access to both data and code, providing comprehensive instructions in the supplemental material to faithfully reproduce the main experimental results. The code and data can be found in the abstract. The instructions can be found in the README.md of the code fold.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: We provided the experiment settings including platform information, dataset source, hyperparameter settings, etc. in the Experiment section and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] .

Justification: This paper thoroughly considers statistical experimental studies, calculating the variance for each experiment to obtain more precise experimental data results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: We discuss the time efficiency of the proposed method in the Methodology section and provide the information for the experimental platform in the Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: This paper strictly adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] .

Justification: See the discussion in the Introduction and Conclusion sections.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: All the baseline model code and open source data are well-cited and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes] .

Justification: In this paper, the newly introduced algorithmic assets are well documented, and these assets are provided after being anonymized. The detailed documentation is presented through structured templates, covering key information such as the training process, licensing details, and potential limitations.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The paper does not address any potential risks to participants. All authors and related institutions have been anonymized.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA] .

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a combination of Bayesian optimization (BO) with a GP-UCB acquisition function, an energy-based model (EBM), and a Proximal Policy optimization (PPO) Reinforcement Learning Agent as an optimization strategy for expensive-to-evaluate black-box functions, similar to Bayesian Optimization.\nIn contrast to other BO methods, the introduced framework enables a trade-off between global and local exploration, allowing for multistep planning. The idea is that the EBM is a better model to capture the global structure of the problem, while a Gaussian process is used for local modeling. The PPO agent is then used for multistep look-ahead, mitigating the standard myopic setting of BO. Experimental synthetic benchmarks demonstrate that the introduced method performs better than other BO approaches in terms of a metric called pseudo-regret. This metric considers, in addition to the classical regret formulation, global exploration properties, penalizing the non-exploration of parts of the parameter space where potential optima could be located. \n\nContributions are: \n- Combination of Energy-based models and GP model \n- Multistep look ahead mechanism with PPO \n- Introduction of the pseudo-regret metric\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**S1: Relevant Problem: Multistep look ahead BO.**\nLeaving the myopic setting of BO is a relevant research field and also has an impact on the field of batch-BO methods.  \n\n**S2: Combination of reinforcement learning (RL) and probabilistic modeling.**\nThe approach to combine RL with probabilistic models generally makes sense. \n\n## Weaknesses\n\n**W1: Results are not reproducible.** \nThe provided code cannot be used to reproduce the results as neither the baselines nor the introduced method are implemented. It is mainly dummy code, where, for example, every baseline acquisition function only returns 0.5 of the state space. Therefore, the results are not reproducible. It is unclear which implementation was used for the given results. Furthermore, it is unclear which implementation of the baselines is used, which would be important for the analysis of the computational overhead. \n\n\n**W2: Pseudo-regret metric.**\nThe performance of the introduced algorithm is mainly evaluated based on the pseudo-regret metric, which is introduced in the paper. While this metric can be used as one evaluation criterion, it would also be helpful to have a comparison in terms of standard regret in the main part of the paper.  \n\nAn additional comparison against the standard regret metric would improve the results and give a hint if the introduced method is also helpful in a standard BO setting. This comparison is partly made in \"C.8 Performance Metrics Heatmap\"; however, here, TurBO is missing, which is the only locally searching BO method. \n\nFurthermore, in the experiments, it is unclear which value of $\\alpha$ was chosen when using the pseudo-regret metric. The supplementary material suggests a grid search for $\\alpha$ but does not state how this was applied in the paper. \n\n\n**W3: Introduction of many hyperparameters.**\nBy combining multiple methods, many hyperparameters for the method arise; the question arises if there is one algorithm setting that can be easily transferred to real-world problems or if extensive tuning is necessary in each application. Even the newly introduced acquisition function EBM-UCB has two hyperparameters ($\\beta$ and $\\gamma$ ), and it is not clear how they need to be chosen. \n\n\n**W4: Noise-free experiments.**\nA key property of Bayesian Optimization is the possibility to deal with noisy function evaluations. However, in this paper, only one benchmark function includes noise. An open question is how sensitive this approach is to noisy function evaluations. \n\n\n## Summary of evaluation\nWhile the premise of the paper makes sense, and the idea of combining the GP model, an EBM, and PPO can make sense, the paper has some weaknesses. It is not ready for publication, and even though it is claimed in the checklist, there is no code to reproduce the experiments. \n\n\n## Minor comments: \n\nAbbreviations like REMBO (p.1, line  30), TURBO (p.1, line 31), MCMC (p.2, line 42), MLMC (p.2, line 75) are not introduced,  \n\np.2 BO is introduced as an abbreviation. It could also be used in line 38 and 43\n\nFigure 1: This is not a UML-Diagram \n\nline 106: $f: \\mathcal{X} \\rightarrow \\mathbb{R}$ or just $f$  instead of $f(x)$\n\nline 236 missing space\n\nequation line 246-247: $n$ is not introduced\n\nequation line 254-255: $\\epsilon$ not introduced\n\nThe paper would profit from a clearer problem formulation, which could be introduced as a separate section. There are the preliminaries of Online Black-Box Optimization; however, restating the full objective once more clearly and formally would help clarify the understanding of this paper. \n\nReferences: Bayesian optimization is not capitalized correctly in many references\n\nAppendix: Theoretical results build on Srinivas (2010), and some derivations are taken from Gaussian Processes for Machine Learning by Rasmussen and Williams, which could be referenced more clearly. \n\nIt would be helpful to include numbers to the equations, especially if they are referenced later.\n\nQUESTIONS:\nQ1: How was $\\alpha$ chosen for the experiments?\n\nQ2: How can $\\alpha$ be chosen for new settings? \nIs there one algorithm configuration that can easily deployed for new use cases without extensive hyperparameter tuning? \n\nQ3: Is the EBM-UCB acquisition used in the algorithm, or is this an alternative to the PPO evaluation? If EBM-UCB is an alternative acquisition function introduced to demonstrate the theoretical properties of the method, it would also make sense to compare it against this acquisition function in the experiments. From the pseudo-code, it seems like the models are trained, and then the PPO chooses the next evaluation.\n\nQ4: Figures 2 and 3: Why are all algorithms initialized differently? For a fair comparison, all algorithms should be initialized with the same initial data points. \n\nQ5: How does the proof in Appendix I differentiate from the results in \"Gaussian Process Optimization in the Bandit Setting:\nNo Regret and Experimental Design\", Srinivas et al. 2010?\n\nLIMITATIONS:\nThe lack of theoretical analysis of the combination of RL and probabilistic modeling is clearly addressed in the conclusion. \n\nLimitations that are currently not addressed include the number of hyperparameters required to employ the algorithm and the lack of analysis considering the algorithm's sensitivity to noisy function evaluations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes REBMBO, a framework for black-box optimization that combines three fundamental components: (1) a Gaussian Process (GP) to model the uncertainty, (2) an Energy-Based Model (EBM) trained using short-run MCMC to learn the global energy landscape, and (3) a Reinforcement Learning with Proximal Policy Optimization (PPO) approach to enable multi-step, non-myopic decision making. A new acquisition function, EBM-UCB, is introduced to unify GP-based local exploration with EBM-informed global guidance. The method is evaluated on several synthetic and real-world benchmarks, consistently outperforming standard Bayesian optimization (BO) and multi-step RL-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\nStrengths:\n* The paper is technically sound and presents thorough research.\n* Each module (GP, EBM, PPO) is well-motivated, clearly explained, and supported by solid theoretical foundations.\n* The experimental section is comprehensive, covering both synthetic benchmarks and real-world applications.\n\nWeaknesses:\n* The paper only includes one high-dimensional benchmark (HDBO 200D). Expanding the evaluation with more high-dimensional tasks would strengthen the contribution.\n\n**Clarity**\n\nStrengths:\n* The paper is well-written, well-organized, and easy to follow.\n* The modular structure (Modules A–C) is intuitive and clear.\n\nWeaknesses:\n* The acronym UCB (Upper Confidence Bound) is introduced late in the paper. \n* Figure (2) could be improved with smoother curves, higher-contrast color schemes, and bolder lines to better distinguish among methods.\n* While Figure (1) is visually appealing, it lacks informativeness. Replacing the illustrations with more precise, less decorative icons and clearer annotations could improve its clarity.\n\n**Significance**\n\nStrengths:\n* The method addresses a known limitation of Bayesian Optimization: its tendency to greedily exploit local optima rather than exploring under-sampled regions that may potentially contain better solutions.\n* The proposed framework applies to many real-world settings involving black-box functions.\n\n**Originality**\n\nStrengths:\n* To the best of my knowledge, the combination of GP, EBM, and PPO-based reinforcement learning used for black boxed optimization problems is novel and well-motivated.\n* The proposed EBM-UCB acquisition function is both theoretically justified and empirically proven.\n* The pseudo-regret metric, which incorporates a global exploration component derived from the EBM, is an innovative approach to enhancing the metric of success in optimization.\n\nQUESTIONS:\n* Could reporting the pseudo-regret be misleading? Does it guarantee that the found solution x is also better? What if non-optimal objective values are compensated by low energy?\n* Have you considered situations with severe scale mismatch between the objective and energy values? If their ranges differ significantly, it could affect reward computing and PPO training stability.\n* What is the practical limit on dimensionality for REBMBO? Are there degradation patterns or bottlenecks as dimensionality increases beyond what was tested?\n* Are there specific problem types or objective landscapes where REBMBO performs worse than traditional BO or pure RL-based methods?\n* Can a trained model generalize across similar tasks or domains?\n\nLIMITATIONS:\nThe discussion of limitations is relatively shallow, consisting of a single sentence in the conclusion: \"There may have been unavoidable training errors in EBM, and RL may have influenced theoretical convergence rates, leaving comprehensive analysis for future research.\" A more thorough discussion of the method’s limitations—such as computational cost, potential failure cases, lack of convergence guarantees, or scaling behavior—would provide a more balanced and transparent assessment of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces REBMBO (Reinforced Energy-Based Model for Bayesian Optimization), which combines three key components: (1) Gaussian Processes for local modeling, (2) Energy-Based Models (EBMs) for global exploration guidance, and (3) Proximal Policy Optimization (PPO) for multi-step lookahead planning. The method formulates each BO iteration as a Markov Decision Process and uses a novel EBM-UCB acquisition function that integrates GP uncertainty with global energy signals from short-run MCMC training. The authors propose three variants (REBMBO-C, REBMBO-S, REBMBO-D) using different GP configurations and evaluate on synthetic and real-world benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nQuality: The paper presents a technically sound approach that addresses a real limitation in Bayesian Optimization - the one-step myopia problem. The integration of EBMs with GP-based BO is novel and the mathematical formulation appears correct. The experimental evaluation covers diverse benchmarks from 2D to 200D problems.\nOriginality: The combination of EBMs, GPs, and PPO for BO is original. The EBM-UCB acquisition function and the MDP formulation of BO iterations represent meaningful contributions. The pseudo-regret metric incorporating global exploration penalties is also novel.\nSignificance: The work addresses an important problem in BO and demonstrates consistent improvements over strong baselines across multiple dimensions and problem types. The multi-step lookahead via RL could influence future BO research.\nClarity: The paper is generally well-written with clear motivation and methodology sections. Figure 1 effectively illustrates the overall framework.\n\n### Weaknesses\n\nQuality Issues:\n- Limited analysis of computational overhead - the method adds significant complexity with EBM training and PPO optimization\n- The choice of hyperparameters (especially the balance between GP, EBM, and RL components) lacks principled justification\n- Experimental setup details are insufficient for reproducibility\n\nClarity Problems:\n- The notation is sometimes inconsistent (e.g., switching between different symbols for the same concepts)\n- The relationship between the three modules could be explained more clearly\n- Some mathematical formulations are unclear (e.g., the exact form of the reward function)\n\nExperimental Limitations:\n- Only 5 runs per experiment is quite limited for statistical significance\n- Missing important baselines like BORE, qEI, or more recent RL-based BO methods\n- The pseudo-regret metric is non-standard and makes comparison with other work difficult\n- Computational cost comparison is missing despite the method's obvious overhead\n\nTechnical Concerns:\n- The EBM training via short-run MCMC may not converge properly, especially early in optimization when data is limited\n- The method has many hyperparameters (GP kernel parameters, EBM architecture, PPO settings) that need careful tuning\n- No analysis of how sensitive the method is to these hyperparameter choices\n\nQUESTIONS:\n- Computational Overhead Analysis: Can you provide a detailed computational complexity analysis comparing REBMBO to baseline methods? How does the overhead scale with problem dimension and number of iterations?\n- Hyperparameter Sensitivity: How sensitive is REBMBO to the choice of key hyperparameters (α, β, γ in the acquisition function and reward)? Can you provide guidelines for setting these in practice?\n- EBM Convergence: How do you ensure the EBM training converges properly, especially in early iterations with limited data? What happens if the EBM provides poor global guidance?\n- Theoretical Guarantees: Can you clarify the theoretical convergence guarantees? The paper mentions \"sublinear regret under mild assumptions\" but these aren't clearly stated in the main text.\n- Comparison Fairness: Given that REBMBO uses a non-standard pseudo-regret metric, can you provide results using standard regret metrics for fair comparison with existing methods?\n\nMy evaluation could increase if: (1) comprehensive computational cost analysis is provided, (2) theoretical guarantees are clearly stated and proven, (3) more extensive experiments with standard metrics are conducted, (4) hyperparameter sensitivity analysis is included, and (5) the method's limitations and failure cases are honestly discussed.\n\nLIMITATIONS:\nThe authors briefly mention limitations in the conclusion but don't adequately address potential negative societal impacts or failure cases. A more thorough discussion of when REBMBO might fail and its computational requirements is needed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces REBMBO, a novel Bayesian optimization framework that integrates Gaussian Processes (GPs), Energy-Based Models (EBMs), and Reinforcement Learning (RL). The key innovation lies in modeling the BO process as a Markov Decision Process (MDP), utilizing PPO-based multi-step planning to overcome the \"one-step myopia\" of standard BO methods. REBMBO also introduces a new EBM-UCB acquisition function that combines local GP uncertainty and global EBM energy to better explore complex, high-dimensional search spaces. Extensive synthetic and real-world experiments show that REBMBO outperforms state-of-the-art BO approaches across various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The combination of GP, EBM, and PPO into a unified BO framework is conceptually rich and addresses the long-standing limitation of one-step lookahead in classical BO.\n- The paper clearly explains each module (GP, EBM, PPO), how they interact, and provides algorithmic pseudocode for reproducibility.\n- REBMBO variants consistently outperform several baselines (TuRBO, BALLET-ICI, EARL-BO, KG, etc.) across synthetic (2D to 200D) and real-world (robotics, protein design) tasks.\n- Thoughtful ablation studies and detailed benchmark explanations strengthen empirical credibility.\n\nWeaknesses: While components are well-explained, the paper would benefit from open-sourcing code and including more reproducibility metrics (e.g., number of PPO iterations, sensitivity to λ in reward).\n\nQUESTIONS:\n- It is unclear why the author fixes the kernel for GP as a combination of RBF+Matern.\n- How sensitive is REBMBO to the choice of λ in the reward function?\n- Can the method scale to asynchronous or batched BO settings, which are common in real applications?\n\nLIMITATIONS:\nThe authors discussed the limitations of their method. However, incorporating discussion for concerns below would help improve the paper. The method assumes a fixed surrogate model (GP); it's unclear how flexible REBMBO is when the GP is inaccurate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}