EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization
Authors: Mujin Cheon
Jay H Lee
Dong-Yeun Koh
Calvin Tsay
Venue: ICML 2025 poster (2025)
Keywords: Bayesian Optimization
Reinforcement Learning
High-dimensional Optimization
Nonmyopic Bayesian Optimization
Primary area: optimization->zeroorder_and_blackbox_optimization
TL;DR: We propose an Attention-Deepsets encoder with actor-critic RL and a GP virtual-environment training procedure to provide an end-to-end solution for multi-step lookahead BO.

Abstract: To avoid myopic behavior, multi-step lookahead Bayesian optimization (BO) algorithms consider the sequential nature of BO and have demonstrated promising results in recent years. However, owing to the curse of dimensionality, most of these methods make significant approximations or suffer scalability issues. This paper presents a novel reinforcement learning (RL)-based framework for multi-step lookahead BO in high-dimensional black-box optimization problems. The proposed method enhances the scalability and decision-making quality of multi-step lookahead BO by efficiently solving the sequential dynamic program of the BO process in a near-optimal manner using RL. We first introduce an Attention-DeepSets encoder to represent the state of knowledge to the RL agent and subsequently propose a multi-task, fine-tuning procedure based on end-to-end (encoder-RL) on-policy learning. We evaluate the proposed method, EARL-BO (Encoder Augmented RL for BO), on synthetic benchmark functions and hyperparameter tuning problems, finding significantly improved performance compared to existing multi-step lookahead and high-dimensional BO methods.

EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization
Mujin Cheon 12
Jay H. Lee
Dong-Yeur Koh Calvin Tsay !
Abstract
2021). Many of these settings involve costly data acquisition (i.e., experiments) and no information about the gradient. Bayesian Optimization (BO) is popular in such problems due to its ability to query data-cfficient samples (Brochu et al., 2010; Paulson & Tsay, 2024). BO leverages a sur- rogate model with uncertainty quantification, typically a Gaussian process, or GP (Williams & Rasmussen, 2006), in conjunction with an acquisition function, which contains the "philosophy" about how to balance exploration and ex- ploitation, to identify and optimize the underlying function in a sequential sampling process.
To avoid myopic behavior, multi-step lookahead Bayesian optimization (BO) algorithms consider the sequential nature of BO and have demon- strated promising results in recent years. How- ever, owing to the curse of dimensionality, most of these methods make significant approxima- tions or suffer scalability issues. This paper presents a novel reinforcement learning (RL)- based framework for multi-step lookahead BO in high-dimensional black-box optimization prob- lems. The proposed method enhances the scala- bility and decision drdiiinn-makinggggg quality of multi-step lookahead BO by efficiently solving the sequen- tial dynamic program of the BO process in near- optimal manner using RL. We first introduce an Aetestion-DeepSets encoder to represent the state of knowledge to the RI. agent and subsequently propose a multi-task, fine-tuning procedure based on end-to-end (encoder-RL) on-policy learning. We evaluate the proposed method, EARL-BO (En- coder Augmented RL for BO), synthetic bench- mark functions and hyperparameter tuning prob- lems, finding significantly improved performance compared to existing multi-step lookahead and high-dimensional BO methods.
Traditional BO algorithms deploy acquisition functions such as Expected Improvement (Jones et al., 1998). Probability of Improvement (Kushner, 1964), and Upper Confidence Bound (Srinivas et al., 2010) in one-step lookahead poli- cies. In other words, these policies ignore the effect(s) of the choice made at current iteration on future steps and in- stead solely focus on the immediate maximization of the acquisition function. Previous studies (Ao & Li, 2024; Lam et al., 2016; Osborne et al., 2009; Wu & Frazier, 2019) have demonstrated that these so-called myopic policies can result in sub-optimal performance. Moreover, anticipating the se- quential nature of BO can enable consideration of the path of samples taken (Folch et al., 2022; Yang et al., 2024).
Along these lines, Ginsbourger & Le Riche (2010) observe that the decision-making process of BO can be seen as par- tially observable Markov decision process (POMDP), and therefore policies can be improved by multi-step lookahead decisions. Several lookahead methods have since been pro- posed. GLASSES (González et al., 2016) involves approx- imating the ideal lookahead loss function in an open-loop approach. Rollout-based strategies (Lam et al., 2016; Lee et al., 2020; Osborne et al., 2009) approximate the POMDP value function by "rolling-out heuristic pocision-making policies (e.g., maximizing Hl) over future time steps to ap- proximate long-term gains. In this category, Wu & Frazier (2019) suggest practical two-step lookahead methods using an envelope-function estimator to improve computational tractability. More recently, Cheon et al. (2024) introduce a reinforcement learning (RL) strategy to solve the multi-step lookahead POMDP of BO in near-optimal way.
1. Introduction
Optimization of an unobserved, "black-box" function un- derlies engineering applications such as hyperparameter tuning (Snoek et al., 2012), material discovery (Shahriari et al., 2015), reaction chemistry (Folch et al., 2022), energy systems (Thebelt et al., 2022), and robotics (Muratore al.,
'Department of Computing, Imperial College London, UK 2 Depaarmennt of Chemical and Biomolecular Engineering, Korea Advanced Institute of Science & Technology (KAIST). South Korea "Mork Family Department of Chemical Engineering and Materials Science, University of Southern California, USA. Corre- spondence to: Calvin Tsay <c.tsay @ imperial.cc.uk>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Multi-step lookahead methods generally face two critical

---

EARL-BO: RL for Multi-Step Lookahead BO
challenges: they must (1) simplify the Stochastic Dynamic Programming (SDP) problem, leading to a sub-optimal pol- icy, and/or (2) suffer from scalability issues that hinder applicability to larger and higher-dinge dishee-ddmensional problems.
between nearby points (e.g.. assumptions about the smooth- ness of the function), and dataset Dj {( }:) (i=1)" (i= the posterior distribution of the function value at new query point x is Gaussian, N NN4 (a; D&), K (x, xx; Dx)), where:
Given these challenges, this work introduces a novel frame- work combining end-to-end encoder architectures with re- inforcement learning to create a scalable, RL-based BO framework tailored for multi-step lookahead optimization black-box optimization problems.
(x; D2) = m(z) + k(z) (K + 12 p(x)) K x;; (xx,)) x)) kK (K+
Here, Is denotes the 1e k identity matrix. The mean " (a; D2) represents the posterior prediction of the function at x, and K ("xxx r;; Dx)) reflects the covariance.
in higher-dimensionn Our Encoder Augmented RL for BO (EARL--O) frame- work leverages the representational power of an Attention- DeepSets-band based encoder to parameterize the current knowl- edge of the state within a latent space that is amenable to RL. The RI. agent learns in a model-based fashion, using a GP virtual environment to receive rewards based on multi- step performance, completing the end-to-end learning struc- ture. The proposed integration enables the RL algorithm to solve the SDP of the BO decision-making process in a near-optimal, yet tractable way over an extended horizon.
The subsequent point J6+1 for evaluation is selected by optimizing an acquisition function A(r|Da), such that:
F4+1 = argmax (zx(Da). =EQ
(1)
For example, one popular acquisition function, Expected Improvement (EI), is defined as:
AEr (x) E
f(xx)())
(2)
In summary, our contributions include:
where Mx+ is the current best observation. The point that maximizes the acquisition function A(x) is selected as the next evaluation point, noting that this optimization prob- lem itself may be non-trivial to solve (Ament et al., 2023; Xie et al., 2024). While this one-step lookahead strategy can be effective in many scenarios, it is inherently myopic, focusing only on the immediate benefits and neglecting the long-term effects of decisions.
An integrated and modular RL-based BO method that efficiently solves the SDP inherent in multi-step BO.
The introduction of an Attention-DeepSets based en- coder, which provides a scalable representation of the knowledge state in BO, enhancing the model's 's ability to handle high-dimensional spaces.
An end-to-end, model-based learning procedure for the combined encoder-RR RL pair using a GP-based virtual environment.
2.1. Multi-step Lookahead Approaches
To overcome the limitations of one-step lookahead meth- ods, multi-step lookahead strategies seek to account for the impacts of current decisions subsequent evaluations.
Comprehensive evaluations of EARL-BO across both synthetic benchmark functions and real-world hyper- parameter tuning. against other multi-step lookahead and high-dimensional optimization methods.
Rollout-Based Bayesian Optimization. Rollout methods are a class of multi-step lookahead strategy, where future decisions are approximated using a base policy тg- This base policy could involve applying a heuristic or a compu- tationally simpler, sub-optimal strategy, such as policy that maximizes EI (2). The objective is to enhance the base policy by maximizing the cumulative expected reward over user-defined horizon H:
Our work advances the state of the art in non-myopic BO by addressing the limitations of existing approaches and provid- ing a scalable and modular framework for high-dimensional black-bor box optimization problems.
2. Background and Related Works
Arollout Arollouex:|DD)
EET Uk+ .
Σ r Bk+n)
Consider the problem of globally maximizing a continu- ous function f fxx) defined over a compact domain Q c Rdd When evaluating fz) is costly, it becomes crucial to min- imize the number of function evaluations.
Here, r(zk+n Mk++n is the reward step (I: + n), which is often defined as the improvement in the objective function, and E denotes the expectation over possible future trajec- tories given a policy T ~ π (Dx), with yk+1 = f(*x+1).
BO uses Gaussian process (GP) to model the objective function f(x). Based on the GP prior f(x) ~ GP(H, K), where ::Ω + R is the mean function and K f x Ω + R is the covariance (or kernel) function that encodes the correlation
The seminal work of Osborne et al. (2009) demonstrates the possibility of computing this rollout acquisition function
2

---

EARL-BO: RL for Multi-Step Lookahead BO
for I-D optimization problem with two-step lookahead horizon (H=2). Later, Lam et al. (2016) reduce the com- putational burden by approximating the expected future rewards using Gauss- performance on 2-D synthetic functions with up to five-step lookahead horizons. More recently, Lee et al. (2020) suggest rollout-based BO with variance reduction, i.e., reducing the uncertainty (variance) in the predictions made during each rollout to produce more reliable estimates of future rewards. The authors demonstrate this strategy on traditional BO methods such as HI, UCB, KG in 2-D and 4-D benchmark functions with up to six-step lookahead horizons.
trust region and modeled with GP. This approach has been extensively evaluated on diverse high-dimensional synthetic and real-world problems, such as the 200-dimensional Ack- ley function and robot control tasks, where TuRBO outper- forms traditional BO methods and alternative optimization techniques, including evolutionary algorithms.
quadrature, enabling good
SAASBO (Eriksson Jankowiak, 2021) introduces Sparse Axis-Aligned Subspaces to effectively handle the curse of dimensionality. By learning which dimensions are most rel- evant to the optimization objective, SAASBO can focus on the most influential parameters and scale to problems with hundreds of dimensions. Given their relative popularity and strong performance, we use TuRBO and SAASBO as bench- marks for comparison in higher-dimensional problems.
Despite the benefits of rollout-bas based BO and the above strategies, several challenges remain:
In similar vein, VAB-BO methods address high- dimensional BO using Variational Autoencoders (VAEs) to learn low-dimensional latent representations of the search space (Gómez-Bombarelli et al., 2018). LBO (Tripp et al., 2020) improves upon this by incorporating weighted re- training of the VAE, assigning higher weights to better- performing points and periodically updating the VAE. More recently, Chen al. (2024) propose PG-LBO, which intro- duces pseudo-label training to leverage unlabeled data and GP guidance to directly integrate labeled data.
Computational Complexity: Even with variance re- duction, the computational cost of evaluating multi- step lookahead strategies is substantial, particularly as the dimensionality of the search space increases.
Approximation Errors: The need to approximate future decisions using a base (heuristic) policy will inevitably lead rollout methods to sub-optimal policies.
To mitigate the latter approximation errors, Cheon et al. (2024) use RL to make multi-step lookahead decisions, showing improved performance compared to rollout-based methods on 2-D benchmark functions. However, the algo- rithm employs latticized representation of GP the RL agent input, which suffers from the curse of dimensionality.
2.3. Learning to Optimize (L20)
This work falls within the broader framework of Learn- ing to Optimize (I.20), an emerging paradigm that lever- ages machine learning techniques to improve traditional optimization methods. Specifically, instead of relying on predefined heuristics, L2O approaches learn optimization strategies from data, enabling them to adapt to different problem classes and potentially discover superior optimiza- tion behaviors (Chen et al., 2017; Ma et al., 2025).
Noting that the above works are limited to din-dimesioona (approx. <4-D) problems, this work focuses on multi-step lookahead BO for higher-dimensional search spaces. Due to the computational infeasibility of rollout methods for such settings, we compare performance against rollout-based BO for oow-dimesiional problems but against scalable (one-step lookahead) BO methods for higher-dimensional problems.
Recent advances in L.20 for black-box optimization include RIBBO (Song et al., 2024), which employs transformer architectures with regret-to-go tokens to learn end-to-end algorithms from offline datasets, and POM (Li et al., 2024), which combines population based optimization with trans- former encoders for zero-shot black-box optimization. A more comprehensive review of L.20 can be found in Ma et al. (2025). While these methods demonstrate promising re- sults, they again generally remain fundamentally limited to single-step, myopic decision making (the above approaches generate query points based solely on the current state).
2.2. High Dimensional Approaches
Several recent works address challenges posed by inereas- ing dimensionality in BO (Wang et al., 2023). Hoang et al. (2018) propose the Decentralized High-dimensional Bayesian Optimization (DEC-HBO) algorithm, which em- ploys a sparse factor graph representation of the objective function to exploit interdependent effects of input compo- nents while maintaining scalability. The authors propose a decentralized strategy for optimizing the acquisition func- tion and demonstrate DEC-HBO's performance on various tasks, including synthetic functions of up to ten dimensions.
3. Methodology
3.1. Preliminaries
Another popular method is Trust Region Bayesian Optimiza- tion, or TuRBO (Eriksson et al., 2019). TuRBO partitions the search space into multiple local regions, each defined as
Bayesian Optimization as MDP. Lam et al. (2016) con- ceptualize the decision-making process of BO as finite-
3

---

EARL-BO: RL for Multi-Step Lookahead BO
horizon dynamic program. The key idea is that, given a set of observed data Dg, the data-acquisition order and pro- cedure are irrelevant as long as the same data points are obtained. Thus, the BO setting satisfies the Markov prop- erty. Here, we express this framework using the equivalent Markov Decision Process (MDP) formulation, adopting the standard notation from Puterman (2014).
seek to represent data in a size- and sermutation-invariaa way using tailored encoder structure.
Reinforcement Learning (RL). RL is paradigm in ma- chine learning where an agent learns to make decisions by interacting with an environment (Sutton & Barto, 2018). In our context, we employ RL to learn an optimal policy for selecting query points, i..., solving the MDP formulation of the BO problem. Several studies use RL to learn such policy, e.g., by meta-learning acquisition functions for GPs (Hsieh et al., 2021; Volpp et al., 2019). Later, Shmakov et al. (2023) integrate learning of deep kernels, and Maraval et al. (2024) propose an end-to-end framework based on meta-learning both a transformer and an RL policy. These works are similar in spirit, but consider the 1.20 setting of meta-learning across problems (Chen al., 2017). On the other hand, EARL-BO aims to learn to emulate multi-step lookahead BO on the given task.
An MDP is defined by the tuple U,SSAA,P.A
T is the set of decision epochs, 7= = {0, 1, ....,, 1}.
with h oo here representing finite horizon.
S is the state space comprising the information neces- sary to describe the system at any given time t T.
A is the action space of possible decisions.
P(s'|s, a) is the transition probability, or the likeli- hood of moving to s' by taking action at state s.
In this work, we focus on the general setting where data from other tasks are not available for meta-learning, and instead take a model-based RL approach. Our framework is modular and agnostic to the specific choice of RL algorithm: here we employ Proximal Policy Optimization (PPO), policy gradient algorithm that addresses the challenges of step-size selection and sample efficiency (Schulman et al., 2017). PPO uses clipped surrogate objective to ensure that policy updates are not overly large (which could otherwise lead to performance collapse), defined as:
R(s, a, defines the reward received when transition- ing from state to state s' by taking action a.
Following this notation, decision rule n. : S - A maps states to (a distribution of) actions at time step t. A policy (oo, T1, .... T- is sequence of decision rules, one for each decision epoch. Given a policy æ, an initial state 80, and a horizon h, the expected total reward V (so) is:
hh-1
V (so) =E
Σ 1=
R (st, m((s),
(3)
L (P) = E, mmnn (0)
(5)
In this MDP framework, the objective to determine the optimal policy x² that maximizes the expected total reward:
where:
(arl)))) re( as =ta is the probability ratio between the new and old policies,
argmax V/5 (80); xENl
(4)
where II is the set of all possible policies. This formulation allows us to view BO as a sequential eecision-making prob- lem, where the goal is to maximize the cumulative reward over finite horizon by selecting the best possible actions at each step. Moreover, Paulson & Tsay (2024) observe that choosing acquisition function A can be mapped to re- ward function from the viewpoint of dynamic programming.
Ar is the estimated advantage at time 4, and
e is a hyperparameter that controls the clip range.
The clipping the objective function (often implemented with e = 0.2) ensures that the employed ratio between the new and old policies does not deviate significantly from unity, which helps to stabilize training in practice.
For the BO setting, ST A, A, P. R) can be construed as T: a user-intended lookahead horizon for BO, St data acquired in the BO process (Dt). A: the next query point of BO, P: the change in state when point xz41 is queried, and R: user-defined reward, which, analogously to (2), is com- monly chosen as R(D,, 11+1, D++1) = max Vt+1 - y;) for multi-step lookahead BO, where y denotes the maxi- mum value of y. Notice that directly using the obtained data D, as the state requires continually increasing the dimen- sionality of the state vector as BO proceeds. We therefore
3.2. Overview of EARL-BO
EARL-BO is modular framework comprising: (1) an en- coder module that learns representation of the state of data acquisition and (2) an RL module that learns the lookahead BO policy. An overview of EARL-BO is shown in Figure I. The framework is agnostic to the specific choice of encoder and RL algorithm. Similarly, the off-policy learning step described below can leverage any existing BO method.
4

---

EARL-BO: RL for Multi-Step Lookahead BO
(rg: Mi) are computed as:
CrtooCrttic
Enstroment
Irput
Tapet
CP GPintoration boe /Vittal Envionenenti
0₁ = softmax (an((1(2 V:))),
1
Netrock
Netwook
Bowarl
where fast is a scoring function that measures the relevance of each data point. The final output is then computed as:
DospSets Layer
Steroeyy butter
query e
J Outpe
Action
State Valla
Loss Caéculatos
Outpe
Updas Eucodar.
BABL-BO
Actie Catie
Datr (Di) p
Σ ag Sc..(KI1 M.)
(8)
relnorrss
By using an Attention-DeepSets based encoder, we ensure that the state representation used in EARL-BO is permu- tation invariant, size invariant, and capable of focusing on the salient information in the dataset. This robust encod- ing strategy is essential for effectively navigating the high- dimensional search space in multi-step lookahead BO tasks.
Figure I. An overview the EARL-BO architecture.
Attention- Deeepees- Encoder Module. Permutation invari- ance is crucial in BO, as the order in which data points is acquired should not affect the learning process (i.e., the
MDP). In other words, given a dataset Dr {(1:; ):)) )==r the order of the data pairs (zi, 34) should not influence the agent's s decision- decision-making process. While meta-learning stud- ies have leveraged transformer models (Chen et al., 2022; Maraval et al., 2024), we begin with the simpler DeepSets architecture (Zaheer et al., 2017), which ensures that the encoding function o applied to the data is invariant under permutations. Mathematically, this is expressed as:
PPO Actor-Critic Module. The core of EARL-BO's Serision-making process is the RL module. We employ PPO, which comprises two main components: the actor network and the critic network.
The actor network, denoted as (alss)) with parameters O, represents the policy that maps states to actions. In our case, it outputs the next query point for the BO process. The actor's s objective is to maximize the expected cumulative reward, (3). The critic network, denoted (a(s) with pa- rameters o, estimates the value function of the current state and helps reduce the variance of policy gradient estimates by providing a baseline for advantage estimation:
((D)) p
Σ (E..)EDD,
(a wi)
(6)
A(st, 0z) = Q(ss, ac) - (o())).
99)
where & is function that maps individual data points to a latent space, and p is a function that aggregates these representations into a fixed-size vector. This aggregation achieves permutation invariance of the final representation.
The PPO algorithm updates the actor network by maximiz- ing the clipped surrogate objective: L (0) from (5). The critic is updated to minimize the mean squared error be- tween its predictions and the observed returns
Another requirement for the encoder is size invariance, which refers to the ability to handle growing dataset D, as samples are acquired. As new data points are added, the encoder must adapt without being explicitly retrained or altered. The DeepSets architecture inherently provides this property by summing over the data representations, allow- ing it to naturally handle datasets of varying sizes. Note that the aggregation in (6) could be replaced with other permutation-ing invariian functions, e.g., arithmetic mean.
L VF (() -E, [(Vo(s)) R.))²
(10)
Initialization with Off-Policy Learning. To accelerate the initial training of EARL-BO, we employ an off off-policy learning strategy to warm-start training of the RL module. As noted above, this step can use samples generated by any existing BO method. In this work, we specifically use TuRBO due to its strong performance and popularity in high- dimensional BO, though the framework could seamlessly integrate the other benchmark methods such as SAASBO, EI, etc. The off off-policy learning process is as follows.
While permutation and size invariance are important, data points may not contribute equally to the recision-making process in BO: some points may be more relevant than others in determining the next query point. We therefore incorporate an attention mechanism (Vaswani al., 2017), which assigns weights to each data point based on its learned relevance, effectively allowing the model to "focus" on the critical information. Note that attention-base -ased architectures have found success in encoding general datasets in other applications (Gordon et al., 2020; Simpson et al., 2021).
First, we generate batch of initial training data using the chosen baseline method (TuRBO in our implementa- tion), D bascline {(s a,, Ts. + based the current dataset Dr. We then use these trajectories to pre-train the actor and critic networks using a modified loss function:
Lpretrain (0, 6) = E E( D'DDDart" PPO (0)
I VF (F(()) (c)H(TT)),, (11)
Mathematically, the attention weights a; for each data point
5

---

EARL-BO: RL for Multi-Step Lookahead BO
where (m)) is the entropy of the policy, encouraging ex- ploration, and C1, 02 tunable weights. We then freeze the initial layers of both the actor and critic networks. This step preserves some pre pre-training knowledge, while allowing for fine-tuning in the subsequent on- on-policy learning phase. Specifically, the parameters are partitioned as:
Algorithm 1 EARL-BO
Input: data Dx. action bounds [lb, ub]
Parameters: lookahead horizonn, max.episodes, up- date episodes, off policy episodes
Output: next query point I++1 Initialize RL agent (PPO agent), encoder network, and memory buffer
Otrozen = 101..... 0x): Buainable = 10x-1:-.. 8n}
Fit GP to D3:
Ofrezen {0),- ..., bb Perninable = { (1x+1, )n))
for k: = 1 to max max_ sexiedess do
where only Otrainable and Otrainable contain parameters up- dated in subsequent (on-policy) training. After the off- policy pre-training phase, we switch on-policy learning using the chosen RL algorithm (PPO in our implementation).
Reset environment state s with D; for step = to ookkahad_horizon do Encode state using encoder network if k S off policy episodes then Select action using TuRBO acquisition else
The (optional) modular off-policy learning step provides EARL-BO with flexible initialization options, leveraging the strengths of existing BO methods while maintaining the adaptability of RL. This design choice allows practitioners integrate EARL-BO with their existing BO infrastructure and domain-specific acquisition functions.
Select action a using RL agent end
Sample ύK+1 ~ N (n Da), (x, (; Dx))
Compute reward T R(Dk: Th+1, Dk+1)) using ÿK++1 Update environment state s' Dx+1
Model-Based RL with GP Virtual Environment. Typical RL algorithms learn policies through direct interaction with the enviromment. However, in the context of BO, where sam- pling is expensive, is impractical and inefficient for an RL agent to directly interact with the real environment to find the optimal policy π². In these settings, it advantageous to incorporate (known or learned) model rather than/in addition to the environment directly, paradigm known as model-based RL (Moerland et al., 2023).
Store transition (s, s', to memory buffer f s'
end for kmod update_episodes. = 0 then if k S off policy episodes then Update RL agent with initial policy else
Calculate PPO loss using memory buffer Train actor, critic, and encoder networks
To address this challenge, we propose using the current trained GP at each step as a 'model' for RL. Specifically, we create a virtual environment based on a GP fitted to the current dataset D;- {a yi) Hz and leam the optimal policy from interacting with this virtual environment model rather than the original black-box function.
end
end if
Clear memory buffer end for
Encode state using final encoder network Return F4+1 output from final actor network
In other words, by taking this model-based approach, EARL- BO assumes that the current GP posterior at each BO step a good approximation of the class of functions we seek to optimize, cf. meta-learnigg strategies that require data from given source domains (Maraval et al., 2024). The RL algorithm is therefore provided with (multi-step looka- head) rewards using samples from the GP posterior as the underlying "black-be box function" environment.
space representation, capturing the complex relationships between previously sampled points and their corresponding values. The RL module then makes decisions based this encoded state, and interacts with the GP model simulate the outcomes of its actions by sampling Uk+11 from the GP posterior distribution.
EARL-BO Summary (Algorithm 1). The EARL-BO al- gorithm implements a hybrid model-based and model-free RL approach, loosely following the Dyna framework (Silver et al., 2008; Wu et al., 2023).
This Dyna-style learning process allows the agent to learn from both real data (as samples are collected along the BO process) and simulated experiences (generated through GP posterior sampling). The algorithm initially learns from off-policy data generated by TuRBO (or another baseline algorithm) to establish a reasonable starting policy, then transitions to on-policy learning using PPO. As more BO samples are collected, the approximation quality the GP model improves, and the simulated experiences become
The algorithm maintains GP that serves the world model, enabling virtual interactions for policy learning without re- quiring expensive function evaluations from the "rrll-world" environment, At each iteration, the Attention- Attention-DeepSets en- coder module transforms the current state Dj into a latent-
6

---

EARL-BO: RL for Multi-Step Lookahead BO
Ackley (2D)
Levy Function (2D)
Rosenbrock Function (2D)
Sumsquare Function (2D)
10
10
Rogret
10
10
30
Iteration
Iteration
lieration
Iteration
EI
Random
TuRBO
Rollout, VR
EARL-BO
SAASBO
Ackley Function (SD)
Levy Function (SD)
Rosenbrock Function (5D)
Sumsquare Function (SD)
10
10
10
Regret
10
10
10
10
lieration
leeaatoon
lieration
Iteration
E1
Random
TuRBO
EARL-BO
SAASBO
Ackley Function (8D)
Levy Function (8D)
Rosenbrock Function (8D)
Sumsquare Function (8D)
10
10'
10
10
Regret
10
10
10
10
1-117
10"
20
30
Iteration
Random
TuRBO
EARL-BO(3)
EARL-BO(S)
SAASBO
Figure 2. Optimization performance of various BO methods on synthetic benchmarks.
increasingly reliable for policy learning. This approach enables efficient exploration and policy improvement, while maintaining the sample efficiency that is crucial for BO.
optimization objective to minimize these functions. We initialize the BO algorithms using 30 random points within the search space and evaluate performance using simple re- gret (yopt - vτ). Each method is tested for ten replications by resampling the initial dataset.
4. Results and Discussion
The results for 2-D, 5-D, and 8-D are shown in Figure 2, where the solid lines represent the mean performance across the ten runs, and the shaded area shows standard devi- ation. For all problems, we compare performance against random search, standard one-step El maximization, TuRBO, and SAASBO. For the 2-D problems, we also compare against rollout with variance reduction, denoted as 'Roll- out_ VR' (Lee al., 2020). This method was impractical run in higher dimensions (see Section 2.1). For Rollout. VR and EARL-BO, To investigate the effects of the lookahead horizon, we fur- ther test EARL-BO with a lookahead horizon of H = 5 on the 8-D benchmark functions.
This section compares the performance of EARL-BO against existing lookahead and high-dimensional BO al- gorithms on both synthetic benchmark functions and real- world hyperparameter optimization tasks. The details of the implementation and experiment are given Appendix A. In particular, while PPO itself contains several hyperparam- eters, we found EARL-BO to perform well consistently and use standard/default values. The exception the learning rate, which be linked the learning rate of the encoder. Appendix B contains an ablation study on learning rates.
select lookahead horizon of I/ 3.
4.1. Synthetic Benchmark Functions
We first evaluate EARL-BO four popular synthetic benchmark functions: Ackley, Levy, Rosenbrock, and Sum Squares (Surjanovic & Bingham, 2013). These functions were tested in 2-D, 5-D, 8-D, and 30-D configurations, with a fixed search space of [-15, 15) for all dimensions. The
The results for the 30-D setting are presented in Figure 3, using the same experimental setup (search space of [- 15,15] for all dimensions, 30 initial data points) the lower- dimensional experiments. For a better visibility of the figure, we excluded random search from the 30-D comparisons,

---

EARL-BO: RL for Multi-Step Lookahead BO
Ackley Function (30D)
Levy Function (30D)
Rosenbrock Funetion (30D)
Sum squares Function (30D)
6-117
1.3-0
18.
1-
10
Rogee
1.3-
10"
122-10
10
11f
10
441
50
0
10
31
1/1
10
20
30
441
10
10
20
Iteration
Iteration
Iteration
Iterasion
EARL-BO
EI-DO
TuRBO
SASSDO
Figure 3. Optimization performance of various BO methods 30-D synthetic benchmarks.
focusing on the more competitive BO methods.
benchmark functions, EARL-BO significantly outperforms EI, TuRBO, and SAASBO, with performance gaps that are substantially larger than those observed in lower dimensions. This pronounced improvement supports our hypothesis that, as problem dimensionality increases, the optimization land- scape becomes more complex and challenging, making non- myopic decision-making increasingly valuable. The ability to plan multiple steps ahead becomes crucial for navigating high- nigh-dimensional spaces effectively, where myopic policies are more likely to become trapped in suboptimal regions or fail to exploit promising areas of the search space efficiently.
Results. As shown in Figure 2, in the 2-D setting, EARL- BO demonstrates competitive performance, often closely matching, slightly improving on, the performance of Rollout_VR. This alignment is intuitive, as both methods implement multi-step lookahead strategies, suggesting that EARL-BO is able to properly learn the three-step lookahead policy. This notable given the relative computational scalability of EARL-BO, as described in Appendix C. For most functions, the performance gap between EARL-BO and single-step lookahead methods such as EI was not pronounced in this oow-dimesiiona setting, likely due to the relatively high information density provided by the 30 initial points in a 2-D space, which may reduce the advantages gained from extended exploration and lookahead strategies. In other words, myopic policies are less detrimental since fewer samples are required.
Notably, even SAASBO, which performs competitively on certain lower- dimensioal problems such as Sum Squares, is consistently outperformed by EARL-BO in the 30-D) setting across all benchmark functions. This suggests that the bene- fits of multi multi--tep lookahead planning become dominant over specialized high-dimensional techniques as the complexity of the optimization problem increases.
A notable pattern emerges with the Sum Squares func- tion across all dimensionalities (2-D, 5-D, and 8-D), where SAASBO consistently demonstrates superior performance compared to all other methods, including EARL-BO. This result also intuitive, given the additive nature of the black- box objective. The performance highlights SAASBO's par- ticular effectiveness on smooth, unimodal functions, where its sparse axis- alignel approach can efficiently navigate the search space. However, for functions with more compli- cated landscapes featuring multiple local optima or irregular surfaces, the relative performance changes significantly.
Regarding lookahead horizons, the three-step lookahead variant of EARL-BO shows similar or slightly better per- formance compared to its five-step counterpart in the 8-D experiments. This finding aligns with empirical observa- tions from previous multi-step lookahead studies (Lam et al., 2016; Lee et al., 2020), which report optimal performance at intermediate horizons of H 3, 4 steps compared to H = 5, 6. A more comprehensive study on more lookahead horizon lengths can be found in Appendix B.
The enhanced performance of EARL-B in higher dimen- sions may be attributed to several factors:
In the 5-D and 8-D settings, FARL-BO's advantages be- come increasingly evident on the more complex functions. Across the Ackley, Levy, and Rosenbrock functions, EARL- BO consistently outperforms baseline methods including EL, Random Search, and TuRBO. This trend underscores the effectiveness of our multi-step lookahead approach in navi- gating more complicated functions and dimees-aimen landscapes (relative to other lookahead methods).
For complex and nigher-dimensional functions, the ability to plan mul- tiple steps ahead becomes more valuable for effective exploration and exploitation balance.
Increased exploration benefits:
Efficient use of information: EARL-BO's Attention- DeepSets encoder better captures and exploits relation- ships between more informed Recision-making
The 30-D experimental results, shown in Figure 3, demonstrate FARL-BO's outstanding performance in high- dimensional optimization settings. Across all four 30-D
samples, enabling
8

---

EARL-BO: RL for Multi-Step Lookahead BO
Search Space ID 5889 (6D)
Scarch Space ID 5968 (8D)
Scurch Space ID 7200 (19D)
10
30"
Regret
UCB
Regeet
UCB
Ragest
30'
TKB
26
EARL-BO
EARI-HO
EARL-BO
Tult5o
TaltDO
SAAS500
SAJS500
SAA580
10
20
30
40
50
10
30
40
50
10
20
10
40
beration
Demiton
Beertion
Figure 4. Optimization performance of various BO methods on HPO problems.
In addition the aforementioned heuristic observations, the plateauing, or slight degradation performance with increased lookahead (from three to five in 8-D), may also be attributed to errors in the GP model, a phenomenon we term "planning delusion." " As EARL-BO does not interact with the true objective function, but rather with samples from the GP model (which then interacts with the true objective function), looking many steps ahead may compound errors in the predicted outcomes. In essence, the true objective function may not correspond exactly to a GP posterior sam- ple, reducing the effectiveness of model-based learning. The three-step variant seemingly allows for meaningful planning without excessive reliance on imperfect models.
dataset. We compare performance against random search, standard El and PI maximization, TuRBO, and SAASBO.
For EARL-BO, select a lookahead horizon of H = 3.
Results. As shown in Figure 4, EARL-BO consistently outperforms the baseline methods on the HPO tasks, Ap- pendix visualizes these results with non- non-logarithmic seal- ing and contains an ablation experiment for the 19-D prob- lem starting from only five initial samples. For all problems, EARL-BO performs equivalently (or worse than) competing methods early iterations, and then overtakes them quickly later, including TuRBO and SAASBO. This suggests that single-step methods are indeed overly myopic, and thus fo- cused on initial rewards; the multi-step lookahead enables EARL-BO to achieve the best long-term performance.
We speculate that this "planning delusion" effect could be more pronounced in EARL-BO compared to traditional rollout-based methods. FARL-BO leamns policy that di- rectly optimizes for multi-step performance, potentially making it more susceptible to biases in long-term predic- tions. In contrast, rollout methods typically use myopic policies as base strategies, which may provide some inher- ent robustness against long-horizon planning errors.
In the 19-D Search Space (ID 7200) setting with only five initial points (Appendíx B), EARL-BO performs similarly to some of the baseline methods. In this case, the initial data are extremely sparse, leading to a poor GP model initially, and as a result, large variation in samples from the GP posterior. In this scenario, EARL-BO suffers again from model errors, and adopting its multi-step lookahead policy may not provide the expected benefits and could potentially lead to suboptimal decisions due to errors in long-term predictions.
Further results on the synthetic benchmarks, including abla- tion studies on learning rates, planning delusion, RL training convergence, and permutation invariance, are given in Ap- pendix B. Computational requirements and scalability are discussed in Appendix C.
5. Conclusions
4.2. Hyperparameter Optimization Experiments
This paper introduces EARL-BO, a novel Bayesian Opti- mization approach based on Encoder- -uggeee- Reinforce- ment Learning for multi-step lookahead in high-dimensional spaces. The approach combines Attention- Attention-Deepsets en- coder module with model-based actor-critic RI. to provide an end-to-end solution for the sequential decision- process of BO. Our experiments on synthetic benchmarks and hyperparameter optimization tasks demonstrate EARL- BO's superior performance in moderate high-dimensional spaces compared traditional and state-of-the-art methods. Overall, this study demonstrates the potential of reinforce- ment learning in handling the sequential nature of BO.
We next evaluate EARL-BO in real-world scenarios using the Hyperparameter Optimization Benchmarks (HPO-B) dataset (Arango et al., 2021). HPO-B contains datasets of classification model hyperparameters and their correspond- ing accuracies across multiple types and search spaces, pro- viding realistic testbed for optimization algorithms. We focus on nigher-dimensioaal optimization problems, and select search space IDs 5889 (6-D), 5968 (8-D), and 7200 (19-D) as test problems. We initialize the BO algorithms using five random points for 6- and 8-D and 50 for the 19-D problem. We again evaluate performance using simple re- gret (Jopt - v and ten replications by resampling the initial
9

---

EARL-BO: RL for Multi-Step Lookahead BO
Acknowledgements
Chen, Y., Song, X., Lee, C., Wang. Z., Zhang, R., Dohan, D., Kawakami, K., Kochanski, G., Doucet, A., Ranzato, M., et al. Towards learning universal hyperparameter optimiz- ers with transformers. Advances in Neural Information Processing Systems, 35:32053-32068 2022.
The authors thank Dr Mark van der Wilk for discussions and insightful feedback, especially in the conceptualization of this project. CT gratefully acknowledges support from BASF/Royal Academy of Engineering Senior Research Fellowship. This work was partially supported by the Basic Science research program through the National Research Foundation of Korea (NRF) funded by the Ministry of Sci- ence, ICT & Future Planning under the contract No. 2021RIC1C101200 as well as USC ECET"s funding under CYPRES award as part of the distribution of a settlement relating to fuel economy for gasoline- -owwered vehicles.
Cheon, M., Byun, H., and Lee, J. H. Non-myopic Bayesian optimization using model-free reinforcement learning and its application to optimization in electrochemistry. Computers & Chemical Engineering, 184:108624, 2024.
Friksson, D. and Jankowiak, M. High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Un- certainty in Artificial Intelligence, pp. 493-503. PMLR, 2021.
Impact Statement
Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., and Poloczek, M. Scalable global optimization via local Bayesian optimization. Advances in Neural Information Processing Systems, 32, 2019.
This paper presents work whose goal is to advance the field of Machine Learning through improved Bayesian optimiza- tion techniques. The potential societal consequences align with those generally associated with advancing machine learning methodologies.
Folch, J. P., Zhang, S., Lee, R., Shafei, B., Walz. D., Tsay, C., van der Wilk, M., and Misener, R. SnAKe: Bayesian optimization with pathwise exploration. Advances in Neural Information Processing Systems, 35, 2022.
References
Ament, S., Daulton, S., Eriksson, D., Balandat, M., and Bakshy, E. Unexpected improvements to expected im- provement for bayesian optimization. Advances in Neural Information Processing Systems, 36:20577-20612, 2023.
Ginsbourger, D. and Le Riche, R. Towards Gaussian process-based optimization with finite time horizon. In International Workshop in
Design and Analysis, pp. 89-96. Springer, 2010.
Ao, Z. and Li, J. On estimating the gradient of the expected information gain in Bayesian experimental design. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 2031 1 1-20319, 2024.
Gómez-Boobarelli, R., Wei, J. N., Duvenaud, D., Hermández-Lobato, J. M., Sánchez-Lengeling B., She- berla, D., Aguilera- Iparraguirre, J., Hirzel, T. D., Adams, R. P., and Aspuru-Guzik, A. Automatic chemical de- sign using a data-driven continuous representation of molecules. ACS Central Science, 4(2):268-276, 2018.
Arango, S. P., Jomaa, H. S., Wistuba, M., and Grabocka, J. Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml. arXiv preprint arXiv:2106.06257, 2021.
González, J., Osborne, M., and Lawrence, N. GLASSES: Relieving the myopia of Bayesian optimisation. In Ar- tificial Inteligence and Statistics, pp. 790-799. PMLR, 2016.
Brochu, E., Cora, V. M., and De Freitas, N. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning, arXiv preprint arXiv: 1012.20/ 2010.
Gordon, J., Bruinsma, W. P., Foong, A. Y., Requeima, J., Dubois, Y., and Turner, R. E. Convolutional conditional neural processes. In International Conference on Learn- ing Representations, 2020.
Chen, T., Duan, Y., Li, D., Qi, L., Shi, Y., and Gao, Y. PG- LBO: Enhancing nigh-dimensional bayesian optimiza- tion with pseudo-label and Gaussian process guidance. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 1 11381-11323 2024.
Hoang, T. N., Hoang. Q. M., Ouyang, R., and Low, K. H. Decentralized nigh-dimensional Bayesian optimization with factor graphs. In Proceedings of the AAAI Confer- ence Artificial Intelligence, volume 32, 2018.
Chen, Y., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Lillicrap. T. P.. Botvinick, M., and Freitas, N. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning (ICML), pp. 748-756. PMLR, 2017.
Hsich, B.-J., Hsich, P.-.,, and Liu, X. Reinforced few-shot acquisition function learning for Bayesian optimization. Advances in Neural Information Processing Systems, 34: 7718-7731, 2021.
10

---

EARL-BO: RL for Multi-Step Lookahead BO
Jones, D. R., Schonlau, M., and Welch, J. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455-492, 1998.
Shahriari, B., Swersky, K., Wang. Z., Adams, R. P., and De Freitas, N. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, (04(1):148-175 2015.
Kushner, H. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering. 86(1):97-106, 1964.
Shmakov, A., Naug, A., Gundecha, V., Ghorbanpour, S., Gutierrez, R. L., Babu, A. R., Guillen, A., and Sarkar, S. Rtdk-bo: High dimensional Bayesian optimization with reinforced transformer deep kernels. In 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pp. 1-8. IEEE, 2023.
Lam, R., Willcox, K., and Wolpert, D. H. Bayesian opti- mization with a finite budget: An approximate dynamic programming approach. Advances in Neural Information Processing Systems, 29, 2016.
Silver, D., Sutton, R. S., and Müller, M. Sample-based learning and search with permanent and transient memo- ries. In International Conference on Machine Learning (ICML), pp. 968-975, 2008.
Lee, E., Eriksson, D., Bindel, D., Cheng, B., and Mccourt, M. Efficient rollout strategies for Bayesian optimization. In Conference on Uncertainty in Artificial Intelligence, pp. 260-269, PMLR, 2020.
Simpson, F., Davies, I., Lalchand, V., Vullo, A., Durrande, N., and Rasmussen, C. E. Kernel identification through transformers. Advances in Neural Information Processing Systems, 34:10483-1049, 2021.
Li, X., Wu, K., Zhang. X., Wang, H., Liu, J., et al. Pretrained optimization model for zero-shot black box optimization. Advances in Neural Information Processing Systems, 37: 14283-14324, 2024.
Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2012.
Ma, Z., Guo, H., Gong, Y.J.,, Zhang, J., and Tan, K. To- ward automated algorithm design: A survey and practical IEEE Transac-
guide to wwwp-bllackooooopiiiiaeeee
tions on Evolutionary Computation, 2025.
Song, L.,, Gao, C.,, Xue, K., Wu, Li, D.,, Hao, J., Zhang, Z., and Qian, C. Reinforced in-context black-box opti- mization. arXiv preprint arXiv:2402. 17423, 2024.
Maraval, A., Zimmer, M., Grosnit, A., and Bou Ammar, H. End-to-end meta-Bayesian optimisation with trans- former neural processes. Advances in Neural Information Processing Systems, 36, 2024.
Srinivas, N., Krause, A., Kakade, S., and Seeger, M. Gaus- sian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), Pp. 1015-1022, 2010.
Moerland, M., Brockens, J., Plaat, A., Jonker, C. M., et al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):
Surjanovic, S. and Bingham, D. Virtual library of simulation experiments: test functions and datasets. Simon Fraser University, Burnaby, BC, Canada, accessed May, 13: 2015, 2013.
1-118, 2023.
Muratore, F,, Filers, C., Gienger, M., and Peters, J. Data- efficient domain randomization with Bayesian optimiza- tion. IEEE Robotics and Automation Letters, 6(2):911- 918. 2021.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Osborne, M. A., Garnett, R., and Roberts, S. J. Gaussian processes for global optimization. 2009.
Thebelt, A., Tsay, C., Lee, R. M., Sudermann- N., Walz, D., Tranter, T., and Misener, R. Multi-objective constrained optimization for energy applications via tree ensembles. Applied Energy, 306:1 18061, 2022.
Paulson, J. A. and Tsay, C. Bayesian optimization as flexible and efficient design framework for sustainable process systems. Current Opinion in Green and Sustain- able Chemistry, pp. 100983, 2024.
Tripp. A., Daxberger, E., and Hernández-Lobicd J. M. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33:1 1259- 1 11272, 2020.
Puterman, M. I. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Atten- tion is all you need. Advances in Neural Information Processing Systems, 30:5998-660 6008, 2017.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint reXiiv/707.06e 2017.

---

EARL-BO: RL for Multi-Step Lookahead BO
Volpp, M., Fröhlich, L. P., Fischer, K., Doerr, A., Falkner, S., Hutter, F., and Daniel, C. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations, 2019.
Wang, X., Jin, Y., Schmitt, S., and Olhofer, M. Recent advances in Bayesian optimization. ACM Computing Surveys, 55(13s):1-36 2023.
Williams, C. K. and Rasmussen, E. Gaussian processes for machine learning, volume MIT press Cambridge, MA, 2006.
Wu, G., Fang, W., Wang, J., Ge, P., Cao, J., Ping, Y., and Gou, Dyna-PPO reinforcement learning with gaus- sian process for the continuous action decision-making in autonomous driving. Applied Intelligence, 53(13):16893 16907, 2023.
Wu, J. and Frazier, Practical two-step lookahead Bayesian optimization. Advances in Neural Information Processing Systems, 32, 2019.
Xie, Y. Zhang, S., Paulson, J., and Tsay, C. Global opti- mization of gaussian process acquisition functions using piccewise-lineee kernel approximation. arXiv preprint 1xXi:22410.16893 2024.
Yang. A. X., Aitchison, L., and Moss, H. MON- GOOSE: Path-wise smooth bayesian optimisation via meta-learning. ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R., and Smola, J. Deep sets. Ad- vances in Neural Information Processing Systems, 30, 2017.
12

---

EARL-BO: RL for Multi-Step Lookahead BO
A. Experimental Setup
In this section, we provide details of the experimental setup, focusing on the dataset, its preprocessing, the hyperparameters of RL, and the hardware used for experiments aid in reproducibility.
A.1. Tasks
The real-world Hyperparameter Optimization dataset is sourced from the HPO-B dataset (Arango et al., 2021), a collection of HPO datasets grouped by search space and tasks. Each search space ID refers to the set of hyperparameters for specific machine learning (ML) model, e.g. Ranger, XGBoost. Table 5 of (Arango al., 2021) gives the details for each search space ID, including number of evaluations, number of datasets, number of dimensions, and the name of search space (i.e., the name the number of combined datasets, some search spaces contain duplicate measurements (i.e.
of ML model). Depending multiple/differeet response recorded for the exact same input values). To mitigate this issue, (Arango et al., 2021) randomly select one data sample for each input when duplicates exist; however, in this research, we only use search space IDs which do not contain data duplication for cleaner BO comparisons. All in all, due to our research topic in high-dimennsioal problems and for the clean-data, we have selected search space IDs of 5889 (6-D), 5968 (8-D), and 7200 (19-D).
A.2. Baselines
For baseline optimization performance on the synthetic benchmark functions, we compare against EI, Random, TuRBO, and Rollout. VR in this work. For EI, Random, and Rollout VR, we use implementations from (Lee et al., 2020) found at: release. For TuRBO, we use the current implementation
https: / //g/tttuuuu com/ wwwarrchhsseeeeooaaeaeeeee from the Uber research group: https: /tte:////ttttt www./o/eooreeeeeeeeeeee research TuRBO. To keep experiments consistent and reproducible, optimization was performed in the same search space with same initial data for each problem repetition. Note that we found TuRBO can give slightly different optimization performance even with the same set of initial data. Ten sets of initial data were provided to optimization methods, and the average performance is reported in this paper.
A.3. EARL-BO
On top of the pseudo code (i.e., Algorithm 1) presented in the main text, EARL-BO contains one additional detail -imilly the stopping criterion for RL training. For this, if the PPO agent fails to learn policy over some episodes (e.g., average reward becomes zero for 15 consecutive updates) or the final trained agent's average reward is less than 1e-5, EARL-BO aborts the learned policy and returns the off-policy suggestion provided by the TuRBO algorithm. We found this to happen in practice only seldom, particularly when EARL-BO had already discovered the optimum point.
Hyperparameters. Table displays comprehensive list of hyperparameters for EARL-BO. We would like underline that none of these presented values for hyperparameters were tuned across problems. In other words, across various dimensions and function forms, we have kept the same hyperparameters with most basic PPO and Encoder values. However, setting learning rates for the RL and encoder modules (their relative magnitudes in particular) was a very important question. We employed different learning rates for the RL agent (0.001) and the encoder (0.01). This design choice is justified by the distinct roles and complexities of these components within the algorithm, allowing them to train in dynamically decoupled fashion. The RL agent, tasked with learning a policy and value function in dynamic environment, benefits from lower learning rate to maintain stability and prevent performance collapse due rapid policy changes. Conversely, the encoder, which learns static representation of the search space, can adapt more quickly with a higher learning rate, improving the quality of state representations rapidly. As an ablation study below, we present EARL-BO with 4 different selections of learning rates and compare the optimization performance.
A.4. Hardware
We conducted our experiments on computing server with AMD EPYC 7742 processors equipped. The specific allocation for each job was as follows: 16 CPUs and max memory of 100 GB. With this configuration, the average time required to complete each experiment was approximately 25 hours. This configuration was chosen in order to parallelize the multiple repetitions for each experiment. Notably, experiments could potentially be sped up using GPU acceleration.
13

---

EARL-BO: RL for Multi-Step Lookahead BO
Table 1. EARL-BO hyperparameter values.
PPO
Learning rate epochs
0.001
100
Epsilon clip e
0.2
values for Adam
(0.9, 0.999)
Discount factor y
0.95
Value function coefficient
Entropy coefficient
0.1
# layers frozen
Max episodes
4000
Update frequency
50
off-policy episodes
400
No-improvement threshold
15
Horizon
Encoder
Hidden dimension
64
Output dimension
16
Learning rate
0.01
GP
Kemel
RBF + WhiteKernel
RBF length-scale bounds
(le-2, 1c2)
Noise bounds
(Ie-10, 1el)
B. Additional Results
In this section, we describe several ablation studies to characterize the proposed EARL-BO algorithm. Specifically, should the relative learning rate of RL be slower than that of the encoder? Does "planning delusion" occur only when the lookahead horizon is long? How stable is the RL training across BO iterations? Is permutation invariance important?
What is the effect of different learning rates? Figure Al shows two different cases for 8-D optimization of the Ackley function: first, when the learning rates for the RL and encoder modules are, respectively, (0.001, 0.01). This configuration denoted as 'standard." We also reverse the learning rates to (0.01, 0.001), respectively, which indicated as 'reverse." Figure A1 I shows that having higher learning rate for the RL module leads to both unstable BO performance (as we can observe the increasing standard deviation over time), but also lower optimization performance.
Ablation Study: Ackley Function (8D)
Rogret
6x10
EARL-BE (ALLL-BOtttnndad)
EARL (Btevere))))
4x 10
40
Iteration
Figure A1. Optimization performance of EARL-BO with different relative learning rates.
When can planning delusion happen? In the Results and Discussion section of the main text, we discuss that "planning delusion" may happen if the lookahead horizon for RL is large, due to high uncertainty of the GP virtual environment. Astute readers will notice that, if the high-uncertainty of the GP is the source this planning delusion, having extremely scarce data for high-dimensional optimization might also cause the planning delusion even when the lookahead horizon
14

---

EARL-BO: RL for Multi-Step Lookahead BO
is small. Figure A2 repeats the HPO-B search space ID 7200 (19-D) problem with only five initial samples instead of 50. The results confirm that, if we employ EARL-BO in high-dimensional search spaces with sparse initial data, it may indeed underperform compared to other BO methods, suffering from planning delusion.
Search Space ID 7200 (19D)
0.4
0.3
EARL-BO
Regret
0.2
0.1
0.0
Iteraion
Figure A2. Optimization performance of methods HPO-B search space ID 7200 with 5 initial samples.
Additional Figures. Figure 3 in the main text might suggest that the standard deviations of performance do not decline as BO progresses for the HPO-B dataset. This can be attributed to the logarithmic y-axis scale in Figure 3. Thus, we include here the same graph with a linear y-axis scale. Figure A3 more easily visualizes that the standard deviation of regrets in all the BO methods are decreasing over as BO progresses. More specifically, in most of cases, EARL-BO displays the smallest standard deviation after approximately eight iterations among all compared BO methods. This suggests that EARL-BO exhibits relatively stable performance regardless of initial point distribution.
Search Space ID 5889 (6D)
Search Space ID 5968 (SD)
Search Space ID 7200 (19D)
0.15
LCB
0.15
E1
P1
0,10
T,R50
BARI-BO
Begget
02-
EARI-90
Regnet
0.10
EARE-BO
Sax.BO
Saa30
SAASBO
0,65
0.1
0.05
60-
0.00
Iteration
Iteration
Deeration
Figure A3. Performance of various BO methods on henchmark functions with non-logarithmic y-axis.
How stable is the RL training across BO iterations? Figure A4 presents the RL training curves at several BO steps (Ist, 11th, and 21st) during the optimization of the 8-D benchmark functions. It suggests two possible characteristics of EARL-BO. First, the RL training appears stable across all BO iterations, evidenced by the smooth convergence of the loss function. Second, we observe that the RL training may become slightly easier as BO progresses-the the loss functions for the 11th and 21st BO steps start at and converge to lower values compared to the first step. This suggests that as EARL-BO accumulates more information about the optimization landscape, the RL agent may more easily learn effective decision-making policies, possibly also due to the improved quality of the learned encoded representations.
Step Ackley Function (80)
BO Step Losse: Liny Function
Step
Step Lomes: square
1.7
20
30
Auwage Vave
Suy
1.5
1,5
1.45
125
20
Trainne
Trenane
Figure A4 Convergence of RL training at various BO iterations.
15

---

EARL-BO: RL for Multi-Step Lookahead BO
Is permutation invariance important? Figure A5 presents an ablation study investigating the importance of permutation
invariance in the encoder design. We compare EARL-BO with encoder architectures the 81) Ackley function:
Acidey Function Ablation Study (SD)
BAIL--OC Seep)
EARL 30X52ep)
10.0
10
50
Isention
Figure A5. Performance of EARL-BO with different encoders.
The two tested architectures are both attention-based, but differ in that one includes the DeepSets architecture for permutation invariance, while the other is 'attention-only" We evaluate both designs using three- and five-step lookahead horizons. The results show that EARL-BO with the Attention- Attention-DeepSets encoder consistently outperforms the attention-only variant across both lookahead settings. This validates our architectural choice and confirms that explicit permutation invariance beneficial for BO making,, intuitive since the order of data acquisition should not affect the learning process.
Effect of different lookahead horizons Figure A6 shows the optimization performance of EARL-BO across different lookahead horizons (1, 3, 5, and 7 steps) on the 8D Ackley function.
EARL-BO Lookshead Horizon Ablation Study Ackley)
(1-9op)
EARL-BC BLL-BB0/ (3 -appp
EARL -B0 (5-eup)
(7-50p
16
k
4-10
19
50
Bention
Figure A6. Performance of EARL-BO with different lookahead horizons.
The one-step lookahead variant demonstrates strong initial performance but exhibits the worst optimization results by the end, consistent with the intuition that myopic behavior prioritizes immediate gains over long-term exploration. The three-step and five-step lookahead variants achieve the best overall performance. The seven-step variant exhibits generally poor performance throughout most iterations, though with some improvement toward the end. These results align with empirical observations from previous multi-step lookahead studies (Lam et al., 2016; Lee al., 2020), which report optimal performance using intermediate lengths for lookahead horizon.
C. Computational Cost Analysis
While EARL-BO demonstrates superior optimization performance on most test problems, computational efficiency represents an important practical consideration. In this section, we provide a detailed analysis of the computational overheads associated with different BO methods.
C.1. Runtime Comparison on 8D Ackley Function
Table 2 presents the computational cost results for various BO methods on the 8D Ackley function. All given times represent the average runtime (i.e., time to compute the next sample location) per BO iteration across 10 independent runs.
16

---

EARL-BO: RL for Multi-Step Lookahead BO
Table 2. Computational cost comparison on 8D Ackley function.
Method
Average Runtime (s)
Std Dev (s)
Notes
EI
0.28
0.05
TuRBO SAASBO
0.27
0.20
168.6
27.6
Rollout (3-step)
>3600
1000 MC iterations
EARL-BO (3-step)
840
147
4000 episodes
EARL-BO (5-step)
1075
134
4000 episodes
These computational cost results reveal several important insights:
Myopic vs. non-myopic methods: As expected, myopic methods (EI, TuRBO) exhibit significantly faster per-iteration computation times compared to multi-step lookahead approaches. However, SAASBO, despite being a myopic method, requires substantially more computation time due to its sparse axis-aligned subspace optimization.
EARL-BO rollout methods: EARL-BO achieves multi-step lookahead performance with considerably less compu- tational overhead than traditional rollout- based methods. The rollout-based method with variance reduction exceeds our 3600-second timeout threshold even with only 1000 Monte Carlo iterations, while EARL-BO completes its 3-step lookahead training in approximately 840 seconds. Note that significantly more MC iterations may be required for good performance in higher dimensional settings or longer lookahead horizons.
Horizon effects: The computational cost scales moderately with increased lookahead horizon (cf. rollout), with 5-step EARL-BO requiring only 28% more time than the 3-step variant.
C.2. Scalability Higher Dimensions
The results on the 30-dimensional Ackley function further demonstrate the computational scalability of EARL-BO. In particular, the computational costs here highlight the relative efficiency of our framework:
Rollout methods would require exponentially more Monte Carlo iterations to handle the expanded search space, making them computationally prohibitive.
EARL-BO runs in 1600 seconds for 30D problems, representing only 100% increase compared 8D optimization.
Myopic methods El and TuRBO require 533% and 215% more CPU time, respectively, when scaling from 8D to 30D.
These results demonstrate that EARL-BO's computational scaling is more favorable than both traditional rollout methods and some myopic approaches when moving to higher-dimenniooall problems.
C.3. Practical Considerations
While multi-step lookahead methods inevitably require more computation time compared myopic approaches, several factors justify this overhead in practical BO applications:
Expensive function evaluations: BO is typically applied to problems where individual function evaluations costly (e.g., materials discovery, hyperparameter tuning of large models, chemical laboratory experiments). In these settings, single point evaluation can require hours or days, making the BO overhead relatively insignificant.
Scalability advantages: Unlike rollout-based methods that become computationally intractable in higher dimensions, EARL--BO maintains reasonable computational requirements while sealing to practically relevant dimensionalities.
The computational analysis confirms that while EARL-BO introduces additional overhead compared to myopic methods, its superior optimization performance and favorable scaling properties make it particularly suitable for expensive black-box optimization problems where the evaluation costs dominate the total optimization budget.
17


Reviews:
[{"review_id": "IPaDTG2zAO", "reviewer": "Reviewer_euH5", "summary": "The paper introduces EARL-BO, a novel reinforcement learning (RL)-based framework for multi-step lookahead Bayesian Optimization (BO) in high-dimensional black-box optimization problems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review and valuable suggestions for improving our work. Your feedback is extremely helpful, and we are grateful for your careful consideration of our work.\n\n### **Planning Delusion and Uncertainty-Aware Mechanisms**\n\nRegarding 'planning delusion,' or the effect of over-reliance on GP with high uncertainty, we agree this is an important component that warrants further exploration. We believe planning delusion occurs because EARL-BO repeatedly samples from the GP model, adds them to the dataset, updates the GP, and so on. This iterative process leads to increasingly less reliable GP models and RL environments further into the lookahead horizon. This explains the sub-optimal performance with excessively long lookahead horizons.\n\nWe will include additional experiments testing 1 (i.e., standard myopic), 3, 5, and 7-step lookahead EARL-BO variants on the benchmark functions. If our hypothesis is correct, we expect the smaller lookahead horizons to perform better at earlier iterations, when the initial GP is more uncertain, but longer lookahead horizons to dominate at later iterations, when the initial GP is more representative of the black-box function. Our preliminary findings on the 8D Ackley function indeed follow this pattern: in earlier iterations, the 3-step lookahead method performs best; however, as optimization progresses, the 5-step lookahead method shows superior performance. The 7-step lookahead method performs worse consistently, suggesting the GP environment never reaches an accuracy amenable to such a long lookahead horizon. We will include and expand on these ablation study results in a revised version.\n\nIncorporating uncertainty-aware lookahead mechanisms is an excellent idea for future expansion. We agree that real and virtual data should be treated with different levels of trust. Uncertainty quantification would help mitigate planning delusion by accounting for the reliability of predictions, e.g., with a robust BO approach. Additionally, an adaptive horizon approach (using longer lookahead when uncertainty is low) could be a promising direction for future research.\n\n### **Computational Cost Analysis**\n\nWe agree that runtime comparisons are also a crucial consideration that warrant detailed investigation. We provide computational cost results for the 8D Ackley function below:\n\n| Method | Average Runtime (s) | Std Dev (s) | Notes |\n| --- | --- | --- | --- |\n| EI | 0.28 | 0.05 | - |\n| TuRBO | 0.27 | 0.20 | - |\n| SAASBO | 168.6 | 27.6 | - |\n| Rollout_EI (3-step) | >3600 | - | 1000 MC iterations |\n| EARL-BO (3-step) | 840 | 147 | 4000 episodes |\n| EARL-BO (5-step) | 1075 | 134 | 4000 episodes |\n\nWhile multi-step lookahead methods inevitably require more computation time, we note that Bayesian optimization is typically applied to problems where function evaluations are expensive (e.g., materials discovery, hyperparameter tuning of large models, etc.). Nevertheless, it is important to highlight these performance tradeoffs, and we will include a comprehensive table of CPU times in the final version.\n\n### **Performance with Poorly Trained GPs**\n\nWe assume that poorly trained GPs affect all BO methods. However, we hypothesize the effects are more pronounced for multi-step lookahead methods, because looking multiple steps ahead via an untrustworthy environment may cause sub-optimal performance. However, as shown in our appendix experiment with the 19D optimization problem, using only 5 initial points (creating a challenging sparse data scenario, which can lead to poor choice of hyperparameters), EARL-BO initially struggles but eventually finds the global optimum faster than traditional methods such as EI, PI, UCB, and TuRBO. This demonstrates the robustness of our approach even in these challenging scenarios.\n\n### **Alternative RL Methods**\n\nThe question about alternative RL methods is rather insightful. In fact, off-policy methods might offer sample efficiency advantages in certain scenarios. As our intention is to present EARL-BO as a novel framework, rather than to tune to a particular application, we select PPO due to its well-documented stability, robustness to hyperparameters, and strong empirical performance in continuous action space tasks similar to our application. We will emphasize that the EARL-BO framework is agnostic to the choice of RL method, and the optimal design of an RL method represents an interesting direction for future research that could yield further improvements.\n\nWe appreciate your constructive feedback and will incorporate your suggestions to strengthen the paper.", "claims_and_evidence": "The paper makes several key claims:\n- EARL-BO improves multi-step lookahead BO performance in high-dimensional settings.\n  - Evidence: Experiments on synthetic and real-world HPO tasks show that EARL-BO outperforms single-step BO methods and achieves comparable or better results than rollout-based BO in low dimensions.\n- EARL-BO is scalable to high-dimensional problems.\n  - Evidence: The use of an Attention-DeepSets encoder and RL-based optimization allows it to handle up to 19D problems, unlike traditional rollout-based methods which are limited to low-dimensional cases.\n- EARL-BO mitigates myopic behavior in BO.\n  - Evidence: Experimental results demonstrate that multi-step lookahead strategies improve long-term optimization results, particularly in complex and higher-dimensional search spaces.\n- The GP-based virtual environment enables efficient learning.\n  - Evidence: The model-based RL approach significantly improves sample efficiency, reducing reliance on expensive function evaluations.\n\nWhile these claims are largely supported, some concerns include:\n- Planning Delusion: The paper acknowledges that EARL-BO can suffer from poor decision-making when the GP posterior has high uncertainty, especially in high-dimensional settings with sparse data.\n- Computational Cost: Although EARL-BO is scalable, the training overhead of RL-based BO is not extensively compared to other methods.", "methods_and_evaluation_criteria": "The proposed methods are well-suited for multi-step lookahead BO and high-dimensional black-box optimization. The evaluation criteria include:\n- Synthetic benchmark functions (Ackley, Levy, Rosenbrock, Sum Squares) tested in 2D, 5D, and 8D.\n- Real-world HPO tasks (HPO-B dataset) with search spaces 6D, 8D, and 19D.\n- Comparison against state-of-the-art BO methods including EI, TuRBO, SAASBO, and rollout-based methods.\n\nThe benchmarks and criteria are appropriate, but additional details on computational efficiency and runtime analysis would strengthen the evaluation.", "theoretical_claims": "The paper formulates BO as a finite-horizon Markov Decision Process (MDP) and proposes an RL-based solution. The theoretical framework is sound, leveraging:\n- Dynamic programming to justify multi-step optimization.\n- Attention-DeepSets encoders for permutation invariance.\n- PPO-based RL policy optimization.\n\nThe theoretical justification appears correct, though more formal convergence guarantees for the RL policy would be beneficial.", "experimental_designs_or_analyses": "The experiments are well-structured and include:\n- Multiple replications (10 runs per experiment).\n- Ablation studies (e.g., lookahead horizon effects, learning rate sensitivity).\n- Comparisons against strong baselines.\n\nHowever, there are some limitations:\n- Lack of runtime comparisons: How does EARL-BO’s computational cost compare to TuRBO or SAASBO?\n- Planning Delusion Analysis: The effect of high uncertainty in GP modeling on EARL-BO's decision-making could be explored further.", "relation_to_broader_scientific_literature": "The paper is well-situated within the Bayesian optimization and reinforcement learning literature. The connections are well-documented, but a deeper discussion on the trade-offs between model-based vs. model-free RL approaches in BO would be valuable.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- Innovative integration of RL with BO for multi-step lookahead.\n- Scalability to high-dimensional problems.\n- Strong empirical validation across synthetic and real-world tasks.\n- Comprehensive ablation studies.\n\nWeaknesses:\n- Computational cost is not analyzed in depth.\n- Limited discussion on theoretical RL convergence properties.", "other_comments_or_suggestions": "- Provide a runtime comparison with existing BO methods.\n- Analyze how EARL-BO scales computationally with increasing dimensions.\n- Explore potential improvements to mitigate planning delusion (e.g., uncertainty-aware lookahead adjustments).", "questions_for_authors": "- How does EARL-BO’s training cost compare to TuRBO or SAASBO?\n  - A direct runtime comparison would clarify its practical applicability.\n- Would an uncertainty-aware lookahead mechanism help mitigate planning delusion?\n  - Given that long-horizon lookahead can lead to compounding GP errors, would incorporating uncertainty quantification help?\n- How does EARL-BO perform when initialized with a poorly trained GP?\n  - If the GP model is unreliable in early iterations, does EARL-BO suffer significant performance degradation?\n- Could alternative RL methods (e.g., model-free RL) be competitive?\n  - PPO is used, but would Q-learning or offline RL approaches provide benefits?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jzxAlzNRMM", "reviewer": "Reviewer_JCQV", "summary": "The paper proposes a new framework for combining RL (via meta-learning) with BO to provide effective optimisation. They propose a NN architecture that deals well with the sequential nature of data-collecting. For each BO step, off-line policy learning attempts to learn how to mimic TURBO, and then model-based RL is used to fine-tune the policy. Unlike exisiting RL-based optimisers, the RL agent is trained fro scratch from each BO step allowing a higher level of specificity. In particular, they focus on the task of non-myopic optimisation, providing empirical results.\n\n## update after rebuttal\n\nThanks for your convincing replies to the rebuttal. I maintain my already high score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and positive feedback on our work. We appreciate the time taken to understand our contribution and the valuable suggestions to strengthen the paper.\n\n### **Regarding the \"planning delusion\" concept:**\n\nWe will conduct further experimentation on this concept as suggested. The text below is repeated from our response to Reviewer euH5:\n\nRegarding 'planning delusion,' or the effect of over-reliance on GP with high uncertainty, we agree this is an important component that warrants further exploration. We believe planning delusion occurs because EARL-BO repeatedly samples from the GP model, adds them to the dataset, updates the GP, and so on. This iterative process leads to increasingly less reliable GP models and RL environments further into the lookahead horizon. This explains the sub-optimal performance with excessively long lookahead horizons.\n\nWe will include additional experiments testing 1 (i.e., standard myopic), 3, 5, and 7-step lookahead EARL-BO variants on the benchmark functions. If our hypothesis is correct, we expect the smaller lookahead horizons to perform better at earlier iterations, when the initial GP is more uncertain, but longer lookahead horizons to dominate at later iterations, when the initial GP is more representative of the black-box function. Our preliminary findings on the 8D Ackley function indeed follow this pattern: in earlier iterations, the 3-step lookahead method performs best; however, as optimization progresses, the 5-step lookahead method shows superior performance. The 7-step lookahead method performs worse consistently, suggesting the GP environment never reaches an accuracy amenable to such a long lookahead horizon. We will include and expand on these ablation study results in a revised version.\n\nIncorporating uncertainty-aware lookahead mechanisms is an excellent idea for future expansion. We agree that real and virtual data should be treated with different levels of trust. Uncertainty quantification would help mitigate planning delusion by accounting for the reliability of predictions, e.g., with a robust BO approach. Additionally, an adaptive horizon approach (using longer lookahead when uncertainty is low) could be a promising direction for future research.\n\n### **On the comparison with other non-myopic algorithms:**\n\nWe aimed to compare against N-step lookahead BO methods (N>2) that are general and can be applied to non-trivial black-box optimization problems (e.g., handling ~10 dimensions). The Rollout method with variance reduction represents the most recent approach meeting these criteria. Many other non-myopic methods face scalability challenges in higher dimensional spaces, making fair comparisons difficult (though we appreciate it if the reviewer can point us to suitable methods). Nevertheless, we appreciate this feedback and will clarify this justification in the revised manuscript.\n\n### **Regarding the RL agent trained on GP prior only:**\n\nWe are admittedly uncertain what the reviewer means by \"no step-specific training.\" We took this to mean RL-based BO without lookahead steps. Our ablation study for planning delusion will include experiments with exactly 1-step RL-based BO. We expect these results to show that performance is comparable to EI-based BO, since both methods effectively optimize for immediate improvement (RL effectively serves as the \"optimizer\" for the EI acquisition function). Adding this ablation study will support that the multi-step lookahead approach is the main contributor to EARL-BO's enhanced optimization efficiency rather than the RL component. We will include this helpful analysis in a revised version.\n\n### **On using SAASBO priors:**\n\nThank you for this insightful suggestion. We envision EARL-BO as a flexible and modular framework for RL-based BO, which can be fine-tuned to particular problem settings. It can conduct off-policy learning with any state-of-the-art BO method, including SAASBO, and suggest points that will have optimal multi-step lookahead effects. Moreover, other aspects could also be switched, e.g., the choice of RL algorithm (see response to Reviewer eu5H). These directions indeed represent exciting avenues for future work, and we will highlight this in the revised manuscript.\n\nWe thank the reviewer again for their constructive feedback and are pleased that they found our paper well-written and a \"substantial step forward\" in the field that can help with many difficult optimization problems.", "claims_and_evidence": "The paper provides convincing empirical evidence that their proposed method works. Most smaller claims along the way are well supported, except the quick but interesting discussion about \"planning delusion\". I would like to see some more robust discussion / specific experimentation backing up this point (more that provided in the Appendix), which feels quick weak.", "methods_and_evaluation_criteria": "The banchmark datasets and experimental protocols seem valid + experiemntal details seem clear.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Aside from Rollout VR, why are there not more non-myopic algorithms compared agaisnt? This requires justification in my eyes.", "relation_to_broader_scientific_literature": "This paper sits within a popular trend of armortising certain prohibitively expensive operations with GP models, particulary in the context of optimisation. Scholarship is mainly good, but a couple of references are missing (see below).\n\nI do feel like this is a substantial step forward, providing a general framework for deploying complicated BO. I imagine that this work could spin out into lots of interesting future work, applying this framework to other difficult optimisation problems.", "essential_references_not_discussed": "I believe that there are a couple of missing refereces, but scholarship is on the whole good.\n1. The classic meta-learning optimiser paper: https://arxiv.org/abs/1611.03824\n2. Work using a very similar NN arcitecture + meta-training over GP samples, but with the goal of accelerating model fitting https://arxiv.org/abs/2106.08185\n3. Again, similar meta-training models attempting to mimic GPs through a similar NN architecture: https://arxiv.org/abs/1910.13556\n4. A non-myopic meta-learning approach to BO: https://arxiv.org/abs/2302.11533", "other_strengths_and_weaknesses": "The paper seems novel and makes an important contribution. The paper is also very well written and was a pleaseure to read.", "other_comments_or_suggestions": "Have you though about building your RL environment using the SAASBO priors to sample your GP trajectories? This baseline seems to do well and it would be interesting if your method could learn to mimic some aspects of the behaviour + make it nonmyopic?", "questions_for_authors": "I would be tempted to increase my score if the following two aspects could be addressed:\n1) I want to see performance if you just train the RL agent on the GP prior (i.e. no step-sepcfici training) and add this as a baseline. This would really help provide context for how well your step-based episodic environments work\n2) improved exploration of your hypothesis of planning delusion", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MTjTHWQLSq", "reviewer": "Reviewer_Kq9J", "summary": "This paper propose a quite interesting and promising way to combine strength of RL with effective BO optimizer. The authors first pose the limitations of current BO optimizer and hence locate the \"myoptic\" issue wihtin these BO methods. To address that, the authors propose modelling the BO optimization iterations as MDP~(at least POMDP) and learn a sampling neural network by k-step PPO. A series of novel designs have been proposed by the authors to facilitate this technical objective. First, the authors give a reasonable MDP definition that align well with BO optimziation process, including the state reprsentation, action, max(.,.) reward and transition dynamics. With the MDP definition, an attention-based state extraction mechanism is introduced  with a weighted scoring function, which could help extract order-invariant and size-invariant information from gradually enlarged query points set D. The RL agent in this paper serve as a sampling method to sample next query point and the reward function rewards actions with maximum EI value. Moreover, the authors carefully designed the training procedure, which include two stages: 1) off-policy learning from advanced BO method; 2) further on-policy fine-tuning with parts of the network parameters. The effectiveness of the learned BO policy is tested on both synthetic and reaslistic optimziation scenarios and shows improved performance against several baselines.\n\n## after the rebuttal \n\nThanks for the responses. Overall, I still think the added validation and further explanation are not convincing enough. I keep my boardline rejection opinion on this paper. However, I respect the other two reviewers and AC/PC and would leave the final decision to them.\nThe writing of this paper is clear and well organized.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed assessment of our work and for raising important questions that help strengthen the paper. We appreciate the time taken to thoroughly review our submission and provide constructive feedback.\n\n### **Regarding the claim about \"efficiently solving SDP\":**\n\nWe apologize for the lack of clarity in our terminology. Our claim here refers to the relative efficiency of RL in (approximately) solving SDPs, compared to other *multi-step lookahead* BO methods, as the traditional single-step BO approaches do not solve the SDP. For example, our 3-step lookahead EARL-BO demonstrates superior optimization performance across various benchmark functions compared to the Rollout_BO, while also having smaller computational time requirements. Efficiency is indeed an important aspect, which we will explore in a revised version. Please see the response to Reviewer eu5H for our preliminary analysis of CPU time for each BO method on the 8D Ackley function.\n\n### **On addressing \"high-dimensional problems\":**\n\nWe should clarify that we tried to emphasize in our writing that by \"high-dimensional,\" we mean higher-dimensional again in comparison to other *non-myopic* BO methods, which typically handle only very low-dimensional problems (≤6D). EARL-BO successfully scales to practically relevant dimensionalities (up to 19D in our experiments), which represents an advancement for non-myopic BO approaches. Moreover, EARL-BO is a modular framework, and scalability depends mostly on the chosen RL method, providing avenues for future improvement. We will revise our claims in the introduction to be more precise about this contribution.\n\n### **Regarding the ablation study on permutation invariance:**\n\nWe appreciate this excellent suggestion for understanding not only the architecture of EARL-BO, but also the general permutation invariance of Bayesian optimization as an SDP. Following your recommendation, we will include an ablation study comparing an attention-only encoder (with position encodings) to our attention-DeepSets based encoder to exactly explore this point. We have preliminarily run this experiment on the 8D Ackley function for both 3-step and 5-step lookahead settings. Results can be found here: https://anonymous.4open.science/r/icml_2025_review-B587. The results confirm that our proposed method achieves significantly better optimization performance across different BO settings compared to the attention-only encoder, supporting our claims about the importance of permutation invariance. We will include these findings, which support our EARL-BO framework, in the revised version.\n\n### **On evolution strategy baselines:**\n\nWe thank the reviewer for suggesting additional baselines. We want to clarify that our primary aim is to extend the capabilities of non-myopic Bayesian optimization, rather than to compete with methods designed for very high-dimensional problems (100s-1000s of dimensions), and thus large numbers of samples (on the order of 10^6 in the provided references). We will clarify our contributions in the introduction and reference these evolution strategies and Learning to Optimize (L2O) as alternative approaches to BO for different problem settings. This important distinction will help position our work more accurately within the literature.\n\n### **Regarding the minor point about aggregation functions:**\n\nThis comment raises an important point about the potential trade-off between bias and learnability when choosing aggregation functions. While we mentioned arithmetic mean as a permutation-invariant alternative to summation, we agree that maintaining an appropriate level of inductive bias is crucial for effective RL training. Our neural network-based approach provides sufficient bias to facilitate learning, as you noted. We will clarify this discussion in the revised manuscript.\n\n### **Additional improvements:**\n\nWe thank the reviewer for the detailed feedback and will address the other suggestions in a revised version, including: fixing the inconsistent \"K\" symbols in Section 2, correcting the directory titles in the supplementary source code, improving clarity of Figure 1, and expanding our experimental evaluations with the ablation studies suggested throughout the reviews.\n\nWe sincerely appreciate the reviewer's constructive feedback, which has helped us identify areas for improvement and clarification. We believe these changes will strengthen our paper significantly.", "claims_and_evidence": "I acknowledge and agree one of the authors:\n\na) BO has myoptic issue and using MDP modelling and RL for solving the MDP could address this issue. This claim is clearly supported by the elaboration of motivation, related works and methodology in Section 2 and Section 3.\n\nThere are three claims that I think are not fairly validated and supported:\n\na) In page 2, left column, line 073, the authors claim that \"An integrated RL-based BO method that efficiently solves the SDP inherent in multi-step BO\". However, no empirical evidence is provided to demonstrate that point. Is the efficiency here means the computational complexity? If so, this method proposed in this paper involve RL training, isn't is less efficient than traditional BOs?\n\nb) The authors claim that the proposed method aims to address high-dimensioanl problems. However, the problem with maximum dimensions in the experiment is 19-D, which is apparently not a high-dimensional problem.\n\nc) In page 4, right column, line 205-208, the authors claim that \"Permutation invariance is crucial in BO, as the order in which data points\nis acquired should not affect the learning process\". I wonder whether this claim is correct since the sampling trajectories show clear time-dependent features as the GP approximation transforms from a state to another. I strongly request the authors to add an ablation where position encodings are added to the sampling point trajectories and compare with the current version. This would definitily help back up this claim you have posed and seriously affect the correctness of your neural network designs.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experimental protocols are somehow insufficient:\n\na) At least one evolution strategy baseline should be compared since ES also show robust optimziation performance on expensive and high-dimensional problems. I give two suggestions: Sep-CMAES (https://link.springer.com/chapter/10.1007/978-3-540-87700-4_30) and MMES (https://arxiv.org/pdf/2203.12675).\n\nb) The authors only present three problem instances in HPO-B, which makes me wonder the perforamnce of the propsoed method on many other HPO isntances in the benchmark. More instances should be involved and average performance and error bars are needed to demonstrate the efrectiveness. Besides, instances in HPO-B can not fully represent challenging high-dimensional optimziation problems, consider NeuroEvolution benchmarks.\n\nc) There are no ablation studies to explore the true effectiveness of the proposed designs.", "relation_to_broader_scientific_literature": "I notice that this paper show relation with a broader research domain: Learning to Optimize (L2O). Within this scope, the researches widely dicuss how learning techniques such as reinfircement learning can be leveraged to boost traditional optimization methods.", "essential_references_not_discussed": "As I said above, since the topic discussed in this paper falls into Learning to Optimize domain, at least some latest development within this scope should be cited in the beginning of the paper. In particular, since this paper focus on continuous black-box optimization domain, I suggest the following related papers:\n\n[1] https://dl.acm.org/doi/abs/10.1145/3321707.3321813\n\n[2] https://openreview.net/forum?id=h3jZCLjhtmV\n\n[3] https://arxiv.org/abs/2402.17423\n\n[4] https://ui.adsabs.harvard.edu/abs/2024arXiv240503728L/abstract\n\n[5] https://arxiv.org/abs/2411.00625", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "a) I suggest shorten the lengthy text content about related works in Section 2.1, about RL and PPO algorithms in Section 3.1 and Section 3.2. Instead, provide sufficient ablation study results to reinforce the scientific integrity of this paper.\n\nb) The begining of Section 2, there are many \"K\" symbols, with different meanings. Correct them please.\n\nc) Figure 1 is comprehensive yet might be too comprehensive to understand the concept of the propsoed work properly. Simplifying this figure would help readers indeed. \n\nd) a small question, in page 5, left column, line 232-233, the authors claim that an arithmetic mean could also be used as aggregation function. However, it might reduces the bias hence makes it hard for RL to learn, isn't it? The design now, which is a neural network could provide sufficient bias, from my opinion.", "questions_for_authors": "See my comments in other blocks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This manuscript proposes a new methodology (called EARL-BO) for nonmyopic Bayesian optimization grounded in reinforcement learning. The authors study the performance of this method in an empirical study involving both synthetic and real objective functions, where it was competitive with alternative approaches.

The initial response to this work from the reviewers was somewhat mixed, although leaning positive. The reviewers generally found the material to be of interest to the ICML audience and for the proposed methodology to be well-motivated and clear. The reviewers did have some questions and concerns, which were mostly addressed satisfactorily by the author responses.

This paper generated significant discussion during the discussion phase. The primary issue raised, which was seen as a blocker by one reviewer, was the computational cost of the prosed methodology. However, nonmyopia by its very nature comes with increased computational cost, and EARL-BO does appear to be competitive in performance and runtime to other nonmyopic alternatives. Thus it likely lies somewhere on the Pareto frontier of runtime vs performance. This, combined with the potentially interesting ideas put forward in the manuscript, is sufficient to warrant publication in my opinion.
Decision: Accept (poster)