BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL
Authors: Yu Heng Hung
Kai-Jie Lin
Yu-Heng Lin
Chien-Yi Wang
Cheng Sun
Ping-Chun Hsieh
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Objective Bayesian Optimization
Transformers
Hyperparameter Optimization
Reinforcement Learning
Acquisition Function
Primary area: reinforcement learning

Abstract: Bayesian optimization (BO) offers an efficient pipeline for optimizing black-box functions with the help of a Gaussian process prior and an acquisition function (AF). Recently, in the context of single-objective BO, learning-based AFs witnessed promising empirical results given its favorable non-myopic nature. Despite this, the direct extension of these approaches to multi-objective Bayesian optimization (MOBO) suffer from the hypervolume identifiability issue, which results from the non-Markovian nature of MOBO problems. To tackle this, inspired by the non-Markovian RL literature and the success of Transformers in language modeling, we present a generalized deep Q-learning framework and propose BOFormer, which substantiates this framework for MOBO via sequence modeling. Through extensive evaluation, we demonstrate that BOFormer constantly achieves better performance than the benchmark rule-based and learning-based algorithms in various synthetic MOBO and real-world multi-objective hyperparameter optimization problems.

Published as a conference paper at ICLR 2025
BOFORMER: LEARNING TO SOLVE MULTI-OBIECTIVE BAYESIAN OPTIMIZATION VIA NON-MARKOVIAN RL
Yu-Heng Hung'
Kai-Jie Lin't Yu-Heng Lin' Chien-Yi Wang Cheng Sun Ping-Chun Hsieh
NNational Yang Ming Chiao Tung University, Hsinchu, Taiwan 'NVIDIA Research
(hungyh.com cs08, pinghs @nycu. edy.cc.eedu....
ABSTRACT
Bayesian optimization (BO) offers efficient pipeline for optimizing black-box functions with the help of a Gaussian process prior and an acquisition function (AF). Recently, in the context of singlc-objective BO, learning-based AFs witnessed promising empirical results given its favorable non-myopic nature. Despite this, the direct extension of these approaches multi-objective Bayesian optimization (MOBO) suffer from the hypervolume identifiability issue, which results from the non-Markovian nature of MOBO problems. To tackle this, inspired by the non- Markovian RL literature and the success of Transformers in language modeling, we present a generalized deep Q-learning framework and propose BOFormer, which substantiates this framework for MOBO via sequence modeling. Through extensive evaluation, we demonstrate that BOFormer constantly outperforms the benchmark rule-based and beariing-based algorithms in various synthetic MOBO and real- world multi-objective hyperparameter optimization problems. We have made the source code publicly available to encourage further research in this direction."
INTRODUCTION
Bayesian optimization (BO) offers a sample-efficient pipeline for optimizing black-box functions in various practical applications, such hyperparameter optimization (Lindauer et al., 2022: Snoek et al., 2012; Klein et al., 2017), analog circuit design (Lyu et al.. 2018; Zhou et al., 2020), and automated scientific discovery (Ueno et al., 2016; Homnez-Bombarelli al., 2018). Notably, these real-world engineering tasks usually involve multiple objective functions, which are potentially conflicting. To search for the set of candidate solutions under sampling budget, multi-objective BO (MOBO) integrates the following two components: (i) MOBO utilizes Gaussian processes (GP) as a surrogate function prior for capturing the underlying structure of each objective function and thereby offering posterior predictive distributions in a compact manner; (ii) MOBO then iteratively determines the samples through an acquisition function (AF), which induces an index-type strategy based on the posterior distributions. The existing AFs are built on various design principles. such as maximizing one-step expected improvement (Emmerich & Klinkenberg, 2008: Yang et al., 2019) and maximizing one-step information gain Bernández-LLobto et al., 2022). However, the existing AFs for MOBO are mostly handcrafted and myopic, iee, greedily optimize one-step surrogate and lack long-term planning capability. With that said, one important and yet under-explored challenge of MOBO lies in the design of non-myopic AFs.
al., 2016; Belakaria al., 2019; Tu
In single-objective BO (SOBO), one promising non-myopic approach is recast BO a rein- forcement learning (RL) problem, and several RL-based algorithms (Volpp et al., 2020; Hsich et al., 2021; Shmakov al., 2023) have recently witnessed competitive empirical results. Specifically, AFs could be parameterized by neural networks and learned by either actor-critic (Volpp et al., 2020) or valued-based RL (Hsieh al., 2021), where the state-action representation consists of the posterior mean and variance of candidate point as well as the best function value observed so far, and the reward defined as a function of negative simple regret, as shown in Figure 1. However, a direct
'Equal contribution.
Equua advising.
"https:: httpps/:///hnuuuhenggg gittggbittt...o.. 10/ /O

---

Published as a conference paper at ICLR 2025
Ri-Based Arquistion Function
MOBO: Scenario
SO80
Sample og
Sumple ag
o(44)
RL-AF
T(AA)
Sample xg
Sample
Befarence
Acoulsition Function to MOBQ
Direct Extinnsion of BlBased
MORG: Scenario
Sample xg
Increased HV
Sample Xg
(4% ASUK]
Sample Xq
(o))
Y,000
(y (eik)
f000
Reference point
Figure 1: Left: In SOBO, an RL-based AF (e.g.. FSAF (Hsich al., 2021)) takes the posterior mean and standard deviation (Ne (z), oz) (z)) and the best function value observed so far Vē as input and then outputs the AF value T2 (2)). An direct extension to MOBO simply takes into account the same set of information about all the K objective functions. Right: The hypervolume identifiability issue can be illustrated by comparing the hypervolume improvement incurred by the sample I3 in the two different scenarios above. Clearly, despite that the AF inputs at 23 are the same in both scenarios, the increases in hypervolume upon sampling 23 are rather different. Hence, the increase in hypervolume is not identifiable solely based on the AF input (44 (z), Oz (x)), De (5)*
(1) (4)
]εE[K] of the existing RL-based AFs.
extension of these single-objective RL-based AFs to MOBO could suffer from the fundamental hypervolume identifiability issue. To better illustrate this, we provide motivating example in Figure 1, which shows that one can construct pair of scenarios that cannot be distinguished based solely on the current posterior distributions and the current best values. This example also highlights that the identifiability issue actually results from the inherent non-Markovianty in MOBO as the improvement in hypervolume is cestory-dependen setting since good candidate point (i.e., with a high function value) remains good regardless of the history. As a result, there remains one important open challenge in MOBO:
Note that this identifiability issue is much milder in the SOBO
How learn a non-myopic AF for MOBO without suffering from the above identifiability issue?
To tackle the above challenge, we propose to rethink MOBO from the perspective of non-Markovian RL via sequence modeling. Motivated by the optimality equations in the general non-Marke Mon-Markovian RL (Dong al., 2022), we tackle the hypervolume identifiability issue by presenting the Markoviann version of deep Q-network termed Generalized DQN, which extends the standard Markovian DQN (Mnih et al., 2013) by learning the generalized optimal Q-function defined on the history of ob- servations and actions. To implement Generalized DQN, inspired by the significant success of the Transformers in language modeling, we propose BOFormer, which leverages the sequence modeling capability of the Transformer architecture and thereby minimizes the generalized temporal difference loss. As a general-purpose multi-objective optimization solver, the proposed BOFormer is trained solely on synthetic GP functions and can be deployed optimize unseen testing functions.
Moreover, to facilitate the training process, we present several useful and practical enhancements for BOFormer: (i) Q-augmented observation representation: Regarding the representation of per-step observation, we propose to use the posterior of the candidate point augmented with its Q-value, which serves as an informative indicator of the prospective improvement hypervolume. Under this design. the representation is completely domain-agnose agnostic and memory effforeeefficient in the sense that it does not increase with the domain size. (ii) Prioritized trajectory replay buffer and off-policy learning: To improve convergence and data efficiency during training, we utilize prioritized trajectory replay buffer, which can be viewed as a generalization of the typical replay buffer vanilla DQN. Through this buffer, BOFormer naturally supports off-policy learning and more flexible reuse of training data. (iii) Demo-porre comoo-plicy-eee policy-guided exploration: While randomized exploration (e.g., ppsilon-greede popular in many RL algorithms, this exploration scheme can be very inefficient as the sampling budget in BO is usually much smaller than the domain size (i.e., the number of actions). To achieve efficient exploration, propose to collect part of the training trajectories through helper demo
remains
2

---

Published as a conference paper at ICLR 2025
policy. In practice, one resort to a policy induced by any off-the-shelf rule-based AFs, such as Expected Hypervolume Improvement (Emmerich & Klinkenberg, 2008).
Notably, general-purposs multi-objective optimization solver, BOFormer enjoys the following salient features: (i) No hypervolume identifiability issue: Built on the proposed Generalized DQN framework, BOFormer systematically addresses the identifiability issue such that it can approximately recover the Pareto front under a small sampling budget. (ii) Zero-shot transfer: BOFormer is trained solely on synthetic GP functions and can achieve zero-shot transfer to other unseen testing functions. With that said, it does not require any fine-tuning or any metadata during inference at deployment. (iii) Cross-domati transfer capability: BOFormer nicely supports cross- cross- domain transfer in the sense that the domain size and dimensionality of the black- box functions for training can be different from those of the testing functions. (iv) No Monte-Care estimation needed during inference: As learning-based AE, BOFormer completely obviates the need for the computationally heavy Monte-Carlo estimation required by many rule-based AFs and thereby enjoys efficient inference during deployment. The main contributions could be summarized as follows:
We identify the critical hypervolume identifiability issue due to the inherent non-Markovianity in learning the AFs for MOBO. To resolve this, inspired by the literature of general RL, we present the Generalized DQN framework for nan-Markovian environments.
To substantiate the Generalized DQN framework, we propose BOFormer, which leverages the Transformer architecture and reinterprets MOBO as a sequence modeling problem. To the best of our knowledge, BOFormer serves as the first RL-based AF for MOBO. Moreover, several practical enhancements are proposed to facilitate the training of BOFormer.
We construct a hypervolume optimization dataset called HPO-3DGS, consisting of 68000 different parameters on 3D Gaussian Splatting (Kerbl et al., 2023) for 3D object reconstruction problems with different scenes. We evaluate the proposed BOFormer on a variety of black-box functions, including both synthetic optimization functions and real-world hyperparameter optimization HPO-3DGS. We demonstrate that the proposed AF significantly outperforms both existing rule- based AFs and other Transformer-based RI. benchmark methods.
2
RELATED WORK
2.1
MUL.TI-OBJECTIVE BAYESIAN OPTIMIZATION
Random Scalarization. To leverage the plethora of AFs for SOBO in the MOBO setting, random scalarization addresses MOBO via iteratively solving single-objective BO subproblems under scalarization function, such as a direct weighted sum or the Tchebycheff scalarization function (Miettinen, 1999). Notably, random scalarization was originally developed for recovering the Pareto front under evolutionary methods, such as the celebrated ParEGO (Knowles, 2006), MOEA/D (Zhang & Li, 2007), and RVEA (Cheng et al., 2016), and has been subsequently adapted to solving MOBO (Zhang et al., 2009; Paria et al., 2020). Despite its simplicity, as random scalarization enforces exploration mainly by the random sampling of the scalarization parameters, this approach is known to be sensitive to the scale of the different objective functions and could suffer under high-dimensional search spaces (Daulton et al., 2022).
Improvement Maximization. Another popular class of AFs is built on the maximization of improvement-based metrics, such as the expected one-step improvement (Emmerich & Klinkenberg, 2008; Emmerich et al., 2011; Hupkens et al., 2015; Yang et al., 2019) (also known as the S-metric in (Beume et al., 2007: Ponweiser et al., 2008)), sequential uncertainty reduction (Picheny, 2015), and the hypervolume knowledge gradient (Daulton et al., 2023). However, evaluating the one-step expected improvement typically involves a multi-dimensionee is difficult to derive directly and hence needs to be approximated by the costly Monte Carlo estimation. Accordingly, to tackle the above computational complexity issue, differentiable methods have recently been developed to enable fast parallel evaluations of these AFs, such gEHVI (Wada & Hino, 2019; Daulton et al., 2020) and NEHVI (Daulton et al., 2021) in BoTorch (Balandat et al., 2020).
hypervolume (EHVI) in
integral, which
criteria have been utilized in the context of MOBO. For example, Hernández- Lobato et al. (2016) proposes Predictive Entropy Search for MOBO (PESMO), which selects the candidate point with maximal reduction in the entropy of the posterior distribution over the Pareto-optimal input set, and is subsequently extended to the constrained setting Saarido-Merchal & Hernández-LLobaoo, 2019). Subsequently, Belakaria et al.
Information- Cntommation-Thooettiee Search Methods. Various information-
3

---

Published as a conference paper at ICLR 2025
(2019) propose MESMO, which extends the Max-value Entropy Search approach (Wang & Jegelka, 2017) to the principle of output space entropy search for MOBO, i.e, utilizes the information gain computationally tractable sampling criterion (Belakaria
about the Pareto-optimal output set as et al., 2021). Suzuki et al. (2020) propose Pareto-Frontier Entropy Search, which utilizes information gain of the Pareto front in the AF design. Moreover, Joint Entropy Search (JES) further takes into account the joint information gain of the Pareto-optimal set of inputs and outputs (Tu et al., 2022; Hvarfner et al., 2022). On the other hand, USeMO (Belakaria al., 2020) uses the volume of the uncertainty hyper-rectangle as an alternative uncertainty measure for sampling.
Despite the plethora of AFs developed for MOBO, most of them are built on optimizing one-step information-theoree metrics and do not explore the possibility of multi-step look-ahead policies. By contrast, the proposed BOFormer takes the overall long-term effect of each sample into account through non-Markovian RI. and sequence modeling.
2.2 SINGLE-ORIECTIVE BLACK-BOX OPTIMIZATION VIA LEARNING
Several recent attempts have tackled SOBO problems from the perspective of RL-based AFs. Volpp et al. (2020) propose MetaBO, which leverages actor-critic RL to learn AFs from GP functions for transfer learning, Subsequently. Hsieh et al. (2021) proposes meta-RL framework termed Few-Shot Acquisition Function (FSAF), which learns a Bayesian deep Q-network as a differentiable AF and adapts the Bayesian model-agnostic meta-learning (Yoon et al., 2018) in order enable few-shot fast adaptation to various black-box functions based on metadata. (Shmakov et al., 2023) proposes solve SOBO through combination of transformer- acquisition functions. Despite the above, the existing solutions all focus on SOBO under the standard RL formulation and therefore cannot be directly applied the non-Markovian problem of MOBO. Moreover, as optimization of single-objective black-box functions is essentially a sequential decision making problem, several recent attempts manage to learn sequence models in an end-to-end manner. For example, Chen et al. (2022) propose OptFormer, which focuses on hyperparameter optimization (HPO) and leverages Transformers through fine-tuning on an offline dataset to enable adaptation to the HPO tasks. More recently. Maraval et al. (2023) propose Neural Acquisition Process (NAP), which is multi-task variant of Neural Process (NP) simultaneously learning an acquisition function and the predictive distributions, without using the surrogate GP model. To the best of our knowledge, our paper offers the first learning-bast based solution to MOBO.
deep kernels and RL-bbased
Remarks on Application Scope and Objectives. Notably, there are two salient differences between BOFormer and the above two works: (i) Application scope: BOFormer is positioned as a general- purpose multi-objective black-bos optimization solver with superior cross-domain capability (i.e. the size and the dimensionality of the input domains can be different between the training and deployment phases). In contrast, OptFormer is designed specifically for HPO, and NAP is built on the idea of transfer learning in BO and has limited cross-domain transferability. (ii) Multiple objectives: Both OptFormer and NAP focus on single-objective problems and are not readily applicable to recovering the Pareto front in the multi-objective setting. By contrast, BOFormer directly tackles mujtivobeective optimization and addresses the inherent identifiability issue. Therefore, we consider the above OptFormer and NAP as orthogonal directions to ours. Moreover, the proposed BOFormer can also benefit from the learned neural process in NAP and other variants of NPs (Garnelo et al., 2018; Kim et al., 2018) as surrogate models beyond GPs.
Due to the page limit, we defer the related works on sequence modeling for RL to Appendix B.
3 PRELIMINARIES
In this section, we present the formulation of MOBO and the background of general RI. Throughout this paper, we let (2 denote the set of all probability distributions over a set Z and use [K] as a shorthand for {1, K}.
3.1 MULTI-OBIECTIVE BAYESIAN OPTIMIZATION
The goal of MOBO is to design algorithm that sequentially takes samples from the input domain
X c R to jointly optimize a black-box vector-vale valued function f X - RKK under a sampling budget
T E N. For ease of exposition, we also write f(x) = (fi(z)... .fx(z) as the tuple of the K
4

---

Published as a conference paper at ICLR 2025
scalar objective functions, for each X. At each step t. the algorithm selects a sample point 2, E X and observes the corresponding function values y: == (y/ (1)
) where // f1())) + Et, is the noisy observation of the i-th entry of the function output and Ex,, are i..d. zero-mean Gaussian noises. For notational convenience, we use Fr == {(x,, y:) to denote the observations up to t.
Pareto Front and Hypervolume. To construct a (partial) ordering over the points of the input domain, we say that f(x) dominates fa') if f((x) 2 f((z') for all E [K] and fa( > fy (2') for at least one element j. For simplicity, we write z > x' if f(z) dominates f(x'). Based on this, the Pareto front (denoted by X²) is defined as the subset of X that cannot be dominated by any other point in X. Le., X" = {ar E X|x' y x, Va' E X). An alternative description of the goal of MOBO is to discover the Pareto front. Accordingly, MOBO algorithms are typically evaluated from the perspective of hypervolume, which offers natural performance metric for capturing the inherent trade-off among different objective functions. Specifically, given a reference point u E R and any subset X' S X. the hypervolume of X' is defined as (Zitzler & Thiele, 1999):
HV(X; := X U (v]f(z) EEX
(1
> y uu))
where X(-) is the K-dimensional Lebesgue measure and ys are those elements that satisfy (() > > u E X. In practice, the reference point can be configured as u (min (ar)..... min fx())). To evaluate policy, we consider the simple regrer defined as R(t)= == HX(X)= - H)(X)). which measures the overall performance of the samples up to time step For brevity, we simply use HV(X) as a shorthand for HV(X'; u. in the sequel.
Gaussian Process Surrogate Model. To maximize hypervolume in sample-efficient manner, MOBO imposes function prior through GP, which serves as surrogate probabilistic model for capturing the underlying structure of the objective functions. Specifically, as Bayesian approach, the GP assumes that for each objective function fi(+), the function values at any set of input points form multivariate Gaussian distribution, which can be fully characterized by mean function and a covariance kernel. Therefore, under a GP prior, given the observations F, up to time t, the posterior predictive distribution of each f((z) (æ E X) remains Gaussian and can be written as
N N(
(6) (z), of (6) (z)²). where vuبi) (x) == [x) [ Fe] and of (z) := /V]]((z)] Fi] can be derived in closed form through matrix operations (Williams & Rasmussen, 2006). For notational convenience, (K)
we let ( (z) == 2 (z))
(x), ... .44 (k) (x)) and or (x) == (a (x), ... .o, (x))
Acquisition Functions. With the help of GPs, one natural way to address BO is through planning as in optimal control (e.g., via dynamic programming). However, finding the exact optimal policy for BO (either single- or multi-objective) is known to be computationally intractable in general due to the curse of dimensionality. To tackle this issue, BO resorts to index-type strategies induced by an acquisition function ( (x)), (x)), which takes the posterior mean and variance input and outputs an indicator for quantifying the usefulness of each potential candidate sample point x E X, typically based on some handcrafted design criteria. For example, the celebrated Expected Hypervolume Improvement (EHVI) method constructs an AF as THHVI(H)(T)) o (x)) == EHH, Ufz}) HV(X))) Fil, which involves a multi-dimensional integral with respect the posterior distribution characterized by μu (z) and (z))
3.2 GENERAL RL IN NON-MARKOVIAN ENVIRONMENTS
To achieve RL without Markovianity, several generalizations of the standard Markov decision process (MDP) have been proposed, such as the classic cartially-observabe MDPs (Monahan, 1982; Jaakkola al., 1994), the early works on general RI. (Lattimore et al., 2013; Leike, 2016; Majeed, 2021), and the more recent attempts on RL for arbitrary environments (Dong et al., 2022; Lu et al., 2023; Bowling et al., 2023). In this paper, we consider the general RL formulation in (Dong al., 2022: Lu et al., 2023) to address policy learning beyond Markovianity.
Environment. The general interaction protocol of the agent and the environment can be described as follows. Let A and O denote the set of actions and observations, respectively. At each time t E N. the agent first receives new observation O2 E O from the environment and takes an action A, E A based on the history H, := (Ao, O₁, AI, ... Ar-1. Ot) observed so far. For simplicity, we let the initial history Ho be empty. We also define the set of all n-step histories H()) (A x 0)a
5

---

Published as a conference paper at ICLR 2025
and accordingly define the set of all finite histories H. := Un2o H(00 The transition dynamics of the environment is captured by the transition function p:
x A A(O), which determines the transition probability p(olh, a) = P(O = H, = A, a) of observing o upon applying action a under history h. Moreover, r H A
[-Tmaxx T'mxx] denote the reward function. Notably, the reward function r in non-Markovian environments is allowed to be history-dependent and hence better suits the MOBO problems. Let E [0, 1) denote the discount factor for the rewards.
Policies and Value Functions. The agent specifies its strategy through policy я H - ((A),, which maps each history to a probability distribution over the action set. Let II denote the set of all policies. Similar to the MDP setting, we define value functions that reflect the long-term benefit of following policy и. Given a 7-step history h E H.
(h): E.
(H,, A,, O++1) Hr in]
@²( = E. Σ rrH,, A,, Oc+1) Hr - h1 A, a}
Moreover, we extend the definitions the optimal value functions in MDPs to the non-Markovian setting
("(h) = sup V* Q" = sup Q" (h, a). well
(I)
sell
The proposition below offers a generalized version of the Bellman optimality equations and charac- terizes V and Q*
Proposition 3.1 (Dong al. (2022)). The pair of V*,Q") is the unique solution to the following system of equations:
((h) max Q(h,a') d'E
(2)
a,)) Q(h, a) 1o.- +
(3)
where V : H -> IR and Q H. x A - R are bounded real-valued functions.
METHODOLOGY
4.1
GENERALIZED DQN FOR NON-MARKOVIAN PROBLEMS
Motivated by the optimality equations in (2)-(3), we convert these fundamental properties into a learning algorithm.
Loss Function. To learn Q*, we adapt the loss function of the standard DQN to the generalized non-Markovian version by minimizing the residual of the optimality equation. Let Q& a) denote the parameterized Q-function. Then, the loss function of the generalized DQN is designed as
E(hao,)) (rlth (,) + max
Qa a') Qo(h,a)) 27 dea
(4)
where D is the underlying distribution of the observed histories during training, h = (h, a, o) is the history for the next Q-value, and Q3 is copy of Qo with parameters frozen.
Remark 4.1. The above loss function bears some resemblance to that of the POMDP variant of DQN, such as Deep Recurrent Q-Networks (DRQN) in (Hausknecht & Stone, 2015). Despite this, one fundamental difference is: The POMDP formulation presumes that there exists hidden true state, which determines the transitions and the reward function, and the hidden state is to be learned and deciphered by recurrent neural networks in DRQN. By contrast, Generalized DQN does not make this presumption and involves only the history of observations and actions.
Direct Implementation of Generalized DQN. To implement (4), one natural design is to leverage sequence modeling (e.g., Transformers) and directly use the full observations the input of the sequence models. This design principle is widely adopted Transformer- based RL (Chen et al., 2021; Janner et al., 2021; Chebotar et al., 2023) for various popular RL benchmark tasks (e.g., locomotion

---

Published as a conference paper at ICLR 2025
and robot arm manipulation in MuJoCo (Todorov et al., 2012)). In the context of learning AFs for MOBO, one can apply this design principle and extend the representation design of AF for SOBO (cf. Figure 1) to the MOBO setting, and this amounts to taking the posterior distributions of all K objective
Ji=i the best function values functions at all the domain points along with {y/ (8)* = argmax, SS (6) 1
( (r), o (0) (x), ( J While being a natural variant of Transformer-basel RL, this implementation of the Generalized DQN framework can be problematic in MOBO for two reasons: (i) Limited cross-domain transferability: As the observation representation is domain-depender under this design, the learned model is tied closely to the training domain and has very limited transferability. As result, retraining or customization is needed for every task at deployment. (ii) Scalability issue in sequence length and memory requirement: Under this design, the sequence length would grow linearly with the number of domain points and pose a stringent requirement on the hardware memory for training. Indeed, the domain size is at least on the order of thousands in practical BO problems (e.g, circuit design (Lyu al., 2018) and hyperparameter optimization (Lindauer et al., 2022)).
observed so far as the per-step observation, i.e.,
To tackle the above issues, we propose an alternative design that better substantiates the Generalized DQN framework for MOBO with comain-agnoste representations and several practical enhance- ments, as detailed Section 4.2.
4.2 BOFORMER: AN ENIIANCED IMPLEMENTATION GENERALIZED DQN
To avoid the issues of the direct imple- mentation, we propose BOFormer, which is built on the following enhancements. The pseudo code is in Algorithm in Ap- pendix C.
BOFormer (Policy)
Osservation-Action
HVV()) DVV))-)) Hv(*) UV(())
#4, (Q Target)
Q-Augmented Representation. Define
+4 (Q-Pulicy)
y1 (0)+ max Vi Vi E [L,...,,K]] 1S5St
Elament-Wis Addition
BOFormer (Target)
as the best observed function value of i-th objective at time f. Moreover, for each domain point X. let oz(z) denote the observation for x as (x) p (z), (x). (1), L "T] [E(K]]
Posiionaa Encocing
Figure 2: BOFormer comprises two distinct networks as shown above: The upper network functions as the policy network, utilizing the historical data and the Q-value predicted by the target network to estimate the Q-values for action selection. The lower network serves as the target network, responsible for constructing Q- Q-values for past observation-action pairs.
Moreover, in BOFormer, we use the nor- malized hypervolume improvement as the reward, i.e.,
HV( - (V)
Tt HV(X*) HV(X)*
Then, h4s the history up to time f, is the concatenation of past obwervation-acteeee pair representation defined as follows:
(9) (1z). 1 (1z)) w/ j/. Q7
h4
(5)
ie (e)))-t((-
Notably, under this design, the representation is domain-agnostic and memory in the sense that its dimension does not increase with the domain size.
BOFormer
Acquisition Function for MOBO. The model structure of BOFormer is provided
in Figure 2. Denote Qo(-) (((-1) x + R to be the function of BOFormer parameterized by and let i represent the parameters of BOFormer. The selected point 2; satisfies that x, == argmaxnex QB (he, or (x)).
Then, Qi considered in hy can implemented by target network, as is commonly done in Deep Q-learning. In non- Markovian version, Qg can be defined recursively, where
Qh or (xx))) = Qa ((o,(1)), -0(0)-1 o((2))
7

---

Published as a conference paper at ICLR 2025
Remark 4.2. To handle continuous domains, BOFormer approximate the maximum Q-value using Sobol grids, as adopted by (Volpp et al., 2020; Hsich et al., 2021). However, this could be a lim- itation in high- -igh-dimessional tasks. To address this, empirical study on high-dimensional problems and the detailed procedure of employing Sobol grids are in Appendix D.3 and C.2, respectively.
Off-Policy Learning and Prioritized Trajectory Replay Buffer. We extend the concept of Priori- tized Experience Replay (PER) (Schaul et al., 2016) and introduce the Prioritized Trajectory Replay Buffer (PTRB). The detailed modifications are as follows: (i) Elements pushed into this buffer are entire trajectories т = {or (x.), r- (ii) The TD-error considered in PER is replaced by o(Qe,, T), which is the summation of the TD-error of the policy network for all transitions in this trajectory, ie.,
T-1
8Q. = Σ (h, (zi)) - (rs + y max Qo(h,+1, 04x0
(6)
Let B denote the batch sampled from PTRB. The loss function of BOFormer is defined as [(0)- == Eres (Q,,
5 EXPERIMENTS
In this section, we evaluate the proposed BOFormer against popular MOBO methods on both synthetic and hyperparameter optimization on 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023). Unless stated otherwise, we report the average attained hypervolume at the final step over 100 evaluation episodes in the main text. Due to the space limit, all the statistics, including the hypervolume at each step and performance profiles (Agarwal et al., 2021) are provided in the Appendix.
Benchmark Methods. We compare BOFormer with various classic and state-of-the-ar benchmark methods, including: (i) Rule-based methods: NEHVI (Daulton et al., 2021), ParEGO (Knowles, 2006), NSGA-II (Deb et al., 2002), HVKG (Daulton et al., 2023), and JES (Tu et al., 2022; Hvarfner et al., 2022). Regarding NEHVI, ParEGO, and HVKG, we use the differentiable Monte- Carlo version, namely NEHVI, gParEGO, and gHVKG, provided by BoTorch (Balandat et al., 2020). (ii) Learning- based methods: Given that BOFormer is the first learning- based MOBO method, we consider the direct multi-objective extension of FSAF (Hsich et al., 2021), which achieves state-of-the-arr results in SOBO. To showease the design of BOFormer, we also adapt a popular RL Transformers, namely Decision Transformer (DT) (Chen et al., 2021), to the MOBO setting. Moreover, we also include O-Transformer (QT) (Chebotar et al., 2023), a more recent Transformer design RL that uses a similar DQN loss (termed Autoregressive Discrete Q-L.earning in their paper) and can be viewed as variant of BOFormer without Q-augmented representation. To further demonstrate the competitiveness of BOFormer, we also compare it with OptFormer (Chen et al., 2022), a recent Transformer-basee method designed specifically for hyperparameter optimization. All the learning-base based methods are trained on GP functions under with the lengthscales drawn randomly from [0.1, 0.4] for faimess. The detailed configuration is provided in Appendix A.
Q: Does BOFormer achieve sample-efficient MOBO on variety of optimization problems?
Synthetic Functions. We answer this question by first evaluating BOFormer extensively on a diverse collection of synthetic black-box functions: (i) Combinations of functions with many local optima, including Ackley Rasttiinn (ARa): (ii) Combination of smooth functions, including Branin- (BC) and Chanin-Currin-Be Dixon, (iii) Combination of non-smooth and smooth functions, including Ackley-Rooenbroock (AR) and Dixon-Rastrigin (DRa) Table and Figure 6 shows the averaged hypervolume on synthetic problems. Based on Figure 6, we can observe that BOFormer constantly achieves the largest or among the largest hypervolume among all methods. In the cases of Branin and Currin, the performance of BOFormer not as expected, which we attribute the larger length scales of these functions compared to others. Then, QT, a variant of the generalized DQN but without Q-augmented representation, does not perform well. This indicates that sequence modeling itself does not necessarily guarantee an efficient search for the Pareto front, and Q-augmented representation are needed for solving MOBO, as showcased by the proposed BOFormer. Figure shows the performance profiles, which are meant to more reliably present the performance variability of BOFormer and other baselines across testing episodes than the interval estimates of aggregate metrics. We observe that in most tasks, the profiles of BOFormer sit the top right of the other baselines and hence enjoy a statistically better performance in hypervolume. Please see Appendix D for more performance profiles of the final attained hypervolume.
8

---

Published as a conference paper at ICLR 2025
Multi-Objective Hyperparameter Optimization 3D Gaussian Splatting. We also evaluate our method on 3D object reconstruction problems. This task takes multiple images, which captures an object, as input, and the goal is to reconstruct 3D representation for novel-view rendering. 3D Gaussian Splatting (3DGS) by Kerbl et al. (2023) is the recent state-of-the-art on this task. One practical issue of 3DGS is their sensitivity to the hyperparameters. To achieve the best quality, we typically require manual tuning for each capture, which is time consuming and requires expert knowledge. An automatically hyperparameters tuning pipeline thus become very useful. To evaluate on this task, we construct dataset by consulting domain expert and gense-grid searching 1440 hyperparameters of 3DGS. We perform 30 samples for this task where we defer the details in the supplementary. The objectives are negative model size and novel-view rendering quality measured by PSNR. We use different objects from (Mildenhall et al., 2021) and 64 different chairs from (Yu et al., 2023) to compare the performance of different hyperparameter tuning methods. Again, from Table 2 and Figure 7, we observe that BOFormer remains the best or among the best in all the tasks. This demonstrates the wide applicability of the proposed BOFormer. The detailed hypervolume per step and performance profile of the final hypervolume for 3DGS dataset is provided in Appendix D.
Table 1:
The average attained hypervolume at the I 100th sampling step under synthetic functions.
Boldface and underlining denote performance within 1% of the best-performing method.
AR
ARa
BC
DRa
RBF
Matern
BCD
Rule-haxed Methods
qHVKG (Daulton et al., 2023)
0.5787
0.7008
0.4751
0.9499
0.8646
0.8506
0.3547
NSGA-II (Deb et al., 2002)
0.5557
0,7122
0.4271
0,9573
0.8603
0.8569
0.3400
QNEHVI (Daulton et al., 2021)
0.5428
0.6290
0.4773
0.9333
0.8731
0.8696
0.3697
JES (Tu et al., 2022)
0.4930
0.6207
0.4487
0.9392
0.8661
0.8594
0.3345
qPrFEGO (Knowles, 2006)
0,5410
0.6111
0.4597
0.9347
0,8730
0,8684
0.3564
Learning-based Methods
BOFomme (Ours)
0.5900
0.7377
0.4476
0,9461
0.8751
0.8642
0,3617
FSAF (Hsich et al., 2021)
0.4424
0.4800
0.4175
0.9193
0,8381
0,8566
0.3578
OptFormer (Chen et al., 2022)
0,4448
0.4835
0.4143
0.8927
0,8558
0,8488
0.3437
DT (Chen et al., 2021)
0.4397
0.4855
0.4159
0.9141
0.8409
0.8359
0.2980
QT (Chebotar et al., 2023)
0,4630
0.4953
0.4086
0.8584
0,8685
0,8550
0.3320
Table 2: The average hypervolume at the 30th step under 3DGS hyper-parameter optimization scenarios. Boldface and underlining denote performance within 1 5 of the best-performing method.
Chairs
Lego
Materials
Mic
Ship
Rule-based Methods
qHVKG (Daulton et al., 2023)
0.8508
0.9365
0.9088
0.8234
0.9630
NSGA-II (Deb al., 2002)
0.8500
0.9192
0.8831
0.8051
0.9615
QNEHVI (Daulton al., 2021)
0.9159
0.9344
0.9041
0.8353
0.9661
JES (Tu et al., 2012)
0.9003
0.9409
0.9217
0.8251
0.9656
ParFGGO (Knowles, 2006)
0.9103
0.9335
0.9098
0.8263
0.9635
Learning-based Methods
BOFormer (Ours)
0.9162
0.9508
0.9224
0.8816
0.9745
FSAF (Hsich et al., 2021)
0.9120
0.9504
0.9213
0.8871
0.9737
OptFormer (Chen et al., 2022)
0,8200
0.7875
0.7838
0.7865
0.9575
DT (Chen et al.. 2021)
0,8602
0.9315
0,9005
0.8786
0.9409
QT (Chebotar et al.. 2023)
0.9119
0.9368
0,9142
0.8731
0.9551
1-2 and Figures 3-8, we observe that the performance of BOFormer surpasses that of OptFormer. We conjecture that the reasons are two-fold: (i) OptFormer takes supervised learning perspective to learn the context that describes the HPO information while BOFormer leverages Mar--iannv RL for better long-term planning. (ii) OptFormer reinterprets HPO as a language modeling problem in a text-to-text manner. This approach necessitates that the training dataset closely resembles the testing domain, and this requirement does not hold here (e.g., dimensionality of the training domain differs from that of the testing domain).
O: The comparison between BOFormer and OptFormer. From Tables
Q: A study on the effect of sequence length of BOFormer. We also conducted an ablation study on evaluating how the sequence length (denoted by w) would affect the hypervolume performance of BOFormer. Figure 4 shows that the hypervolume of non-Markovian BOFormer (w > 1) is superior to that of Markovian BOFormer (w = 1). The results also demonstrate that sequential modeling can successfully solve the hypervolume identifiability issue.
@: A study on computational efficiency. We provide computation times per step in Table 3. BOFormer and gNEHVI are competitive in terms of final hypervolume (Tables 1-2), with the computation times

---

Published as a conference paper at ICLR 2025
of BOFormer being less sensitive to the number of objective functions and shorter than those of NEHVI under the 3-objective function setting.
Q: How is the transfer ability of BOFormer under diffferent numbers of objective functions? Although lcarning-bast based methods generally require training multiple models for different numbers of objective functions due mismatched model structures, to determine whether it is possible to train only the state-action embedding layer while utilizing a pre-trained transformer model from different number of objective functions. The results, shown in Figure 8, demonstrate that BOFormer exhibits excellent transfer ability across different numbers of objective functions. The detailed implementation of transferring on BOFormer in Appendix C.2.
conducted experiment BOFormer
BOFormer (Oun)
Opt
PPrBCO
Fautinn too ogooniee with BVV'
Fratimm of upionbee with HTV'
0.3
3.5
of upsentre
6.6
9.0
0.5
0.5
0.5
Attained HV the linal step (+)
Attalned EV at the final stop
Attained BV the findl step (r)
(a) AR
(b) ARa
(c) BC
Paucian of spactia with BEV
0.5
0.8
Paurian enjostio
Seaph
2.4
04
0.4
Artained IIV the final step (r)
Attained LDV at the final step (r)
Attained the step (r)
(d) Chairs
Materials
(f) Lego
Figure 3: Performance profiles of hypervolume at the final step.
31
0.55
0.85
0.8
22
0.25
0.7
0.65
0.00
0.6
0.6
0.6
Samples
Samples
Samppea
(a) DRa
(b) RBF
(c) Matern52
Figure 4: Attained hypervolumes of BOFormer under various sequence lengths.
CONCLUSION
In this paper, we address MOBO problems from the perspective RL-based AF by identifying and tackling the inherent hypervolume identifiability issue. We achieve this goal by first presenting generalized DQN framework and implementing it through BOFormer, which leverages the sequence modeling capability of Transformers and incorporates multiple enhancements for MOBO. Our experimental results show that BOFormer is indeed promising approach for general-purpose multi-objective black-box optimization.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This material is based upon work partially supported by NVIDIA grant and technical involvement from NVIDIA Taiwan Al R&D Center (TRDC), National Science and Technology Council (NSTC), Taiwan under Contract No. NSTC 113-2628-EL 2---888--449-22 and the Higher Education Sprout Project of the National Yang Ming Chiao Tung University and Ministry of Education (MOE), Taiwan. We also thank the National Center for High-performance Computing (NCHC) for providing computational and storage resources.
BIBLIOGRAPHY
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34:29304-29320 2021.
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. BoTorch: A framework for efficient Monte-Cario Bayesian optimization. Advances in Neural Information Processing Systems. 11221122-215355 2020.
Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi- objective Bayesian optimization. Advances in Neural Information Processing Systems, 32, 2019.
Syrine Belakaria, Aryan Deshwal, Nitthilan Kannappan Jayakodi, and Janardhan Rao Doppa. -ncertainty-aware search framework for multi-objective Bayesian optimization. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10044-10052, 2020.
Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Output space entropy search framework for multi-objective Bayesian optimization. Journal of Artificial Intelligence Research, 72:667-715. 2021.
Nicola Beume, Boris Naujoks, and Michael Emmerich. SMS-EMOA: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research, 181 (3): 2007.
Michael Bowling, John Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In International Conference Machine Learning, pp. 3003-3020, 2023.
Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-Transformer: Scalable offline reinforeement learning via autoregressive q-functions. Conference on Robot Learning, pp. 3909-3928, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems,
2021.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Mare' aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Process- ing Systems, 35:32053 -20668. 2022.
Ran Cheng, Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. A reference vector guided evolutionary algorithm for many obje-ttje optimization. IEEE Transactions on Evolutionary Computation, 2075):773-791, 2016.
Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning, Advances in Neural Information Processing Systems, 30, 2017.
Sam Daulton, Maximilian Balandat, and Eytan Bakshy. Hypervolume knowledge gradient: A looka- head approach for multi-objective Bayesian optimization with partial information. In International Conference Machine Learning, pp. 7167-7204, 2023.

---

Published as a conference paper at ICLR 2025
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. Advances Neural Information Processing Systems, 33:9851 196644 2020.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems, 34:2187-2200,, 2021.
Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Multi-objective Bayesian optimization over high-dimensional search spaces. In Uncertainty in Artificial Intelligence, pp. 507-517, 2022.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multi- objective genetic algorithm: NSGA-IL. IEEE Transactions on Evolutionary Computation, 6(2): 182-197, 2002.
Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23(1):11627- 11680, 2022.
Michael Emmerich and winlwillem Klinkenberg. The computation of the expected improvement in dominated hypervolume of Pareto front approximations. Rapport technique, Leiden University, 2008.
Michael TM Emmerich, André H Deutz, and Jan Willem Klinkenberg, Hypervolume- expected improvement: Monotonicity properties and exact computation. In IEEE Congress of Evolutionary Computation (CEC), pp. 2147-2154, 2011.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2021.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 1704-1713, 2018.
Eduardo
and Daniel Hernández-Lobato. Predictive entropy search for multi- objective Bayesian optimization with constraints. Neurocomputiigg 361:50-68, 2019.
Rafael Bómeez-Bomarrelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández- Benjamín Sänchez-Lengeling Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Timothy D Hirzel,
Matthew Hausknecht and Peter Stone. Deep recurrent Q-learnnng for partially observable MDPs. In AAAI Fall Symposium Series, 2015.
Daniel Hernández-Looaae Jose Hernandez-LLoato, Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective Bayesian optimization. In International Conference on Machine Learning, pp. 1492-1501, 2016.
Bing-Jing Hsieh, Ping-Chun Hsieh, and Xi Liu. Reinforced few-shot acquisition function learning for Bayesian optimization. Advances in Neural Information Processing Systems, 34:7718-7731, 2021.
Iris Hupkens, André Deutz, Kaifeng Yang, and Michael Emmerich. Faster exact algorithms for computing expected hypervolume improvement. In International Conference on Evolutionary Multi-Criterion Optimization, pp. 65-79. Springer, 2015.
Carl Hvarfner, Frank Hutter, and Luigi Nardi. Joint entropy search for muximally-informed Bayesian optimization. Advances in Neural Information Processing Systems, 35:1 1494-1150666 2022.
Tommi Jaakkola, Satinder Singh, and Michael Jordan. Reinforcement learning algorithm for partially observable Markov decision problems. Advances in Neural Information Processing systems, 1994.
12

---

Published as a conference paper at ICLR 2025
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in Neural Information Processing Systems, 34:1273-1 1286, 2021.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast Bayesian optimization of machine learning hyperparameters on large datasets. In International Conference on Artificial Intelligence and Statistics, pp. 528-536, 2017.
Joshua Knowles. ParEGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1): 50-66, 2006.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Tor Lattimore, Marcus Hutter, and Peter Sunehag. The sample-complexity of general reinforcement learning. In International Conference on Machine Learning, pp. 28-36, 2013.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang. Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 25:27921-27925 2022.
Jan Leike. Nonparametric general reinforcement learning. PhD thesis, The Australian National
University (Australia), 2016.
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp. Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. SMAC3: A versatile Bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(1):2475- 2483, 2022.
Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen, et al. Reinforcement learning, bit by bit. Foundations and Trends@ in Machine Learning, 16(6);
733-865, 2023.
Wenlong Lyu, Fan Yang, Changhao Yan, Dian Zhou, and Xuan Zeng. Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design. In International Conference on Machine Learning, pp. 3306-3314, 2018
Sultan J Majeed. Abstractions of general reinforcement learning: An inquiry into the scalability of generally intelligent agents. PhD) thesis, The Australian National University (Australia), 2021.
Alexandre Maraval, Matthieu Zimmer, Antoine Grosnit, and Haitham Bou Ammar. End-to-end Baata-aa optimisation with Transformer neural processes. Advances in Neural Information Processing Systems. 2023.
Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 6561) 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 112.5211 2013.
13

---

Published as a conference paper at ICLR 2025
George E Monahan. A survey of partially observable Markov decision processes: Theory, models, and algorithms. Management Science, 28(1):1-16, 1982.
Gergely Neu and Ciara Fike-Burke. A unifying view of optimism in episodic reinforcement learning. Advances in Neural Information Processing Systems, 33:1392-1444 1403, 2020.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi- objective Bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelli- gence, pp. 766-776, 2020.
Victor Picheny. Multiobjective optimization using Gaussian process emulators via stepwise uncer- tainty reduction. Statistics and Computing, 2566): 2015.
Wolfgang Ponweiser, Tobias Wagner, Dirk Biermann, and Markus Vineze. Multiobjective optimiza- tion limited budget of evaluations using model-assisteddmmetr selection. In International Conference on Parallel Problem Solving From Nature, pp. 784-794. Springer, 2008
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference for Learning Representations, 2016.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-Just map them to actions. arXiv preprint arXiv:1912.02877 2019.
Alexander Shmakov, Avisek Naug. Vineet Gundecha, Sahand Ghorbanpour, Ricardo Luna Gutierrez, Ashwin Ramesh Babu, Antonio Guillen, and Soumyendu Sarkar. RTDK-BO: High dimensional Bayesian optimization with reinforced transformer deep kernels. In IEEE International Conference Automation Science and Engineering (CASE), pp. 1-8, 2023.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2012.
Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama. Multi-
objective Bayesian optimization using Pareto-frontier entropy. In International Conference Machine Learning, pp. 9279-922999 9288, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference Intelligent Robots and Systems, pp- 5026-5033, 2012.
Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi-objectie Bayesian optimization. Advances in Neural Information Processing Systems, 35:9922-9999 9938, 2022.
Tsuyoshi Ueno, Trevor David Rhone, Zhufeng Hou, Teruyasu Mizoguchi, and Koji Tsuda. COMBO: An Efficient Bayesian Optimization Library for Materials Science. Materials Discovery, 4:18-21, 2016.
Michael Volpp. Lukas P Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations, 2020.
Takashi Wada and Hideitsu Hino. Bayesian optimization for multi objei-obeecctve optimization and multi-point search. arXiv preprint arXiv: / 119900203770 2019.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning. pp. 3627-3635, 2017.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. volume 2. MIT Press, 2006.
Kaifeng Yang. Michael Emmerich, André Deutz, and Thomas Bäck. Multi-objective Bayesian global optimization using expected hypervolume improvement gradient. Swarm and Evolutionary Computation, 44:945-956 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. Advances in Neural Information Processing Systems, 31, 2018.
14

---

Published as a conference paper at ICLR 2025
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. MVImgNet: large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9150-9161, 2023.
Qingfu Zhang and Hui Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposi- tion. IEEE Transactions on Evolutionarr Computation, 11(6):712-73, 2007.
Qingfu Zhang, Wudong Liu, Edward Tsang, and Botond Virginas. Expensive multiobjective optimiza- tion by MOEA/D with Gaussian process model. IEEE Transactions on Evolutionary Computation, 14(4)456-474, 2009.
Qinging Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference Machine Learning, pp. 27042-27059, 2022.
Jinzhu Zhou, Zhanbiao Yang, Yu Si, Le Kang, Haitao Li, Mei Wang, and Zhiya Zhang. A trust-region parallel Bayesian optimization method for simulation-driven antenna design. IEEE Transactions on Antennas and Propagation, 69(7):396633981, 2020.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257-271, 1999.
15

---

Published as a conference paper at ICLR 2025
APPENDICES
Table of Contents
A Detailed Experimental Configuration
16
A.1 Environment
16
Hyperparameters of Learning- Based Approaches
A.3 Hyperparameters of Rulc-Based Approaches
17
B Additional Related Work
17
B.1 Transformers and Sequence Modeling for RI.
B.2 A More Detailed Comparison to Q-Transformer
17
C Pseudo Code and Additional Implementation Details
18
C.I. Pscudo Code BOFormer
18
C.2 Additional Implementation Details of BOFormer
18
C.3 About Temporal Information in the History Representation of BOFormer
21
Issues With the Direct Implementation Section 4.1
21
C.5 Implementation Details of OptFormer...
22
D Detailed Experimental Results
22
Detailed Hypervolume Statistics and Inference time
22
D.2 Ablation Study on the Demo Policy in BOFormer
22
D.3 Scalability of BOForerr High- figh-Dimensiooal Problems
26
Performance of BOFormer More Challenging Problems
27
E About the Non-Markovian Nature of MOBO
27
A DETAILED EXPERIMENTAL CONFIGURATION
A.1 ENVIRONMENT
In each testing episode, the agent interacts with the environment for total of 7 100 for synthetic functions and T' = 30 time steps for HPO-3DGS. The observed function values subject noise <~ (0, 0.1) for synthetic functions, while the fuucciin values for HPO-3DGS are observed without noise. During testing and training, the objective functions of each episode are scaled by perturbation noise which is sampled from Uniform(0 0.1) for synthetic functions and Uniform 0.01) for other functions.
For HPO-3DGS, there are about 68,000 data points related to 4 different objects: Lego, Materials, Mic, Ship, and Chairs. Additionally, there are different scenes involving chairs. While testing on chairs, each episode is conducted with an individual scene.
Configuration of the Surrogate Model:
Training: For all the learning-basee algorithms, we used the same GP surrogate model with an RBF kernel for obtaining the posterior distributions. Regarding the generation of GP synthetic functions as objective functions during training, each function is generated randomly under either RBF Matemn-52 kernel with lengthscale sampled from [0.1, 0.4).
Testing: During testing, all algorithms utilized surrogate model based on the Matern52 kernel. The length scale was estimated online by maximizing the marginal likelihood by following the same approach as BoTorch.
16

---

Published as a conference paper at ICLR 2025
A.2 HYPERPARAMETERS OF LEARNING- BAANEN APPROACHES
BOFormer:
hidden size: 128 for all linear layers used to embed positional encodings. state- action pairs, rewards, and Q-values, learning rate: 10 10-5, weight decay: 10 Tdemo: 0.01, batch size: 8, number attention layer: 8, number of head of attention layer: 4, window size w 31, dropout: 0.1, buffer size: 64, training episode: 3000.
DT" hidden size: 128, learning rate: 10-4, weight decay: 10-4, batch size: 16, number attention layer: 3, number of head attention layer: embed dim: 500, dropout: 0.1,
warmup steps: 10, max length: 100.
QT: hidden size: 128, learning rate: 10-6, weight decay: 10-5, Tdemo: 0.01, batch size: 8, number of attention layer: 8, number of head of attention layer: 4, window size w 21, dropout: 0.1, buffer size: 64, training episode: 2000.
FSAF: alpha: 0.8, hidden size: 100, learning rate: 0.01, batch size: 128, few shot step: 5, number of particles: total task: 3, size of meta data: 100, use demo: True, early terminate:
False, select type: average, training episode: 300 for K = 2 and 500 for K 3.
OptFormer: string length: 128, learning rate: 10-2, weight decay: 10-2, batch size: 1,
window size w 10, training episode: 1000.
Common Hyperparameter: optimizer: Adam (Kingma & Ba, 2015), e-greedy rate: 0.1
Transformer Architecture: GPT-2-based Transformer architecture.
A.3
HYPERPARAMETERS OF RULE-BASED APPROACHES
NNEHVI: q(batch selection): 1
qHVKG: q(batch selection):
qParEgo: q(batch selection): 1, acquisition function: Expected Improvement
JES: # of samples: 64, estimation type: LB
NSGA2: population size: 10, # of generations: 10, sampling: random
B ADDITIONAL RELATED WORK
B. TRANSFORMERS AND SEQUENCE MODELING FOR RL
Given the success of requeene-to-sequeee models in language processing, RL has recently been addressed through the lens of sequence modeling, especially transformers. For example, Chen et al. (2021) proposed Decision Transformer (DT), which addresses offline RL by mapping return-to-go to actions via transformers and thereby substantiating the concept of Upside Down RL (Schmid- huber, 2019). The design of DT has been subsequently extended to various settings, including online RL (Zheng et al., 2022), multi-game setting (Lee et al., 2022), and general information matching (Furuta et al., 2021). Concurrently, to tackle offline RL, Janner et al. (2021) propose Trajectory Transformer (TT), which serves as a predictive dynamics model and uses beam search as a trajectory optimizer. More recently, Chebotar et al. (2023) introduced Q- T-Tansforrer (QT), which is trained by offline temporal difference updates and achieves scalable representation for Q-functions by discretizing action dimensions tokens contrast, the proposed BOFormer is built on the (online) generalized DQN framework and designed for addressing MOBO based on multiple enhancements.
Transformer, with focus robotic tasks. By
B.2 A MORE DETAILED COMPARISON TO Q-TRANSFORMER
Q- Q-Transformer (QT) (Chebotar et al., 2023) similarly incorporates the idea of utilizing Transformer in learning the Qffunction. Despite this high-level design resemblance, the proposed BOFormer is different from QT in multiple aspects:
Problem Setting: QT is designed mainly to address offline RL, where the performance is highly correlated the quality of prior data, robotic learning approach and required to address
"We the open source implementation from http://shhicccoommmmmmmmiiirroot
17

---

Published as a conference paper at ICLR 2025
high-dimensional action spaces. By contrast, work introduces BOFormer, which is trained to solve MOBO by learning a generalized Q function based on the online interactions with synthetic GP functions, and hence this can be viewed an instance of online RL.
Network Architecture: QT uses state-action sequences as the input of the Transformer. By contrast, to resolve the hypervolume identifiability issue and achieve cross-domain transferability simultaneously, the proposed transformer of BOFormer implements generalized DQN and uses the Q-aummented observation representation as the input of the transformer. This approach can better address the non-Mait Mon-Markovian property in MOBO.
Training Algorithm: Compared to QT, the proposed BOFormer incorporates multiple practical cahancements, including Q-augmented representation, reward normalization, and demo policy and prioritized trajectory replay buffer for off- off-policy learning.
C PSEUDO CODE AND ADDITIONAL IMPLEMENTATION DETAILS
C.1 PSEUDO CODE OF BOFORMER
The detailed pscudo code of the training processes for BOFormer under off-policy learning and on-policy learning setting are provided in Algorithms and 2, respectively.
C.2
ADDITIONAL IMPLEMENTATION DETAILS OF BOFORMER
Reward Signal With Normalization. In the MOBO setting, one natural reward design is the one-step improvement in hypervolume, i.e., ft := HV(X) - HV However, as the achieved hypervolume increases, the reward signal f. can get weaker in the later stage of an episode, making it difficult to recover the whole Pareto front. To address this, we construct rr. which f but scaled by the difference between the current hypervolume and optimal hypervolume, the reward signal for RL in MOBO, i.e.,
HV(A)) HV Pt HV(X** HX))
Remark C.1. The information about HV (2**) in (7) is used only during training and can be easily pre-computed approximated given the knowledge about the domain and the black-box functions. Demo-Policy-Guider Exploration. To facilitate the off-policy learning of BOFormer, one natural approach is to adopt behavior policy with randomized exploration (e.g.. opsilon-greedy: for col- lecting trajectories from the environment. However, such a randomized exploration scheme can be very inefficient as the sampling budget in MOBO is usually much smaller than the domain size (i.e., the number of actions). To better guide the exploration, we propose to use a demo policy, which is possibly sub-optimal but of sufficient strength in exploring regions near the Pareto front. In practice, we set Tdemo to be the probability of using demo policy induced by an off-the-shelf AF for MOBO (e.g., EHVI) in this training episode. The training processes of BOFormer for off-policy learning and on-policy learning are provided in Algorithms and 2, respectively.
Transfer Learning Across Different Numbers of Objective Functions. As discussed in the main text, the input dimension of BOFormer varies depending on the number of objective functions. For instance, a BOFormer model trained on objective functions cannot be directly applied objective functions due to the mismatch in input dimensions, which requires reshaping of the embedding layer. To address this issue and reduce the computational burden of training BOFormer, we developed an efficient transfer learning strategy that enables model transfering across different numbers of objective functions.
Our approach first loads the pre-trained weights for the transformer component from another trained BOFormer, then initializes a linear embedding layer to match the dimension of the state-action pair. Only the linear embedding layer is trained. This strategy significantly reduces training episodes and computational requirements by allowing us to leverage pre-trained models for different numbers of objective functions with minimal adjustments.
The efficiency of this transfer learning method evident in the reduced number of training episodes. While the original model requires approximately 3000 episodes to converge, the transfer model
18

---

Published as a conference paper at ICLR 2025
achieves comparable performance after only 400 episodes of fine-tuning. This substantial reduction in training episodes demonstrates the effectiveness of our approach in rapidly adapting pre-trained models to objective function configurations.
Positional Encoding. The positional encoding in BOFormer is designed specifically for the context of sequential modeling in RL.. Unlike the standard positional embeddings used in traditional Trans- formers, where each position corresponds to a single token, our method assigns an embedding to each timestep that is shared across multiple tokens (i.e., {(5,, (e); Te, (st,, at)} for state-action pairs, rewards, and Q-values). This approach allows the Transformer to capture the temporal context effectively by providing explicit time indices for each token.
Temporal Information in State-Action Representation. Temporal information is also incorporated into the state-action representation, specifically through terms like j/t in (5). As BOFormer is an RL-based method, this essentially corresponds to the episodic RL setting (Dann et al., 2017; Neu & Pike-Burke, 2020), where the length of each episode fixed. episodic RL, the value functions (T, a) and V (s, a) do depend on this temporal information & because the RL agent shall make decisions based on the remaining time budget (i.e., 1). In the context of MOBO, this temporal information also serves
budget indicator, similar to that in episodic RL. This is critical for learning MOBO policies that can achieve high hypervolume in as few queries as possible by maximizing cumulative reward within the given budget. In summary, the motivations for these two temporal components differ:
Positional Encoding ensures the Transformer can capture temporal information in the historical queries.
State-Action Temporal Representation directly informs the RL agent of the progress within sequential decision-makinn with respect to the remaining budget, helping maximize future cumulative rewards.
Regarding the ablation studies for validating the effects of positional encoding and temporal infor- mation, conduct preliminary ablation study BOFormer by comparing several variants of BOFormer models:
1. With Positional Encoding and Temporal Information: This is the original BOFormer.
2. Without Temporal Information: Temporal information is removed from BOFormer during training.
3. Without Positional Encoding nor Temporal Information: Both components were excluded from BOFormer during training.
To better observe the training behavior, we collected all training data using the demo policy and monitored the evolution of the loss curve during training. The rest of the configuration, such as hyperparameters and neural network architecture, was consistent across all three variants. The evolution of the training loss for the three variants can be found in Figure 5 We observe both second and third BOFormer appear to suffer from divergence in terms of training loss. These results appear to align with the requirements of episodic RI, where knowing the budget is important for estimating Q-values. Temporal information plays a vital role in the need of this budget-related representation, while positional encoding essential for enabling the Transformer to recognize each position as corresponding to single token. Together, these components are helpful to BOFormer's 's stability and convergence during training.
Off-Policy Learning and Prioritized Trajectory Replay Buffer These components are introduced to address exploration, sample efficiency and learning stability:
Off-Policy Learning enables BOFormer to leverage previously collected data, reducing the need for additional costly sampling. This improves sample efficiency and facilitates the use of a demo policy, allowing for more effective learning and faster convergence.
Prioritized Trajectory Replay Buffer ensures that the most informative transitions (e.g., those with significant temporal-difee difference errors) are replayed more frequently during training. accelerating convergence.
19

---

Published as a conference paper at ICLR 2025
Loss Curves
Temporel BoFarmer we Tamporal information & Pasitional Ercading Borormer wi Temporal Intormation Positional Encoding
1000
1O55
200
230
300
150
3000
12:0
1500
3750
2020
Gradlent Steps
Figure 5: Training loss for variants of BOFormer models
Handling Continuous Domains Under BOFormer. As BOFormer is a learning- based acquisition function (AF) implemented as neural network, it can nicely handle continuous domains by leveraging the common ways of finding an (approximate) AF maximizer, just like the existing learning- based methods and neural AFs. Specifically:
Approximate maximization by Sobol grids (commonly used by learning-batc based methods): One can find an approximate maximizer of each learning-basee -baed AF by grid search with the help of Sobol sequences, as suggested by (Volpp et al., 2020; Hsieh et al., 2021). Specifically, one can (i) first selects N points spanning over the entire domain with the help of Sobol sequence, (ii) chooses the top-M M points in terms of AF value among the N candidates, (iii) and then builds local Sobol grid of K points for each of these M points. This is the default choice of BOFormer.
Gradient-based optimization with automatic differentiation (used by the sule-based meth- ods in BoTorch): One can also optimize the AF by using gradient-based optimization (e.g., multi-start L-BFGS-B in BOTorch available at https : //btttrcchhhhhccc orgg docs/ / optimi zation. ion, with the help of automatic differentiation provided by deep learning frameworks (e.g., Py PyTorch). This is the default approach adopted by BoTorch.
Q-Value Computation by Target Network for Trajectory. Given a trajectory 7 {81, 71,71, $2, 8t-1; rt-13 (i) BOFormer computes Qhg, $1, a1) using ($1, @1) as the input. (ii) Then, it computes Q(h1, $2, 02) by letting ($1, a1, 71, $2, a2) as the input. (iii) This process continues sequentially until Q(h--2, S1-1, ac-1) is computed.
Action Selection by Policy Network at time t. To select a, at time I, BOFormer sam- ples actions from softmax policy where the logits correspond to Q-values: Pr(a[he-1,
exp ((hh-1,)t,a)) Enex exp (Q(h--1.t..a)))
Other Training Details.
Each trajectory is collected using the demo policy with probability of raema: otherwise, it is collected by BOFormer itself.
Demo policy:
Batch sampling: BOFormer samples batch from the prioritized trajectory experience replay buffer, with each batch containing batch of trajectories.
Optimization: Gradient descent is performed on the policy network using the optimizer with dropout.
Target network: The target network is used to compute (Q(h4-1, 8t, [= and synchro- nized with the policy network every 5 episodes.
20

---

Published as a conference paper at ICLR 2025
C.3 ABOUT TEMPORAL INFORMATION IN THE HISTORY REPRESENTATION OF BOFORMER
As BOFormer is a non-Markovian RL method for BO, and hence the design shall take both RL and BO into account. Specifically:
Temporal information in state-action representation serves as budget indicator, similar to episodic RL: In MOBO, we do want to attain high hypervolume by using as few samples possible, and hence each episode only has small number of queries (or time steps), e.g.. T 100. As BOFormer is an RL-based method, this essentially corresponds to the episodic RL setting (e.g.. [Dann et al., 2017; Neu and Pike-Burke, 2020J), where the length of each episode is fixed. In episodic RL, the value functions Q7 (8, a) and Vi" a) do depend on this temporal information because the RL agent shall make decisions based on the remaining time budget (i.e. T - t). In the context of MOBO, this temporal information also serves as a budget indicator, similar to that in episodic RL.
Anothe notable fact is that in BO, the historical queries shall be permutation-invarian in the sense that the order of previous queries should not affect the posterior distributions under GP.
permulaatoo-invarinnce of queries in posterior distributions:
By taking both into account, choose include the temporal information in the state-action representation.
C.4 ISSUES WITH THE DIRECT IMPLEMENTATION IN SECTION 4.1
Recall that the direct implementation in Section 4.1 is designed to implement Generalized DQN with the per-step observation that consists of two parts:
1. The posterior distributions of all K objective functions all the domain points.
2. The best function values observed so far (denoted by y(( i 1,.., K).
Hence, the per-step observation is (11()) ( (x), be)x,)]]]))) As described in Section 4.1, this representation design is subject to two practical issues:
Scalability issue in sequence length: Under this design, the sequence length of the per-step observation would grow linearly with the number of domain points and pose a stringent requirement on training.
Limited cross-domain transferability: As this observation representation is domain- dependent, the learned model is tightly coupled with the training domain and has very limited transferability.
We have tried this direct implementation, and we found that this design severely suffers from the scalability issue and requires extremely long training time. Below we report the training time that we observed:
Even under fairly small input domain of only 50 points (Le.. X 50), observe that 20 training iterations already take more than 36 hours of wall clock time to complete. **oo finish least 1000 training iterations (based our training experience with BOFormer), this direct implementation would require about 75 days to complete one training run.
Notably, the above training was already on high-end GPU server with NVIDIA RTX 6000 Ada Generation GPUs and Intel Xeon Gold 5515+ CPU.
The above issue will be more severe under input domain with more domain points (i.e., larger XX). This manifests that this direct implementation suffers significantly from the scalability issue and is rather impractical. This also motivates the proposed BOFormer for substantiating the Generalized DQN framework.
21

---

Published as a conference paper at ICLR 2025
C.5 IMPLEMENTATION DETAILS OF OPTFORMER
Recall that the OptFormer and BoFormer have different application scopes: BOFormer is designed to serve as a general- general-purpose purpose optimization solver for MO black-box optimization, whereas OptFormer is more task-specific as it is designed specifically for single-objective HPO and requires offline HPO data for training. To adapt OptFormer to the general MOBO setting, there are some modifications needed: (i) The core idea of OptFormer is to recast HPO as language modeling. and we adapt this idea to MOBO. For instance, when training OptFormer for a 2-objective MOBOsetting we modify the input to include representation that is more relevant to the MOBO problem, such
"Ist_ ji_ddmmensio_x = 0.031 2nd_dimension_> = 0.531
1st lst_objective__ = 0.254 2nd_objective_y 0.258
1st st__dimension_x = 0.604..."
(8)
To ensure the strength of OptFormer, we retrain the OptFormer model upon handling different numbers of objective functions. (ii) Additionally, the metadata includes information derived from the GP surrogate model, which is estimated online by maximizing the marginal likelihood (just like all the other rule-based AFs and BOFormer). Given that OptFormer requires an offline dataset for training, we collected the training dataset using rule-based AFs (e.g., Expected Hypervolume Improvement), to ensure the model effectively tailored to the MOBO setting.
D DETAILED EXPERIMENTAL RESULTS
D.1 DETAILED HYPERVOLUME STATISTICS AND INFERENCE TIME
In this section, we provide the additional and detailed statistics of the per-sample inference time and the final hypervolume follows.
Table 3: Average per-sample inference time of BOFormer and other benchmark methods.
Time (millisecond)
Algorithms
Objectives
Objectives
Objectives
BOFormer
14.20
14.20
14.29
QNEHVI
6.15
8.042
224.10
JES
46.66
50.15
57.87
ESAF
5.49
6.55
6.92
DT
1.30
1.28
1.28
NSGA2
0.17
0.19
0.20
gParEGO
1.10
1.42
1.75
OptFormer
237.33
366.15
485,51
qHVKG
371.66
533.37
662.76
QT
9.37
9.71
9.73
The messages from Table 3 mainly three-fold:
We observe that for the baselines that competitive in the attained hypervolume (such as QNEHVI and qHVKG), the inference time scales significantly with the number of objectives, whereas BOFormer maintains lower and nearly constant inference time across different number of objectives.
Compared to the Markovian approach like FSAF, BOFormer only needs a slightly higher per-step inference time and remains rather efficient. Hence, the non-Markovian nature of BOFormer still consumes an inference time comparable to a Markovian approach.
BOFormer enjoys much lower inference time than the sophisticated OptFormer, which views black-box optimization as a language modeling problem in a text-to-text manner and hence involves text-based input representation.
D.2 ABLATION STUDY ON THE DEMO POLICY IN BOFORMER
22

---

Published as a conference paper at ICLR 2025
Algorithm 1 Off-Policy BOFormer
Input: 0,. Training Episodes E, Time Horizon T, Demo Rate Idemo- Target Rate Target Target Network Qg. Batch Size B, Replay buffer B, environment &
2: for 1,2, E do
3: E.rese(()
4: of Etstate
5: demo random. random.binomian binomial((ammm))
6: पł = Expeceattyyrvolumemmpovmma
7:
for t 1,2, ,T' do
8:
demo then o or (sf)
9:
10:
else
for i 1,2, ,t- do
12:
Compute Qo(hs of = Qi ((((())))
of (x2))
13:
end for Select x; == argmax Q3 (h, (x)) end if
14:
15:
16:
1 rf E.tep((x;)
17:
end for
18:
if mod Tharget then
19:
QB Qo.
end if
21:
B.pppedd(T. {o (xf), r rf
22:
Sample batches from
23:
for b = 1....... B do
24:
Compute (2 of (x+) = Qi (o((z))) for i 1, ... T do
25:
26:
Compute Qg(,, (z(2)) = Qi (cothhh+rr.000033,,,,,0,,1111,13
Compute 02h (x!))) = Q0 ((kkeh)er3.00000..o0h11111311111
27:
end for end for
29:
30: Loss 0) = E 1= EtS
03 of(())) (r + max
31:
Oc+1 argmin Loss(0)
32: end for
Algorithm 2 On-Policy BOFormer
1: Input: 0,, Training Episodes E, Time Horizon T', environment E,
2: for 2, do
3: E.reset()
4: $1 Esstate
5: for ( = 1,2,... ... T do
6: Select F4 == argmaxxex Qo, (he, oe()))
7:
(4+1()), = E.step(r:)
8: 0.+1 argmin Qo (( - (rr max Qo, (1e41,04+11)))
xEX
9: end for
10:
end for
To further investigate the connection between BOFormer's performance and the choice of demo policy, conducted additional experiments comparing QNEHVI (the original choice) and NSGA2 as demo policies, as well as baseline where no demo policy is used.
The results available Figure 10. Based on these results, observe the following:
23

---

Published as a conference paper at ICLR 2025
DOPormer (Oun)
Optiormn
N5GAZ JJ3
FSAF
gVVKK
NEHV
0.45
0.5
0.4
0.4
Hypervolune
Hypervohme
0:25
0.2
0.2
0.15
10
0.1
100
100
Samples
Number Samples
Number Samples
(a) AR
(b) ARa
(c) BC
0.8
0.8
0.7
0.8
Hyperoluue
Hypervolume
0.7
0.6
0.4
0.4
0.5
0.3
Number Samples
Number of Sumples
Number of Samples
(d) DRa
(e) RBF
(f) Matern52
Figure 6: Averaged attained hypervolume under synthetic functions with two objectives.
BiFurrae (Oux)
Opt Parmur
NSGA2
DT
PSAF
gVKK
NEIVI
ParBGG
0.9
0.9
0.85
0.9
0.85
0.86
0.8
0.75
Hypervolume
0.8
Hypervolum
Hypervoluma
0.7
0.75
0.65
0.7
0.65
0.6
0.5
0.55
20
10
30
10
Numher Samples
Numher Samples
Number Samples
(a) Chairs
(b) Materials
(c) Lego
Figure 7: Averaged attained hypervolume under HPO-3DGS with two objectives.
BOFormer (w/i QNEHVI) achieves favorable improvement in attained hypervolume than that without using demo policy, This demonstrates the benefit of off-policy learning via a demo policy in the context of MOBO. By contrast, the improvement offered by the NSGA2-basel demo policy appears minimal.
Recall that NNEVVI performs generally better than NSGA2 on these tasks (f.e., RBF, Matern, and BC) as shown Table of the original manuscript and the performance profiles in Figure 3. Intuitively, would expect that the trajectories contributed by qNEHVI can
24

---

Published as a conference paper at ICLR 2025
D.Former [Oun)
Optiormn gVVK
NSGAZ -..163
FSAF
NEHV
0.35
0.3
0.95
0.8
0.25
0.75
Hypevolane
Hypervolume
0.7
0.9
0.85
0.6
0.1
0.5
50
100
10
30
10
20
Number of Samples
Number of Samples
Number of Samples
(a) BCD
(b) Mic
(c) Ship
Figure Averaged attained hypervolume under synthetic and HPO-3DGS with three objectives.
BUFurraer (Ouns)
Opt Farmur
NSGA2
-ES
PSAF
&NEIVV
Fraction of episodes with IVV
Fraction of episoees with HV>
0.8
0.8
0.8
Fraction of eplaodes with
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.5
0.5
0.5
Attained IIV at the final step (r)
Atsained IIV at the final step (r)
Attoined IIV at the final step (r)
(a) DR
(b) Matern52
(c) RBF
Fraction of episodes with HV> 7
Fraction of episodes with HV>
Fraction opisodes with HV>
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0
0.5
0.3
0.4
0.5
0.6
0.7
0.7
0.8
0.9
Attained at final step (7)
Attained HV at the final step (v)
Attained at the step (r)
(d) BCD
(e) Mic
(f) Ship
Figure 9: Performance profiles in terms of hypervolume the final step under synthetic functions and HPO-3DGS.
better help BOFormer explore the regions with higher hypervolume. This intuition also resonates with the general understanding that RL performance can be correlated with the data quality (Kumar et al., 2019).
25

---

Published as a conference paper at ICLR 2025
deso palcy)
(CNELVVI)
0.7
0.5
0.0
0.45
0.5
0.4
0.3
Hypervolume
0.35
Hypervolume
0.4
Hypervoluee
0.25
0.3
0.2
0.25
0.2
0.1
0
50
100
50
Number of Samples
Number of Samples
Number Samples
(a) AR
(b) ARa
(c) BC
0.9
0.8
0.8
0.85
0.7
0.7
0.8
Hyperolune
Hypervolumee
0.7
0.65
0.6
0.4
0.4
0.53
0.3
0.5
Number of Samples
Number of Samples
Number of Samples
(e) DR
(f) Matern52
(g) RBF
Figure 10: Averaged attained hypervolume under synthetic functions with two objectives.
D.3 SCALABILITY OF BOFORMER TO
PROBLEMS
We conducted additional experiments on black-box functions with higher-dimennsonal domains, including:
Matern52 and RBF functions under 10 and d 30.
(Ackley & Rosenbrock) and (Ackley & Rastrigin) under d = 40.
DTLZ functions under d 100. Notably, DTLZ is family of synthetic functions for MOBO and included in the pymoo library.
DOFormer (Ours)
NETVI PPPE0O JES FSAF
0.7
0.5
0.8
0.5
2.45
0.4
040
05
20
100
20
100
50
100
100
Number of Sampies
Number of Sucpples
Ntamber of Scapples
Number Samples
(a) RBF 10D
(b) Matern52 10D
(c) RBF 30D
(c) Matern52 30D
Figure 11: Averaged attained hypervolume under challenging GP functions with higher- dimensional domains.
26

---

Published as a conference paper at ICLR 2025
+ ROFormer (Oung) + NRHVI QPoRGG -
0.20
0.55
0.15
1120
0.14
Hyparsalume
0.20
Hypurvaluano
0..0
0.12
0.18
0.11
0.16
GO
100
50
100
50
200
150
200
Number Samples
Number Sumples
Sumplees
(a) AR 40D
(b) ARa 40D
DTL.Z 100D
Figure 12:
Averaged attained hypervolume under various challenging functions, including synthetic functions with 40-dimensional domains and the DTLZZ function with 100-dimensional domain.
Figures 11-12 demonstrate that BOFormer remains competitive compared to other MOBO baseline algorithms on these ligh-dimensiona tasks, while FSAF requires metadata for few-shot updates,. Notably, all the above results are obtained under the same BOFormer model without any fine-tuning, and hence this further demonstrates the strong cross-domain transferability of BOFormer.
D.4 PERFORMANCE BOFORMER ON MORE CHALLENGING PROBLEMS
We conducted additional experiments on various challenging problems with more than 100 samples, including scemarios with (1) higher domain dimensionality and (2) increased perturbation noise. Accordingly, we set T = 200 or all the following experiments to better evaluate BOFormer and other baselines. Figure 13 demonstrates that BOFormer remains competitive compared to other MOBO baseline methods across various types of challenging scenarios. These findings highlight BOFormer's scalability, making well-suited for more challenging black-box functions.
Remark D.I. make the problems more challenging as suggested, for the HPO-3DGS tasks, the perturbation noise is set 0.1 make the black-box functions more non-smooth.
Remark D.2. In the above, we compare BOFormer with the rule-based methods such as NNEHVI, JES, and qParEGO as well as the learning- based method like FSAR These MOBO methods have been shown to be the most competitive among all the baselines (cf. Tables 1-2)
Remark D.3. FSAF is metaRL-based method and requires some metadata for few-shot adaptation. By contrast, BOFormer evaluated on unseen testing functions in zero-shot manner.
E ABOUT THE NON-MARKOVIAN NATURE OF MOBO
Defining the state representation the complete of historical queries along with the posterior distribution of candidate points would indeed endow the problem with full observability. However, by the standard definition of Markov property, this design is not Markovian (and hence does not result in an MDP) because the definitions of the Markov property and MDP presume that the state space is of fixed dimensionality. Specifically, this design involves two fundamental issues:
The dimensionality of the state space would increase across time steps as more samples are taken: To illustrate this more concretely, let us consider an example of K K-objective MOBO problem with objective functions f(x) = (f)( ... fx (x)) with f(z) : X - R where the domain X discrete and has N candidate points (a similar argument can be used for continuous domains). Recall that we use (z).ys) to denote the query and the resulting observed function values at time step i. In this design, at the t-th step of an episode, the state representation would consist of: (i) The complete of historical queries
27

---

Published as a conference paper at ICLR 2025
+ ROFormer (Ours)
NRRVI GPoRGG -
1.05
Hypavoluma
0.9
Hypervoluza
Hypervolume
0.95
0.95
0.8
0.9
0.9
0.7
0.85
0.8
0.8
0.8
0
50
100
150
200
50
100
150
200
0
50
100
150
200
Number Samples
Number Samples
Number Samples
(a) chairs
(b) lego
(c) materials
1.1
1.2
0.38
Hypavolune
0.9
Hypervolune
1.15
0.36
0.8
1.1
0.34
0.7
0.32
0.6
150
200
50
100
150
50
100
150
200
Number Samples
Number Samples
Number Samples
(e) mic
(f) ship
(g) DTLZ 100D
Figure 13: Averaged attained hypervolume under more challenging problems with 200 samples
is y1 yi),..., (Ir, y:)) The dimensionality of this part is ((K 1). (ii) The posterior
distribution of candidate points can be captured by (11(2), a (x)) ( The dimensionality of this part is 2.N. Therefore, we can see that the dimensionality of the state space would increase with time step t.
The state representation would be tightly coupled with the input domain: Based on the above first point, can that under this design, part of the state representation involves { y.), (Iz, y:)}- This suggests that the state space is coupled with the domain of each specific task. As result, re-training is typically needed upon handling new task. This issue motivates the need and design for BOFormer, which has the favorable cross-domain capability (i.e., the size and the dimensionality of the input domains can be different between the training and testing).
28


Reviews:
[{"review_id": "52A2E6eVOd", "reviewer": "Reviewer_91dP", "summary": "The paper introduces BOFormer, a novel approach for tackling multi-objective Bayesian optimization (MOBO) challenges, particularly the non-Markovian nature and hypervolume identifiability issues. BOFormer employs non-Markovian reinforcement learning and sequence modeling, leveraging the Transformer architecture to optimize long-term outcomes. The method demonstrates superior performance in various synthetic and real-world multi-objective optimization scenarios.", "questions": "1. How does the time efficiency of your proposed algorithm compare to the baseline algorithms?\n2. What is the intended meaning of the experimental results in your Figure 3?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces an innovative approach by framing multi-objective Bayesian optimization (MOBO) as a non-Markovian reinforcement learning problem. This represents a creative combination of existing ideas from non-Markovian RL and Transformer-based sequence modeling, marking a fresh perspective in the field.\nThe use of diagrams and examples, such as the hypervolume identifiability issue, aids in understanding complex concepts.", "weaknesses": "1. The discussion of shortcomings in the paper is relatively brief and does not clearly articulate the innovative aspects of the work. Furthermore, the contributions appear to be somewhat incremental rather than groundbreaking.\n2. The use of Transformers may require substantial computational resources and memory, which could limit accessibility for some users.\n3. The explanations regarding the experimental section lack clarity. The paper does not specify how the proposed algorithm's time efficiency compares to that of other algorithms. Additionally, it is unclear whether the non-Markovian nature of the process consumes more time than a Markovian approach. The improvements over baseline results also do not appear to be significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Py501MvFCN", "reviewer": "Reviewer_62Xe", "summary": "This paper introduces BOFormer, a novel learning-based acquisition function for multi-objective Bayesian optimization (MOBO) that combines reinforcement learning and sequence modeling with Transformers. BOFormer addresses the hypervolume identifiability issue in MOBO, which stems from its non-Markovian nature, by presenting a Generalized DQN framework and substantiating it through sequence modeling.\nSeveral practical enhancements, such as Q-augmented observation representation and prioritized trajectory replay buffer, are incorporated to facilitate the training of BOFormer. Extensive experiments on synthetic and real-world hyperparameter optimization problems demonstrate that BOFormer consistently outperforms various benchmark methods, exhibits cross-domain transfer capabilities, and efficiently transfers learning across different numbers of objective functions.", "questions": "How does the Generalized DQN framework differ from the standard DQN in terms of theoretical guarantees and convergence properties? Can the authors provide more insights into the theoretical foundations of their approach?\n\nThe paper introduces several practical enhancements, such as Q-augmented observation representation and prioritized trajectory replay buffer. How do these enhancements contribute to the performance of BOFormer individually, and are there any potential synergies or trade-offs between them?\n\nThe demo-policy-guided exploration is an interesting concept. How sensitive is BOFormer's performance to the choice of the demo policy, and what are the characteristics of a good demo policy for MOBO problems?\n\nThe paper showcases BOFormer's cross-domain transfer capabilities and efficient transfer learning across different numbers of objective functions. Can the authors elaborate on the potential limitations of this transfer learning approach and discuss scenarios where it might not be applicable or effective?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "One of this paper's key strengths is its novel approach to addressing the hypervolume identifiability issue in multi-objective Bayesian optimization (MOBO). By presenting the Generalized DQN framework and implementing it through BOFormer, the authors tackle MOBO's inherent non-Markovian nature. This innovative perspective of reinterpreting MOBO as a sequence modeling problem using Transformers allows for a more effective and efficient solution to the identifiability issue.\nAnother strength lies in the practical enhancements introduced to facilitate the training of BOFormer. The Q-augmented observation representation provides an informative indicator of the prospective improvement in hypervolume while maintaining a domain-agnostic and memory-efficient representation. The prioritized trajectory replay buffer and off-policy learning enable better convergence and data efficiency during training. Furthermore, the demo-policy-guided exploration ensures efficient search space exploration, which is particularly important given the limited sampling budget in MOBO.", "weaknesses": "Limited theoretical analysis: Although the paper introduces the Generalized DQN framework and provides empirical evidence of its effectiveness, it lacks an in-depth theoretical analysis of the proposed approach. A more rigorous theoretical foundation could help better understand BOFormer's convergence properties, optimality guarantees, and limitations.\n\nScalability to high-dimensional problems: While BOFormer performs well on the tested problems, its scalability to high-dimensional MOBO problems is not explicitly addressed. As the number of dimensions increases, the search space grows exponentially, potentially impacting the algorithm's performance. Further investigation into the scalability of BOFormer and performance on high-dimensional problems would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Zf1L8Nbxh", "reviewer": "Reviewer_tuhd", "summary": "This paper introduces BOFormer, a novel approach to multi-objective Bayesian optimization (MOBO) that leverages non-Markovian reinforcement learning. The authors' primary contribution is addressing the hypervolume identifiability issue in MOBO through a generalized DQN framework implemented via a Transformer architecture. The work introduces several practical enhancements, including Q-augmented observation representation, prioritized trajectory replay buffer, and demo-policy-guided exploration. The method is comprehensively evaluated on both synthetic functions and real-world hyperparameter optimization tasks for 3D Gaussian Splatting, demonstrating competitive performance against existing approaches. The authors also show promising transfer learning capabilities across different numbers of objective functions.", "questions": "Could you provide details about the experimental methodology, specifically:\n- How many random repetitions were performed to ensure statistical significance?\n- Were the same network architectures used consistently across all experiments?\n- What are the specific architectural details of the Transformer implementation?\n- How was stability analysis conducted? ie, stability wrt different transformer models and also different initialization.\n\nWould it be possible to present results on more challenging problems requiring significantly more than 100 samples? The current evaluation up to 100 samples may not fully reflect real-world BO scenarios.\n\nThe method's effectiveness with limited data (approximately 10 samples) is surprising given the typically large parameter count of Transformer architectures. Could you elaborate on how this is achieved?\n\nHow does the method's performance scale with increasing problem complexity and dimensionality?\n\nCould you provide a more detailed justification for choosing a Transformer architecture in this small data regime compared to simpler alternatives?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors have identified and thoroughly addressed a fundamental issue in learning-based MOBO approaches - the hypervolume identifiability problem. This represents a significant contribution to the field. The theoretical framework connecting non-Markovian RL to MOBO is rigorously developed and mathematically sound. \n\nThe experimental evaluation is comprehensive, comparing the method against both classical and learning-based baselines across diverse scenarios.", "weaknesses": "The contribution appears somewhat incremental relative to existing approaches like OptFormer and NAP. While the authors introduce novel elements, the core methodology builds heavily on established techniques.\n\nThe motivation for using Transformers in this context needs stronger justification. Given the small data regime typical in Bayesian optimization, the choice of a Transformer architecture, which typically requires substantial data for effective training, requires more thorough explanation.\n\nThe theoretical justification for computational intractability due to \"curse of dimensionality\" (line 244) requires more precise argumentation and supporting references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bnwtdhfSzx", "reviewer": "Reviewer_UwND", "summary": "This paper introduces BOFormer, a learning-based approach for Multi-Objective Bayesian Optimization (MOBO). The key idea is to replace the traditional handcrafted acquisition function with a Transformer-based network trained through non-Markovian reinforcement learning. The authors claim that MOBO is inherently non-Markovian due to the \"hypervolume identifiability issue\" and propose a generalized DQN framework incorporating historical information. The method is evaluated on both synthetic functions and real-world hyperparameter optimization tasks.", "questions": "* Regarding OptFormer Implementation, how did you adapt OptFormer for MOBO? To my understanding it was originally designed for single-objective HPO tasks. Did you retrain the model from scratch for MOBO or fine-tune the existing model? It would be great if you could add some details for the baselines you compared with.\n* Have you tried to compare the direct implementation you mentioned in Section 4.1 with your proposed method?\n* I assume BOFormer can only accept discrete inputs, how does the method handle continuous domains in practice? Are there any discretization issues?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* While there are several existing works on meta-learned acquisition functions for single-objective BO, BOFormer is the first work to apply a learning-based acquisition function to MOBO.\n* The domain-agnostic representation, which only incorporates historical query information rather than maintaining information for all domain points, enables cross-domain transfer while being memory-efficient.\n* Well-structured related work with clear categorization of existing approaches and insightful comparison with similar works, especially in distinguishing the unique aspects of BOFormer from OptFormer and NAP.\n* The paper provides a comprehensive evaluation on various synthetic functions and real-world hyperparameter optimization tasks, both show strong performance of BOFormer. Besides, the paper conducts an additional ablation study on sequence length.", "weaknesses": "* I have some doubts about the non-Markovian nature of MOBO. If we define the state as the complete set of historical queries along with the posterior distribution of candidate points, the problem naturally becomes Markovian, as the hypervolume improvement can be uniquely determined from this state representation. The non-Markovian property in your approach appears to be artificially induced by your choice of a simplified state representation. Please correct me if I am wrong or if I misunderstood something, as I am not from the field of reinforcement learning.\n* A major weakness lies in the methodology section, particularly in the insufficient details about BOFormer. Figure 2 is not thoroughly discussed; for instance, positional encoding appears in the architecture diagram but is never mentioned in the text, while Equation 5 already seems to contain temporal information. Moreover, it's unclear how j/t in Equation 5 is combined and encoded with other inputs, as it appears to be merely an arithmetic sequence (e.g., numbers from 0.1 to 0.9 when t=10). Additionally, the introduction of Off-Policy Learning and Prioritized Trajectory Replay Buffer feels abrupt - what problems are they addressing? There's no proper introduction or motivation. I suggest the authors improve this section to present the problems and their solutions more clearly to readers.\n* Another concern is the temporal information in the history representation. In BO, the historical queries should be permutation-invariant - the order of previous queries should not affect future decisions. However, by incorporating explicit temporal information, the method might be breaking this consistency. Or have you observed any benefit of incorporating temporal information? And is there any ablation study showing the impact of it?\n* The paper lacks some experimental details. E.g., no configuration of the surrogate model such as the choice of the kernel and hyperparameters; Training cost for meta-training not reported.\n* The paper contains several mathematical formulation issues and inconsistencies. E.g., Line 345, shouldn't it be $max$?; Line 347, I guess it should be $i$-th objective as $j$ is the time step; Inconsistent index notation, in line 344, $i$ indexes objective functions: $i \\in [1,···,K]$, while in the equation at line 374, $i$ represents time steps; Line 225, the hypervolume formula contains an unexplained $x'$ that appears redundant; Line 819, shouldn't it be uniform distribution?\n* There is no discussion about Figure 3, it might be better to give an introduction about performance profiles.\n* There is no experiment to support the idea of demo-policy-guided exploration, thus it is hard to see the effect of this enhancement.\n* Minor issues: Line 201: Incorrect appendix reference (B.2 instead of B); Line 820: Incomplete sentence ending abruptly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)