MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions
Authors: Lam Ngo
Huong Ha
Jeffrey Chan
Hongyu Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Bayesian optimization
multi-objective optimization
Gaussian process
Primary area: probabilistic_methods
TL;DR: We propose a multi-objective Bayesian Optimization algorithm that shows state-of-the-art performance on a wide set of benchmark problems.

Abstract: Bayesian Optimization (BO) is a powerful tool for optimizing expensive black-box objective functions. While extensive research has been conducted on the single-objective optimization problem, the multi-objective optimization problem remains challenging. In this paper, we propose MOBO-OSD, a multi-objective Bayesian Optimization algorithm designed to generate a diverse set of Pareto optimal solutions by solving multiple constrained optimization problems, referred to as MOBO-OSD subproblems, along orthogonal search directions (OSDs) defined with respect to an approximated convex hull of individual objective minima. By employing a well-distributed set of OSDs, MOBO-OSD ensures broad coverage of the objective space, enhancing both solution diversity and hypervolume performance. To further improve the density of the set of the Pareto optimal candidate solutions without requiring an excessive number of subproblems, we leverage a Pareto Front Estimation technique to generate additional solutions in the neighborhood of existing solutions. Additionally, MOBO-OSD supports batch optimization, enabling parallel function evaluations to accelerate the optimization process when resources are available. Through extensive experiments and analysis on a variety of synthetic and real-world benchmark functions with two to six objectives, we demonstrate that MOBO-OSD consistently outperform the state-of-the-art algorithms.

MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions
Lam Ngo
Huong Ha
School of Computing Technologies
School of Computing Technologies
RMIT University, Australia
RMIT University, Australia huong. ha@rmit...... edu. au
tung. lam. ngo@rmit edu.
Jeffrey Chan
Hongyu Zhang School of Big Data and Software Engineering Chongqing University, China hyzhangdcqu. edu. cn
School of Computing Technologies
RMIT University, Australia jeffrey cww.ncmiinntttccc edu. au
Abstract
Bayesian Optimization (BO) is a powerful tool for optimizing expensive black- box objective functions. While extensive research has been conducted on the single-objective optimization problem, the multi-ohjective optimization problem remains challenging. In this paper, we propose MOBO-OSD, multi-objective Bayesian Optimization algorithm designed to generate a diverse of Pareto opti- mal solutions by solving multiple constrained optimization problems, referred to MOBO-OSD subproblems, along orthogonal search directions (OSDs) defined with respect to approximated convex hull of individual objective minima. By employ- ing a well-distributed set of OSDs, MOBO-OSD ensures broad coverage of the objective space, enhancing both solution diversity and hypervolume performance. To further improve the density of the set Pareto optimal candidate solutions without requiring excessive number of subproblems, we leverage Pareto Front Estimation technique to generate additional solutions in the neighhorhood of exist- ing solutions. Additionally, MOBO-OST supports batch optimization, enabling parallel function evaluations to accelerate the optimization process when resources are available. Through extensive experiments and analysis on a variety of synthetic and real-world benchmark functions with two to six objectives, we demonstrate that MOBO-OSD consistently outperforms the state-of-the-art algorithms. Our code implementation can be found at htttps://githhie cow www.Lammmoo/moobo.....ooo osd
1 Introduction
Multi-objective Bayesian Optimization (MOBO) has recently attracted much attention 5
13
[] MOBO extends Bayesian Optimization (BO) a powerful framework for optimizing expensive black-box functions - to settings where multiple, often conflicting, objectives must be optimized
simultaneously. Applications of MOBO include but
limited machine learning 44
29
material design [88 56]. agriculture 27 28 robotics [54] 34 and vehicle design [32
[8]56).
There several challenges when working with MOBO problems. First, the objective functions often conflicting, meaning that improving one objective may detcriorate another. Hence the goal is not to identify single optimal solution, but rather set of Pareto optimal solutions. Second, maintaining the diversity among these Pareto optimal solutions is crucial to capture the wide range of trade-off across all objectives. This becomes more challenging under the limited evaluation budget in MOBO settings, where there is less budget to balance between exploration and exploitation while
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

covering the Pareto front. Third, batch optimization plays an important role in reducing the time and cost through parallel evaluations of objective functions. However, in MOBO, it introduces additional complexity, as it requires modeling the interactions among unobserved points across multiple objective functions. Fourth, the computational cost remains significant drawback, as the cost for training surrogate models and computing multi-objective acquisition functions, e.g.. EHVI [33, become increasingly expensive with the number of objectives. Despite the growing body of MOBO research, there exists substantial room for improvement. Many existing approaches rely on scalarization techniques [5 9 4T, which often fail to capture a diverse Pareto front, leading to suboptimal performance in terms of the hypervolume indicator. Moreover, several studies are unable to scale to a large number of objectives (M > 3) [3] or suffer from high computational cost in batch
optimization settings B 152.
In this paper, our goal is to develop a novel MOBO algorithm that prioritizes Pareto front diversity, resulting in an improvement in hypervolume performance and scalability to an arbitrary number of objectives in both sequential and batch optimization settings. To achieve this, we build upon the key insight of the Normal Boundary Intersection (NBI) method [[I] the intersection points between the boundary of the objective space and the vectors orthogonal to the convex hull of individual minima (CHIM) of the objectives could be Pareto optimal solutions. This geometric approach has been shown to generate a well-distributed (diverse) set of Pareto optimal solutions and scale well to large number of objectives in multi-objective optimization problems with large evaluation budget. To incorporate this idea under the limited evaluation budget in MOBO problems, we first introduce a technique to effectively approximate the CHIM. We then generate well-distributed set of orthogonal search directions (OSDs) with respect to the approximated CHIM and define a tailored search procedure that solves a constrained single-objective optimization problem along each OSD) (MOB)- -ODD subproblem) to identify intersection points with the boundary that are likely to be Pareto optimal. By constructing well-distributed set of OSDs, MOBO-OSD achieves broad coverage of the objective space, enhancing both solution diversity and hypervolume performance. To further enrich the set of Pareto optimal candidate solutions without the need solve large number of MOBO-OSD subproblems, we incorporate Pareto Front Estimation (PFE) technique [46] that locally explores the neighborhood of existing Pareto optimal candidates. All candidates are aggregated and evaluated using the hypervolume improvement acquisition function [19], and those with the highest scores are selected as the next batch of data points for evaluation. Finally, to support batch optimization, we incorporate the Kriging Believer technique (22 while considering the diversity of exploration regions, i.e., ensuring that selected points for observation originate from different exploration spaces. We name our method Batch Multi- culfi-Obbeectie Bayesian Optimization via Orthogonal Search Directions (MOBO-OSD). Our thorough analysis and extensive experiments on various synthetic and real-world benchmark problems, in both sequential and batch optimization settings, show that MOBO-OSD outperforms the state-of-the-art MOBO methods. We summarize our contributions as follows.
We propose MOBO-OSD, a novel MOBO algorithm that generates diverse set of Pareto optimal candidate solutions by solving multiple optimization subproblems defined along search directions orthogonal to the approximated CHIM of the objective space. To further enhance the diversity, we propose to locally explore the Pareto set for additional Pareto solutions via a PFE technique.
We develop MOBO-OSD in such a way that can perform effectively in the batch optimiza- tion setting by leveraging the Kriging Believer technique and exploration space information. We demonstrate that MOBO-OSD can outperform the sfate-of-the-art MOBO baselines on a comprehensive set of synthetic and real-world multi-objective benchmark problems with wide range of numbers of objectives, in both sequential and batch settings.
2 Background
2.1 Multi-Objective Optimization
We consider multi-objectivve optimization (MOO) problem involving a vector-valued objective function f. : X' + y' with f = (f1..... fas) where X' C RR is a D-dimensional input space, and y C A is an M-dimensional output space (M > 1). Without loss of generality, we assume the goal is to minimize all objectives of f. In MOO, it is generally not possible to find a single solution that optimizes all objectives simultaneously. Instead, the aim to identify the set of Pareto optimal solutions, where no objective can be improved without deteriorating at least one of the others.
2

---

A solution f(x(() is said to Pareto dominate another solution f(x(1),, i.., f(x(()) > f(x(11). if
9x(()) 2 fm (x()) Vm = , . and there exists m' E [ } such that f' (x(()) fx(()))) The set of Pareto optimal solutions is called the Pareto front P, {f(x) Ex' E
X f(x')= f(x)). and the corresponding set of Pareto optimal inputs is called the Pareto ser
P. fx E X f(x) E p,t. Formally, the MOO problem is expressed as finding the Pareto front
Py and Pareto set Ps such that,
Py min xEX
(I)
To measure the quality of Pareto front Py, the Hypervolume (HV) indicator (59 is one of the most widely used metrics [5] 5 9 13 I. Given a reference point E RM the HV indicator of a finite approximated Pareto front P. is defined as the M -ddimensional Lebesgue measure AM of the space dominated by solutions p in py and upper bounded by the reference point Formally, HV r) = AM PEP, r. Pl). where [r. p] denotes the hyperrectangle bounded by the reference
point r and E Py 9 13 14). The higher the HV, the better Py approximates the true Pareto front.
2.2 Multi-Objective Bayesian Optimization
Bayesian Optimization [21] is common tool for optimizing expensive black-box objective functions f. Given a minimization problem, the goal is to find the global optimum of the function ff using the least function evaluations. BO sequentially selects observation data via an iterative process. Each BO iteration trains a probabilistic surrogate model, builds an acquisition foniitionnnnco function, then selects the acquisition function's 's maximizer as the next observation. The most common type of surrogate model for BO is a Gaussian Process (GP) (55), which provides a posterior distribution over the objective function given the observed dataset D. The acquisition function a X - TR. is constructed from the surrogate model and an optimization policy to quantify the utility of each unobserved data point. Single-objective BO has many common acquisition functions such as EI (37), UCB (48] and TS (51).
Multi-objective Bayesian Optimization (MOBO) extends the capabilities of BO to optimize expensive black-box vector-valued objective functions f X' + ) with f = (fi,..., fM). Given a minimiza- tion problem, the goal is to find the Pareto set P. and the corresponding Pareto front Py, using the least function evaluations. In MOBO, the most common surrogate model is set of GPs, where each GP independently represents an objective function fm [31 9 41 [3]. For acquisition functions,
while several works leverage existing single-objective acquisition functions 31
41 5 others propose
new acquisition functions tailored for the MOO setting 23 52 13
3 Related Work
There have been various works aiming to develop MOBO algorithms. Many works have attempted to adapt singlc-objective (SO) acquisition functions to the multi-objective optimization (MOO) framework. One of the earliest methods is ParEGO [31], which randomly scalarizes the objectives and applies the EI acquisition function [37] to determine the next data points for observation. Paria et al. [1 systematically generalize the random scalarization technique, allowing different scalarization techniques, e.g., weighted sum and Tchebyshev (39). as well as different SO acquisition functions, e.g. TS [51 UCB [48,, be integrated into the MOBO framework. TSEMO 9 formulates a multi- objective TS acquisition function (51) by sampling all objectives, employs NSGA-II [16] to solve the resulting optimization problem, and selects the next evaluations by maximizing hypervolume improvement. USeMO [5] aims to maximize the uncertainty reduction of the candidate points generated by a MO acquisition function, which is defined by applying SO acquisition functions to each objective GP. Recently, PDBO [1] uses multi-armed bandit technique to select SO acquisition function and generate a candidate pool, followed by a Determinantal Point Process to select diverse set of solutions. Our proposed method operates directly on a multi-objective (MO) acquisition function, the Hypervolume Improvement, which is more suitable for the MOBO setting. In the experimental section, we compare against ParEGO, USeMO and PDBO- one of the latest state-of-the the-tartt methods.
There are also many works that have focused on developing new acquisition functions tailored for the MOO setting. Information-theoree (IT) MO acquisition functions aim to maximize the information gain of the next observation about the objective functions. Examples include PESMO 23
3

---

(a)
(c)
(d)
True Pareto Front
Nadir Point
Observed data
Approximated CHIM
uixosoy
x p(Xnnett
Boundary Points
Ideal Point
OSDs
U(i)noints
u(x=FE)
Figure 1: Summary of the MOBO-OSD algorithm: (a) The boundary points are defined via the
nadir and ideal points; (b) The approximated CHIM is defined from the boundary points and the OSDs
defined each ((3) point and orthogonal to the approximated CHIM: (c) The MOBO-OSD subproblem (Eq. (2) is optimized for x OSD candidates; (d) Additional candidates PFE are generated by locally exploring around x xSSD Finally, X is chosen based on Eq. (4).
MESMO PFES 49). and JES PAL is another algorithm based on information theory,
but it is only applicable to input spaces with finite set of discrete points. Another widely used approach define MO acquisition functions is through the HV indicator. The EHVI acquisition function [19] extends the EI concept from SOO to compute the expected improvement in the hypervolume of
the Pareto front. Daulton et al. [33 propose novel formulation of EHVI in parallel setting, namely qEHVI, based on Monte-Carlo sampling. Despite being differentiable, even for M > 2 objectives, qEHVI can suffer from high computational cost. Unlike qEHVI, which computes the expected hypervolume improvement under the posterior distribution, DGEMO (33] relies solely on the hypervolume improvement
the posterior mean, yet its integration with the Pareto Front Estimation technique [46] renders the method competitive. Nonetheless, unlike proposed method, its search strategy for Pareto optimal solutions is entirely random and does not ensure adequate coverage of the Pareto front. Moreover, DGEMO requires specialized data structure that must be designed separately for each specific number of objectives, thereby limiting its scalability beyond problems with three objectives. Recently, the HVKG [12] acquisition function, which extends the SO Knowledge Gradient acquisition function, has been proposed to tackle multi-fidelis and decoupled MOO problems. Although proposed method uses the Hypervolume Improvement acquisition function, it is compatible with alternative MO acquisition functions, rendering it applicable to new developments. In the experiment section, compare against JES- state-of-the-art IT-based method, as well as qEHVI and DGEMO, as they also leverage the HV-based acquisition functions.
Evolutionary Algorithms (EA) are also capable of tackling MOO problems. Examples of multi- objective Evolutionary Algorithms include MOEA/D [57). CMS-EMOA (6), and NSGA-II [16] 26). Generally, EA-based algorithms are less sample-efficient than BO-baser methods, making them unsuitable for settings with a limited evaluation budget. We compare against NSGA-II, the most widely used EA baseline in MOBO works.
4 Proposed Method
In this section, we present our proposed algorithm MOBO-OSD. See Fig. [or a summary of the proposed MOBO-OSD method. The core idea of MOBO-OSD is to find the Pareto optimal solutions by estimating the intersection points between the boundary the objective space and the vectors orthogonal to the convex hull of individual objective minima (CHIM) [Л.. To achieve this in the MOBO setting with limited evaluation budget, we first approximate the CHIM using bounded hyperplane (the approximated CHIM), and then construct set of well-distributed orthogonal search directions (OSDs) w.r.t. this approximated CHIM (Sec. 4.I). Then we propose the MOBO-OSD subproblem for each OSD (Sec. 4.2) followed by the incorporation of the Pareto Front Estimation technique (Sec. 4 Finally, we propose the batch selection process (Sec. 4.4). For simplicity, we assume that all the objective values are non-negative, which can be achieved by offsetting the current
worst value found so far each objective.

---

4.1 The MOBO-OSD Components
Approximated CHIM. As is not possible to obtain the individual min- ima of the objectives in MOBO settings with limited evaluation budget. we propose to approximate the CHIM as a convex hull of M boundary points, which are the extreme points computed from the observed dataset. Denote the ideal point yidcal = (mingeD fi (x),.... minxeD S (x)) and the nadir point M-dimensional points
ynadir (maxxeD f1 fx)))..... maXxeD fas whose components are the best and the worst values observed so far for each objective, respectively. Then, the m-th boundary point Pm is computed by replacing the m-th component of the ideal point with its corresponding nadir value, leaving the rest unchanged, i.e., Pm satísfies [pml] - [ynadir], jj if j m and [yldal], otherwise, for j 1, ...... M.. This approximated CHIM serves as a replacement for the true CHIM, which is not available as individual minima cannot be found efficiently in MOBO settings. Compared
True Parcto Front Boundary Points
*
Individual Minima
Observed data
simpler alternative, which approximates the CHIM via individual minima found so far in the observed dataset, our approach avoids prematurely shrinking the search region for the Pareto optimal solutions. In the alternative method, if good Pareto optimal solution is found early, the individual minima found so far in the observed dataset immediately shrink the search region, overlooking other potential unexplored regions of the objective space to find more Pareto optimal solutions. In contrast, the proposed approximated CHIM maintains a broader search region, allowing our proposed method MOBO-OSD explore more promising regions. See the inset for an example where our proposed approximated CHIM provides a larger search region (green area) than that the alternative method (purple area).
Orthogonal Search Directions. Having obtained the approximated CHIM, the Orthogonal Search Direction (OSD) is defined as one-dimensional line in the objective space that follows the unit normal vector n of the approximated CHIM and passes through a point on the approximated CHIM. Denote G as an M-dimensional convex combination vector 3 E RM >m= 8... = 1 and 8... > O, which defines point on the approximated CHIM (3)) Formally, 2(33) (PB = 1 3m Pm B RM, .EM= 3m = 1, 3m > 0} such that P = [P1,.... PM] is the matrix whose columns are the M boundary points pm- We then denote the OSD
M
line (()( n) that goes through the point U(3) in the direction of the normal vector n. In order to generate a well-distributed set of OSDs, it is essential to construct well-distributed set of ((3) points. This is equivalent to generating well-distributed set of {3} over an M -imiennsional unit simplex, as (33) is, by definition, the linear transformation of /3 from unit simplex to the approximated CHIM. To achieve this, we propose to employ the Riesz s-Energy method 8 to arrange the set of (3 in well-distributed fashion. Following the physics principle that the minimum potential energy state of set of points corresponds diverse distribution of those points, the Riesz s-Energy aims to minimize the sum of a potential energy function defined over (.3). which effectively generates
rell-distributed set (3. To sum up, in practice, we generate nıg well well-distributed OSDs by first producing ng well-distributed convex combination weight vectors [.3.). then linearly mapping them to the corresponding points (133)) on the approximated CHIM, and finally defining the set of OSDs (3,) n)), where i 1 ng.
See Fig. Db for an illustration of the OSDs. More details on the OSDs are in Appendix A.3
4.2 The MOBO-OSD Subproblem
We now present the MOBO-OSD subproblem, which is constrained optimiza- tion problem designed to obtain the Pareto optimal solutions by finding the intersection between an OSD and the boundary of the objective space. Given a convex combination weight vector B3, point the approximated (B). and an OSD L = (VV(3)) n). the intersection point L between the line L and the boundary of the objective space can be parameterized as L' 21(3)+ + A'n, where the scalar parameter X satisfies (x*, max À subject to U(B) + An = f(x). This is the ideal optimization subproblem used in the NBI method. More details on the NBI technique and its subproblem can be found Appendix A. In MOBO settings, however, it is not feasible to solve this ideal
40x0s0)
Oix)
vix050)
OSD
constrained problem due to the high evaluation cost of the constraints. There- fore, we propose the MOBO-OSD subproblem, which aims to approximate X
UIB) point
5

---

and x". We modify the ideal problem as follows. First, we replace the objective function values f(x) with the estimated function values, the posterior mean vector (x).. Second, we relax the equality constraint that forces (x(x) x) to be exactly on the OSD, (3) + An fx(x), and only require (4(x) to be within a certain distance from the OSD. This can be achieved by constraining the projection of jx(x)
within its confidence bounds, i.e., (x; B3 n) E Q(x) where ((x; 33 n) is the projection function of (x() onto the OSD C, and Q(x) = {y p(() - )α S y S p((x) + {x) represents a hyper-rectangular confidence region defined by the posterior standard deviation vector and a scaling factor [6]. Finally, we maximize the scalar X, which represents the scalar parameter corresponding to the projection point the OSD i... ((x; 3. m) = ((33+ + An. See the inset for an illustration of the MOBO-OSD subproblem. Note that À is not necessarily the distance |x((x) (3( 2. as À can be negative in the case of concave Pareto front, e.g., the DTLZ2 benchmark function [17]. Ultimately, the MOBO-OSD subproblem is defined as the following constrained optimization problem,
g1 (x) = (x: m) - [xx) + 6x) 2 0
(x OSD AOSD) E mmax À subject to XEA
(2)
(x) = fx(x) + 50 (x: 3. n) 0
A detailed formula for À can be derived from the definition of the projection operator, i.e., (xx; B, n) = (roj = Proj, ((xx))-((3) (1(3)) (((x))=2((3))))))))))))))))))))) n. Therefore, À can be expressed as X(x: B) = (m(x) - ((3)) n. as |||||
nnn
1. Solving Eq. ( results in (x05D, XOSD), which approximates the ideal (x*, as Eq. (2) ensures that the obtained solution has competitive estimated function values while remaining close to the current OSD L. The point (u(xOSD) serves as an approximation of the intersection point L' Given a set of ns well-distribtted OSDs as defined in Sec. 4.1 solving a MOBO-OSD subproblem for each OSD C generates well-distributed set of solutions {x D. See Fig. L for an illustration.
The detailed optimization routine for the proposed MOBO-OSD subproblem in Eq. proceeds as follows. We solve Eq. 2 using gradient- based off-the-shelf optimizer, e.g., SLSOP 35). To compute the gradient vector Jx (x) and Jacobian matrix JIm.,Ka (x), we require the Jacobian matrix of the posterior mean and standard deviation vectors of the GPs, which can be computed either in analytic form under common kernels [33 38) by automatic differentiation via computational graphs
42. Moreover, since the problem is highly non-convex, we solve Eq. 2 via multiple starting
points {x; Ti-o and select the most promising solution among the resulting solutions. Specifically, for each starting point x; we obtain a solution (3) = OSD that exhibits a different trade-off between maximizing A(x) and minimizing the distance from p(x) to L. i.e., 1(x) = llı(x) ->(x) 22 Since it is not feasible to find the promising solution x(SDD that maximizes A(x(SD) while enforcing 1(x,5D)= = 0, we opt to select the most potential solution by formulating a bi-objective selection step to identify the best trade-off among the n, candidates |[ characterized by X, = >(x(SD) and 1, = I(x?SD). We use the hypervolume indicator for this selection, computing the hypervolume contribution for each pair [A.. [4] as the reduction in hypervolume when that pair is removed from the solution set S
Formally, HVC 41) = HV (S,r,) HV (S \ [x, l..r... where r, nadir + 0.1 msadir gideal) is the reference point computed from the nadir point s naadr and the ideal point gi العان of the set S. The final solution for Eq. (2) is chosen as x(SD((3) (B) x where 1 = arg max; !I..... HVC ([>,, 4.)). This procedure ensures the selected solution achieves a balance between a high X value and a close proximity to the line L. Since each MOBO-OSD subproblem Eq. ( solved independently, all subproblems (for different (3) can be processed in parallel to improve computational efficiency.
4.3 Pareto Front Estimation
For each MOBO-OSD subproblem (corresponding to a point on the approximated CHIM (3)).. we obtain a solution x OSD (in Eq. 2 Instead of discretizing the approximated CHIM into a large number of points U(3), we propose to leverage a Pareto Front Estimation (PFE) technique to explore the local region around the current solution XSD (13) and generate additional data points that are expected to be Pareto optimal. One successful approach is the First Order Approximation technique [46] (3), which computes a local exploration space T around a current Pareto optimal solution. Specifically, we consider the First Order Approximation problem, which the surrogate model is used to estimate the exploration space T around the solution xOSD(3). See Appendix A.2 for an overview of the First Order Approximation method. In particular, solving the First Order
Approximation problem results in set of directions which define the space T. Then we estimate

---

Algorithm The MOBO-OSD Algorithm
Input: Objective function f()., evaluation budget 7', batch size b, number of weight vectors n
2: Output: The Pareto set Ps
3: Initialize data points and append to the observed dataset D
4: while t v ' do
5: Compute upproximated CHIM and define n,3 OSDs
Sec.
6: Train GPs for each objective function f.
7:
for each point (13) on the approximated CHIM do
8:
Optimize the MOBO-OSE subproblem to generate a candidate (3) Dr Eqq.
9:
Estimate the Pareto front around x (B) to explore more candidates x x
(3) P Eq.
10: Append X. +- X U x (B)
11: end for
12: Select batch of b solutions from X. and evaluate them; Increase ++
Eq.
13:
end while
14:
Return P. from dataset D
the Pareto Front by randomly sampling ne data points in T such that,
X (g) [x, ((x,F((())) (3)} (x (g3) + v.v for i 1,..., fles
where u; is the i-th random perturbation term to shift the solution xOSD along the directions v. This PFE step generates additional candidate points around the current expected Pareto optimal candidates 0SD from the previous step. This also helps to avoid solving an excessive number of MOBO-OSD subproblems that are close to one another. See the ablation study in Sec. S.c. for details.
This step repeated for all MOBO-OSD subproblems, i.e., for each point on the approximated CHIM U(3), resulting in set of candidates X, U X (B,)), where n. denotes the number of weight vectors on the approximated CHIM. In general. X n2 X ne.
4.4 Batch Selection Strategy
Having generated ng X ne candidate solutions Xe, we use the Hypervolume Improvement (HVT) of the posterior mean as the acquisition function to determine the most promising solutions to be evaluated solutions that are expected to maximize the hypervolume contribution 9 33 Given the current approximate Pareto front P computed from the observed dataset 'D and a reference point the HVT acquisition function defined as onvi(( Py, r) = HV(H(x U Pr, r) HV(P,: r). where pa()) is the posterior mean function. For batch selection, the goal is to pick a batch of b data points X5 that achieve high HVI values while maintaining diversity among the chosen solutions. To address the problem, first, apply the Kriging Believer (KB) method [22) to improve the posterior mean estimation given unobserved data points during batch selection. particular, after selecting data point x; for i E [1..... b). we re-train the GP models on the aggregated dataset D U {x,, (xxx)) }. Secondly, we boost the diversity by choosing candidate points coming from different exploration spaces T. This is because the approximated candidate points from single exploration space T tend to be close to one another [46), therefore exhibiting similar contributions to the hypervolume. Denote (3,, X) function that counts the number of candidates x E X originating from the exploration space T around OSD (3,). Hence, the batch selection mechanism is formulated as follows,
X, argmax (xv Py,r) such that max X.EX,
(BB,, X.) i=.....
min ((,, Xb) S 1. (4)
1=1....,Tig B2
The constraints in Eq. 4 are designed to ensure that the number of data points selected from different exploration spaces differs by most one. practice, we solve Eq. (4) sequentially select data points X,, from the candidate set X In each iteration, after re- re-training the GP models (as required by KB) and selecting the best candidate point based on αиvı, we remove from X, other candidates originating from the same exploration space, then repeat the process to select the next point in the batch. If the candidate set X, becomes empty, all previously removed but unselected candidate points reintroduced, and the process is repeated.
7

---

MO (M-2)
7DT1 (M-2)
DT172-M2
Log ItV difference
10
10
10
10
10
50
100
150
200
50
100
150
200
50
100
150
200
MOBO-OSD
DTL22-M3
DTL22-M4
Speed Reducer
(ours)
Log HIV difference
PDRO
10
DGEMO
QEHVI
10"
10"
JES
10
USEMO-EI
10
qParEGO
100
200
50
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
NSGA-II
Log ItV difference
10
NBI
10
10"
50
100
150
200
50
100
150
200
50
100
150
200
Number evaluations
Number of c'valuaations
Number of evaluations
Figure 2: Comparison MOBO-OSD against the SOTA baselines on 5 synthetic and real-world benchmark problems in sequential setting (batch size 1). Note that DGEMO does support problems with M objectives. Overall, MOBO-OSD outperforms the baselines.
4.5 Overall Algorithm
The overall MOBO-OSD algorithm is described in Alg. MOBO-OSD operates in iterative fashion. In each iteration, based on the observed dataset D, we compute AM boundary points P. the approximated CHIM U, and the ng OSDs (line 5). Subsequently, for each point U(33) the approximated CHIM, we optimize the MOBO-OSD subproblem to generate Pareto optimal candidate xSD and then locally explore the space around x to generate additional Pareto optimal candidates PFF (lines 8 99. We then use the aggregated set of candidates X, from all ny OSDs to select batch of b solutions using the omvi acquisition function and the exploration space constraint (line 12). The process is repeated until the evaluation budget is exhausted, and the final Pareto front Py and Pareto set P, computed from the observed dataset D.
Experiments
Now we empirically evaluate proposed method, MOBO-OSD, against the state-of-the-art methods on an extensive set of synthetic and real-world benchmark problems.
Experimental Settings and Baselines. We evaluate the proposed algorithm MOBO-OSD against comprehensive set of baselines: qParEGO (31, USeMO (5). DGEMO (33, PDBO (1) JES [22 qEHVI [13, NSGA-II (16] and NBI [1.. For USeMO, we select the El acquisition function as it has good performance and is commonly used in other works. For qParEGO, we use the batch implementation developed by Daulton et al. [13]. For DGEMO, we note that the authors' implementation only supports problems with M 2 and M objectives. For JES, due to its prohibitive computational cost (Appendix A. we only compare under the sequential optimization setting (batch size 1). For NBI, to the best of our knowledge, no open-source implementation of the method currently exists; therefore, we re- ce-mplemented it following the procedure described in Detailed implementations of MOBO-OSD and the baselines can be found in Appendix A.7
For the comparison metrics, we compute the logarithmic hypervolume difference between hypervolume of the best accumulated observed Pareto front and the maximum hypervolume. The hypervolume is calculated using the reference points r specified in Appendix A.6 We report the mean and the standard error across 10 independent runs.

---

MO. (M-2)
ZDT1 (M-2)
DT172-M2
Log ItV difference
10
10
10
10
50
100
150
200
50
100
150
200
50
100
150
200
DTL22-MS
DTL22-M4
Speed Reducer (M=3)
MOBO-OSD
Log HIV difference
(OurS)
10
PDBO
10"
10"
qEHVI
10
USEMO-EI
10
qParEGO
100
200
200
50
Car Side Design (M=4)
Marine Design (M-4)
Water Pl Planning (M-6)
NSGA-II
Log LogitV difference
10
10
10"
50
100
150
200
50
100
150
200
50
100
150
200
Number ovaluations
Number of ovaluations
Number of evaluations
Figure 3: Comparison of MOBO-OSD against the SOTA baselines 5 synthetic and 4 real-world benchmark problems in batch setting (batch size 4). Overall, MOBO-OSD outperform the baselines.
Benchmark Problems. We conduct experiments five synthetic and four real-world multi- objective benchmark problems. The number objectives ranges from two to six, which is common in the MOBO literature. For synthetic benchmark problems,
DTLZ2 with different objective settings M E {2, 2,334) 4} 17). ZDT/ [5] and VI.MPP2 [53] For real-world benchmark problems, we various problems from the RE problem suite (00] including Speed Reducer, Car Side Design, Marine Design, and Water Planning. The dimensionality and number of objective settings for each function are given Table 2 These problems are widely used in the MOBO literature (5 12 Details of the benchmark problems can be found in Appendix A.6
333.
5.1 Comparison with Baselines
Sequential Optimization. Fig. 2 shows the performance of all baselines all nine benchmark problems in the sequential setting (batch size 1). Across all benchmark problems, both synthetic and real-world, MOBO-OSD consistently outperforms other ttate-of-the-art methods. qEHVI shows competitive performance in most cases, yet eventually finds suboptimal solutions. DGEMO also shows strong performance, but outperformed by MOBO-OSD on VLMOP2 and Speed Reducer. Moreover, it is limited to problems with at most three objectives M < 3. NSGA-II often cannot compete with the MOBO algorithms the case of limited evaluation budget. These results indicate the efficiency of MOBO-OSD in achieving fast convergence towards the Pareto front.
Batch Optimization. We conduct experiments in batch settings, with batch size (4, 8, 10). Fig. 3 shows the performance of all baselines across all nine benchmark problems with batch size whereas additional batch results can be found in Appendix [A.8] Similar to the batch size setting, MOBO-OSD consistently outperforms other state-of-the -aee--ff-hheeaeee art methods. qEHVI and DGEMO remain the two strongest baselines after MOBO-OSD, achieving hypervolume results close those of MOBO-OSD, This result further emphasizes the efficiency of MOBO-OSD, even in the batch setting. Furthermore, we provide a theoretical time complexity analysis for MOBO-OSD, along with runtime comparison between MOBO-OSD and the baselines in Appendix II demonstrating the scalability of MOBO-OSD to an arbitrary number of objectives in both sequential and batch settings.
5.2 Ablation Study
In this section, we conduct study on the effect of the na parameter the number of points on the approximate CHIM ((3) on the performance of MOBO-OSD. larger n, corresponds to a denser set of points on the approximated CHIM. We evaluate MOBO-OSD using varying

---

Log HV difference
22-1 (b-1)
Marine Design (b-1)
D311.22-M4 (b-4)
Marine Design (b=4)
10
10"
10"
10"1
10
10" 11
100
200
100
200
100
200
100
200
Number evaluations
Number of evaluations
Number evaluations
Number evaluntions
n_ beta = 10
n beta = 20 (default)
n beta = 50
D. bcta 100
Figure Ablation study on the effect of parameter ni.. Full results can be found in the Appendix A.9 Overall, MOBO-OSD is robust against ng--
values of ng E {10, 50, 100) all nine benchmark problems and compare the default setting (n,3 = 20). Four representative results are shown in Fig. while the remaining results can be found in Appendix A.9 Fig. shows that MOBO-OSD's overall performance remains stable across different values of nig indicating the robustness of the proposed algorithm with respect to this parameter.
We hypothesize that this robustness can be attributed to the Pareto Front Estimation (PFE) component (Sec. 4.3). which facilitates the generation of additional Pareto optimal candidates. Without the component, MOBO-OSD would require more (()) points, thus solving larger number of MOBO- OSD subproblems to achieve a comparable set of Pareto optimal candidates. To validate this hypothesis, conduct study by removing the PFE component and run the resulting MOBO-OSD variant with varying values of n,s E (20, 100, 200, 500). As shown in Table increasing n, leads to performance improvements. This finding suggests that although PFE enhances the efficiency of MOBO-OSD it is not an essential component, as similar performance can be attained by increasing the density of the ((3) set.
Table 1: Ablation results (in HV) on the effect of ng on MOBO-OSD without the PFE component.
MOBO-OSD method
DTLZ2-M2
VLMOP2
Car Side Design
W/o PFE (mg- = 20)
0.4041 + 0.0004
0.2713 + 0.0020
145.1195 = 0.3340
W/o PFE (n3 100)
0.4118 0.0001
0.2978 = 0.0011
154.3249 = 0.2662
W/o PFE (ng = 200)
0.4142 + 0.0001
0.3076 0.0006
157.2311 = 0.2095
W/o PFE (ng 500)
0.3159 = 0.0004
160.2797 = 0.2118
Default (with PFE)
0.4217 + 0.0000
0.3383 + 0.0000
177.4782 + 0.2310
6
Conclusions
In this paper, we address the multi-objective Bayesian optimization problem for expensive black-box, vector-valued objective functions. We propose MOBO-OSD, a novel algorithm that aims to generate well-distributed set of solutions via multiple subproblems defined along orthogonal search directions. To further enrich the diversity of the solutions, we perform local exploration around current Pareto optimal candidates, generating additional Pareto optimal candidate solutions. These candidates are scored using the Hypervolume Improvement acquisition function, while batch selection guided by the Kriging Believer strategy and the exploration space information. Our experimental results show that MOBO-OSD outperforms state-of-the-ard methods across various synthetic and real- world benchmark problems with varying number of objectives, in both sequential and batch settings.
Limitations. One limitation of our approach is that focuses on noiseless observations, a common assumption in various existing works. Future work could address this limitation by employing acquisition functions that can handle noise and by improving the MOBO-OSD subproblem for example, by integrating the uncertainty of previous observations when defining the projection.
Acknowledgments
The first author (L.N.) would like to thank the School of Computing Technologies, RMIT University, Australia and the Google Cloud Research Credits Program for providing computing resources for this project. Additionally, this project was undertaken with the assistance of computing resources from RMIT Advanced Cloud Ecosystem (RACE).
10

---

References
[1] Alalch Ahmadianshallhi,, Syrine Belakaria, and Janardhan Rao Doppa. Pareto front-diverse batch multi-objective bayesian optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10794-10794, 2024.
[2] Siwakorn Anosri, Natee Panagant, Pakin Champasak, Sujin Bureerat, Chinnapat Thipyopas, Sumit Kumar, Nantiwat Pholdee, Betül Sultan Yildız, and Ali Riza Yildiz. A comparative study of state-of-the-ar metaheuristics for solving many-objective optimization problems of fixed wing unmanned acrial vehicle conceptual design. Archives of Computational Methods in Engineering, 30(6):3657-3671, 2023.
[3] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020.
[4] Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi-objective bayesian optimization. Advances in neural information processing systems, 32, 2019.
[5] Syrine Belakaria, Aryan Deshwal, Nitthilan Kannappan Jayakodi, and Janardhan Rao Doppa. Uncertainty-awwae search framework for multi-objective bayesian optimization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages
2020.
[6] Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa: Multiobjective selection based on dominated hypervolume. journaa of operational research, 181()):1633-166) 2007.
[7] Blank and K. Deb. pymoo: Multi-objective optimization in python. IEEE Access, 8:89497- 89509, 2020.
[8] Julian Blank, Kalyanmoy Deb, Yashesh Dhebar, Sunith Bandaru, and Haitham Seada. Generat- ing well-spaced points on a unit simplex for evolutionary many-objective optimization. IEEE Transactions Evolutionary Computation, 25(1):48-66 -0, 2020.
[9] Eric Bradford, Artur M Schweidtmann, and Alexci Lapkin. Efficient multiobjective optimization employing gaussian processes, spectral sampling and a genetic algorithm. Journal of global optimization, 71 71(2):407-4338 2018.
[10] Carlos A Coello Coello and Gary B Lamont. Applications of multi-objective evolutionary algorithmss, volume 1. World Scientific, 2004.
[11] Indraneel Das and John E Dennis. Normal- boundary intersection: A new method for gener- ating the pareto surface in nonlinear multicriteria optimization problems. SIAM journal on optimization, 8(3):631-65 1998.
[12] Sam Daulton, Maximilian Balandat, and Eytan Bakshy. Hypervolume knowledge gradient: a lookahead approach for multi-objective bayesian optimization with partial information. In International Conference Machine Learning, pages 7167-7204. PMLR, 2023.
[13] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. Advances in Neural Information Processing Systems, 33:9851 -96664 2020.
[14] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement. Advances in Neural Information Processing Systems, 34:2187-2200, 2021.
[15] Kalyanmoy Deb. Multi-objective genetic algorithms: Problem difficulties and construction of test problems. Evolutionary computation, 7(3):205 230, 1999.
[16] Kalyanmoy Deb and Himanshu Jain. An evolutionary majy-objective optimization algorithm using seference-point-basseee nondominated sorting approach, part solving problems with box constraints. IEEE transactions evolutionary computation, 18(4):577-601, 2013.

---

[17] Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, and Eckart Zitzler. Scalable test problems for evolutionary multiobjective optimization. In Evolutionary multiobjective optimization: theoretical advances and applications, pages 105-145. Springer, 2005.
[18] Aryan Deshwal, Cory M Simon, and Janardhan Rao Doppa. Bayesian optimization of nanoporous materials. Molecular Systems Design & Engineering, 6(12): 1066-10666, 2021.
[19] Michael Emmerich and Jan-willem Klinkenberg. The computation of the expected improve- ment in dominated hypervolume of pareto front approximations. Rapport technique, Leiden University, 34:7-3, 2008
[20] Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix- matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems, 31, 2018.
[21] Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
[22] David Ginsbourger, Rodolphe Le Riche, and Laurent Carraro. Kriging is well-suited to paral- lelize optimization. In Computational intelligence in expensive optimization problems, pages 131-162. Springer, 2010.
[23] Daniel Hleránndez-Lobate Jose Hernandez-Lohare Amar Shah, and Ryan Adams. Predictive entropy search for multi-objective bayesian optimization. International conference on machine learning, pages 1492-1501. PMLR, 2016.
[24] Claus Hillermeier. Nonlinear Multiobjective Optimization: A Generalized Homotopy Approach, volume 135. Springer Science & Business Media, 2001.
[25] Hisao Ishibuchi, Hiroyuki Masuda, Yuki Tanigaki, and Yusuke Nojima. Modified distance cal- culation generational distance and inverted generational distance. In International conference on evolutionary multi-criterion optimization, pages 110-125. Springer, 2015.
[26] Himanshu Jain and Kalyanmoy Deb. An evolutionary many-objective optimization algorithm using reference-point based nondominated sorting approach, part i:: Handling constraints and extending an adaptive approach. IEEE Transactions on evolutionary computation, 18(4): 602-622, 2013.
[27] Shan Jiang, Hongyan Zhang, Wenfeng Cong, Zhengyuan Liang, Qiran Ren, Chong Wang, Fusuo Zhang, and Xiaoqiang Jiao. Multi-objective optimization of smallholder apple production: Lessons from the bohai bay region. Sustainability, 12(16):6496, 2020.
[28] Faranak Karamian, Ali Asghar Mirakzadeh, and Arash Azari. Application of multi-objective genetic algorithm for optimal combination of resources to achieve sustainable agriculture based on the water-energy-food nexus framework. Science of The Total Environment, 860: 860:160419, 2023.
[29] Florian Karl, Tobias Pielok, Julia Moosbauer, Florian Pfisterer, Stefan Coors, Martin Binder, Lennart Schneider, Janek Thomas, Jakob Richter, Michel Lang, et al. Multi-objective hyperpa- rameter optimization in machine learning-an overview. ACM Transactions on Evolutionary Learning and Optimization, 3(4):1-50, 2023.
[30] Johannes Kirschner, Manuel Nonnenmacher, Mojmír Mutný, Andreas Krause, Nicole Hiller, Rasmus Ischebeck, and Andreas Adelmann. Bayesian optimisation for fast and safe parameter tuning of swissfel. In FE1.2019, Proceedings of the 39th International Free-Flectron Laser Conference, pages 707-710. JACoW Publishing, 2019.
[31] Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expen- sive multiobjective optimization problems. IEEE transactions evolutionary computation, 10 (1):50-66, 2006.
[32] Takehisa Kohira, Hiromasa Kemmotsu, Oyama Akira, and Tomoaki Tatsukawa. Proposal of benchmark problem based on real-world car structure design optimization. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 183-184, 2018.
12

---

[33] Mina Konakovic Lukovic, Yunsheng Tian, and Wojeiech Matusik. Diversity- guided multi- objective bayesian optimization with batch evaluations. Advances in Neural Information Processing Systems, 33:17708- 2020.
[34] Sallam A Kouritem, Mohammed I Abouheaf, Nabil Nahas, and Mohamed Hassan. A multi- objective optimization design of industrial robot arms. Alexandria Engineering Journal, 61(1)): 12847-12867, 2022.
[35] Dieter Kraft. Deutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt, 1988.
software package for sequential quadratic programming. Forschungsbericht-
[36] Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalarization for multi-objective optimization. In International Conference on Machine Learning, pages 30479-30509. PMLR, 2024.
[37] Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference Novosibirsk, July 1-7, 1974 6, pages 400-404. Springer, 1975.
[38] Sarah Müller, Alexander von Rohr, and Sebastian Trimpe. Local policy search with bayesian optimization. Advances in Neural Information Processing Systems, 24:20708-20770, 2021.
[39] Hirotaka Nakayama, Yeboon Yun, and Min Yoon. Sequential approximate multiobjective optimization using computational intelligence. Springer Science & Business Media, 2009.
[40] Lam Ngo, Huong Ha, Jeffrey Chan, and Hongyu Zhang. Boids: High-dimensions bayesian optimization via incumbent -uided direction lines and subspace embeddings. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 39, pages 19659-19667, 2025.
[41] Biswajit Paria, Kirthevasan Kandasamy, and Barnabás P6czos. A flexible framework for multi-objective bayesian optimization using random scalarizations. In Uncertainty in Artificial Intelligence, pages 766-776. PMLR, 2020.
[42] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W- W, 2017.
[43] Michael JD Powell. direct search optimization method that models the objective and constraint functions by linear interpolation. Springer, 1994.
[44] Jixiang Qing. Henry B Moss, Tom Dhaene, and Ivo Couckuyt. Pf2es: Parallel feasible pareto frontier entropy search for multi-objective bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 2565-2588. PMLR, 2023.
[45] Nery Riquelme, Christian Von Lücken, and Benjamin Baran. Performance metries in multi- objective optimization. In 2015 Latin American computing conference (CLEI), pages 1-11. IEFE, 2015.
[46] Adriana Schulz, Harrison Wang, Eitan Grinspun, Justin Solomon, and Wojeiech Matusik. Interactive exploration of design trade-offs. ACM Transactions on Graphics (TOG), 37(4) 2018.
[47] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
[48] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimenta design. In Proceedings of the 27th International Conference on Machine Learning, pages 1015-1022. Omnipress, 2010.
[49] Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama Multi-objective bayesian optimization using pareto-frontier entropy. In International conference on machine learning, pages 9279-9288. PMLR, 2020.
[50] Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 89:106078, 2020.
13

---

[51] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285 1933.
[52] Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi- objective bayesian optimization. Advances in Neural Information Processing Systems, 35: 9922-9938, 2022.
[53] David A Van Veldhuizen and Gary B Lamont. Multiobjective evolutionary algorithm test suites. In Proceedings of the 1999 ACM symposium Applied computing, pages 351-357, 1999.
[54] Changyun Wei, Ze Ji, and Boliang Cai. Particle swarm optimization for cooperative multi-robot task allocation: a multi-objective approach. In IEEE Robotics and Automation Letters, volume 5, pages 2530-2537. IEEE, 2020.
[55] Christopher Williams and Carl Edward Rasmussen. processss for machine learning, volume 2. MIT press Cambridge, MA, 2006.
[56] Pengeheng Xu, Yingying Ma, Wencong Lu, Minjie Li, Wenyue Zhao, and Zhilong Dai. Multi- objective optimization in machine learning assisted materials design and discovery. Journal of Materials Informatics. 5(2):-A, 2025.
[57] Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decom- position. IEEE Transactions on evolutionary computation, 1 11(6):11-771,, 2007.
[58] Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multi- objective black box optimization. In International conference on machine learning. pages (1096-11105, PMLR, 2020.
[59] Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4): 257-271, 2002.
[60] Eckart Zitzler, Lothar Thiele, Marco Laumanns, Carlos M Fonseca, and Viviane Grunert Da Fonseca. Performance assessment of multiobjective optimizers: An analysis and review. IEEE Transactions on evolutionary computation, 7(2): 17-132, 2003.
[61] Marcela Zuluaga, Guillaume Sergent, Andreas Krause, and Markus Püichel. Active learning for multi-objective optimization. In International conference on machine learning, pages 462-470. PMLR, 2013.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a multi-objective Bayesian Optimization algorithm, which can outperform the state-of-the-ar methods. We provide experiments and analysis to support our claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limitation in the Conclusion section (Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the settings for proposed method and baselines in the Appendix
A.7
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code in the supplementary materials. We will also make the code public after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guudees/Coeeebbbmsss itw.oeeeSuubiissiooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuaaaeoeaec
wetpeeee/oooSSeeesssSeeeeeee www.ssssiicogsooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment settings, including the hyperparamters the methods and the details of benchmark functions are presented in Sec. and Appendix A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In all our experiments, we provide the error bar to represent the randomness of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We present the computing resources used for all experiments in the Appendix A.10
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpsisstpseeitisscooo/o//// http://wwwppppcoouuuuui/c/chsaaa
Answer: [Yes]
Justification: Our work comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper propose optimization algorithm. There should be no direct societal impacts that must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe our work does not pose any risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the baseline implementation open access. All the real-world benchmark problems are also open access.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:The paper does not release new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not pose any risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Technical Appendices and Supplementary Material
A.1 Normal Boundary Intersection
In this section, we provide more details on the Normal Boundary Intersection (NBI) method, which is general MOO technique that aims to generate a uniformly distributed set of Pareto optimal solutions. The key insight of the NBI method is that, given a MOÔ problem, the intersection points between the boundary of the objective space and the vectors orthogonal to the convex hull of individual minima (CHIM) of the objectives could be Pareto optimal solutions, Given the MOO problem in Eq.
II the NBI method first determines the individual optima of each objective C - [(x],, ) where X;,
argmin xEX fm (x) for m = I,.... M. Denote a column matrix F = [[ 62
IM the CHIM constructed by a set of points {3) such that ((3) (F* 3 >i=1 f.f. G RRRR,, -'=
af
1. 3. > 0}. where B = [3] [s,................... 3as] is a convex combination weight vector to construct points on CHIM. For each point (()3). NBI defines constrained optimization problem, referred as the NBI subproblems, along the direction normal to CHIM, i.e. (x"., X²) E maxxeR,xEx À subject to U(3)+ An f(x), where n is the normal vector of CHIM. Solving this constrained optimization problem results in data point x" corresponding to the intersection point f(x") ) that is expected to be Pareto optimal. As a result, by solving multiple NBI subproblems defined over a set of well- distributed points (33 on the CHIM, the NBI method is expected to result in set of well-distributed ns Pareto optimal solutions for the MOO problem in Eq. []. The geometric intuition ensures uniform spacing of solutions and is particularly suitable for problems with complex, non-convex Pareto fronts. However, solving the NBI problem requires evaluating the objective functions f(x) for the constraint term, which is impractical for expensive objective functions and limited evaluation budget as in MOBO.
In NBI, the set of follows. Given an integer number denoting the number of partition ny. possible values for each 3, are {0,6 8, 26, 1} where ō = 1/n,, is the step-size. Then, we sequentially select each 3; such M
/ = 1B..... Bar] is constructed using structured approach as
In practice, the possible values for B3 corresponding to 3 mæę for i - .. . and j 2, . M 1 are { 26,. (p- --- m=)² }. Because of this formulation, the number of convex combination weight vectors ,3 (and the number of points on CHIM ((3)) is determined via the binominal coefficients as n2 (M+P-1). As result, this structured formulation cannot generate any arbitrary number of n.s as it ultimately depends on the number of partitions p instead. Our proposed MOBO-OSD algorithm leverages a different point generation method (Sec 4.1) that can handle any arbitrary number of ng, while maintaining the desired well-distributed property.
that they satisfy Di=1 B.
A.2 Pareto Front Estimation via First Order Approximation
This section summarizes the First Order Approximation technique [46] to estimate the Pareto front of a MO problem. In MOO, discovering the entire Pareto front Py can be challenging, however, once we find a Pareto optimal solution x* E Py. it is easier to find nearby Pareto optimal solutions by locally Pareto Front First Order Approximation technique 46 leverages this fact to approximate the Pareto front from a previously discovered Pareto optimal solution. Given the MOO problem in Eq. i and an observed Pareto optimal input x" found so far, the key idea is to construct a local exploration space T C X' around x". In particular, T is linear combination of a set of exploration vectors v which are obtained by solving the following equation:
exploring around x* 46 33
Hv E Im (Jr(x*)) e Im (Ja(x*)), TT(x')v== 0,
(5)
where H - ,xH)x(x))+++ (x H. (x*) corresponds the derivatives of the stationarity
(x*)+
KTT condition; Jr and To are the Jacobian matrix of all objectives f and K active constraints g. respectively; Hi is the Hessian matrix of an arbitrary function and a [aj..... avy and /
[BI...., 3x] are the Lagrange multipliers (dual variables) associated with the KKT conditions [24 of the MOO problem in Eq. !I The first equation in Eq. (5) ensures that moving x" along direction still preserves the stationarity KKT conditions, ensuring the Pareto optimality of the resulting solutions. Additionally, the second equation ensures the feasibility of active constraints g that
appear when x" is very close to boundary of X. Overall, this technique can help to discover more
22

---

solutions nearby the current Pareto optimal solution f(x'). Schulz et al. [46] shows that, generally, d7m min(M 1, D). Having the local exploration space T with exploration vectors v, we can generate additional Pareto optimal solutions around the previously discovered Pareto optimal solutions x".
A.3 Details on the Proposed Orthogonal Search Directions
This section presents the details when computing the normal direction of the approximated CHIM (Sec.F. [.T). MOBO-OSD, following Das and Dennis [IT] instead using the exact normal direction where Pn O, we employ quasi-normal directions for consistent scaling across all objectives, preventing potential ill-conditioning. Specifically, the proposed orthogonal search direction ń for MOBO-OSD is defined as ñ -Pe where is column vector of all ones. The formula computes an equally weighted linear combination of the boundary points Pm. then multiplied by -1 to ensure the normal vector points to the origin. Then normalize normal vector to unit length n h /|][] The quasi-normal direction can be interpreted as applying a normalization to remove the difference in scaling among the objectives, while maintaining similar results of subproblems, i.e., the intersection points found.
A.4 Advantages of OSD Subproblems Compared to Other Scalarization Techniques
In this section, we provide discussion on the advantages of our propose OSD subproblem formulation compared to the most closely related scalarization technique linear scalarization (LS) [41. First, LS generates search directions at random, which can be less efficient than the data-driven search directions proposed in our OSD formulation. Second, even when LS weight vectors are well- distributed (e.g., using Riesz s-Energy (8). LS is limited to finding solutions on the convex regions of the Pareto front [58). In contrast, our proposed OSD formulation is capable of identifying solutions on the Pareto front of arbitrary shape, including both convex and non-convex regions. Note that due to this limitation, LS has been superseded by the most widely used scalarization technique Tehebychev scalarization (TCH) 41 TCH can handle non-convex PF and has been widely selected in other works when evaluating scalarization techniques 41 13 33 However, one critical drawback of TCH lies in its non-smooth formulation caused by the maximization operator, which makes TCH suffers from non-differentiabiliy and slow convergence [36). On the other hand, the MOBO-OSD subproblem formulation differentiable, either analytically under common kernels or by automatic differentiation via computational graphs, described in Sec. 4.2 Empirically, proposed method consistently outperforms baselines employing TCH, such as qParEGO. Figures 2 5 and show that MOBO-OSD outperforms qParEGO across different benchmark functions with diverse PF characteristics, including convex (ZDTI) and concave (DTLZ2).
A.5 Comparison To Other Lise-based BO Algorithms
As our proposed MOBO-OSD algorithm conducts search for Pareto optimal solution along one- dimensional guiding lines, in this section, we discuss the key differences compared to other recent line-based BO algorithms, such as LineBO (30 and BOIDS [40]. The primary distinction lies in the formulation of these one-dimensional lines: both LineBO and BOIDS construct search directions with some degree of randommess, whereas MOBO-OSD relies on deterministic OSD directions that are orthogonal to the approximated CHIM. Additionally, another key difference is in the search domain: LineBO and BOIDS define one-dimensional lines in the input space, while our MOBO- OSD formulates one-dimensional search directions in the output space. Furthermore, LineBO and BOIDS are primarily designed to make high-dimensional problems tractable, while MOBO-OSD focuses on generating well-distribe
set of Pareto optimal points on the Pareto front. Therefore, LineBO and BOIDS are fundamentally different approaches and are not compatible with the MOBO settings. In contrast, MOBO- BEBO-OSD is specifically designed for MOBO problems, with the core purpose of generating orthogonal search directions relative to the approximated CHIM ensure a well-distributed coverage of the Pareto front.
A.6 Details of Benchmark Problems
We present the details of all benchmark problems in Table 2 The reference points for synthetic problems are similar to many MOBO works [33 To compute the reference point for real-world
23

---

problems, we follow common rule [3 33 50
and generate a pool of observed data points to
compute the reference point via the pool's nadir and ideal points as r = y madii + 0.1 (ynadir y yideal). All baselines use similar reference points both when running the code implementation (if required) and when computing the hypervolume difference as comparison metric.
Table 2:
Details of synthetic (ZDT1, VLMOP2, DTLZ22-2,, DTL.22-M3, DTLZ2-M4) and real-world (Speed Reducer, Car Side Design, Marine Design, Water Planning) benchmark problems.
Problem
D
M
Reference Point
DTLZ2-M2
5
2
111
DTLZ2-M3
5
11,1,
DTLZ2-M4
5
4
1.1.1I
ZDTI
5
(11.0, 11.0)
VLMOP2
5
2
(1.0,
Speed Reducer
7
(6735.9, 1761.17, 402.34)
Car Side Design
7
(38.89, 4.44, 12.94, 8.87)
Marine Design
6
(-210.44, 18970.82, 24111.07, 11.36)
Water Planning
3
(84349, 1461, 3101484, 12442800, 67030, 1.59)
A.7 Detailed Implementation
We implemented MOBO-OSD and all baselines in Python (version 3.10). The detailed implementation are as follows.
MOBO-OSD. For the surrogate model, we implement the GPs via GPyTorch (0 and BoTorch 3 We follow [33 and use Matérn 5/2 kernel with the ARD length-scales in the interval v10-3 V10 and signal variance in the interval v v10-3, v103, The Gaussian likelihood is modeled with standard homoskedastic noise in the interval [10 10-3).
For the number of points on approximated CHIM, we set the default value n, = 20 and present an ablation study of other settings in Sec. 5.2 For the scaling of confidence region, we the common 95% confidence interval, i.e., 5 1.96
(21. For the number of starting points when 4. Our code implementation can be found
solving MOBO-OSD subproblem, we set n,
https://g/ teg.t.t..... chtub..cm//ammlgg//mmoooooose
We use the default hyperparameter settings from the paper, including the Hedge algorithm for selecting acquisition functions, the pool of acquisition functions EI, UCB, TS and Identity (i.e., the posterior mean function). We use the open-sourced implementation at https //gi //github.. caaabb.ccommaallaa com/Alaleeh/PDBO
PDBO 0.
GEHVI 13.
We use the default hyperparameter settings from the paper. This includes the batch selection strategy that sequential greedy approach integrate over the unobserved outcomes. We use the open-sourced implementation at
use the default hyperparameter settings from the paper, including the number of buffer cells, max number of samples in each cell, buffer origin, graph-cut hyperparameters and solver NSGA-II hypeparameters. We use the open-sourced implementation at hhttps://githubb Com/yyushhegtiian/DGNNN
DGEMO
USeMO S
We use the default hyperparameter settings from the paper. For the acquisition function, use the EI 37 as it has the overall best performance and is widely compared in other works. We use the open-sourced implementation at jhttps://github www/bew..accccccccccoccccccoo.ee weeeakaaiiaatae i//MSMMM
that developed by Daulton et al. 13 to leverage batch setting. We use the settings follows: augmented Tchebychev scalariza-
qParEGO (1]. qParEGO a novel extension from ParEGO 31
tion [39], acquisition function with gradient solver and sequential greedy batch selection strategy.
We use the open-soureed implementation https: http::://bbooooro org/ http//oc//tttrriii
24

---

We use the default hyperparameter settings from the paper. This includes the number of random initialization points for optimizing JES, and the NSGA-II hyptrparrpperraeeetttr settings for solving the Pareto front. We use the open-sourced implementation at httttps: /enttps://githutte htt///gtthuu...... com www.coww//o.aeee..cte/ benml tu/ JES
JES [52).
NSGA-II [6]. We use NSGA-II implementation from рymoo 77. We use the default settings as follows: population size of 100, binary tournament selection, simulated binary crossover (probability p = 0.9, exponential distribution parameter 1) = 15) and polynomial mutation (probability p = 0.9,
exponential distribution parameter 1 20). We the open-sourced implementation at https
//pymoo. org/ arg/algors alg.r wawerrittmmmooomeeeeeeo nsga2. html
To the best of our knowledge, there is no available open-source implementation for NBI, therefore we recreated the implementation based on [[T] as follows. At first, we sequentially optimize each objective fn to obtain the individual minima. Then the convex hull of individual minima (CHIM) can be computed via the convex combination weight vectors B. We use the same Riesz. s-Energy method B as in MOBO-OSD to generate exact n. = 20 combination weight vectors, since the default point generation strategy cannot work with arbitrary ng. See Appendix A. for details. Then, for each of n, NBI subproblems defined at each point on the CHIM, we solve the NBI subproblem to compute the intersections between the CHIM and the NBI normal search directions. Until the evaluation budget depletes, we solve n.g NBI subproblems again (with new random initialization) to obtain solutions. We aggregate all observations found, including when optimizing for the individual minima, when optimizing NBI subproblems and when evaluating the NBI subproblem constraints. Finally, from the observed dataset, we construct the approximate Pareto front for hypervolume comparison against other baselines.
NBI
[..]
As NBI requires evaluating objective functions for the constraints, the method is not sample-efficient In fact, in our experiments, NBI often depletes the budget even when not having finished the first round of ng subproblems. We illustrate this statistics by presenting the number of function evaluations required to complete the individual minimum optimization and the first round of n NBI subproblems given two types of solver, gradient-based SQLSP (35) and gradient-free COBYLA 43 Details are shown in Table
We also present the performance NBI given these two solvers in Fig.
A.8 Additional Results for Baseline Comparison
On top of the comparison results between MOBO-OSD and the baselines shown in Sec. 5.1 we provide extra results for other batch settings b = {8, 10). which are shown in Fig respectively, Note that qEVHI has limited iterations due to the prohibitively high computational cost (exceeding 12 hours runtime) and memory required (exceeding 64 GB) on problems with M 2 4 objectives and batch size b {8, 10). MOBO-OSD and other baselines can fimish the experiments within the given similar resources.
A.9 Additional Results for Ablation Study
Figs and S present full results of the ablation study on ni parameter on all benchmark problems, on both sequential and batch settings, respectively. Overall, all variants have relatively similar performance, indicating the robustness of MOBO-OSD with respect to ng parameter.
A.10 Computing Infrastructure
We run experiments computing server with Dual CPU of type AMD EPYC 7662 (total of 128 Threads, 256 CPUs). Each experiment is allocated 8 CPUs and 64GB Memory. The server is installed with Ubuntu (20.04.3 LTS) Operating System.
A.11 Computational Complexity
We provide the theoretical computational complexity analysis as follows. Let N and denote number of surrogate model training and testing points, respectively, and let and M denote the number of dimensions and objectives, respectively. The computational complexity for the approximated CHIM and OSD formulation O(NM) as these steps require iterating over N data points with A dimenniionl output. The GP training cost is well-known to scale cubically with the
25

---

VLMOP2 (M-2)
(M-2)
DTLZ2-M2
Log trv difference
10
10"
10
10
10
100
200
200
150
200
DTLZ2-M3
DTLZ2-MM M4
Speed Reducer (M-3)
MOBO-OSD
Log IVV difference
(ours)
10
PDBO
DGEMO
10"
QEHVI
10
USEMO-EI
10
qParEGO
50
100
1500
200
50
100
150
200
50
100
150
200
Car Side Design (M-4)
Marino Design (M=4)
Water Planning (M-6)
NSGA-II
Log HIV difference
10"
10
10"
100
200
100
200
50
150
Number of evaluations
Number of evaluations
Number of evaluations
Figure 5: Comparison of MOBO-OSD against the SOTA baselines on 5 synthetic and 4 real-world benchmark problems in batch setting (batch size 8). Note that qEVHI has limited iterations due the prohibitively high computational cost and memory required on problems with M 2 4 objectives.
VLMOP2 (M-2)
ZDT1 (M-2)
D1LZ2-M2
Log tTV difference
10
10"
10
10"3
10"
100
200
150
200
50
150
200
DTLL2-M3
D'LL2--M4
Speed Reducer (M-3)
MOBO-OSD
Log IVV difference
(ours)
10
PDBO
DGEMO
10"
ĄEHVI
10"
USEMO-EI
10
qParEGO
50
100
150
200
50
100
130
200
50
100
150
200
Car Side Design (M=4)
Marine Design (M=4)
Water Planning (M=6)
NSGA-11
Log HV difference
10
10"
10
100
200
200
50
Number of evaluations
Number of evaluations
Number of evaluations
Figure 6: Comparison of MOBO-OSD against the SOTA baselines on 5 synthetic and 4 real-world benchmark problems in batch setting (batch size 10). Note that qEVHI has limited iterations due to the prohibitively high computational cost and memory required on problems with M 2 4 objectives.
26

---

DTLZ2-M2
DTLZ2-M3
DTLZ2-M4
Log HV difference
10
10"
50
100
150
200
50
100
150
200
50
100
150
200
VLMOP2 (M-2)
ZDT1 (M-2)
Speed Reducer (M-3)
Log difference
10"
10
10"
100
150
200
100
150
200
Car Side Design (M-4)
Marine Design (M-4)
Water Planning (M-6)
Log difference
10a
10"
10a
10'
10"
50
100
150
200
50
100
150
200
50
100
150
200
Number of ovaluations
Number of evaluations
Number of evaluations
n beta = 10
n beta = 20 (default)
- n beta 50
n_ beta 100
Figure 7: Ablation study the effect of n, parameter (batch 1). Overall, all variants have similar performance, indicating the robustness MOBO-OSD with respect to ng parameter.
DTLZ2-M2
DTLZ2-M3
DTLZ2-M4
Log HV difference
10
10
10"
50
100
150
200
50
100
150
200
50
100
150
200
VLMOP2 (M-2)
ZDTI (M-2)
Speed Reducer (M-3)
Log LogHV differnnce
10
10
10
10"
10"
100
150
200
100
130
200
150
200
Car Side Design (M-4)
Marine Design (M-4)
Water Planning (M-6)
Log HV difference
1
10"
10"
100
150
200
100
150
200
50
100
150
200
Number of evaluations
Number of evaluations
Number of evaluations
n beta - 10
n beta 20 (default)
n. beta 50
beta 100
Figure 8: Ablation study the effect of ns parameter (batch 4). Overall, all variants have similar performance, indicating the robustness of MOBO-OSD with respect to ny parameter.
27

---

Table 3: Number of required function evaluations to complete the first round of n. NBI subproblems, including finding individual optima and solving ni NBI subproblems. Gradient-basee solver cost significantly more budget due to gradient computation via finite difference method. Generally, NBI depletes all 200 evaluation budget before finishing the first round of optimization.
Individual Objectives
Subproblems
Subproblems (Gradient (Gradient-based solver)
(Gradient Srradient-frre solver)
VLMOP2
139.20 + 32.06
259.20 = 32.06
8049.70 70 = 1750.33
ZDTI-M2
24.00 0.00
144.00 = 0.00
720.20 = 29.50
DTLZ2-M2
30.00 = 9.30
150.00 = 9.30
898.20 = 22.52
DTLZ2-M3
36.60 + 1.80
156.60 + 1.80
1118.90 + 213.97
DTLZ2-M4
54.00 + 8.90
174.00 8.90
2050.80 + 1176.62
Speed Reducer
72.00 + 25.04
232.00 = 25.04
7680.70 + 1023.34
Car Side Design
109.60 + 10.15
269.60 = 10.15
5141.30 + 1105.41
Marine Design
139.30 + 22.45
279.30 + 22.45
7859.30 + 1032.65
Water Planning
44.00 + 0.00
124.00 + 0.00
124.00 + 0.00
VLMOP2 (M-2)
ZDT1 (M-2)
DTLZ2-M2
Lo HV difforence
0.4
0.3
0.3
10
0.2
10
10
100
200
100
200
100
200
DTT 733M3
DTT.72-M4
Speed Reducer (M=3)
Log HV difforence
0.8
10
0.7
0.94
10
NBI
0.8
(grad-free)
0.5
0.7
NBI
0.6
(grad-based)
0.4
200
200
100
Car Side Design (M-4)
Marine Design (M-4)
Water Planning (M-6)
Log HV difforence
6x 10
x 10
5 10
4 10
4 10
30
10
20
100
200
100 Number of evaluations
200
100
200
Number of evaluations
Number of evaluations
Figure 9: Performance of NBI when using different solvers for the NBI subproblems. Overall, no variant is consistently better than the other.
number of training samples, resulting in a computational complexity of O(V3). The computational complexity for solving each MOBO-OSD subproblem is O(D + N2),, which comprises of two dominated components: the SLSQP solver with cost of (D3)) and the approximated cost of O (N)') associated with evaluating the mean vector, standard deviation vector, and their corresponding gradients. For the Pareto Front Estimation step, following Schulz et al. [46] and Konakovic Lukovic al. [3] , the computational cost is O(D3 Nn). Finally, for the batch Hypervolume Improvement acquisition function, following Daulton et al. [3], the computational complexity is ((M K(25 1)), where is the number of disjoint partitions for box decompositions and is the batch size.
Additionally. we provide the empirical runtime for the proposed method and all baselines in Ta- bles 4 5 6 and 7 for batch size = {1, 4, 8, 10), respectively. Note that qEHVI is significantly more computationally (in terms of runtime and required memory) than other baselines and MOBO-OSI especially when = (8. 10), hence can only have partial results on problems with M 2 4 objectives. All NSGA-II runs cost less than 0.0 so we exclude the method from the results. Moreover, symbol "_" indicates the method not applicable for the corresponding baseline, according to the authors' implementation.
28

---

Table 4: Runtime comparison (seconds per iteration) for batch size 1. Overall, our proposed method, MOBO-OSD, have affordable time complexity. JES is significantly more computational expensive even in sequential setting. Symbol indicates the method is not applicable for the corresponding benchmark problems.
MOBO-OSD
USeMO (EI)
PDBO
qEHVI
JES
DGEMO
qParEGO
(Ours)
VLMOP2
16.18
32.02
2.14
67.30
9.46
1.89
0.60
ZDTI-M2
7.50
22.47
1.89
40.67
11.84
2.28
0.50
DTLZ2-M2
8.34
23.74
2.95
46.99
8.74
1.91
0.50
DTLL2-M3
11.72
15.38
10.32
53.69
10.63
2.75
0.68
DTLZ2-M4
19.85
14.66
55.27
126.97
3.75
0.88
Speed Reducer
17.18
13.38
6.91
115.07
16.31
2.67
0.71
Car Side Design
24.84
15.65
16.31
105.03
4.00
0.92
Marine Design
22.72
17.03
17.17
91.56
3.50
1.81
Water Planning
30.16
73.66
150.93
846.36
5.34
1.72
Table Runtime comparison (seconds per iteration) for batch size Overall, our proposed method, MOBO-OSD, have affordable time complexity. qEHVI is significantly more computational expensive with increasing batch size. Symbol "_" indicates the method is not applicable for the corresponding benchmark problems.
MOBO-OSD
USeMO (EI)
PDBO
ĄEHVI
DGEMO
qParEGO
(Ours)
VLMOP2
4.34
8.23
2.20
2.33
0.69
0.45
ZDT1-M2
2.31
5.86
1.63
2.81
0.65
0.39
DTL.Z2-M2
2.51
7.23
2.60
2.13
0.53
0.39
DTLZ2-M3
3.80
4.95
8.82
2.86
0.78
0.52
DTLZ2-M4
10.58
4.45
31.73
1.14
0.65
Speed Reducer
4.65
3.84
14.06
3.96
0.72
0.53
Car Side Design
13.90
5.11
71.46
1.23
0.66
Marine Design
10.03
5.38
57.27
1.05
1.83
Water Planning
24.25
21.47
272.16
2.06
1.11
Table 6: Runtime comparison (seconds per iteration) for batch size Overall, our proposed method, MOBO-OSD, have affordable time complexity. qEHVI is significantly more computational expensive with increasing batch size. Symbol "_" indicates the method is not applicable for the corresponding benchmark problems.
MOBO-OSD
USeMO (EI)
PDBO
qEHVI
DGEMO
qParEGO
(Ours)
VLMOP2
1.69
3.98
7.25
1.07
0.28
0.35
ZDT1-M2
1.11
3.05
4.47
1.35
0.42
0.34
DTL.Z2-22
1.14
3.56
10.00
1.03
0.33
0.34
DTLZ2-M3
1,71
2.22
100.07
1.43
0.55
0.46
DTLZ2-M4
2.42
2.33
449.26
0.66
0.56
Speed Reducer
2.42
2.36
66.35
2.16
0.36
0.46
Car Side Design
2.50
3.22
554.83
0.78
0.55
Marine Design
3.13
2.97
374.85
0.50
1.64
Water Planning
11.64
11.97
651.57
0.97
0.79
29

---

VI.MOP2 (M-2)
ZDT1 (M-2)
DTT
10
10
IGD
10
10
10"
10"
MORO-OSD
100
200
100
200
100
200
(ours)
DTLZ2-M3
DTLZ2-M4
Speed Reducer (M-3)
10
10"
PDBO
DGEMO
qEHVI
1GD
JES
USEMO-EI
qParEGO
10
200
NSGA-II
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
10
10"
(gra--base)
NBI
1GD
10"
10
10
100
200
100
200
100
200
Number evaluations
Number of ovaluations
Number of ovaluations
Figure 10: Comparison using IGD indicator of MOBO-OSD against the baselines on 5 synthetic and 4 real-world benchmark problems with batch size 1.
Table 7: Runtime comparison (seconds per iteration) for batch size 10. Overall, our proposed method, MOBO-OSD, have affordable time complexity. qEHVI is significantly more computational expensive with increasing batch size. Symbol "_" indicates the method is not applicable for the corresponding benchmark problems.
MOBO-OSD
USeMO (EI)
PDBO
qEHVI
DGEMO
qParEGO
(Ours)
VLMOP2
1.71
3.29
45.84
0.90
0.38
0.41
ZDTI-M2
1.13
2.57
18.72
1.11
0.67
0.41
DTLZ2-M2
1.15
2.92
57.72
0.82
0.45
0.40
DTLZ2-M3
1.78
1.89
229.09
1.25
0.46
0.55
DTI.72-M4
2.73
2.07
446.87
1.00
0.70
Speed Reducer
2.49
1.89
158.63
1.67
0.42
0.54
Car Side Design
2.77
2.47
498.51
0.87
0.67
Marine Design
3.48
2.35
468.98
0.61
2.42
Water Planning
21.78
11.20
710.14
1.19
0.94
A.12
Other Comparison Metrics
compare the performance of MOBO-OSD and the baselines using other comparison metries, including IGD 10, IGD+ (25] and e-indicator [60] Figs 10 12 and 13 show the IGD metrics. Figs
14
15
16 and show the IGD+ metrics. Figs 18 20 and 21 show the e-indicator. Overall,
MOBO-OSD shows competitive performance regardless of performance metrics.
30

---

VI MOP2 (M-2)
7DT1 (M-2)
DTT
10
10
10
1G0
10"
10"
10
10"
10
100
200
100
200
100
200
Speed (M-3)
MOBO-OSD
10
10'
(ours)
PDRO
1GD
qEHVI
10
USEMO-EI
qParEGO
100
200
200
200
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
NSGA-II
10
10
1GD
10
10'
10
10"
100
200
100 e'evaluations
200
100
200
evaluations
Number
Figure 11: Comparison using IGD indicator of MOBO-OSD against the baselines on 5 synthetic and 4 real-world benchmark problems with batch size 4.
VLMOP2 (M-2)
ZDTI (M-2)
DTLZ2-M2
10
10
10
IGD
10"
10
10
10
100
200
100
200
100
200
D1LL2-M3
D'LLZ--M
Speed Reducer (M-3)
MOBO-OSD
10a
(ours)
10
PDBO
1GD
DGEMO
qEHVI
10"
USEMO-FI
100
200
10"
100
200
200
qParEGO
Car Side Design (M-4)
Marine Design (M-4)
Waterr Planning (M-6)
NSGA-11
10
1GD
10"
10
10
100
200
100
200
100
200
Number evaluaaioons
Number of d'evvluations
Number of evaluations
Figure 12: Comparison using IGD indicator of MOBO-OSD against the baselines on 5 synthetic and 4 real-world benchmark problems with batch size 8.
31

---

VI. (M-2)
7DT1 (M-2)
DTT 72-M2
10
10
1GD
10"
10
10
100
200
100
200
100
200
Speed Reducer (M-3)
MOBO-OSD
10"
(ours)
10
PDRO
1GD
qEHVI
10
USEMO-EI
qParEGO
100
200
200
100
200
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
NSGA-II
10
10"
1GD
10
10
10
10"
100
200
100 d'evaluations
200
100
200
evaluations
Number
Figure 13: Comparison using IGD indicator of MOBO-OSD against the baselines on 5 synthetic and 4 real-world benchmark problems with batch size 10.
VLMOP2 (M-2)
ZDTI (M-2)
DTLZ2-M2
10
10
10
1GD
10
10"
10
10"
MOBO-OSD
100
200
100
200
100
200
(ours)
D1LL2-M3
D'LLZ--A
Speed Reducer (M-3)
10
10"
PDBO
DGEMO
IGD1
QEUII
JES
10
USEMO-EI
qPaEEGO
100
200
200
200
NSGA-II
Car Side Design (M-4)
Marine Design (M-4)
Waterr Planning (M-6)
NB1
(grad-based)
IGD
10
10a
10
100
200
100
200
100
200
Number evaluaaions
Number of d'evaluations
Number of evaluations
Figure 14: Comparison using IGD+ indicator of MOBO-OSD against the baselines on 5 synthetic and 4 real-world benchmark problems with batch size 1.
32

---

VI MOP2 (M-2)
7DT1 (M-2)
DTT
10
10
10
10"
1GB
10"
10
10"
10
100
200
100
200
100
200
Speed (M-3)
MOBO-OSD
10
10'
(ours)
PDRO
1GDI
qEHVI
10
USEMO-EI
qParEGO
100
200
200
200
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
NSGA-II
10
10%
IGDI
10
10'
10
10"
100
200
100 d'evaluations
200
100
200
evaluations
Number
Figure 15: Comparison using IGD+ indicator of MOBO-OSD against the baselines on 5 synthetic and real-world benchmark problems with batch size
VLMOP2 (M-2)
ZDTI (M-2)
DTLZ2-M2
10
10
10
1GD
10"
10
10
10
100
200
100
200
100
200
D1LL2-M3
D'LLZ--M
Speed Reducer (M-3)
MOBO-OSD
10a
(ours)
10
PDBO
IGDI
DGEMO
qEHVI
10
10"
USEMO-FI
100
200
10"
100
200
200
qParEGO
Car Side Design (M-4)
Marine Design (M-4)
Waterr Planning (M-6)
NSGA-11
10
IGD
10"
10
10
100
200
100
200
100
200
Number evaluaaioon
Number of evvlluations
Number of evaluations
Figure 16: Comparison using IGD+ indicator of MOBO-OSD against the baselines on 5 synthetic and 4 real-world benchmark problems with batch size 8.
33

---

MOP (M-2)
7DT1 (M-2)
DTT 72-M2
10
10
1GD
10"
10
10
100
200
100
200
100
200
DTLZ2-M3
Speed Reducer (M-3)
MOBO-OSD
10"
(ours)
10
PDRO
1GDI
qEHVI
10
USEMO-EI
qParEGO
100
200
200
200
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
NSGA-II
10
10"
IGDI
10
10
10
10"
100
200
100 d'evaluations
200
100
200
evaluations
Number
Figure 17: Comparison using IGD+ indicator of MOBO-OSD against the baselines on 5 synthetic and real-world benchmark problems with batch size 10.
VLMOP2 (M-2)
ZDTI (M-2)
DTLZ2-M2
10
10
10'
Eppsiln
10
10a
10
10
10
-2
MOBO-OSD
100
200
100
200
100
200
(ours)
DTLLZ-M4
Speed Reducer (M-3)
PDBO
10
10"
10
DGEMO
Epsilon
QEUII
10
JES
10
USEMO-EI
qPaEEGO
100
200
200
200
NSGA-II
Car Side Design (M-4)
Marine Design (M-4)
Water Planning (M-6)
NB1
10
(grad-based)
10
Epsilln
10
10a
100
200
100
200
100
200
Number evaluaaionn
Number of d'evaluations
Number of evaluations
Figure 18: Comparison using e-indicator of MOBO-OSD against the baselines on 5 synthetic and real-world benchmark problems with batch size 1.
34

---

MOP2 (M-2)
7DT1 (M-2)
D2
10
10
Eppllon
10
10"
10
10"3
10
100
200
100
200
100
200
Speed (M-3)
MOBO-OSD
10"
(ours)
PDRO
Epsilan
10'
qEHVI
10
USEMO-EI
10
10
qParEGO
100
200
200
200
Car Side Design (M=4)
Marine Design (M-4)
Water Planning (M-6)
NSGA-II
10"
Epeilon
10
10'
10
100
200
100 d'evaluations
200
100
200
Number evaluations
Number evalluation
Figure 19: Comparison using e-indicator of MOBO-OSD against the baselines on 5 synthetic and real-world benchmark problems with batch size 4.
VLMOP2 (M-2)
ZDTI (M-2)
DTLZ2-M2
10
10
Eppilln
10a
10
10
10
10
100
200
100
200
100
200
DILL2-M3
DTLLZ-MA
Speed Reducer (M-3)
MOBO-OSD
10"
(ours)
10
PDBO
Epsilon
DGEMO
qEHVI
USEMO-FI
100
200
10"
200
10
200
qParEGO
Car Side Design (M-4)
Marine Design (M-4)
Waterr Planning (M-6)
NSGA-11
Epsillen
10
10
10
100
200
100
200
100
200
Number evaluaaions
Number of e'evaluations
Number of evaluations
Figure 20: Comparison using e-indicator of MOBO-OSD against the baselines on 5 synthetic and real-world benchmark problems with batch size 8.
35

---

VLMOP2 (M-2)
ZDT1 (M-2)
DTLZ2-M2
10
10
10
Epailem
10
10
10
10
100
200
100
200
100
200
DTLZ2-M3
DYLL22-4
Speed Reducer (M-3)
MOBO-OSD
10"
(ours)
PDBO
Epsilon
10'
qEHVI
USEMO-EI
10
qParEGO
100
Car Side Design (M=4)
Marine Design (M=4)
Water Planning (M=6)
NSGA-II
10
Epailon
10
100
200
100
200
160
200
Number evaluaaions
Number of evaluations
Number of evaluations
Figure 21: Comparison using e-indicator MOBO-OSD against the baselines on synthetic and real-world benchmark problems with batch size 10.
36


Reviews:
[{"review_id": "gSBYC6EBWu", "reviewer": "Reviewer_6cNg", "summary": "This paper introduces MOBO-OSD, a novel algorithm for batch multi-objective Bayesian optimization (MOBO). The core idea is to adapt the Normal Boundary Intersection (NBI) method to the limited-budget BO setting. The algorithm works by first defining an approximated convex hull of individual objective minima (CHIM). It then generates a set of well-distributed Orthogonal Search Directions (OSDs) and solves a series of constrained single-objective subproblems along these directions to identify candidate solutions on the Pareto front. To enhance solution density, the method incorporates a Pareto Front Estimation (PFE) technique to explore the local neighborhood of these candidates. Finally, a batch selection strategy using the Hypervolume Improvement acquisition function, Kriging Believer, and a diversity-promoting mechanism is employed to select points for parallel evaluation. Extensive experiments show that MOBO-OSD consistently outperforms state-of-the-art baselines on a variety of problems.", "questions": "1. Could you further elaborate on the fundamental advantages of the OSD-based subproblem formulation compared to a well-designed scalarization method that uses a similarly diverse set of weight vectors for batch selection? While empirically superior, the conceptual distinction could be clarified.\n\n2. Your approach of searching along 1D Orthogonal Search Directions in the objective space is conceptually similar to methods that use 1D subspaces for high-dimensional single-objective BO (e.g., Kirschner et al., 2019; BOIDS). Have you considered this connection, and do you see any potential for cross-pollination of ideas between these domains?\n\n**References**\n- Ngo, Lam, Huong Ha, Jeffrey Chan, and Hongyu Zhang. \"BOIDS: High-Dimensional Bayesian Optimization via Incumbent-Guided Direction Lines and Subspace Embeddings.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, pp. 19659-19667. 2025.\n- Kirschner, Johannes, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. \"Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces.\" In International Conference on Machine Learning, pp. 3429-3438. PMLR, 2019.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Novel Geometric Approach for Diversity: The paper's main strength is its novel adaptation of the NBI method to MOBO. By defining search directions geometrically (the OSDs), the algorithm has a structured and principled mechanism for generating a diverse set of candidate solutions, directly addressing a key challenge in MOBO.\n- Effective Combination of Techniques: MOBO-OSD intelligently combines multiple powerful techniques. The use of Riesz s-Energy to generate well-distributed OSDs, the PFE technique to enrich the candidate pool without solving an excessive number of subproblems, and a thoughtful batch selection strategy all contribute to a robust and effective algorithm.\n- Strong and Comprehensive Empirical Validation: The authors conduct extensive experiments on nine benchmark problems (with two to six objectives) against seven state-of-the-art baselines in both sequential and batch settings. The results consistently demonstrate the superiority of MOBO-OSD in terms of hypervolume difference, which is a strong piece of evidence for the method's effectiveness.\n- Designed for Batch Optimization: The algorithm is designed from the ground up to support batch optimization. The batch selection strategy, which combines Kriging Believer with a mechanism to ensure candidates are drawn from different exploration zones, is a practical and well-reasoned approach to accelerate the optimization process with parallel resources.\n\n**Weakness**\n-  Incomplete Contextualization within the MOBO Landscape: The paper's positioning could be improved with a broader discussion of the existing MOBO literature.\n      \n      - Conceptual Overlap with Scalarization: The core mechanism involves solving multiple single-objective subproblems. While the generation of these subproblems is geometrically motivated by OSDs, the paper could provide a deeper discussion on how this fundamentally differs from advanced scalarization techniques that also generate diverse weight vectors for batch optimization.\n      -  Omission of related MOBO baseline: The literature review focuses entirely on methods that approximate the full Pareto front but omits the following work: Zuluaga, Marcela, Andreas Krause, and Markus Püschel. \"e-pal: An active learning approach to the multi-objective optimization problem.\" Journal of Machine Learning Research 17, no. 104 (2016): 1-32.\n\n- Limited Discussion of Related Single-Objective Methods: The paper misses an opportunity to connect its directional search approach to similar ideas in high-dimensional single-objective BO. Methods like those by Kirschner et al. (2019) and Ngo et al. (2025) also leverage 1D line searches to make complex problems tractable. Discussing these parallels would better situate MOBO-OSD within the broader BO literature.\n- Algorithmic Complexity and Hyperparameters: The complete MOBO-OSD algorithm is a multi-stage pipeline with several components and associated hyperparameters . While an ablation study on $n_\\beta$ is provided, a more comprehensive analysis of the method's complexity and sensitivity to these different moving parts would be beneficial.", "was_revised": true, "final_justification": "This paper is recommended for acceptance. Its core contribution, MOBO-OSD, is a novel and effective algorithm for batch multi-objective Bayesian optimization. The method's strength lies in its concrete geometric approach: adapting Normal Boundary Intersection (NBI) with Orthogonal Search Directions (OSDs) to ensure a diverse and well-structured exploration of the Pareto front. This is intelligently combined with a Pareto Front Estimation (PFE) technique for density and a Kriging Believer strategy for efficient batch selection.", "rebuttal": "We thank the reviewer for their insightful review. We address the questions and comments as follows:\n\n# Questions\n\n> 1. Could you further elaborate on the fundamental advantages of the OSD-based subproblem formulation compared to a well-designed scalarization method that uses a similarly diverse set of weight vectors for batch selection? While empirically superior, the conceptual distinction could be clarified.\n\nThere are several advantages of our OSD subproblem formulation compared to the most closely related scalarization technique - linear scalarization (LS) [37]. First, *LS generates search directions at random*, which can be less efficient than the data-driven search directions proposed in our OSD formulation. Second, even when LS weight vectors are well-distributed (e.g., using Riesz s-Energy [8]), *LS is limited to finding solutions on the convex regions of the Pareto front* [A]. In contrast, our proposed OSD formulation is capable of identifying solutions on the Pareto front of arbitrary shape, including both convex and non-convex regions. Note that due to this limitation, LS has been superseded by the most widely used scalarization technique - Tchebychev scalarization (TS) [37]. TS can handle non-convex PF and has been widely selected in other works when evaluating scalarization techniques [29, 37, 12, 31]. However, one critical drawback of TS lies in its non-smooth formulation caused by the maximization operator, which makes TS suffers from non-differentiability and slow convergence [B]. On the other hand, the MOBO-OSD subproblem formulation is differentiable, either analytically under common kernels or by automatic differentiation via computational graphs, as described in Appendix A.4. Empirically, our proposed method consistently outperforms baselines employing TS, such as qParEGO. Figures 2, 3, 8 and 9 show that MOBO-OSD outperforms qParEGO across different benchmark functions with diverse PF characteristics, including convex (ZDT1) and concave (DTLZ2). We will include this discussion in the revised version of the paper. \n\n> 2. Your approach of searching along 1D Orthogonal Search Directions in the objective space is conceptually similar to methods that use 1D subspaces for high-dimensional single-objective BO (e.g., Kirschner et al., 2019; BOIDS). Have you considered this connection, and do you see any potential for cross-pollination of ideas between these domains?\n\nLineBO (Kirschner et al. (2019)) and BOIDS (Ngo et al. (2025)) are fundamentally different approaches and are not directly compatible with our method. The primary distinction lies in the formulation of these 1D lines: both LineBO and BOIDS construct search directions with some degree of *randomness*, whereas MOBO-OSD relies on *deterministic* OSD directions that are orthogonal to the approximated CHIM. Additionally, another key difference is in the search domain: LineBO and BOIDS define 1D lines in the *input space*, while our MOBO-OSD formulates 1D search directions in the *output space*. Furthermore, as noted by the reviewer, LineBO and BOIDS are primarily designed to *make high-dimensional problems tractable*, while MOBO-OSD focuses on the *generating a well-distributed set of Pareto optimal points* on the Pareto front. For these reasons, we argue that cross-pollinating OSD with uniformly random directions (as in LineBO) or incumbent-guided directions (as in BOIDS) might not be feasible in the MOBO problems, as it would compromise the core purpose of OSD: producing orthogonal directions to the approximated CHIM to ensure well-distributed coverage of the Pareto front. \n\n# Weakness\n\n> Incomplete Contextualization within the MOBO Landscape: The paper's positioning could be improved with a broader discussion of the existing MOBO literature…\n\nRegarding the conceptual overlap with scalarization, please refer to our response to Question 1 for a discussion of scalarization techniques. We will include this discussion in the revised version of the paper.  Regarding the omission of the related MOBO baseline ($\\\\varepsilon$-PAL), we would like to emphasize that $\\\\varepsilon$-PAL is specifically designed for *finite discrete input spaces*, which is why it was not mentioned. Furthermore, prior studies [37, 5, 22] have highlighted that $\\\\varepsilon$-PAL shares methodological similarities with [C], which has been empirically outperformed by our baseline USeMO [5]. Nevertheless, we will include a brief discussion of $\\\\varepsilon$-PAL in the revised version of the paper.\n\n> Limited Discussion of Related Single-Objective Methods: The paper misses an opportunity to connect its directional search approach to similar ideas in high-dimensional single-objective BO. Methods like those by Kirschner et al. (2019) and Ngo et al. (2025) also leverage 1D line searches to make complex problems tractable. Discussing these parallels would better situate MOBO-OSD within the broader BO literature.\n\nPlease see the response to Question 2 for detailed explanation. We will include this discussion to the revised version of the paper.\n\n> Algorithmic Complexity and Hyperparameters: The complete MOBO-OSD algorithm is a multi-stage pipeline with several components and associated hyperparameters. While an ablation study on $n_\\beta$ is provided, a more comprehensive analysis of the method's complexity and sensitivity to these different moving parts would be beneficial.\n\nRegarding the algorithmic complexity, we would like to note that a comprehensive runtime comparison between MOBO-OSD and the baselines is already provided in Tables 4, 5, 6 and 7 in Appendix A.9, covering all benchmark problems and batch settings $b=\\\\{1,4,8,10\\\\}$. In addition, we have now conducted a theoretical time complexity analysis, summarized as follows. The time complexity is $\\\\mathcal O(NM)$ for the approximated CHIM and OSD formulation, $\\\\mathcal O(D^3+Nn+N^2n)$ for solving each MOBO-OSD subproblem, and  $\\\\mathcal O(D^3+Nn)$ for the Pareto Front Estimation step, where $N$ and $n$ denote the number of surrogate model training and testing points, respectively, $D$ and $M$ denote the number of dimensions and objectives, respectively. We will include this theoretical time complexity analysis in the revised version of the paper.\n\nRegarding the sensitivity analysis of the hyperparameters in MOBO-OSD’s components, we presented two ablation studies for the two most important components of MOBO-OSD in Section 5.1.2. The first ablation study targets the OSD component, specifically examining the impact of the number of OSDs, $n_\\\\beta$, on overall performance. Figures 4 and 6 demonstrate that the number of OSDs, $n_\\\\beta$, has minimal effect on MOBO-OSD’s overall performance, indicating the robustness of the method. The second ablation study focuses on the Pareto Front Estimation (PFE) component, evaluating its necessity within MOBO-OSD. Table 1 demonstrates that although PFE enhances the efficiency of MOBO-OSD, it is not a required component, as comparable performance can still be achieved by increasing the density of the $\\\\mathcal{U}(\\\\boldsymbol\\\\beta)$ set. We will consider adding more sensitivity analysis for other minor components in the revised version of the paper.\n\n## References\n\n[A] Zhang, Richard, and Daniel Golovin. \"Random hypervolume scalarizations for provable multi-objective black box optimization.\" *International conference on machine learning*. PMLR, 2020.\n\n[B] Lin, Xi, et al. \"Smooth Tchebycheff Scalarization for Multi-Objective Optimization.\" *International Conference on Machine Learning*. PMLR, 2024.\n\n[C] Ponweiser, Wolfgang, et al. \"Multiobjective optimization on a limited budget of evaluations using model-assisted-metric selection.\" *International conference on parallel problem solving from nature*. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008."}, {"review_id": "1SPMy3toy8", "reviewer": "Reviewer_51fE", "summary": "This paper proposes a novel algorithm for multi-objective Bayesian optimization (MOBO) that also supports batched evaluations. MOBO is the problem setting in which one aims to optimize multiple, potentially competing objectives. Therefore, instead of searching for solutions that minimize all objectives, one aims to find Pareto optimal solutions, that is, solutions where one cannot find a better solution for an objective without worsening another objective.\n\nSeveral algorithms for MOBO have been proposed, but they are often either not scalable in the number of objectives or suffer from high computational cost. \n\nThis paper aims to fill this gap by proposing an algorithm that is more scalable in the number of objectives and does not exhibit high computational cost. To this end, it approximates the convex hull of individual object minima (CHIM) to define search directions orthogonal to the CHIM. These orthogonal search directions are then used to find Pareto-optimal points, each of which is further augmented to find a dense set of points close to the Pareto front. \n\nThese points then serve as candidates for the next Bayesian optimization iteration. Batch BO is realized by conditioning the GP on previously selected points, using the posterior mean as a hallucinated outcome (Kriging believer).\n\nThe proposed method is evaluated on a wide range of benchmarks and shows promising performance.", "questions": "* How many objectives can MOBO-OSD handle? Can you construct (synthetic) benchmarks with arbitrarily many objectives?\n* What's the significance of the batching? Can't the Kriging believer be combined with other MOBO AFs as well?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths__\n\n* The paper is excellently written, and the presentation of the relatively complex algorithm is clear.\n* The algorithm outperforms the competitors throughout the bank.\n* The paper has high-quality figures that help understand the algorithm.\n\n__Weaknesses__\n\n* The methodology for batching seems disconnected from the paper and is orthogonal to the multi-objective design.\n* Not all runs in the empirical evaluation have converged or are close to convergence, so it's not always clear if MOBO-OSD actually beats the other methods in the end.\n\n__Minor Comments__\n\n* Typo in the Figure in Section 4.1 ('Inidividual')\n* The small Figures in 4.1 and 4.2 also deserve a number and a caption.", "was_revised": true, "final_justification": "This paper provides an effective and well-designed solution for a relevant problem. I see room for improvement in the empirical evaluation and in the presentation of the batching mechanism but recommend the acceptance of the paper.", "rebuttal": "We thank the reviewer for their insightful review. We address the questions and comments as follows:\n\n# Questions\n\n> How many objectives can MOBO-OSD handle? Can you construct (synthetic) benchmarks with arbitrarily many objectives?\n\nMOBO-OSD is designed to handle *any arbitrary number of objectives*, as it does not require any specific data structure for the objective space and is still maintained feasible computational cost even in batch settings. While it is possible to construct synthetic benchmark functions that scale to arbitrarily many objectives, such as the DTLZ series [16], we follow the standard practice in many MOBO papers [5, 22, 48, 12, 13, 31] by performing evaluation using benchmark problems with up to 6 objectives and batch size of 10. Note that several state-of-the-art methods face scalability limitations under these settings. For example, DGEMO [31] requires a carefully designed performance buffer tailored to each specific number of objectives, and its current open-access implementation cannot scale beyond 3 objectives. Similarly, qEHVI [12] becomes prohibitively expensive in batch settings, as demonstrated in the runtime analysis provided in Tables 4, 5, 6 and 7 of Appendix A.9.\n\n> What's the significance of the batching? Can't the Kriging believer be combined with other MOBO AFs as well?\n\nAs mentioned in Section 1 (line 31), batching plays an important role in reducing the time and cost of expensive black-box optimization by *enabling parallel evaluations of objective functions*. Therefore, it is common practice for MOBO algorithms to be able to support batch settings [1, 5, 12, 31]. Regarding Kriging Believer (KB), it generally cannot be applied to other MOBO AFs because many MOBO AFs have their own batch mechanisms. For example, the qEHVI [12] AF sequentially integrates over unobserved data points when working with batch setting.\n\n# Weakness\n\n> The methodology for batching seems disconnected from the paper and is orthogonal to the multi-objective design.\n\nWe would like to emphasize that our methodology for batching (batch selection strategy) is tightly integrated with the core design of our proposed method. Specifically, *the batching mechanism directly follows the construction of OSD subproblems*: each point in the batch is associated with a distinct OSD and its corresponding exploration space $\\mathcal{T}$, as described in Section 4.4 and formalized in Eq. (4). This design ensures the batch maintains both strong hypervolume performance and diversity across the Pareto front. Therefore, the batching methodology is not orthogonal but rather fundamentally coupled to the multi-objective design.\n\n> Not all runs in the empirical evaluation have converged or are close to convergence, so it's not always clear if MOBO-OSD actually beats the other methods in the end.\n\nIn MOBO, where objective functions are *expensive* and *black boxes*, the goal is to find the Pareto set of optimal solutions using the least number of function evaluations [29, 5]. Figures 2, 3, 8 and 9 show that MOBO-OSD consistently find better Pareto sets - measured by the hypervolume indicator - and uses fewer evaluation budget to reach such performance compared to all baselines. It is also worth noting that an evaluation budget of 200 iterations is standard practice in recent state-of-the-art MOBO works [5, 22, 48, 12, 31].\n\n## Minor Comments\n\nThank you for the comments. We will fix these in the revised version of the paper."}, {"review_id": "WhmuSAv0Oi", "reviewer": "Reviewer_xpJo", "summary": "Their method is based on NBI, which computes the CHIM (convex hull of the individual minima) and works well when there is plenty of budget. Since MOBO problems have limited budget, they propose a technique to efficiently approximate the CHIM. Starting from this approximated CHIM, they generate orthogonal search directions (OSDs) evenly distributed, defined as constrained single-objective subproblems in which they look for the intersection points (candidates) with the Pareto front generated by the GPs. After solving these subproblems, they employ the Pareto Front Estimation (PFE) technique to increase the number of candidates (instead of solving more subproblems). Then, for each candidate they calculate its HVI, and if batch > 1 they use Kriging Believer to choose the rest of the batch candidates one by one, ensuring diversity among candidates. Finally, they claim that their method performs as well as or better than the state-of-the-art and that it can scale to any number of objectives in both sequential and batch evaluations.", "questions": "Have you considered using IGD [4], IGD^+ [5], and the ε-indicator [6] alongside HVI for batch selection? HVI maximizes dominated volume, IGD/IGD^+ ensure uniform coverage, and the ε-indicator guarantees worst-case bounds. What are your thoughts?\n\n[4] Coello, C. A. C., & Lamont, G. B. (2004). Applications of multi-objective evolutionary algorithms (Vol. 1). World Scientific.\n\n[5] Ishibuchi, H., Masuda, H., Tanigaki, Y., & Nojima, Y. (2015). Modified distance calculation in generational distance and inverted generational distance. In Evolutionary Multi-Criterion Optimization: 8th International Conference, EMO 2015, Guimarães, Portugal, March 29--April 1, 2015. Proceedings, Part II 8 (pp. 110-125). Springer International Publishing.\n\n[6] Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C. M., & Da Fonseca, V. G. (2003). Performance assessment of multiobjective optimizers: An analysis and review. IEEE Transactions on evolutionary computation, 7(2), 117-132.\n\n--\n\nThese sentences sound strange:\n\n - Many existing approaches rely on scalarization techniques [5, 8, 36], which often fail to capture a diverse Pareto front, leading to suboptimal **performance in performance** such as hypervolume.\n\n- In order to generate a **well-distributed** set of OSDs, it is essential to construct a **well-distributed** set of U(β) points. This is equivalent to generate a **well-distributed** set of {β} over a M -dimensional unit simplex, as U(β), by definition, is the linear transformation of β from a unit simplex to the approximated CHIM.", "limitations": "I have described the limitations I found in the Weaknesses section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n - Their explanatory figures help to understand their method.\n\n - They have run several benchmarks from the literature to test their method in both sequential and batch scenarios.\n\n - Their method outperforms all other methods on all the benchmarks they present.\n\nWeaknesses:\n\n - Their method only handles problems without noise.\n\n - They mention JES [1] and PFES [2] in the \"Related Work\" section but they do not include them in the experiments. While these methods can be somewhat costly, I think it is important to include them for M <= 4 in the sequential, non-batch, problems.\n\n- The greedy selection for the batch of points to evaluate is not the best option. Although the Hypervolume Subset Selection Problem is #P-hard, there are better alternatives [3].\n\n[1] Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi- objective bayesian optimization. Advances in Neural Information Processing Systems, 35: 9922–9938, 2022.\n\n[2] Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama. Multi-objective bayesian optimization using pareto-frontier entropy. In International conference on machine learning, pages 9279–9288. PMLR, 2020.\n\n[3] Guerreiro, Andreia P., Fonseca, Carlos M., & Paquete, Luís (2021). The Hypervolume Indicator: Computational Problems and Algorithms. ACM Computing Surveys (CSUR), 54(6), 1–42.", "was_revised": true, "final_justification": "After reading the authors' responses to me and the other reviewers, I've decided to raise my score.", "rebuttal": "We thank the reviewer for their insightful review. We address the questions and comments as follows:\n\n# Questions\n\n> Have you considered using IGD, IGD+, and the $\\varepsilon$-indicator alongside HVI for batch selection? HVI maximizes dominated volume, IGD/IGD+ ensure uniform coverage, and the $\\varepsilon$-indicator guarantees worst-case bounds. What are your thoughts?\n\nThe Hypervolume Improvement (HVI) metric is widely recognized as the most commonly used performance metric in general multi-objective optimization (MOO) and, in particular, multi-objective Bayesian optimization (MOBO) [29, 37, 5, 1, 22, 48, 12, 31]. Nonetheless, we agree that presenting additional metrics can provide further insights into our proposed method. We have now conducted comparisons using IGD, IGD+ and $\\varepsilon$-indicator metrics across all baselines, all benchmark problems, and batch settings. The IGD and IGD+ results show that MOBO-OSD converges rapidly towards 0, indicating the uniform coverage of the discovered Pareto fronts. The $\\varepsilon$-indicator results exhibit convergence toward 0, reflecting strong worst-case guarantees. Compared to the baselines, overall, MOBO-OSD still outperforms them under these new metrics.  We will include these additional metrics in the revised version of the paper.\n\n> These sentences sound strange…\n\nThank you for the comment. We will fix these sentences in the revised version of the paper.\n\n# Weakness\n\n> Their method only handles problems without noise.\n\nWe agree with this limitation, which has already been acknowledged in the Conclusion section of our manuscript. It is important to note that this limitation is not unique to our method but is also commonly encountered in existing MOBO approaches, including recent state-of-the-art baselines such as qEHVI [12] and DGEMO [31]. Reviewer RbX4 also agrees that this is a very minor weakness. Nonetheless, addressing noisy settings remains an important challenge and is a promising direction for our future work.\n\n> They mention JES [1] and PFES [2] in the \"Related Work\" section but they do not include them in the experiments. While these methods can be somewhat costly, I think it is important to include them for M <= 4 in the sequential, non-batch, problems.\n\nThe main reason we did not include PFES and JES is that they are information-theoretic (IT)-based acquisition functions (AFs), whereas our proposed MOBO-OSD uses a hypervolume-based (HV-based) AF. As shown in Figures 2, 3, 8 and 9, we have already empirically demonstrated that MOBO-OSB outperforms other state-of-the-art HV-based baselines such as qEHVI and DGEMO. Nonetheless, we appreciate the reviewer’s suggestion and would like to note that we have now evaluated JES (the most recent IT-based AF) on all nine benchmark problems under sequential settings (batch size 1). The results are summarized in the following table, which reports the mean and standard error of the log HV difference between JES and MOBO-OSD at the end of the optimization run (200 iterations). Overall, *MOBO-OSD significantly outperforms JES on all nine problems*.\n\n| | MOBO-OSD            | JES                 |\n|:----------------|:--------------------|:--------------------|\n| DTLZ2-M2        | 2.34e-03 ± 4.42e-05 | 3.32e-02 ± 6.46e-04 |\n| DTLZ2-M3        | 3.78e-02 ± 2.33e-04 | 1.31e-01 ± 1.70e-03 |\n| DTLZ2-M4        | 5.45e-02 ± 8.00e-04 | 2.14e-01 ± 3.36e-03 |\n| VLMOP2          | 2.39e-03 ± 3.59e-05 | 3.11e-02 ± 8.67e-04 |\n| ZDT1            | 4.03e-03 ± 5.55e-04 | 3.38e-02 ± 2.41e-03 |\n| Speed Reducer   | 1.24e+06 ± 7.70e+04 | 8.85e+06 ± 5.75e+05 |\n| Car Side Design | 1.10e+01 ± 2.62e-01 | 4.44e+01 ± 5.89e-01 |\n| Marine Design   | 1.19e+11 ± 3.42e+09 | 5.80e+11 ± 2.36e+10 |\n| Water Planning  | 3.24e+24 ± 5.82e+23 | 1.49e+25 ± 8.10e+23 (*) |\n\nAdditionally, as the reviewer noted, JES suffers from high computational cost and is unable to complete the Water Planning problem ($M=6$). The asterisk in the table denotes results based on iteration data available before termination due to the time limit. Specifically, JES exceeds the time budget of 12 hours per run, which is the same time budget applied to all other methods. We will include these new results (plots across all iterations) in the revised version of the paper.\n\n> The greedy selection for the batch of points to evaluate is not the best option. Although the Hypervolume Subset Selection Problem is #P-hard, there are better alternatives (Guerreiro et al. (2021)).\n\nWe would like to argue that the alternative Hypervolume Subset Selection Problem (HSSP) algorithms mentioned in Guerreiro et al. (2021) *may not be directly suitable* for our batch selection strategy. As the reviewer did not specify a particular HSSP method, we are unsure which algorithm should be the main focus in our response. To the best of our knowledge, these HSSP algorithms *do* *not guarantee satisfaction of a key criterion* in our design: the selected batch points must come from different exploration spaces $\\mathcal T$ (Section 4.4, line 266). This criterion is essential for promoting diversity in the estimated Pareto front. Additionally, we find a practical concern regarding the *computational cost* of these HSSP algorithms. Specifically, as discussed in Section 6 of Guerreiro et al. (2021), for problems with more than 3 objectives, selecting the optimal subset generally requires evaluating all $N=\\binom{b}{N_c}$ combinations of $b$ points among the $N_c$ candidate solutions. In our experimental setup, where $N_c=2000$, $N$ increases exponentially with the batch size (up to $b=10$), making HSSP computationally expensive for large batches. Nonetheless, we appreciate the reviewer’s suggestion and agree that, with further comprehensive investigation, HSSP methods may be adapted to align with the specific constraints of our approach."}, {"review_id": "gdt5ntVd5R", "reviewer": "Reviewer_RbX4", "summary": "The authors propose “Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions” (MOBO-OSD), a novel Bayesian optimization method for expensive, multi-objective black-box optimization problems. MOBO-OSD solves multiple constrained optimization problems along orthogonal search directions (OSDs) in order to obtain a final diverse set of Pareto optimal solutions. These multiple constrained problems are called MOBO-OSD subproblems. Three key strategies are used by MOBO-OSD to obtain good performance. First, to maintain a broad search region, MOBO-OSD approximates the Convex Hull of Individual Objective Minima (CHIM) using the best (ideal) and worst (nadir) observed points. Second, MOBO-OSD generates a well-distributed set of OSDs using the Riesz s-Energy method. Third, to further enhance density and diversity of solutions without requiring an excessive number of subproblems, MOBO-OSD uses Pareto Front Estimation (PFE) for local exploration around candidate solutions. The authors also extend MOBO-OSD to the batch optimization setting and use the Kriging Believer method of considering distinct exploration regions in order to enhance diversity in the selected batch. The authors provide extensive experiments on various synthetic and real-world benchmark tasks with between 2 and 6 objectives. Results demonstrate MOBO-OSD consistently outperforms SOTA baseline methods in both the sequential and batch optimization settings.", "questions": "Question 1: \nMOBO-OSD approximates the CHIM with an ideal point and a nadir point to define boundries. Can the authors elaborate on the choice of using the nadir point (the worst observed value for each objective)? How does this specific choice encourage a \"broader search region\"? \n\n\nQuestion 2: \nThe authors provide an ablation study showing that MOBO-OSD is robust to the number of OSDs, largely due to the PFE component. What are the practical trade-offs a practitioner should consider when applying MOBO-OSD in real-world problem settings. In particular, when deciding between using PFE with fewer OSDs, versus using more OSDs possibly without PFE, how might this impact computational cost and diversity/performance of the final solutions obtained? \n\n\nQuestion 3: \nThe authors cite their focus on settings with noiseless observations as a limitation of MOBO-OSD. What do you foresee as the primary challenges in adapting the proposed MOBO-OSD to handle noisy observations effectively?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: \nThe work is high quality. Figures and text are well polished. Ablation studies are extensive and provide useful insight. \n\nClarity: \nThe work is very well written and the author's method and motivation are clearly explained. \n\nSignificance: \nThis work is significant largely due to the strength of the empirical results. The proposed MOBO-OSD clearly outperforms SOTA baselines in the literature across tasks. This is further strengthened by the inclusion of the 4 real-world benchmark problems. Since multi-objective black-box optimization is particularly relevant and applicable to many areas of the broader research community, this work will have significant impact. The primary weakness of the method is that it cannot be readily applied to settings with noisy observations and this is the case in many real-world settings. However, the authors acknowledge this limitation and I think that this is a very minor weakness of the work overall. \n\nOriginality: \nThis work is clearly novel since it prevents a novel strategy to handle the problem of finding a diverse set of pareto-optimal solutions for multi-objective black-box optimization problems.", "was_revised": true, "final_justification": "In their rebuttal, the authors answer all of my questions sufficiently. I maintain that this work should be accepted for all of the reasons stated in my initial review. Additionally, the other reviews and the author's rebuttals to them served to confirm my initial assessment, it seems that all the reviewers are in agreement that this paper should be accepted. I therefore maintain my score of 5.", "rebuttal": "We thank the reviewer for their insightful review. We address the questions and comments as follows:\n\n# Questions\n\n> Question 1: MOBO-OSD approximates the CHIM with an ideal point and a nadir point to define boundaries. Can the authors elaborate on the choice of using the nadir point (the worst observed value for each objective)? How does this specific choice encourage a \"broader search region\"?\n\nAs mentioned in Section 4.1, our proposed approximated Convex Hull of Individual Minima (CHIM) - constructed from the proposed boundary points defined from ideal and nadir points - can generate a larger search region compared to the alternative CHIM constructed from the individual minima of the current observed dataset. Note that we only consider the alternative CHIM as the true CHIM requires the unknown true individual minima. By definition, both individual minima and boundary points contain one component of the ideal point (as illustrated by the green and purple stars in the inset of Section 4.1). Therefore, by setting the remaining components of each boundary point to the worst (largest) value possible in the dataset, i.e., the nadir points, our proposed boundary points, and consequently the approximated CHIM, can define a larger search region than that generated using the individual minima and the alternative CHIM. We will update this explanation in the revised version of the paper.\n\n> Question 2: The authors provide an ablation study showing that MOBO-OSD is robust to the number of OSDs, largely due to the PFE component. What are the practical trade-offs a practitioner should consider when applying MOBO-OSD in real-world problem settings. In particular, when deciding between using PFE with fewer OSDs, versus using more OSDs possibly without PFE, how might this impact computational cost and diversity/performance of the final solutions obtained?\n\nA practical trade-off worth considering is that increasing the number of OSDs results in better performance but also incurs higher computational cost, hence *we recommend using PFE with fewer OSDs* as the default setting for all real-world problems. This is because, as shown in the two ablation studies in Section 5.1.2, incorporating PFE enables MOBO-OSD to maintain strong performance without having to solve an excessive number OSD subproblems. \n\n> Question 3: The authors cite their focus on settings with noiseless observations as a limitation of MOBO-OSD. What do you foresee as the primary challenges in adapting the proposed MOBO-OSD to handle noisy observations effectively?\n\nAs mentioned in in the Limitations section of the Conclusion, the primary challenge lies in the acquisition function (AF). This is because the Hypervolume Improvement AF depends on the posterior mean, which is sensitive to noise. Another challenge lies in the MOBO-OSB subproblems (Eq. (2)), as solving Eq. (2) requires estimating the posterior mean, standard deviation, and their gradients, all of which are sensitive to noise. Additionally, the current integrated PFE might pose another limitation, as the algorithm in [42] was originally designed for true objective function values (not surrogate model estimates). Therefore, applying it directly to noisy surrogate predictions may compromise the quality of the estimated Pareto front."}]

Author rebuttal:

Meta-review: In this paper, the authors propose NBI-MOBO, a multi-objective Bayesian Optimization algorithm inspired by the Normal Boundary Intersection framework. NBI-MOBO decomposes the multi-objective problem into a series of single-objective subproblems, referred to as NBI-MOBO subproblems, each constrained along a predefined search direction in the objective space. The experiments and analysis on various synthetic and real-world benchmark functions with various objective functions, ranging from 2 to 6 objectives, show that NBI-MOBO can outperform sota methods. According to the reviewers, the paper's main strength is its novel adaptation of the NBI method to the MOBO setting. The paper has a strong and comprehensive empirical validation. Moreover, the paper is excellently written. Furthermore, the proposed method outperforms all other methods on all the benchmarks considered. Finally, the work is clearly novel. Thus, I belive this is a nice and interesting paper that will get the attention of the BO community.
Decision: Accept (poster)