Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemble
Authors: Hanyang Wang
Juergen Branke
Matthias Poloczek
Venue: NeurIPS 2025 poster ()
Keywords: Bayesian Optimization
Preference Learning
Multi-objective Optimization
Neural Network
Primary area: optimization

Abstract: Many real-world black-box optimization problems have multiple conflicting objectives. Rather than attempting to approximate the entire set of Pareto-optimal solutions, interactive preference learning, i.e., optimization with a decision maker in the loop, allows to focus the search on the most relevant subset.  However, few previous studies have exploited the fact that  utility functions are usually monotonic. In this paper, we address the Bayesian Optimization with Preference Exploration (BOPE) problem and propose using a neural network ensemble as a utility surrogate model. This approach naturally integrates monotonicity and allows to learn the decision maker's preferences from pairwise comparisons. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches and exhibits robustness to noise in utility evaluations. An ablation study highlights the critical role of monotonicity in enhancing performance.

Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemble
Hanyang Wang
Juergen Branke
University of Warwick Coventry, CV4 7AL, United Kingdom Hanyang Wang@warwick. ac.uk
University of Warwick
Coventry, CV4 7AL, United Kingdom
Juergen. Branke@wbs...... ac. uk
Matthias Poloczek*
Amazon
San Francisco, CA 94105, USA
natpol@amazon.c com
Abstract
Many real-world black-box optimization problems have multiple conflicting ob- jectives. Rather than attempting approximate the entire set of Pareto-optimal solutions, interactive preference learning, i.c., optimization with a decision maker in the loop, allows to focus the search the most relevant subset. However, few previous studies have exploited the fact that utility functions are typically monotonic. In this paper, address the Bayesian Optimization with Preference Exploration (BOPE) problem and propose using a neural network ensemble as utility surrogate model. This approach naturally integrates monotonicity and allows learning the decision maker's preferences from pairwise comparisons. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches and is robust to noise in utility evaluations. An ablation study highlights the critical role of monotonicity in enhancing performance.
Introduction
Global optimization of complex and expensive-to-eevluuae functions is a fundamental challenge in various scientific and engineering fields. Bayesian Optimization (BO) has emerged as a powerful framework for addressing such problems in sample-efficient way. BO has also been extended to multi-objective optimization (MOO), where multiple, often conflicting, objectives need to be optimized simultancously. It has demonstrated its effectiveness in real-world applications such as protein design, where both protein stability and additional desired propertics like surface area need to be considered, see, c.g., Stanton et al. 2022 or Fromer and Colcy [2023].
In MOO, there is usually not a single optimal solution, but range of so-called Pareto optimal or non-dominated solutions with different trade-offs. Several approaches have been developed for finding good representations of Pareto-optimal solutions, with two methods standing out: ParEGO [noolees 2006], which employs random augmented Chebyshev scalarizations for optimization in each iteration, and expected hypervolume maximization (Yang et al., 2019 Daulton et al.) 2020), which directly maximizes the hypervolume of the Pareto front. Although both methods can generate a Pareto front (a set of Pareto optimal trade-off solutions), eventually one is usually interested in a single solution, and while it may be convenient for a decision maker (DM) to pick solution from the Pareto front, it means that lot of computational effort spent searching for solutions that are eventually discarded.
'This research does not relate to Matthias' work at Amazon.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To make optimization more efficient, interactive multi-objective optimization algorithms have been proposed that repeatedly elicit preference information from the DM, and then guide the search towards the most preferred solution. Combining expensive MOO and interactive preference learning. Lin jet al. [022] proposed problem setting called Bayesian Optimization with Preference Exploration (BOPE). In BOPE, the objective values are from an expensive black-box multi-output function fiue called an output function, while the DM preferences are determined by an unknown utility function Srme. We observe solution's 's objective function values by evaluating the expensive black box function Sime. The DM preference is usually elicited through pairwise comparisons between alternatives, as this is known to have low cognitive burden (Guo and Sanner, 2010J. The ultimate goal of BOPE is to efficiently identify the optimal solution x² that maximizes the DM's s utility over the outputs, i.e. x E argmax e Xtue (fuue(z)) where x is the decision variable and X' IR is the design space. In the original BOPE paper [Lin et al. 2022]. a Gaussian process (GP) is utilized to model the utility function given pairwise preference information.
However, a fundamental property of utility functions in conventional economic theory is monotonicity as increasing objective function values should not decrease the utility (Mas-Colel et al. 1995). Existing approaches usually do not account for monotonicity, potentially hindering the performance BO. To address this limitation, in this paper, we introduce the Monotonic Neural Network Ensemble (MoNNE), a neural network-based approach to modelling utility functions in BOPE. The proposed approach offers several advantages: it naturally incorporates monotonicity constraints through positive weight transformations, efficiently handles pairwise comparison training data using Hinge loss, and quantifies uncertainty through ensemble methods. We also adjust the acquisition function Expected Utility of the Best Option (RUBO) [Lin et al., 2022] that chooses the pairwise comparison for MoNNE models make it more robust under noise. The experimental results demonstrate that the proposed combination of techniques effectively learns utility functions in BOPE, significantly enhancing performance in multi-objective settings that incorporate DM preferences.
This work makes two key contributions. First, we introduce a model that accounts for monotonicity of utility functions, which achieves better performance over SOTA algorithms. Second, we provide the first comprehensive investigation of neural networks with uncertainty estimation in pairwise comparison settings. Although neural networks have been used in preference learning with pairwise comparison data, uncertainty modelling in this context remains understudied. We compare the commonly used techniques such Bayesian Neural Networks (BNN) using Hamiltonian Monte Carlo, BNN by Variational Inference, and ensembles. Our experiments reveal that although Bayesian Neural Networks using Hamiltonian Monte Carlo perform well with standard (X, Y) observations, they face significant challenges in pairwise comparisons. As a result, we recommend and validate a neural network ensemble approach better alternative preference learning.
The structure of this paper is as follows. Section surveys the literature on MOO with interactive preference learning. Section 3 introduces the BOPE problem setting, Section 4 presents the neural- network-basse modelling approach for the DM's utility as well as the adjusted acquisition function. Section 5 reports on the experimental results, demonstrating the empirical advantage of using MoNNE as the utility surrogate model in BOPE. The paper concludes with a summary and suggests avenues for future work. The code is available at
2 Literature Review
This paper focuses on Bayesian Optimization with Preference Exploration (BOPE), which falls within the broader domain of expensive MOO with interactive preference learning. One common assumption in this domain is that the DM's preference can be modelled by an underlying utility function. Typically, this function unknown, thus a crucial step is to acquire knowledge of this utility function through pairwise comparisons Fürnkranz and Hüllermeier) 2010). Astudillo and Frazier 2020) assume that the utility function family is known and only its parameters are unknown. They utilize Bayesian estimation to learn the parameters and subsequently determine the next solution x to be evaluated, based on an acquisition function called Expected Improvement under Utility Uncertainty (EIUU). Ungredda and Branke 2023) also assume a known utility family but unknown parameters. However, the assumption of knowing the utility function family might be too strong in real life. In the work of Lin et al. [2022]. it is assumed that the utility function is entirely unknown, and GP is employed to model it [Chu and Ghahramani] 2005). They also extend ETUU to a batched setting, proposing acquisition function called qNEIUU, and propose acquisition function called

---

Expected Utility of the Best Option (EUBO) to select the pair of alternatives to be shown to the DM. Besides GPs, there are other surrogate models. For example, [Sinha et al. [2010] use a generalized polynomial value function and point estimation for the model parameters. However, both studies ignore the monotonicity of the utility: increasing values of objective functions should not decrease the DM's utility. To fill this gap. CChin et al. (2018) consider GPs with monotonicity in preference learning. They combined linear model and a GP as the surrogate model. However, it requires known kernel hyperparameters instead of learning them. There is another possible model called constrained GP (cGP) by Lopez-Lopera et al. [2022]. cGP adapts finite-dimensional GPs (a piece-wise linear approximation of GPs), and then trains the finite- dineessiinnii GPs with constraints on the order of knot values. Barnett et al. 2024] propose a monotonic GP based on cGP. The main difference is that Barnett et al.) [024] ' method does not treat monotomicity
constraint the order; instead, they prescribe a monotonic transformation on the knot value. Although these two approaches explicitly model monotonicity, it is not obvious how to use them given pairwise comparison information. In addition, both methods adopt an additive kernel, limiting expressiveness for multiplicative effects in utility functions, such as the Cobb- Tobb-Douglas utility function. Another approach to incorporate monotonicity is through monotonic parametric utility models such as the Chebyshev scalarization function, demonstrated by [Ozaki al., 2024). However, these methods sacrifice flexibility to achieve monotonicity, and the value of this trade-off is not clear. Concurrent work by HHbber et al. [025] also addresses monotonicity by augmenting the training data with virtual queries in which dominated solutions are labeled as less preferred. However, their approach encourages rather than enforces monotonicity in the model.
A similar problem setting is Preferential Bayesian Optimization (PBO) [González et al. 2017]. PBO aims at optimizing single black-box function f(x) given pairwise comparison observa- tions. González et al. [017]] use a GP model built on the space of pairs of designs X' x X' as the surrogate model and propose two acquisition functions, Copeland Expected Improvement and sampling. Subsequent research has built upon and enhanced the PBO framework. Takeno al. [2023 adapt skewed Gaussian processes [Benavoli et al. 2020] as a surrogate model to improve performance. Furthermore. Siivola et al.] [2021] expand PBO to a batched setting, allowing parallel preference queries and potentially accelerating the optimization process. Besides,/Astudilloo
et al. [2023) extend EUBO to the batched PBO setting. There are other variants of PBO. such as amortized PBO Zhang et al. 2025]. projective PBO [Mikkola et al. 2020). and multi-objective PBO Astudillo et al. 2024]. Recent work by Xu et al. [2024] provides theoretical analysis of PBO, demon- strating that preference-based BO can achieve information-tee -tthorretccc regret bounds. Furthermore, Arun Kumar et al.) [2024) show that PBO can be used to accelerate standard BO by incorporating expert knowledge about abstract, unmeasurable properties. Those later variants no longer build GP models over the space of pairs of designs X' x X but directly in the (single) design space X. It should be noted that while PBO and BOPE share conceptual similarities, they differ fundamentally in the type of comparisons being made- comparison of designs based on decision variables in case of PBO, and based on objective function values in case of BOPE. though PBO can be applied in the BOPE setting if one assumes that the DM can indeed compare sets of decision variables (which may not be true - a customer may be able to sensibly compare two cars based on their observed features such speed, power, fuel consumption, but not based on their abstract design variables). However, given that both of them use pairwise comparisons, PBO [González et al.) 2017 Astudillo et al. 2023 with modification will serve benchmark algorithm in our experimental evaluations.
In this paper, we use neural networks as the utility surrogate model due to their flexibility and ability to handle complex, high-dimensional spaces (Snoek et al. 2015 Springenberg et al.) 2016 W White ct al. 2021 Li et al., 2024]. When the number of parameters is large, neural networks can capture intricate patterns in the objective function landscape without making strong assumptions about its underlying structure. In the BO framework, a surrogate model needs to provide both predicted mean and uncertainty estimates. Neural networks offer various approaches quantifying uncertainty, each with its own strengths and weaknesses. [Snoek et al. (2015) introduce the use of Bayesian linear layers top of deep neural networks, allowing efficient uncertainty estimation while maintaining the expressive power of deep architectures. [Springenberg et al. (2016), Kim et al. [2021) and Li al. 20244 propose a fully Bayesian treatment of neural network parameters, enabling a more comprehensive uncertainty quantification at the cost of increased computational complexity. |Witt et al. T02T demonstrate the effectiveness of neural network ensembles in capturing predictive uncertainty for Neural Architecture Search, offering a balance between computational efficiency and robust uncertainty estimates. In this paper, we extend the application of neural network ensembles
3

---

to BOPE, where the training set consists of pairwise comparisons. Additionally, we enforce model monotonicity align with the properties of utility functions.
Problem Setting
Given & different objective functions represented by an expensive black-box multi-output function fune : Rd + F Furthermore, assume that there is decision maker (DM) whose utility of solution depends only on these objective values and follows an unknown utility function Jime R - R. The aim is to find
xx arggmx [Ame (Sime (x)))
where x is the decision variable and X' C Rd is the design space.
The solution process consists of two alternating stages: Experimentation and Preference Exploration. In the t-th iteration:
Experimentation. The experimenter selects an ie for evaluation and observes the values of the objective function fiwe (xg). Given n observations of the objective function values, we denote the set of observations by D, = ((11,1))))............))))>.)))) y1.,.., (Iz, }.
Preference Exploration. The experimenter asks the DM express preference between two outputs, f1,t and 12.t. chosen by the experimenter. The DM response is coded as +1 if yı, is preferred and -1 if 1/22 is preferred. At the 1-th iteration, when m pairwise comparisons have been observed, the pairwise comparisons are denoted by Pe == ((11,1.92.1). (f),,, f2.m)) and R2 = {PI, P2; -... Pm where p; +1 is the preference result.
The workflow is illustrated in Figure T and comparison pairs.
It supports batch acquisition by selecting multiple solutions
Inilialiee
Dijn
Preference Exploration
No
Decision Maker
Yiza
Stapping Conditionn
Esppeimenntttion
free
Black Funcson
Figure 1: The experiment alternates between the Preference Exploration and the Experimentation stages.
In prior work, Lin al. 2002) assume that during the Preference Exploration stage, experimenters could present DMs with arbitrary output pairs, including those that were unobserved or even non- existent. However, presenting such unattainable solutions risks causing disappointment when DMs later discover these options are unavailable. Therefore, our work constrains experimenters to only present output pairs that have been previously observed.
In addition, to use more realistic model of DMs, we allow for noise in the decision process. Lin et al. [2022] discuss two types of noise: (1) constant probability of the DM making an error, and (2) noise in the utility values. We argue that the first approach oversimplifies real-worle decision-making as errors often related the utility gap between options. For example, when the utilities of outputs y1 and f2,t are very close, the DM is highly likely to make a mistake, but when the utility gap is large, the DM can easily tell which alternative is preferred. Specifically, we assume the decision result is Pt = 2 1 (ws.)) (tue(g)))+c>0 - 1, where the noise ~ A' o2 2 for some variance We call noise of this type 'utility noise' in this paper. In addition to the two noise types
22 2 discussed above, third type of noise can be introduced by the Bradley-Terry model. Since the noise characteristics of the Bradley-Terry model are similar to that of Gaussian noise, we do not conduct separate experiments with the Bradley-Terry model, but we provide brief introduction to the Bradley- Terry model and compare it with Gaussian noise model in Appendix A.5

---

4 Utility Surrogate Model and Pairwise Comparison Acquisition Function
The problem described above was originally proposed by Lin et al. 2022]. and they handle it in BO framework. This paper follows their framework (albeit with different assumptions about permissible outputs for the Preference Exploration stage) but adjusts the surrogate model and the corresponding acquisition function.
Specifically, their BO framework consists of two surrogate models: GP model f given D, for the objective function, and another GP model g given Pm and Rm as the surrogate model for the DM's utility function.
In the Experimentation stage, using these two surrogate models, they select the next evaluation point based on an acquisition function qNEIUU. For a batch of points F1:4 = (F1,..., Fgg E X', QNEIUU is expressed as: qNEIUU (I1:.) = Em,. {max g (f (z1:4)) - max g (f (Xx)))
where Em | = E| Pm, Rm, Dnl] denotes the conditional expectation given the data from m queries and experiments, teenotes the positive part function, and following their notation, max g (f (z1:g) = max 9 (f (z:)) and max (f (Xa)) = (E)) ()()))
In the Preference Exploration stage, the output pair is chosen according to an acquisition function called the Expected Utility of the Best Option (EUBO) [Lin et al. 2022]. EUBO is defined as EUBO (/2,), IE.. [maxx {g (y1.)), . (12,,)Hl, where the expectation is over the posterior of the utility surrogate model g at the time the query is chosen. Lin et al. [2022] assume that the experimenter can show the DM any pair for their preference decision even if the solutions do not exist.
In this case, it is not straightforward to determine the pair using EUBO because in the acquisition optimization, the search range of the output f1, 32 is unknown. Hence, to find the pair of outputs, EUBO needs to be transformed as EUBO (I1,,, x2,2) = E. [max (g (f(11,)) (f(22)))) to find the pair (2 22,2) ) E X², and the experimenter can use a posterior sample of (f( (x2,)) as the comparison pair (21,,, V2.x).
In this paper, we follow the steps outlined above. However, we propose using a new surrogate model for the utility function instead of using GPs, as GPs cannot easily incorporate monotonicity constraints. In addition, we adjust the EUBO acquisition function for the new model so that the pairwise comparison can be chosen more efficiently.
4.1 Monotonic Neural Network Ensemble as Utility Surrogate Model
An ideal utility surrogate model g for this problem setting should possess the following properties: sufficient flexibility to model realistic utility functions; 2. monotonicity; 3. trained using pairwise comparisons (rankings can also be expressed set of pairwise comparisons); 4. capable of handling uncertainty.
We propose using a neural network as the surrogate model. The advantages of neural networks are manifold. First, neural networks with sufficient capacity can approximate complex utility functions without requiring explicit assumptions about their functional form, providing the desired modelling flexibility. Second, by transforming the weights into positive space, we ensure the model is monotonically increasing, as suggested by Sill [197]. Specifically, we will apply an exponential transformation to the weights, guaranteeing that all weights are positive. The following theorem states that a monotonic neural network is able to model any bounded monotonically increasing function with bounded partial derivatives:
Theorem 1. Sill 1997)) Let m(x) be any continuous, bounded monotonic function with bounded par- tial derivatives, mapping [0, 1] to R. Then there exists a function mner(x) which can be implemented by a monotonic network and is such that, for any e and any x E [0, 1]d, [m(x) mnee (z)) <
Third, using Hinge loss as the loss function enables us to use pairwise comparisons for training. While Hinge loss is typically used in classification problems, it can also be effectively applied to ordinal regression [Rennie and Srebro 2005). Hinge loss is defined as
((P:)
Σ
ma 1 a (g(v1.) 9(v2.)) p.).
where p; = +1 is the corresponding preference of y1.1 vs. V2.1 and a is positive parameter that learned during training. Additionally, while the BOPE problem setting primarily deals with binary
5

---

preferences (=1), we can extend it to handle tied preferences (denoted by p: = 0). The Hinge loss can be modified accordingly: ((P:) E(g (p1, pu max 1 a- (g(()).) (322.)) Pi
1p (g( g (v2.;) Lastly, we employ neural network ensembles [Hansen and Salamon 1990
Lakshminarayanan et al. 2017 to obtain uncertainty estimates that are important for the acquisition
criterion.
Combining neural networks, monotonicity constraints, Hinge loss, and ensemble methods described above, we call the resulting model Monotonic Neural Network Ensemble (MoNNE).
To incorporate uncertainty quantification, we explored several established methods, primarily focusing on Bayesian Neural Networks (BNNs) and ensemble techniques. While BNNs with posterior inference performed via Hamiltonian Monte Carlo (HMC) are generally superior for standard supervised learning with direct input-ouiput observations (X, Y) and performs well in 'vanilla' BO (see Figure 5 in Appendix (A.)), it encounters significant challenges in preference learning. Briefly, the independent prior distribution of parameters hinders training due to scale invariance in preference learning (because the training set only shows relative comparisons of function values rather than absolute values). We refer to Appendix [A.3] for more details. Also, given that Monte in underperforms compared to ensemble methods "Lakshminarayania et al. 2017).
Carlo Dropout focus the investigation on ensemble approaches and BNNs trained with variational inference (VI), particularly recommending ensemble methods for uncertainty quantification in preference learning. This recommendation is supported by several observations: recent studies have shown that ensembles provide more reliable uncertainty estimates than BNNs [Lakshminarayanan
al., 2017 Ovadia
et al. 2019]. In the specific context of BO for Neural Architecture Search, White et al. [202] found ensemble methods to outperform Bayesian estimation approaches, and generally ensembles also show a comparable but marginally lower performance than HMC-trained BNNs in 'vanilla' BO [Li al. 2024). Finally, ensembles offer practical advantages through their inherent parallelizability. substantially reducing computational overhead. and as we will demonstrate later, naturally can be combined with normalization handle scale invariance in preference learning.
In a neural network ensemble, Me. (M₃ > 1) base neural networks, denoted as g¹ (j= Me). simultaneously trained with the same training set but from different initial parameters. The predicted mean and variance are derived by aggregating outputs from these Me neural networks. It should be noticed that the output scales of the Me models might be different due to scale invariance. To handle this issue, we normalize neural networks to ensure uniform scaling. Specifically, the output g¹ (z) scaled as g' ( where Paaxx represents the largest estimated utility value of the compared outputs: gimn max (m ( PiiiiPm..iEE BI ( ) (g' (v)} and の¹ denotes the
normalized deviation of compared outputs: 0¹ Eve (22,,, Pm my (V1,) (9) (y) (y)= iPal
where je denotes the average predicted utility of model وو over all outputs in Pm.
Figure shows two surrogate models trained with the same data: a GP and our MONNE (detailed descriptions of these models are provided in Section 5). The left-hand side of Figure 2 displays two distinct 2D utility functions. The first utility (top) is a linear utility function defined as 9une (31, y2) = yi +15g2 and the second (bottom) is more complicated, defined as Ginee (y1, y/2 vyi 0.9sin
In(y2 e ) + /U2- Both functions increase monotonically in the range [0, 10)2. For clearer visualization, we present slices of these functions. The --axis represents either y (value of the first dimension) or 1/2 (value of the second dimension), indicated in the legend. For the 'y1 + 3/2 10' slice, the x-axis shows y/1 values while y/2 = 10 y1- For 'y2 = 2.5', the z-axis represents y1 values with y2 fixed at 2.5. Similarly, for 1/1 2.5', the r-axis shows y/2 values with y/1 fixed at 2.5. Using 15 randomly chosen pairs from y1 + 3/2 = 10, we constructed the GP and MoNNE models. The middle plot illustrates the GP surrogate model, while the right plot depicts the MoNNE surrogate model. As evident in Figure 2, the MoNNE model more accurately captures the underlying utility function with only 15 output pairs. Furthermore, by incorporating monotonicity constraints, MoNNE maintains a strictly increasing relationship for legends '/1 = 2.5' and 'y2 2.5', whereas the GP model exhibits non-monotonic behavior.
Additional visualizations of GPs and MoNNEs with varying numbers of pairwise comparisons are provided in Figure 10 in Appendix A.4

---

Linear Utiity Function
GP surrogate model
McNIEE surrogate model
Uiilty
Y, 1/
(b)
V4 22
V1 y,
surrogate
surrogate
Ubilty
122
15 1--1-110
2ym23
-15
23
23
130
23
(d)
or
(6)
or
11
or
In each row, the left plot depicts the utility function slices; the middle is the GP surrogate model and the right one is the MoNNE model. While the scale varies significantly across models, this does affect our analysis, as care about relative rankings rather than absolute values.
Figure 2:
4.2
Acquisition Functions
4.2.1
Experimentation
After building the two surrogate models f and g. we adopt QNEIUU (Astudillo and Frazier 2020
et al. 2022 to choose next solution be evaluated for the Experimentation stage. Following
Lin et al. 2022 we calculate the qNEIUU by Monte Carlo simulation:
M
N
max (5* (211a) max (5* (Xn)) }
1 N
QNEIUU (II
M
==
where Fing E 2 (in this paper, g set to be 1), X,, is the tensor of evaluated solutions, + +denotes the positive part function, M is the number of samples from the model N is the posterior sample number of f. g'(()) the j-th sample from g. and f*((-) is the k-th posterior sample from f. When the utility surrogate model g is MoNNE, we set M Me-
4.2.2 Preference Exploration
For the Preference Exploration stage, EUBO can be used pick output pairs. As mentioned, in this paper, the experimenter is only allowed to show observed outputs. In this case, EUBO can be adjusted
as EUBO (y1,s, w2,,) = 37 Σe max { (y1,s) 9 (y2.) }. where y1.i, V2, E (yx (204, y:) Dr)
two observed outputs.
In this paper, we propose one modification the EUBO for MoNNEs. Specifically, when calculating max(g(vi.1) .)), (V2.)),, we assume independence between ((v,,)) and g(V2.), instead of treating them as correlated. We term the modified acquisition function Independent EUBO (IEUBO). Specifically, IEUBO is calculated as:
IEUBO (11.t, V2.1) = E[max {g (11.)) (y2,))]l.
where V1.t. V2,t E (2) 3i) D,,} are two observed outputs, and g(v1.)) and ( are two independent Gaussian variables predicted by the surrogate model g. The expectation can be computed analytically: for two independent normal random variables we have m[max(X, Y)] μr ((a) + p2 -a) + azΦ(a), where t is the cumulative distribution function of standard normal distribution, d3 Vo + oh and #1 2008). The derivation details can be found in Appendix A.110 A.10
N(,,, of) and Y N oz).
Nadarajah and Kotz
03
7

---

Our experiments demonstrate that IEUBO generally outperforms EUBO in the presence of utility noise. Detailed experimental results are provided in Appendix ( This performance difference may be attributed to how noise affects DM preferences: under noisy conditions, the base model of MoNNE might learn incorrect correlations between outputs, potentially making the independence assumption of IEUBO more robust than EUBO's consideration of correlations. Hence, we use IEUBO as the acquisition function.
The use of independence has precedent: White et al. [002]] employed independent Thompson Sampling for neural network ensemble surrogate model in the Neural Architecture Search (NAS) problem, where they similarly assumed that posterior distributions are independent.
Note that GP models do not suffer from this unreliable correlation issue. Therefore, when using GP utility surrogate models, we still employ EUBO for selecting output pairs to present to the DM, albeit constrained to the observed outputs.
Algorithm iin Appendix A summarizes the BOPE algorithm using MoNNE as the utility surrogate model and IEUBO to decide the output pairs for comparison.
5 Experimental Results
In this section, we compare the proposed method with benchmark algorithms on six test prob- lems. The performance of different algorithms is measured by simple regret, defined as the gap between the true optimal utility and the best utility over the observations: max {9ruc (ffuc())) (a.)erea
(guue (z)))
We use all four multi-objective problems that were used benchmarks in Lin al. 2022], J. plus two additional ones. We test a wide range of utility functions from Lin et al. [2022], Astudillo and Frazier [2020]. plus the Cobb-Douglas function, which is standard function economics, and Linear Expnentxp a sum of a linear component and an exponential product. Details can be found in Appendix A.3 Following the experiment setting of Lin al. 2022 the initial number of output observations. that is, the initial dataset, is 16 (for d < 5) or 32 (for d 2 5), and the number of initial pairwise comparisons is 10 (for d < 5) or 20 (for d 2 5). In each iteration, we evaluate one solution, observing faue (z) and ask the DM for his preference over one pair of outputs. We compare the proposed method against the following benchmark algorithms: Random, qNPrEEGO (Daulton et al. 2020). qNEHVI [Daulton et al. 2020], PBO AAstudillo et al. 2023 BOPE-Linear (using linear model as the utility surrogate model). BOPE-GP [Lin et al. 2022]. BOPE-MoNNE (Ours), BOPE-BMNN (Ours, using Variational Inference-traine trained Bayesian Monotonic Neural Network). Known Utility (the utility function known, so the problem becomes a Composite BO [Asuudillo and Frazier 2019). For more details about benchmark algorithms, see Appendix A.5
We allow for noise in DM's decision, as discussed in Section 3. By default, noose is set 0.1 (noise is a setting of the test problem, not the algorithm). Although 0.1 may seem small, considering that BOPE-MoNNE can achieve regret less than 0.001 shown in Figure 3 0.1 utility noise can cause enough trouble (Figure 2 Appendix A.9 shows how many preference errors 0.1 utility noise can make). We also conduct tests with o moise and noise 0.5 to evaluate how this noise affects the performance of MoNNEs and GP surrogate models, and the result is shown in Figure 22 in Appendix A.9
The MoNNE implementation employs an ensemble of 8 three-layer neural networks. The three layers have dimensions d x 100, 100 x 10. 10 x 1, where the weights are transformed using an exponential function. This relatively shallow architecture was chosen due to the limited training set size in the BOPE problem setting. For initialization, the network parameters are drawn uniformly random from a range that depends on the number s of inputs to a node. Specifically, weights are sampled from [- (-(1/s) 1/s] and biases from [-1/s, 1/s]. The weight initialization uses a shifted range with reduced lower bound to account for the subsequent exponential transformation that ensures positive weights and thus monotonicity. In terms of activation function, we utilize Swish [Ramachandran et al. 2017]. A study of the influence of activation functions and layer numbers can be found in Appendix A.6 and A.7 We use the Adam optimizer (Ir = 0.01) combined with CosineAnnealingg.. (T T__max= = 1600, eta_ min = 0.0001) for MoNNE training, and the training epoch number is 1600. Also, to save computation time, we finish the training when the Hinge loss value reduces to zero. The acquisition function optimization uses the multi-start L-BFGS method [Balandat et al. 2020 and all
8

---

2011
Lenaton
0.00
14
Random
GPrTGG BOPE MORINE
PBO
DOPE-Linear
BOPS-GP
BOPE-BMMZI
Udiliy Known
Figure 3: Experimental results show that among all BOPE algorithms, BOPE-MoNNE demonstrates superior performance, followed by BOPE-BMNN and BOPE-GP.
hyperparameters are identical (initial sample number 256, restart number 12) for all algorithms. The output GP surrogate model uses Matérn kernel with Automatic Relevance Determination (ARD). All experiments are replicated 20 times and the plots show the mean #1 standard error. To save computational cost, we stop
early if the simple regret drops below 10-5, If this happens, the value for this replication is set to 10-5 for this and all subsequent iterations.
Figure 3 compares the performance of different methods, with subplots titled 'A+B' (A: output function, B: utility function). Excluding Known Utility algorithm, among all the BOPE algorithms, BOPE-MONNE consistently achieves the best performance, followed by BOPE-BMNN and BOPE- GP, aligning with the uncertainty quality of these models reported in Table (Appendix (.5)). We evaluated uncertainty quality using modified cross-entropy loss function: -p=1 P(p = +)+ 11=-11
P(p= -)), where IP(p = 1) and IP(p = -1) represent the model's predicted probabilities for the DM's preferences. This metric takes into account both the accuracy of the model and its uncertainty, and the larger the value, the better. From Table MoNNE exhibits the highest uncertainty quality, with BNN following and GP ranking last. It should be noticed that the poor uncertainty quality of GPs is mainly due to scale invariance, while the GP prediction accuracy is still good, although lower than MoNNE and BNN (see Table 2 in Appendix (..5n. The experiment details about these tables reported in Appendix A.5 In addition, BOPE-Lines works slightly better than BOPE-MoNNE when the utility function is Tinear, which not surprising. However, in the other problems, the MoNNE model consistently outperforms the linear model.
Furthermore, we find that constraining output comparisons to observed values diminishes BOPE-GP's performance (see Figure 19 in Appendix (.8). While removing this constraint improves BOPE-GP's effectiveness, BOPE-GP GP still underperforms compared to BOPE-MoNNE.
To gain more insight into the effectiveness of the proposed method, we performed an ablation study to isolate the impact of two key components of MoNNE: the monotonicity constraint and the ensemble technique.
Figure Tillustrateee the results (20 repetitions) of the ablation experiments across various test problems, consistently demonstrating that the full BOPE-MoNNE model, incorporating both monotonicity constraints and ensemble learning, outperforms its ablated variants and competing methods (without ensemble can be considered as case where we use Monte Carlo simulation with only one sample for acquisition function calculation). The inclusion of monotonicity information significantly enhances the model's performance by leveraging domain knowledge about the underlying preference structure. Similarly, the use of ensemble techniques proves beneficial, likely due to its ability to capture broader range of potential preference patterns and reduced over-fitting. Note that even when the monotonicity constraint is removed, BOPE-MoNNE still outperforms BOPE-GP, indicating that the neural network ensemble provides advantages beyond just its ability to incorporate monotonicity information.

---

30"
30
Queentt
Carcasledpp CanaCoaga
removed)
Imonstonidty removed)
removed)
Figure 4: The ablation study shows that MoNNE performs best, highlighting the importance of all components. Regret values capped at 10-5
Conclusion
This paper introduces BOPE-MONNE (Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemble), a novel approach to multi-ohjective optimization that learns the decision maker's preferences, leveraging monotonicity of the underlying utilities. The experimental results demonstrate the effectiveness of this method across a range of test problems. BOPE-MoNNE consistently outperforms other methods, including BOPE-GP [Lin et al. 2022 and PBO [Astudillo et al,, 2023], across various test problems. A key strength of the proposed approach lies in its ability incorporate monotonicity of utilities through our neural network design, addressing a crucial property of utility functions that is often overlooked in existing approaches.
Looking ahead, there several promising directions for future work. This paper identified challenges faced by HMC-trained BNNs preference learning, particularly due to the scale invariance of preference learning and the independent prior distributions. An important area of exploration is how to effectively utilize HMC to train BNNs given pairwise comparisons, as an alternative to ensemble- based approaches. Additionally, making scenarios, where multiple decision makers are involved. Preference-based multi-objective optimization has wide range of applications, from science engineering business problems that we expect to benefit from the advancements presented in this work.
interested in extending the work to group decision-
Acknowledgments
We sincerely thank the amonymous reviewers for their valuable comments and constructive sug- gestions, which have greatly helped improve the quality of this paper. The first author gratefully acknowledges support by the Engineering and Physical Sciences Research Council through the Mathematics of Systems II Centre for Doctoral Training at the University of Warwick (reference EP/S022244/1).
References
AV Arun Kumar, Alistair Shilton, Sunil Gupta, Santu Rana, Stewart Greenhill, and Svetha Venkatesh. Enhanced Bayesian optimization via preferential modeling of abstract properties. In Joint Euro- pean Conference on Machine Learning and Knowledge Discovery in Databases, pages 234 250. Springer, 2024.
Raul Astudillo and Peter Frazier. Bayesian optimization of composite functions. In International Conference Machine Learning, pages 354-363. PMLR, 2019.
10

---

Raul Astudillo and Peter Frazier. Multi-attre autti- Bayesian optimization with interactive preference learning. In International Conference on Artificial Intelligence and Statistics, pages 4496 4507. PMLR, 2020.
Raul Astudillo, Zhiyuan Jerry Lin, Eytan Bakshy, and Peter Frazier. qEUBO: A decision-theoretic acquisition function for preferential Bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 1093-1114. PMLR, 2023.
Raul Astudillo, Kejun Li, Maegan Tucker, Chu Xin Cheng, Aaron D Ames, and Yisong Yue. Preferential multi-objective Bayesian optimization. arXiv preprint arXiv:2406. 14699, 2024.
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. Advances in Neural Information Processing Systems, 0121524-21533 2020.
Steven D Barnett, Lauren Beesley, Annie Booth, Robert B Gramacy, and Dave Osthus. Monotonic warpings for additive and deep Gaussian processes. arXiv preprint 1rXiv:2408.0/540 2024.
Alessio Benavoli, Dario Azzimonti, and Dario Piga. Skew Gaussian processes for classification. Machine Learning, 109(9):1877-1902, 2020.
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman, Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 2018.
Robert Chin, Chris Manzie, Alex Ira, Dragan Nesic, and Iman Shames. Gaussian processes with monotonicity constraints for preference learning from pairwise comparisons. In IEEE Conference on Decision and Control, pages 1150-112 1155. IEEE, 2018.
Wei Chu and Zoubin Ghahramani. Preference learning with Gaussian processes. In International Conference on Machine Learning, pages 137-144, 2005.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. Advances Neural Information Processing Systems, 33:9851-9866 2020.
Kalyanmoy Deb and Himanshu Jain. An evolutionary many-objective optimization algorithm using reference-poinn point-based nondominated sorting approach, part I: solving problems with box constraints. IEFE Transactions on Evolutionary Computation, 18(4):577-601, 2013.
Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, and Eckart Zitzler. Scalable test problems for evolutionary multiobjective optimization, In Evolutionary Multiobjectiive Optimization: Theoretical Advances and Applications, pages 105-145. Springer, 2005.
Jenna C Fromer and Connor W Coley. Computer-aided multi-objective optiniiztion in small molecule discovery. Patterns, 4(2), 2023.
Johannes Fürnkranz and Eyke Hüllermeier. Preference learning and ranking by pairwise comparison. In Preference learning, pages 65-82. Springer, 2010.
Javier González, Zhenwen Dai, Andreas Damianou, and Neil D Lawrence. Preferential Bayesian optimization. In International Conference on Machine Learning, pages 1282-1291. PMLR, 2017.
Shengbo Guo and Scott Sanner. Real-timo multiattribute Bayesian preference elicitation with pairwise comparison queries. International Conference on Artificial Intelligence and Statistics, pages 289-296, 2010.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEFE Transactions on Pattern Analysis and Machine Intelligence, 12(10):933-0011, 1990.
Matthew Hoffman, Andrew Gelman, et al. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1):1593-1633, 2014.

---

Felix Huber, Sebastian Rojas Gonzalez, and Raul Astudillo. Bayesian Preference Elicitation for Decision Support in Multi-Objective Optimization. Journal of Multi-Crrieria Decision Analysis, 32(3):e70019, 2025.
Samuel Kim, Peter Y Lu, Charlotte Loh, Jamie Smith, Jasper Snoek, and Marin Soljačić. Deep learning for Bayesian optimization of scientific problems with high-dimensional structure. arXiv preprint arXiv:2 104. 1667, 2021.
Joshua Knowles. ParBGO: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1): 50 50-66, 2006.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Sungyoon Lee, Hoki Kim, and Jaewook Lee. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2645-2611, 2022.
Yucen Lily Li, Tim GJ Rudner, and Andrew Gordon Wilson. A Study of Bayesian Neural Network Surrogates for Bayesian Optimization. In International Conference Learning Representations. 2024.
Xingtao Liao, Qing Li, Xujing Yang, Weigang Zhang, and Wei Li. Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Struciural and Multidisciplinary Optimization, 35:561-569, 2008.
Zhiyuan Jerry Lin, Raul Astudillo, Peter Frazier, and Eytan Bakshy. Preference exploration for efficient Bayesian optimization with multiple outcomes. In International Conference Artificial Intelligence and Statistics, pages 4235 0235-4258. PMLR, 2022.
Andrés López-Lopera, François Bachoc, and Olivier Roustant. High-dimensional additive Gaussian processes under monotonicity constraints. Advances in Neural Information Processing Systems, 35:8041-8053, 2022.
Andreu Mas-Colell, Michael Dennis Whinston, Jerry R Green, et al. Microeconomic theory, volume 1. Oxford university press New York, 1995.
Petrus Mikkola, Milica Todorović, Jari Järvi, Patrick Rinke, and Samuel Kaski. Projective preferential Bayesian optimization. In International Conference on Machine Learning, pages 6884-6892 PMLR, 2020.
Saralees Nadarajah and Samuel Kotz. Exact distribution of the max/min of two Gaussian random variables. IEFF Transactions on Very Large Scale Integration (VLSI) Systems, 16(2):210-212, 2008.
Chang Yong Oh, Efstratios Gavves, and Max Welling. BOCK: Bayesian optimization with cylindrical kernels. In International Conference Machine Learning, pages -6687 PMLR, 2018.
Andrzej Osyezka and Sourav Kundu. A new method to solve generalized multicriteria optimization problems using the simple genetic algorithm. Structural Optimization, 10:94-99, 1995.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Ryota Ozaki, Kazuki Ishikawa, Youhei Kanzaki, Shion Takeno, Ichiro Takeuchi, and Masayuki Karasuyama. Objti-Objective Bayesian Optimization with Active Preference Learning. In AAAI Conference on Artificial Intelligence, volume 38, pages 11490-14499 2024.
Du Phan, Neeraj Pradhan, and Martin Jankowiak. Composable effects for flexible and accelerated probabilistic programming in NumPyro. arXiv preprint arXiv: 2019.
12

---

Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Jason DM Rennie and Nathan Srebro. Loss functions for preference levels: Regression with discrete ordered labels. In IJCAI Mullidisciplinary Workshop on Advances in Preference Handling, volume 1. AAAI Press, Menlo Park, CA, 2005.
Eero Siivola, Akash Kumar Dhaka, Michael Riis Andersen, Javier González, Pablo Gareía Moreno, and Aki Vehtari. Preferential batch Bayesian optimization. In IEEE International Workshop Machine Learning for Signal Processing, pages 1-6. IEEE, 2021.
Joseph Sill. Monotonic networks. Advances in Neural Information Processing Systems, 10, 1997.
Ankur Sinha, Kalyanmoy Deb, Pekka Korhonen, and Jyrki Wallenius. Progressively interactive evolutionary multi-objective optimization method using generalized polynomial value functions. In IEEE Congress on Evolutionary Computation, pages 1-8. IEFE, 2010.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning, pages 2171-2180. PMLR, 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust Bayesian neural networks. Advances in Neural Information Processing Systems, 29, 2016.
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning, pages 20459- 20478. PMLR, 2022.
Shion Takeno, Masahiro Nomura, and Masayuki Karasuyama. Towards practical preferential Bayesian optimization with skew Gaussian processes. In International Conference on Machine Learning, pages 33516-33533, PMLR, 2023.
Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 89:106078 2020.
Juan Ungredda and Juergen Branke. When to elicit preferences in multi-objective Bayesian opti- mization. In Genetic and Evolutionary Computation Conference Companion Proceedings, pages 1997-2003, 2023.
David A Van Veldhuizen and Gary B Lamont. Multiobjective evolutionary algorithm test suites, In ACM Symposium on Applied Computing, pages 351-357, 1999.
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural archi- tectures for neural architecture search. In AAAI Conference on Artificial Intelligence, volume 35, 2021.
pages
Bing Xu. Empirical evaluation of rectified activations in convolutional network. arXiv preprint WrXv::505500083, 2015.
Wenjie Xu, Wenbin Wang, Yuning Jiang, Bratislav Svetozarevic, and Colin N Jones. Principled preferential Bayesian optimization. arXiv preprint 40X5v:240..05367, 2024.
Kaifeng Yang, Michael Emmerich, André Deutz, and Thomas Bäck. Multi-objective Bayesian
global optimization using expected hypervolume improvement gradient. Swarm and Evolutionary Computation, 44:945-95 -566, 2019.
Xinyu Zhang, Daolang Huang, Samuel Kaski, and Julien Martinelli. PABBO: Preferential Amortized
Black- Optimization. arXiv preprint 2rXIv:2503.00924 2025.
Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary Computation, 832):173-195, 2000.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
All contributions and scope are well reflected in the paper's abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Yes, we have acknowledged that the proposed method incurs modest compu- tational overhead compared to baseline approaches.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This work presents an empirical study of neural network applications in Bayesian optimization without introducing novel theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary experimental details have been provided, and the code publicly available ensure reproducibility.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, the code is publicly available and implements standard Python operations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.:bimiiiiiooooooo.iooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All necessary details have been provided and the code is available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are shown to indicate the significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All necessary details about computer resources have been provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: btttap:://euuriie
Answer: [Yes]
Justification:
The paper strictly follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Preference-based multi-objective optimization has wide range of applications, from science over engineering to business problems that we expect to benefit from the advancements presented this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
18

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper does not utilize extemal data. All prior work and Python packages used have been properly credited through citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwpprswithhaeeooo thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: source.
This paper does not introduce new datasets. The code is provided and open-
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
19

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicabll, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM does not impact the core methodology, scientific rigorousness, origi- nality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM attt::://berrrpssccc////ofteeerp for what should or should not be described.
20

---

A Appendix
A.1 Algorithm
Algorithm 1 BOPE-MONNE
Initial observation Do, initial pairwise comparisons Po. and Ro
2: for t = 0, 1...., do
3: Train a GP output model given D.
4: Train a MoNNE utility model g given P, and R.
5:
Experimentation Stage:
6:
Find 1 E arg max (4NEIUU(()(.9g)
7:
Query objective function: Vt+1 = Sime (Tt+1))
8:
Update Di+1= Dr U (2++1, Preference Exploration Stage:
9:
10:
Find (V1,0+1, 1/2c+11) arg f a2 IEUBO V2,119)3.
11: where 31,1, E {Mil (24, 3/) E Dn
12:
Query preference: Pt+1 = 2 (11, (1)) (uu ~~ No0, )
13: Update Pe+1 = Pe U (V1,r+1 (2.4+1) and R+41 = RR {ptt1}
end for
A.2
'Vanilla' BO by Neural Network Ensemble and HMC-trained BNN
In this section, we evaluate two surrogate models in BO: neural network ensemble and Bayesian Neural Networks (BNNs) with posterior inference performed via Hamiltonian Monte Carlo (HMC). All neural networks are of 4 layers with size 128 (as suggested by Li et al. (2024) and Swish as the activation function. For BNN training, utilize the pyro package [Bingham et al. 2018 Phan et al., 2019] with the No-U-Turn Sampler (NUTS) algorithm [Hoffman et al. 2014]. a variant of HMC. The NUTS hyperparameters include 50 warmup iterations and 50 posterior samples.
The experiment involves four common synthetic objective functions, with an initial sample size of 3d (d being the input dimension) and 10 repetitions. We employ the qEI acquisition function from Bo and is set to be 5. Our results demonstrate that -rrCned BNNs exhibit performance comparable ensemble methods, aligning with the findings of et al. 2024
Ackley
Beale 2D
Branin
Hartmann
Meuraal Natworkk Braemble
Neural hatwork Ensemble
HBCC srined ShM
HBC tra nod BHN
HMCC BNA
Sirole regret.
Single regret
simp regret
(a) evaluuaionn
(b) number of evalustions
number of evaluations
[d] number evaluations
Figure 5: HMC-trained BNNs share similar performance as neural network ensemble in 'vanilla BO' setting.
A.3 HMC-trained BNN Given Pairwise Comparisons
In this section, we show empirical results demonstrating that HMC does not work well in training BNNs given pairwise comparisons. We attribute this to the scale invariance of preference learning and the independent prior distribution neural network parameters. Specifically, in preference learning, due to scale invariance, the most important thing to learn is the relationship between parameters rather than their absolute value. However, in Bayesian estimation, the prior distributions of different parameters are independent. Hence, there is big prior conflict which hinders the training.
21

---

To clarify this issue, we first consider the simplest case: linear utility function (1n 21 + 2ng and linear model f(z1, 22) 2 + W222. To be simple, we assume no utility noise, and we are given pairwise comparisons. In this case, we do not care about the absolute values of and u2,, but rather their relationship, i.e., 101
w22
Using Maximum Likelihood Estimation (MLE), given 20 pairs, we can learn w1 R 2- However,
with HMC using standard normal priors for each parameter, we can quickly learn that p(an) R 2 (where uu-) denotes the posterior mean), but their estimated correlation is only 0.75, far from the ideal case of p = 1. This relatively low correlation cannot guarantee wi = when sampling w1 and W2 from their posterior distribution. Increasing the training size to 40 improves the correlation to 0.85, which is higher but still far from p = 1. The relationship between correlation and sample size is illustrated in Figure 6 As shown, the correlation approaches as the training set grows, requiring 100 data points to achieve p 0.95, which is large number of samples.
p(ar))
wa
Correlation of and Wa
170
Correlation
678
100
180
Training Size
Figure 6: A large number of samples are needed to learn the relationship between wh and w2.
The key difference between MLE and Bayesian estimation by HMC lies the independence of the prior: in preference learning, parameters should be highly correlated due to scale invariance, but the prior assumes independence. Consequently, the prior hinders the training, necessitating a much larger training dataset to learn the underlying relationship.
Turning to BNN models, we maintain the setup from Section 5 (but remove monotonicity) with independent standard normal prior distributions for weights and biases. Our goal is to model one- dimensional function fiue(2) (2) v 0.42sin using pairwise comparisons. With 100 randomly sampled comparisons, the model's fitness remains poor (as shown in the Figure 7 a)). Examining the posterior distribution of weights in the last layer reveals a distribution nearly identical to the prior (mean O, variance 1). indicating that the prior distribution may be too restrictive. By increasing the training set to 2000 samples, the model fits the underlying function well (see (b)). At this point, the posterior distributions diverge significantly from the prior. This more complex example reinforces our earlier explanation: scale invariance and the independence prior impede effective training with HMC.
22

---

100 samples
2000 samples
100
predictive mean
predictvve mean true funct
true function
(6)
[b)
Figure 7: (a) With 100 pairwise comparisons, the BNN model exhibits poor fit; (b) With 2000 data points, the fit improves significantly. Note: The large uncertainty arises from scale invariance.
W
Layor3
Layer
Woigh
Layerr Weiggn
Layer Weight
Layer Weighn
Layer 3 Widggne
Layer Weight
50
230
30
52
25
100
50
15
200
15
200
Figure 8: After 100 pairwise comparisons, the posterior distribution of the weights remains nearly indistinguishable from the prior distribution NO, 1).
Layer 3 We ghc.
Layer 3 Woughn
Layor 3 Weight
Layer We
Layer 3 Weight
5.2
2.5
2.5
0.0
We
Welghts
We
Welghrt
0.0
100
130
105
166
Figure 9: With 2000 pairwise comparisons, the posterior distribution deviates significantly from the prior distribution.
Consequently, HMC-trained BNNs are unsuitable for our problem setting, as they require a large number of training data points. This is particularly challenging BO problems, where data limited.
All the experiments above have been executed using the pyro package [Bingham et al., 2018 Phan al. 2019], and we use NUTS [Hoffman et al. 2014] the HMC algorithm.
23

---

A.4 Model Visualization Given Different Numbers of Pairwise Comparisons
This section presents GP and MoNNE models applied the complex function described in Section 3, with varying numbers of pairwise comparisons, as illustrated in Figure T0 The top row displays the GP model, while the bottom row showcases the MoNNE model. For GP models, the 'two-peak" pattern for y/1 + 1/2 = 10 becomes apparent only after 30 samples, whereas for MoNNE, this pattern is evident with just 15 samples.
GP surrogate model samples)
surrogate model samples)
surrogate model (45 samples)
10
y: 100
Y1 =0
2.5
2.5
2.5
11
=
y1
Y1 or Y2
V1 or
Y1 of Y2
MoNNE surrogate model (15 samples)
MoNNE surrogate model (30 samples)
MoNNE surrogate model (45 samples)
-15
-
21 y
12.8
2.5
2.5
2.5
Y1 or 92
Y1 or 9/2
Y1 or y2
Figure 10: GPs vs MoNNEs given different number of pairwise comparisons
A.5
Experiment Setting
A.5.1
Noise Model
In this paper, we consider noise in utility values, which can lead to errors in the DM's choices. Another type of noise be introduced through the Bradley-Terry model.
The Bradley-Terry preference model defines the probability preferring item over item j. given their utility values U, and Uy, as P(i > -) "5 T where å = U U, denotes the utility gap. A fundamental limitation of the standard Bradley-Terry model is its sensitivity to utility scale. For example, if a utility function ranges over [0. 1] and we scale all utility values by 10 to obtain even though the preference ordering remains the same. To address this issue, a more flexible formulation introduces scale parameter (3 yielding P(i >
function ranging over [0, 10). the predicted probabilities change significantly,
Ite Dr- With the additive Gaussian
noise used our experiments, i.e., ~ (0, o the probability of selecting item becomes
P + > 0) +
where o(-) denotes the standard normal cumulative distribution
Dastux
function.
Both the Gradley-Terry and Gaussian noise models exhibit similar behavior: as 5 0, the selection probability approaches 0.5, and as - oo. it approaches 1. Moreover, for a given noise level Onoise: one can choose a corresponding such that the two models closely align. For instance, when Onnise 1, setting/ 3 1.8 results in almost identical probability curves (see Figure(T]) Therefore, we expect that our results will not change if our Gaussian-noise based formulation would be replaced by the very similar Bradley-Tere -eery choice model.

---

Bradiey-Terry Model vs. Gaussian Noise Model
Brodley-Tery Terry Model p-1.8
Caussian Noise Model duw
4.-
22.0
Figure 11: Bradley-Terry and Gaussian noise models exhibit similar behaviour.
A.5.2
Test Problems
The test output problems are taken from [Lin al. 2022 Astudillo and Frazier 2020]. combined with different utility functions. An interesting fact is that, in terms of VehicleSafety, OSY, and CarCabDesign, a global optimum x located at the corner of the search range. Hence, although these three problems are of higher dimension, it is relatively easier to find the optimal solution as BO tends to explore more on the boundary [Oh et al. 2018)
DTL.Z2 (3D) Linear (2D):: The output function DTL.72, introduced by Deb et al.
2005
is a two-objective problem. Its objective functions are defined as f (x)
h(x))
cos(#o T/2) and fiue (x) = (1 + h(x)) sin(xo- T/ T/2). where h(()
i=m ( 0.5)
The utility function is Juue (y) Σi=1 0, Yi. where [3.5, 6.5].
VLMOP3 (2D) Exponential (3D): The output function VLMOP3 Van Veldhuizen and (x + x3)) + sin (5f + 2z), , Sime
Lamont 1999 has three objectives: flue (x) 0.5
(3m) 2-) 11) + 15 and fi (x) マ + 1.1 exp (- (Λî + x²))
27
The utility function is Pune (y) > iml 1-exp(-8y:
where 0 0.35.
ZDTI (10D) + LinearExponential (2D): The output function ZDTI, introduced by Zitzler
al. 2000), two-objective benchmark problem. Its objective functions defined as
[ To and / (x) -(x) (1 / /////(())))))))) where g(x) = 1 + - d-1 Tj.
The utility function is (y) = 5(yo + 5(y1 + + ex..2ex00075505))0 2)).
exp(1.25(y1 -
OSY (6D) + Quadratic (8D): The output function, the OSY problem, originally introduced as a constrained optimization task by Osyezka and Kundu [1995), was reformulated as multi-objective problem by redefining its constraints objectives.
The utility function is Juue (y) Σ Ei 87-
VehicleSafety (5D) + Kumaraswamy CDF (3D): For details of the output function fime definition, we refer readers to Liao et al. 20088, Tanabe and Ishibuchi (2020). In addition,
each dimension of fime is normalized to be within the range [0. 1].
The utility function is Jaue (y) I
(3:). where F.(-) is the CDF of Kumaraswamy distribution with shape parameters a = [0.5, 1.5) and b [1.0, 2.0, 3.0)
CarCabDesign (7D) + CobbDouglas (9D): Details can be found in Deb and Jain (0013) Tanabe and Ishibuchi 22020]. To ensure deterministic ground-truth outcome function, the stochastic components of the original problem were excluded. Each dimension of fune was also normalized the range [0, during the simulation.
The utility function is Jrme (y) = C. I + 0.), where C = 50 and [0.75, 0.5, 0.5, 0.3, 0.7, 0.5, 0.65, 0.8, 0.55].
25

---

A.5.3 Benchmark Algorithms
We compare the proposed method against the following benchmark algorithms:
Random: A baseline approach that randomly selects the next solution with SobolEngine.
qNarEGO [Daulton et al. 2020]: A multi-objeectvve optimization algorithm that operates without learning DM preferences. Multi-objective optimization algorithms can be considered as strategies where the experimenter tries to find promising solutions on the Pareto front and asks the DM for their preference at the end.
qNEHVI [Daulton et al. 2020]: Another multi cojett-objective optimization algorithm that, like qNParEGO, does not incorporate DM preference learning.
PBO (Lin al. 2022 Astudillo et al., 2023]: An approach that learns DM pref- erences directly in the solution space X. It constructs a model g(x) to predict the DM preference for solution x. The acquisition functions are defined as QEEUU (Z1:g) = M Ej=1 {max g (I1:g) mmxxa (Xn)))* and EUBO (Z1.1 22,1)
1 Eÿe max (x),s)) o (12.)). It is important note that we still require an output surrogate model f as DM can only compare the output pair rather than the input pairs. As the true output function is unknown, we use the posterior samples of (f)),,, (x2,))) )) as output pairs. As mentioned in the literature review, PBO is different from BOPE as their information sources are different. In the BOPE problem setting, DMs compare solutions based on their objective function values, while in PBO, comparisons are based on their design variables, making PBO not directly applicable. Therefore, in our benchmark PBO algorithm implementation, we construct a surrogate model of the output function and use output samples from pairs selected by PBO to elicit DM preferences in the Preference Exploration stage. the Experimentation stage, we use the Expected Improvement acquisi- tion function to pick solution to be evaluated. Its result is shown in Figure 3 However, this setting may show the DM any arbitrary output pairs instead of only observed ones. Another way to make PBO applicable is to do two experimentation evaluations combined and then one preference exploration. This approach is necessary for PBO because DM cannot directly compare two different inputs; instead, we must first obtain the outputs from both methods and subsequently elicit the DM's preferences between these outputs. We conducted experiments using this more equitable framework (one preference exploration and two experimentation evaluations in every iteration). Results presented in Figure 12 show that the order of performance of PBO, BOPE-GP and BOPE-MONNE remains the same.
BOPF-LLinear: Using linear model of the utility function. Similar to MONNE, constrain the parameters of the linear model to be positive to guarantee utility to be monotonically increasing. We use Laplace approximation as an approximation of the Bayesian estimation. This model is monotonically increasing but lacks the flexibility to represent more complex utility functions.
BOPE-GP (Lin et al., 2022]: A method specifically designed for BOPE problems that models DM preferences using GPs and employs EUBO to select pairwise comparisons from the observed outputs.
BOPE-MoNNE (Ours): Building upon BOPE-GP's framework, our method replaces the GP utility surrogate model with MoNNE and uses IEUBO for pairwise comparison selection, as detailed in Section 4.
BOPE-BMNN (Ours): We implemented a Bayesian Monotonic Neural Network (BMNN) as a benchmark surrogate model evaluate the effectiveness of different uncertainty sources. As discussed, HMC faces some challenges given the pairwise comparison training set, so VI is utilized to train the model. The implementation is based on the torchbnn package [Lee] let al., 2022]. We define the loss function as a weighted sum of the expected Hinge loss and the KL-ddivegeecee, Eo~g(C(P))] + X DKL ((()))) where 0 represents the neural network parameters, i.e. weights and biases, ((0) is the posterior distribution of 6, and p(0) is its prior. The weight is set to 10-5; we use such a small positive value to balance the scales of the KI KL-divergence and Hinge loss (see visualization of BMNNs with various À values in Figure [3]. To ensure comparability with BOPE-MoNNE we use IEUBO to select output pairs.
26

---

Utility Known: The utility function is known, so the problem becomes a Composite BO
[Astudillo and Frazier 2019]. This algorithm is considered the potential best benchmark that one can achieve.
OTti2 1800 Liverr 1201
VLMOPS (28) Experentill (30)
PD
evaluadore
evalustions
648 uatuatons
USt (60) Quadroti (BD)
Cartabslasig (70) Cottloug in (96)
SRor mOp 13
0.13
6.2
6.0
23
esslustions
Figure 12: The order of performance of PBO, BOPE-GP and BOPE-MONNE does not change even if we allow two experimental evaluations in every iteration (which more natural for PBO).
DMWN
SNINN
DWVVN
BWNN (3 D)
0.8
0.75
413
0.00
43
0.1
4.13
030
23
22-25
-1
-.-
-15
21-25
02
(b)
Figure 13: Bayesian Monotonic Neural Network training with different À given 15 randomly sampled pairs from the slice 3/ + 3/2 10 from the complicated utility function: trade-off between the Hinge loss and uncertainty. A value of À = 1 x 10-5 achieves good balance between the prior and uncertainty.
Tables J and Tables present the model uncertainty and accuracy evaluations in the following experimental setup: For problems of dimension d < 5, we use 160 samples, while for > 5, we use 320 samples. The initial number of pairwise comparisons is set to 10 for d < 5 and 20 for d 2 5, with a utility noise level of 0.1. After constructing the utility surrogate model, we evaluate its performance by randomly selecting pairs and comparing the model's predictions against the true preferences. This process is repeated 50 times. Table reports the training time for the three utility surrogate models under these experimental conditions, with all computations performed on an Intel Core 17-1355U CPU.
In addition, we visualize an example of the optimization trajectories of NEHHVI, BOPE-GP, and BOPE-MoNNE on the ZDTI test problem over 25 iterations (steps 1-25). As shown in Figure [4 a), qNEHVI selects points that maximize hypervolume, but these chosen points can be distant from optimal solutions. According to Figure 14]b), BOPE-GI GP gradually selects points closer to the Pareto front but remains far from the optimal regions. While allowing BOPE-GP to compare arbitrary output pairs (rather than only observed ones) improves its performance (Figure [4cc), c)), it still underperforms compared to BOPE-MONNE (Figure d)),, which directly converges toward optimal points. We further demonstrate Appendix A.8 that even when BOPF-GP allowed arbitrary output comparisons, it continues to perform worse than BOPE-MoNNE
27

---

Table 1: Assessment of the quality of the uncertainty estimates provided by the models
GP
MoNNE
BMNN (VI)
DTLZ2
0.622(+0.01)
0.924(+0.014)
0.855(#0.019)
VLMOP3
0.625(+0.01)
0.915(+0.022)
0.731(#0.027)
ZDTI
0.674(=0.012) 0.951(+0.025)
0.911(#0.034)
VehicleSafety
0.63(=0.016) 0.863(=0.045)
0.848(+0.04)
OSY
0.64(=0.015) 0.878(#0.022)
0.886(#0.021)
CarCabDesign
0.658(=0.008)
0.951(=0.08)
0.901(#0.02)
Table Surrogate Model Accuracy (accuracy is measured as the percentage of correct predictions)
GP
MoNNE
BMNN (VI)
DTLZ2
0.9
0.92
0.9
VLMOP3
0.86
0.92
0.84
ZDTI
0.92
0.96
0.94
VehicleSafety
0.82
0.86
0.86
OSY
0.82
0.88
0.88
CarCabDesign
0.84
0.96
0.94
Fous Poogen NA
Potet Drogrante (BOTE
Prggeasee (BO7E
180
641
Figure 14: Optimization Trajectories of Different Algorithms
A.6 Activation Function Comparison
The selection of an appropriate activation function is crucial to the performance of the BNNs [Li et al. 2024]. While prior work has compared smooth (Tanh) and non-smooth (ReLU) activation functions in BNNs, no clear consensus emerged regarding their relative effectiveness. In our investigation, we initially considered five activation functions MoNNEs: Tanh, Sigmoid, Swish [Ramachandran
et al. 2017], ReLU, and Leaky ReLU Xu 2015].
During the empirical investigation, observed frequent numerical instabilities when using Tanh and ReLU activation functions, which led us to focus our analysis on three alternatives: Sigmoid, Swish, and Leaky ReLU. These functions provide instructive comparison between smooth (Sigmoid, Swish) and non-smooth (Leaky ReI.U) activation functions. For Leaky ReLU, the negative slope parameter is set to 0.25. Figure 16 presents the performance comparison of MoNNE in these activation
Table 3: Computation Time (seconds)
GP
MoNNE
BMNN (VI)
DTLZ2
0.39(20002)
3.53(+0.91)
61.34(+0.17)
VLMOP3
0.42(+0.02)
4.19(+0.91)
60.66(10.01)
ZDTI
0.40(+0.03)
3.53(+1.09)
63.52(10.11)
VehicleSafety
0.49(
5.58(+1.05)
60.88(+0.02)
OSY
0.49 +0.003)
3.83(+0.8)
123.59(+0.04)
CarCabDesign
0.28 (0.00)
4.92(=0.98)
124.32(#0011)
28

---

functions. While all three activation functions demonstrate comparable performance, Swish exhibits marginally superior results.
Although Leaky ReI.U achieves performance comparable to that of Swish in the previous experiments, it exhibits limitations in modelling complex utility functions. Figure T3] illustratess these limitations by displaying the base models of MoNNE with different activation functions, trained on 30 randomly sampled pairs from the constraint surface y1 + 1/2 10. We examine the same complex function previously analyzed in Figure d) of Section 3, plotting the model values along the constraint surface y11 + 3/2 10. The visualization reveals that Leaky ReLU struggles to capture the intricate patterns of the underlying function.
MoNNE surrogatc model (Leaky RolU)
surrogate gmpic]
surrogace
STANT
Lock reade
Dus TRAM
Figure 15: Given the shallow architecture of the neural network, Leaky ReLU activation functions may not be ideal for modelling complex underlying functions.
VLMOPS (29) Expereetiall
2011 (100) LiteeCapeteetttlliioo (20)
10
Ledyy Rell)
Sigeal
15
Eati
10
vehidesalery (3D) KumerapaamyCUF (30)
OCT Quadratte (BD)
CarCabDerign (20: CobaDoug kel
Ledty Bell)
Saln
0.22
605
Ieeaton
heration
10
Figure 16: While all three activation functions demonstrate comparable performance, Swish exhibits marginally superior results.
A.7 Layer Number Comparison
In this section, we analyze the relationship between network depth and model performance. We evaluate architectures with two layers (d 100, 100 1), three layers (d x 100, 100 10, 10 1), illustratees [ the behavior of MoNNE and four layers (d x 100, 100 x 100, 100 x 10, 10 x 1). Figure
models trained on 30 randomly sampled pairs from the constraint surface 3/1 + 3/2 = 10, using the complicated utility function described in Section 3. Figure 17 reveals that the learned representations maintain consistent patterns across varying network depths, with no clear correlation between the number of layers and the model's mean and variance.
We evaluate three different network depths across six test problems, as shown in Figure 18 While the optimal number of layers varies across problems. we observe that the 3-layer MoNNE consistently achieves intermediate performance. Notably, the parameter count increases substantially with network depth: the 2-layer MoNNE contains hundreds of parameters, the 3-layer variant introduces approx- imately 1,000 additional parameters, and the 4-layer architecture further adds 10,000 parameters. Moreover, the layer MoNNE exhibits significantly longer training times compared to shallower
29

---

architectures. Based on this trade-off between performance and computational efficiency, we adopt the 3-layer MoNNE architecture for all subsequent experiments in this paper.
NcNNE layers!
MoNNE
McNIEE
PROOU
bese
Figure 17: The patterns of MoNNEs with different layer numbers are similar.
(28) Enperental
2013 (100) LinerrCxponertallin 1201
kyyn kya
20
50
teration
verideSalery Kumeramaayyorr (B))
OSY (80) + Guadratic (50)
CarCabDesign (2):
6.2
Smgee IOTTT
Iteraton
50
Figure 18: While the optimal number of layers remains unclear, the 3-layer MoNNE consistently
achieves intermediate performance.
A.8 EUBO Experiments
In this section, we compare two versions of the acquisition function EUBO, one that can choose arbitrary outputs for comparison, and one that is restricted to choose among previously observed outputs. Both use GP as utility surrogate model. The number of posterior samples of the GP is set N 32. In addition, the utility noise is set to 0.1.
In Figure 19 EUBO (arbitrary output) works better than EUBO (observed output) given the GP models. However, MoNNE+IEUBO still works the best.
30

---

DTLZ2 (30) Unber 200
VUJOP (20) Exponerdal (D)
Lusting
55
PRC PU00 2300
3331
Rocadonn
Seration
USY (60) Guadroti (5D)
Sire roges 13
0hv0000 IOSERNTTINNC
CNCCI ioserve
10
Parsatio
Keration
Figure 19: Allowing arbitrary outputs EUBO for GPs improves the BOPE-GP performance, yet still remains inferior to BOPE-MoNNE.
In Figure 20 we compare IEUBO and EUBO with and without utility noise and given the MoNNE model. Our experiments demonstrate that IEUBO generally outperforms EUBO in the presence of utility noise. However, when utility noise is absent, EUBO achieves comparable or occasionally superior performance to IEUBO.
OTLE2 150) 6 LINEE (20)
VUMOPS (201 Exponcrtal (20)
2073, 390 LIF LirsorEsponenttl
HAG F3
18-1
Reredon
/WicckGsfeey KumarRARmycCE (10)
OST (40) Qundraic
CarCabüedgn (79) CahaDsugle (90)
HMC
EAD/12
100 se
Reration
Dreation
keradon
Figure 20: IEUBO in most cases outperforms EUBO in the presence of utility noise, but when utility noise is absent, EUBO achieves comparable or occasionally superior performance to IEUBO.
A.9 Influence of Utility Noise
we show the number of preference enror that the DM made during the experiment. Note that the error number for the VehicleSafety problem and the CarCabDesign problem plateaus due to several runs reaching the emror threshold and stopping.
In Figure 21
31

---

20122 Liweaa
2.0
9-
0.0
Und De mennty DNT1 wewoneorggoooo
MUAIE
1299
80%
3.20
Lanaton
Comparison of Preference Errors in BOPE-GP and BOPE-MONNE BOPE-MONNE generally has more errors. Notably, in the VehicleSafety problem and the CarCabDesign problem, the error count for BOPE-MoNNE plateaus due to several runs reaching the error threshold and stopping.
Figure 21:
We conducted experiments with different noise levels (Tnoise = O, noise = 0.1 and Tnoise = 0.5) to investigate the impact of noise on both the GP model and the MoNNE model, with results shown in Figure 22 Our analysis reveals that BOPF-MoNNE consistently outperforms BOPE-GP across all noise levels. This performance difference is particularly noteworthy in the VehicleSafety problem, where noise level moise 0.5 represents significant disruption, given that the output range of this problem is approximately 0.6. Under these conditions, pairwise comparisons become nearly random. While BOPE-GP performance deteriorates substantially in this high-noise scenario, BOPE-MoNNE maintains reasonable effectiveness by preserving the underlying monotonicity constraints of the problem.
DT-22 (SU) Liwar (20)
VLMOPS (28) Experential (30)
2011 (100) Lineor Exwonertiitlnlloooooooooo (ZU)
Oreba
CP vano
GP-C2 hosa
WAKE Fil
KoMeC ho1e
20
Vehidasallery (SD) Kameranaamyya CDC (30)
Quadrosic
CoaaDo-gian
2.12
6:8
lteestion
Rarazion
baration
Figure 22: MoNNE models are robust under utility noise.
A.10 Derivation of Expected Maximum of Two Normal Distributions
This result can also be found in Nadarajah and Kotz 2008
Let N(n, af) and Y ~ N( oz) be independent normal random variables. We derive E|max X, Y)] as follows:
32

---

First, we apply the law of total expectation:
E|max(X, Y) B[ > YJP(X > Y) + EJYYY > XJP(Y= > X).
Define == X Y. Since X and Y are independent normals:
NN111 - 12, oî + o²).
Let 033 Vot and 12 03
The probabilities are:
P(X > Y) P(Z 0) 2(a),
P(Y > X) P(Z < 0) ((-a) =1 =1-((a)
where (.) the standard normal CDF We first need to calculate:
E| > Y E > 0j,
where o() is the standard normal PDF.
(X, 2) follows a bivariate normal:
x
N
(of (M
[11 - p22)
ln
)
of + 03
of
)
of
The correlation coefficient between X and Z is p a a of +
a1
03
For bivariate normal, E = =] is linear in 2:
X 2 = 2| = M1 +
01 (z-11))
03
where μ/= 111 122
Hence,
E|X| >
E - > 0]
Elu + -d 1z)) 0]
Tr + (E|Z 0 - k12)
tl + p 01 (142 ((a) 12))
(a)
((a)
P1 + Lod ((a)
in + of ②(a)
03 ((a)
of d(a) da P(a)'
That is, E[XXX YJ A1
Similarly:
E|yY > x| = + of Φ(-a)
03 (-a)
Substituting back:
E Ejmax (X,Y)) = (m) + of Ma)
$(a) + 1/2 + oz (-a)
Φ(-a) $ $(-a).
:(a))
The closed-fone -fom expression is:
E [[maax (rnnax(X,)) Y)] nm ((x) + (-xx) + 03δ(a))
33


Reviews:
[{"review_id": "mDfEsHAGOt", "reviewer": "Reviewer_3gJh", "summary": "In interactive multi-objective optimization, the utility functions are usually monotonic. The paper considers this fact and introduces a new method BOPE-MoNNE. BOPE-MoNNE uses neural network ensemble model with positive weight constraints to ensure monotonicity while modeling the mean and variance. The model is trained on pairwise preferences queried from the decision maker (DM). Experimental results on several established low-dimensional benchmark problems demonstrate that BOPE-MoNNE outperforms all baselines, and achieves performance close to the ideal case where the true utility is known.", "questions": "Could you provide visualizations the Pareto front of different methods? This could offer additional insight into the performance and trade-offs.", "limitations": "The main limitations are the potential inefficiency due to frequent DM queries and the unclear scalability of the method to higher-dimensional or more challenging scenarios.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The proposed method effectively incorporates the monotonicity property of utility functions, which is often overlooked in previous work.\n- Experimental results show that the proposed approach consistently achieves lower simple regret compared to existing methods.\n- The paper is clearly written and easy to follow. \n\nWeaknesses\n\n- The method requires a preference query from the DM at every iteration, which could become impractical or inefficient for complex or real-world scenarios where querying is expensive.\n- Experiments are limited to a few standard low-dimensional tasks. It is unknown whether the method performs well on more challenging tasks with higher dimensions.\n- Although the authors state that code is available in an anonymous GitHub repository, I was unable to find the link. The lack of accessible code may hinder reproducibility of the results.", "was_revised": true, "final_justification": "Thank you for the response addressing each of the concerns. I have changed the score accordingly.", "rebuttal": "Thank you for your thoughtful review and insightful comments. We provide detailed responses to your questions below and include the additional experimental results according to your suggestion. We will modify the camera-ready paper accordingly if the paper is accepted.\n\n> The method requires a preference query from the DM at every iteration \n\nWhile we alternate between preference elicitation and optimization steps, it is straightforward to change the ratio, e.g. have one preference query every two optimization steps. In our paper, we already test this case (Figure 10) and we find that BOPE-MoNNE still works best.\n\n> whether the method performs well on more challenging tasks with higher dimensions\n\nWe evaluated our approach on five benchmark problems with dimensions ranging from 2 to 7 and number of objectives ranging from 2 to 9. This experimental setup is consistent with the scale and scope commonly used in the existing literature. In particular, we use all four multi-objective problems that were used in the benchmark paper [1], plus an additional one. Regarding the utility functions used, we already test a wide range of utility functions from [1] and [2], plus the Cobb-Douglas function which is a standard function in economics. We have not observed a significant influence of the utility function on results, and wouldn’t expect so since ANNs are at least in principle capable of representing any utility function.\n\nTo enhance the comprehensiveness of our evaluation, we have added an additional test case: the well-known multi-objective benchmark ZDT1 with 10 dimensions and 2 objectives, using a linear-exponential utility function defined as $U(y_0,y_1) = 5(y_0+2) + 5(y_1+2) + 2\\exp(0.75(y_0+2))\\exp(1.25(y_1+2))$, i.e., a sum of a linear component and an exponential product. The experimental result (20 repetitions, 50 iterations and all the experiment setting follows the paper) is shown in the table below:\n\n| Method        | 25th                 | 50th                 | 75th                 | 100th                |\n|---------------|----------------------|----------------------|----------------------|----------------------|\n| Random        | 57.58 (0.6948)       | 57.01 (0.6948)       | 56.75 (0.5361)       | 56.75 (0.5361)       |\n| qNEHVI        | 35.15 (1.514)        | 11.77 (1.913)        | 6.319 (1.136)        | 4.816 (1.047)        |\n| qParEGO       | 51.72 (0.7307)       | 48.53 (1.331)        | 47.80 (1.322)        | 46.44 (1.273)        |\n| PBO           | 52.62 (1.496)        | 48.01 (1.886)        | 38.66 (2.450)        | 24.75 (3.359)        |\n| BOPE-Linear   | 13.78 (1.283)        | 0.8842 (0.1596)      | 0.8842 (0.1596)      | 0.8842 (0.1596)      |\n| BOPE-GP       | 32.43 (2.314)        | 3.403 (0.8208)       | 2.327 (0.6852)       | 1.674 (0.6571)       |\n| BOPE-BMNN     | 14.08 (1.799)        | 0.3960 (0.0804)      | 0.2231 (0.0439)      | 0.2231 (0.0439)      |\n| BOPE-MoNNE    | 9.588 (0.8750)       | 0.2127 (0.0899)      | 0.0025 (0.0010)      | 0.0020 (0.0007)      |\n| Utility Known | 8.676 (1.016)        | 0.0558 (0.0183)      | 0.000125 (0.000034)  | 0.000059 (0.000018)  |\n\n\nFrom the table, we observe that BOPE-MoNNE achieves the best performance, followed by BOPE-BMNN and BOPE-Linear. The strong performance of the linear model can be attributed to the structure of the utility function, where the first component is linear. While BOPE-GP performs worse than these three methods, it still significantly outperforms the remaining baselines. PBO exhibits weaker performance, which can be explained by the challenge of navigating a 10d space using only pairwise comparisons. \n\nWe also conducted an ablation study as shown below (same experimental setting):\n\n| Method                          | 25th                   | 50th                   | 75th                   | 100th                  |\n|----------------------------------|-------------------------|-------------------------|-------------------------|-------------------------|\n| BOPE-GP                          | 32.43 (2.314)           | 3.403 (0.8208)          | 2.327 (0.6852)          | 1.674 (0.6571)          |\n| BOPE-MoNNE (monotonicity removed) | 17.98 (1.787)           | 1.275 (0.3492)          | 0.2872 (0.1461)         | 0.09786 (0.08952)       |\n| BOPE-MoNNE (ensemble removed)     | 9.195 (1.257)           | 0.4128 (0.1342)         | 0.2087 (0.1192)         | 0.1204 (0.0893)         |\n| BOPE-MoNNE                        | 9.588 (0.8749)          | 0.2127 (0.0899)         | 0.0025 (0.0010)         | 0.0020 (0.0007)         |\n\n\n\nThe results demonstrate that  incorporating monotonicity information improves the performance significantly, achieving better function values at a lower cost. \n\n\n\n[1] Lin, Z.J., Astudillo, R., Frazier, P. and Bakshy, E., 2022, May. Preference exploration for efficient Bayesian optimization with multiple outcomes. In International Conference on Artificial Intelligence and Statistics (pp. 4235-4258). PMLR.  \n[2] Astudillo, R., & Frazier, P. (2020, June). Multi-attribute Bayesian optimization with interactive preference learning. In International Conference on Artificial Intelligence and Statistics (pp. 4496-4507). PMLR\n\n>  unable to find the code link \n\nThe link is provided on line 71 of our paper and can be accessed by clicking \"the anonymous GitHub repository\". If the link is not functional, perhaps it was removed by the system to ensure anonymity, even though it was an anonymous GitHub repository. NeurIPS rules don’t allow us to provide a link in the rebuttal. \n\n> visualizations the Pareto front of different methods\n\nThank you for your suggestion. NeurIPS rules don’t allow us to put figures into the rebuttal, but we already prepared a revised version of the paper with added visualization. We observed that MOBO methods (ParEGO and EHVI) exhibit more uniform exploration across the objective space, while BOPE methods concentrate their evaluations around high-utility regions. Furthermore, BOPE-MoNNE demonstrates closer convergence to the global optimum compared to BOPE-GP, as MoNNE provides better modeling of the utility function.\n\n> the potential inefficiency due to frequent DM queries and the unclear scalability of the method to higher-dimensional or more challenging scenarios\n\nAs explained above, the paper already demonstrates that BOPE-MoNNE maintains its superiority if the DM is queried less frequently, and also on an additional 10-dimensional problem."}, {"review_id": "zD8zIYh2nW", "reviewer": "Reviewer_jmqt", "summary": "The paper introduces BOPE-MoNNE, a method for Bayesian Optimization with Preference Exploration (BOPE) that learns a decision maker’s preferences from pairwise comparisons. BOPE-MoNNE models the utility function with a monotonic neural network ensemble, which enforces monotonicity through exponential weight transformations and trains on pairwise data. They show neural network ensembles provide better-calibrated uncertainty estimates than Bayesian neural networks, which struggle with scale invariance in pairwise preference learning. The method adapts standard acquisition functions. Across five benchmark problems, BOPE-MoNNE consistently outperforms previous methods, demonstrating that incorporating monotonicity and ensemble-based uncertainty estimation significantly improves performance.", "questions": "How does the method perform over a larger variety of utility functions and benchmarks? \n\nAre there cases where BOPE-MoNNE fails/performs worse than baselines?", "limitations": "The authors could further address the limitations of their work. They mention the additional computational overhead but do not discuss any other possible limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The method shows strong performance compared to baselines on the test functions presented. Additionally, their contributions include adding monotonicity to the utility model, which could better mimic real-world scenarios than other methods that do not enforce this condition. There are extension ablations provided in the appendix detailed network choices and comparisons to other ways of modeling utility.\n\nHowever, the overall evaluation is fairly limited and more robust evaluation on a larger variety of benchmarks would be more convincing. This includes scaling to more dimensions or more complex utility functions, as well as a greater quantity of evaluations.", "was_revised": true, "final_justification": "The authors have cleared my concerns. Keeping the original score.", "rebuttal": "Thank you for your thoughtful review and insightful comments. We provide detailed responses to your questions below and include the additional experimental results according to your suggestion. We will modify the camera-ready paper accordingly if the paper is accepted.\n\n> How does the method perform over a larger variety of utility functions and benchmarks?\n\nWe evaluated our approach on five benchmark problems with dimensions ranging from 2 to 7 and number of objectives ranging from 2 to 9. This experimental setup is consistent with the scale and scope commonly used in the existing literature. In particular, we use all four multi-objective problems that were used in the benchmark paper [1], plus an additional one. Regarding the utility functions used, we already test a wide range of utility functions from [1] and [2], plus the Cobb-Douglas function which is a standard function in economics. We have not observed a significant influence of the utility function on results, and wouldn’t expect so since ANNs are at least in principle capable of representing any utility function.\n\nTo enhance the comprehensiveness of our evaluation, we have added an additional test case: the well-known multi-objective benchmark ZDT1 with 10 dimensions and 2 objectives, using a linear-exponential utility function defined as $U(y_0,y_1) = 5(y_0+2) + 5(y_1+2) + 2\\exp(0.75(y_0+2))\\exp(1.25(y_1+2))$, i.e., a sum of a linear component and an exponential product. The experimental result (20 repetitions, 50 iterations and all the experiment setting follows the paper) is shown in the table below:\n\n| Method        | 25th                 | 50th                 | 75th                 | 100th                |\n|---------------|----------------------|----------------------|----------------------|----------------------|\n| Random        | 57.58 (0.6948)       | 57.01 (0.6948)       | 56.75 (0.5361)       | 56.75 (0.5361)       |\n| qNEHVI        | 35.15 (1.514)        | 11.77 (1.913)        | 6.319 (1.136)        | 4.816 (1.047)        |\n| qParEGO       | 51.72 (0.7307)       | 48.53 (1.331)        | 47.80 (1.322)        | 46.44 (1.273)        |\n| PBO           | 52.62 (1.496)        | 48.01 (1.886)        | 38.66 (2.450)        | 24.75 (3.359)        |\n| BOPE-Linear   | 13.78 (1.283)        | 0.8842 (0.1596)      | 0.8842 (0.1596)      | 0.8842 (0.1596)      |\n| BOPE-GP       | 32.43 (2.314)        | 3.403 (0.8208)       | 2.327 (0.6852)       | 1.674 (0.6571)       |\n| BOPE-BMNN     | 14.08 (1.799)        | 0.3960 (0.0804)      | 0.2231 (0.0439)      | 0.2231 (0.0439)      |\n| BOPE-MoNNE    | 9.588 (0.8750)       | 0.2127 (0.0899)      | 0.0025 (0.0010)      | 0.0020 (0.0007)      |\n| Utility Known | 8.676 (1.016)        | 0.0558 (0.0183)      | 0.000125 (0.000034)  | 0.000059 (0.000018)  |\n\n\nFrom the table, we observe that BOPE-MoNNE achieves the best performance, followed by BOPE-BMNN and BOPE-Linear. The strong performance of the linear model can be attributed to the structure of the utility function, where the first component is linear. While BOPE-GP performs worse than these three methods, it still significantly outperforms the remaining baselines. PBO exhibits weaker performance, which can be explained by the challenge of navigating a 10d space using only pairwise comparisons. \n\nWe also conducted an ablation study as shown below (same experimental setting):\n\n| Method                          | 25th                   | 50th                   | 75th                   | 100th                  |\n|----------------------------------|-------------------------|-------------------------|-------------------------|-------------------------|\n| BOPE-GP                          | 32.43 (2.314)           | 3.403 (0.8208)          | 2.327 (0.6852)          | 1.674 (0.6571)          |\n| BOPE-MoNNE (monotonicity removed) | 17.98 (1.787)           | 1.275 (0.3492)          | 0.2872 (0.1461)         | 0.09786 (0.08952)       |\n| BOPE-MoNNE (ensemble removed)     | 9.195 (1.257)           | 0.4128 (0.1342)         | 0.2087 (0.1192)         | 0.1204 (0.0893)         |\n| BOPE-MoNNE                        | 9.588 (0.8749)          | 0.2127 (0.0899)         | 0.0025 (0.0010)         | 0.0020 (0.0007)         |\n\n\nThe results demonstrate that  incorporating monotonicity information improves the performance significantly, achieving better function values at a lower cost. \n\n\n\n[1] Lin, Z.J., Astudillo, R., Frazier, P. and Bakshy, E., 2022, May. Preference exploration for efficient Bayesian optimization with multiple outcomes. In International Conference on Artificial Intelligence and Statistics (pp. 4235-4258). PMLR.  \n[2] Astudillo, R., & Frazier, P. (2020, June). Multi-attribute Bayesian optimization with interactive preference learning. In International Conference on Artificial Intelligence and Statistics (pp. 4496-4507). PMLR\n\n>Are there cases where BOPE-MoNNE fails/performs worse than baselines?\n\nWe have not seen such examples and can’t think of a particular case where BOPE-MoNNE would perform worse. Conceptually and as shown in an ablation study, the incorporation of monotonicity should be beneficial, and the MoNNE seems to work well as a surrogate model. An ablation study also shows that the performance is robust with respect to small algorithmic variations. \n\n\n\n\n>discuss any other possible limitations\n\nWe cannot think of any major limitations relative to the benchmark algorithms. MoNNE is a black box, so one doesn’t really know what utility function the algorithm has learned - but so is the GP used in the benchmark algorithm. \nOur study doesn’t involve any human decision makers, but doing so would not only be prohibitively expensive but also not be reproducible. Instead, we use artificial decision makers that share characteristics with human decision makers, in particular sometimes selecting the less preferred solution if the solutions have similar utility.\nThe algorithm assumes a single stakeholder, and an extension to multiple stakeholders would be interesting - but other methods assume a single stakeholder as well. We will mention multiple stakeholders in the future work section."}, {"review_id": "fZAKg9OoX0", "reviewer": "Reviewer_cbyB", "summary": "This paper studies the problem of mulit-objective Bayesian optimization with preference exploration. It proposes a Monte-Carlo based method to solve the problem, using surrogate model of monotonic neural network ensemble. Experimental results indicate that it can achieve superior performance as compared to other baseline methods.", "questions": "1. What is the benefit of doing separate modelling for g and f? What if we just run preferential BO over the input to preference composite function? What is the intuition why the proposed method outperform PBO in Fig. 3？\n2. Can the method be extended to the popular Bradley-Terry preference model?\n3. How is the NN initialized? I expect this initialization method determines the prior of the model. (Similar to the lengthscale of the GP model.) \n4. In Sec. 4.1, \"Firstly, neural networks could be interpreted as nonparametric when their number of parameters is large enough, providing the desired flexibility.\" I feel this claim is a bit too strong.\n5. What do you specifically mean by \"scale invariance in preference learning\"?", "limitations": "1. NN usually requires more data to train. While BO setting has quite small dataset (especially in the initial steps). Can the authors comment on this issue?\n2. The survey would be more complete with more recent works on preferential BO, since similar techniques can be easily transferred to multi-objective setting with theoretical guarantees.\n3. The justification for only using $y$ in the historical data set for comparison seems weak to me. Indeed, if the preference elicitation is cheap, we can even build a preference model for $g$ offline. \n4. Some typos and font issues:\n- Line 28, \"it may\" ---> \"it may be\"\n- Fig. 3, the font for xticks and yticks look small.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality: Overall the proposed method is solid with convincing experimental result. But there are some key concerns I feel the paper needs to address.  \nClarity: The paper is mostly well-written except some minor typo and font issues.\nSignificance: The paper combines several existing techniques. The siginificance of the contribution is medium to me. \nOriginality: As far as I know, this work is original.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We provide detailed responses to your questions below and include the additional experimental results according to your suggestion. We will modify the camera-ready paper accordingly if the paper is accepted.\n\n\n> there are some key concerns I feel the paper needs to address \n\nWe are not sure which key concerns the reviewer is referring to, as only minor typo and font issues as well as medium significance are mentioned.\n\n\n>What is the benefit of doing separate modelling for g and f? What if we just run preferential BO over the input to preference composite function? What is the intuition why the proposed method outperform PBO in Fig. 3?\n\nOur results show that PBO alone is not as good. Compared to PBO, our algorithm exploits the inherent structure of the problem. For nested BO problems $\\text{max}~g(h(x))$ where $h$ is expensive-to-evaluate and $g$ is cheap to compute, modeling $h$ separately with GPs should outperform modeling $g(h(x))$ as a single GP, according to [1]. \n\nBesides the nested structure, our approach also allows to leverage the monotonicity of the utility function, which in turn allows to recognize dominance in multi-objective optimization. The benefit of explicit monotonicity is also clearly visible in our ablation study.\nTherefore, BOPE-MONNE's superiority over PBO for this type of problem can be expected.\n\n[1] Astudillo, R., & Frazier, P. (2019, May). Bayesian optimization of composite functions. In International Conference on Machine Learning (pp. 354-363). PMLR.\n\n\n>Can the method be extended to the popular Bradley-Terry preference model?\n\nThank you for this comment. The Bradley-Terry model is widely used to model choice behaviour in pairwise comparisons. It is, however, less a utility model (which aggregates different objective values into a scalar utility value), than a choice model which specifies probabilities of selecting one or the other alternative given their underlying utilities. We model this in our paper by adding a normally distributed random value to the true utility difference. Nevertheless, the two show very similar behavior in pairwise comparisons. \n\nThe Bradley-Terry preference model defines the probability of preferring item $i$ over item $j$, given their utility values $U_i$ and $U_j$, as $P(i \\succ j) = \\frac{e^{U_i}}{e^{U_i} + e^{U_j}} = \\frac{1}{1 + e^{-\\delta}}$, where $\\delta = U_i - U_j$ denotes the utility gap. A fundamental limitation of the standard Bradley-Terry model is its sensitivity to utility scale. For example, if a utility function ranges over $[0, 1]$ and we scale all utility values by 10 to obtain a new function ranging over $[0, 10]$, the predicted probabilities change significantly, even though the preference ordering remains the same. To address this issue, a more flexible formulation introduces a scale parameter $\\beta$, yielding $P(i \\succ j) = \\frac{1}{1 + e^{-\\beta \\delta}}$.\nWith the additive Gaussian noise used in our experiments, i.e., $\\epsilon \\sim \\mathcal{N}(0, \\sigma^2)$, the probability of selecting item $i$ becomes $P(U_i - U_j + \\epsilon > 0) = \\Phi\\left(\\frac{\\delta}{\\sigma}\\right)$, where $\\Phi(\\cdot)$ denotes the standard normal cumulative distribution function.\n\nBoth the Bradley-Terry and Gaussian choice models exhibit similar behavior: as $\\delta \\to 0$, the selection probability approaches $0.5$, and as $\\delta \\to \\infty$, it approaches $1$. Moreover, for a given noise level $\\sigma$, one can choose a corresponding $\\beta$ such that the two models closely align. For instance, when $\\sigma = 1$, setting $\\beta = 1.8$ results in almost identical probability curves. Therefore, we expect that our results will not change if our Gaussian-noise based formulation would be replaced by the very similar Bradley-Terry choice model. We will mention the similarity of the two models in our paper, as indeed the Bradley-Terry model is very popular and making this link is helpful for many readers.\n\n>How is the NN initialized? \n\nThe initial parameters are set by:\n\n        stdv = 1. /self.weight.size(1)\n        self.weight.data.uniform_(-stdv-6, stdv) \n        self.bias.data.uniform_(-stdv, stdv)\n\nThe bias initialization follows standard practice. For weights, we use a smaller range with a shifted lower bound (-stdv-6 instead of -stdv) because the subsequent exponential transformation amplifies the values. We will mention it in the revised paper.\n\n>\"Firstly, neural networks could be interpreted as nonparametric when their number of parameters is large enough, providing the desired flexibility.\" I feel this claim is a bit too strong.\n\nANNs are universal function approximators [1]. But we agree with the reviewer that our claim was a bit too strong. We have revised the sentence to: \"First, neural networks with sufficient parameters can approximate complex utility functions without requiring explicit assumptions about their functional form, providing the desired modeling flexibility.\" \n\n[1] Hornik, K., Stinchcombe, M. and White, H., 1989. Multilayer feedforward networks are universal approximators. Neural networks, 2(5), pp.359-366.\n\n\n>meaning of \"scale invariance in preference learning\"?\n\nJust from pairwise preference information, we cannot uniquely determine the underlying utility function. Any monotonic transformation will result in the same preference statements. \n\n\n>NN usually requires more data to train. While BO setting has quite small dataset (especially in the initial steps).\n\nAs our paper shows in Appendix A.3, given pairwise comparisons, if the network is trained using Hamiltonian Monte Carlo, the size of the dataset is indeed a problem. For this reason, we use the ensemble method.\n\nNeural networks have been used as surrogate models also in other traditional BO settings, such as [1][2][3].  The key difference of NNs in BO (when compared with NNs in Natural Language Processing or Computer Vision) is their typical shallow architecture (3-4 layers), which reduces data requirements and makes them well-suited for BO's limited data regime.\n\nBesides, we do compare to strong baselines designed for a low data regime (such as GP-based methods), and our approach works better. \n\n[1] Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., ... & Adams, R. (2015, June). Scalable bayesian optimization using deep neural networks. In International Conference on Machine Learning (pp. 2171-2180). PMLR.\n\n[2] Li, Y. L., Rudner, T. G. J., & Wilson, A. G. (2024). A study of Bayesian neural network surrogates for Bayesian optimization. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=SA19ijj44B\n\n[3] Brunzema, P., Jordahn, M., Willes, J., Trimpe, S., Snoek, J., & Harrison, J. (2025). Bayesian optimization via continual variational last layer training. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=1jcnvghayD\n\n\n>more recent works on preferential BO\n\nWe now cover a few more references in the related work section:\n\nXu, W., Wang, W., Jiang, Y., Svetozarevic, B. and Jones, C.N., 2024. Principled preferential Bayesian optimization. arXiv preprint arXiv:2402.05367.\n\nArun Kumar, A. V., Shilton, A., Gupta, S., Rana, S., Greenhill, S., & Venkatesh, S. (2024, August). Enhanced Bayesian optimization via preferential modeling of abstract properties. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 234-250). Cham: Springer Nature Switzerland.\n\nMikkola, P., Todorović, M., Järvi, J., Rinke, P., & Kaski, S. (2020, November). Projective preferential bayesian optimization. In International Conference on Machine Learning (pp. 6884-6892). PMLR.\n\nHowever, preferential BO doesn’t benefit from information about the nested problem structure and monotonicity of the utility function and thus is expected to perform worse in our setting.\n\n>The justification for only using y in the historical data set for comparison seems weak. If the preference elicitation is cheap, we can even build a preference model for g offline.\n\nWe can extend pairwise comparison to arbitrary outputs. In our paper we have decided to only compare outputs from historical observations because we are concerned that presenting non-existent options to a decision maker might lead to complaints in practice, when a solution examined and desired by the decision maker doesn’t exist.\n\nLearning preferences is usually not cheap, as it requires time from a human expert. Besides, without knowledge of the output values or ranges, offline preference learning becomes challenging as we cannot determine which outputs to present for comparison.\n\n>Some typos and font issues\n\nThank you for carefully pointing this out. We have corrected the typo and updated Figure 3 by adjusting the font size of the x-axis and y-axis tick labels in the revised version."}, {"review_id": "rPczxO5Hjj", "reviewer": "Reviewer_v6yY", "summary": "This paper introduces BOPE-MoNNE (Bayesian Optimization with Preference Exploration via Monotonic Neural Network Ensemble), a novel multi-objective optimization method that learns decision-makers’ preferences by leveraging the monotonicity of underlying utility functions. A key advantage of the proposed approach is its ability to incorporate utility monotonicity through neural network design—an often-overlooked yet critical aspect of preference modeling. Extensive experiments demonstrate the efficiency of the proposed method.", "questions": "1. Regarding lines 161–162: While constraining preference queries to only previously observed output pairs enhances realism, could this restriction hinder the efficiency of preference learning by limiting exploration?\n2. The ablation study section is completely moved to the appendix and somewhat limit. I would encourage the authors to include it in the main text, because the ablation study provides important insights into the contribution of key components of the proposed method.\n3. MoNNE requires training an ensemble of 8 networks, which is more expensive than GP-based methods. Is the performance gain worth the computational cost?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method uses neural networks as the utility surrogate model and incorporates the monotonicity of utility functions.\n2. This paper proposes one modification acquisition function IEUBO for MoNNEs, experiments demonstrate that IEUBO generally outperforms EUBO in the presence of utility noise.\n3 . The experimental results are comprehensive and verify the effectiveness of the proposed method.\n\nWeaknesses:\nWhile the paper demonstrates strong performance on synthetic benchmarks, the evaluation does not include real-world applications. A discussion of how the method might adapt to such settings—or a case study on real-world data—would strengthen the practical relevance of the contributions.", "was_revised": true, "final_justification": "The review's detailed response helped clarify my concerns. I will keep my score unchanged.", "rebuttal": "Thank you for your thoughtful review and insightful comments. We provide detailed responses to your questions below, and will make corresponding changes to the camera-ready version of the paper.\n\n>the evaluation does not include real-world applications\n\nWe would like to highlight that VehicleSafety, OSY, and CarCabDesign are generally considered real-world applications. Problems with a real decision maker are not considered as they would not be reproducible.\n\n\n>While constraining preference queries to only previously observed output pairs enhances realism, could this restriction hinder the efficiency of preference learning by limiting exploration?\n\nAsking a real decision maker to compare solutions that don’t exist may lead to dissatisfaction, which is why we do not allow it. But theoretically, allowing the comparison of arbitrary outputs may improve efficiency slightly, as we have demonstrated for BOPE-GP in Figure 17. For BOPE-MoNNE, allowing arbitrary outputs was slightly worse. In any case, BOPE-MoNNE which is constrained to historical observations still achieves superior performance compared to BOPE-GP which is allowed to use arbitrary outputs. \n\n> include ablation study in the main text\n\nThank you for the suggestion! We placed this content in the appendix due to space constraints. We will revise the paper layout to include at least a paragraph highlighting the main results in the main text.\n\n>Is the performance gain worth the computational cost of an ensemble of 8 networks?\n\nThis depends on the cost of training the MoNNE vs. the cost of running a simulation or asking an expert to provide preference information. Additionally, although MoNNE requires training 8 networks, we are able to  train them in parallel rather than sequentially, which provides significant speedup."}]

Author rebuttal:

Meta-review: This paper addresses multi-objective black-box optimization by integrating monotonicity into a neural network ensemble for preference learning. The main strengths lie in the clear presentation, comprehensive experiments, and demonstrated performance gains from exploiting monotonicity. The weaknesses concern limited evaluation on low-dimensional synthetic tasks and the potential impracticality of repeated preference queries in real-world scenarios. Nevertheless, all reviewers provided positive assessments, and the rebuttal effectively addressed most concerns. Overall, the contribution is solid and original, and I recommend acceptance, with the expectation that the authors incorporate reviewer feedback into the final version for improved completeness and clarity.
Decision: Accept (poster)