Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandits
Authors: Tian Huang
Shengbo Wang
Ke Li
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-objective optimization
dueling bandit
interactive multi-objective optimization
preference learning
Primary area: optimization

Abstract: The ultimate goal of multi-objective optimization (MO) is to assist human decision-makers (DMs) in identifying solutions of interest (SOI) that optimally reconcile multiple objectives according to their preferences. Preference-based evolutionary MO (PBEMO) has emerged as a promising framework that progressively approximates SOI by involving human in the optimization-cum-decision-making process. Yet, current PBEMO approaches are prone to be inefficient and misaligned with the DM’s true aspirations, especially when inadvertently exploiting mis-calibrated reward models. This is further exacerbated when considering the stochastic nature of human feedback. This paper proposes a novel framework that navigates MO to SOI by directly leveraging human feedback without being restricted by a predefined reward model nor cumbersome model selection. Specifically, we developed a clustering-based stochastic dueling bandits algorithm that strategically scales well to high-dimensional dueling bandits, and achieves a regret of $\mathcal{O}(K^2\log T)$, where $K$ is the number of clusters and $T$ is the number of rounds. The learned preferences are then transformed into a unified probabilistic format that can be readily adapted to prevalent EMO algorithms. This also leads to a principled termination criterion that strategically manages human cognitive loads and computational budget. Experiments on $48$ benchmark test problems, including synthetic problems, RNA inverse design and protein structure prediction, fully demonstrate the effectiveness of our proposed approach.

Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandits
Tian Huang Shengho Wanglt, Ke Li School of Computer Science and Engineering University Electronic Science and Technology of China
Department of Computer Science, University of Exeter
tianhuang uestc@gmail. com
shnbo. wang@foxmail com k. x.11@exeter ac uk
Abstract
The ultimate goal of multi-objective optimization (MO) is to assist human decision- makers (DMs) in identifying solutions of interest (SOI) that optimally reconcile multiple objectives according to their preferences. Preference-based evolutionary MO (PBEMO) has emerged as a promising framework that progressively approxi- process.
mates SOI by involving human in the www.iiaatiooccmmddeiiiioooo Yet, current PBEMO approaches are prone to be inefficient and misaligned with the DM's true aspirations, especially when inadvertently exploiting mis-calibrated reward models. This is further exacerbated when considering the stochastic nature of human feedback. This paper proposes novel framework that navigates MO to SOI by directly leveraging human feedback without being restricted by a prede- fined reward model nor cumbersome model selection. Specifically, we developed clustering based stochastic dueling bandits algorithm that strategically scales well high-dimensional dueling bandits. The learned preferences then transformed into a unified probabilistic format that can be readily adapted to prevalent EMO algorithms. This also leads a principled termination criterion that strategically manages human cognitive loads and computational budget. Experiments on 48 benchmark test problems, including the RNA inverse design and protein structure prediction, fully demonstrate the effectiveness of our proposed approach.
1 Introduction
Multi-objecttiv optimization (MO) represents a fundamental challenge in artificial intelligence [49], with profound implications that span virtually every sector sector-froot from scientific discovery [67] en- gineering design [22), and societal governance [23]. In MO, there is no single ulopian solution that optimizes all objectives; instead, the Pareto front (PF) comprises non-dominated solutions, each representing an efficient yet incomparable trade-off between objectives. The goal of MO is assist human decision-makers (DMs) in identifying solutions of interest (SOI) that optimally reconcile these conflicting objectives according to their preferences. This field has been subject of rigorous study within the multi-criterion decision-makinn (MCDM) community [ for more than half century. Over the past two decades, we have witnessed seamless transition from purely analytical methodologies
burgeoning interest interactive evolutionary meta- neta-heuristics. known
as preference-basse based evolutionary MO (PBEMO) 39).
As in Figure 1 a). PBEMО method involves three building blocks. The optimization module uses a population-based meta-heuristics to explore the search space. The preference information is progressively learned by periodically involving human DM in the consul tation module to provide
"Corresoondence: k. k.lieexeter. uk; Equal contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Optimization
Parametric/ Structured
Reward/ Ranking
Direct
((x)).(x))
X; > Xj
X; X
DM
DM
Consultation
X; > Xj
Clustering
e (VV eV
B(I,))
Preference Elicitation
PBEMO
1
PBEМO
2
PBÉМO
(a)
(b)
Figure 1: (a) Flow chart of conventional PBEMO. (b) Conceptual illustration of reward-basee model-based, and direct preference learning strategies.
preference feedback. The learned preference representation is then transformed into the format that guides the evolutionary search progressively towards SOI in the preference elici tation module. The overall PBEMO process is DM-oriented and is an
process. While PBEMO has been extensively studied in the literature the past three decades 36 59 F&), there are several fundamental issues unsolved, especially in the consul consuttation and preference elicitation modules, that significantly hamper the further uptake in real-world problem-solving scenarios.
(e.g., [4 15 g
The consultation module serves as the interface for DM interaction with the optimization module. It queries the DM about their preferences effectively recommend solutions. This closely aligns with preference learning in the machine learning community. There are three strategies accomplish this. The first, as shown in Figure FigureT b--)), involves this approach, DMs often pro- vide scores rankings for a large set of solutions. However, this overburdens the DM and risks introducing errors into the optimization process due to overlooked human cognitive limitations. The other strategies are grounded in the dueling bandits settings [66], utilizzig pairwise comparisons in consultation but with different assumptions about human feedback. Specifically, the second strategy relies on parameterized transitivity model [5] or a structured utility function for dueling feedback, such as the Bradley-Tere Terry-Luce model [1] in Figure 2b-2). While theoretically appealing. this method faces challenge in model selection that can be complex the original problem, posing difficulties in practical applications. Different from the previous two strategies, the last one tackles human feedback stochastic events such as Bernoulli trials [63], and it learns DM's preferences from their feedback as shown in Figure [b-3). However, a key bottleneck here is the souring number of DM queries required when considering population of solutions in PBEMO.
reward models [51] or ranking mechanisms (29] 27 38
The creference elicitation module acts as a catalyst, transforming the preference information learned in the consultatin module-usualle not directly applicable-ing inte--int> format usable in the underlying EMO algorithm. The stochastic nature of human feedback M can lead to a misuse of learned preferences that adversely disturb search processes. This issue is pronounced in PBEMO contexts, where the number of consultations is constrained reduce the human cognitive burden. Additionally, there is no thumb rule for determining the frequency of DM interactions or terminating such interactions, where current methods are often heuristics (e.g., setting a fixed number of interactions (36]
In this paper, propose novel direct PBEMO framework (dubbed D-PBEMO) that directly leverages DM's feedback to guide the evolutionary search for SOL. Note that it neither relies on any reward model nor cumbersome model selection. Our D-PBEMO framework consists of two key features.
Given the stochastic nature of human feedback, we develop a novel clustering tic dueling bandits algorithm in the consultation module. It is model-free and its regret
stochas-

---

is O T), where K is the number of clusters and T is the number of rounds. This overcomes the challenge of substantial queries inherent in conventional dueling bandits [4]
The preference elicitation module transforms the learned preferences from the consultation module into unified probabilistic format, which the associated uncer- tainty represents the stochasticity involved in preference learning. This not only streamlines the incorporation of learned preferences into the optimization module to guide EMO search for SOL, but also constitutes a principled termination criterion that strategically manages human cognitive burden and the computational budget.
2 Preliminaries
2.1 Multi-Objective Optimization Problem
The MO problem is formulated as: min F(x) (m. T where (x) ( Fn) is xEI2 an n-dimensional decision vector and F(x) is an m-dimensional objective vector whose i-th element is the objective mapping f: 52 - I, where 52 is the feasible set in the decision space IR.. Without considering the DM's preference information, given x" x² E Q, x is said to dominate x² (denoted have fi S fx(x) and F(x') ) # F(x"). A solution x E f is said to be Pareto-optimal iff AI'' E Ω such that x' 3 x. The set of all Pareto-optimal solutions is called the Pareto-optimal set (PS) and their corresponding objective vectors constitute the PF.
as x < x²) iff vi E {1...., m)
The ultimate goal of MO is to identify the SOI from the PS satisfying DM's preference. It consists of two tasks: ① searching for Pareto-optimal solutions that cover SOI, and @ steering these solutions towards the SOL. PBEMO addresses the task by employing an EMO algorithm as a generator of an evolutionary population of non-dominated solutions S {x J striking a balance between convergence and diversity for coverage. For the task @, PBEMO actively queries DM for preference information regarding these generated solutions, then it leverages the learned preferences to guide the EMO algorithm to approximate the SOI.
2.2 Preference Learning as Dueling Bandits
Since human feedback from relative comparisons is considerably more reliable than absolute labels 48 we focus on pairwise comparisons as a form of indirect preference information. In PВEМO, DM is asked to evaluate pairs of solutions (x', x²) selected from S, where i.j E (1..... N} and i + j. The DM's task is to decide, based on her/his preferences, whether x' is better, worse, or equivalent to x², denoted as x" >pp x², x' Xp x', or x' 2p x². Regarding stochastic preference, there a preference matrix for K Kaareed dueling bandits defined as = [pt,] where Pi.j the winning probability of the i-th arm over the j-th arm [65]. In particular, we have pi.j Pj.i = with Pt.; = 0.5. The i-th arm is said to be superior to the j-th one iff p., > 0.5. Simply considering each solution as individual arm will yield K N, which suffers efficiency in targeting the SOI when the evolutionary population is large [3). The ranking of all arms determined by their Copeland scores, where the SOI should be the Copeland winners that have the biggest Copeland scores.
Definition 2.1 (60)) ) The normalized Copeland score of the i-th arm, E {1,..., K}, given by:
C K-1
Σ
TI (pis > 0.5).
(+..E(L.....)
where 11 (-) is an indicator function. Arm K* satisfying 1s argmax Si is the Copeland winner. :E(1...,,K
The goal of dueling bandits algorithm is to identify the Copeland winner among all candidate arms with no prior knowledge of P. To this end, winning matrix introduced as B [bx_]l |x record the pairwise comparison labels, where by.j is the number of time-slots when the i-th arm is preferred from pairs of i-th and j-th arms. Consequently, we can approximate the preference probability with
whose upper confidence bound tt,, and lower confidence bound 4,, can be b mean Pa.j be. quantified as:
bii Mi.j = bis + big bij + bas""
a logg
b biss + bjg
a logt
+
lis
(2)
Disj + bj,a'
3

---

where a > 0.5 controls the confidence interval, and t is the total number of comparisons so far. The performance of a dueling bandits algorithm is often evaluated by the regret defined as follows. Definition 2.2. The expected cumulative regret for dueling bandits algorithm is given as:
(C*
T = - 2 ((
RT
2
where T's the total number of pairwise comparisons, (1) and (1) denote the pair to be compared at the 1-th (1 S S 7) round, (* represents the Copeland score of the Copeland winner.
3 Proposed Method
Our proposed D-PBEMO framework follows the conventional PBEMO flow chart in Figure Ta)) In the following paragraphs, we mainly focus on delineating the design of D-PBEMO with regard to both consultation tation and preference elicitation modules, while leaving the design of the optimization module open.
3.1 Consultation Module
As the interface by which the DM interacts with an EMO algorithm, the consultation module mainly aims to collect DM's preference information from their feedback upon identify the SOL, We employ the stochastic dueling bandits (74), to directly derive preferences from human feedback without relying on further assumptions such as contextual priors [41] or structured models [ In this setting. natural choice is to consider each candidate solution
arm to play. However, since the size of is usually as large as over 100 in the context of EMO, the conventional dueling bandits algorithms will suffer from a large amount of preference comparisons to converge [73] 35). This is impractical in PBEMO when involving DM in the loop. To address this problem, we propose clustering -aaee stochastic dueling bandits algorithm that consist of the following three steps.
Ji=1 based solution features in the context of EMO. Such partitioning is implemented as a clustering method based on the Euclidean distances between solutions of S in the objective space. Instead of viewing each solution as individual arm, we consider each subset S as an arm in our proposed dueling bandits. We denote the solution-level preference matrix as P.- = ipi,llxx. where PÝ,, represents the probability that x' >p x². Then, the preference matrix P in the subset-level can be calculated by Pijj = | Ex-E Ex' Pu,e,
Step 1: Partition S into K subsets (S*)
where |S*] stands for the size of St. This probability is well-defined since satisfies pr.j + Pj. 1. So far, we have reformulated subset-level dueling bandits problem. Accordingly, the subset-level Copeland winner is the subset S- that beats others on average.
Step 2: Subset-level dueling sampling and solution-level pairwise comparisons. We employ the double Thompson sampling algorithm [62] to determine the subset pairs, and then select solutions from the pairs to query DM preferences. We introduce two vectors
(eI...., uN) T and é = (L, ...,.., ( to record the winning and losing times of each solution respectively, initialized by i/ = 0, & = i = NJ. We perform the following steps within a given budget T.
Step 2.1: Determine the subset S' that most likely covers the SOL. We first narrow candidates to the subsets having the highest upper confidence Copeland scores, denoted as C² ( argmax, (.), where Č কー Ejæt T(un. > 0.5), i,j E {1....,K) Then, v C), apply Thompson sampling ( Beta(bi, + 1, 1) to sample the winning probability of S' over other subsets Š1, where j E (1,..., K} and j + i. Finally, we apply the majority voting strategy to determine the candidate by S¹ + argmax Ejet [(O! 0.5), where ties broken randomly.
Step 2.2: Select the subset S" that can be potentially preferred over S'. To promote exploration, we narrow candidates to the subsets whose lower-confident winning probability over Š' is at most 0.5, denoted C² 18%14,, 0.5}. Note that S' e C because 4, P., 0.5.

---

* Soi
* SOI
S
0.8
0.8
$1
0.6
0.6
Sa
0.4
00.00
0.4
S
0.2
0.2
8"
X
<pX
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(a)
(b)
Figure 2: (a) The evolutionary population of an EMO algorithm is divided into three subsets, where S2 covers the SOI (denoted as a +). (b) After a PBEMO round, in the next consultatioo session, all solutions steered towards the SOI and their spreads become more tightened towards the SOL.
Then, ASa E C?, we apply Thompson sampling as ( ~ Beta(br, 1,b, 1) sample the winning probability of Š* over S', and fix o/? = 0.5 according to the definition of preference matrix. Finally, the candidate is determined by S" +- argmax 0.0
Step 2.3: Select two representative solutions x' S' and x" E to query DM. We conduct uniform sampling to obtain solutions from the least-informative perspective. The DM is asked to evaluate the pair of solutions x"). If we observe >p x", we update br," t- b,, + 1, 8, + 1, and ln <- lu + 1, and vice versa. Note that other strategies to obtain solutions x' and x" can be used to improve the query efficiency.
Step 3: Output the learned preferences. The output is triplet (S* v. e), where St argmaxg; & is the optimal subset that most likely covers SOI, and candidate solutions are considered to be the SOI with uncertainty encoded by their winning and losing times.
The pseudo codes of the above algorithmic implementation are detailed in Appendix Figure gives illustrative example of the consultation process.
Remark 1. PBEMO is optimization-cum deciiiiii-un-eiiii------i----ee- Pareto-optimal candidate solutions upfront. PBEMO starts with a coarse-graiin representation of the PF. Then, it gradually steers incumbent solutions towards the learned SOI, which may be inaccurate initially. Subsequent consultations then serve as a refinement process. In this context, different from the dueling bandits, which are designed for identifying the single best solution in each round, our proposed method aims to recognize the SOI with a progressively refined fidelity.
process. Instead of having set of
Remark 2. Based the Remark 7 we intend explore the dependency among solutions 44 531.
This involves performing pairwise comparisons for solution groups, rather than for a single candidate, enable more efficient interaction. Consequently, in the preference elicitation stage, it becomes essential not only to rely on the learned preference but also to consider the uncertainty introduced by the coarse-graiie gaarie- representation.
Theoretical Analysis. We present a rigorous analysis of the regret bound of our proposed algorithm. To this end, we introduce the following two assumptions derived from the current literature.
Assumption 3.1 662 The winning probability between two arms satisfies Pı, * 0.5, Vi +
Assumption 3.2 (Tight clustering
33
All solutions in S* are the Copeland winners over other
solutions in the sub-optimal subsets.
5

---

Winning Voctor
Gaussian
Approcimation
Losing Vecior
Winning
Probability
Losing
Probability
0.215
Density Ratio
25
X²
SOI
(b)
Figure The density ratio between (x and pe( is shaded in blue, while its estimation
shaded in red. The SOI falls within the estimated Gaussian distribution for 95% confidence interval.
Theorem 3.3. Under the Assumptions 3.1 and 3.2 for any (0, 1] and 0.5, the regret of our clustering-base based stochastic dueling bandits algorithm is bounded by:
RT(T) S
Σ * E., <0.5
((++c)
log T
4a log T
+0
K²
DKI(P,, 0.5)
0.5)²
E2
Definitions of P.g and proof can found Appendix B.I
Remark 3. Theorem 3.3 reveals that the expected regret of our algorithm is bounded by O(K² log T). Compared with the dueling bandits algorithms based on Thompson sampling and their extensions to large arms [62 35J, whose regret is bounded by O(N2 log'),, our proposed clustering- stochastic dueling bandits algorithm more efficient in searching for the SOI, given K N.
3.2 Preference Elicitation Module
The preference elicitation module plays as bridge that connects consultation and optimiza- tion. It transforms the preference learned from the consultation on module into the configurations used in the underlying EMO algorithm, thus steering the EMO process progressively towards the SOI. In addition, this module maintains the preference information to inform strategic termination criterion of consultation, thus reduces DM's workload. We present the design of this module by addressing the following three questions.
How to leverage the preference learned from the consultation session? Let us assume the DM's feedback collected in the consultation session is drawn from preference distribution as the density ratio (x) where p, (x) and (x( is respectively the winning and losing probability of solution X sampled from the PS, see an illustrative example in Figure aa). After the r-th round of BauueBBar
the consultation session, we perform density-ratio estimation based on and l. By using moment matching techniques (see Appendix B B.2), we obtain Gaussian distribution with mean X and covari- ance D0 Let o, take the largest value of diagonal elements of Σ. The pref erence elici tation module maintains a Gaussian mixture distribution by a convex combination of Gaussian distributions from multiple consultation sessions:
N Noansult
Pr(x)
N(x Tr ΣT),
Z
where E Ω N Noonsult is the total number of consultation sessions conducted so far, and Z Neameuls 1//- is the normalization term. Figure b)) b) gives an illustrative example of Gaussian 2+=1 distribution approximated by the DM's feedback collected one consultation session.
How to adapt Pr(x) to EMO algorithms? We believe Pr(x) can applied in any EMO algorithms with few adaptation in their environmental selection. For proof-of-concept purpose, this paper takes NSGA-II [14 and MOEA/D 68 two most popular algorithms the EMO literature, as examples and we design two D- PBEMO instances, dubbed D- P--PBNNSAAT II and D D-PBMOEA/D.

---

At each generation of the original NSGA-II, it first uses non-dominated sorting to divide the combination of parents and offspring into several non-domination fronts F...... Fi- Starting from F1, one front is selected at time to construct a new population, until its size equals or exceeds the limit. The exceeded solutions in the last acceptable front will be eliminated according to the crowding distance metric to maintain population diversity. In replace the crowding distance with Pr(x). As result, the solutions
D- D-PBNSGA-II, close to the SOI will survive to the next generation.
The basic idea of the classic MOEA/D is to decompose the original MOP into a set of subproblems using weight vectors. Then, these subproblems tackled collaboratively In D-PBMOEA/D. we progressively transform the
using population-bie bauued originally uniformly distributed weight vectors W {w' } used in the original MOEA/D to the preference distribution Pr(x). In practice, the transformed weight vector w (w"), where Pr (.) is weighted sum of inverse Gaussian distribution, defined
Pr Pr"(")) in equation 19)
Detailed implementation D-PBNSGA- II and D-PBMOEA/D are in Appendix
When to stop querying DM? There is no principled termination criterion in exiting PBEMO, but is often set as a pre-defined number of consultation sessions. This not rationale and likely incur unnecessary workloads to DM, even when the evolutionary population is either converged to the SOI or being trapped by local optima. Under our D-PBEMO framework, we have the following theoretical result about the convergence property of Pr(x).
Theorem 3.4. Assume the preference distribution Pr(x) around the SOI follows Gaussian mixture distribution. It becomes stable when Neonsult increases.
The proof of Theorem 3.4 is in Appendix B.3 In D-PBEMO, we apply Theorem 3.4 to adaptively terminate the consultation session when Pr(x) becomes stable. In practice, this happens when the Kullback-Leible (KL) divergence of Pr(.) between two consecutive consultation sessions is smaller than a threshold e:
N,
log - (x,)
DKL Pr || Pr.)) = Σ
Pr, (8:)
(5)
log Pr, (S::)
where 1 1< S Neonsult. X; is sampled from St. No is the number samples when calculating the KL divergence. Here we use = 10 and its parameter sensitivity is studied in Section
Experiments
4.1 Experimental Setup
This section outlines some key experimental setup including benchmark test problems and perfor- mance metrics. More detailed information can be found in Appendix D
Benchmark problems Our experiments considers 33 synthetic test instances including ZDTI to ZDT4 and ZDT6 WFG5, and WFG7 30 (m 3). These problems are with various PF shapes and challenges such In addition, we also consider two scientific discovery problems
72 (m 2), DTLZI to DTLZ66 [8] where m {3. 5. 8, 10), and WFG1, WFG3,
as bias, plateau, and including 10 two-objective RNA inverse design tasks [55) 70 and 5 oour-objective protein structure prediction (PSP) task (69 The problem formulations are detailed Appendix D.I
Performance metrics As discussed in 37.. quality assessment of non-dominated solutions is far from trivial when considering DM's preference information regarding conflicting objective. In our experiments, we consider two metrics to serve this purpose. One is approximation accuracy c (S) that evaluates the closest distance of E S regarding the DM preferred solution in the objective space, denoted as 'golden point' x E Rm while the other is average accuracy E(S) that evaluates the average distance z'. Note that unknown to the underlying algorithm in practice, and the corresponding settings used in experiments are in Appendix D.2 Due to the stochastic
7

---

(())
E(S)
Scott Kio-KKott Rank
6
Seatt- Seott-Knott Knott Rank
2
5
3
5
6
Algorichm
Algorithm
Figure Box plot for the Scott-Knott test rank D-PBEMO and peer algorithms achieved by test problems running for 20 times. The index of algorithms are as follows: ~ue D-PBNSGA-II, 2~
D-PBMOEA/ D, I-MOEA/D BELAADDPPLVIE PLLV,, I-NSGA- II/LTR, IEMO/D.
Target Structure
D PBNSGA--I II
D-PBOOEA/D I- I-MOEA/D-PLVP I-NSGA- II/LTR
IEMO/D
Eterna ID: 852950
MFF. 24.1
MFE 25..7
MFF 23.2
MFE 24.5 0.77
MFE -44.2
σ 0.92
0.85
0.77
0.77
Figure 5: Comparison result of D S--PBNSGA-II II against the other three state-of-the-arr PBEMО algorithms on selected RNA inverse design task (Eterna ID: 852950). The target structure is shaded in blue color while the predicted structures obtained by different optimization algorithms are highlighted in red color. In this experiment, the preference is set to a 1. The closer a is to 1, the better performance achieved by the corresponding algorithm. When the shares the same biggest value, the smaller M E the better the performance is. Full results can be found in Appendix
nature of evolutionary computation, each experiment is repeated 20 times with different random seeds. To derive statistical mcaning of comparison results, we consider Wilcoxon rank-sum test and Scott-Knott test in our experiments. They are briefly introduced Appendix [D.3]
Comparison Results with State-of-the-art PBEMO algorithms
To validate the effectiveness of our proposed D-PBEMO framework, we first compare the per- formance of D-PENSGA-IT and D-PBMOEA/D against three state-of-the-art PBEMO algorithms, IEMO/D 58
I-MDEAD-PLVE 36 I-NSGA2/LTR
For the synthetic benchmark test problems, the comparison results of (S) and ε(S) in Tables A5 and A6 have demonstrated the competitiveness of our proposed D-PBEMO algorithm instances. In particular, D-PBNSCA-II and D-PBMOEA/D have achieved the best metric values in 77 out of 96 comparisons according the Wilcoxon rank-sum box plots of the ranks derived from the Scott-Knott test of D- D----NNA-AA---- II and D-PBMOEA/D compared against the other three peer algorithms further consolidate our observation about the effectiveness of our D-PBEMO framework for searching for SOI within a given computational budget. From the plots of the final non-dominated solutions obtained by different algorithms shown Figures AK to A9 we can see that the solutions found by our D-PBEMO algorithm instances are more concentrated on the 'golden point' while the others are more scattered. Further, the superiority of our proposed D D-PBEMO algorithm instances becomes more evident when tackling problems with many objectives, ie.. 2 Similar observations can be found in the scientific discovery problems, as the results shown in Tables A22 A23 A24 and A25 For the RNA inverse design tasks, the sequences identified by our proposed D-PBEMO a algorithm instances have good match regarding the targets as the selected results shown in Figure (full plots in Figures A14 and A13). As for the PSP problems, from the results in Figure 6 it clear to see that D-PBEMO algorithm instances significantly outperform the other peers. The protein structures predicted by our algorithms are more aligned with the native protein structure. Full results can found Appendix
the 0.05 significant level. In Figure 4 the

---

PERBOA 1I
PERKEA/D
I MODA/D ELVE
SEGA 1I/LIR
1ES0/D
ROISD 7,87G
RMSD -2.75
RMBD 9.551
JLMSD 10.420
JMSD 11.022
1836
RMSD 1.958
RMSD -1.822
RMSD 14.930
RSED 8.308
RMSD 2,032
2M/T
RAISD 1.450
RMSD 00.633
RMSD 1978
RMSD 1.39
JMSD 1.066
3PУK
-2.495
RMSD 1.821
RMSD -2.000
SVIA
TUMSD -7.005
DMSD -1.636
RMSD 10.360
RMSD 8.050
TMSD 12,061
Figure Experiments results for comparison results between D-PBEMO and the other three ofate-of-the-art PBEMO algorithms the PSP problems. In particular, the native protein structure is represented in a blue color while the predicted one obtained by different optimization algorithms are highlighted in red color. The smaller RMSD as defined in Equation (29) of appendix, the better performance achieved by the corresponding algorithm.
4.3 Investigation of the Effectiveness of Our Consultation Module
The consultation module, which learns the DM's preferences from their feedback, is one of the most important building blocks of our proposed D-PBEMO framework. To validate its effectiveness, we designed D-PBEMO variant (denoted as D-PBEMO-DTS) that uses the double Thompson sampling (DTS) widely used in conventional stochastic dueling bandits 62 as an alternative our proposed clustering based stochastic dueling bandit algorithm. From the results in Tables A7 and A8 it is clear see that 1--PBEO--DT observation can be attributed to the ineffectiveness of the traditional dueling bandit algorithms for tackling a large number of arms. In our PBEMO context, DTS requires at least thousands comparisons when encountering more than 100 candidate solutions. This is not feasible under the limited amount of computational budget. We envisage the same results will be obtained when considering other ducling bandits variants such as [74]]
always outperformed by our three D-PENSGA- and PBMDEA/D, This
Further, we designed another variant (denoted as OPPPMM---PB) that uses a parameterized preference learning model in Bayesian optimization (25) as alternative in the consul tation module. From the comparison results shown in Tables A7 and A8 we find that the performance of D D-PBEMO-PBO is competitive on problems with small number of objective (m 2). However, its performance degenerate significantly with the increase of the number of objectives. This can be attributed to the exponentially increased search that renders D-PBEMO-PEI ineffective by suggesting too many solutions outside of the region of interest. contrast, the clustering strategy in our proposed method strategically and significantly narrow down the amount of comparisons without compromising the learning capability. Additionally, the enlarged search space also makes the model selection in D-PBEMO-PER difficult. Our proposed method, on the other hand, learns human preferences directly from their feedback, thus scalable to a high-dimensional space.

---

4.4 Parameter Sensitivity Study
This section discusses the sensitivity study of two hyperparameterss of the D-PBEMO framework.
K' is the parameter that controls the number of subsets used in our clustering- stochastic dueling bandits algorithm. From the results shown from Tables and A12 considering K = {2. 10), is interesting to note that D-PBEMO is not sensitive to the setting of K when the dimensionality is low (i.e., m71 = 2). This implies that we can achieve a reasonably good performance even when using a smaller K, i.e., a coarser- -aaanneegr approximation of SOL By doing so, we can further improve the efficiency of our D-PBEMO framework.
As the dimensionality increases (i.e., m {3, 5, 10)), the population size will increase. Generally, as is shown in Table A13 A20 with larger populations, higher K tends to yield better results, aligning with our intuition. Furthermore, our significance analysis across 20 repeated experiment (Figure + reveals that the optimal K does not show significant differences in performance. In summary, K does not significantly impact the performance of our proposed D-PBEKO framework. For most problems, we do not recommend choosing a very small/large K (c.g.. K 2, K = N), as it may inefficiently narrow down the ROI.
To certain extent, the threshold plays an important role for controlling the budget of consulting the DM. From the results shown in Tables A9 and A10 we find that all three settings of 10-1 may lead to a
{10-1, 10-3, 10-6) have shown comparable results. However, a too large premature convergence risk. On the other hand, a too small y 10-6 may be too conservative to terminate. This renders more consultation iterations, thus leading to a larger amount of cognitive workloads to DMs. In contrast, we find that D-PBEMO algorithm instances can converge with less than consultation iterations with = 10 10-3
5 Limitations
Our proposed D-PBEMO directly leverages DM's feedback to guide the evolutionary search for SOI, which neither relies on any reward model nor cumbersome model selection. However, there are several potential limitations of D-PBEMO that warrant discussion here:
The regret analysis of our proposed clustering-basee stochastic dueling bandits is for the optimal subset, i.e., the region of interest on the PF. not yet directly applicable to identify the exact optimal solution of interest. As part of our future work, we will work on efficient algorithms and theoretical study on the best arm identification in the context the preference-based EMO.
This paper only analyzes the regret of the consultation module. How to further analyze the convergence of the D-PBEMDas a whole remains unknown. This will also lead to the next step of our research. In particular, if it is successful, may provide radically new perspective to analyze the convergence of evolutionary multi-objective optimization algorithms.
Conclusion
This paper introduced the D-PBEMO framework that is featured in a novel clustering- based stochastic dueling bandits algorithm. It enables learning DM's preferences directly from their feedback, neither relying on predefined reward model nor cumbersome model selection. Additionally, we derived unified probabilistic format adapt the learned preference prevelant EMO algorithms. Meanwhile, such probabilistic representation also contributes to principled termination criterion of DM interactions. Experiments demonstrate the effectiveness of our proposed D-PBEMO algorithm instances. In future, we will investigate more principled approaches to obtain K subsets, such as fuzzy clustering with overlapping. Further, we plan extend this current K K-armed bandits setup to a best arm identification. By doing so, we can directly obtain the solution of interest, with potential explainability for MCDM. Moreover, we will extend our theoretical analysis of the termination criterion to convergence analysis of PBEMO, even generalizable to the conventional EMO. Last but not the least, we will collaborate with domain experts to promote a emerging 'expert-in-the-lone scientific discovery platform, contributing to the prosperity of Al for science.
10

---

Acknowledgements
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here. Li was supported by UKRI Future Leaders Fellowship (MR/X0I 1135/1, MR/S0I 7062/1), NSFC (62376056, 62076056), the Kan Tong Po Fellowship (KTP/R1/231017), Alan Turing Fellowship, and Amazon Research Award.
References
[1] M. Agranov and P. Ortoleva. Stochastic choice and preferences for randomization. Journal of Political Economy, 125(1):40-68, 2017. 2
[2] Agrawal and N. Goyal. Further optimal regret bounds for thompson sampling. AISTATSS 13: Proc. of the 16th International Conference on Artificial Intelligence and Statistics, volume 31, pages 99-107, 2013. 19
[3] R. F. Alford, A. Leaver-Fay, J. R. Jeliazkov, M. J. O'Meara, F. DiMaio, H. Park, M. V. Shapovalov, P. D. Renfrew, V. K. Mulligan, K. Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation. 13(6):3031-3048, 2017. 25
[4] V. Bengs, R. Fakste, A. Mesaoudi-Pain Paul, and E. Hüllermeier. Preference- online learning with dueling bandits: A survey. J. Mach. Learn. Res., 22(1):278-385, 2021.
[5] V. Bengs, A. Saha. and E. Hüllermeier. Stochastic contextual ducling bandits under linear stochastic transitivity models. ICML'22. Proc. of the 39th international conference on Machine learning, volume 162, pages 1764-1786. PMLR, 2022.
16
[6] K. S. Bhattacharjee, H. K. Singh, M. J. Ryan, and T. Ray. Bridging the gap: Many-objective optimization and informed decision-making IEEE Trans. Evol. Comput., 21(5):813 820, 2017.
[7] J. Branke and K. Deb. Integrating user preferences into evolutionary multi-objective optimiza- tion. In Knowledge Incorporation Evolutionarr Computation, volume 167, pages 461 461-477. Springer Berlin Heidelberg, 2005. 16
[8] J. Branke, S. Greco, R. Slowinski, and P. Zielniewicz. Learning value functions in interactive evolutionary multiobjective optimization. IEEE Trans. Evol. Comput., 19(1):88-102, 2015. 2
16
[9] Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li. Learning to rank: from pairwise approach to listwise approach. In /CM1.107: Proc. of the 24th international conference Machine learning, pages 129-136, 2007. [6]
[10] E. Carlsson, D. P. Dubhashi, and F. D. Johansson. Thompson sampling for bandits with clustered arms. In IICAI'21: Proc. of the 30th International Joint Conference on Artificial Intelligence, pages 2212-2218, 2021. [8
[11] W. Chu and Ghahramani. Preference learning with gaussian processes. ICML '05: Proc. of
the 22nd international conference Machine learning, pages 137-144, 2005. 4 16
[12] K. Deb. Multi-objective optimization using evolutionary algorithms. In Wiley Interscience ser. syst. optim.. 2001. I6
[13] K. Deb and S. Gupta. Understanding knee points in bicriteria problems and their implications
as preferred solution principles. Eng. Optim., 43(1 ((:1 175-1144, 2011. 16
[14] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and clitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans. Evol. Comput., 6(2):182-197, 2002. 6
[15] K. Deb, A. Sinha, P. J. Korhonen, and J. Wallenius. An interactive evolutionary multiobjective optimization method based on progressively approximated value functions. IEEE Trans. Evol.
Comput., 14(5):723-739, 2010. 2 16

---

[16] K. Deb and J. Sundar. Reference point based multi-objective optimization using evolutionary algorithms. In GECCO 06: Proc. of the 8th anmual conference on Genetic and evolutionary computation, pages 635-642, 2006. 21 22
[17] K. Deb, L. Thicle, M. Laumanns, and E. Zitzler. Scalable multi-objective optimization test problems. In CEC 02: Proc. of the 2002 Congress on Evolutionary Computation., volume 1, pages 825 830 vol.1, 2002.
[18] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. Scalable test problems for evolutionary multi- objective optimization. In Evolutionary Multiobjectivve Oprimization, Advanced Information and Knowledge Processing, pages 105-145. Springer, 2005.
[19] J. Derrac, S. Gareía, D. Molina, and F. Herrera. A practical tutorial on the use of nonparametric statistical tests as methodology for comparing evolutionary and swarm intelligence algorithms.
Swarm Evol. Comput., 1(1):3 18, 2011. 27
[20] Y. Ding and Z. Zhou. Preference based adaptation for learning objectives. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 7839-7848, 2018.
26
[21] R. L. Dunbrack and F. E. Cohen. Bayesian statistical analysis of protein side-chain rotamer preferences. Protein science, 6(8):1661-1681, 1997. 25
[22] T. Erps, M. Foshey, M. K. Luković, W. Shou, H. H. Goetzke, H. Dietsch, K. Stoll, B. von Vacano, and W. Matusik. Accelerated discovery of 3D printing materials using data-driven multiobjective optimization. Sci. Adv., 42(7):1-10, 2021. [
[23] A. S. Flecker, Q Shi, R. M. Almeida, H. Angarita, J. M. Gomes-Selman R. Garefa-Villacortaa
S. A. Sethi, S. A. Thomas, N. L. Poff, B. R. Forsberg, S. A. Heilpern, S. K. Hamilton, J. D. M. C. nas, O. Dangles, C. Encalada, A. S. Fleischmann, M. Goulding, J. Higgins, C. Jezequel, E. I. Larson, P. B. McIntyre, J. M. Melack, M. Montoya, Т. Oberdorff, R. Paiva, G. Perez, B. H. Rappazzo, S. Steinschneider, S. Torres, M. Varese, M. T. Walter, X. Wu, Y. Xue, X. E. Zapata-Ríos, and C. P. Gomes. Reducing adverse impacts of amazon hydropower expansion. Science, 375(6582)753-766, 2022.
Abad, P. Anderson, N. Barros, I. C. Bernal, R. Bernstein,
[24] J. Fürnkranz and E. Hüllermeier. Pairwise preference learning and ranking. In Machine Learning: ECML 2003, pages 145-156, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. 16
[25] González, Z. Dai, A. C. Damianou, and N. D. Lawrence. Preferential bayesian optimization. In 1CML'17: Proc. of the 34th international conference on Machine learning, volume 70, pages 1282-1291. PMLR, 2017.
g 16 23
[26] S. Greco, M. Ehrgott, and J. R. Figueira. Multiple Criteria Decision Analysis. Springer New York, NY, 2016. [6
[27] D. J. Hejna and D. Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 2014-2025 PMLR, 2022. 2
16
[28] J. Honda and A. Takemura. An asymptotically optimal bandit algorithm for bounded support models. In COLT, pages 67-79. Citeseer, 2010.
[29] N. Houlsby, J. M. Hernández-LLobtto, F. Huszár, and Z. Ghahramani. Collaborative gaussian processes for preference learning. In NIPS'12: Proc. of the 25th International Conference on Neural Information Processing Systems Volume 2, pages 2096-2104, Curran Associates Inc.,
2012. 16
[30] Huband, Hingston, I.. Barone, and While. review of multiobjective test problems and scalable test problem toolkit. IEEE Trans. Evol. Comput., 17(5):777-506, 2006.
12

---

[31] A. Hüyük and C. Tekin. Multi-objective multi armed bandit with lexicographically ordered and satisficing objectives. Mach. Leam., 11006):1233-1263 2021.
[32] E. Jacquet-Lagreze and Siskos. Assessing a set additive utility functions for multicriteria decision-making the uta method. Eur. J. Oper. Res., 10(2):51-1664 1982. T6
[33] M. Jedor, V. Perchet, and J. Louëdec. Categorized bandits. In NIPS' 19: Proc. of the 32 Anmual Conference on Neural Information Processing Systems, pages 14399-14409, 2019. 5
[34] J. Komiyama, J. Honda, H. Kashima, and H. Nakagawa. Regret lower bound and optimal algorithm in ducling bandit problem. In COLT Proc. of The 28th Conference on Learning Theory, pages 1111-1154 154. PMLR, 2015. 9
[35] C. Li, I. Markov, M. D. Rijke, and M. Zoghi. Mergedts: A method for effective large-scale online ranker evaluation. ACM Trans Inf Syst, 38(4):1 2020.
[36] K. Li, R. Chen, D. Savić, and X. Yao. Interactive decomposition multiobjective optimization via progressively learned value functions. IEEE Trans. Fuzzy. Syst., 27(5) 2019. 2 I6
[37] K. Li, K. Deb, and X. Yao. R-metric: Evaluating the performance of preference-based evo- lutionary multiobjective optimization using reference points. IEEE Trans. Evol. Comput., (16))821-835. 2018. 7 16
[38] K. Li, G. Lai, and X. Yao. Interactive evolutionary multi-objective optimization via learning-to- rank. IEEE Trans. Evol. Comput., 2023. 2 16
[39] K. Li, M. Liao, K. Deb, G. Min, and X. Yao. Does preference always help? a holistic study on preference- based evolutionary multiobjective optimization using reference points. IEEE Trans. Evol. Comput., 24(6))
2020. 16
[40] K. Li, H. Nie, H. Gao, and X. Yao. Posterior decision making based on decomposition-ddiven knee point identification. IEEE Trans. Evol. Comput., (406)))3499-2333, 2022. 16
[41] D. Lindner, S. Tschiatschek, K. Hofmann, and Krause. Interactively learning preference constraints in linear bandits. In ICML'22: International Conference on Machine Learning, volume 162, pages 13505-13527. PMIR, 2022. 4
[42] M. Luque, L. V. Contana-Quinters A. G. Hernández-Díaz, C. A. C. Coello, and R. Caballero. g-dominance: Reference point based dominance for multiobjective metaheuristics. Eur. J. Oper. Res., 197(():685-692))) 2009. [6
[43] K. Miettinen. Nonlinear Multiobjective Optimization. Kluwer Academic Publishers, Boston, USA, 1999.
[44] K. Miettinen and M. M. Mäkelä. Interactive multiobjective optimization system www-nimbus on the internet. Comput. Oper. Res., 27(7):709-7233 2000.
16
[45] N. Mittas and I.. Angelis. Ranking and clustering software cost estimation models through multiple comparisons algorithm. IEEE Trans. Software Eng., (9(4):337-551, 2013. 27
[46] T. T. Nguyen and H. W. Lauw. Dynamic clustering of contextual multi-armed bandits. In Proceedings of the 23rd ACM International Conference on Conference Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pages 1959-1962. ACM, 2014. 17
[47] S. Pandey, D. Chakrabarti, and D. Agarwal. Multi-armed bandit problems with dependent arms. In Machine Learning, Proceedings of the Eventy-Fourts International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24. 2007, volume 227 of ACM International Conference
Proceeding Series, pages 721-728. ACM, 2007. 17
[48] F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM'08. 08: Proc. of the 17th ACM Conference on Information and Knowledge Management, pages 43-52. ACM, 2008
3 16
13

---

[49] S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Pearson, 4 edition, 2021.
[50] A. Saha and Gaillard. Versatile dueling bandits: Best-of- world analyses for learning from relative preferences. In ICML'22: Proc. of the 39th international conference on Machine learning, volume 162, pages 19011-19026. PMLR, 2022. [6
[51] A. Saha and S. Ghoshal. Exploiting correlation to achieve faster learning rates in low-rank preference bandits. In AISTATS 22: International Conference on Artificial Intelligence and Statistics, volume 151, pages 456-482. PMLR, 2022. 2
[52] M. Sugiyama, T. Suzuki, and T. Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012. 20
[53] Y. Sui, M. Zoghi, K. Hofmann, and Y. Yue. Advancements in dueling bandits. In IJCAL, pages 5502-5510, 2018.
[54] R. Tanabe and K. Li. Quality indicators for preference- based evolutionary multi-objective optimization using reference point: review and analysis. IEEE Trans. Evol. Comput., 2023. accepted for publication. 16
[55] A. Taneda. Modena: a multi-objective ma inverse folding. Advances and Applications in Bioinformatics and Chemistry, pages 1-12, 2010. 7 24
[56] C. Tekin and E. Turgay. Multi-objectivv contextual multi-armed bandit with a dominant objective. IEEE Trans. Signal Process., 66(14):3799-3813 2018. 17
[57] L. Thiele, K. Miettinen, P. J. Korhonen, and J. M. Luque. A preference-based evolutionary algorithm for multi-objeccivv optimization. Evol. Comput., 17(3):411-436, 2009. [6
[58] M. K. Tomezyk and M. Kadziński. Decomposition-basl interactive evolutionary algorithm for multiple objective optimization. IEEE Trans. Evol. Comput., 24(2):320-334, 2019. 8
[59] M. K. Tomezyk and M. Kadzinski. Decomposition-basee interactive evolutionary algorithm for multiple objective optimization. IEEE Trans. Evol. Comput., 24(2):320-334, 2020.
2 16
[60] Urvoy, F. Clerot, Féraud, and Naamane. Generic exploration and k-armed voting bandits. In (CML'133 Proc. of the 30th International Conference Machine Learning, pages 91-99. PMLR, 2013.
3 17
[61] F. Wilcoxon. Individual comparisons by ranking methods. Springer, 1992. 27
[62] H. Wu and X. Liu. Double thompson sampling for dueling bandits. Adv. neural inf. process. syst, 29, 2016.
4 5
16 17 19
[63] X. Yan, C. Luo, C. I.. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In SIGIR '22: Proc. of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 567-577, 2022. 16
[64] Y. Yang and Y. Zhou. Specific interactions for ab initio folding of protein terminal regions with secondary structures. Proteins: Structure, Function, and Bioinformatics, 722)),733803,, 2008.
25
[65] Y. Yue, J. Broder, R. Kleinberg, and Joachims. The k-armed dueling bandits problem. J. Comput, 78(5):1538-) 1556, 2012. 3
16 17
[66] Y. Yue and T. Joachims. Beat the mean bandit. In ICML'11: Proc. of the 28th international conference on machine learning, pages 241-248. Citeseer, 2011.
[67] H. Zhang, I. Zhang, Lin, C. Xu, Z. Li, Liu, Liu, Ma, Zhao, H. Jiang, C. Chen, H. Shen, H. Li, D. H. Mathews, Y. Zhang, and L. Huang. Algorithm for optimized mRNA design improves stability and immunogenicity. Nature, 621:396 103339-403,, 2023. [
[68] Q. Zhang and H. Li. Moea/d: A multiobjective evolutionary algorithm based on decomposition.
IEEE Trans. Evol. Comput., 11(6):112-73 2007. 6 22
23
14

---

[69] Z. Zhang, S. Gao, Z. Lei, R. Xiong, and J. Cheng. Pareto dominance archive and coordinated selection strategy-basee many-objective optimizer for protein structure prediction. IEEE/ACM Transactions on Computational Biology and Bioinformaticx, 2023.
26 29
[70] T. Zhou, N. Dai, S. Li, M. Ward, D. H. Mathews, and L. Huang. RNA design via structure-aware multifrontier ensemble optimization. Bioinform., 39Supppementt-1> 1): 13: 571, 2023. 7 24
[71] L. M. Zintgraf, D. M. Roijers, S. Linders, C. M. Jonker, and A. Nowé, Ordered preference elicitation strategies for supporting multi-objective decision making. In AAMAS 18: Proc. of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1477-1485. IFAAMAS, 2018. T6
[72] E. Zitzler, K. Deb, and L. Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evol. Comput., 8(2):173-195, 2000. 7
[73] M. Zoghi, S. Whiteson, and M. de Rijke. Mergeruch: A method for large-scale online ranker evaluation. In WSDM 15: Proc. of the 8th ACM International Conference on Web Search and Data Mining, pages 17-26, 2015.
17
[74] M. Zoghi. S. Whiteson, R. Munos, and M. Rijke. Relative upper confidence bound for the k-armed ducling bandit problem. In JCML'I44 Proc. of the 31st International Conference on Machine Learning, pages 10-18. PMLR, 2014. 4
19
[75] M. Zoghi, S. A. Whiteson, M. De Rijke, and R. Munos. Relative confidence sampling for efficient on-line ranker evaluation. In WSDM' 14: Proceedings of the 7th ACM international
conference on Web search and data mining, pages 73-82, 2014. 16 17
15

---

A Related Works
A.1 Multi-objective optimization
In the context of MO, to align with DM's preference, existing methods fall into three main categories: the a posteriori methods, the priori methods, and the interactive methods [12). The a posteriori methods first generates extensive efficient solutions through MO algorithms, then ask DM to select SOI from these solutions. In this way, DM is involved only when solution generation from the optimization module is finished, where the decision making process can be finished according to multi-criterion decision analysis [26] or certain SOI analysis [13 6 40). However, the absence of DM's preferences in solution generation can paradoxically complicate pinpointing the SOI, and the sheer number of generated solutions can be cognitively overwhelming for DM choose from. The a priori method, leverages pre-defined DM's preference to guide the search of SOI 7
37.. In other words, the decision making happens before optimization. This is more related to offline preference learning such as reinforcement learning from human feedback (RLHF) [38 However, given the black-box nature of real-world problems, eliciting reasonable preferences priori can be pointed out that the a priori method may lead to disruptive preferences and result in faulty decisions. Moreover, elaborating the DM labeled dataset used for offline preference learning such as RLHF is also difficult task. In contrast, the interactive method also known as preference-basee evolutionary multi-objective optimization (PBEMO) (39] presents a valuable opportunity for the DM to gradually comprehend the underlying black-box system and consequently
controversial. [39] [54]
refine user preference information 44 15
38
A.2 Preference learning in PBEMO
The consultation module in PBEMO collects DM's preference by actively querying DM's preference towards recommended candidates. Based on the requirements of human feedback, methods for consultation can be categorized into the following three types, as also presented in Figure lb).
The first type requires DM to provide scores or rankings for a bunch of (typically more than three) solutions. In [38], a net was introduced to conduct consultation in PBEMO. In a ranking model was required in the context of RLHF. In addition, as presented in relative preference can only be leveraged by ranking on more than three candidates. However, providing reward or ranking on a bunch of solutions will not only increase the workload of DMs, but will also introduce uncertainty and randomness into optimization
27
50
processes due to the limited capacity of DMs (88)
Algorithms in the second type rely utility function for dueling feedback [3], such as the Tradley-Terry-Laee model [T]] which has been widely explored in the literature 32 24 21) These algorithms can be traced back to 1982, when (32) introduced the UTT (UTillités Additives) method for deducing value functions based provided ranking of reference set. Following up work included (24) that employed pairwise preference to predict a ranking for potential labels associated with new training examples, (II) that utilized GP for pairwise preference learning (PGP) within a Bayesian framework, and [9] that proposed the ListNet which was an NN-based learning-to-rank method to model preference feedback. In addition, 29 extended the
parameterized transitivity model structured
work in [9] to handle multi-user scenarios by introducing weight vector for each user and combining multiple preference latent functions. More recently. (25] proposed a novel pairwise preference learning method using the concept of Bayesian optimization based on a structured surrogate model assumption, named preferential Bayesian optimization (PBO). Moreover, in 771. comparative study towards four prominent preference elicitation algorithms was conducted, and results shown that ranking queries outperformed the pairwise and clustering approaches in terms of utility models ad human preference.
Algorithms that belong to the third type consider human feedback as stochastic events such as Bernuuli trails, enabling quantifying randomness regarding the DM's capability. Dueling bandits algorithms are known for its efficiency towards interactive preference learning (65] has improved dueling bandits algorithms in both
In recent years, work like 775 62 x querying efficiency and computational complexity. Further investigations on dueling bandits are presented in A.3
16

---

Algorithms in the first two types burden the development of PBEMO, reward/ranking method fails take human capability into account and will navigate solutions an inaccurate direction, while the parameterized method relies heavily on model selection. It is well known that, in real-world problems, selecting an appropriate model is as difficult as solving the original problems. We appreciate the model-free settings in dueling bandits algorithms.
A.3 Bandit algorithms
The ducling bandits problem involves sequential decision-making process where learner selects two out of K "arms" in each round and receives binary feedback about which arm is preferred. Following [4] dueling bandits algorithms can be classified into three categories: MAB-rrlated, merge sort/quick sort, and tournamen/ccalllnnge.. In this paper, we focus on traditional dueling bandit algorithms falling in the MAB-related category. comprising four distinct methodologies for handling pairwise comparisons.
The first method known as explore then commit (ETC), which is utilized by algorithms such as interleaved filtering (IF) [65]. beat the mean (BTM) [66] and SAVAGE [60]. ETC methods kick out solutions that are unlikely to win, but this approach may lead to lower accuracy of predictive probability.
The second method involves using the upper confidence bound (UCB), for example relative upper confidence bound (RUCB) (74), MergeRUCB [73], and relative confidence sampling (RCS) 75). MergeRUCB. an extension of RUCB, is particularly designed for scenarios with a large number of arms. RCS combines UCB and Beta posterior distribution to recommend one arm for each duel in each iteration step.
The third method employs Thompson sampling, as demonstrated by double Thompson sampling (DTS) [62 and MergeDDTSS (35). Similar MergeRUCB, MergeDTS is designed for dealing with substantial number of arms. It is worth nothing that UCB methods assume the existence of Condorcet winner, whereas Thompson sampling methods assume a Copeland winner, representing a fundamental distinction between these two types.
The fourth method involves using the minimum empirical divergence, introduced by relative minimum empirical divergence (RMED) 34] and deterministic minimum empirical divergence (DMED) [28). RMED and DMED employ KL divergence metric to evaluate candidate arms.
Overall, these four methods represent different approaches to pairwise comparison traditional dueling bandit algorithms. However, the key bottleneck here is the sourcing number of DM queries required when considering a population solutions in PBEMO.
To our knowledge, there is no clustering teastering-bbsed dueling bandit algorithms. While in the field of multi-armed bandit (MAB), there has already exists clustered MABs which consider the correlation of arms [47 46). they relied heavily on complex model selection that is not applicable for weal-world scenarios.
Additionally, it is noted that multi-objective multi-armed bandit (MOMAB) is potential to address PBEMO problems. However, as pointed out in [1 (56), MOMABs belong to the a posteriori MO method, recalling I Specifically, the goal of MOMABs is to find an optimal multi-objective arm by sampling potential winners. However, MOMABS do not involve optimization for PF generation nor active query for preference learning. Thus it's not feasible to use MOMABs as our peer algorithms or consultation module.
B Derivation and Theoretical Analysis
B.1 Regret bound of clustering dueling bandits
Proof of Theurem 3.3 We follow the proof of double Thompson sampling for dueling bandits [62]. different from which the clustering operation will lead to non-stationary environment. As shown subsequently, the Assumption [.2 is important to build an auxiliary problem that is stationary, facilitating to apply the results in 162).
17

---

According to the property of Beta distribution, the estimated mean of probability for subset beats subset j is Pa.j (t) b Unfortunately, the pairwise comparison feedback does not follow stationary distribution. Specifically, Pi.j represents winning probability between the best solutions in S and $2, while (t) is computed from comparison among different solutions in the two subsets due to limited knowledge of the best solutions at the beginning. As result, the reward distribution is drifting [T]. To handle this, we introduce an auxiliary problem with stationary reward distributions. Based on comparison principles, the number of plays on each subset in the original problem can be upper bounded by the ones of the auxiliary problem. Note that the auxiliary problem are introduced merely for theoretical analysis.
(t,
Without loss of generality, we assume S to be St. Since we are most interested in locating the optimal subset, we define the underestimation of winning probability of Pı.j as Li,"" which is the
winning probability between: i) the solution x² E $1 with the lowest Copeland score over all solutions in S³, and it) the solution X E S with the highest Copeland score over all solutions in 5 = -.....K K. Likewise, we define Ba, as the winning probability between: i) the solution x' E St with the lowest Copeland score over all solutions in 83, and ii) the solution x E 8 with the highest Copeland score over all solutions in S* Pi.j > 0.5, ij 1,... K. For Pt. < 0.5, denote L,, 2,.; Lastly, we denote Li,; = 0.5, Vi = 1,..., K. According to Assumption 3.2 the optimality remains, i.e., the optimal subset in the context of Ptş is also optimal in the context of Lij'
With the auxiliary problem, we introduce the virtual Thompson sampling strategy that generates samples as 0f Beta(b),, 1, (t,(() - 1). where Ki, is the number of time-slots when subsets Si beats S under Pi.j" is easy verify two facts in expectation: i) bbg S br.j. and ii) b/,, 2 6z,.- Consequently, when Pi.j > 0.5 or i = 1, for samples (t) ~ Beta(b,,; + 1, 1,65, + 1) and (t) ~ Beta( + 1, 1), we have 6i,, dominates 0/ in expectation [10]
In what follows, we are going to quantify the number of plays of subset pair St and & To avoid ambiguity, let (t) denote the case where S' = St and S" =.'
When the first subset for comparison has been determined as St, the number of plays for St j + i, is
5
(T)
P S" = 5t)
t=1
T
S"" = St P3 < <0.5 IP
+ Σ PP (S" t=1
SS" S$ S ,ps P..)
(6)
When 2.g < 0.5. consider the following facts: i) 03.s are sampled from Beta(by,, + 1, b.j + 1) when / i; (i) 0z,, 0.5; : 0z. > Oc,, since subset &t is selected by sampling for comparison.
From these observations, the problem is equivalent to multi-arm bandit problem where the best arm indexed by i. Then, the left item of equation (6) can be further divided by
T
P S" = =S S,, < <0.5)
T
T
(5" = S Bys <c<0s)= + + p (5" = - c < Bys < <0.5)
Σ
T
P (5" = - .4, <c,N,,,(t-1) -1) DkL(cl10.5)
log T
1=1
log T
T
Σ
P 5" 5 B, <c.Nsilt 1) > DkI(cl(0.5)
+
Σ p S5"
$5,c < c Bzs <0.)
18

---

When Pj < 00.5, if p..s > 0.5, we have Pj.s > 23 thus B3 dominates 8/, in expectation. Therefore,
we have P (N - S (e.. 0.5)
Dxr.(cl0.33 where N -)) is the number of plays of pair S against S using the virtual Thompson sampling strategy. Otherwise, if
log
SP
N N - 1) S log
Pjá < 0.5, we have Pj.i < -,,- thus P (pJ.. <c) 2 P Piái <c cc) which follows the regular form as
Lemma in 62 According to the concentration property of Thompson samples
2 we have
Σ 8" = Lg, < x..5)
(1) -1, log T (c((0))0.5)
S" St Lzi
P2 C,
T
IP
8" = Sü Las < < -
log T
+
S5" = P
-S, s' Bz,t <0.5)
(.I(c(0.55)
1=1
log
DKL (k) +1.
(8)
(..55
As shown in 22 for any E E (0, 1]. c can be chosen such that DKL (c] = DKL (D, [105)/(1++))
and 1/DrI ppd) (1) Therefore the left item in equation 6 is bounded from above by
Σ P S" = S' < 0.5) S (1 (1++c) +
log T
O
(7)
(
1
(2 (0.5)
To account for the second term in equation
(6). the concentration property of relative upper confidence
bound 77] 62 will be used. For 2/ > 0.5, define AS =,,, 0.5 > 0, thus the right side of
equation 6 can be divided into
Σ
P
8"
= S', > 0.5)
,2 > 0.5, 0.5,N, N2,((1 1) 4a log T A
P
S" $5 L,
1=1
T S"
S 2,. > 0.5, N₁, - 1) 2 4a log T
(10)
Note that the subset S can be selected as S" only if 1,5 0.5. For the right item in equation 10)
Nt. 1) 2 4n log T is sufficient to yield 1,,5 + A,, 2 uj, finally we have uz, S Pj. Applying
concentration properties in Lemma 6 of 622. we have
T
Σ
P 5"
0.5, Nt. 1)
4a log T Aju
E S =
t=1
As result, we can also have a upper bound of the left item in equation 1T) as
T P
Σ
5" = S". Pn. > 0.5, N5,,N,,(t -
4a log
4ax log T A j.
(11)
A
1=1
Altogether, for j + i, have
T
E J.T Σ P (5" = ) s (1 c1 DKL DK 00.))
log T
4a log T
1
+
O
A
2
For the case where Š' and Š" are selected to be the same subsets, an upper bound of O(K) has been proven in Lemma 3 of 622. Therefore we omit the proof of this case for brevity.
19

---

B.2 Density-ratio estimation of preference distribution
Consider the DM's preference distribution determined by the density ratio of winning distribution over losing distribution, in other word, preference solution X is determined by p.. (x)/px(R) From consultation module, we have obtained the winning and losing time vectors v and & of all candidate solutions, which in turn gives us density estimation of Pe and pe. We are going to perform density- ratio estimation based upon and obtain the density ratio, namely the preference distribution pr (X)), which can be mathematically formulated as
(x pr (x))pr())
(12)
We use finite-order moment matching approach without conducting complex density estimation of each (x( and pe (x). Denote a vector-valued nonlinear function Φ - R Consider the first
k-th order moments, we have ((x) (xx) ..... x()),, where (*)) stands for element-wiss k-th
power of x. Then, the moment matching problem for density-ratio estimation can be formulated as
arg min
/ (x)px)))))
(x)
(13)
Equivalently, we can reformulate the problem by
/ (x dx
2 2 (X)))) dx, / (x)
di dx).
arg min Pr
(14)
where (.) denotes the inner product. Viewing each component in equation (4 as the expectation operation of function (X) over distribution (x) and (x()),
sample averages to esti-
mate the expectation [52] To this end, we construct two sample vectors 4 (Φ(*)) x(x%.))
and 4; (()))..... ((x; () based on the wining and losing time vectors and t. Specifically
x and X are solutions that has won or lost one time during consultation. Note that solution may appear in P or 4 more than one time since its winning or losing time is greater than 1. N, and N, the number of winning and losing solutions, calculated by the sum of all winning losing times for all solutions. Correspondingly, we denote p. (x)..........
An estimator p. for
can be calculated by solving the following problem:
p. arg min T. E.E.A, NZ Pr th D.P.
N.N: P. 0. I,INN,
(15)
where 1N. is the Ng-dimensional vector whose elements are all valued by The optimality is reached by when the derivative of equation (15) is zero, i.c..
2 d.p. N,N, D p. = -k;
(16)
N
where Og is the k-dimensional vector whose elements are all valued by O. Correspondingly, we have
N N
pr
(Φ]
8, I.IN.
(17)
Note that is the estimation of true density ratio pr, namely the probability density function, on solutions X = 1,...,,N, By assuming Gaussian distribution around SOI, we can further approximate the mean and variance by
N₂
N
Σ = diag
D.
x
(18)
where we dismiss the covariance between any two dimensions in solution space for brevity.
Remark 4. The sample averages can also be performed by first estimating the winning and losing density distributions based on and É, then sample solutions on them to increase the sample numbers. This will improve the estimation in equation (14) at the cost of complex density estimation. addition, perturbations solutions can be introduced for legal inverse operation of o Dy in equation (17).
20

---

B.3 Convergence of preference distribution
Proof of Theorem 3.4 Form Theorem 3.3 when Nconsult increases, the clustering dueling bandits can grasp DM's preference towards parwise comparisons with increasing accuracy. Therefore, the convergence of preference distribution Pr(x) will be mainly influenced by the behaviors of EMO algorithms as well as the landscape of MO problems. To this end, we consider two extreme cases: i) the population distribution of candidate solutions from EMO generators remains unchanged, and it) the population of solutions are steered into shrinking regions by EMO algorithms.
For the first case, we let the unchanged distribution be Nx X E.,), Vr Nu. where N2 > denotes a threshold number of consultation sessions. The convergence is presented by the following truth:
No Σ r=1 0
N Nommall Σ
Pr(x)
N(x X +
N(x X Σn)
Z
0u
7= No+1
N(x X Σn)
N. 1 + (Nconsult Nu) 7= 0,
Bu
(Nconsult Nu) X En))
Σ (Neonsult Nu) T..
Note that 0 and is constant. When Neonault >> therefore Pr(x) is convergent to (x X E...
No. we have Pr(x) + 0 N(x X ).).
For the second case, shrinking regions with increasing Nconsult result in uniform distribution in the context of Gaussian distribution around SOI. In this case winning and losing probability are close to each other. As a result, a large o, will be computed. Therefore this distribution hardly contributes to the mixture distribution. Consequently, with shrinking regions, effect of consultation sessions decreases to zero, thus Pr(x) convergent ultimately.
Based on these observations, two extreme cases can result in convergent stable mixture distribution, covering all other cases in PBEMO between the two extreme ones. The proof is complete.
C Algorithmic Implementation of Our D-PBEMO Instances
The structural pscudocode for D-PBEMO is shown in Algorithm I The D-PBEMO framework is made up of three component, consultation, preference elicitation and optimization module. In this paper, contribution is mainly in consultation and preference elicitation module (highlighted by rhd). In consultation module, we proposed a clustering based stochastic dueling bandit algorithm (pseudocode can be referenced in Algorithm(2 (). While in this section, will delineate the step-by-step process of three different preference elicitation module. The code of our algorithms and peer algorithms are available at https: http:::://ggtuu.... com Laab//tooo--arartrrrrrorrereeeee EMOC//
C.1 D DI
This section will discuss how the learned preference information can be used in NSGA-II 17 Based on [16), solutions from the best non con-dominttinn levels are chosen front-wis, wise as before and modified crowding distance operator used choose a subset of solutions from the last front which cannot be entirely chosen to maintain the population size of the next population, the following steps are performed:
Step Before the first consultation session, the NSGA-II runs as usual without considering the preference information.
Step If time to consult user preferences (e.g., when we have evaluated the population for 50% of the total generation), we will update the preference distribution Pr(x) defined in equation (4)
Step 3: Between two interactions, the crowding distance of each solution will be evaluated by the predicted preference distribution Pr(x) learned from the last consultation session.
21

---

usual without considering the Algorithm D-PBEMO structure
Input: G maximum generation, N solutions in population.
1: 0; // represents the consultation session count.
2: current_generalion <- 0.
3: Pro(-) < 0.
4: while current seveenttggnerttionn G do
5: I/ Phase I:Consult user using C-DTS.
6: if time consult (e.g. perrent_generrrtii > 0.5 G. and DEL (Pr
Pr,)) then
7: Drunn blustering-baseee stochastic dueling bandit (Algorithm 2
8:
T+ +
9:
end if'
I/ Phase 2: Preference elicitation and optimization.
11: if = 0 then
Optimization;
13: else
Preference Elicitation, update (x,(
15:
Optimization;
end if
17: end while
Output: Solutions x; (,, (1,2
C.2 D-PBMOEA/D
Following [6). MOEA/D 668 is designed to use a set of evenly distributed weight vectors W {w') to approximate the whole PE. The recommendation point learned from the consultation module is to adjust the distribution of weight vectors. The following four-step process is to achieve this purpose.
Step 1: Before the first consultation session, the EMO algorithm runs as usual without considering any preference information.
Step 2: If time to consult (e.g., when we have evaluated the population for 50% of the total gen- eration), the whole population is fed to consultation module and the three outputs will be recorded and used to update the predicted preference distribution for the current population Pr(x) defined in Equation (4).
Step 3: The weight vectors W {wi Jizı will be projected to the preference distribution Pr by using the weighted sum of inverse Gaussian distribution. For a specific weight vector w
Noonsutt
" Pr (w"") = Z
1
ay-1(w' w St.):
(19)
where i = 1,2 .,.,, N and w is the corresponding weight vector of x in Pr(x) (Equa- tion Output the adjusted w as new weight vectors W" for next generation.
C.3 D-PBEMO-DTS
This section will discuss how to utilize DTS consultation module in ablation experiment. In each consultation session, the budget for pairwise comparison is limited to 100. The optimization module is Pareto-based EMO algorithm, i.e., NSGA-II [17]. Based [16]. solutions from the best non-domination levels are chosen front-wise as before and a modified crowding distance operator is used choose subset of solutions from the last front which cannot be entirely chosen to maintain the population size of the next population, the following steps are performed:
Step 1: Before the first consultation session, the NSGA-II preference information.
Step time to consult (e.g., when we have evaluated the population for 50% of the total generation), then the whole population will be fed into the DTS and the optimal arm will be
22

---

Algorithm 2 Clustering-based Stochastic Dueling Bandit (C-DTS)
Input: T E (1,2...) (xo), N arms, K subset.
1: v -- 0x, 4- ON:
2: B <- θхж:: "i
1,, 44, for i,j (......K' K}:
3: for = I..... .T do
4: I/ Step Choose the first candidate subset S ber
5: ui,j = bbj + v a log (br, + bj.s). 4,5 b..
big
V log t/ (bi, + bj.s). if + j.
and u4 li, 1/2, Vi; let z/0 := for any æ, when bis bg.5 0.
6:
+ 1/(K - jラ I(u, > 1/2); // Upper bound of the normalized Copeland score
7:
c (i: { max, {.):
8:
j = L....., with i < do
Sample (1 ~ Beta(bi, + 1, by.a +1;
9:
(1) // + 1116 i.5
10:
11:
end for
S arg max Ci+i:((0)) 1/2); //Choosing from C¹ to eliminate likely non-winner
arms: Ties broken randomiy.
13:
I/Step 2: Choose the second candidate subset S"
14:
C² (Sr14,, S 0.5);
Sample 0!2) ~ Beta(b,, br., + +1, +1,b,, 1) for all o S' and let 0 = 1/2:
15:
S" <- g 0(2)
16:
I/ Step Query and update
Randomly select an arm x' from S¹, and x" from S" And observe the result.
19:
if x >p x" then
20:
br b1 1 and u, bu t tu
21: else
b + b and Un t Un 1, l l +;
23:
end if
end for
Output: (S* v. L} where S* argmaxg: E
recorded and used to update the predicted preference distribution Pr(x) as Equation (4) for the current population.
Step 3: Between two interactions, the crowding distance of each solution will be evaluated by the predicted preference distribution Pr(x) learned from the last consultation session.
C.4 D-PBEMO-PPI PBD
In this section, PBO (25] is utilized as the consultation module for D-PBEMO. The PBO is initialized with 4 pairwise comparisons and its query budget is limited to 10, Also the acquisition function is set to Thompson sampling. The optimization module is chosen to be decomposition- based EMO algorithm, i... MOEA/D (68]. Following (36), the decomposition- set of evenly distributed weight vectors W {w') ) to approximate the whole PF. The recommendation point learned from the consultation module is to adjust the distribution of weight vectors. The following four-step process is achieve this purpose.
EMO algorithm (e.g..
MOEA/D (6)) is designed
Step 1: Before the first consultation session, the EMO algorithm runs as usual without considering any preference information.
Step time to consult (e.g., when we have evaluated the population for 50% of the total genera- tion), the whole population is fed consultation module and the recommendation solution will be recorded and used to update the predicted preference distribution for the current population Pr(x) as Equation (4).
23

---

Step 4: The weight vectors W {w' He will be projected to the preference distribution Pr by using the weighted sum of inverse Gaussian distribution as defined in Equation (9). The adjusted weight vectors W' = [w")
Ji=1 will be the new weight vectors for next generation.
We present the lookup table for key notations in the following table
D Experimental Settings
D.1 Benchmark Test Problems
This section introduces the problem definitions of two real-world scientific discovery problems considered in our experiments. Further, we summarize the key parameter settings in our experiments.
D.1.1 Inverse RNA design
An RNA sequence of length is specified string of base nucleotides x1 .n. where I; E {A, C, G. U} for = 1.2..... n. A secondary structure P for x is a set of paired indices where each pair (i,j) P indicates two distinct bases æ,z, {CG, GC, AU, UA, GU, UG} and each index from to n can only be paired once. For example, given target secondary structure ((..)", the predicted sequence 2 2122 should satisfied that the 1st and 5th nucleotides in paired set {CG, GC, AU,
A, GU,UG)
Reference from
55
70,, our inverse RNA design problem adopts two objective functions.
The ensemble of an RNA sequence is the set of all secondary structures that x can possibly fold into, denoted as Vxx). The free energy AG(x, y) used characterize the stability y E (x). The lower the free energy AG(x, y), the more stable the secondary structure for The structure with the minimum free energy (MFE) is the most stable structure the ensemble, i.c. MFE structure.
Stability
fi = MFF(FF = argmin Ax(x,y)
(20)
yEy
Note that ties for argmin are broken arbitrarily, thus there could be multiple MFE structures for given x. Technically, MFE should be set. our experiment, the MFE is calculate by ViennaRNA package.
Similarity
The second objective function is to assess the similarity between the best secondary structure of our predicted sequence and the target structure.
(n d)/n.
(21)
where d is the Hamming distance between our predicted structure and target structure, and a E [0, 1|. For example, when the target structure " (...)" and the predicted secondary structure is "...." "....", d and σ = (3 /5 0.6. The bigger the similarity the more precise our predicted structure is. Since our MO problems are minimization problems, the second objective should be:
d/n.
(22)
D.1.2 Protein Structure Prediction
Protein structure prediction (PSP) is the inference of the fitee-dimensional structure of a protein from its amino acid sequence that is, the prediction of its secondary and tertiary structure from primary structure.
To computationally address PSP, the initial step involves constructing protein conformations using a suitable model. Traditional Protein Data Bank files contain Cartesian coordinates of all atoms in protein, often totaling 1000 atoms for modest 70-amino acid sequence. This abundance of coordinates renders Protein Backbone Exploration with EMO algorithms nearly impracticable. To mitigate this, protein structures are represented using torsion and dihedral angles, obtained by
II Table A1 we use the i-th arm to denote the i-th subset in context PREMO.
24

---

converting atom coordinates along consecutive bonds, effectively reducing the search space while retaining essential structural information. Protein backbones are primarily defined by three torsion angles: Φ (around the -N - C. bond), V (around the C C - bond), and f2 (around the -C- N peptide bond). Additionally, each amino acid residue may feature varying numbers of side- chain torsion angles (x. i E 1, 2, 3, 4) depending on its structure. Despite this, the conformational search space remains extensive. To address this, secondary structures and ceckbone-independarr rotamer libraries [21] are utilized to further constrain backbone and side-chain angles, respectively. These libraries also integrate biological insights from protein secondary structures. Moreover, this approach necessitates less optimization space compared to those relying contact maps. For instance, reconstructing the main and side chains of 50-amino acid sequence requires only 250- 300 angles using dihedral angles, whereas contact maps demand 2500 distances for the main chain alone.
In this paper, we consider PSP tasks as a four-objective optimization problem. Each objective function is an energy function that determines regulatory property of a protein.
CHARMM
The first objective function fi is CHARMM that calculates several energy terms that determine protein properties. The corresponding energy function is formulated as:
fi Ec
Σ
ko(b bu)²
Σ angles
ko(O 0²
bounds
Σ dshedrals
kipl1 + cos(ne - 6)]+
(23)
Σ improper
kw w
Σ
kulu 2
Uvey- Brodly
Σ
Evil( Tíj
Rig )
Raj
y0 rij
99
e e-rs
Van-der- Voals
charge
where b is the bound length, bo is the ideal length, 0 is the angle formed by three atoms involved in two connected bonds, is the torsion angle, w is the improper angle, u is the distance between two atoms of nonbonded interactions in an angle, ki, kg hig, ō, n. liw wo, ky, U0 are constants. The bonds, angles dihedrals, improper, and Srey-Brradley terms belong the bond term, while the non-bond term includes Wanlds and charge terms. Apart from these, Van-der-Waals and charge terms used to calculate the Van der Waals force and electrostatic energy between a pair of atoms (i.j), respectively.
dDFIRE
The second one f2 is dDFIRE that follows interactions of atom pairs, i.e., that between polar and nonpolar atoms, and that between polar and non non-hydrogen-bone bonded polar atoms, to get excellent solution for potential. The energy of atoms (p.4) q) pair calculated as:
{
RT (0.
N.(0p.0..000..1 Nol0, Be.
T < Yeuts
pD(r, 0g. pp))
(24)
2 Feut:
where (0p. fg. 8pq, r) is the number of polar atom pair (p.g) at distance 6p. 0g and 6pg orientation angles of polar atoms. R is the gas constant, temperature T is set as 300 K, r is the distance between an atom pair, the cutoff distance Teut is set to 14.5A, and Ar(Areud is the bin width (Feat). The value of parameter is proven to be 1.51 (64]. The total dDFIRE potential is the summation of energy of all possible atom pairs (p. q) in the protein structure:
f2 ED
Σ Vpg 9pp
PD(Tpg 0p Ogr 0pg)
(25)
The third objective function fa is Rosetta, composite energy energy function based on physics and knowledge, each potentials of is intricately designed:
Rosetta
1== = Σ w: Ex, P),
(26)
where w and are the weight and degree of freedom of each energy term, respectively. P the protein structure. Details on Rosetta can be referenced in (3]
25

---

RWplus The last objective function fa is RWplus, whose calculation is the same as that of dDFIRE. However, emphasizes more about the potential of short-range interactions:
= ERW Σ -IT Nola, 3. R) (a B, R)
-0.16((, In N (A, AA
Σ A,B
(27)
N [, B,
0,,
where k is the Boltzmann constant and T' the temperature in units of Kelvin. No 3, R) and Ne (a, 3, It) are the observed and expected number of atom pairs, with and 3 atom types within distance R. respectively. Similarly, (. B, OAB) and (e,A, B, OAB) are the observed and expected number of atom vector pairs with (A, B) type within relative orientation OAB respectively. The type of atom vector pairs between atom types and / (A, B). 8(A, B) when vector pairs A and B are not contact, and 1 vice versa. The total RWplus potential is the sum of these atom pairs' energies.
The problem formulation of PSP can be referenced in
69
D.2 Parameter Setting
This section lists several parameters used in experiments, including the parameters in EMO algorithms, and other parameters we need in D-PBEMO:
The probability and distribution of index for SBX: p. 1.0 and 7/c 20;
The mutation probability and distribution of index for polynomial mutation operator: Pm
and 1m 20;
The population size for different problems can be referenced in Table
The maximum number of generation G can be referenced in Table A3
For I-MOEA/D-PLVF and I-NSGA2/LTR, the number of incumbent candidate presented to decison maker (DM) for consultation: μ = 10;
For I-MEA//D-PLVV and I-NSGA2/LTR, there exists the number of consecutive consultation session 7 = 25;
The step size of reference point update for MOEA/D-series PBEMO algorithms are set to: = 0.3.
The reference points in different test problems be referenced in Table A4 The reference point of real-world problems will be introduced their experiment results.
While the winning probability is usually known in advance in dueling bandits, it however does not exist in the context of MO. In this paper, to project a MO solution to arm in the bandit setting, we define the winning probability pi, of a pair of candidate solutions (x;, Vi,j (I....,,AN N} 20
P&j = ((P((x)) - Pr(x,)),
(28)
where u(a) = 1/(1 exp(-a)), Pr denotes the real preference distribution of each arm. For example, we set Pr(x) = (x|x" [eilx*,=** ,²²), where x" is decision vector of the reference point z' (Table A4) and o* is constant value initially set 0.1. Note that Ptj can take other forms in case it satisfies the conditions in Section
The subset number K:
10 if m 2,
8
if m 3,
K
12
if
m = 8,
18 if m = 10.
The budget for dueling bandits algorithm is set as 100.
The parameter úi, and 4,, set as a 0.6.
"hanss/githubb.om//zaannnoo Proteei---iii---- Protein-Structure- Crrencttrre-eeeictiionn
26

---

D.3 Statistical Test
Wilcoxon rank-sum test To offer a statistical interpretation of the significance of comparison results, we conduct each experiment 20 times. To analyze the data, we employ the Wilcoxon signed-rank test [6] in our empirical study.
The Wilcoxon signed-rank test, a non parametric statistical test, is utilized to assess the significance makes minimal assumptions about the data's underlying
of our findings. The distribution. It has been widely recommended in empirical studies within the EA community [9] In our experiment, we have set the significance level to p 0.05.
advantageous
Scott-Knott rank test Instead of merely comparing the raw (S)) and ε(S) values, apply the Scott-Knott test [45] to rank the performance of different peer techniques over runs on each experiment. In a nutshell, the Scott-Knott test uses statistical test and effect size to divide the performance peer algorithms into several clusters. In particular, the performance of peer algorithms within the same cluster is statistically equivalent. Note that the clustering process terminates until no split can be made. Finally, each cluster can be assigned rank according to the mean e e"(S) or ((S) values achieved the peer algorithms within the cluster. In particular, since smaller e (8) and E(S) value is preferred, the smaller the rank is, the better performance of the technique achieves.
E Experiment on Test Problem Suite
E.1 Population Results
In this section, show the results of our proposed method running ZDT, DTLZ, and WFG test suites.
From the selected plots of the final non-dominated solutions obtained by different algorithms shown Figure A7 (full results are in Figures A8 to A9). we can see that the solutions found by our D-PBEMO instances more concentrated the 'golden solution' while the others more scattered. Furthermore, the superiority of our proposed D-PBEMO algorithm instances become more evident when tackling problems with many objectives, i.e., m 3.
The population results of D-PBEMO running on ZDTI~ZDT4, and ZDT6 are shown Figure A8 The population results running on DTI.ZI (m {3, 5, 8, 10}) are shown in Figure A10 rig- ure
AII Figure A12
Figure A13 respectively. The results running on WFG (m = 3) are shown
in Figure A9
The performance metries are shown Table A5 and Table A6
F.2 Parameter influence
In this section, testify the influence of different hyperparamettrrs, including KL threshold and subset number K.
We set in BBEMO-MOEA, to three different values, E {10-1, 10-3 2,100), The experiment results of e* (5) and ((S) can referenced A9 and A10
We set K D-PBEMO- -MDA/D to three different values running on ZDT test problems 10). The experiment results of e*(S) and E(S) can be referenced
(N 100), K
22.
in A1I A13
A15
A17
A19
A12
A14
A16
A18 and A22
F
Experiments on Scientific Discovery Problems
F1
Experiment on Inverse RNA Design
This section shows the experiment result of our proposed method, specifically -SSAMM--NNGAAA Our method and peer algorithms run on Eternal I 000-viennal benchmark? which contains 100 RNA sequences. We selectively run our method on 10 short sequences 36). For inverse RNA
27

---

PERIZAD 20130
11-00011 2U11
0.5
0.5
0,3
04
11
11:
112
D'LEZII
0.2
0.
EARE DT12
LASCCA-B/LIE DELZIGI
IEMO.D DTLRI
Figure A7: Selected plots of the final populations of D-PBEMO instance and peer algorithms. The red star (*) or line represents reference point, the gray lines (-) represent PF, and the black points (+) or lines represent the non-dominated solutions obtained by algorithm.
design problems, the most important task is to predict accurate sequence whose secondary structure the same as target structure, f2 = O. However, in real-world scenarios, we need to tradE-off between stability and similarity. When f2 O, the stability D-PBEMO framework can do more than those singlE-objective algorithms for RNA. Users can guide the search D-PBEMO algorithms and finally reach point which sacrifices little similarity but has better stability. So our experiment are divided into two parts. In the first part, D-PBEMO-NSGA- only focuses finding the most similar solutions, whose reference point locates on f2 0. In another part, user prefers a solution in the middle of PF, f2 E (0, 1). The reference points of our two-session experiments can be referenced in A2I
reach its global optimum in PF. Our
The experiment results of running on reference point 1 are listed in A22 and A23 The predicted secondary structures of each algorithm shown in A14 and A15 Each algorithm repeatedly for 10 times. As we can see from these two tables, D-PBEMO NSGA- II has best performance for times from the perspective of (S) while 6 times from ((S).
The experiment results of running on reference point are listed A24 and A25 As we can see from these two tables, PPPBEMO-NSGA- II has best performance for 8 times with reference to e (S) and E(S).
F.2 Experiment on PSP
In this section we listed the results implementing our proposed method, specifically D-PBEMO-NSGA-TE on PSP problems. We implement RMSD as the performance metric for PSP problems:
Ratom d²
RMSD
i=
(29)
Beatom
28

---

0.5
05
0.5
0122
012
0.2
2- 20713/6
D-7INEEA/S 2012
D-9380600 21112 21
0- ZUTI0
D-2200EUS 20 101
0.5
0.5
0.4
0.4
0.4
0.
0.2
02
111
56
115
02
04
B4
0.
022
04
116
113 04 R3
BJ 98 Rt
1-M0LEA/DDPPNN7 20113(m
AMOOEA//DEESY 40130
D3 201 -
1MOEA D.FOY 2DT
0.5
0.3
11.
3.0
0.4
112
114
0.2
024
800
12.
116 30 01 07 105
L.NGAA-1: 2DT
LNSGA-D/LLE
LNSCAEE/LTE 20T5
1.8804 2DT
Drpiidion
11.0
85
11.5
0.2
6.0
0.5
0.3
2073
1E380/0 20T4/
[E300, 20TW
0.4
0.3
11.
dead
0.4
-0.5
Figure A8: The population distributions of D-PBEMO and peer algorithms running on ZDT test suite
(m 2).
where Ratom is the total number of matched atoms between the two protein structures and d, is the distance between each pair of atoms. The four energy settings and predicted results our proposed method are in Table A26 Other parameters in PSP problems align with [69].
The population results are shown in Figure A16 and the secondary structures are dipicted in Figure The RMSD comparison results are shown in Table A27 As we can see our proposed method have better convergence and accuracy than synthetic problems. This may be caused by two reasons:
The first one is the PSP problem is only conducted 4-dimensional objective spaces, In synthetic problems, our proposed method shows better performance results when dimension 2 3 while peer algorithms may collapse.
The second reason is the formulation of PSP problems. In this paper, we adopt utilizing 4 energy function to represent, which are empirically proved to be more accurate than in 1 I-dimensional objective function[69)
29

---

rESSCA WEGHIN
223024 WPPGE
TEEDGA WEGT
D-SEREA/O WPOINE 50
WEGSGN 41
DAPEE? WEGNI =3
L-BEEA---ELV WEGHIS
D-RDEA-D-PUUP WPNN S1
J.MKA HULTR J
J-NSEAA-H HELTT IOSSOOI
2-N5DA-1/LIII WEGTES
WEGLEN
DEW0yDD
IEM0yD -5)
IEW0/ WEGTIN
Figure A9: The population distribution of D-PBEMO algorithms and peer algorithms running on WFG suite (m
3).
(4)
60C
Figure A10: The population distribution of proposed method (i.e., D-PBMOEA/D) running DTLZ test suite (m 3).
30

---

Table A1: Lookup Table for Key Notations
Notation
Dimension
Description
Rn
n-dimensional decision vector the i-th decision variable m-dimmnnsional objective vector the i-th objective function preference matrix of K arms
I;
R
F Si
R
R
P
RK*K
winning probability of the i-th arm over the j-th arm
Pij
[o, 11
the normalized Copeland score of the i-th arm
C
[o, 11
k*
N
the Copeland winner in K arms constructed winning matrix of K arms in dueling bandits algorithm
B
RK*
number of time-slots when the i-th arm is preferred from the pair of the i-th and the j-th arms
bs.j
[0, 1]
approximated preference probability the i-th arm over the j-th arm upper confidence bound of pi.j lower confidence bound of Ps.j parameter for confidence level total number of comparisons so far expected cumulative regret maximum number of comparisons solution-level preference matrix winning probability of x' is preferred over x
Pis
[0, 1
[0,
IR
t
N+
Rr
R+
T
N+
Pa
R N
[0, 1]
NW
winning time vector of population of solu- tions
N
winning times of the i-th solution
NV
losing time vector of population of solu- tions
N
losing times of the i-th solution upper confidence Copeland score of the i-th solution
3
[o, 1]
winning probability of St subsets & sam- pled from Thompson sampling
00
[0, 1]
winning probability of Š' over S" sampled from Thompson sampling
[0, 1]
R"
feasible set in the decision space population non-dominated solutions the i-th subset of population of non- dominated solutions
finite set
si
finite set
subset of population of non-dominated so- lutions with the highest upper confidence Copeland score
Ca
finite set
Si
finite set
subset most likely covers the SOI subset of population of non-dominated so- lutions with lower-confident winning prob- ability over S'
C
finite set
S"
finite set
subset potentially preferred over Š'
31

---

Table A2: Population size in different problems
Problem
m
N
ZDT
100
64
128
DTLZ
224
10
288
WFG
64
PSP
50
Inverse RNA
100
Table A3: Maximum generation in different problems
Problem
G
ZDT
250
DTL.Z1
500 + 50(m - 2)
DTLZ2
200 50(m 2)
DTLZ3
1000 + 50(m 2)
DTLZA
200 50(m - -2)
WFG
1000 + 50(m 2)
PSP
300
Inverse RNA
250
Punto Fruzt Population
DTT127
DTLI2
DTTL24/0
0.5
0.4
0.5
0.8
0.6
0.2
0.4
0.4
0.4
0.1
0.2
0.2
0
0
abjectices
abjendanca
obycetenca
objectices
(8)
(b)
(e)
(d)
Figure A1l: The population distribution of proposed method (i.e., D D-PBMOEA/D) running DTLZ test suite (m 5)
Purcto Frout Popeiation
DTL21 8)
DTLZ2(m 8)
DTL2Mk -
DTLZA/m B)
0.5
0.3
0.6
0.6
0.4
11.0
0.4
0
24
36
0
44
47
44
5
abjections
obyectives
objoctions
objectioes
(8)
(b)
(e)
(d)
Figure A12: The population distribution of proposed method (i.e., D-PBMOEA/D) running DTLZ test suite (m 8).
32

---

Table A4: The settings of the "golden solution' z" for different benchmark problems (represented in the objective space).
problem
reference point
ZDT1
(0.3, 0.4)
ZDT2
2
(0.2.0.8)
ZDT3
2
(0.15,0.4)
7DT4
2
(0.3,
0.4)
7DT6
(0.9.0.3) 2
WFGI
00.2,0.5.0.)
WFG3
(0.6,0.8,0.8
WFG5
3
(0.3.0.7,033
WFG7
3
(0.7,0.40..)
3
00.3)
0.2.0.1.011.033,0...
DILZI
(011002.0.1.00444001111 0.3.0.00.
(0.02, 0.000..0.. 0.02, 0.03, 0.05, 0.08)
3
00.
0.5)
5
(0.7, 0.6, 0.3, 0.8
DTLZ2
(0.6, 0.5, 0.75, 0.2, 0.55. 0.7
10
(0.3, 0.3, 0.3,0.1,0.,, 0.55, 0.35, 0.35, 0.45)
3
(0.7,0.8, 0.5)
(0.7,0.6, 0.3, 0.5)
DTLZ3
8
(0.6,005, 0.75, U.2, 0.3, 0.55, 0.7, O.6)
10
(0.3,0.3, 0.3, 0.3, 0.1, 0.55, 0.35, 0.35, 0.45)T
(0.7,0.8, 0.5)T
(0.7,0.6, 0.3, 0.8,
DTLZ4
(0.6, 0.5, 0..6, 0.75, 0.2, 0.3, 0.55
10
(0.3, 0.3, 0.3, U.1,0, 0.55, 0.35, 0.35, 0.45)
3
(0.2, 0.3, 0.6)
(0.12, 0.12, 0.77.
DTL.55
(0.04, 0.04, 0.0566, 0.8, 0.113.11.,,.. 0.16, 0.2263, 0.68)
10
0,,0, 0.0096, 0.027,0.082, 0.25, 0.75, 0.08)
3
(0.2, 0.3, 0.6)
5
(0.12, 0.17, 0.24, 0.7)
DTL.76
8
(0.04, 0.04, 0.0566, 0.8, 1.3.0.1...16 0.2263, 0.68)
10
0,,0, 0.0096, 0.027, 0.082, 0.25, 0.75, 0.08)
Population
D/TL,21(>> t
D/TL22( 10)
DTLZ3( 10)
DTTL24( 11)
0.4
0..
0.5
0.8
0.3
0.6
0.0
0.1
0.2
0.2
0.2
objectives
abycetives
objectioes
(6)
(b)
(e)
(d)
Figure A13: The population distribution of our proposed method (i.e., D-PBMOEA/D) running on DTLZ test suite (m 10).
33

---

Table A5: The mean(std) of E" (S) obtained by our proposed D-PBEMO algorithm instances against the peer algorithms.
PROBLEM
m
D-PENSGA-II
D-PBMOEA/D
I-HOEA/D-PLVF
I-NSGA-II/LTR
IEMO/D
ZDTI
2
0.039(2 406-
0.067(1 851-3)
0,054(1 23B-331
0.064(5.99E
0.066c8 831-3)
ZDT2
2
0.1731.71--33:
0.208(1.26B-3))
0.274(1.07-2))
0.3214433B-4)
(294(5,59EE2)
7DT3
2
0.086(5.87E:418
0.098(21178.3)
0.362(4.32F-2)
0.5(4.11
0.117(6993F:4)
ZDT4
2
0.141(1 91E-2)
(1114(788EE-3
0.331(3)938-3)
0.078 34B-3)
0.086(5.37E-3)
0.046(1.766 761.6)
0.055(2.16FF5)
0.106(
0.04((7.62F55)
0.052(5.77F-6))
WFG1
2.2466.44E-D
2.366317E-DD8
210333811B33)
2.2(2.30E-2)
2.48(9.35E-2)
WFG3
L7(L 1)8
L01((4A8E-2y5
0.658(L03E-3)
0.835(2009E-3)
0.743(6.97EE5)
WFG5
74E-1)
1.62(1.3e+000)
2.24(2277-11-
1.59(.86B-211
2.64(2.28E-4)f
WFG7
L.13.78E-1)
1.79(L)8E-1)
1.12(3119--))
1.5(4.39E-4)
L49(4453E-3)t
0.14 481-3)
0.194(3. 11E-4)
0,171(L
04E-3y+
0,178(4.41E-2)
0.171(.11E-1)
0.4-7(220E-2)::
0.318(3.99E-Df
(9)-2))))2))
0,1984.928-33
0,302(3.38E-5
DIL.ZI
0.526 62E-1)
0.512(2.11-D)
0.364(3.348)
0.862))78
0,473(7 (6E-5)
0.5(4.91B-3)5 10
0.25(6.59E-4)
1.20866.88E-3
0,246(4.95E-3)
0.224(2.97E-4)
3
(0237(1.44E-33 3)8
0.213(1.77E:33 77E
0.256(1.038) 0.03 2)
0.177(5.09E-5)
0.176(1 677
2.669/2.78E-240 78E-216
0.507(4.02E-3))
0.504(8.40E-3)
0.5943.744-2)
0.356 34E-4)
DTL22
(.1
0.714(2.53E-3)
0.623(2.87B-2 2yt
1.067744B-3) 3)1
0.647(4.33E 3)F
10
0.675(L007BI)
0.336(4.8EE-3)
0.684(2.87E-2)
0.845(L74E-2)-
0.447(8.80E-3)
0.552(8.88 2)8
0.265(6.54E:33 3)8
1.((242e=00)
0.
0.912(3.99F:
0. 0.772255.80 80E-1)
0.498(7.69E-3))
1,1345 (155.84--1 1)
0.985(5. (88-1)
0.355(6.75E-5)
DTTI33
0.592(1 (8E-1)
037013.39FF33
0786(3777B-21 2yt
2,28(3 64c+00)
0.625(4.35E 3)F
10
0.361(4.24B5 (2)-))
0.385(L01E:2)
0.232(
135((.79E-I)
0.4317.60EE33)
3
0.551 (.:5(99944::000 2)8
0.618(9.0355 2)
09.21 2)
0,474(9.66B
0.553(8. 923388992222 2)
0.877(3 90F-2)
1612(2.75H-22
0.583(2.278:22
116(L155-22
166466.381-40
DIT.74
0.875(2. 1SE 2)8
0.851(1.17:2)
0,807(7442 (217-42E-2)
1.5(2.09E-6
0.835(3.24F
10
0.773(1.88E-11 D)
0.6(2.26E-2)
0.668(1.69E-2)
122(LL
0.556CL.5EE22
3
0.336 22B-5) 5)
0.312(6. 14B-71
0.327(1. 15B-3)
0.312(3.10E.
0.321(5.68E 9yf
(.22668
0.223(1 (909-66)
0.344(4.26F.22)
52E-229
0.23(3.000 (06-6))
DTLZ5
8
0.756(3.11E5 1.11E-1)
0.721 12F-4)
0.8(7.53B-3)
0.736(1.42E-77
10
0.538(7 42IF.
0.581(1.175:
0.779(1 49FF-2y
1.31
1.22(1.377
3
0.444(3. 49E 2)S
0 1.00--3033
0.459 (3333)))33
1.54(1
0.424(7 0.424(7199:4)
0.449(7.5015
20.
0.4(4.59F:2)
17e4001)
DTL.76
8
0.941(S.30E-I)
0.735(3.90) 99E-4)
111 1)*
2.76(5.79E- 1)f
0.741 75E-4)
1.03(3711E1))
0.645(1 1.62
0.889(2.11E-2)
1006
1.29().01B:4))
I denotes proposed method significantly outperforms other peer algorithms according to the Wilcoxon's 's rank sum test at 0.05 significance level;
t denotes the corresponding peer algorithm outperforms proposed algorithm.
34

---

Table A6: The mean(std) of ((S) obtained by our proposed D-PBEMO algorithm instances against the peer algorithms.
PROBLEM
m
D-PBNSGA-II
D-PBMOBA/D
I-NOEA/D-PLVE
MNSGA-II/LIX
IEMO/D
ZDTI
0.0466477-77
0.126(144E-3)
0.146f3.77E-3f
0.162(
0.082(1. 14E-2)
ZDT2
0.173(1.33E-3
0.683(9.54E:2 2)
0.426( 188E-221
(.208(1.644B3)
0.366 233-))
80FF))
0.209(3.26F: 26F..
0.593(4.02E-2)
0.188(2.155) 15F.3) 3y+
0.129(8. 78F:4))
ZDT4
2161(2.29E-2)
(.1666678EE33)
0.145(7.41B5
0.337(2.00B2)
0.11C1 28E-2)
7D)T6
0.052(2.30F: 4)
0.134(4.59E:5)
(.226(1.81F))
0.33(1.21B-39)
0.053(6.69F 6)9
WFGI
2.3
13373.19E-11.
2,08/6 04E-3)
2.17(3733-44
2.52(8.62E-2)
WFG3
1.70.17--13 1)$
1.1 1
1.12(.288-2)
1.27Q.38E-3)
0.74516.44E-5)
WFGS
2.57(3.74E-1)
22e400)
3. 3.14(3. 15E-1)
2
2.64(2.11B-)f
WFG7
2.19(378E-II
L8(L57cH0013
3.33(2.32E-1
2.9(9.81B-3)
L51(4105E-3)
0.223(1.42B:
0.239(2.02B-3)
0.2168711B44ff
0.17299.39E:33
51F-2
(3.91(1122F.4))
(266(3.571
0.305(1 65FE-5)
DILZI
3.38/8.02++01)
0.54(2.00E-4)
50
L69E-22
0.548(1.40E-3)
13.3(5.29e+022) 10
0.209(4.21F 4)
8.87(7.74F. 1y1
0,285(7.48BB3)
0.255(9.3088 30F-4)
0.2544( 225433664--3))
0.416 08E-3)4
63E-3)
(33((1388-3)
0.17611 66E-5)
0.669/2 72E-216
0.626(3. 14E-3)5
0.779(1.32B. (32-22)
0.70113.59E2 53
0.40612 66E-2)
DTLZ2
1.25(2.68E-22 68
0.798(2. 30E:3)
1.03(2.51E:2 2)f
1.12(3.55E-1) 3yt
0.941 2)F
10
(6)().644-1)
0.498())77E-2)
0.968(1.13E2)
0.905(1.41E22)
0.831(4.02E- -2F
0.574(8.00B:2
0.447C
158.00 1.8.0(1.42=+02)1
0.502 2)
1.04(4.56E:
6.03(5. 18c+0114
0.63366.21B 3)2
19.4(7002e-00)
1.46(2655+000)
014292255112
DTL.33
48.5(7.50e+00
0.795(7.39E-3
329 229.018.0 25c+01)
182(1.14+402)
0.903(4.1EE-21
10
1)
0.565(4. (2F:2)
49.7(5. (00
2.25(3.56++00))
0.814(244FF225
0. 0.566(9.57B44
0.716(5.49E-2)
0.834(5. 17B-2)
0.615(499EE-2)
0.563().02E-2)
1.98994883E-21 218
0.684(2.42E2 (2222)
0.777(1.455 45B-2)
1.178. 117(8.18E-3)
0.67416.00E-2)
DTLZ4
1.21(2.31E
0.935C1.03E:22
L,19(4.
2)
L.(0000e-00)
1.0(.23E
10
(2(())54E-1)
1.68332.30E--
0.979(6.08E5 08EE-311
1.26(5699-8)t
(.092(.
0.337 5)8
0.345(1.377 .377 4)d
0.552(5. 70F 3)
0.344(2.33 4)
(.321
1.54(8. 14E-1) Dy
0.261 (9.2944)
0.9099 1 488E-) I)
0.576(1 99E-2)
DTL.75
L.6933 331-1)
0.758( I. 83E-4)
10
0.772(4 92E-3)
0.776(4.04E-8FF
2.3(3.47E-1)
70F.3)
(.992().111B1))
1.79(5.38F-21)
1.22(1.41F.9)
1.54(7. 80E-2yt
0.489(1. 64E-3)t
1881(4499-33
188-22
0.424(7.21E-4
8.99(8.0) 13E-1)8
0.421(2.94B331
1.57(4.90 90E- I)
3,86(2.41e405
0.352(1
DTLZ6
9.57(5.51E-11 1)$
(999:3)
3.79(
3.12(7.00F
0.798(4.12F.4
10
10.5(2.09E-1)
1.08(7.44 1.08(7.47B-2) 47B-2)
1.53(3755E-1) 1)
1518.622--1f
1.3(1.66-3))
t denotes our proposed method significantly outperforms other peer algorithms according to the Wilcoxon's is rank sum 0.05 significance level;
t denotes the corresponding peer algorithm outperforms our proposed algorithm.
35

---

Table A7: The mean(std) of (5) obtained by our proposed D-PBEMO algorithm in distillation experiments.
PROBLEM
D-PBNSGA-I1
D-PEMOEA/D
D-PBEMO-DTS
D-PBEMO-PBO
ZDTI
0.039(2.40E)
0.067(1.85E-3)
0.721(4.88E-3)
0.04(3.53E-6)
ZIT2
0.172(1.718-3
0.208(1.26B-3
0.915(4.78F-2)
0.153(7.07F
ZDT3
2
0.08615 87C-4)
0.098(2.17E-3)
1,16(2.38D-2)f
0.75(4.251-2))
ZDYT4
(.911()
0.114(7.88F-3)
0.146(2.07F-2)
0.107(6.81F-3)
ZDT6
0.046(1.76F.6)
0.055(2.16B55)
0.13(1-418-3)
0.058(2.591-55)
WFG1
2.2416
2.36(3.17E-1)
2.26(8.56E-4)
2.28(2.03E-2)
WFG3
1.7(L
L8E(4448E22)
0.913(3.05E-33
0.926(3.82E-4)
WFG5
1.29(3744-1)
1.62(1.23c+00)
1.72(1.21B33f
1.97(.74B-3)t
WFG7
(3.7318--1)
1.79(1.88-1)
1.33(2.30E-3)f
2.04(2335E-21
0.144( (8E:3)
0.194(3. 11E-4)
0,34(3.26E-3f
0,204(2.32B
0,417(2.20E-2 20B-2)
(318(19999-4
0.572(105E-I)$
0,327(1 93D-5)
DTLZI
0.526 62.2-)1-
0.512(281E-4)
0.5(4.911-3)
(225665911-44
5.99(5.99++01)
0.27555.33DD-2)
0.237(L.74B-3)
0.213(
0.56(1889F-2)
0.17(2 74F:4)
0.669(2.78E-2)
0.507(4.0EE-3)
1,6(L162E-2V
0.528(8.98E-5))
DTLZ2
0.589(3.31822
0.714)2.53E-3)
1.42(9.76E-3ff
0.793(1.73E-4))
(.675(L.078-1)
0.33(4.38F-33
1.13(1.75B-3)f
0.457(1.73F. 6)f
(.352(
(4E-3)))
0,827(1 920-2
0. (.29(88.77E-S)
0.702(5 80E-1)8
0.498(7.60
69E-3)8
2.26(3.21e+00
1351(1.29E-3
DTLZ3
(.5E-))))
O. 0.701))39
133.0(8.89c 89c+03)f
(77093.62E-33)
10
(4.4-11)24
0.38((.011-22)
881.0(5 08)+
0,391(2.83D-3
0.551(9.04F-2)
0.618(9.03F.2)
0.72(3.47R-2))
0.563(1. 20F--))
0.877(3.90E
0.612(275E-22
(88662292E-22)
0.698(8.33E-2
DTLZ4
0.875(2.18F)
0.85((.17E-2)
1.47(4.52F-3))
0.988(3.71E- 1)f
0.773(1.88F-1
0.6(2.26F.2)
1.23(4.61 F--31
0.684(1.12F-4)
0.336(6. 22E-5)
1312/6.14E-7
3392(5.13E-3))
0.328(1.20E-2
0.226(8. 72E-1)
0.223(1.09E-6)
1.65(2.44E-11
0.36(3.38E-5)f
DTL.75
0.756(3. 11--1
0.721(1.12E4)
1.43(2.99E I-f
0.749(4.55E-3
10
(2426))
0.581(..17E22)
0.923(2.1SE-1)5
07803(7.29E-2))
49E-22
0.425(1.06B:
1.39(2.19E-2)
0,421(4.11-33
0. 0.449(7.3 5OC-1)
2.69(6.11E-I
0,383(1 0,383(1.48B-3)
DTLZ6
0,941 (5
0.735(3.99F) 99F
4.64(1 58c+00y
0.798(2.75F-2)
1.03(3711B-)
0.645(1 62E-2)
4.33(6.03E-1)f
1.23(1.12B4))
t denotes our proposed method significantly outperforms other peer algorithms according to the Wilcoxon's 's rank sum test at 0.05 significance level;
$ denotes the corresponding peer algorithm outperforms proposed algorithm.
36

---

Table A8: The mean(sid) of E(S) obtained by proposed D-PBEMO algorithm in distillation
experiments.
PROBLEM
D-PENSGA-II II
D-PBMOEA/D
D-PBEMO-DTS
D D-PSEMO-PED
ZDTI
0.04(6.47E-7)
(11--))
0.73(3.89E-3)
08.61 99F-5)t
ZDT2
0.173(1.73B-3)
0.683(9.54E-2)
0.939(4445E-2)f
0.174(3.00E-3)
ZDT3
2
0.088(5.80F
0.209(3.266)
1.21(6.54E-3)
0.879(3.97E-2))
ZDT4
2
0.161(2.29B-2)
(.166)6.78B-3)
0,231(2.66E-2
0.1387741E5
ZDT6
0.052(2.300E-4)
0.134(4.59B-5)
0.211(4.04E-4f
0.108(2.02E-5)
WFGI
1.31(5.73E-4)$
2.37(3.19E: 1)$
2.32(5.82E
2.21(7.32E-5)
WFG3
1.7(1.17--1)
1.15(1.66-11) 1)$
0,914(3.0SE33)
(.2((838E-3)
WFG5
3
2.57(3.74E-1)
1.72(1.11E-3)
1.69(9.87B-5)
WFG7
2.19(3.78E-1)
1.8(1.7e+000)
1.33(2.00--3)
1.2(2.44-3)
0.223(142B--)
0.143(2.55E-2)
0.393(5.06E-2)
2.1(3.80E-4f
0.56(8.51E-2)
0.391(1.12E4)
2.04(9.56++00)
3.01(185E-44
DTLZI
3.38(8.022 011+00
0.54(2.00F--)
0,571(4.80F-))
10
13.3(5.29c+1))
0.209(4.2 BB-44
0.269(8.22E.)
0.254(3.64D: 64)-3)
0.416(2.08E-3)
0.585(2.20E-21)
0.499(3.41B-3))
0.669(2.72B-2)
162663.14B-3)
1.15(9.99E3)) 99E-3)
(699(3.95E-3
DTLZ2
1.25(2.68E-2)
0.798(2.06-3
1.7(3.08E 3)
1.21(4.87E-5
10
1.65(3.64B-1)
0.498(1.77B.)
1.16(188E-3)f
1.0(187E-44f
1.574(8.00E-)
0.447(1.06E-2)
0,86(1.66E-2)
0.573(1.74E-4)
18++00)))
0.633(6.21E-3)
5.21(3.88++01)
0.672(6.73E-3)
DTLZ3
48.5(7.50c+03)
0.795(7.399 -3
428.002.75++05)
0.798(7.85B)
1.7(3372E-1)
0.565(4.12B-2)
1170.0(4.1ec005)
0.609(1.78E-3))
0.566(9.57E-2)
0.716(5.49B-22
0.746(3.5EE-2)
0.578(
0.989(4.83E-2)8
0.684(2.44 42E-2)
(05(1.94E-2)
0.649(2.17E.4)
DTLZ4
1.21(2.31B-2)
0.935(1.0RE
1.49(8.32F-4)
L02((660FF33f
1.6((4.54--1)
0.683(2.30E-2
1.2(6601E-4)f
0.703(1.72E-4))
0,337(4.33I2
0.345(1.37E-4)
0.404(8.620-)
0.348(4.73D-4)
1.54(8.14E-1
2261(66299-4)
1.74(2.58E-1)
0.261(2.188:4)
DTLZ5
1.69(3.34E-1)
0.758(1.83B
1,543.07E- l)f
0.78(1.82E-3)
2.3(3.47E-1
(.951(4.70B-3)
0.992(2.13E-1))
1.44(2.0EE33)
1.54(7.80E-2)
0.489(1 0.489(1644B3) 64B-3)
161((.54E-2)
0.509(1.91E-3))
8.99(8.03E-1 1)$
0.421(2.94B-3)
3.1(8.25E-1)
0.402((.78E-4))
DTLZ6
9557()
0.798( .999-3)
5.03(1 44c+000)
(80119.81E-)) 1.1((380E-4)
10.5(2.09E-1)
1.08(7.47B22)
4.75(6.14E-1))
I denotes our proposed method significantly outperforms other peer algorithms according to the
Wilcoxon's 's rank sum test at 0.05 significance level;
# denotes the corresponding peer algorithm outperforms our proposed algorithm.
37

---

Table A9: The statistical comparison results of c* (S) obtained by D-PBEMO with different KI
threshold E.
PROBLEM
10
6 10 3
ZDTI
0.0416.35E-7)
0.066(1.76E33
0.04(1.94E-6)
ZDT2
0.188(2021--)
(228())261--)
0.187(1 84E-3)
ZDT3
0.072(1 29E-5)
0.0-3)
0.073(1.9EE-5)
71Y14
0.048(1.71:
0.114(7.8N:33)
0.058(2.90H:-))
ZDT6
0.055(248B-6)
0.055(2.16E-5)
0.04(4.34E77)
WFG1
(22L06E-2)
2.36(3.17E-1)
2.25(4.02E-2)
WFG3
(03(4.74B-2)
1.1((448E-2)
1.03(372BB22)
WFG5
3.7(1. 90e+00)
3,24(1.23eHH)
1.29(1.17++00)
WFG7
3.5+01)3c+00)
3.5+0))
2:96(7.92E-2)
0.18(2
0.194(3.11E- 1IE-4)
0.182(7.28E-5)
0,304(3 951-4)
0.318(3.991-4
9)E-4)
0,313(4.33E-44
DTLZI
0,511(2.67E44
0,512(2.81E-4)
0.516(2099E4)
20)663)
0.258(7.161:4)
0.211(2.04E33
0.213)
0.2188
0.442(4.39E-3 39E-3)
0,507(4.2EE-3)
0.5
DTLZZ
0.757(6.388 38B-3)1
0.714(2.5B1-3)
0,50 84E-3)
0.569( 29F-2)1 2)1
0.3770.226--37
(.22(3))3
0.265)6.54E:3)
3.
0.457 82E-3)
0.498(7.69E-33
0.467(2.76E-3)
DTLZ3
0.775(8. 161-3)
0 3915-3)
355-33
0.576(1.57B
0.38((.018-2)
33
0.645(1.168) 161-1)1
0.618190318-2)
0.4441
18E:1)
0.654(2.14B. 14E-2)f
0.612(2.55
0.666
29H
DTLZZ
(926(1.60E-2))
0.851(..17-22)
0.80(1553--2
(663()))7
0.6(2.26B-2)
02E-2)
0.312(8 36F-7)
700:7)4.708:7
0.223(3.71E-6)
2223L109E-6
DTLZS
0.718(2.03E-5
0.721
0.724( 46E-4)
(1.27133
0.581(.117-22)
0.617
19B-2))
0.42 338E-3
(425(1106H-3)
8:F::
0.339(1 11E-3)
0.33(9 20E-4)
0.341 (311(155333 53E-3)
DTTI76
77733)
0.735(3.99F
0.742(9.50E-5) 50E:5)
0.562(2. 30E-3)
0.663(9244B 0.645(
It denotes our proposed method with this setting significantly outperforms other settings
according to the Wilcoxon's 's rank sum test at a 0.05 significance level.
38

---

Table A10: Statistical comparison results of ((S) obtained by D-PBEMO with different settings.
PROBLEM
10
10
10
ZDTI
0.11316.44E 441-6
(.4003))
0.11(1177-5)
ZDT2
0.293(3.50E-3
54E-2)
0.278(2. 19E-3)
ZDT3
0.22(5.51E:4)f
(.209(3.26F-3)
0.201(3.05E:
ZDT4
0.121CL40E 40E-4)
0.16616.78E-3)
0.127)3.7E--4
ZDT6
0.139(1.1E-43)
0.134(4.59E-3)
0.133(..94E-7)
WKG1
621266662133)
2.37(3.19611)
2,37(3.
19E-1)1
2.27(3-571-22)
WFG3
(.15(1633--)
L15(L.66E-L)
1.12(1 20E-
WIGS
3.72(1 86e+00)1
22700.22-+00)
3.51(1.6eeH00)
WRG7
3.6(1.51++00)
3.6(1.5c+00fF
3.0(1.01-11
0,243(4.500 50E-4)1
(.243(2.05c+022
0.23)())0E-4) (6E-4)
0.394(3.00g 04E-4))
0.39101122-44
0.384(7 81E-S)
DTLZI
0.561(3.85F-4
0.56(2.00FF))
0.555(2.68F:4)
10
0.318(2.49E-4))
0.314)).11E-4)
0.29
0.416(3.42E-3)
0.416(2.08E-3)
0.306(5.66E-4)
(6655332888-))
0.62613. 141-3)
0.627().67
DTLZ2
0.929(7.71BBB))
3.
0.794 20F:3)
0.709(4.3613) 3613-3)
(198(1.7753
1.58(.6411-22
()3-()
0.447(1.06B (6066))))
0.436(7
0.686(4.82E-3)
(.3-)))
0,509 89E-3)
DTLZ3
39
222-3)
0.744(9. (474499.40H-3))
0.565(4.12H--)
0.504(1 32F.
0.753(5.75B) 7575335555--)1
0.716(5.49FF 2)
1664(6636E:2
0.836(L.39E-2)
0.68412 42E-2)
(.758)
DTLZ4
1.09(8. 10E-3)
0.935(1.03EE .33E-2)
0.916(2005E-2)
(44991124FF-2) 2y1
0.683(2.308 30F 2)
0.634(1 5.33
0.359 06E-4)
0.34501.37 37E-4)
0.347(1. 14E-4)
(1301(9.21
0.261 241
()
DTLZ5
1.83(2.45E-2)
0,758(1 1. 83E-4)
0,779(3.25E
0.964(4445B-3)
0,951(4.70B) 70E-3)
0.991(1.12E-2)
(479922922-33)
0,489(1.64)) 64E-3)
0.479(.73E-3)
0.458(3.48 3y1
0.421(2.948
0.439(2.02F3)
DTLZ6
(884433811E33)
0.798( 1. (99)-))
0.821(L75E-22f
1.04 E222)
1.08(7 47E-2)
1.07(2.99E22)
9 denotes our proposed method with this setting significantly outperforms other settings according to the Wilcoxon's rank sum test 0.05 significance level.
Table A11:
The statistical comparison results c" (S) obtained by D-PBMOEA/D with different number of subsets K (m 2).
PROBLEM
K=2
K
ZDT1
2
0.041(9.27E-6)
0.04(3.33-66
(004((1110E55)
ZDT2
2
0.189(1.93--3
0.195(2.67E-3)
0.178(1.55E-3)
ZDT3
2
0.075(6.53E-5)
1.0
0.072(3.79E-5)
ZDT4
2
(0662(7.22E-4
0.054(8 45E.4)
0.075(3.28E-3)
ZDT6
0.054(5.82E-7)
0.054(5. 20E-7)
0.054(5. 18E-7)
$ denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's rank sum 0.05 significance level.
Table A12:
The statistical comparison results of E(S) obtained by D-PBMOEA/D with different number of subsets K (m 2).
PROBLEM
m
K
==
K 10
ZDT1
2
0.111((.08B55)
0.112(2.38D-5)
0.112(1. 528-5))
0.283(1733--3)
0.283(2.13E-3)
0.26912. (TE-3)
ZDT3
2
0.2(2.37E-4
0.208(1000E4)
0.206(3.72E-))
ZDT4
2
0.121(5.99E-4)
0.144(2.85E-3) (5(66688-4
ZDT6
2
0.133(3.58E)
0.13333333E-6 33E-6)
0.135(9.88E-6)
* denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's 's rank sum test at a 0.05 significance level.
39

---

Table A13:
The statistical comparison results of c* (S) obtained by D-PBMOEA/D with different
subsets K (m = 3).
PROBLEM
173
K 2
K === 8
N
WFGI
3
(194F3)))))3)
1,791 30F-4) 4)
18177(152F-3)F
2.26(8.50F:-44)
WFG3
3
0.907(2877-22)
0.942(1.83E-2)
0.813(1.83-22)
0,913(3.05E33)1
WFG5
1.694(2.61E-4)
1.708(3.87E-3
1.716(5.24E-3)
1.72(1211--3)
WFG7
1.3(7.86B-4)
1.324(1.36-22)
1.369(1.00B-2)
1.33(2.0EE-)
DTLZI
0.212(2.11E-4)1
0.187(2.6885-4)
0.184(1.57E-))
0.34(3.26B-3)f
DTLZ2
0.3636.72E-3311
0.224(2.41B-3))
0.213(5.776-3)
0.56(1.
DTLZ3
0,402(4,42E-3Jf
0,247(5 88E-3)
0,262(2.68E-2
0,827(1.926-2)
DTLZA
0.741(7.07F-2)
0.717(6.11-2)
0.699(L077-
0.72(3447-2)
DTLZS
0.313(1.39F-5) 0.312()00E-6)
0.313(L.06F-5)
0.392(5.13E-3)f
DTLZ6
0.417(6.82E-4)
0.418(1.13E-3)
(.413(L.25E-3)
1.39(2.19E-2)+
9 denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's rank sum test a 0.05 significance level.
Table A14: The statistical comparison results of ((S) obtainted by D-PBMOEA/D with different K
(m 3).
PROBLEM
K ==
K ==4
K 8
K = N
WFG1
(.048(1.39E-3)
2.001(3.36B-3
2.026(4.10E-3
2.32(5.82EE44))
WFG3
1.374( 95FF-2)
1.299(2.68F-22)
L043(2.60F-2)
1.914(3055F-3)f
WFG5
2.628(8.61B-3)
2:278(9.09E-3)
2.542(7.15E-
2.72(1.21E-3)
WFG7
3
2.498(7.02B-3)
2.499(4.10E-1)
2.264(7.411- 1)
2.63(2.30E-3)
DTLZI
3
0.286(3.20F:44)
0.234((.18BF-4))
0.214(3.22F-44)
0.393(5.06E-2)f
DILZ2
3
0.678(5.52L--))
0,417(3.00B-3) -01
0,585(2.20E-2) 0.321(4.721:3)
DTLZ3
3
0.67(2.42E-3))
0.454(8.3
0.382(2.11B2)
0.86(1.66--2)
DTLZA
0.857(2.50E-2)
0.775(3.55E-2)
0,742(7.41E-2)
0.746(3.355) (355)))
DTLZS
0,373(2. 14E-4)
0.342(2.98E-)
0.338(7.66 (7.5))55))
0.404(8 62E-3)
DTL.76
0,503(1. 14F-3)+
(.471(.)55F-3)
0.434(1.80F-)
1.61(2.54F-25
$ denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's rank sum test at a 0.05 significance level.
Table A15: The statistical comparison results of (*)) obtainted by D-PBMOEA/D results with different K (m = 5).
PROBLEM
K 2
K
K 16
K= N
DTLZI
0.303 83E.4)
0.317(4.75B-47
0.329(6.42E-44
0.572(1.0SE- 1)f
DTLZ2
0.606(8.00E-3)
0.492(4.14B-3) 14E-3))
0.436(4.15B-3)
1.06(1.62E-2)
DTL
0.624(1.065 26FF-2)
(68F8(6.88F-3)
2.260.21E++00)
DTLZ4
0.724( (.00E2)
0.684(3.20E-2
0.657(3.27B-2)
(16622.9EE-2)+
DTLZ5
0.223(1.62E.4
0.222(7.60E-)
1.65(2444-1I)
DTLZ6
3.33(1.34B-3
0.343(1.55E-3)
0.333(1.19B-33
2.69(6.31E-1))
$ denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's rank sum test at a 0.05 significance level.
Table A16: The statistical comparison results of E(S) obtainted by D-PBMOEA/D with different subsets K (m 5).
PROBLEM
K 2
K 8
K- 16
K K- N
DTLZI
0.418(1.77E-ff
0.393(2.48E-4)
0.373(1.2855 21)
DTLZ2
0.879(4.95E-31
0.627(2776--3)
0.557(3.96E-3)
1.15(9.99 99E-3)f
DTLZ3
0.878(4.22F-3)f
0.624(5.26E-3))
0.578(3.56B-3)
5.21(G.88F+01)
DTLZ4
0.942(7.57E-3)f
0.765(2.15E-22
0.72(2.426-2)
L05(1.944E-2)
DTLZS
0.4(1.88E-3)
0.28(3.81E-4)
0.27().55E-4)
1,74(2.58B-1)
DTLZ6
0.607(5.81E-3)
(42883.25E-3)1
0.39(2.47E-3)
3.1(8.25E-1ff
f denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's 's rank sum test at a 0.05 significance level.
40

---

Table A17: The statistical comparison results of E² (S) obtained by D-PBMOEA/D with different K
(m = 8).
PROBLEM
m
K 2
K 8
K' = 16
K = N
DTLZI
8
4146))
0.50
0:516(2.53F-44+
DTLZ2
8
0.857(4
(.698(9911444)
(.555(
1 (42(9.76B-3)
DTLZ3
0.811(.61E-3)f
0.708(6.81E-3)
0.689(3.39E-3)
133.018 89E+03)
DTLZ4
45
0.841(1.65F-2)
0.806C1 16R--22
(2.5F3))))
DTLZS
0.721(77336-97 0.717(3.77B-5ff 0.714(4611:9)
143(2.89E-1)
DTLZ6
0.714(1 81E-4)
0.742(7.55E-)) 0.732)3,54B-44+
f denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's ' rank sum test at 0.05 significance level.
Table A18: The statistical comparison results of E(S) obtained by D-PBMOEA/D with different K
(m = 8).
PROBLEM
m
K 2
K
K 16
K N
DTLZI
8
0.58(1.52B-4)f
0.563(L.16B-4)
0.554(9 18E-5)
11.8(132EE+03f
DTLZ2
1.101(359E-3ff
0.814(1 24E-33
0.83(1.33E-2)
1,4((3088E33)
DTLZ3
1.0663315F-331
0.831(4.918-33)
0.767(2.58F-3)
428.0(2755++05)
DTLZ4
L1884(.36E-33)
0.947(.12E-2)
0.888()54
1.49(8.22-4)t
DTLZ5
0.83(8.00E-4)f
0.757)..7EE-1)
0.758(L43E-4)
1.54G.07E-Df
DTLZ6
0.9(2.9EE-399
0.802(9.58B-44
0.794(5.72E-44
5.03(1.44++00)
& denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's rank sum test 0.05 significance level.
Table A19: The statistical comparison results of (S) obtained by D-PBMOEA/D with different K
(m 10).
PROBLEM
K
K 6
K 24
K = N
DTLZI
0.223(1888-4)
0.248(4.11E-4f
0.244(6.34B-4))
99905.99++01f
DTLZ2
10
0,537(2.58E-33)
0.437(8.59E-3)
0.412(1441-2)
1.13(1.75B-3)
DTLZ3
10
(.496(3.9E-31)
0.485(1.17E-21f
0.382(6.49E-3)
48
DTLZ4
0.639(1.72E-2)
0.611(3.25E-2)
0.561(2.38E-2
1.23(4.61E-3)
DTLZS
0.885)).188-6)
0.411(3.24E-411
0,597(1 0,597(1
0.923/2.15E-I1 0.659(L43B-2))
DTLZ6
0.527(4.77B-4)
0.562(2.03E-3)f
4.33(6.03E-1)f
+ denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's 's rank sum test 0.05 significance level.
Table A20: The statistical comparison results of E(S) obtained by D-PBMOEA/D with different K
(m 10).
PROBLEM
K =
K =
K ===3 24
K = N
DTLZI
10
0.342(1.41E-))
0.321 60E-4)
0.323(5.62E-4)
A.97(1.29c+02)
DTLZ2
10
0.804(4.06E -3)
0.567(4.54E-3)
0.562(3.63E-2)
1.16(1
DTLZ3
10
0.791(3.31E-31
0.614(L.03E-2
0.544(2.29B-2
15c=05
DTL.ZA
0.934(2.92E-3)) 03901(3.29F-3)1
0.647(3.01E-2
1.25(6.01E-D)
DS
10
(8818(121B2)
1.89688005E-3
0.992(2.
DTLZ6
10
0.978(7.61B33
0.981(5.44E-3)
1.008(1.94B.4
& denotes our proposed method with this K setting significantly outperforms other settings according to the Wilcoxon's 's rank sum test at a 0.05 significance level.
41

---

Table A21: RNA experiment settings.
RNA
Eterna
Target Strueture
Reference Point
Reference Point
Sample Solution
1074756
((I...))
(-63,0)
(-7,,02)
GAUAAAAUAUCAA
12
2
20111
(-9.1,0)
[-13.8, 0.125)
GOOCRGAAAAACCOEC
16
3
997382
((....)(((..))
(-4,0)"
(-8.9,0.6)
CUGGAAAGAGGUEGGGGGG
17
652950
-((((((((...))))))))))
(-12,0)
(-24,0.23)
AAAUGUGAAUGAA
AAAUAUTAUAUAA
26
GCCGCOAAAMGCAAC
727172
(-9,0)7
(-17.70033))
CGAAAAAAAAGOOGC
30
MAAAAAAGGGGGGCGAA
7
477402
(-24,0)
(-31,0.24)
AEGACUCCCCTCCIAAAAAA
34
GAGAAAUCAGAGAAAUUC
997391
(-13,0)7
(-28,0.57)
AGUGAAAGCAGGGAAACU
35
CULGACAAUUAJAMAAAIC
15819
(0000-000--00))))
(-15,0)
(-38,0.88)
AGLCGAUGAAAAAAKAA
UAMAACGAGAAAAACCCA
816170
(-26.7,0)
(-45,0.17)
AGUUUCTAATCCRGGGUUEEEE
36
The column reference point lists our first session experiment (/2 0).
The column reference point lists reference points in the second-session (f2 E (0, 1)).
The sample solution is the possible sequence for the given target structure provided by benchmark.
Table A22: The mean(std) of e' (S) comparing our proposed method with peer algorithms on inverse RNA design problems given reference point
RNA D-PBNSGA-II D-PBMOEA/D
I-MOEA/D-PLVE
I-NSGA2/LTR.
IEMD/D
0.289(0.0)
0.679(U.27)
0.971
0.35(U.05)
0.605(0.21)
0.33(0.01)
1.0((.11)
1.31(1.09)
1.342(1.51)F
1.943(213))
2.454(11.6)
7,89(3.39)
7.102(13.05)
1.332(0.19)
1.819(2.62)
4
4.514(17.76)
2.052(2.08)
1.612(1.62)
1.848(0.86)
2.96(2.05)
5
(.843
5.497(21.22)
7.663(14.46))
7.557(31.47)
6,951 (199ff
1.764(4.51)
4.878(11.1))
5.699(21.21ff
7.656(12.38)
4.1(10.83)f
4.522(7.7)
6.956(11.14)
6.51(16.64)
4.216(12.95)
5.284(13.19)
6.604(26.97)
12.114((3.74)
13.638(28.74)
5.945(15.62)
3.959(15.97)
1445(1.73)
3.656(5.75)
4.506(10.720f
4.424(14.0)
6.731(40.42)F
10
3.988(11.25)
8.506(13.14)
3.86(10.51)
4.983(20.48)
7.128(23.01)
t denotes proposed method significantly outperforms other peer algorithms according to the Wilcoxon's 's rank sum test at 0.05 significance level;
$ denotes the corresponding peer algorithm outperforms proposed algorithm.
Table A23: The mean(std) of E(S) our proposed method with peer algorithms on inverse RNA design problems given reference point
RNA D-PBMSGA-II
D-PBKOEA/D
I-MOEA/D-PLIH
I-NSGA2/LTR
IEMO/D
0.35(0.05)
0.685(0.26)
1.063(0.65)f
0.357(0.03)+
0.605(0.21)#
0.33(0.01)
1.259(1.14)
1.832(1.04f
1.501(1.6)
1.945(2.13)$
3
2.45((11.6)
8.667(2.27)
9.409(2.68)
1.697(0.39)
3.234(8.49)
4
4.514(17.76)
2.569(3.85)
1.758(1.35)
(.849(0.86)
3.226(2.22)
5
2.871(4443)
6.894(15.14)
9.261
8.439(22.87ff
7,206(12.81)
6
1.764(4.51)
5.372(10.74)
6,451(19455f
9.077(8.67)
4.833(8.18)
4.522(7.7)
7.226(17.14)
9.602(10.75)
4.216(12.55)
6.579(10.64)
6.604(26.97)
12.775(24.08)
14.764(25.1))
7.105(13.47)
7.131(1481)
1.46(1.44)
4.94(5.79)
4.965(10.91)f
5.332(11.64)1
7.876(34.82)
10
4.001(11.16)
9.077(17.27)
4.057(9.73)
5.014(20.4)
7.133(22.96)
t denotes our proposed method significantly outperforms other peer algorithms according to the Wilcoxon's 's rank sum test at 0.05 significance level;
$ denotes the corresponding peer algorithm outperforms proposed algorithm.
42

---

Table A24: The mean(std) (S) comparing our proposed method with peer algorithms on inverse RNA design problems given reference point 2
RNA
D-PENSGA-II D-PEMOEA/D
I-MCEA/D-PLVV
I-NSGA2/LTR
IEMO/D
0.459
0.469(0.09)
0.333(0.00)
0.821(0.06)
1.64((133)
0.752(0.54)
0.74(0.22)
1.1
3.143
4.131(1.38)
0.409(0.11)
0.064(0.0)
2,095(254)f
5.037(2.93)F
6.343(5.02f
4
1.459(3.0)
1.72(2.1)
4.25(5.64)
3.702(14.69)
9.43(14.85)f
5
2.247(2.1))
0.84(0.65)
7.28(13.67Vf
5,405(23.99)1
5.934(12.8)f
1.846(2.53)
3.91(2.5)
3.592(10.88)
7.571(19.21ff
7.94(14.85)f
2.612(4.65)
1.518(L.65)
(6601
8.53133114)f
10.78(33.49)
2.1(2.57)
1.388(0.99)
4.953(31.37)
7.202(30.66)
9.016/23.66 6.)
5.559(25.5)
4.071(4.5)
4.276(17.19)
8.706(28.4)f
13.123G1161f
4.25(11.95)
7.42(2.38)
8.94(13.94)t
7.089(16.7)
15.29(49.85)c
i denotes proposed method significantly outperforms other peer algorithms according to the Wilcoxon's s rank sum test at 0.05 significance level;
$ denotes the corresponding peer algorithm outperforms proposed algorithm.
Table A25: The mcan(std) of E(S) our proposed method with peer algorithms on inverse RNA design problems given reference point 2
RNA D-PBNSGA-II D-PEMOEA/D
I-MOEA/D-PLIY
I-NSGA2/LTR
IEMO/D
0.459)0.07)
0,523(0.0))
0.425(0.07)
0.821(0.06)
1.614(1.3)
0.753(0.54)
0.745(0.22)
2.295(1 1.2)
3.143(0.82)f
4.165(1.37)
0.409(0.1T)
0.688(0.03)
3,402(1 19)f
5.072(2.92f
6.874(2.45)f
4
1.462(3.0)
2.51(4.3)
4.393(5.65)f
3.797(14.2)f
9.436(14799)
5
2.247(2.19)
2.918(1.27)
8. R.126(12 126((2.68)f
5.535(23.02)f
7.108(15.5)f
1.846(2.5))
4.442(3.9)
3.851(11.13f
8.347(24.1)f
8.001(14.62)=
2.616(4.64)
4.066(5.7)
3.944(6.39)
8.924(26.44f
11.35(31 19))
2.141(2.47)
3.988(2.57)
9.281(12.21If
7.267(49.92)
10.736(28.62f
5.724(24.65)
5.729(7.45)
4.701(15.3))
8.736(28.2)
14.447(50.17)
10
4.272(12.0)
7.477(2.3)
8.954(13.95)f
7.345(13.88)
15.295(49.80) 86)t
t denotes proposed method significantly outperforms other peer algorithms according to the Wilcoxon's s rank sum test at 0.05 significance level;
$ denotes the corresponding peer algorithm outperforms proposed algorithm.
Table A26: The difference between native and predicted protein in energy
ID
Type
Bound
dDFIRE
Rosetta
RWplus
Native Predicted
431.51
-52.84
293.70
-5059.39
IK36
431.75
.1.66
402.33
3990.52
Native 297.18
-74022
-27.73
-660.188
IZDD
Predicted
328.84
-63.03
63.03
3986.78
Native 269.76
33.51
-10.82
-3113.84
2M7T
Predicted
276.12
22.98
210.47
22111.19
Native
379.04
-104.15
-11.29
-6140.81
3P7K
Predicted
413.47
-91.21
184.17
-3399.93
Table A27: The mean(std) of RMSD comparing our propsoed emthod with peer algorithms on PSP problems.
ID
D-PBNSGA-IT III
D-PEMOEA/D
I-MOEA/D-PLVF
I-NSGA2-LTTR
IEMO/D
1K36
583.20 583.29(117.088
302.82(138.59)
682.23(182.63
597.19(284.911
610.62(402.35
1ZDD
(46.83)542..33)
360.25(17.73)
623.14(394.14)
450.23(582.19)
488.28(518.4))
2M7T
350.51(8.95)
477.56(76.93)
671.45(372.01)
721.73(502.31)
823.46(1023.54)
3P7K
719,90(
(0090001122.992 (202.92)
152.53(7.45)
663.29(802.99
692.31(823.15
818.93(923.88 87))
3VIA
687.07 .33)
584.43(28.34)
887.68(391.74)
191.13(304.22)
823.28(528.87)
43

---

Torget
D-PENSGA-II
D-PRMODA/D
T-MORA
UNSGA-I/LLTR
TEMO,D
2
MFE 10.3.
MPE 10.4 0.57
MFE 11.2. 0876
MFE
10011
MFE 10.1.
63..
MFE 24.1. 0.2
MPE 26.7 0.8
MFE 202. 0,77 MFE
21,00 07T MFD 2212,0 0.7
MFE -26.0-0002
MFE 0.
MFE 117..0 0.47
MFE 5.5. MFD 23.6
The comparison results of PBEMO against the other three tat-offtheeann PBEMO algorithms on inverse RNA design problems. particular, the target structure is sample of possible solution represented in blue color while the predicted one obtained by different optimization algorithms are highlighted
Figure A14:
color. In this graph, the reference point set as a 1. The closer is to 1, the better performance achieved by the corresponding algorithm. When the share the same biggest value, the smaller M E the better the performance is.
44

---

RNA
Tirget Structure
IEPRNSGA-II
I&PRMORA/D
EMORA//>PINT
ENSGA-TIILR
TRMO;
MFR 26.1, 0.94
MFF -2.T,
0.88 MFF
-35., - 0.6 MFE
-273, 0.74 MFE
-2090 0.87
2 088
28.. 065 - 355 0.77 154, 0.82
MFF 244, 06
MFE 28.1 0.50 IFF -35
031 MFF
13.1, 0.72 MFR 12.2 0.51
MFE - 0.5
MFB - 078 MFE -26., 0.58 MPE
2600 0.88 MFB
-318, a 0.53
10
-33.2
-318,
-3.2,
-3..0
33.00
The comparison results of D-PBEMO against the other three state-of-the-ar PBEMO algorithms on inverse RNA design problems. In particular, the target structure is sample of possible solution represented blue color while the predicted one obtained by different optimization algorithms are highlighted in red color. In this graph, the reference point is set as a 1. The closer is to 1, the better performance achieved by the corresponding algorithm. When a share the same biggest value, the smaller M F E the better the performance is.
Figure A15:
45

---

Paplick
Natie Exery
D-P01D24-12 LaatDin 4)
2-FER0OI-11 1ic3jje
D-PEIOOL- 2MTTE00
D- JVSAmm 6
D-728504-12 Te/t//s
Tatee (00911
Vesta 2000
Boees
Dippr Boaa
Routta Diastiins
BB Hlene
dIF-E Rostla
Empiie Eated
dBl
Pacile WWyle
(Ajertive
Disectioos
3-99KDRA,0 1200//
SVIA
D-POWSTA/3 JPKkj
Volwi-1100
Vollant 1000)
Voohec (OMR)
25
Bond JESBR Roac
RWylas fland
IFFIE
Bosta
blunt
dielix Roetle WWWller Bore:
4UF-RE
Hos
EW Blod
JEFIS
Roeeta RWy
ESUURA/DUEY 12 0
AMEDRAAD-PLL PLWE Jelkoo i
DAUDEA ANTT 4
AMIDEAAEE-PA 3V1A/ m
Valoo 1000)
Vallec LOND
ffand
Claccino
ANSCA-IL/LTR
1890
I/LIR JV1Afe 4)
durin 3Ph// 4)
Voluet (000)
Skai de3Bi Roc
BRL Okgedtene
Boedia
Bigaa Bhend dPFIRE Rostta
RW Munn
4F Boseto
Emples Fard
AIFRE Roitta
1E30// LEUGRr
IEADU/D
3P7K/sm
Tatoee 0900
Falus LOOG
(Pjection
IBI (ljechess
100 bximl
Houtte
Chyedias
Hosett
400 BLS
Paantaa
Figure A16: The population distribution of D-PBEMO and peer algorithms running PSP problems (m 4).
46

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the computational efficiency of proposed algorithms and how scale with dataset size. Possible Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
47

---

Answer: [Yes]
This paper includes theoretical results in Appendix B. B and theorems, formulas, and proofs in the paper numbered and cross-referenced. All assumptions are clearly stated or referenced in the statement of any theorems.
Justification:
B.3
All
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The steps of our proposed algorithms are described step by step. The repro- ducibility of experiments can be accomplished in various ways.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
48

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: This paper includes experiments requiring code. Code link is provided in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.ccppubcccguuuceeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The vital experimental settings are presented in the core of the paper, Section And the full details can be referenced in D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiment results are accompanied by statistical significance tests. The statistical significance tests are introduced in D.3 The significance level settings are also included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
49

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: This paper doesn't 't indicate the type of compute wokers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. cctpiiipppccpp/ciiiu http://wwwppp.cppp.oo..ccccccaaa
Answer: [Yes]
Justification:
The authors make sure preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper is foundational research and not tied to particular applications, let alone deployments.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
50

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
51

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
52

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
53


Reviews:
[{"review_id": "VTbgJoHcpC", "reviewer": "Reviewer_PNDi", "summary": "One challenge and potential advantage of multi-objective optimization (MO) is to adapt the dynamic human preferences while outputting an optimum. Although Preference-based evolutionary MO (PBEMO) is a promising framework, current approaches are inefficient and may not interpret the decision makers' true intentions accurately. One reason is that the decision makers' true intentions were not precisely \"expressed\" and \"acknowledged\" by the predefined reward model. This paper, intending to solve this reason, designs a framework that directly leverages human feedback, using a clustering-based stochastic dueling bandits algorithm.", "questions": "In Section 3.2 \"Preference Elicitation Module\", why is Equation (4) the resulted mixture distribution? If it was computed, were there any justifications that this shall be the result? More explanations on this may be appreciated because, to the best of my understanding, this part plays a critical role in justifying the framework since it directly works on the preferences.", "limitations": "Yes.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper tackles an interesting and important problem, and the method is novel. In particular, it is certainly valuable to directly express the human feedback into the framework, as human feedback is the critical information for preferences outputs.\n\n2. The application of multiple arms and Copeland winner as the main decision-making criteria indeed makes sense to me. Although different, it resembles the rank computation in many RLHF schemes to some extent.", "weaknesses": "Due to my very limited knowledge in this area, I might not be able to find valuable weaknesses in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Q1: Is the Gaussian mixture model computed**\n\nThank you for this question. First of all, the Gaussian mixture model is not computed. Instead, it is a model assumption which we contend to be reasonable to represent user preference distribution within the solution of interest (SOI) region. We justify this from the following three aspects.\n\n1. A key assumption of using the Gaussian mixture model is that user preference distribution within the SOI region follows a Gaussian distribution. However, we do not enforce such Gaussian distribution assumption to the other areas out side of the SOI region.\n2. Theoretically, a Gaussian mixture model can simulate any distribution. Therefore, using a Gaussian mixture model to simulate user preference distributions is more general compared to other methods for expressing preferences, such as Gaussian process for ranking used in Bayesian optimization [1].\n3. In addition, Gaussian mixture model is potent to be used to derive the uncertainty of the preference elicitation model.\n\n[1] Zintgraf, Luisa M., et al. \"Ordered preference elicitation strategies for supporting multi-objective decision making.\" *AAMAS'18*: 1477-1485, (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdnFyDbz9C", "reviewer": "Reviewer_hDtX", "summary": "Preference-based evolutionary multi-objective optimization (PBEMO) methods involve optimization (explore the space), consultation (learn human preference), and elicitation (guide evolutionary search). Existing PBEMO methods may suffer from inaccurate reward models, which is likely to happen given that human feedback exhibit a lot of randomness. The authors propose to directly leverage human feedback without using a reward model to guide the evolutionary search. Specifically, given that human feedback from relative comparison is better than absolute labels, the authors employ dueling bandits to compute preference metrics.", "questions": "Could the authors provide some insights on the advantage of starting with a coarse-grained representation, which may yield an initially inaccurate SOI, compared to having a set of Pareto-optimal candidate solution upfront?", "limitations": "The limitations are not clear. The scope of the claims should be discussed. I encourage the authors to create a separate limitations section (see NeurIPS paper checklist).", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors prove the regret bound of proposed algorithm, which is better than that of Thompson sampling based dueling bandit algorithms. Empirical evaluations on 33 settings showcase the effectiveness of proposed approach.", "weaknesses": "In step 1 of consultation module, one needs to choose K, which is the number of subsets to partition S into. The choice of K would greatly affect how close the solutions are within a subset. Thus, it is important to discuss methods to choose K and provide justifications. \n\nThe runtimes are not reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1: Choice of $K$**\n\nWe agree with the reviewer that $K$ can impact the crowdedness of solutions within a subset. In particular, the larger the $K$ is, the smaller the distances between solutions within each subset are. As for the choice  and sensitivity of $K$, our justifications are presented in the **Author Rebuttal** attached with a PDF containing additional experiment results.\n\n**Response to W2: Runtimes**\n\nBecause we are not sure whether the *runtimes* referred by the reviewer is the CPU wall clock time (actual execution time) or the run time analysis used to analyze the convergence of an evolutionary algorithm, we provide discussions on both aspects.\n\n1. In terms of CPU wall clock time, our D-PBEMO algorithms are the fastest one in the experiments. However, we are concerned about whether it is a fair comparison if we report the CPU wall clock time results. This is mainly attributed to the programming languages used to implement different algorithms. In particular, our clustering-based stochastic dueling bandits algorithm is implemented in C++ and it runs super fast (usually within $2$  seconds). In contrast, the other peer algorithms are mainly implemented in Python. Therefor, they are often much slower than ours. Further, algorithm like I-NSGA-II/LTR [1] involves neural network training that can even make itself very slow (often over $10$ minutes).\n\n2. As for the run time analysis, while there have been some attempts on the evolutionary multi-objective optimization (EMO) algorithms (e.g., [2]), it is far from mature compared to the rich literature for single-objective optimization, not to mention the preference-based EMO. Further, the run time analysis highly depends on the baseline algorithm while our D-PBEMO framework is EMO algorithm agnostic. In this paper, our key theoretical contribution is the regret bound for the preference learning part (i.e., the consultation module) when considering multiple conflicting objectives. We believe this contribution is original and will be valuable to multiple communities including but not limited to preference learning, multi-objective decision-making, and EMO. As part of our future works, we plan to further analyze the complexity for EMO when using the preference learned from our consultation module.\n\n[1] Li, Ke, et al. \"Interactive evolutionary multiobjective optimization via learning to rank.\" *IEEE Transactions on Evolutionary Computation* 27(4): 749-763, 2023.\n\n[2] Bian, Chao, et al. \"A General Approach to Running Time Analysis of Multi-objective Evolutionary Algorithms.\" *IJCAI'18*: 1405-1411, (2018).\n\n**Response to Q1: Advantage of starting with coarse-grained representation**\n\nWe respectively check whether the reviewer is concerned about the statement in our *Remark 1*. If this is the case, we address the reviewer’s question from the following three aspects.\n\n1. In EMO, solutions at the early stages of the evolutionary search are often far from the Pareto-optimal front (PF). Further, the search direction of EMO is not deterministic. Instead, given the diversity and spread requirements of an evolutionary population, the search direction is rather stochastic in the early stages. Therefore, it can be misleading and noisy if we ask decision-makers to elicit their preferences regarding such solutions.\n2. On the other hand, we do not intend to wait until the end of EMO, i.e., when obtaining a set of solutions well approximate the PF (this also corresponds to the reviewer mentioned \"$\\cdots$ *a set of Pareto-optimal candidate solution upfront*\"). This represents the posteriori decision-making whose drawbacks are discussed in Appendix A.1. In particular, it is hard to guarantee that we can have the solution(s) meet the decision-makers’ preference upfront given the dense characteristics of PF. Therefore, it is a paradox if the underlying EMO algorithm ends up with no solutions lying in the region of interest. This is not uncommon since the range of PF can be too huge to have a full coverage when having many objectives. Differently, since preference-based EMO is an optimization-cum-decision-making process, it is designed to search for the solution of interest interactively, rather than approximating the whole PF.\n3. Our strategy in D-PBEMO is to start consulting decision-makers neither too early nor too late as justified in the above two bullet points. This is because: 1) such coarse-grained PF sufficiently represents the PF shape, and 2) the search direction of an evolutionary population is largely determined. In this case, we believe decision-makers can already elicit meaningful preferences regarding such solutions. This hypothesis is also empirically validated through our experiments, i.e., our proposed D-PBEMO instances outperform the other peer algorithms. As some experiments in [3, 4], it is suggested to elicit decision-makers' preferences in the later half of an EMO process. Here, to make our comparison fair, we heuristically set the timing for consulting decision-makers as the middle of EMO for all peer algorithms. \n\n[3] Lai, Guiyu, et al. \"Empirical studies on the role of the decision maker in interactive evolutionary multi-objective optimization.\" *CEC'21*: 185-192, (2021).\n\n[4] Marquis, Jon, et al. \"Impact of number of interactions, different interaction patterns, and human inconsistencies on some hybrid evolutionary multiobjective optimization algorithms.\" *Decision Sciences* 46(5): 981-1006, 2015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k01GcjW2UO", "reviewer": "Reviewer_FUFf", "summary": "This paper focused on the problem of multi-objective optimization in the dueling bandits settings.  A clustering-based stochastic dueling bandits algorithm was developed and analysis. The performance is further validated via experiments.", "questions": "See weakness above.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- A new framework via combining dueling bandits and evolutionary multi-objective optimization was presented. \n- The proposed algorithm has been applied to real-world problem, i.e., protein structure prediction, which is interesting and promising.", "weaknesses": "- In dueling bandits literature, one of the most widely assumed winner or the most general winner is the Condorcet winner. In this paper, for example, in definition 2.1, the authors defined the Copeland winner. What is the intuition behind this or in other words, does your framework requires this specific winner, or can it be generalized to the general Condorcet winner (or other winners such as Borda Winner, Neumann Winner that are widely used in the dueling bandits literature)?\n- In your algorithms and theorem, $\\alpha>0.5$ is assumed. Is this a natural option in practice? \n- The proposed D-PBEMO framework consists of three modules as shown in Figure 1(a), while the regret is characterized for the consultation modules. Not sure if I miss anything, can this regret be claimed as that of D-PBEMO? \n- The writing of the paper can be significantly improved. It is hard to follow the paper with many mathematical notations not rigorously defined, and some details are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to W1: Choice of winner**\n\nWe address the reviewer’s concerns from the following three aspects.\n\n1. The Copeland winner is more universally applicable than the Condorcet winner. The Copeland winner includes the Condorcet winner. That is to say the Condorcet winner is always a Copeland winner, whereas the Copeland winner is not necessarily a Condorcet winner. In some scenarios, a Condorcet winner may even not exist, but a Copeland winner always does. \n2. Following the first bullet point, in our multi-objective optimization scenario, a Condorcet winner may not exist. Specifically, user preferences, which are represented as reference points or golden points in our experimental settings, may not be reachable because sometimes user preferences lie beyond the PF (see some examples in Figure A7 on the ZDT3 test problem in our Appendix E.1). In such cases, the non-dominated solution set found by the evolutionary multi-objective optimization (EMO) algorithm may contain multiple solutions that are the same closest to the golden point. These solutions are optimal arms and are Copeland winners but not Condorcet winners. Because the Condorcet winner should be unique by definition, the Condorcet winner is not suitable for our multi-objective optimization context.\n3. As for the reviewer mentioned two other winners, our justifications are as follows.\n- For the Borda winner, it belongs to the family of positional voting where the preferences are elicited as a full rank regarding all solutions. As reported in [1], such preference elicitation method can be cognitive intensive for humans. Note that preference-based EMO itself is an optimization-cum-decision-making process, which involves multiple runs of consultations with humans. Therefore, one of our design principles is to reduce the human's cognitive load as much as we can during each consultation. Given this justification, we think the Borda winner can in principle be applied in our consultation module, but it is not recommended, at least not as good as the dueling bandits using pairwise comparisons in our D-PBEMO.\n- For the Neumann winner, it is designed for scenarios involving potential clones of arms [2]. However, because each subset (i.e., arm) is different in our D-PBEMO, the preference elicitation module will suspend preference learning before subsets become clones. Further, the Neumann winner is originally designed for contextual dueling bandits, it is not directly applicable to our EMO context which is stochastic.\n\nGiven the above justifications, we believe our choice of Copeland winner in our current version of D-PBEMO is rational.\n\n[1] Zintgraf, Luisa M., et al. \"Ordered preference elicitation strategies for supporting multi-objective decision making.\" AAMAS'18: 1477-1485, (2018).\n\n[2] Dudík, Miroslav, et al. \"Contextual dueling bandits.\" *Conference on Learning Theory*. PMLR, 2015.\n\n**Response to W2: $\\alpha$ setting**\n\nWe confirm to the reviewer that $\\alpha > 0.5$ is a natural option in practice. The requirement $\\alpha > 0.5$ in the double Thompson sampling proof can be traced back to the RUCB paper [3]. This parameter range ensures the exploration capability in Thompson sampling or UCB processes, which is a fundamental assumption to guarantee the algorithm’s convergence.\n\n[3] Zoghi, Masrour, et al. \"Relative upper confidence bound for the k-armed dueling bandit problem.\" *ICML'14*: 10-18, (2014).\n\n**Response to W3: Regret analysis**\n\nWe confirm that the reviewer understood well. The regret analysis is for the consultation module, i.e., the clustering-based stochastic dueling bandits algorithm used for preference learning. We justify this further from the following three aspects.\n\n1. The regret represents the uncertainty in the consultation module. In this paper, we are not yet to provide an uncertainty quantification for the D-PBEMO algorithm. This is mainly attributed to the stochastic nature of the reproduction operators (i.e., crossover and mutation) which introduce additional uncertainty difficult to quantify.\n2. Further, we believe our regret analysis for the preference learning in the context of preference-based EMO is an original contribution, which has not yet been addressed so far to the best of our knowledge.\n3. Note that our proposed D-PBEMO framework is algorithm agnostic. That is to say any EMO algorithm can be applied with minor modification in our optimization module. In this paper, we applied two most influential EMO algorithms as the baseline for a proof-of-concept purpose. Because the convergence property highly depends on the algorithmic behavior of the underlying EMO algorithm (partially justified in our first bullet point), it is hard to provide a universal convergence analysis applied for all EMO algorithms. Yet, we also believe this is part of our future endeavours.\n\n**Response to W4: Writing of the paper**\n\nWe apologize for our presentation when tackling many mathematical notions. In the camera-ready version, we will take two actions to improve the readability of this paper.\n\n1. Carefully double check all mathematical notions used in this paper and prepare a lookup table at the beginning of the Appendix.\n2. We will also augment some important definitions and preliminary knowledge statement about dueling bandits to be more self-contained.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFSIDTdkA1", "reviewer": "Reviewer_dDig", "summary": "This article focuses on the preference-based evolutionary multiobjective optimization. First of all, such problems are widely found in real engineering application scenarios, thus becoming one of the most popular research directions in the field of multi-objective optimization. \n\nOverall, in terms of the presentation of the article, this article is well-structured and clearly expressed. From the aspect of methodological design, this article is somewhat innovative. However, from the aspect of experimental design, this article seems to have some defects.", "questions": "1-In Section 4.1, I am curious as to what criteria the authors used to select the benchmark problems. For example, why didn't the authors choose the same series of benchmark problems as DTLZ7, WFG2, WFG4, etc.?\n2-In Section 4.2, I am concerned that the comparison algorithm used in the experiment is SOTA in the domain.\n3-Two variants, DPNSGA-II and DP MOEA/D, were designed in the article. However, the discussion between these two variants seems to be a bit thin. What are their respective strengths and weaknesses?\n4-Similarly, I found the discussion in the experimental section to be overly simplistic, seeming to restate the results of the experiment at face value. But it doesn't analyze the reasons behind the results too much.\n5-Population size, number of search iterations, and other such parameter settings do not seem to be given by the authors.", "limitations": "The experimental chapter needs to be further upgraded.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The designed method aims to address the issues exists in both consultation and preference elicitation modules. \n\nTo be specifically, the authors design a clustering-based stochastic dueling bandits algorithm to overcome the problem of convergence due to a large amount of preference comparisons. Additionally, the authors applies the Gaussian mixture distribution to leverage the preference learned from the consultation session.", "weaknesses": "1-In Section 4.1, I am curious as to what criteria the authors used to select the benchmark problems. For example, why didn't the authors choose the same series of benchmark problems as DTLZ7, WFG2, WFG4, etc.?\n2-In Section 4.2, I am concerned that the comparison algorithm used in the experiment is SOTA in the domain.\n3-Two variants, DPNSGA-II and DP MOEA/D, were designed in the article. However, the discussion between these two variants seems to be a bit thin. What are their respective strengths and weaknesses?\n4-Similarly, I found the discussion in the experimental section to be overly simplistic, seeming to restate the results of the experiment at face value. But it doesn't analyze the reasons behind the results too much.\n5-Population size, number of search iterations, and other such parameter settings do not seem to be given by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to weaknesses and questions**\n\nWe address the reviewer’s concerns one by one as follows.\n\n1. The choice of benchmark test problems follow the existing literature [1] and it also adheres to the criteria outlined in [2,3]. This ensures that the chosen benchmark test problems have different Pareto-optimal front (PF) shapes and various characteristics (e.g., multiple local optima and concave/convex PF, etc) while we omitted some benchmark problems having duplicated properties. In addition, we included more real-world problems (protein structure prediction and RNA inverse design) to promote AI4Science.\n    \n    We do not consider DTLZ7 and WFG2 because they are with disconnected PF segments. As discussed in [1], it is difficult for decision-makers to elicit their preferences in the disconnected regions, which also represent infeasible parts. Further, since WFG4 shares the same characteristics with WFG7 (i.e., separable, unimodal and with a concave PF), we consider WFG7 in our experiments.\n    \n2. In particular, according to the experimental results reported in a recent paper [4], we chose three most competitive algorithms (i.e., I-MOEA/D-PLVF, I-NSGA-II/LTR, and IEMO/D) as our peer algorithms. We believe the chosen algorithms represent the current state-of-the-art in preference-based evolutionary multi-objective optimization (EMO). Further, to demonstrate the flexibility of our framework, we chose a preferential Bayesian optimization [5], which was designed only for single-objective optimization problems, as a baseline and adapt it to our D-PBEMO framework by replacing its preference learning part by our consultation module. As the results shown in Section 4.3, this variant also works well for tackling multi-objective optimization problems.\n\n3. Because our proposed D-PBEMO framework is algorithm-agnostic. That is to say any existing EMO algorithm can be used in the optimization module with minor adaptation. For proof-of-concept purpose, we choose NSGA-II and MOEA/D, two influential EMO algorithms in the literature, as two instances. As reported in the EMO literature [6], NSGA-II performs better on 2-objective problems, while MOEA/D scales well to many-objective problems ($m\\ge3$). This characteristic is retained in our D-PBNSGA-II and D-PBMOEA/D algorithms.\n\n4. We apologize for the succinct discussion on the experimental results. This is partially because our proposed D-PBNSGA-II and D-PBMOEA/D have shown constantly superior performance in almost all comparisons, as well as the strict page limit. We promise to strengthen this part in the camera-ready version. Specifically, we plan to focus on discussing the different characteristics of D-PBNSGA-II and D-PBMOEA/D for handling low- and high-dimensional problems, respectively. These characteristics are related to the internal mechanisms of of NSGA-II and MOEA/D.\n\n    i) NSGA-II is a representative algorithm based on Pareto dominance in its environmental selection. Since its diversity maintenance strategy mainly depends on the distances among solutions, it is relatively robust and fast to find a reasonably good Pareto-optimal front (PF) approximation. However, because NSGA-II relies on Pareto dominance, it does not scale well to many-objective problems.\n\n    ii) In contrast, the search of MOEA/D is mainly determined by the weight vectors. It is naturally more scalable to many-objective cases.\n    \n    The above properties are all reflected in the performance of D-PBNSGA-II and D-PBMOEA/D.\n\n5. All experiment parameter settings, including population size, number of search iterations, and other parameter settings, are provided in our Appendix D.2.\n\n[1] Li, Ke, et al. \"Does preference always help? A holistic study on preference-based evolutionary multiobjective optimization using reference points.\" *IEEE Transactions on Evolutionary Computation* 24(6): 1078-1096, 2020.\n\n[2] Huband, P. et al. \"A review of multiobjective test problems and a scalable test problem toolkit,\" in IEEE Transactions on Evolutionary Computation, 10(5):477-506, 2006, doi: 10.1109/TEVC.2005.861417.\n\n[3] Zapotecas-Martínez, Saúl, et al. \"A review of features and limitations of existing scalable multiobjective test suites.\" *IEEE Transactions on Evolutionary Computation* 23(1): 130-142, 2018.\n\n[4] Li, Ke, et al. \"Interactive evolutionary multiobjective optimization via learning to rank.\" *IEEE Transactions on Evolutionary Computation* 27(4): 749-763, 2023.\n\n[5] J. González, et al. \"Preferential bayesian optimization.\"  In ICML’17: Proc. of the 34th international conference on Machine learning, volume 70, pages 1282–1291. PMLR, 2017.\n\n[6] Zhang, Qingfu, and Hui Li. \"MOEA/D: A multiobjective evolutionary algorithm based on decomposition.\" *IEEE Transactions on evolutionary computation* 11(6): 712-731, 2007.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to reviewer hDtX about the choice of $K$
We agree with the reviewer hDtX that $K$ can impact the crowdedness of solutions within a subset. In particular, the larger the $K$ is, the smaller the distances between solutions within each subset are. As for the choice of $K$, we justify this from two perspectives:

1. As shown in Theorem 3.3, $K$ is involved in the regret of our clustering-based stochastic dueling bandits algorithm. A larger $K$ results in a larger uncertainty in preference learning for the same number of comparisons, leading to a slower convergence rate. However, it also narrows the final region of interest (ROI). There is no definitive guideline for selecting $K$ for different problems. Users can adjust $K$ to an appropriate value based on the accuracy of preference learning.
2. Further, we have conducted a sensitivity study on $K$ for $2$-objective problems in Section 4.4. In particular, we set the population size as $100$ and $K\in\{2,5,10\}$. From the results therein we can see that our proposed D-PBEMO framework is not sensitive to the choice of $K$. In addition, we have conducted additional experiments on problems with more objectives. The results can be found in the attached PDF file. 
As the dimensionality increases, the population size will increase. Generally, with larger populations, a higher $K$ tends to yield better results, aligning with our intuition. Furthermore, our significance analysis across 20 repeated experiments reveals that the optimal $K$ does not show significant differences in performance. 
In summary, $K$ does not significantly impact the performance of our proposed D-PBEMO framework. For most problems, we do not recommend choosing a very small/large $K$ (e.g., $K=2$, $K=N$), as it may inefficiently narrow down the ROI.

Meta-review: This paper focuses on the application of dueling bandits to evolutionary multi-objective optimization. In particular, reference-based evolutionary multi-objective optimization (PBEMO) methods involve optimization (explore the space), consultation (learn human preference), and elicitation (guide evolutionary search). Existing PBEMO methods may suffer from inaccurate reward models, which is likely to happen given that human feedback exhibit a lot of randomness. The authors propose to directly leverage human feedback without using a reward model to guide the evolutionary search. Specifically, given that human feedback from relative comparison is better than absolute labels, the authors employ dueling bandits to compute preference metrics and to choose the winner using the Copeland score.

In summary the authors have done a great job explaining the unclear parts during the rebuttal phase, convincing several reviewers to raise their final scores. While the paper does not contribute that much to the core ML literature, the application of dueling bandits to the multi-objecive optimisation domain is significant.
Decision: Accept (poster)