Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandits

Authors: tian  huang
shengbo  wang
ke  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:27:17

Abstract: The ultimate goal of multi-objective optimization (MO) is to assist human decision-makers (DMs) in identifying solutions of interest (SOI) that optimally reconcile multiple objectives according to their preferences. Preference-based evolutionary MO (PBEMO) has emerged as a promising framework that progressively approximates SOI by involving human in the optimization-cum-decision-making process. Yet, current PBEMO approaches are prone to be inefficient and misaligned with the DM’s true aspirations, especially when inadvertently exploiting mis-calibrated reward models. This is further exacerbated when considering the stochastic nature of human feedback. This paper proposes a novel framework that navigates MO to SOI by directly leveraging human feedback without being restricted by a predefined reward model nor cumbersome model selection. Specifically, we developed a clustering-based stochastic dueling bandits algorithm that strategically scales well to high-dimensional dueling bandits, and achieves a regret of $\mathcal{O}(K^2\log T)$, where $K$ is the number of clusters and $T$ is the number of rounds. The learned preferences are then transformed into a unified probabilistic format that can be readily adapted to prevalent EMO algorithms. This also leads to a principled termination criterion that strategically manages human cognitive loads and computational budget. Experiments on $48$ benchmark test problems, including synthetic problems, RNA inverse design and protein structure prediction, fully demonstrate the effectiveness of our proposed approach.

Main paper: 
 introduction: 
 Multi-objective optimization (MO) represents a fundamental challenge in artificial intelligence [49] , with profound implications that span virtually every sector-from scientific discovery [67] to engineering design [22] , and societal governance [23] . In MO, there is no single utopian solution that optimizes all objectives; instead, the Pareto front (PF) comprises non-dominated solutions, each representing an efficient yet incomparable trade-off between objectives. The goal of MO is to assist human decision-makers (DMs) in identifying solutions of interest (SOI) that optimally reconcile these conflicting objectives according to their preferences. This field has been a subject of rigorous study within the multi-criterion decision-making (MCDM) community [43] for more than half a century. Over the past two decades, we have witnessed a seamless transition from purely analytical methodologies to a burgeoning interest in interactive evolutionary meta-heuristics, known as preference-based evolutionary MO (PBEMO) [39] .As in Figure 1 (a), a PBEMO method involves three building blocks. The optimization module uses a population-based meta-heuristics to explore the search space. The preference information is progressively learned by periodically involving human DM in the consultation module to provide preference feedback. The learned preference representation is then transformed into the format that guides the evolutionary search progressively towards SOI in the preference elicitation module. The overall PBEMO process is DM-oriented and is an optimization-cum-decision-making process. While PBEMO has been extensively studied in the literature in the past three decades (e.g., [44, 15, 8, 36, 59, 38] ), there are several fundamental issues unsolved, especially in the consultation and preference elicitation modules, that significantly hamper the further uptake in real-world problem-solving scenarios.• The consultation module serves as the interface for DM interaction with the optimization module. It queries the DM about their preferences to effectively recommend solutions. This closely aligns with preference learning in the machine learning community.There are three strategies to accomplish this. The first, as shown in Figure 1 (b-1), involves reward models [51] or ranking mechanisms [29, 27, 38] . In this approach, DMs often provide scores or rankings for a large set of solutions. However, this overburdens the DM and risks introducing errors into the optimization process due to overlooked human cognitive limitations. The other two strategies are grounded in the dueling bandits settings [66] , utilizing pairwise comparisons in consultation but with different assumptions about human feedback. Specifically, the second strategy relies on a parameterized transitivity model [5] or a structured utility function for dueling feedback, such as the Bradley-Terry-Luce model [11] in Figure 1 (b-2). While theoretically appealing, this method faces a challenge in model selection that can be as complex as the original problem, posing difficulties in practical applications. Different from the previous two strategies, the last one tackles human feedback as stochastic events such as Bernoulli trials [63] , and it learns DM's preferences from their feedback as shown in Figure 1 2 3 ). However, a key bottleneck here is the souring number of DM queries required when considering a population of solutions in PBEMO. • The preference elicitation module acts as a catalyst, transforming the preference information learned in the consultation module-usually not directly applicable-into a format usable in the underlying EMO algorithm. The stochastic nature of human feedback [1] can lead to a misuse of learned preferences that adversely disturb search processes. This issue is pronounced in PBEMO contexts, where the number of consultations is constrained to reduce the human cognitive burden. Additionally, there is no thumb rule for determining the frequency of DM interactions or terminating such interactions, where current methods are often heuristics (e.g., setting a fixed number of interactions [36] ).In this paper, we propose a novel direct PBEMO framework (dubbed D-PBEMO) that directly leverages DM's feedback to guide the evolutionary search for SOI. Note that it neither relies on any reward model nor cumbersome model selection. Our D-PBEMO framework consists of two key features.• Given the stochastic nature of human feedback, we develop a novel clustering-based stochastic dueling bandits algorithm in the consultation module. It is model-free and its regret is O(K 2 log T ), where K is the number of clusters and T is the number of rounds. This overcomes the challenge of substantial queries inherent in conventional dueling bandits [4] . • The preference elicitation module transforms the learned preferences from the consultation module into a unified probabilistic format, in which the associated uncertainty represents the stochasticity involved in preference learning. This not only streamlines the incorporation of learned preferences into the optimization module to guide EMO to search for SOI, but also constitutes a principled termination criterion that strategically manages human cognitive burden and the computational budget. 
multi-objective optimization problem: 
 The MO problem is formulated as:min x∈Ω F(x) = f 1 (x), . . . , f m (x)⊤ , where x = (x 1 , . . . , x n ) ⊤ is an n-dimensional decision vector and F(x) is an m-dimensional objective vector whose i-th element is the objective mapping f i : Ω → R, where Ω is the feasible set in the decision space R n . Without considering the DM's preference information, given x 1 , x 2 ∈ Ω, x 1 is said to dominate x 2 (denoted as x 1 ⪯ x 2 ) iff ∀i ∈ {1, . . . , m} we have f i (x 1 ) ≤ f i (x 2 ) and F(x 1 ) ̸ = F(x 2 ). A solution x ∈ Ω is said to be Pareto-optimal iff ∄x ′ ∈ Ω such that x ′ ⪯ x. The set of all Pareto-optimal solutions is called the Pareto-optimal set (PS) and their corresponding objective vectors constitute the PF.The ultimate goal of MO is to identify the SOI from the PS satisfying DM's preference. It consists of two tasks: ① searching for Pareto-optimal solutions that cover SOI, and ② steering these solutions towards the SOI. PBEMO addresses the task ① by employing an EMO algorithm as a generator of an evolutionary population of non-dominated solutions S = {x i } N i=1 , striking a balance between convergence and diversity for coverage. For the task ②, PBEMO actively queries DM for preference information regarding these generated solutions, then it leverages the learned preferences to guide the EMO algorithm to approximate the SOI.Since human feedback from relative comparisons is considerably more reliable than absolute labels [48] , we focus on pairwise comparisons as a form of indirect preference information. In PBEMO, a DM is asked to evaluate pairs of solutions ⟨x i , x j ⟩ selected from S, where i, j ∈ {1, . . . , N } and i ̸ = j. The DM's task is to decide, based on her/his preferences, whether x i is better, worse, or equivalent to x j , denoted as x i ≻ p x j , x i ≺ p x j , or x i ≃ p x j . Regarding stochastic preference, there is a preference matrix for K-armed dueling bandits defined as P = [p i,j ] K×K , where p i,j is the winning probability of the i-th arm over the j-th arm [65] . In particular, we have p i,j + p j,i = 1 with p i,i = 0.5. The i-th arm is said to be superior to the j-th one iff p i,j > 0.5. Simply considering each solution as an individual arm will yield K = N , which suffers efficiency in targeting the SOI when the evolutionary population is large [35] . The ranking of all arms is determined by their Copeland scores, where the SOI should be the Copeland winners that have the biggest Copeland scores. Definition 2.1 ( [60] ). The normalized Copeland score of the i-th arm, i ∈ {1, . . . , K}, is given by:EQUATIONwhere I (•) is an indicator function. Arm k ⋆ satisfying k ⋆ = argmax i∈{1,...,K} ζ i is the Copeland winner.The goal of dueling bandits algorithm is to identify the Copeland winner among all candidate arms with no prior knowledge of P. To this end, a winning matrix is introduced as B = [b i,j ] K×K to record the pairwise comparison labels, where b i,j is the number of time-slots when the i-th arm is preferred from pairs of i-th and j-th arms. Consequently, we can approximate the preference probability with mean pi,j = bi,j bi,j +bj,i , whose upper confidence bound u i,j and lower confidence bound l i,j can be quantified as:EQUATIONwhere α > 0.5 controls the confidence interval, and t is the total number of comparisons so far. The performance of a dueling bandits algorithm is often evaluated by the regret defined as follows.Definition 2.2. The expected cumulative regret for a dueling bandits algorithm is given as:EQUATIONwhere T is the total number of pairwise comparisons, ζ ′ (t) and ζ ′′ (t) denote the pair to be compared at the t-th (1 ≤ t ≤ T ) round, ζ ⋆ represents the Copeland score of the Copeland winner. 
proposed method: 
 Our proposed D-PBEMO framework follows the conventional PBEMO flow chart as in Figure 1 (a).In the following paragraphs, we mainly focus on delineating the design of D-PBEMO with regard to both consultation and preference elicitation modules, while leaving the design of the optimization module open.As the interface by which the DM interacts with an EMO algorithm, the consultation module mainly aims to collect the DM's preference information from their feedback upon S to identify the SOI. We employ the stochastic dueling bandits [74] , to directly derive preferences from human feedback without relying on further assumptions such as contextual priors [41] or structured models [11] . In this setting, a natural choice is to consider each candidate solution as an arm to play. However, since the size of S is usually as large as over 100 in the context of EMO, the conventional dueling bandits algorithms will suffer from a large amount of preference comparisons to converge [73, 35] . This is impractical in PBEMO when involving DM in the loop. To address this problem, we propose clustering-based stochastic dueling bandits algorithm that consist of the following three steps.Step 1: Partition S into K subsets { Si } K i=1 based on solution features in the context of EMO. Such partitioning is implemented as a clustering method based on the Euclidean distances between solutions of S in the objective space. Instead of viewing each solution as an individual arm, we consider each subset Si as an arm in our proposed dueling bandits. We denote the solution-level preference matrix as P s = [p s i,j ] N ×N , where p s i,j represents the probability that x i ≻ p x j . Then, the preference matrix P in the subset-level can be calculated by p i,j =1 | Si || Sj | x u ∈ Si x v ∈ Sj p s u,v, where | Si | stands for the size of Si . This probability is well-defined since it satisfies p i,j + p j,i = 1. So far, we have reformulated a subset-level dueling bandits problem. Accordingly, the subset-level Copeland winner is the subset S⋆ that beats others on average.Step 2: Subset-level dueling sampling and solution-level pairwise comparisons. We employ the double Thompson sampling algorithm [62] to determine the subset pairs, and then select solutions from the pairs to query DM preferences. We introduce two vectors v = (v 1 , . . . , v N ) ⊤ and ℓ = (ℓ 1 , . . . , ℓ N ) ⊤ to record the winning and losing times of each solution respectively, initialized by v i = 0, ℓ i = 0, i = {1, . . . , N }. We perform the following steps within a given budget T .Step 2.1: Determine the subset S′ that most likely covers the SOI. We first narrow candidates to the subsets having the highest upper confidence Copeland scores, denoted as C 1 = { Si |i = argmax i ζi }, where ζi = 1 K-1 j̸ =i I(u i,j > 0.5), i, j ∈ {1, . . . , K}. Then, ∀ Si ∈ C 1 , we apply Thompson sampling as θ (1) i,j ∼ Beta(b i,j + 1, b j,i + 1) to sample the winning probability of Si over other subsets Sj , where j ∈ {1, . . . , K} and j ̸ = i. Finally, we apply the majority voting strategy to determine the candidate by S′ ← argmax Si ∈C 1 j̸ =i I(θ (1) i,j > 0.5), where ties are broken randomly.Step 2.2: Select the subset S′′ that can be potentially preferred over S′ . To promote exploration, we narrow candidates to the subsets whose lower-confident winning probability over S′ is at most 0.5, denoted as C 2 = { Si |l i,′ ≤ 0.5}. Note that S′ ∈ C 2 because l ′,′ ≤ p ′,′ = 0.5. x 00 < l a t e x i t s h a 1 _ b a s e 6 4 = " n B N Y f B l O 2 + e t k m A X D K T A I 6 X s 2 v M = " > A A A B + n i c b V C 7 T s M w F L 0 p r 1 J e K Y w s F h U S U 5 U g V B g r W B i L R B 9 S E y r H d V q r d h L Z D l C F f g o L A w i x 8 i V s / A 1 O 2 w F a j m T p 6 J x 7 d Y 9 P k H C m t O N 8 W 4 W V 1 b X 1 j e J m a W t 7 Z 3 f P L u + 3 V J x K Q p s k 5 r H s B F h R z i L a 1 E x z 2 k k k x S L g t B 2 M r n K / f U + l Y n F 0 q 8 c J 9 Q U e R C x k B G s j 9 e y y J 7 A e B m H 2 O L n z E s k E 7 d k V p + p M g Z a J O y c V m K P R s 7 + 8 f k x S Q S N N O F a q 6 z q J 9 j M s N S O c T k p e q m i C y Q g P a N f Q C A u q / G w a f Y K O j d J H Y S z N i z S a q r 8 3 M i y U G o v A T O Z B 1 a K X i / 9 5 3 V S H F 3 7 G o i T V N C K z Q 2 H K k Y 5 R 3 g P q M 0 m J 5 m N D M J H M Z E V k i C U m 2 r R V M i W 4 i 1 9 e J q 3 T q l u r 1 m 7 O K v X L e R 1 F O I Q j O A E X z q E O 1 9 C A J h B 4 g G d 4 h T f r y X q x 3 q 2 P 2 W j B m u 8 c w B 9 Y n z / v 3 5 R 1 < / l a t e x i t > x 0< l a t e x i t s h a 1 _ b a s e 6 4 = " s W D z O S v q h D m b s g L h a b 1 o u u 2 t w q g = " > A A A C A Q O e D h j H H K s R Z M H j A B F D F p 5 o Q K p j e F d M x E Y Q q H V 9 B h / B 1 K f 6 f t C p l 2 y k 7 l y e l + t k i j j w 6 Q E V 0 h G x U R X V 0 g R q o i S i 6 Q w / o C T 0 b 9 8 a j 8 W K 8 z k t z x q J n H / 2 A 8 f Y J K k S c b w = = < / l a t e x i t > S00 < l a t e x i t s h a 1 _ b a s e 6 4 = " N v Q q Z G X Y 7 p l E Q K B P k k 3 M P k u u d B s = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G T b A I r k o i U l 0 W 3 b i s a B / Q x D K Z T N q h k 0 m Y u R F K z M J f c e N C E b f + h j v / x k n b h b Y e G D i c c y / 3 z P E T z h T Y 9 r d R W l p e W V 0 r r 1 c 2 N r e 2 d 8 z d v b a K U 0 l o i 8 Q 8 l l 0 f K 8 q Z o C 1 g w G k 3 k R R H P q c d f 3 R V + J 0 H K h W L x R 2 M E + p F e C B Y y A g G L f X N A x c Y D 2 j m R h i G B P P s N s / v n U r f r N o 1 e w J r k T g z U k U z N P v m l x v E J I 2 o A M K x U j 3 H T s D L s A R G O M 0 r b q p o g s k I D 2 h P U 4 E j q r x s k j + 3 j r U S W G E s 9 R N g T d T f G x m O l B p H v p 4 s Y q p 5 r x D / 8 3 o p h B d e x k S S A h V k e i h M u Q W x V Z R h B U x S A n y s C S a S 6 a w W G W K J C e j K i h K c + S 8 v k v Z p z a n X 6 j d n 1 c b l r I 4 y O k R H 6 A Q 5 6 B w 1 0 D V q o h Y i 6 B E 9 o 1 f 0 Z j w Z L 8 a 7 8 T E d L R m z n X 3 0 B 8 b n D / Z i l h I = < / l a t e x i t > S1 < l a t e x i t s h a 1 _ b a s e 6 4 = " c 1 5 m / F e J T G S m R J n C Y 3 0 t L V 0 j 0 0 Q = " > A A A B / 3 i c b V D L S s N A F J 3 4 r P U V F d y 4 G S y C q 5 I U q S 6 L b l x W t A 9 o Y p l M J u 3 Q y S T M 3 A g l d u G v u H G h i F t / w 5 1 / Y 9 J m o a 0 H B g 7 n 3 M s 9 c 7 x Y c A 2 W 9 W 0 s L a + s r q 2 X N s q b W 9 s 7 u + b e f l t H i a K s R S M R q a 5 H N B N c s h Z w E K w b K 0 Z C T 7 C O N 7 r K / c 4 D U 5 p H 8 g 7 G M X N D M p A 8 4 J R A J v X N Q w e 4 8 F n q h A S G l I j 0 d j K 5 r 5 X 7 Z s W q W l P g R W I X p I I K N P v m l + N H N A m Z B C q I 1 j 3 b i s F N i Q J O B Z u U n U S z m N A R G b B e R i U J m X b T a f 4 J P s k U H w e R y p 4 E P F V / b 6 Q k 1 H o c e t l k H l P P e 7 n 4 n 9 d L I L h w U y 7 j B J i k s 0 N B I j B E O C 8 D + 1 w x C m K c E U I V z 7 J i O i S K U M g q y 0 u w 5 7 + 8 S N q 1 q l 2 v 1 m / O K o 3 L o o 4 S O k L H 6 B T Z 6 B w 1 0 D V q o h a i 6 B E 9 o 1 f 0 Z j w Z L 8 a 7 8 T E b X T K K n Q P 0 B 8 b n D / f n l h M = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " V y j s q / e l c w i 14 0 C K 9 h x k m t 8 t N V 4 = " > A A A B / 3 i c b V D L S s N A F J 3 4 r P U V F d y 4 C R b B V U l U q s u i G 5 c V 7 Q O a W C a T m 3 b o Z B J m J k K J W f g r b l w o 4 t b f c O f f O G m 7 0 N Y D A 4 d z 7 u W e O X 7 C q F S 2 / W 0 s L C 4 t r 6 y W 1 s r r G 5 t b 2 + b O b k v G q S D Q J D G L R c f H E h j l 0 F R U M e g k A n D k M 2 j 7 w 6 v C b z + A k D T m d 2 q U g B f h P q c h J V h p q W f u u 4 q y A D I 3 w m p A M M t u 8 / z + t N w z K 3 b V H s O a J 8 6 U V N A U j Z 7 5 5 Q Y x S S P g i j A s Z d e x E + V l W C h K G O R l N 5 W Q Y D L E f e h q y n E E 0 s v G + X P r S C u B F c Z C P 6 6 s s f p 7 I 8 O R l K P I 1 5 N F T D n r F e J / X j d V 4 Y W X U Z 6 k C j i Z H A p T Z q n Y K s q w A i q A K D b S B B N B d V a L D L D A R O n K i h K c 2 S / P k 9 Z J 1 a l V a z d n l f r l t I 4 S O k C H 6 B g 5 6 B z V 0 T V q o C Y i 6 B E 9 o 1 f 0 Z j w Z L 8 a 7 8 T E Z X T C m O 3 v o D 4 z P H / l s l h Q = < / l a t e x i t > S3 < l a t e x i t s h a 1 _ b a s e 6 4 = " N v Q q Z G X Y 7 p l E Q K B P k k 3 M P k u u d B s = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G T b A I r k o i U l 0 W 3 b i s a B / Q x D K Z T N q h k 0 m Y u R F K z M J f c e N C E b f + h j v / x k n b h b Y e G D i c c y / 3 z P E T z h T Y 9 r d R W l p e W V 0 r r 1 c 2 N r e 2 d 8 z d v b a K U 0 l o i 8 Q 8 l l 0 f K 8 q Z o C 1 g w G k 3 k R R H P q c d f 3 R V + J 0 H K h W L x R 2 M E + p F e C B Y y A g G L f X N A x c Y D 2 j m R h i G B P P s N s / v n U r f r N o 1 e w J r k T g z U k U z N P v m l x v E J I 2 o A M K x U j 3 H T s D L s A R G O M 0 r b q p o g s k I D 2 h P U 4 E j q r x s k j + 3 j r U S W G E s 9 R N g T d T f G x m O l B p H v p 4 s Y q p 5 r x D / 8 3 o p h B d e x k S S A h V k e i h M u Q W x V Z R h B U x S A n y s C S a S 6 a w W G W K J C e j K i h K c + S 8 v k v Z p z a n X 6 j d n 1 c b l r I 4 y O k R H 6 A Q 5 6 B w 1 0 D V q o h Y i 6 B E 9 o 1 f 0 Z j w Z L 8 a 7 8 T E d L R m z n X 3 0 B 8 b n D / Z i l h I = < / l a t e x i t > S1 < l a t e x i t s h a 1 _ b a s e 6 4 = " c 1 5 m / F e J T G S m R J n C Y 3 0 t L V 0 j 0 0 Q = " > A A A B / 3 i c b V D L S s N A F J 3 4 r P U V F d y 4 G S y C q 5 I U q S 6 L b l x W t A 9 o Y p l M J u 3 Q y S T M 3 A g l d u G v u H G h i F t / w 5 1 / Y 9 J m o a 0 H B g 7 n 3 M s 9 c 7 x Y c A 2 W 9 W 0 s L a + s r q 2 X N s q b W 9 s 7 u + b e f l t H i a K s R S M R q a 5 H N B N c s h Z w E K w b K 0 Z C T 7 C O N 7 r K / c 4 D U 5 p H 8 g 7 G M X N D M p A 8 4 J R A J v X N Q w e 4 8 F n q h A S G l I j 0 d j K 5 r 5 X 7 Z s W q W l P g R W I X p I I K N P v m l + N H N A m Z B C q I 1 j 3 b i s F N i Q J O B Z u U n U S z m N A R G b B e R i U J m X b T a f 4 J P s k U H w e R y p 4 E P F V / b 6 Q k 1 H o c e t l k H l P P e 7 n 4 n 9 d L I L h w U y 7 j B J i k s 0 N B I j B E O C 8 D + 1 w x C m K c E U I V z 7 J i O i S K U M g q y 0 u w 5 7 + 8 S N q 1 q l 2 v 1 m / O K o 3 L o o 4 S O k L H 6 B T Z 6 B w 1 0 D V q o h a i 6 B E 9 o 1 f 0 Z j w Z L 8 a 7 8 T E b X T K K n Q P 0 B 8 b n D / f n l h M = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " V y j s q / e l c w i 14 0 C K 9 h x k m t 8 t N V 4 = " > A A A B / 3 i c b V D L S s N A F J 3 4 r P U V F d y 4 C R b B V U l U q s u i G 5 c V 7 Q O a W C a T m 3 b o Z B J m J k K J W f g r b l w o 4 t b f c O f f O G m 7 0 N Y D A 4 d z 7 u W e O X 7 C q F S 2 / W 0 s L C 4 t r 6 y W 1 s r r G 5 t b 2 + b O b k v G q S D Q J D G L R c f H E h j l 0 F R U M e g k A n D k M 2 j 7 w 6 v C b z + A k D T m d 2 q U g B f h P q c h J V h p q W f u u 4 q y A D I 3 w m p A M M t u 8 / z + t N w z K 3 b V H s O a J 8 6 U V N A U j Z 7 5 5 Q Y x S S P g i j A s Z d e x E + V l W C h K G O R l N 5 W Q Y D L E f e h q y n E E 0 s v G + X P r S C u B F c Z C P 6 6 s s f p 7 I 8 O R l K P I 1 5 N F T D n r F e J / X j d V 4 Y W X U Z 6 k C j i Z H A p T Z q n Y K s q w A i q A K D b S B B N B d V a L D L D A R O n K i h K c 2 S / P k 9 Z J 1 a l V a z d n l f r l t I 4 S O k C H 6 B g 5 6 B z V 0 T V q o C Y i 6 B E 9 o 1 f 0 Z j w Z L 8 a 7 8 T E Z X T C m O 3 v o D 4 z P H / l s l h Q = < / l a t e x i t > S3 < l a t e x i t s h a 1 _ b a s e 6 4 = " s W D z O S v q h D m b s g L h a b 1 o u u 2 t w q g = " > A A A C A 3 i c d V D L S g N B E J y N r x h f U W 9 6 G Q y C p 7 A b d Z N j 0 I v H i O Y B 2 T X M T j r J k N k H M 7 N C W A J e / B U v H h T x 6 k 9 4 8 2 + c T V Z Q 0 Y K G o q q b 7 i 4 v 4 k w q 0 / w w c g u L S 8 s r + d X C 2 v r G 5 l Z x e 6 c l w 1 h Q a N K Q h 6 L j E Q m c B d B U T H H o R A K I 7 3 F o e + P z 1 G / f g p A s D K 7 V J A L X J 8 O A D R g l S k u 9 4 p 6 j G O 9 D 4 v h E j S j h y d V 0 e u N E g v n Q K 5 b M s l 2 x r e o x N s v m D C m p 1 S q n N r Y y p Y Q y N H r F d 6 c f 0 t i H Q F F O p O x a Z q T c h A j F K I d p w Y k l R I S O y R C 6 m g b E B + k m s x + m + F A r f T w I h a 5 A 4 Z n 6 f S I h v p Q T 3 9 O d 6 a n y t 5 e K f 3 n d W A 1 q b s K C K F Y Q 0 P m i Q c y x C n E a C O 4 z A V T x i S a E C q Z v x X R E B K F K x 1 b Q I X x 9 i v 8 n r U r Z s s v 2 5 U m p f p b F k U f 7 6 A A d I Q t V U R 1 d o A Z q I o r u 0 A N 6 Q s / G v f F o v B i v 8 9 a c k c 3 s o h 8 w 3 j 4 B Z m e Y r g = = < / l a t e x i t > S0 < l a t e x i t s h a 1 _ b a s e 6 4 = " f A L u C O z e X C E p G h R 5 S S p G O 0 J a 9 7 Y = " > A A A C C 3 i c d V C 7 S g N B F J 2 N r x h f q 5 Y 2 Q 4 J g F X a j b l I G b S w j m g d k Y 5 i d 3 C R D Z h / M z A p h 2 d 7 G X 7 G x U M T W H 7 D z b 5 x N I q j o g R k O 5 9 z L v f d 4 E W d S W d a H k V t a X l l d y 6 8 X N j a 3 t n f M 3 b 2 W D G N B o U l D H o q O R y R w F k B T M c W h E w k g v s e h 7 U 3 O M 7 9 9 C 0 K y M L h W 0 w h 6 P h k F b M g o U V r q m 0 V X M T 6 A x P W J G l P C k 6 s 0 v U n c S D A f 5 n / a N 0 t W 2 a k 4 d v U Y W 2 V r h o z U a p V T B 9 s L p Y Q W a P T N d 3 c Q 0 t i H Q F F O p O z a V q R 6 C R G K U Q 5 p w Y 0 l R I R O y A i 6 m g b E B 9 l L Z r e k + F A r A z w M h X 6 B w j P 1 e 0 d C f C m n v q c r s 5 X l b y 8 T / / K 6 s R r W e g k L o l h B Q O e D h j H H K s R Z M H j A B F D F p 5 o Q K p j e F d M x E Y Q q H V 9 B h / B 1 K f 6 f t C p l 2 y k 7 l y e l + t k i j j w 6 Q E V 0 h G x U R X V 0 g R q o i S i 6 Q w / o C T 0 b 9 8 a j 8 W K 8 z k t z x q J n H / 2 A 8 f Y J K k S c b w = = < / l a t e x i t > S00 < l a t e x i t s h a 1 _ b a s e 6 4 = " Y o q v / V K p A e L 3 Q u 0 r w q D w f B j M F k 4 = " > A A A C A n i c b V C 7 T s M w F H X K q 5 R X g A m x W F R I T F W C U G G s Y G E s E n 1 I T a g c 1 2 m t 2 k 5 k O 4 g q i l j 4 F R Y G E G L l K 9 j 4 G 5 w 2 A 7 Q c y d b R O f f q 3 n u C m F G l H e f b K i 0 t r 6 y u l d c r G 5 t b 2 z v 2 7 l 5 b R Y n E p I U j F s l u g B R h V J C W p p q R b i w J 4 g E j n W B 8 l f u d e y I V j c S t n s T E 5 2 g o a E g x 0 k b q 2 w c e R 3 o U h O l D d p d 6 s a S c z P 6 s b 1 e d m j M F X C R u Q a q g Q L N v f 3 m D C C e c C I 0 Z U q r n O r H 2 U y Q 1 x Y x k F S 9 R J E Z 4 j I a k Z 6 h A n C g / n Z 6 Q w W O j D G A Y S f O E h l P 1 d 0 e K u F I T H p j K f G E 1 7 + X i f 1 4 v 0 e G F n 1 I R J 5 o I P B s U J g z q C O Z 5 w A G V B G s 2 M Q R h S c 2 u E I + Q R F i b 1 C o m B H f + 5 E X S P q 2 5 9 V r 9 5 q z a u C z i K I N D c A R O g A v O Q Q N c g y Z o A Q w e w T N 4 B W / W k / V i v V s f s 9 K S V f T s g z + w P n 8A l e S Y N g = = < / l a t e x i t >x 00 ′,′ = 0.5 according to the definition of preference matrix. Finally, the candidate is determined by< l a t e x i t s h a _ b a s e = " n B N Y f B l O + e t k m A X D K T A I X s v M = " > A A A B + n i c b V C T s M w F L p r J e K Y w s F h U S U U g V B g r W B i L R B S E y r H d V q r d h L Z D l C F f g o L A w i x i V s / A O w F a j m T p J x d Y P k H C m t O N W W V b X j e J m a W t Z f P L u + V J x K Q p s k r H s B F h R z i L a E x z k k k x S L g t B M r n K / f U + l Y n F q c J Q U e R C x k B G s j e y y J A e B m H O L n z E s k E d k V p + p M g Z a J O y c V m K P R s + f k x S Q S N N O F a q z q J j M s N S O c T k p e q m i C y Q g P a N f Q C A u q / G w a f Y K O j d J H Y S z N i z S a q r M i y U G o v A T O Z B a K X i / V S H F G o i T V N C K z Q H K k Y R g P q M m J m N D M J H M Z E V k i C U m r R V M i W i e J q T q l u r m O K v X L e R F O I Q j O A E X z q E O C A J h B g G d h T f r y X q x q P W j B m u c w B Y n z / v R < / l a t e x i t > x 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " V E v i X Y b M w B 1 n H 1 R L 6 i 8 h H u f 5 V R A = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i Q i 1 W X R j c s K 9 g F N C J P p p B 0 6 m Y S Z S a G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x z m X t P m H K m t O N 8 W 5 W N z a 3 t n e p u b W / / 4 P D I P j 7 p q i S T h H Z I w h P Z D 7 G i n A n a 0 U x z 2 k 8 l x X H I a S + c 3 B d + b 0 q l Y o l 4 0 r O U + j E e C R Y x g r W R A t v 2 T J w E X o z 1 W M Z 5 O g / s u t N w F k D r x C 1 J H U q 0 A / v L G y Y k i 6 n Q h G O l B q 6 T a j / H U j P C 6 b z m Z Y q m m E z w i A 4 M F T i m y s 8 X m 8 / R h V G G K E q k e U K j h f p 7 I s e x U r M 4 N M l i Q 7 X q F e J / 3 i D T 0 a 2 f M 5 F m m g q y / C j K O N I J K m p A Q 2 b O 1 n x m C C a S m V 0 R G W O J i T Z l 1 U w J 7 u r J 6 6 R 7 1 X C b j e b j d b 1 1 V 9 Z R h T M 4 h 0 t w 4 Q Z a 8 A B t 6 A C B K T z D K 7 x Z u f V i v V s f y 2 j F K m d O 4 Q + s z x 8 s + Z Q I < / l a t e x i t > p < l a t e x i t s h a 1 _ b a s e 6 4 = " T U b 0 U W I L T I 5 m a d R W z V 2 j Q z l r P x A = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i Q i 1 W X R j c s K 9 g F N C J P p p B 0 6 m Y S Z S a G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O m H K m t O N 8 W 5 W N z a 3 t n e p u b W / / 4 P D I P j 7 p q i S T h H Z I w h P Z D 7 G i n A n a 0 U x z 2 k 8 l x X H I a S + c 3 B d + b 0 q l Y o l 4 0 r O U + j E e C R Y x g r W R A t v 2 V E Z I 4 M V Y j 2 W c p / P A r j s N Z w G 0 T t y S 1 K F E O 7 C / v G F C s p g K T T h W a u A 6 q f Z z L D U j n M 5 r X q Z o i s k E j + j A U I F j q v x 8 k X y O L o w y R F E i z R M a L d T f G z m O l Z r F o Z k s E q p V r x D / 8 w a Z j m 7 9 n I k 0 0 1 S Q 5 a E o 4 0 g n q K g B D Z m k R P O Z I Z h I Z r I i M s Y S E 2 3 K q p k S 3 N U v r 5 P u V c N t N p q P 1 / X W X V l H F c 7 g H C 7 B h R t o w Q O 0 o Q M E p v A MS′′ ← argmax Si ∈C 2 θ (2) i,′ .Step 2.3: Select two representative solutions x ′ ∈ S′ and x ′′ ∈ S′′ to query DM. We conduct uniform sampling to obtain solutions from the least-informative perspective. The DM is asked to evaluate the pair of solutions ⟨x ′ , x ′′ ⟩. If we observe x ′ ≻ p x ′′ , we update b ′,′′ ← b ′,′′ + 1, v ′ ← v ′ + 1, and ℓ ′′ ← ℓ ′′ + 1, and vice versa. Note that other strategies to obtain solutions x ′ and x ′′ can be used to improve the query efficiency.Step 3: Output the learned preferences. The output is a triplet { S⋆ , v, ℓ}, where S⋆ = argmax Si ζi is the optimal subset that most likely covers SOI, and candidate solutions are considered to be the SOI with uncertainty encoded by their winning and losing times.The pseudo codes of the above algorithmic implementation are detailed in Appendix C. Figure 2 gives an illustrative example of the consultation process.Remark 1. PBEMO is a optimization-cum-decision-making process. Instead of having a set of Pareto-optimal candidate solutions upfront, PBEMO starts with a coarse-grained representation of the PF. Then, it gradually steers incumbent solutions towards the learned SOI, which may be inaccurate initially. Subsequent consultations then serve as a refinement process. In this context, different from the dueling bandits, which are designed for identifying the single best solution in each round, our proposed method aims to recognize the SOI with a progressively refined fidelity.Remark 2. Based on the Remark 1, we intend to explore the dependency among solutions [47, 53] . This involves performing pairwise comparisons for solution groups, rather than for a single candidate, to enable more efficient interaction. Consequently, in the preference elicitation stage, it becomes essential not only to rely on the learned preference but also to consider the uncertainty introduced by the coarse-grained representation.Theoretical Analysis. We present a rigorous analysis of the regret bound of our proposed algorithm.To this end, we introduce the following two assumptions derived from the current literature.). The winning probability between two arms satisfies p i,j ̸ = 0.5, ∀i ̸ = j.Assumption 3.2 (Tight clustering [33] ). All solutions in S⋆ are the Copeland winners over other solutions in the sub-optimal subsets. Figure 3 : The density ratio between p ν (x) and p ℓ (x) is shaded in blue, while its estimation is shaded in red. The SOI falls within the estimated Gaussian distribution for 95% confidence interval.Theorem 3.3. Under the Assumptions 3.1 and 3.2, for any ϵ ∈ (0, 1] and α > 0.5, the regret of our clustering-based stochastic dueling bandits algorithm is bounded by: R T (T ) ≤ i̸ =j, p i,j <0.5(1 + ϵ) log T D KL (p i,j ∥ 0.5) + 4α log T (p i,j -0.5) 2 + O K 2 ϵ 2 .Definitions of p i,j and proof can be found in Appendix B.1. Remark 3. Theorem 3.3 reveals that the expected regret of our algorithm is bounded by O(K 2 log T ). Compared with the dueling bandits algorithms based on Thompson sampling and their extensions to large arms [62, 35] , whose regret is bounded by O(N 2 log T ), our proposed clustering-based stochastic dueling bandits algorithm is more efficient in searching for the SOI, given K ≪ N .The preference elicitation module plays as a bridge that connects consultation and optimization. It transforms the preference learned from the consultation module into the configurations used in the underlying EMO algorithm, thus steering the EMO process progressively towards the SOI. In addition, this module maintains the preference information to inform a strategic termination criterion of consultation, thus reduces DM's workload. We present the design of this module by addressing the following three questions.How to leverage the preference learned from the consultation session? Let us assume the DM's feedback collected in the consultation session is drawn from a preference distribution as the density ratio p v (x)/p ℓ (x), where p v (x) and p ℓ (x) is respectively the winning and losing probability of a solution x sampled from the PS, see an illustrative example in Figure 3 (a). After the τ -th round of the consultation session, we perform density-ratio estimation based on v and ℓ. By using moment matching techniques (see Appendix B.2), we obtain a Gaussian distribution with mean x * τ and covariance Στ . Let στ take the largest value of diagonal elements of Στ . The preference elicitation module maintains a Gaussian mixture distribution by a convex combination of Gaussian distributions from multiple consultation sessions:EQUATIONwhere x ∈ Ω, N consult is the total number of consultation sessions conducted so far, and Z =N consult τ =11/σ τ is the normalization term. Figure 3 (b) gives an illustrative example of a Gaussian distribution approximated by the DM's feedback collected at one consultation session.How to adapt Pr(x) to EMO algorithms? We believe Pr(x) can be applied in any EMO algorithms with few adaptation in their environmental selection. For proof-of-concept purpose, this paper takes NSGA-II [14] and MOEA/D [68] , two most popular algorithms in the EMO literature, as examples and we design two D-PBEMO instances, dubbed D-PBNSGA-II and D-PBMOEA/D.• At each generation of the original NSGA-II, it first uses non-dominated sorting to divide the combination of parents and offspring into several non-domination fronts F 1 , . . . , F l . Starting from F 1 , one front is selected at a time to construct a new population, until its size equals to or exceeds the limit. The exceeded solutions in the last acceptable front will be eliminated according to the crowding distance metric to maintain population diversity. In D-PBNSGA-II, we replace the crowding distance with P r(x). As a result, the solutions close to the SOI will survive to the next generation. • The basic idea of the classic MOEA/D is to decompose the original MOP into a set of subproblems using weight vectors. Then, these subproblems are tackled collaboratively using population-based meta-heuristics. In D-PBMOEA/D, we progressively transform the originally uniformly distributed weight vectors W = {w i } N i=1 used in the original MOEA/D to the preference distribution Pr(x). In practice, the transformed weight vector is w i′ = Pr -1 (w i ), where Pr -1 (•) is the weighted sum of inverse Gaussian distribution, defined in equation (19) .Detailed implementation of D-PBNSGA-II and D-PBMOEA/D are in Appendix C.There is no principled termination criterion in exiting PBEMO, but is often set as a pre-defined number of consultation sessions. This is not rationale and likely to incur unnecessary workloads to DM, even when the evolutionary population is either converged to the SOI or being trapped by local optima. Under our D-PBEMO framework, we have the following theoretical result about the convergence property of Pr(x).Theorem 3.4. Assume the preference distribution Pr(x) around the SOI follows a Gaussian mixture distribution. It becomes stable when N consult increases.The proof of Theorem 3.4 is in Appendix B.3. In D-PBEMO, we apply Theorem 3.4 to adaptively terminate the consultation session when Pr(x) becomes stable. In practice, this happens when the Kullback-Leibler (KL) divergence of Pr(•) between two consecutive consultation sessions is smaller than a threshold ε:EQUATIONwhere 1 < τ ≤ N consult , xi is sampled from S⋆ . N s is the number of samples when calculating the KL divergence. Here we use ε = 10 -3 , and its parameter sensitivity is studied in Section 4.4. 
experimental setup: 
 This section outlines some key experimental setup including benchmark test problems and performance metrics. More detailed information can be found in Appendix D.Benchmark problems Our experiments considers 33 synthetic test instances including ZDT1 to ZDT4 and ZDT6 [72] (m = 2), DTLZ1 to DTLZ6 [18] where m = {3, 5, 8, 10}, and WFG1, WFG3, WFG5, and WFG7 [30] (m = 3). These problems are with various PF shapes and challenges such as bias, plateau, and multi-modal. In addition, we also consider two scientific discovery problems including 10 two-objective RNA inverse design tasks [55, 70] and 5 four-objective protein structure prediction (PSP) task [69] . The problem formulations are detailed in Appendix D.1.Performance metrics As discussed in [37] , quality assessment of non-dominated solutions is far from trivial when considering DM's preference information regarding conflicting objective. In our experiments, we consider two metrics to serve this purpose. One is approximation accuracy ϵ ⋆ (S) that evaluates the closest distance of x ∈ S regarding the DM preferred solution in the objective space, denoted as 'golden point' z ⋆ ∈ R m , while the other is average accuracy ε(S) that evaluates the average distance to z ⋆ . Note that z ⋆ is unknown to the underlying algorithm in practice, and the corresponding settings used in our experiments are in Appendix D.2. Due to the stochastic < l a t e x i t s h a 1 _ b a s e 6 4 = " P W m p S 5 rI v O H q 1 D n K B 1 9 t / v 9 A O + M = " > A A A C B H i c b V C 7 T s M w F H X K q 5 R X g L G L R Y V U l i p B q D B W s D A W Q R 9 S E y r H d V q r j h 3 Z D l I V d W D h V 1 g Y Q I i V j 2 D j b 3 D a D N B y p C s d n X O v 7 r 0 n i B l V 2 n G + r c L K 6 t r 6 R n G z t L W 9 s 7 t n 7 x + 0 l U g k J i 0 s m J D d A C n C K C c t T T U j 3 V g S F A W M d I L x V e Z 3 H o h U V P A 7 P Y m J H 6 E h p y H F S B u p b 5 c 9 E i v K B L / 3 l E a y 6 k V I j z B i 6 e 3 0 p G 9 X n J o z A 1 w m b k 4 q I E e z b 3 9 5 A 4 G T i H C N G V K q 5 z q x 9 l M k N c W M T E t e o k i M 8 B g N S c 9 Q j i K i / H T 2 x B Q e G 2 U A Q y F N c Q 1 n 6 u + J F E V K T a L A d G Y 3 q k U v E / / z e o k O L / y U 8 j j R h O P 5 o j B h U A u Y J Q I H V B K s 2 c Q Q h C U 1 t 0 I 8 Q h J h b X I r m R D cx Z e X S f u 0 5 t Z r 9 Z u z S u M y j 6 M I y u A I V I E L z k E D X I M m a A E M H s E z e A V v 1 p P 1 Y r 1 b H / P W g p X P H I I / s D 5 / A B F x m G A = < / l a t e x i t > ✏ ? (S)< l a t e x i t s h a 1 _ b a s e 6 4 = " g h 3 r E w q x g d 6 M Z x U / o 8 m a X 3 H g f n w = " > A A A C B H i c b V D L S s N A F J 3 4 r P U V d d n N Y B H q p i Q i 1 W X R j c u K 9 g F N K J P p p B 0 6 m Q k z E 6 G E L N z 4 K 2 5 c K O L W j 3 D n 3 z h p s 9 D W A x c O 5 9 z L v f c E M a N K O 8 6 3 t b K 6 t r 6 x W d o q b + / s 7 u 3 b B 4 c d J R K J S R s L J m Q v Q I o w y k l b U 8 1 I L 5 Y E R Q E j 3 W B y n f v d B y I V F f x e T 2 P i R 2 j E a U g x 0 k Y a 2 B U v Q D L 1 S K w o E z y r e R H S Y 4 x Y e p e d D u y q U 3 d m g M v E L U g V F G g N 7 C 9 v K H A S E a 4 x Q 0 r 1 X S f W f o q k p p i R r O w l i s Q I T 9 C I 9 A 3 l K C L K T 2 d P Z P D E K E M Y C m m K a z h T f 0 + k K F J q G g W m M 7 9 R L X q 5 + J / X T 3 R 4 6 a e U x 4 k m H M 8 X h Q m D W s A 8 E T i k k m D N p o Y g L K m 5 F e I x k g h r k 1 v Z h O A u v r x M O m d 1 t 1 F v 3 J 5 X m 1 d F H C V Q A c e g B l x w A Z r g B r R A G 2 D w C J 7 B K 3 i z n q w X 6 9 3 6 m L e u W M X M E f g D 6 / M H M f a Y d Q = = < / l a tG G G C C G G G G C C G C A G G G C C C C G G C G G G C G A G C C G G C G G G C C A C C G C C G G C U C G G C C G C C C G C C G G G C G G G C G G C C G C A C G A G G C C G G C C C U A U U G G G C C G G C C U G G G C C C G C C G U A G C U C G G C G G G C C C G D-PBMOEA/D < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 2 o 3 9 x a R L U / Q c 2 Y H 0 7 D J S I 6 b 7 V 8 = " > A A A B + X i c b V D L S g N B E J y N r x h f q x 6 9 D A b B i 8 t u C F E P Q k A U L 0 I E 8 4 B k C b O T 2 W T I 7 I O Z 3 m B Y 8 i d e P C j i 1 T / x 5 t 8 4 S f a g i Q U N R V U 3 3 V 1 e L L g C 2 / 4 2 c i u r a + s b + c 3 C 1 v b O 7 p 6 5 f 9 B Q U S I p q 9 N I R L L l E c U E D 1 k d O A j W i i U j g S d Y 0 x t e T / 3 m i E n F o / A R x j F z A 9 I P u c 8 p A S 1 1 T b M D 7 A n S + 9 u b y d V Z q W w 5 X b N o W / Y M e J k 4 G S m i D L W u + d X p R T Q J W A h U E K X a j h 2 D m x I J n A o 2 K X Q S x W J C h 6 T P 2 p q G J G D K T W e X T / C J V n r Y j 6 S u E P B M / T 2 R k k C p c e D p z o D A Q C 1 6 U / E / r 5 2 A f + G m P I w T Y C G d L / I T g S H C 0 x h w j 0 t G Q Y w 1 I V R y f S u m A y I J B R 1 W Q Y f g L L 6 8 T B o l y 6 l Y l Y d y s X q Z x Z F H R + g Y n S I H n a M q u k M 1 V E c U j d A z e k V v R m q 8 G O / G x 7 w 1 Z 2 Q z h + g P j M 8 f x V y S b Q = = < / l a t e x i t > MFE = 24.1 < l a t e x i t s h a 1 _ b a s e 6 4 = " u x z f j + k j R h L O R I Q 2 h g c q K 4 K 3 8 j Q = " > A A A B 8 n i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B 0 7 J b p N q D U P D i s Y L 9 g O 1 S s m n a h i a b J c k K Z e n P 8 O J B E a / + G m / + G 9 N 2 D 9 r 6 Y O D x 3 g w z 8 6 K E M 2 0 8 7 9 s p b G x u b e 8 U d 0 t 7 + w e H R + X j k 7 a W q S K 0 R S S X q h t h T T m L a c s w w 2 k 3 U R S L i N N O N L m b + 5 0 n q j S T 8 a O Z J j Q U e B S z I S P Y W C n o a T Y S + N Z z 6 9 V + u e K 5 3 g J o n f g 5 q U C O Z r / 8 1 R t I k g o a G 8 K x 1 o H v J S b M s D K M c D o r 9 V J N E 0 w m e E Q D S 2 M s q A 6 z x c k z d G G V A R p K Z S s 2 a K H + n s i w 0 H o q I t s p s B n r V W 8 u / u c F q R n e h B m L k 9 T Q m C w X D V O O j E T z / 9 G A K U o M n 1 q C i W L 2 V k T G W G F i b E o l G 4 K / + v I 6 a V d d v + b W H q 4 q j X o e R x H O 4 B w u w Y d r a M A 9 N K E F B C Q 8 w y u 8 O c Z 5 c d 6 d j 2 V r w c l n T u E P n M 8 f 8 W a Q X Q = = < / l a t3 R v A = " > A A A B + X i c b V D L S g N B E J z 1 G e N r 1 a O X w S B 4 c d k N m u h B C I j i R Y h g H p A s Y X Y y m w y Z f T D T G w x L / s S L B 0 W 8 + i f e / B s n y R 4 0 s a C h q O q m u 8 u L B V d g 2 9 / G 0 v L K 6 t p 6 b i O / u b W 9 s 2 v u 7 d d V l E j K a j Q S k W x 6 R D H B Q 1 Y D D o I 1 Y 8 l I 4 A n W 8 A b X E 7 8 x Z F L x K H y E U c z c g P R C 7 n N K Q E s d 0 2 w D e 4 L 0 / v Z m f H V a P L f K H b N g W / Y U e J E 4 G S m g D N W O + d X u R j Q J W A h U E K V a j h 2 D m x I J n A o 2 z r c T x W J C B 6 T H W p q G J G D K T a e X j / G x V r r Y j 6 S u E P B U / T 2 R k k C p U e D p z o B A X 8 1 7 E / E / r 5 W A f + G m P I w T Y C G d L f I T g S H C k x h w l 0 t G Q Y w 0 I V R y f S u m f S I J B R 1 W X o f g z L + 8 S O p F y y l Z p Y e z Q u U y i y O H D t E R O k E O K q M K u k N V V E M U D d E z e k V v R m q 8 G O / G x 6 x 1 y c h m D t A f G J 8 / z / q S d A = = < / l a t e x i t > MFE = 25.7 < l a t e x i t s h a 1 _ b a s e 6 4 = " V L i I 7 6 X 1 M r L X 4 s I j d Q 8 0 X W z V H o I = " > A A A B + X i c b V B N S 8 N A E N 3 U r 1 q / o h 6 9 B I v g x Z B U q X o Q C q J 4 E S r Y D 2 h D 2 W w 3 7 d L N J u x O i i X 0 n 3 j x o I h X / 4 k 3 / 4 3 b N g d t f T D w e G + G m X l + z J k C x / k 2 c k v L K 6 t r + f X C x u b W 9 o 6 5 u 1 d X U S I J r Z G I R 7 L p Y 0 U 5 E 7 Q G D D h t x p L i 0 O e 0 4 Q + u J 3 5 j S K V i k X i E U U y 9 E P c E C x j B o K W O a b a B P k F 6 f 3 s z v j o p n d q l j l l 0 b G c K a 5 G 4 G S m i D N W O + d X u R i Q J q Q D C s V I t 1 4 n B S 7 E E R j g d F 9 q J o j E m A 9 y j L U 0 F D q n y 0 u n l Y + t I K 1 0 r i K Q u A d Z U / T 2 R 4 l C p U e j r z h B D X 8 1 7 E / E / r 5 V A c O G l T M Q J U E F m i 4 K E W x B Z k x i s L p O U A B 9 p g o l k + l a L 9 L H E B H R Y B R 2 C O / / y I q m X b L d s l x / O i p X L L I 4 8 O k C H 6 B i 5 6 B x V 0 B 2 q o h o i a I i e 0 S t 6 M 1 L j x X g 3 P m a t O S O b 2 U d / Y H z + A M V a k m 0 = < / l a t e x i t > MFE = 23.2 < l a t e x i t s h a 1 _ b a s e 6 4 = " Z d 0 n K G U Y + Q i z N e y S y f e D w p D T F W M = " > A A A B + X i c b V B N S 8 N A E N 3 4 W e t X 1 K O X Y B G 8 G J J S q x 6 E g i h e h A r 2 A 9 p Q N t t N u 3 S z C b u T Y g n 9 J 1 4 8 K O L V f + L N f + O 2 z U F b H w w 8 3 p t h Z p 4 f c 6 b A c b 6 N p e W V 1 b X 1 3 E Z + c 2 t 7 Z 9 f c 2 6 + r K J G E 1 k j E I 9 n 0 s a K c C V o D B p w 2 Y 0 l x 6 H P a 8 A f X E 7 8 x p F K x S D z C K K Z e i H u C B Y x g 0 F L H N N t A n y C 9 v 7 0 Z X 5 0 W S / Z Z x y w 4 t j O F t U j c j B R Q h m r H / G p 3 I 5 K E V A D h W K m W 6 8 T g p V g C I 5 y O 8 + 1 E 0 R i T A e 7 R l q Y C h 1 R 5 6 f T y s X W s l a 4 V R F K X A G u q / p 5 I c a j U K P R 1 Z 4 i h r + a 9 i f i f 1 0 o g u P B S J u I E q C C z R U H C L Y i s S Q x W l 0 l K g I 8 0 w U Q y f a t F + l h i A j q s v A 7 B n X 9 5 k d S L t l u 2 y w + l Q u U y i y O H D t E R O k E u O k c V d I e q q I Y I G q J n 9 I r e j N R 4 M d 6 N j 1 n r k p H N H K A / M D 5 / A M t s k n E = < / l a t e x i t > MFE = 24.5 < l a t e x i t s h a 1 _ b a s e 6 4 = " z G y K E l a p V 0 k e M / U Z V n P B x r C I x D o = " > A A A B + X i c b V D L S g N B E J y N r x h f q x 6 9 D A b B i 8 t u C F E P Q k A U L 0 I E 8 4 B k C b O T 2 W T I 7 I O Z 3 m B Y 8 i d e P C j i 1 T / x 5 t 8 4 S f a g i Q U N R V U 3 3 V 1 e L L g C 2 / 4 2 c i u r a + s b + c 3 C 1 v b O 7 p 6 5 f 9 B Q U S I p q 9 N I R L L l E c U E D 1 k d O A j W i i U j g S d Y 0 x t e T / 3 m i E n F o / A R x j F z A 9 I P u c 8 p A S 1 1 T b M D 7 A n S + 9 u b y d V Z q W y V u m b R t u w Z 8 D J x M l J E G W p d 8 6 v T i 2 g S s B C o I E q 1 H T s G N y U S O B V s U u g k i s W E D k m f t T U N S c C U m 8 4 u n + A T r f S w H 0 l d I e C Z + n s i J Y F S 4 8 D T n Q G B g V r 0 p u J / X j s B / 8 J N e R g n w E I 6 X + Q n A k O E p z H g H p e M g h h r Q q j k + l Z M B 0 Q S C j q s g g 7 B W X x 5 m T R K l l O x K g / l Y v U y i y O P j t A x O k U O O k d V d I d q q I 4 o G q F n 9 I r e j N R 4 M d 6 N j 3 l r z s h m D t E f G J 8 / x u C S b g = = < / l a t e x i t > MFE = 24.2 < l a t e x i t s h a 1 _ b a s e 6 4 = " l Z u 3 a Y K 8 l a K a n l O 6 z h 6 F j G 7 K d J g = " > A A A B 8 n i c b V D L S g N B E J y N r x h f U Y 9 e F o P g a d k V j f E g B L x 4 j G A e s F n C 7 G Q 2 G T K P Z a Z X C C G f 4 c W D I l 7 9 G m / + j Z N k D 5 p Y 0 F B U d d P d F a e c G f D 9 b 6 e w t r 6 x u V X c L u 3 s 7 u 0 f l A + P W k Z l m t A m U V z p T o w N 5 U z S J j D g t J N q i k X M a T s e 3 c 3 8 9 h P V h i n 5 C O O U R g I P J E s Y w W C l s G v Y Q O B b 3 6 t d 9 c o V 3 / P n c F d J k J M K y t H o l b + 6 f U U y Q S U Q j o 0 J A z + F a I I 1 M M L p t N T N D E 0 x G e E B D S 2 V W F A T T e Y n T 9 0 z q / T d R G l b E t y 5 + n t i g o U x Y x H b T o F h a J a 9 m f i f F 2 a Q 1 K I J k 2 k G V J L F o i T j L i h 3 9 r / b Z 5 o S 4 G N L M N H M 3 u q S I d a Y g E 2 p Z E M I l l 9 e J a 0 L L 6 h 6 1 Y f L S v 0 m j 6 O I T t A p O k c B u k Z 1 d I 8 a q I k I U u g Z v a I 3 B 5 w X 5 9 3 5 W L Q W n H z m G P 2 B 8 / k D 9 G 2 Q X w = = < / l a t e x i t > = 0.85 < l a t e x i t s h a 1 _ b a s e 6 4 = " k D B E / r 8 + O 9 r 3 s Q 6 o d g i M r 5 G E 9 K U = " > A A A B 8 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 h E W j 0 I B S 8 e K 1 h b S E P Z b D f t 0 t 1 N 2 J 0 I J f R n e P G g i F d / j T f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e V E q u A H P + 3 Z K a + s b m 1 v l 7 c r O 7 t 7 + Q f X w 6 N E k m a a s T R O R 6 G 5 E D B N c s T Z w E K y b a k Z k J F g n G t / O / M 4 T 0 4 Y n 6 g E m K Q s l G S o e c 0 r A S k H P 8 K E k N 5 7 b a P S r N c / 1 5 s C r x C 9 I D R V o 9 a t f v U F C M 8 k U U E G M C X w v h T A n G j g V b F r p Z Y a l h I 7 J k A W W K i K Z C f P 5 y V N 8 Z p U B j h N t S w G e q 7 8 n c i K N m c j I d k o C I 7 P s z c T / v C C D + C r M u U o z Y I o u F s W Z w J D g 2 f 9 4 w D W j I C a W E K q 5 v R X T E d G E g k 2 p Y k P w l 1 9 e J Y 8 X r l 9 3 6 / e X t e Z 1 E U c Z n a B T d I 5 8 1 E B N d I d a q I 0 o S t A z e k V v D j g v z r v z s W g t O c X M M f o D 5 / M H 9 f C Q Y A = = < / l a t e x i t > = 0.77 < l a t e x i t s h a 1 _ b a s e 6 4 = " k D B E / r 8 + O 9 r 3 s Q 6 o d g i M r 5 G E 9 K U = " > A A A B 8 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 h E W j 0 I B S 8 e K 1 h b S E P Z b D f t 0 t 1 N 2 J 0 I J f R n e P G g i F d / j T f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e V E q u A H P + 3 Z K a + s b m 1 v l 7 c r O 7 t 7 + Q f X w 6 N E k m a a s T R O R 6 G 5 E D B N c s T Z w E K y b a k Z k J F g n G t / O / M 4 T 0 4 Y n 6 g E m K Q s l G S o e c 0 r A S k H P 8 K E k N 5 7 b a P S r N c / 1 5 s C r x C 9 I D R V o 9 a t f v U F C M 8 k U U E G M C X w v h T A n G j g V b F r p Z Y a l h I 7 J k A W W K i K Z C f P 5 y V N 8 Z p U B j h N t S w G e q 7 8 n c i K N m c j I d k o C I 7 P s z c T / v C C D + C r M u U o z Y I o u F s W Z w J D g 2 f 9 4 w D W j I C a W E K q 5 v R X T E d G E g k 2 p Y k P w l 1 9 e J Y 8 X r l 9 3 6 / e X t e Z 1 E U c Z n a B T d I 5 8 1 E B N d I d a q I 0 o S t A z e k V v D j g v z r v z s W g t O c X M M f o D 5 / M H 9 f C Q Y A = = < / l a t e x i t > = 0.77 < l a t e x i t s h a 1 _ b a s e 6 4 = " k D B E / r 8 + O 9 r 3 s Q 6 o d g i M r 5 G E 9 K U = " > A A A B 8 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 h E W j 0 I B S 8 e K 1 h b S E P Z b D f t 0 t 1 N 2 J 0 I J f R n e P G g i F d / j T f / j d s 2 B 2 1 9 M P B 4 b 4 a Z e V E q u A H P + 3 Z K a + s b m 1 v l 7 c r O 7 t 7 + Q f X w 6 N E k m a a s T R O R 6 G 5 E D B N c s T Z w E K y b a k Z k J F g n G t / O / M 4 T 0 4 Y n 6 g E m K Q s l G S o e c 0 r A S k H P 8 K E k N 5 7 b a P S r N c / 1 5 s C r x C 9 I D R V o 9 a t f v U F C M 8 k U U E G M C X w v h T A n G j g V b F r p Z Y a l h I 7 J k A W W K i K Z C f P 5 y V N 8 Z p U B j h N t S w G e q 7 8 n c i K N m c j I d k o C I 7 P s z c T / v C C D + C r M u U o z Y I o u F s W Z w J D g 2 f 9 4 w D W j I C a W E K q 5 v R X T E d G E g k 2 p Y k P w l 1 9 e J Y 8 X r l 9 3 6 / e X t e Z 1 E U c Z n a B T d I 5 8 1 E B N d I d a q I 0 o S t A z e k V v D j g v z r v z s W g t O c X M M f o D 5 / M H 9 f C Q Y A = = < / l a t e x i t > = 0.77Target Structure A A A U G U G A A U G A A A A A U A U U A U A U A ATo validate the effectiveness of our proposed D-PBEMO framework, we first compare the performance of D-PBNSGA-II and D-PBMOEA/D against three state-of-the-art PBEMO algorithms, I-MOEAD-PLVF [36] , I-NSGA2/LTR [38] , IEMO/D [58] .For the synthetic benchmark test problems, the comparison results of ϵ ⋆ (S) and ε(S) in Tables A5 and A6 have demonstrated the competitiveness of our proposed D-PBEMO algorithm instances. In particular, D-PBNSGA-II and D-PBMOEA/D have achieved the best metric values in 77 out of 96 comparisons according to the Wilcoxon rank-sum test at the 0.05 significant level. In Figure 4 , the box plots of the ranks derived from the Scott-Knott test of D-PBNSGA-II and D-PBMOEA/D compared against the other three peer algorithms further consolidate our observation about the effectiveness of our D-PBEMO framework for searching for SOI within a given computational budget. From the plots of the final non-dominated solutions obtained by different algorithms shown in Figures A8 to A9, we can see that the solutions found by our D-PBEMO algorithm instances are more concentrated on the 'golden point' while the others are more scattered. Further, the superiority of our proposed D-PBEMO algorithm instances becomes more evident when tackling problems with many objectives, i.e., m ≥ 3. Similar observations can be found in the scientific discovery problems, as the results shown in Tables A22, A23, A24 and A25. For the RNA inverse design tasks, the sequences identified by our proposed D-PBEMO algorithm instances have a good match regarding the targets as the selected results shown in Figure 5 (full plots in Figures A14 and A15 ). As for the PSP problems, from the results in Figure 6 , it is clear to see that D-PBEMO algorithm instances significantly outperform the other peers. The protein structures predicted by our algorithms are more aligned with the native protein structure. Full results can be found in Appendix F. 29) of appendix, the better performance achieved by the corresponding algorithm.The consultation module, which learns the DM's preferences from their feedback, is one of the most important building blocks of our proposed D-PBEMO framework. To validate its effectiveness, we designed a D-PBEMO variant (denoted as D-PBEMO-DTS) that uses the double Thompson sampling (DTS) widely used in conventional stochastic dueling bandits [62] as an alternative of our proposed clustering-based stochastic dueling bandit algorithm. From the results in Tables A7 and A8 , it is clear to see that D-PBEMO-DTS is always outperformed by our three D-PBNSGA-II and D-PBMOEA/D. This observation can be attributed to the ineffectiveness of the traditional dueling bandit algorithms for tackling a large number of arms. In our PBEMO context, DTS requires at least thousands comparisons when encountering more than 100 candidate solutions. This is not feasible under the limited amount of computational budget. We envisage the same results will be obtained when considering other dueling bandits variants such as [74, 34] .Further, we designed another variant (denoted as D-PBEMO-PBO) that uses a parameterized preference learning model in Bayesian optimization [25] as alternative in the consultation module. From the comparison results shown in Tables A7 and A8 , we find that the performance of D-PBEMO-PBO is competitive on problems with a small number of objective (m = 2). However, its performance degenerate significantly with the increase of the number of objectives. This can be attributed to the exponentially increased search that renders D-PBEMO-PBO ineffective by suggesting too many solutions outside of the region of interest. In contrast, the clustering strategy in our proposed method strategically and significantly narrow down the amount of comparisons without compromising the learning capability. Additionally, the enlarged search space also makes the model selection in D-PBEMO-PBO difficult. Our proposed method, on the other hand, learns human preferences directly from their feedback, thus is scalable to a high-dimensional space.This section discusses the sensitivity study of two hyperparameters of the D-PBEMO framework.K is the parameter that controls the number of subsets used in our clustering-based stochastic dueling bandits algorithm. From the results shown from Tables A11 and A12 considering K = {2, 5, 10}, it is interesting to note that D-PBEMO is not sensitive to the setting of K when the dimensionality is low (i.e., m = 2). This implies that we can achieve a reasonably good performance even when using a smaller K, i.e., a coarser-grained approximation of SOI. By doing so, we can further improve the efficiency of our D-PBEMO framework.As the dimensionality increases (i.e., m = {3, 5, 8, 10}), the population size will increase. Generally, as is shown in Table A13 ˜A20, with larger populations, a higher K tends to yield better results, aligning with our intuition. Furthermore, our significance analysis across 20 repeated experiment (Figure 4 ) reveals that the optimal K does not show significant differences in performance. In summary, K does not significantly impact the performance of our proposed D-PBEMO framework. For most problems, we do not recommend choosing a very small/large K (e.g., K = 2, K = N ), as it may inefficiently narrow down the ROI.To a certain extent, the threshold ε plays an important role for controlling the budget of consulting the DM. From the results shown in Tables A9 and A10 , we find that all three settings of ε = {10 -1 , 10 -3 , 10 -6 } have shown comparable results. However, a too large ε = 10 -1 may lead to a premature convergence risk. On the other hand, a too small ε = 10 -6 may be too conservative to terminate. This renders more consultation iterations, thus leading to a larger amount of cognitive workloads to DMs. In contrast, we find that D-PBEMO algorithm instances can converge with less than 10 consultation iterations with ε = 10 -3 . 
limitations: 
 Our proposed D-PBEMO directly leverages DM's feedback to guide the evolutionary search for SOI, which neither relies on any reward model nor cumbersome model selection. However, there are several potential limitations of D-PBEMO that warrant discussion here:• The regret analysis of our proposed clustering-based stochastic dueling bandits is for the optimal subset, i.e., the region of interest on the PF. It is not yet directly applicable to identify the exact optimal solution of interest. As part of our future work, we will work on efficient algorithms and theoretical study on the best arm identification in the context the preference-based EMO. • This paper only analyzes the regret of the consultation module. How to further analyze the convergence of the D-PBEMOas a whole remains unknown. This will also lead to the next step of our research. In particular, if it is successful, we may provide a radically new perspective to analyze the convergence of evolutionary multi-objective optimization algorithms. 
conclusion: 
 This paper introduced the D-PBEMO framework that is featured in a novel clustering-based stochastic dueling bandits algorithm. It enables learning DM's preferences directly from their feedback, neither relying on a predefined reward model nor cumbersome model selection. Additionally, we derived a unified probabilistic format to adapt the learned preference to prevelant EMO algorithms. Meanwhile, such probabilistic representation also contributes to a principled termination criterion of DM interactions. Experiments demonstrate the effectiveness of our proposed D-PBEMO algorithm instances. In future, we will investigate more principled approaches to obtain K subsets, such as fuzzy clustering with overlapping. Further, we plan to extend this current K-armed bandits setup to a best arm identification. By doing so, we can directly obtain the solution of interest, with a potential explainability for MCDM. Moreover, we will extend our theoretical analysis of the termination criterion to a convergence analysis of PBEMO, even generalizable to the conventional EMO. Last but not the least, we will collaborate with domain experts to promote a emerging 'expert-in-the-loop' scientific discovery platform, contributing to the prosperity of AI for science. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset. 
 
 Appendix: In the context of MO, to align with DM's preference, existing methods fall into three main categories: the a posteriori methods, the a priori methods, and the interactive methods [12] . The a posteriori methods first generates extensive efficient solutions through MO algorithms, then ask DM to select SOI from these solutions. In this way, DM is involved only when solution generation from the optimization module is finished, where the decision making process can be finished according to multi-criterion decision analysis [26] or certain SOI analysis [13, 6, 40] . However, the absence of DM's preferences in solution generation can paradoxically complicate pinpointing the SOI, and the sheer number of generated solutions can be cognitively overwhelming for DM to choose from. The a priori method, leverages pre-defined DM's preference to guide the search of SOI [7, 42, 57, 37] .In other words, the decision making happens before optimization. This is more related to offline preference learning such as reinforcement learning from human feedback (RLHF) [38] . However, given the black-box nature of real-world problems, eliciting reasonable preferences a priori can be controversial. [39, 54] pointed out that the a priori method may lead to disruptive preferences and result in faulty decisions. Moreover, elaborating the DM labeled dataset used for offline preference learning such as RLHF is also a difficult task. In contrast, the interactive method also known as preference-based evolutionary multi-objective optimization (PBEMO) [39] presents a valuable opportunity for the DM to gradually comprehend the underlying black-box system and consequently refine user preference information [44, 15, 8, 36, 59, 38] .The consultation module in PBEMO collects DM's preference by actively querying DM's preference towards recommended candidates. Based on the requirements of human feedback, methods for consultation can be categorized into the following three types, as also presented in Figure 1 (b).• The first type requires DM to provide scores or rankings for a bunch of (typically more than three) solutions. In [38] , a rank-net was introduced to conduct consultation in PBEMO. In [27] , a ranking model was required in the context of RLHF. In addition, as presented in [50] , relative preference can only be leveraged by ranking on more than three candidates. However, providing reward or ranking on a bunch of solutions will not only increase the workload of DMs, but will also introduce uncertainty and randomness into optimization processes due to the limited capacity of DMs [48] .• Algorithms in the second type rely on a parameterized transitivity model or a structured utility function for dueling feedback [5] , such as the Bradley-Terry-Luce model [11] which has been widely explored in the literature [32, 24, 11, 29, 71] . These algorithms can be traced back to 1982, when [32] introduced the UTA (UTilités Additives) method for deducing value functions based on a provided ranking of reference set. Following up work included [24] that employed pairwise preference to predict a ranking for potential labels associated with new training examples, [11] that utilized GP for pairwise preference learning (PGP) within a Bayesian framework, and [9] that proposed the ListNet which was an NN-based learning-to-rank method to model preference feedback. In addition, [29] extended the work in [9] to handle multi-user scenarios by introducing a weight vector for each user and combining multiple preference latent functions. More recently, [25] proposed a novel pairwise preference learning method using the concept of Bayesian optimization based on a structured surrogate model assumption, named preferential Bayesian optimization (PBO). Moreover, in [71] , a comparative study towards four prominent preference elicitation algorithms was conducted, and results shown that ranking queries outperformed the pairwise and clustering approaches in terms of utility models ad human preference.• Algorithms that belong to the third type consider human feedback as stochastic events such as Bernoulli trails, enabling quantifying randomness regarding the DM's capability. Dueling bandits algorithms are known for its efficiency towards interactive preference learning [65] .In recent years, work like [75, 62, 63] has improved dueling bandits algorithms in both querying efficiency and computational complexity. Further investigations on dueling bandits are presented in A.3.Algorithms in the first two types burden the development of PBEMO, as reward/ranking method fails to take human capability into account and will navigate solutions to an inaccurate direction, while the parameterized method relies heavily on model selection. It is well known that, in real-world problems, selecting an appropriate model is as difficult as solving the original problems. We appreciate the model-free settings in dueling bandits algorithms.The dueling bandits problem involves a sequential decision-making process where a learner selects two out of K "arms" in each round and receives binary feedback about which arm is preferred. Following [4] , dueling bandits algorithms can be classified into three categories: MAB-related, merge sort/quick sort, and tournament/challenge. In this paper, we focus on traditional dueling bandit algorithms falling in the MAB-related category. comprising four distinct methodologies for handling pairwise comparisons.• The first method is known as explore then commit (ETC), which is utilized by algorithms such as interleaved filtering (IF) [65] , beat the mean (BTM) [66] and SAVAGE [60] . ETC methods kick out solutions that are unlikely to win, but this approach may lead to lower accuracy of predictive probability.• The second method involves using the upper confidence bound (UCB), for example relative upper confidence bound (RUCB) [74] , MergeRUCB [73] , and relative confidence sampling (RCS) [75] . MergeRUCB, an extension of RUCB, is particularly designed for scenarios with a large number of arms. RCS combines UCB and Beta posterior distribution to recommend one arm for each duel in each iteration step.• The third method employs Thompson sampling, as demonstrated by double Thompson sampling (DTS) [62] and MergeDTS [35] . Similar to MergeRUCB, MergeDTS is designed for dealing with a substantial number of arms. It is worth nothing that UCB methods assume the existence of a Condorcet winner, whereas Thompson sampling methods assume a Copeland winner, representing a fundamental distinction between these two types.• The fourth method involves using the minimum empirical divergence, as introduced by relative minimum empirical divergence (RMED) [34] and deterministic minimum empirical divergence (DMED) [28] . RMED and DMED employ KL divergence as a metric to evaluate candidate arms.Overall, these four methods represent different approaches to pairwise comparison in traditional dueling bandit algorithms. However, the key bottleneck here is the sourcing number of DM queries required when considering a population solutions in PBEMO.To our knowledge, there is no clustering-based dueling bandit algorithms. While in the field of multi-armed bandit (MAB), there has already exists clustered MABs which consider the correlation of arms [47, 46] , they relied heavily on complex model selection that is not applicable for real-world scenarios.Additionally, it is noted that multi-objective multi-armed bandit (MOMAB) is potential to address PBEMO problems. However, as pointed out in [31, 56] , MOMABs belong to the a posteriori MO method, recalling A.1. Specifically, the goal of MOMABs is to find an optimal multi-objective arm by sampling potential winners. However, MOMABs do not involve optimization for PF generation nor active query for preference learning. Thus it's not feasible to use MOMABs as our peer algorithms or consultation module.Proof of Theorem 3.3. We follow the proof of double Thompson sampling for dueling bandits [62] , different from which the clustering operation will lead to a non-stationary environment. As shown subsequently, the Assumption 3.2 is important to build an auxiliary problem that is stationary, facilitating to apply the results in [62] .According to the property of Beta distribution, the estimated mean of probability for subset i beats subset j is pi,j (t) = bi,j (t) bi,j (t-1)+bj,i(t-1) . Unfortunately, the pairwise comparison feedback does not follow a stationary distribution. Specifically, p i,j represents winning probability between the best solutions in Si and Sj , while pi,j (t) is computed from comparison among different solutions in the two subsets due to limited knowledge of the best solutions at the beginning. As a result, the reward distribution is drifting [10] . To handle this, we introduce an auxiliary problem with stationary reward distributions. Based on comparison principles, the number of plays on each subset in the original problem can be upper bounded by the ones of the auxiliary problem. Note that the auxiliary problem are introduced merely for theoretical analysis.Without loss of generality, we assume S1 to be S * . Since we are most interested in locating the optimal subset, we define the underestimation of winning probability of p 1,j as p 1,j , which is the winning probability between: i) the solution x 1 ∈ S1 with the lowest Copeland score over all solutions in Sj , and ii) the solution x j ∈ Sj with the highest Copeland score over all solutions in S1 , j = 2, . . . K. Likewise, we define p i,j as the winning probability between: i) the solution x i ∈ Si with the lowest Copeland score over all solutions in Sj , and ii) the solution x j ∈ Sj with the highest Copeland score over all solutions in S1 , p i,j > 0.5, ij = 1, . . . K. For p i,j < 0.5, denote p i,j = 1p j,i . Lastly, we denote p i,i = 0.5, ∀i = 1, . . . , K. According to Assumption 3.2, the optimality remains, i.e., the optimal subset in the context of p ij is also optimal in the context of p ij .With the auxiliary problem, we introduce the virtual Thompson sampling strategy that generates samples asθ ′ j,i ∼ Beta(b ′ j,i (t) -1, b ′ i,j (t) -1), where b ′ i,j is the number of time-slots when subsets Si beats Sj under p i,j . It is easy to verify two facts in expectation:i) b ′ i,j ≤ b i,j , and ii) b ′ j,i ≥ b j,i . Consequently, when p i,j > 0.5 or i = 1, for samples θ i,j (t) ∼ Beta(b i,j + 1, b j,i + 1) and θ ′ i,j (t) ∼ Beta(b ′ i,j + 1, b ′ j,i + 1), we have θ i,j dominates θ ′ i,j in expectation [10] .In what follows, we are going to quantify the number of plays of subset pair Si and Sj To avoid ambiguity, let N i,j (t) denote the case where S′ = Si and S′′ = Sj .When the first subset for comparison has been determined as Si , the number of plays for Sj , j ̸ = i, isEQUATIONWhen p i,j < 0.5, we consider the following facts: i) θ j,i are sampled from Beta(b j,i + 1, b i,j + 1)when j ̸ = i; ii) θ i,i = 0.5; iii) θ j,i > θ i,i since subset Sj is selected by sampling for comparison. From these observations, the problem is equivalent to a multi-arm bandit problem where the best arm is indexed by i. Then, the left item of equation ( 6) can be further divided byEQUATIONWhen p j,i < 0.5, if p j,i > 0.5, we have p j,i > p j,i , thus θ j,i dominates θ ′ j,i in expectation. Therefore, we have P N j,i (t -1) ≤ log T DKL(c∥0.5) ≤ P N j,i (t -1) ≤ log T DKL(c∥0.5) , where N j,i (t -1) is the number of plays of pair Sj against Si using the virtual Thompson sampling strategy. Otherwise, if p j,i < 0.5, we have p j,i < p j,i , thus P (p j,i < c) ≥ P p j,i < c which follows the regular form as Lemma 1 in [62] . According to the concentration property of Thompson samples [2] , we haveEQUATIONAs shown in [2] , for any ϵ ∈ (0, 1], c can be chosen such that D KL (c∥0.5) = D KL (p j,i ∥0.5)/(1 + ϵ), and 1/D KL (c∥p j,i ) = O(1/ϵ 2 ). Therefore the left item in equation ( 6) is bounded from above byEQUATIONTo account for the second term in equation ( 6), the concentration property of relative upper confidence bound [74, 62] will be used. For p j,i > 0.5, define ∆ j,i = p j,i -0.5 > 0, thus the right side of equation ( 6) can be divided intoEQUATIONNote that the subset Sj can be selected as S′′ only if l j,i < 0.5. For the right item in equation (10) ,N j,i (t -1) ≥ 4α log T ∆j,iis sufficient to yield l j,i + ∆ j,i ≥ u j,i , finally we have u j,i ≤ p j,i . Applying concentration properties in Lemma 6 of [62] , we haveT t=1 P S′′ = Sj , p j,i > 0.5, N j,i (t -1) ≥ 4α log T ∆ j,i ≤ T t=1 P (u i,j ≤ p i,j ) = O(1).As a result, we can also have a upper bound of the left item in equation (10) asEQUATIONAltogether, for j ̸ = i, we haveE[N ′ j,T ] = T t=1 P S′′ = Sj ≤ (1 + ϵ) log T D KL (p j,i ∥0.5) + 4α log T ∆ j,i + O 1 ϵ 2 .For the case where S′ and S′′ are selected to be the same subsets, an upper bound of O(K) has been proven in Lemma 3 of [62] . Therefore we omit the proof of this case for brevity.Consider the DM's preference distribution is determined by the density ratio of winning distribution over losing distribution, in other word, preference on solution x is determined by p v (x)/p ℓ (x). From consultation module, we have obtained the winning and losing time vectors v and ℓ of all candidate solutions, which in turn gives us a density estimation of p v and p ℓ . We are going to perform densityratio estimation based upon v and ℓ to obtain the density ratio, namely the preference distribution p r (x), which can be mathematically formulated asEQUATIONWe use finite-order moment matching approach without conducting complex density estimation of each p v (x) and p ℓ (x). Denote a vector-valued nonlinear function ϕ : v → R k . Consider the first k-th order moments, we have ϕ(x) = x(1) , . . . , x(k) , where x(k) stands for element-wise k-th power of x. Then, the moment matching problem for density-ratio estimation can be formulated asEQUATION)Equivalently, we can reformulate the problem byEQUATIONwhere ⟨•⟩ denotes the inner product. Viewing each component in equation ( 14) as the expectation operation of function ϕ(x)p r (x) over distribution p v (x) and p ℓ (x), we use sample averages to estimate the expectation [52] . To this end, we construct two sample vectorsΦ v = ϕ(x 1 v ), . . . , ϕ(x Nv v ) and Φ ℓ = ϕ(x 1 ℓ ), . . . , ϕ(x N ℓ ℓ )based on the wining and losing time vectors v and ℓ. Specifically xi v and xi ℓ are solutions that has won or lost one time during consultation. Note that a solution may appear in Φ v or Φ ℓ more than one time since its winning or losing time is greater than 1. N v and N ℓ are the number of winning and losing solutions, calculated by the sum of all winning or losing times for all solutions. Correspondingly, we denote p r = p r (x 1 ℓ ), . . . , p r (x N ℓ ℓ ) . An estimator pr for p r can be calculated by solving the following problem:EQUATIONwhere 1 Nv is the N v -dimensional vector whose elements are all valued by 1. The optimality is reached by when the derivative of equation ( 15) is zero, i.e.,EQUATIONwhere 0 k is the k-dimensional vector whose elements are all valued by 0. Correspondingly, we haveEQUATION)Note that pr is the estimation of true density ratio p r , namely the probability density function, on solutions xi ℓ , i = 1, . . . , N ℓ . By assuming Gaussian distribution around SOI, we can further approximate the mean and variance byEQUATIONwhere we dismiss the covariance between any two dimensions in solution space for brevity. Remark 4. The sample averages can also be performed by first estimating the winning and losing density distributions based on v and ℓ, then sample solutions on them to increase the sample numbers. This will improve the estimation in equation ( 14) at the cost of complex density estimation. In addition, perturbations on solutions can be introduced for legal inverse operation of Φ ⊤ ℓ Φ ℓ in equation (17) .Proof of Theorem 3.4. Form Theorem 3.3, when N consult increases, the clustering dueling bandits can grasp DM's preference towards pairwise comparisons with increasing accuracy. Therefore, the convergence of preference distribution Pr(x) will be mainly influenced by the behaviors of EMO algorithms as well as the landscape of MO problems. To this end, we consider two extreme cases: i) the population distribution of candidate solutions from EMO generators remains unchanged, and ii) the population of solutions are steered into shrinking regions by EMO algorithms.For the first case, we let the unchanged distribution be N (x | x * u , Σu ), ∀τ > N u , where N u > 1 denotes a threshold number of consultation sessions. The convergence is presented by the following truth:Pr(x) = 1 Z Nu τ =1 1 στ N (x | x * τ , Στ ) + 1 Z N consult τ =Nu+1 1 σu N (x | x * u , Σu ) = 1 Nu τ =1 1 στ + (N consult -N u ) 1 σu Nu τ =1 1 στ N (x | x * τ , Στ ) + 1 σu (N consult -N u )N (x | x * u , Σu ) Nu τ =1 1 στ + (N consult -N u ) 1 σu . Note that 1 σu > 0 and is constant. When N consult ≫ N u , we have Pr(x) → 0 + N (x | x * u , Σu ), therefore Pr(x) is convergent to N (x | x * u , Σu ).For the second case, shrinking regions with increasing N consult result in a uniform distribution in the context of Gaussian distribution around SOI. In this case winning and losing probability are close to each other. As a result, a large στ will be computed. Therefore this distribution hardly contributes to the mixture distribution. Consequently, with shrinking regions, effect of consultation sessions decreases to zero, thus Pr(x) convergent ultimately.Based on these observations, two extreme cases can result in a convergent stable mixture distribution, covering all other cases in PBEMO between the two extreme ones. The proof is complete.The structural pseudocode for D-PBEMO is shown in Algorithm 1. The D-PBEMO framework is made up of three component, consultation, preference elicitation and optimization module. In this paper, our contribution is mainly in consultation and preference elicitation module (highlighted by rhd). In consultation module, we proposed a clustering-based stochastic dueling bandit algorithm (pseudocode can be referenced in Algorithm 2). While in this section, we will delineate the step-by-step process of three different preference elicitation module. The code of our algorithms and peer algorithms are available at https://anonymous.4open.science/r/DPBEMO/.This section will discuss how the learned preference information can be used in NSGA-II [17] . Based on [16] , solutions from the best non-domination levels are chosen front-wise as before and a modified crowding distance operator is used to choose a subset of solutions from the last front which cannot be entirely chosen to maintain the population size of the next population, the following steps are performed:Step 1: Before the first consultation session, the NSGA-II runs as usual without considering the preference information.Step 2: If it is time to consult user preferences (e.g., when we have evaluated the population for 50% of the total generation), we will update the preference distribution Pr(x) defined in equation ( 4).Step 3: Between two interactions, the crowding distance of each solution will be evaluated by the predicted preference distribution Pr(x) learned from the last consultation session. Following [36] , MOEA/D [68] is designed to use a set of evenly distributed weight vectors W = {w i } N i=1 to approximate the whole PF. The recommendation point learned from the consultation module is to adjust the distribution of weight vectors. The following four-step process is to achieve this purpose.Step 1: Before the first consultation session, the EMO algorithm runs as usual without considering any preference information.Step 2: If time to consult (e.g., when we have evaluated the population for 50% of the total generation), the whole population is fed to consultation module and the three outputs will be recorded and used to update the predicted preference distribution for the current population as Pr(x) defined in Equation (4).Step 3: The weight vectors W = {w i } N i=1 will be projected to the preference distribution Pr by using the weighted sum of inverse Gaussian distribution. For a specific weight vector w i :EQUATIONwhere i = 1, 2, . . . , N and w * τ is the corresponding weight vector of x * τ in Pr(x) (Equation (4)). Output the adjusted w i′ as new weight vectors W ′ for next generation.This section will discuss how to utilize DTS as consultation module in ablation experiment. In each consultation session, the budget for pairwise comparison is limited to 100. The optimization module is Pareto-based EMO algorithm, i.e., NSGA-II [17] . Based on [16] , solutions from the best non-domination levels are chosen front-wise as before and a modified crowding distance operator is used to choose a subset of solutions from the last front which cannot be entirely chosen to maintain the population size of the next population, the following steps are performed:Step 1: Before the first consultation session, the NSGA-II runs as usual without considering the preference information.Step 2: If it is time to consult (e.g., when we have evaluated the population for 50% of the total generation), then the whole population will be fed into the DTS and the optimal arm will be Algorithm 2 Clustering-based Stochastic Dueling Bandit (C-DTS) u i,j = bi,j bi,j +bj,i + α log t/(b i,j + b j,i ), l i,j = bi,j bi,j +bj,iα log t/(b i,j + b j,i ), if i ̸ = j, and u i,i = ℓ i,i = 1/2, ∀i; // let x/0 := 1 f or any x, when b i,j + b j,i = 0.Input: T ∈ {1, 2, . . .} ∪ {∞}, N arms, K subset. 1: v ← 0 N , ℓ ← 0 N ; 2: B ← 0 K×K , u i,j = 1, l i,j = 0 for i,ζi ← 1/(K -1) j̸ =i I(u i,j > 1/2); // Upper bound of the normalized Copeland score 7:C 1 ← {i : ζi = max i ζi }; 8:for i, j = 1, . . . , K with i < j do Randomly select an arm x ′ from S′ , and x ′′ from S′′ . And observe the result. 19 :if x ′ ≻ p x ′′ then 20: b ′,′′ ← b ′,′′ + 1 and v ′ ← v ′ + 1, ℓ ′′ ← ℓ ′′ + 1; 21: else 22: b ′′,′ ← b ′′,′ + 1 and v ′′ ← v ′′ + 1, ℓ ′ ← ℓ ′ + 1; 23:end if 24: end for Output: { S⋆ , v, ℓ}, where S⋆ = argmax Si ζi recorded and used to update the predicted preference distribution Pr(x) as Equation (4) for the current population.Step 3: Between two interactions, the crowding distance of each solution will be evaluated by the predicted preference distribution Pr(x) learned from the last consultation session.In this section, PBO [25] is utilized as the consultation module for D-PBEMO. The PBO is initialized with 4 pairwise comparisons and its query budget is limited to 10. Also the acquisition function is set to Thompson sampling. The optimization module is chosen to be decomposition-based EMO algorithm, i.e., MOEA/D [68] . Following [36] , the decomposition-based EMO algorithm (e.g., MOEA/D [68] ) is designed to use a set of evenly distributed weight vectors W = {w i } N i=1 to approximate the whole PF. The recommendation point learned from the consultation module is to adjust the distribution of weight vectors. The following four-step process is to achieve this purpose.Step 1: Before the first consultation session, the EMO algorithm runs as usual without considering any preference information.Step 2: If time to consult (e.g., when we have evaluated the population for 50% of the total generation), the whole population is fed to consultation module and the recommendation solution will be recorded and used to update the predicted preference distribution for the current population Pr(x) as Equation ( 4).Step 4: The weight vectors W = {w i } N i=1 will be projected to the preference distribution Pr by using the weighted sum of inverse Gaussian distribution as defined in Equation (19) . The adjusted weight vectors W ′ = {w i′ } N i=1 will be the new weight vectors for next generation.We present the lookup table for key notations in the following table 2 .This section introduces the problem definitions of two real-world scientific discovery problems considered in our experiments. Further, we summarize the key parameter settings in our experiments.An RNA sequence x of length n is specified as a string of base nucleotides x 1 x 2 . . . x n where x i ∈ {A, C, G, U } for i = 1, 2, . . . , n. A secondary structure P for x is a set of paired indices where each pair (i, j) ∈ P indicates two distinct bases x i x j ∈ {CG, GC, AU, U A, GU, U G} and each index from 1 to n can only be paired once. For example, given a target secondary structure "(...)", the predicted sequence x 1 x 2 x 3 x 4 x 5 should satisfied that the 1 st and 5 th nucleotides in paired set {CG, GC, AU, U A, GU, U G}.Reference from [55, 70] , our inverse RNA design problem adopts two objective functions.Stability The ensemble of an RNA sequence x is the set of all secondary structures that x can possibly fold into, denoted as Y(x). The free energy ∆G(x, y) is used to characterize the stability of y ∈ Y(x). The lower the free energy ∆G(x, y), the more stable the secondary structure y for x.The structure with the minimum free energy (MFE) is the most stable structure in the ensemble, i.e., MFE structure.EQUATIONNote that ties for argmin are broken arbitrarily, thus there could be multiple MFE structures for given x. Technically, M F E(x) should be a set. In our experiment, the MFE is calculate by ViennaRNA3 package.The second objective function is to assess the similarity between the best secondary structure of our predicted sequence and the target structure.EQUATIONwhere d is the Hamming distance between our predicted structure and target structure, and σ ∈ [0, 1]. For example, when the target structure is "(...)" and the predicted secondary structure is ".....", d = 2 and σ = (5 -2)/5 = 0.6. The bigger the similarity σ, the more precise our predicted structure is.Since our MO problems are minimization problems, the second objective should be:EQUATIONProtein structure prediction (PSP) is the inference of the three-dimensional structure of a protein from its amino acid sequence -that is, the prediction of its secondary and tertiary structure from primary structure.To computationally address PSP, the initial step involves constructing protein conformations using a suitable model. Traditional Protein Data Bank files contain Cartesian coordinates of all atoms in a protein, often totaling over 1000 atoms for a modest 70-amino acid sequence. This abundance of coordinates renders Protein Backbone Exploration with EMO algorithms nearly impracticable.To mitigate this, protein structures are represented using torsion and dihedral angles, obtained by converting atom coordinates along consecutive bonds, effectively reducing the search space while retaining essential structural information. Protein backbones are primarily defined by three torsion angles: Φ (around the -N -C αbond), Ψ (around the -C α -Cbond), and Ω (around the -C -Npeptide bond). Additionally, each amino acid residue may feature varying numbers of sidechain torsion angles (χ i , i ∈ 1, 2, 3, 4) depending on its structure. Despite this, the conformational search space remains extensive. To address this, secondary structures and backbone-independent rotamer libraries [21] are utilized to further constrain backbone and side-chain angles, respectively. These libraries also integrate biological insights from protein secondary structures. Moreover, this approach necessitates less optimization space compared to those relying on contact maps. For instance, reconstructing the main and side chains of a 50-amino acid sequence requires only 250 -300 angles using dihedral angles, whereas contact maps demand 2500 distances for the main chain alone.In this paper, we consider PSP tasks as a four-objective optimization problem. Each objective function is an energy function that determines a regulatory property of a protein.CHARMM The first objective function f 1 is CHARMM that calculates several energy terms that determine protein properties. The corresponding energy function is formulated as:EQUATION)where b is the bound length, b 0 is the ideal length, θ is the angle formed by three atoms involved in two connected bonds, ϕ is the torsion angle, ω is the improper angle, u is the distance between two atoms of nonbonded interactions in an angle, k b , k θ , k ϕ , δ, n, k ω , ω 0 , k u , u 0 are constants. The bonds, angles dihedrals, improper, and Urey-Bradley terms belong to the bond term, while the non-bond term includes Van-der-Waals and charge terms. Apart from these, Van-der-Waals and charge terms are used to calculate the Van der Waals force and electrostatic energy between a pair of atoms (i, j), respectively.dDFIRE The second one f 2 is dDFIRE that follows two interactions of atom pairs, i.e., that between polar and nonpolar atoms, and that between polar and non-hydrogen-bonded polar atoms, to get excellent solution for potential. The energy of atoms (p, q) pair is calculated as:EQUATIONwhere N o (θ p , θ q , θ p q, r) is the number of polar atom pair (p, q) at distance r. θ p , θ q and θ pq are orientation angles of polar atoms. R is the gas constant, temperature T is set as 300 K, r is the distance between an atom pair, the cutoff distance r cut is set to 14.5 Å, and ∆r(∆r cut ) is the bin width at r (r cut ). The value of parameter α is proven to be 1.51 [64] . The total dDFIRE potential is the summation of energy of all possible atom pairs (p, q) in the protein structure:f 2 = E D =rpq,θp,θq,θpq μD (r pq , θ p , θ q , θ pq ).Rosetta The third objective function f 3 is Rosetta, a composite energy energy function based on physics and knowledge, each potentials of it is intricately designed:EQUATIONwhere w and Θ are the weight and degree of freedom of each energy term, respectively. P is the protein structure. Details on Rosetta can be referenced in [3] .RWplus The last objective function f 4 is RWplus, whose calculation is the same as that of dDFIRE. However, it emphasizes more about the potential of short-range interactions:EQUATION)where k is the Boltzmann constant and T is the temperature in units of Kelvin. N o (α, β, R) and N e (α, β, R) are the observed and expected number of atom pairs, with α and β atom types within a distance R, respectively. Similarly, N o (A, B, O AB ) and N e (A, B, O AB ) are the observed and expected number of atom vector pairs with (A, B) type within a relative orientation O AB respectively. The type of atom vector pairs between atom types α and β is (A, B). δ(A, B) is 0 when vector pairs A and B are not contact, and 1 vice versa. The total RWplus potential is the sum of these atom pairs' energies.The problem formulation of PSP can be referenced in [69] 4 .This section lists several parameters used in experiments, including the parameters in EMO algorithms, and other parameters we need in D-PBEMO:• • The reference points in different test problems can be referenced in Table A4 . The reference point of real-world problems will be introduced in their experiment results. • While the winning probability is usually known in advance in dueling bandits, it however does not exist in the context of MO. In this paper, to project a MO solution to an arm in the bandit setting, we define the winning probability p i,j of a pair of candidate solutions ⟨x i , x j ⟩, ∀i, j ∈ {1, . . . , N } as [20] :EQUATIONwhere µ(a) = 1/(1 + exp(-a)), Pr denotes the real preference distribution of each arm. For example, we set Pr(x) = N (x|x ⋆ , σ ⋆2 ), where x ⋆ is decision vector of the reference point z * (Table A4 ) and σ ⋆ is a constant value initially set as 0.1. Note that p i,j can take other forms in case it satisfies the conditions in Section 2.2. • The subset number K:K =              10 if m = 2, 8 if m = 3, 12 if m = 5, 14 if m = 8, 18 if m = 10.• The budget for dueling bandits algorithm T is set as 100.• The parameter α in u i,j and l i,j is set as α = 0.6.Wilcoxon rank-sum test To offer a statistical interpretation of the significance of comparison results, we conduct each experiment 20 times. To analyze the data, we employ the Wilcoxon signed-rank test [61] in our empirical study.The Wilcoxon signed-rank test, a non-parametric statistical test, is utilized to assess the significance of our findings. The test is advantageous as it makes minimal assumptions about the data's underlying distribution. It has been widely recommended in empirical studies within the EA community [19] . In our experiment, we have set the significance level to p = 0.05.Scott-Knott rank test Instead of merely comparing the raw ϵ ⋆ (S) and ε(S) values, we apply the Scott-Knott test [45] to rank the performance of different peer techniques over 31 runs on each experiment. In a nutshell, the Scott-Knott test uses a statistical test and effect size to divide the performance of peer algorithms into several clusters. In particular, the performance of peer algorithms within the same cluster is statistically equivalent. Note that the clustering process terminates until no split can be made. Finally, each cluster can be assigned a rank according to the mean ϵ ⋆ (S) or ε(S) values achieved by the peer algorithms within the cluster. In particular, since a smaller ϵ ⋆ (S) and ε(S) value is preferred, the smaller the rank is, the better performance of the technique achieves.In this section, we show the results of our proposed method running on ZDT, DTLZ, and WFG test suites.From the selected plots of the final non-dominated solutions obtained by different algorithms shown in Figure A7 (full results are in Figures A8 to A9 ), we can see that the solutions found by our D-PBEMO instances are more concentrated on the 'golden solution' while the others are more scattered. Furthermore, the superiority of our proposed D-PBEMO algorithm instances become more evident when tackling problems with many objectives, i.e., m ≥ 3. The performance metrics are shown in Table A5 and Table A6 .In this section, we testify the influence of different hyperparameters, including KL threshold ε, and subset number K.We set ε in D-PBEMO-MOEA/D to three different values, ε = {10 -1 , 10 -3 , 10 -6 }. The experiment results of ϵ ⋆ (S) and ε(S) can be referenced in A9 and A10.We set K in D-PBEMO-MOEA/D to three different values running on ZDT test problems (N = 100), K = {2, 5, 10}. The experiment results of ϵ ⋆ (S) and ε(S) can be referenced in A11, A13, A15, A17, A19, A12, A14, A16, A18, and A20.This section shows the experiment result of our proposed method, specifically D-PBEMO-NSGA-II. Our method and peer algorithms run on Eterna100-vienna1 benchmark 5 , which contains 100 RNA sequences. We selectively run our method on 10 short sequences n ∈ [12, 36] . For inverse RNA design problems, the most important task is to predict an accurate sequence whose secondary structure is the same as target structure, f 2 = 0. However, in real-world scenarios, we need to tradE-off between stability and similarity. When f 2 = 0, the stability f 1 can not reach its global optimum in PF. Our D-PBEMO framework can do more than those singlE-objective algorithms for RNA. Users can guide the search of D-PBEMO algorithms and finally reach a point which sacrifices a little similarity but has better stability. So our experiment are divided into two parts. In the first part, D-PBEMO-NSGA-II only focuses on finding the most similar solutions, whose reference point locates on f 2 = 0. In another part, user prefers a solution in the middle of PF, f 2 ∈ (0, 1). The reference points of our two-session experiments can be referenced in A21.The experiment results of running on reference point 1 are listed in A22 and A23. The predicted secondary structures of each algorithm are shown in A14 and A15. Each algorithm repeatedly runs for 10 times. As we can see from these two tables, D-PBEMO-NSGA-II has best performance for 5 times from the perspective of ϵ ⋆ (S) while 6 times from ε(S).The experiment results of running on reference point 2 are listed in A24 and A25. As we can see from these two tables, D-PBEMO-NSGA-II has best performance for 8 times with reference to ϵ ⋆ (S) and ε(S).In this section we listed the results implementing our proposed method, specifically D-PBEMO-NSGA-II on PSP problems. We implement RMSD as the performance metric for PSP problems: where n atom is the total number of matched atoms between the two protein structures and d i is the distance between each pair of atoms. The four energy settings and predicted results of our proposed method are in Table A26 . Other parameters in PSP problems are align with [69] .EQUATIONThe population results are shown in Figure A16 and the secondary structures are dipicted in Figure 6 . The RMSD comparison results are shown in Table A27 . As we can see our proposed method have better convergence and accuracy than synthetic problems. This may be caused by two reasons:• The first one is the PSP problem is only conducted on 4-dimensional objective spaces. In synthetic problems, our proposed method shows better performance results when dimension m ≥ 3 while peer algorithms may collapse.• The second reason is the formulation of PSP problems. In this paper, we adopt utilizing 4 energy function to represent, which are empirically proved to be more accurate than in 1-dimensional objective function [69] . EQUATION)) (-26.7, 0) ⊤ (-45, 0.17) ⊤ UAAAACGAGAAAAACGAA AGUUUUAUCAUGGUUUUAThe column reference point 1 lists our first session experiment (f 2 = 0). The column reference point 2 lists reference points in the second-session (f 2 ∈ (0, 1)). The sample solution is the possible sequence for the given target structure provided by benchmark. G A U A A A A U A U C A Target Structure D-PBNSGA-II I-MOEA/D-PLVF I-NSGA-II/LTR IEMO/D G G C C C A C G G C C A G G G G G A G A C C C C G G G C G C C G C C C G G G G C G C G G C C C G G G G G G G A A A A A C C C C C C C C C C G G C A G A G G G G G C G G G C C C U C G G G C C C G G G G G C G G G U U G G C C C C G C C C C G U U G U G G G G G C C U G A A A A G A G U G A G A G C G C C C C C C G G G C G C G U U U G G C U C C C G U C G G G A G C C G C C U A A G G C G C C C C U C G G G C G G C C U U G G G C G G C C RNA < l a t e x i t s h a 1 _ b a s e 6 4 = " C E q l B + i + p z N i s T e C R r v i T d T 8 j 4 g = " > A A A B / X i c b V D L S g M x F M 3 4 r P U 1 P n Z u g k V w o c O M S u 2 m U B T F j V D B P q A d S i Z N 2 9 A k M y Q Z o Q 7 F X 3 H j Q h G 3 / o c 7 / 8 a 0 n Y W 2 H r h w O O d e 7 r 0 n i B h V 2 n W / r b n 5 h c W l 5 c x K d n V t f W P T 3 t q u q j C W m F R w y E J Z D 5 A i j A p S 0 V Q z U o 8 k Q T x g p B b 0 L 0 d + 7 Y F I R U N x r w c R 8 T n q C t q h G G k j t e z d 2 + u r 4 n H e K R w 1 F e 1 y V H S d w m n L z r m O O w a c J V 5 K c i B F u W V / N d s h j j k R G j O k V M N z I + 0 n S G q K G R l m m 7 E i E c J 9 1 C U N Q w X i R P n J + P o h P D B K G 3 Z C a U p o O F Z / T y S I K z X g g e n k S P f U t D c S / / M a s e 4 U / I S K K N Z E 4 M m i T s y g D u E o C t i m k m D N B o Y g L K m 5 F e I e k g h r E 1 j W h O B N v z x L q i e O l 3 f y d 2 e 5 0 k U a R w b s g X 1 w C D x w D k r g B p R B B W D w C J 7 B K 3 i z n q w X 6 9 3 6 m L T O W e n M D v g D 6 / M H 8 F a S + w = = < / l a t e x i t > MF E = 6.8, = 0.83 < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 + g 7 N h p r 2 1 A I z y e N j / Q k z 4 G J R l Q = " > A A A B + 3 i c b V D L S g M x F M 3 U V 6 2 v s S 7 d B I v g Q o e J S H V T K I r i R q h g H 9 A O J Z N m 2 t B k Z k g y Y h n 6 K 2 5 c K O L W H 3 H n 3 5 i 2 s 9 D W A x c O 5 9 z L v f f 4 M W d K u + 6 3 l V t a X l l d y 6 8 X N j a 3 t n f s 3 W J D R Y k k t E 4 i H s m W j x X l L K R 1 z T S n r V h S L H x O m / 7 w a u I 3 H 6 l U L A o f 9 C i m n s D 9 k A W M Y G 2 k r l 2 8 u 7 m u n C D X Q c c d x f o C V 1 D X L r m O O w V c J C g j J Z C h 1 r W / O r 2 I J I K G m n C s V B u 5 s f Z S L D U j n I 4 L n U T R G J M h 7 t O 2 o S E W V H n p 9 P Y x P D R K D w a R N B V q O F V / T 6 R Y K D U S v u k U W A / U v D c R / / P a i Q 4 u v J S F c a J p S G a L g o R D H c F J E L D H J C W a j w z B R D J z K y Q D L D H R J q 6 C C Q H N v 7 x I G q c O K j v l + 7 N S 9 T K L I w / 2 w Q E 4 A g i c g y q 4 B T V Q B w Q 8 g W f w C t 6 s s f V i v V s f s 9 a c l c 3 s g T + w P n 8 A 6 f i S c w = = < / l a t e x i t > MF E = 10.1, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " V a 4 q g V B q / 8 N v Z 6 i G r z o + x x T Z w V Y = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G z W A R X G h I q r b d F I q i u B E q 2 A e 0 p U y m k 3 b o Z B J m J k K J X f g r b l w o 4 t b f c O f f O G 2 z 0 N Y D F w 7 n 3 M u 9 9 7 g h o 1 L Z 9 r e R W l h c W l 5 J r 2 b W 1 j c 2 t 8 z t n Z o M I o F J F Q c s E A 0 X S c I o J 1 V F F S O N U B D k u 4 z U 3 c H l 2 K 8 / E C F p w O / V M C R t H / U 4 9 S h G S k s d c + / 2 + q p 0 4 p x a u e O W p D 0 f l W y r W D j v m F n b s i e A 8 8 R J S B Y k q H T M r 1 Y 3 w J F P u M I M S d l 0 7 F C 1 Y y Q U x Y y M M q 1 I k h D h A e q R p q Y c + U S 2 4 8 n 9 I 3 i o l S 7 0 A q G L K z h R f 0 / E y J d y 6 L u 6 0 0 e q L 2 e 9 s f i f 1 4 y U V 2 z H l I e R I h x P F 3 k R g y q A 4 z B g l w q C F R t q g r C g + l a I + 0 g g r H R k G R 2 C M / v y P K n l L C d v 5 e / O s u W L J I 4 0 2 A c H 4 A g 4 o A D K 4 A Z U Q B V g 8 A i e w S t 4 M 5 6 M F + P d + J i 2 p o x k Z h f 8 g f H 5 A 9 c V k 3 A = < / l a t e x i t > MF E = 13.2, = 0.875 < l a t e x i t s h a 1 _ b a s e 6 4 = " U o j j D W g + q G d 9 I O c L H F J s p N J k 4 O g = " > A A A B / X i c b V D L S s N A F J 3 U V 6 2 v + N i 5 G S y C C w 2 J 1 e q m U B T F j V D B P q A N Z T K d t E N n k j A z E W o o / o o b F 4 q 4 9 T / c + T d O 2 y y 0 9 c C F w z n 3 c u 8 9 X s S o V L b 9 b W T m 5 h c W l 7 L L u Z X V t f U N c 3 O r J s N Y Y F L F I Q t F w 0 O S M B q Q q q K K k U Y k C O I e I 3 W v f z n y 6 w 9 E S B o G 9 2 o Q E Z e j b k B 9 i p H S U t v c u b 2 + K h 0 V r c J h S 9 I u R y X b O i 2 0 z b x t 2 W P A W e K k J A 9 S V N r m V 6 s T 4 p i T Q G G G p G w 6 d q T c B A l F M S P D X C u W J E K 4 j 7 q k q W m A O J F u M r 5 + C P e 1 0 o F + K H Q F C o 7 V 3 x M J 4 l I O u K c 7 O V I 9 O e 2 N x P + 8 Z q z 8 c z e h Q R Q r E u D J I j 9 m U I V w F A X s U E G w Y g N N E B Z U 3 w p x D w m E l Q 4 s p 0 N w p l + e J b V j y y l a x b u T f P k i j S M L d s E e O A A O O A N l c A M q o A o w e A T P 4 B W 8 G U / G i / F u f E x a M 0 Y 6 s w 3 + w P j 8 A e P 3 k v M = < / l a t e x i t > MF E = 6.3, = 0.53 < l a t e x i t s h a 1 _ b a s e 6 4 = " J d A Y o t N K N j n p 6 k z 6 p S 6 0 e 6 5 7 Q m 4 = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o e Z W q q b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h R x 6 3 e 4 8 2 9 M 2 1 l o 6 4 E L h 3 P u 5 d 5 7 g o h R p V 3 3 2 1 p Y X F p e W c 2 s Z d c 3 N r e 2 7 Z 3 d m g p j i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 1 6 e a d w 0 l K 0 x 1 H J d c 7 c t p 1 z H X c C O E + 8 l O R A i k r b / m p 1 Q h x z I j R m S K m m 5 0 b a T 5 D U F D M y y r Z i R S K E B 6 h H m o Y K x I n y k 8 n 5 I 3 h k l A 7 s h t K U 0 H C i / p 5 I E F d q y A P T y Z H u q 1 l v L P 7 n N W P d v f A T K q J Y E 4 G n i 7 o x g z q E 4 y x g h 0 q C N R s a g r C k 5 l a I + 0 g i r E 1 i W R O C N / v y P K n l H a / o F O 8 L u f J l G k c G H I B D c A w 8 c A 7 K 4 B Z U Q B V g k I B n 8 A r e r C f r x X q 3 P q a t C 1 Y 6 s w f + w P r 8 A U t G k y Y = < / l a t e x i t > MF E = 12.4, = 0.30 < l a t e x i t s h a 1 _ b a s e 6 4 = " c J y z i j M A w 2 R O Z L P E R c 8 8 t q Q H I e U = " > A A A B / X i c b V D L S g M x F M 3 4 r P U 1 P n Z u g k V w o c O M l K q L Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D o U f 8 W N C 0 X c + h / u / B v T d h b a e u D C 4 Z x 7 u f e e I G J U a d f 9 t u b m F x a X l j M r 2 d W 1 9 Y 1 N e 2 u 7 q s J Y Y l L B I Q t l P U C K M C p I R V P N S D 2 S B P G A k V r Q v x z 5 t Q c i F Q 3 F v R 5 E x O e o K 2 i H Y q S N 1 L J 3 b 6 + v i s f n T v 6 o q W i X o 6 L r 5 L 2 W n X M d d w w 4 S 7 y U 5 E C K c s v + a r Z D H H M i N G Z I q Y b n R t p P k N Q U M z L M N m N F I o T 7 q E s a h g r E i f K T 8 f V D e G C U N u y E 0 p T Q c K z + n k g Q V 2 r A A 9 P J k e 6 p a W 8 k / u c 1 Y t 0 5 8 x M q o l g T g S e L O j G D O o S j K G C b S o I 1 G x i C s K T m V o h 7 S C K s T W B Z E 4 I 3 / f I s q Z 4 4 X s E p 3 O V z p Y s 0 j g z Y A / v g E H j g F J T A D S i D C s D g E T y D V / B m P V k v 1 r v 1 M W m d s 9 K Z H f A H 1 u c P 5 b C S 9 A = = < / l a t e x i t > MF E = 9.4, = 0.41 < l a t e x i t s h a 1 _ b a s e 6 4 = " R M K Z f k i m S 8 + g M D L R j j Y m Q d 1 h L F E = " > A A A B + 3 i c b V D L S g M x F L 1 T X 7 W + x r p 0 M 1 g E F 1 p m R K q b Q l E U N 0 I F + 4 B 2 K J k 0 0 4 Y m m S H J i G X o r 7 h x o Y h b f 8 S d f 2 P 6 W G j r g Q s n 5 9 x L 7 j 1 B z K j S r v t t Z Z a W V 1 b X s u u 5 j c 2 t 7 R 1 7 N 1 9 X U S I x q e G I R b I Z I E U Y F a S m q W a k G U u C e M B I I x h c j f 3 G I 5 G K R u J B D 2 P i c 9 Q T N K Q Y a S N 1 7 P z d z X X 5 x H O L 7 n F b 0 R 5 H Z a 9 j F 8 x z A m e R e D N S g B m q H f u r 3 Y 1 w w o n Q m C G l W p 4 b a z 9 F U l P M y C j X T h S J E R 6 g H m k Z K h A n y k 8 n u 4 + c Q 6 N 0 n T C S p o R 2 J u r v i R R x p Y Y 8 M J 0 c 6 b 6 a 9 8 b i f 1 4 r 0 e G F n 1 I R J 5 o I P P 0 o T J i j I 2 c c h N O l k m D N h o Y g L K n Z 1 c F 9 J B H W J q 6 c C c G b P 3 m R 1 E + L X q l Y u j 8 r V C 5 n c W R h H w 7 g C D w 4 h w r c Q h V q g O E J n u E V 3 q y R 9 W K 9 W x / T 1 o w 1 m 9 m D P 7 A + f w D o a 5 J y < / l a t e x i t > MF E = 10.0, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " w 4 Z g 6 X a J F n S Z v v F e 1 g + p 1 z e Z S t Q = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B h Z Z E p b o p F E V x I 1 S w D 2 h C m U w n 7 d C Z J M x M h B I K b v w V N y 4 U c e t P u P N v n L R Z a O u B C 4 d z 7 u X e e 7 y I U a k s 6 9 v I z c 0 v L C 7 l l w s r q 2 v r G + b m V k O G s c C k j k M W i p a H J G E 0 I H V F F S O t S B D E P U a a 3 u A y 9 Z s P R E g a B v d q G B G X o 1 5 A f Y q R 0 l L H 3 H E 4 U n 3 P T 2 6 v r y p H 5 d L J o S N p j 6 O K P e q Y R a t k j Q F n i Z 2 R I s h Q 6 5 h f T j f E M S e B w g x J 2 b a t S L k J E o p i R k Y F J 5 Y k Q n i A e q S t a Y A 4 k W 4 y / m E E 9 7 X S h X 4 o d A U K j t X f E w n i U g 6 5 p z v T i + W 0 l 4 r / e e 1 Y + e d u Q o M o V i T A k 0 V + z K A K Y R o I 7 F J B s G J D T R A W V N 8 K c R 8 J h J W O r a B D s K d f n i W N 4 5 J d L p X v T o v V i y y O P N g F e + A A 2 O A M V M E N q I E 6 w O A R P I N X 8 G Y 8 G S / G u / E x a c 0 Z 2 c w 2 + A P j 8 w f V p p Z g < / l a t e x i t > MFE = 6.3, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 4 3 z N Y E x 9 + D B M V g r T I e T j 5 N p i A c = " > A A A C B X i c b V D L S s N A F J 3 U V 6 2 v q E t d D B b B h Z Z E p b o p F E V x I 1 S w D 2 h C m U w n 7 d C Z J M x M h B K 6 c e O v u H G h i F v / w Z 1 / 4 6 T N Q l s P X D i c c y / 3 3 u N F j E p l W d 9 G b m 5 + Y X E p v 1 x Y W V 1 b 3 z A 3 t x o y j A U m d R y y U L Q 8 J A m j A a k r q h h p R Y I g 7 j H S 9 A a X q d 9 8 I E L S M L h X w 4 i 4 H P U C 6 l O M l J Y 6 5 q 7 D k e p 7 f n J 7 f V U 5 s q 3 S y a E j a Y + j i j 0 q d M y i V b L G g L P E z k g R Z K h 1 z C + n G + K Y k 0 B h h q R s 2 1 a k 3 A Q J R T E j o 4 I T S x I h P E A 9 0 t Y 0 Q J x I N x l / M Y L 7 W u l C P x S 6 A g X H 6 u + J B H E p h 9 z T n e n N c t p L x f + 8 d q z 8 c z e h Q R Q r E u D J I j 9 m U I U w j Q R 2 q S B Y s a E m C A u q b 4 W 4 j w T C S g e X h m B P v z x L G s c l u 1 w q 3 5 0 W q x d Z H H m w A / b A A b D B G a i C G 1 A D d Y D B I 3 g G r + D N e D J e j H f j Y 9 K a M 7 K Z b f A H x u c P g A a W q Q = = < / l a t e x i t > MFE = 10.3, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " w 4 Z g 6 X a J F n S Z v v F e 1 g + p 1 z e Z S t Q = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B h Z Z E p b o p F E V x I 1 S w D 2 h C m U w n 7 d C Z J M x M h B I K b v w V N y 4 U c e t P u P N v n L R Z a O u B C 4 d z 7 u X e e 7 y I U a k s 6 9 v I z c 0 v L C 7 l l w s r q 2 v r G + b m V k O G s c C k j k M W i p a H J G E 0 I H V F F S O t S B D E P U a a 3 u A y 9 Z s P R E g a B v d q G B G X o 1 5 A f Y q R 0 l L H 3 H E 4 U n 3 P T 2 6 v r y p H 5 d L J o S N p j 6 O K P e q Y R a t k j Q F n i Z 2 R I s h Q 6 5 h f T j f E M S e B w g x J 2 b a t S L k J E o p i R k Y F J 5 Y k Q n i A e q S t a Y A 4 k W 4 y / m E E 9 7 X S h X 4 o d A U K j t X f E w n i U g 6 5 p z v T i + W 0 l 4 r / e e 1 Y + e d u Q o M o V i T A k 0 V + z K A K Y R o I 7 F J B s G J D T R A W V N 8 K c R 8 J h J W O r a B D s K d f n i W N 4 5 J d L p X v T o v V i y y O P N g F e + A A 2 O A M V M E N q I E 6 w O A R P I N X 8 G Y 8 G S / G u / Ex a c 0 Z 2 c w 2 + A P j 8 w f V p p Z g < / l a t e x i t > MFE = 6.3, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " w 4 Z g 6 X a J F n S Z v v F e 1 g + p 1 z e Z S t Q= " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B h Z Z E p b o p F E V x I 1 S w D 2 h C m U w n 7 d C Z J M x M h B I K b v w V N y 4 U c e t P u P N v n L R Z a O u B C 4 d z 7u X e e 7 y I U a k s 6 9 v I z c 0 v L C 7 l l w s r q MF E = 23.2, = 0.77 < l a t e x i t s h a 1 _ b a s e 6 4 = " n p 4 g p C j U U w F t c J e 7 s 5 T h Y p p Q U w E = " > A A A B / n i c b V D L S s N A F J 3 U V 6 2 v q L h y M 1 g E F x q S U l s 3 h a I o b o Q K 9 g F t K J P p p B 0 6 k 4 S Z i V B C w V 9 x 4 0 I R t 3 6 H O / / G a Z u F t h 6 4 c D j n X u 6 9 x 4 s Y l c q 2 v 4 3 M 0 v L K 6 l p 2 P b e x u b W 9 Y + 7 u N W Q Y C 0 z q O G S h a H l I E k Y D U l d U M d K K B E H c Y 6 T p D a 8 m f v O R C E n D 4 E G N I u J y 1 A + o T z F S W u q a B 3 c 3 1 5 W z Q t E 6 P + 1 I 2 u e o Y l v l c t f M 2 5 Y 9 B V w k T k r y I E W t a 3 5 1 e i G O O Q k U Z k j K t m N H y k 2 Q U B Q z M s 5 1 Y k k i h I e o T 9 q a B o g T 6 S b T 8 8 f w W C s 9 6 I d C V 6 D g V P 0 9 k S A u 5 Y h 7 u p M j N Z D z 3 k T 8 z 2 v H y r 9 w E x p E s S I B n i 3 y Y w Z V C C d Z w B 4 V B C s 2 0 g R h Q f W t E A + Q Q F j p x H I 6 B G f + 5 U X S K F h O y S r d F / P V y z S O L D g E R + A E O K A M q u A W 1 E A d Y J C A Z / A K 3 o w n 4 8 V 4 N z 5 m r R k j n d k H f 2 B 8 / g B i P Z M 1 < / l a t e x i t > MF E = 24.5, = 0.77 < l a t e x i t s h a 1 _ b a s e 6 4 = " Y L Z d X o W R o a t Z / N H V M P x g V b + P 1 F c = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o e Z U l o 3 h a I o b o Q K 9 g H t U D J p 2 o Y m m S H J C G U o + C t u X C j i 1 u 9 w 5 9 + Y t r P Q 1 g M X D u f c y 7 3 3 B B G j S r v u t 7 W 0 v L K 6 t p 7 Z y G 5 u b e / s 2 n v 7 d R X G E p M a D l k o m w F S h F F B a p p q R p q R J I g H j D S C 4 d X E b z w S q W g o H v Q o I j 5 H f U F 7 F C N t p I 5 9 e H d z X T 7 P F 5 z 8 W V v R P k d l 1 y m V O n b O d d w p 4 C L x U p I D K a o d + 6 v d D X H M i d C Y I a V a n h t p P 0 F S U 8 z I O N u O F Y k Q H q I + a R k q E C f K T 6 b n j + G J U b q w F 0 p T Q s O p + n s i Q V y p E Q 9 M J 0 d 6 o O a 9 i f i f 1 4 p 1 7 8 J P q I h i T Q S e L e r F D O o Q T r K A X S o J 1 m x k C M K S m l s h H i C J s D a J Z U 0 I 3 v z L i 6 S e d 7 y i U 7 w v 5 C q X a R w Z c A S O w S n w Q A l U w C 2 o g h r A I A H P 4 B W 8 W U / W i / V u f c x a l 6 x 0 5 g D 8 g f X 5 A 1 2 N k z I = < / l a t e x i t > MF E = 24.2, = 0.77 < l a t e x i t s h a 1 _ b a s e 6 4 = " w q 3 J Z K D 8 p c h e f 8 F w y G 6 O Z c x J P / Q = " > A A A B / n i c b V D L S g M x F M 3 U V 6 2 v U X H l J l g E F z r M S G n d F I q i u B E q 2 A e 0 Q 8 m k a R u a Z I Y k I 5 S h 4 K + 4 c a G I W 7 / D n X 9 j 2 s 5 C W w 9 c O J x z L / f e E 0 S M K u 2 6 3 1 Z m a X l l d S 2 7 n t v Y 3 N r e s X f 3 2 v r G + b m V k O G s c C k j k M W i p a H J G E 0 I H V F F S O t S B D E P U a a 3 u A y 9 Z s P R E g a B v d q G B G X o 1 5 A f Y q R 0 l L H 3 H E 4 U n 3 P T 2 6 v r y p H 5 d L J o S N p j 6 O K P e q Y R a t k j Q F n i Z 2 R I s h Q 6 5 h f T j f E M S e B w g x J 2 b a t S L k J E o p i R k Y F J 5 Y k Q n i A e q S t a Y A 4 k W 4 y / m E E 9 7 X S h X 4 o d A U K j t X f E w n i U g 6 5 p z v T i + W 0 l 4 r / e e 1 Y + e d u Q o M o V i T A k 0 V + z K A K Y R o I 7 F J B s G J D T R A W V N 8 K c R 8 J h J W O r a B D s K d f n i W N 4 5 J d L p X v T o v V i y y O P N g F e + A A 2 O A M V M E N q I E 6 w O A R P I N X 8 G Y 8 G S / G u / E x a c 0 Z 2 c w 2 + A P j 8 w f V p p Z g < / l a t e x i t > MFE = 6.3, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 x f l u 0 h s U k 3 8 J v k E P Z m u r c y d Z e Y = " > A A A C B n i c b V D L S s N A F J 3 4 r P U V d S l C s A g u N C Q + q p t C U R Q 3 Q g X 7 g K a U y X T S D p 2 Z h J m J U E J X b v w V N y 4 U c e s 3 u P N v n L R Z a O u B C 4 d z 7 u X e e / y I E q k c 5 9 u Y m Z 2 b X 1 j M L e W X V 1 b X 1 s 2 N z Z o M Y 4 F w F Y U 0 F A 0 f S k w J x 1 V F F M W N S G D I f I r r f v 8 y 9 e s P W E g S 8 n s 1 i H C L w S 4 n A U F Q a a l t 7 n g M q p 4 f J L f X V 6 X D Y 9 s 5 8 C T p M l h y 7 O L p s G 0 W H N s Z w Z o m b k Y K I E O l b X 5 5 n R D F D H O F K J S y 6 T q R a i V Q K I I o H u a 9 W O I I o j 7 s 4 q a m H D I s W 8 n o j a G 1 p 5 W O F Y R C F 1 f W S P 0 9 k U A m 5 Y D 5 u j M 9 W k 5 6 q f i f 1 4 x V c N 5 K C I 9 i h T k a L w p i a q n Q S j O x O k R g p O h A E 4 g E 0 b d a q A c F R E o n l 9 c h u J M v T 5 P a k e 0 W 7 e L d S a F 8 k c W R A 9 t g F + w D F 5 y B M r g B F V A F C D y C Z / A K 3 o w n 4 8 V 4 N z 7 G r T N G N r M F / s D 4 / A E 7 / 5 c Q < / l a t e x i t > MFE = 3.0, = 0.65 A A A U G U G A A U G A A A A A U A U U A U A U A A G G G C C G G G G C C G C A G G G C C C C G G C G G G C G A G C C G G C G G G C C A C C G C C G G C U C G G C C G C C C G C C G G G C G G G C G G C C G C A C G A G G C C G G C C C U A U U G G G C C G G C C U G C C G C G A A A A G C AA C C G A A A A A A A A G G G G C G G G G G C C G G G C C C C C C G G G C G C A G G C C C G G G G A G A U C C A G C C A G A G G U A U A C C U C U G G C U C G G G U G C A G C G C U U A C C A C U A A C A G A C C C G C A C C C C G C C G C G G A G G C A C G C G G C G G G G U G < l a t e x i t s h a 1 _ b a s e 6 4 = " w l x d L x b s k l b I y U O v d J J k T C W + W D k = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o e Z K q 2 b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h R x 6 3 e 4 8 2 9 M 2 1 l o 6 4 E L h 3 P u 5 d 5 7 g o h R p V 3 3 2 1 p Y X F p e W c 2 s Z d c 3 N r e 2 7 Z 3 d m g p j i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 3 m 8 0 7 x p K V o j 6 O S 6 x T O 2 n b O d d w J 4 D z x U p I D K S p t + 6 v V C X H M i d C Y I a W a n h t p P 0 F S U 8 z I K N u K F Y k Q H q A e a R o q E C f K T y b n j + C R U T q w G 0 p T Q s O J + n s i Q V y p I Q 9 M J 0 e 6 r 2 a 9 s f i f 1 4 x 1 9 8 J P q I h i T Q S e L u r G D O o Q j r O A H S o J 1 m x o C M K S m l s h 7 i O J s D a J Z U 0 I 3 u z L 8 6 S W d 7 y C U 7 g / z 5 U v 0 z g y 4 A A c g m P g g S I o g 1 t Q A V W A Q Q K e w S t6 i q M J S Y 1 H L J Q N g O k C K O C 1 D T V j D Q j S R A P G G k E w 6 u J 3 3 g k U t F Q P O h R R H y O + o L 2 K E b a S B 3 7 4 O 7 m u n z m l R z 3 t K 1 o n 6 O y 6 x R K H T v v O u 4 U c J F 4 K c m D F N W O / d X u h j j m R G j M k F I t z 4 2 0 n y C p K W Z k n G v H i k Q I D 1 G f t A w VS i J J J w c / 2 G F Z V f k 1 Z T c A = " > A A A B / n i c b V D L S s N A F J 3 U V 6 2 v q L h y M 1 g E F x q S U t p u C k V R 3 A g V 7 A P a U C b T S T t 0 M g k z E 6 G E g r / i x o U i b v 0 O d / 6 N 0 z Y L b T 1 w 4 X D O v d x 7 j x c x K p V t f x u Z l d W 1 9 Y 3 s Z m 5 r e 2 d 3 z 9 w / a M o w F p g 0 c M h C 0 f a Q J I x y 0 l B U M d K O B E G B x 0 j L G 1 1 N / d Y j E Z K G / E G N I + I G a M C p T z F S W u q Z R 3 c 3 1 9 W L Q s U q n X c l H Q S o a l v F c s / M 2 5 Y 9 A 1 w m T k r y I E W 9 Z 3 5 1 + y G O A 8 I V Z k j K j m N H y k 2 Q U B Q z M s l 1 Y 0 k i h E d o Q D q a c h Q Q 6 SM i B H z V G d 6 t J j 1 U v E / r x N L v + I m J I h i i Q M 0 X e T H 1 J C h k W Z i 9 A j H S N K R I h B x o m 4 1 0 A B y i K R K L q 9 C s G d f n i f N M 9 M u m a W 7 8 0 L t I o s j B / b B I T g G N i i D G r g B d d A A C D y C Z / A K 3 r Q n 7 U V 7 1 z 6 m r Q t a N r M H / k D 7 / A F N E J c b < / l a t e x i t > MFE = 5.5, = 0.87 D-PBMOEA/D G G C C G U C G G C C G C C G C C G U A G G G G C G G G C G C G G C C G G G G G G C C G C G G G C C C G C C G U A G C U C G G C G G G C C C G C C C G G U C C C U G G A A G C A G G G A C C G G G G G A U< l a t e x i t s h a 1 _ b a s e 6 4 = " w 4 Z g 6 X a J F n S Z v v F e 1 g + p 1 z e Z S t Q = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d D B b B h Z Z E p b o p F E V x I 1 S w D 2 h C m U w n 7 d C Z J M x M h B I K b v w V Nv r G + b m V k O G s c C k j k M W i p a H J G E 0 I H V F F S O t S B D E P U a a 3 u A y 9 Z s P R E g a B v d q G B G X o 1 5 A f Y q R 0 l L H 3 H E 4 U n 3 P T 2 6 v r y p H 5 d L J o S N p j 6 O K P e q Y R a t k j Q F n i Z 2 R I s h Q 6 5 h f T j f E M S e B w g x J 2 b a t S L k J E o p i R k Y F J 5 Y k Q n i A e q S t a Y A 4 k W 4 y / m E E 9 7 X S h X 4 o d A U K j t X f E w n i U g 6 5 p z v T i + W 0 l 4 r / e e 1 Y + e d u Q o M o V i T A k 0 V + z K A K Y R o I 7 F J B s G J D T R A W V N 8 K c R 8 J h J W O r a B D s K d f n i W N 4 5 J d L p X v T o v V i y y O P N g F e + A A 2 O A M V M E N q I E 6 w O A R P I N X 8 G Y 8 G S / G u / E x a c 0 Z 2 c w 2 + A P j 8 w f V p p Z g < / l a t e x i t > MFE = 6.3, = 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " U Y S R c b t u U d 0 6 G l 0 v 8 4 q g Y g 4 t 8 g U = " > A A A C A X i c b V D L S s N A F J 3 4 r P U V d S O 4 C R b B h Y Z E a t t N o S i K G 6 G C f U A b y m Q 6 a Y f O T M L M R C i h b v w V N y 4 U c e t f u P N v n L Z Z a O u B C 4 d z 7 u X e e / y I E q k c 5 9 t Y W F x a X l n N r G X X N z a 3 t s 2 d 3 b o M Y 4 F w D Y U 0 F E 0 f S k w J x z V F F M X N S G D I f I o b / u B y 7 D c e s J A k 5 P d q G G G P w R 4 n A U F Q a a l j 7 i e 3 1 1 f l U 9 e x 8 y d t S X o M l h 2 7 V D w f d c y c Y z s T W P P E T U k O p K h 2 z K 9 2 N 0 Q x w 1 w h C q V s u U 6 k v A Q K R R D F o 2 w 7 l j i C a A B 7 u K U p h w x L L 5 l 8 M L K O t N K 1 g l D o 4 s q a q L 8 n E s i k H D J f d z K o + n L W G 4 v / e a 1 Y B S U v I T y K F e Z o u i i I q a V C a x y H 1 S U C I 0 W H m k A k i L 7 V Q n 0 o I F I 6 t K w O w Z 1 9 e Z 7 U z 2 y 3 Y B f u 8 r n K R R p H B h y A Q 3 A M X F A E F X A D q q A G E H g E z + A V v B l P x o v x b n x M W x e M d G Y P / I H x + Q O o w p R 7 < / l a t e x i t > MF E = 10.4, = 0.875 < l a t e x i t s h a 1 _ b a s e 6 4 = " m F V G / z K d K T + K d Z / S y k A Q X D l y 3 U U = " > A A A C A X i c b V D L S s N A F J 3 4 r P U V d S O 4 C R b B h Y a k a t V F o S i K G 6 G C f U A b y m Q 6 a Y f O T M L M R C i h b v w V N y 4 U c e t f u P N v n L Z Z a O u B C 4 d z 7 u X e e / y I E q k c 5 9 u Y m Z 2 b X 1 j M L G W X V 1 b X 1 s 2 N z a o M Y 4 F w B Y U 0 F H U f S k w J x x V F F M X 1 S G D I f I p r f u 9 y 6 N c e s J A k 5 P e q H 2 G P w Q 4 n A U F Q a a l l b i e 3 1 1 f F Q z d v n x 8 0 J e k w W H T s / N H J o G X m H N s Z w Z o m b k p y I E W 5 Z X 4 1 2 y G K G e Y K U S h l w 3 U i 5 S V Q K I I o H m S b s c Q R R D 3 Y w Q 1 N O W R Y e s n o g 4 G 1 p 5 W 2 F Y R C F 1 f W S P 0 9 k U A m Z Z / 5 u p N B 1 Z W T 3 l D 8 z 2 v E K j j z E s K j W G G O x o u C m F o q t I Z x W G 0 i M F K 0 r w l E g u h b L d S F A i K l Q 8 v q E N z J l 6 d J N W + 7 B b t w d 5 w r X a R x Z M A O 2 A X 7 w A W n o A R u Q B l U A A K P 4 BQ = " > A A A B / 3 i c b V D L S s N A F J 3 U V 6 2 v q O D G z W A R X G h I q l Y 3 h a I o b o Q K 9 g F t K J P p p B 0 6 k 4 S Z i V B i F / 6 K G x e K u P U 3 3 P k 3 T t s s t P X A h c M 5 9 3 L v P V 7 E q F S 2 / W 1 k 5 u Y X F p e y y 7 m V 1 b X 1 D X N z q y b D W G B S x S E L R c N D k j A a k K q i i p F G J A j i H i N 1 r 3 8 5 8 u s P R E g a B v d q E B G X o 2 5 A f Y q R 0 l L b 3 E l u r 6 9 K R 4 V T q 3 j Y k r T L U c m 2 j o d t M 2 9 b 9 h h w l j g p y Y M U l b b 5 1 e q E O O Y k U J g h K Z u O H S k 3 Q U J R z M g w 1 4 o l i R D u o y 5 p a h o g T q S b j O 8 f w n 2 t d K A f C l 2 B g m P 1 9 0 S C u J Q D 7 u l O j l R P T n s j 8 T + v G S v / 3 E 1 o E M W K B H i y y I 8 Z V C E c h Q E 7 V B C s 2 E A T h A X V t 0 L c Q w J hThe comparison results of D-PBEMO against the other three stat-of-the-art PBEMO algorithms on inverse RNA design problems. In particular, the target structure is a sample of possible solution represented in blue color while the predicted one obtained by different optimization algorithms are highlighted in red color. In this graph, the reference point is set as σ = 1. The closer σ is to 1, the better performance achieved by the corresponding algorithm. When the σ share the same biggest value, the smaller M F E the better the performance is. A U A U A U A A U A U A A A A U A U A U C A A A U A U A U A A G G G C C C C G G G G G G C G A C C C C C G G G G G G G C C G C G G C G G G C C G C C G G G G G A C G G C G G C C C G C C G U C C C C C C U G C G G C G G A G G C C A C A G G G G G G A G G U G U C C C G C C C C G C C G G G G C G G C C G A C A C U U A A A A U A C A G G G C G C G A A A G G A C U C A C U G U A A A A A G A G A A A U C A G A G A A A U C A G U G A A A G C A G G G A A A C U C G U G A C A U U A U A A A A A U G A G U CG A U G A A A A A A A A A A 10 U A A A A C G A G A A A A A C G A A A G U U U U A U C A U G G U U U U A G C C C C C C C C C G G G G C G G A G C G C C G G G G G G G A C G C G G G G G C C C G C C G U A G G G G G G G C C G C C G C G C G G G C U C C G G G C C C U C C U G A G G A G G G C C C G G G G G C U G C G A C G C G C C A C C G C A C C G G G G A U G C C C C G G U G U C G G U G G C G G G G G G C C C A G G C G G G G A C G C C U G G G C C C C C G C A C A A U U A C G G G C C C C C G G G C G G G G G C C C G G G G G C C C G G G C G A G A G C G C A A U C U G U G A U U G C G C U C A U C G C C G U U A C C C G C C U C G U G G U G G G A U G C C C A C C A C G G G C G G G C G G G G G G G G G G G U C U U C U G A C C C C C C C C C C C U C A G C G G U C C C G C C G G G G G G G G G G U G A G C U U A A C A G U A C C C G C C C G C G G U U C U C C G C G G G C G G G G A C C A C U A C G C G G G C C U C C G G C C C G G U U A C G C G G C C G G A G G C C C G G G C G U G G C C A C A G C U G U G G U G G A G A C C U G U G G C C G A G C G G G C C A G G A U A C C C C C C G C G G C C C C U C A C C C G C C C C C U G G C G A G A C C C G G G G U C U C G C C A G G G G G G C G G G G G U C U G C C U C U G A G C A A A G C U C A G A G G C G A C C C C < l a t e x i t s h a 1 _ b a s e 6 4 = " h k q Q I t r f + z H p 9 Q O C w h N i v U a b 7 R U = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o c Z K 7 W b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h Rj i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 3 m 8 4 5 3 0 l K 0 x 1 H J d Q r F t p 1 z H X c C O E + 8 l O R A i k r b / m p 1 Q h x z I j R m S K m m 5 0 b a T 5 D U F D M y y r Z i R S K E B 6 h H m o Y K x I n y k 8 n 5 I 3 h k l A 7 s h t K U 0 H C i / p 5 I E F d q y A P T y Z H u q 1 l v L P 7 n N W P d L f o J F V G s i c D T R d 2 Y Q R 3 C c R a w Q y X B m g 0 N Q V h S c y v E f S Q R 1 i a x r A n B m 3 1 5 n t T O H K / g F O 7 P c + X L N I 4 M O A C H 4 B h 4 4 A K U w S 2 o g C r A I A H P 4 B W 8 W U / W i / V u f U x b F 6 x 0 Z g / 8 g f X 5 A 1 v 9 k z E = < / l a t e x i t > MF E = 33.1, = 0.68 < l a t e x i t s h a 1 _ b a s e 6 4 = " N C r O Y K 8 E A Y u K 2 t 7 E 9 c 0 t 3 R / O 4 g Q = " > A A A B / n i c b V D L S s N A F J 3 U V 6 2 v q L h y M 1 g E F x q S W l o 3 h a I o b o Q K 9 g F t K J P p p B 0 6 k 4 S Z i V B C w V 9 x 4 0 I R t 3 6 H O / / G a Z u F t h 6 4 c D j n X u 6 9 x 4 s Y l c q 2 v 4 3 M 0 v L K 6 l p 2 P b e x u b W 9 Y + 7 u N W Q Y C 0 z q O G S h a H l I E k Y D U l d U M d K K B E H c Y 6 T p D a 8 m f v O R C E n D 4 E G N I u J y 1 A + o T z F S W u q a B 3 c 3 1 5 W z Q t k 6 P + 1 I 2 u e o Y l v l Y t f M 2 5 Y 9 B V w k T k r y I E W t a 3 5 1 e i G O O Q k U Z k j K t m N H y k 2 Q U B Q z M s 5 1 Y k k i h I e o T 9 q a B o g T 6 S b T 8 8 f w W C s 9 6 I d C V 6 D g V P 0 9 k S A u 5 Y h 7 u p M j N Z D z 3 k T 8 z 2 v H y r 9 w E x p E s S I B n i 3 y Y w Z V C C d Z w B 4 V B C s 2 0 g R h Q f W t E A + Q Q F j p x H I 6 B G f + 5 U X S K F h O y S r d F / P V y z S O L D g E R + A E O K A M q u A W 1 E A d Y J C A Z / A K 3 o w n 4 8 V 4 N z 5 m r R k j n d k H f 2 B 8 / g B f R 5 M z < / l a t e x i t >MF E = 27.3, = 0.74 < l a t e x i t s h a 1 _ b a s e 6 4 = " L h u n e S s q h n I o 1 3 A j T B / D V C e 1 4 r g = " > A A A B / n i c b V D L S s N A F J 3 U V 6 2 v q L h y M 1 g E F x q S I m 1 d F I q i u B E q 2 A e 0 o U y m k 3 b o T B J m J k I J B X / F j Q t F 3 P o d 7 v w b p 2 0 W 2 n r g w u G c e 7 n 3 H i 9 i V C r b / j Y y S 8 s r q 2 v Z 9 d z G 5 t b 2 j r m 71 5 B h L D C p 4 5 C F o u U h S R g N S F 1 R x U g r E g R x j 5 G m N 7 y a + M 1 H I i Q N g w c 1 i o j L U T + g P s V I a a l r H t z d X F f O C r Z 1 c d q R t M 9 R x b b K p a 6 Z t y 1 7 C r h I n J T k Q Y p a 1 / z q 9 E I c c x I o z J C U b c e O l J s g o S h m Z J z r x J J E C A 9 R n 7 Q 1 D R A n 0 k 2 m 5 4 / h s V Z 6 0 A + F r k D B q f p 7 I k F c y h H 3 d C d H a i D n v Y n 4 n 9 e O l V 9 2 E x p E s S I B n i 3 y Y w Z V C C d Z w B 4 V B C s 2 0 g R h Q f W t E A + Q Q F j p x H I 6 B G f + 5 U X S K F h O 0 Sr e n + e r l 2 k c W X A I j s A J c E A J V M E t q I E 6 w C A B z + A V v B l P x o v x b n z M W j N G O r M P / s D 4 / A F j u p M 2 < / l a t e x i t > MF E = 20.9, = 0.87 < l a t e x i t s h a 1 _ b a s e 6 4 = " A l l g w N X r L 7 v wI M t 1 Y a g H H V j t f / A = " > A A A B / n i c b V D L S g M x F M 3 U V 6 2 v U X H l J l g E F z r M + G j d F I q i u B E q 2 A e 0 p W T S T B u a Z I Y k I 5 S h 4 K + 4 c a G I W 7 / D n X 9 j 2 s 5 C q w c u H M 6 5 l 3 v v 8 S N G l X b d L y s z N 7 + w u J R d z q 2 s r q 1 v 2 J t b N R X G E p M q D l k o G z 5 S h F F B q p p q R h q R J I j 7 j N T 9 w e X Y r z 8 Q q W g o 7 v U w I m 2 O e o I G F C N t p I 6 9 c 3 t 9 V T o 6 O X O 8 w 5 a i P Y 5 K r l P 0 O n b e d d w J 4 F / i p S Q P U l Q 6 9 m e r G + K Y E 6 E x Q 0 o 1 P T f S 7 Q R J T T E j o 1 w r V i R C e I B 6 p G m o Q J y o d j I 5 f w T 3 j d K F Q S h N C Q 0 n 6 s + J B H G l h t w 3 n R z p v p r 1 x u J / X j P W w X k 7 o S K K N R F 4 u i i I G d Q h H G c B u 1 Q S r N n Q E I Q l N b d C 3 E c S Y W 0 S y 5 k Q v N m X / 5 L a s e M V n M L d a b 5 8 k c a R B b t g D x w A D x R B G d y A C q g C D B L w B F 7 A q / V o P V t v 1 v u 0 N W O l M 9 v g F 6 y P b 1 Y K k y 0 = < / l a t e x i t >MF E = 35.1, = 0.71 < l a t e x i t s h a 1 _ b a s e 6 4 = " / c e 5 U o Ik 7 O U b J Q D G m X + W v j / j G Q o = " > A A A B / n i c b V D L S s N A F J 3 U V 6 2 v q L h y M 1 g E F x o S t a 2 b Q l E U N 0 I F + 4 A 2 l M l 0 0 g 6 d T M L M R C i h 4 K + 4 c a G I W 7 / D n X / j t M 1 C W w 9 c O J x z L / f e 4 0 W M S m X b 3 0 Z m Y X F p e S W 7 m l t b 3 9 j c M r d 3 6 j K M B S Y 1 H L J Q N D 0 k C a O c 1 B R V j D Q j Q V D g M d L w B l d j v / F I h K Q h f 1 D D i L g B 6 nH q U 4 y U l j r m 3 t 3 N d f n k r G A V j t u S 9 g J U t q 1 S q W P m b c u e A M 4 T J y V 5 k K L a M b / a 3 R D H A e E K M y R l y 7 E j 5 S Z I K I o Z G e X a s S Q R w g P U I y 1 N O Q q I d J P J + S N 4 q J U u 9 E O h i y s 4 U X 9 P J C i Q c h h 4 u j N A q i 9 n v b H 4 n 9 e K l X / h J p R H s S I c T x f 5 M Y M q h O M s Y J c K g h U b a o K w o P p W i P t I I K x 0 Y j k d g j P 7 8 j y p n 1 p O 0 S r e n + c r l 2 k c W b A P D s A R c E A J V M A t q I I a w C A B z + A V v B l P x o v x b n x M W z N G O r M L / s D 4 / A F l Y p M 3 < / l a t e x i t > MF E = 35.5, = 0.77 < l a t e x i t s h a 1 _ b a s e 6 4 = " L s M 4 K E B p R 6 4 T l k 5 U 1 0 F n O C A t j k k = " > A A A B / n i c b V D L S s N A F J 3 U V 6 2 v q L h y E y y C C w 1 J 1 d p N o S i K G 6 G C f U A b y m Q 6 a Y f O T M L M R C i h 4 K + 4 c a G I W 7 / D n X / j t M 1 C W w 9 c O J x z L / f e 4 0 e U S O U 4 3 0 Z m Y X F p e S W 7 m l t b 3 9 j c M r d 3 6 j K M B c I 1 F N J Q N H 0 o M S U c 1 x R R F D c j g S H z K W 7 4 g 6 u x 3 3 j E Q p K Q P 6 h h h D 0 G e 5 w E B E G l p Y 6 5 d 3 d z X T 5 x z + 3 T 4 7 Y k P Q b L j l 0 q d M y 8 Y z s T W P P E T U k e p K h 2 z K 9 2 N 0 Q x w 1 w h C q V s u U 6 k v A Q K R R D F o 1 w 7 l j i C a A B 7 u K U p h w x L L 5 m c P 7 I O t d K 1 g l D o 4 s q a q L 8 n E s i k H D J f d z K o + n L W G 4 v / e a 1 Y B S U v I T y K F e Z o u i i I q a V C a 5 y F 1 S U C I 0 W H m k A k i L 7 V Q n 0 o I F I 6 s Z w O w Z 1 9 e Z 7 U C 7 Z b t I v 3 Z / n K Z R p H F u y D A 3 A E X H A B K u A W V E E N I J C A Z / A K 3 o w n 4 8 V 4 N z 6 m r R k j n d k F f 2 B 8 / g B Z D Z M v < / l a t e x i t > MF E = 15.3, = 0.82 < l a t e x i t s h a 1 _ b a s e 6 4 = " I 0 / / T n G V O z Q O e d T l h S q F y 4 n 5 h f Q = " > A A A B / X i c b V D L S g M x F M 3 U V 6 2 v 8 b F z E y y C C x 1 m S q l u C k V R 3 A g V 7 A P a o W T S t A 1 N M k O S E e p Q / B U 3 L h R x 6 3 + 4 8 2 9 M 2 1 l o 9 c C F w z n 3 c u 8 9 Q c S o 0 q 7 7 Z W U W F p e W V 7 K r u b X 1 j c 0 t e 3 u n r s J Y Y l L D I Q t l M 0 C K M C p I T V P N S D O S B P G A k U Y w v J j 4 j X s i F Q 3 F n R 5 F x O e o L 2 i P Y q S N 1 L H 3 b q 4 u y y e F o l M 8 b i v a 5 6 j s O q W O n X c d d w r 4 l 3 g p y Y M U 1 Y 7 9 2 e 6 G O O Z E a M y Q U i 3 P j b S f I K k p Z m S c a 8 e K R A g P U Z + 0 D B W I E + U n 0 + v H 8 N A o X d g L p S m h 4 V T 9 O Z E g r t S I B 6 a T I z 1 Q 8 9 5 E / M 9 r x b p 3 5 i d U R L E m A s 8 W 9 W I G d Q g n U c A u l Q R r N j I E Y U n N r R A P k E R Y m 8 B y J g R v / u W / p F 5 w v J J T u i 3 m K + d p H F m w D w 7 A E f D A K a i A a 1 A F N Y D B A 3 g C L + D V e r S e r T f r f d a a s d K Z X f A L 1 s c 3 4 S C S 8 g = = < / l a t e x i t > MF E = 24.4, = 0.6 < l a t e x i t s h a 1 _ b a s e 6 4 = " h J m K W a s h x 4 u B 7 u r g 7 B z H O T 1 z j e A = " > A A A B / n i c b V D L S g M x F M 3 U V 6 2 v U X H l J l g E F z r M W K 1 u C k V R 3 A g V 7 A P a U j J p p g 1 N M k O S E c p Q 8 F f c u F D E r d / h z r 8 x b W e h 1 Q M X D u f c y 7 3 3 + B G j S r v u l 5 W Z m 1 9 Y X M o u 5 1 Z W 1 9 Y 3 7 M 2 t m g p j i U k V h y y U D R 8 p w q g g V U 0 1 I 4 1 I E s R 9 R u r + 4 H L s 1 x + I V D Q U 9 3 o Y k T Z H P U E D i p E 2 U s f e u b 2 + K h 0 V T p 3 C Y U v R H k c l 1 y l 4 H T v v O u 4 E 8 C / x U p I H K S o d + 7 P V D X H M i d C Y I a W a n h v p d o K k p p i R U a 4 V K x I h P E A 9 0 j R U I E 5 U O 5 m c P 4 L 7 R u n C I J S m h I Y T 9 e d E g r h S Q + 6 b T o 5 0 X 8 1 6 Y / E / r x n r 4 L y d U B H F m g g 8 X R T E D O o Q j r O A X S o J 1 m x o C M K S m l s h 7 i O J s D a J 5 U w I 3 u z L f 0 n t 2 P G K T v H u J F + + S O P I g l 2 w B w 6 A B 8 5 A G d y A C q g C D B L w B F 7 A q / V o P V t v 1 v u 0 N W O l M 9 v g F 6 y P b 1 M W k y s = < / l a t e x i t > MF E = 35.3, = 0.31 < l a t e x i t s h a 1 _ b a s e 6 4 = " s m w 7 W l s 3 8 R n G b R V p C K H F R f E e H M w = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o e Z U q u b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h R x 6 3 e 4 8 2 9 M 2 1 l o 6 4 E L h 3 P u 5 d 5 7 g o h R p V 3 3 2 1 p Y X F p e W c 2 s Z d c 3 N r e 2 7 Z 3 d m g p j i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 1 6 e S d / 0 l K 0 x 1 H J d c 4 K b T v n O u 4 E c J 5 4 K c m B F J W 2 / d X q h D j m R G j M k F J N z 4 2 0 n y C p K W Z k l G 3 F i k Q I D 1 C P N A 0 V i B P l J 5 P z R / D I K B 3 Y D a U p o e F E / T 2 R I K 7 U k A e m k y P d V 7 P e W P z P a 8 a 6 e + E n V E S x J g J P F 3 V j B n U I x 1 n A D p U E a z Y 0 B G F J z a 0 Q 9 5 F E W J v E s i Y E b / b l e V L L O 1 7 R K d 4 X c u X L N I 4 M O A C H 4 B h 4 4 B y U w S 2 o g C r A I A H P 4 B W 8 W U / W i / V u f U x b F 6 x 0 Z g / 8 g f X 5 A 1 F A k y o = < /V D L S s N A F J 3 U V 6 2 v q L h y M 1 g E F x q S q t F N o S i K G 6 G C f U A b y m Q 6 a Y f O J G F m I p R Q 8 F f c u F D EG j K M B S Z 1 H L J Q t D w k C a O c 1 B V V j L Q i Q V D g M d L 0 h p c T v / l A h K Q h vq I + a R k q E C f K T 6 b n j + G x U b q w F 0 p T Q s O p + n s i Q V y p E Q 9 M J 0 d 6 o O a 9 i f i f 1 4 p 1 r + Q n V E S x J g L P F v V i B n U I J 1 n A L p U E a z Y y B G F J z a 0 Q D 5 B E W J v E c i Y E b / 7 l R V I v O F 7 R K d 6 f 5 y u X a R x Z c A i O w A n w w A W o g F t Q B T W A Q Q K e w S t 4 s 5 6 s F + v d + p i 1 Z q x 0 Z h / 8 g f X 5 A 2 C a k z Q = < / l a t e x i t > MF E = 26.0, = 0.88 < l a t e x i t s h a 1 _ b a s e 6 4 = " T T F X Z o Z S t V b M z n W x u u Y D / n o N t q g = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o e Z K q 2 b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h Rj i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 2 e e U 7 + p K V o j 6 O S 6 x S K b T v n O u 4 E c J 5 4 K c m B F J W 2 / d X q h D j m R G j M k F J N z 4 2 0 n y C p K W Z k l G 3 F i k Q I D 1 C P N A 0 V i B P l J 5 P z R / D I K B 3 Y D a U p o e F E / T 2 R I K 7 U k A e m k y P d V 7 P e W P z P a 8 a 6 e + E n V E S x J g J P F 3 V j B n U I x 1 n A D p U E a z Y 0 B G F J z a 0 Q 9 5 F E W J v E s i Y E b / b l e V L L O 1 7 B K d y f 5 8 q X a R w Z c A A O w T H w Q B G U w S 2 o g C r A I A H P 4 B W 8 W U / W i / V u f U x b F 6x 0 Z g / 8 g f X 5 A 1 j l k y 8 = < / l a t e x i t > MF E = 31.2, = 0.67 < l a t e x i t s h a 1 _ b a s e 6 4 = " V + H 7 W S j v / d D g V c sQ 6 + 1 / T / Y U C p o = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o c Z K 7 W b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h R x 63 e 4 8 2 9 M 2 1 l o 6 4 E L h 3 P u 5 d 5 7 g o h R p V 3 3 2 1 p Y X F p e W c 2 s Z d c 3 N r e 2 7 Z 3 d m g p j i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 3 m C 4 5 7 0 l K 0 x 1 H J d Y r 5 t p 1 z H X c C O E + 8 l O R A i k r b / m p 1 Q h x z I j R m S K m m 5 0 b a T 5 D U F D M y y r Z i R S K E B 6 h H m o Y K x I n y k 8 n 5 I 3 h k l A 7 s h t K U 0 H C i / p 5 I E F d q y A P T y Z H u q 1 l v L P 7 n NW P d L f o J F V G s i c D T R d 2 Y Q R 3 C c R a w Q y X B m g 0 N Q V h S c y v E f S Q R 1 i a x r A n B m 3 1 5 n t T O H K / g F O 7 P c + X L N I 4 M O A C H 4 B h 4 4 A K U w S 2 o g C r A I A H P 4 B W 8 W U / W i / V u f U x b F 6x 0 Z g / 8 g f X 5 A 1 q Z k z A = < / l a t e x i t > MF E = 36.0, = 0.83 < l a t e x i t s h a 1 _ b a s e 6 4 = " + V C g H P h + m K g q f 9 d t R jV 8 f V Z c 3 C Y = " > A A A B / n i c b V D L S g M x F M 3 4 r P U 1 K q 7 c B I v g Q o c Z K 7 W b Q l E U N 0 I F + 4 B 2 K J k 0 b U O T z J B k h D I U / B U 3 L h R x 63 e 4 8 2 9 M 2 1 l o 6 4 E L h 3 P u 5 d 5 7 g o h R p V 3 3 2 1 p Y X F p e W c 2 s Z d c 3 N r e 2 7 Z 3 d m g p j i U k V h y y U j Q A p w q g g V U 0 1 I 4 1 I E s Q D R u r B 4 G r s 1 x + J V D Q U D 3 o Y E Z + j n q B d i p E 2 U t v e v 7 u 5 L p 3 m P c c 9 a S n a 4 6 j k O s V 8 2 8 6 5 j j s B n C d e S n I g R a V t f 7 U 6 I Y 4 5 E R o z p F T T c y P t J 0 h q i h k Z Z V u x I h H C A 9 Q j T U M F 4 k T 5 y e T 8 E T w y S g d 2 Q 2 l K a D h R f 0 8 k i C s 1 5 I H p 5 E j 3 1 a w3 F v / z m r H u F v 2 E i i j W R O D p o m 7 M o A 7 h O A v Y o Z J g z Y a G I C y p u R X i P p I I a 5 N Y 1 o T g z b 4 8 T 2 p n j l d w C v f n u f J l G k c G H I B D c A w 8 c A H K 4 B Z U Q B V g k I B n 8A r e r C f r x X q 3 P q a t C 1 Y 6 s w f + w P r 8 A V K / k y s = < / l a t e x i t > MF E = 31.0, = 0.83 < l a t e x i t s h a 1 _ b a s e 6 4 = " f v 7 r s A U e I w 0 v e H V 2 G 8 + E i F p y B / U K C J u g P q c + h Q j p a W u u Z / c 3 V x X T s + L V v m k I 2 k / Q B X b K h f T r l m w L X s C O E + c j B R A h l r X / O r 0 Q h w H h C v M k J R t x 4 6 U m y C h K G Y k z X d i S S K E h 6 h P 2 p p y F B D p J p M H U n i k l R 7 0 Q 6 G L K z h R f 0 8 k K J B y F H i 6 M 0 B q I G e 9 s f i f 1 4 6 V X 3 Y T y q N Y E Y 6 n i / y Y Q R X C c R q w R w X B i o 0 0 Q V h Q f S v E A y Q Q V j q z v A 7 B m X 1 5 n j T O L K d k l e 6 L h e p l F k c O H I B D c A w c c A G q 4 B b U Q B 1 g k I J n 8 A r e j C f j x X g 3 P q a t C 0 Y 2 s w f + w P j 8 A T g Q l E M = < / l a t e x i t > MF E = 34.8, = 0.84X i x + Z Q D g a p k r 2 E o q r S b o L Q = " > A A A C B 3 i c b V D L S s N A F J 3 4 r P U V d S l I s A g u N C S l V l 0 U i q K 4 E S r Y B z S h T K a T d u h k E m Y m Q g n d u f F X 3 L h Q x K 2 / 4 M 6 / c d J m o a 0 H L h z O u Z d 7 7 / E i S o S 0 r G 9 t b n 5 h c W k 5 t 5 J f X V v f 2 N S 3 t h s i j D n C d R T S k L c 8 K D A l D N c l k R S 3 I o 5 h 4 F H c 9 A a X q d 9 8 w F y Q k N 3 L Y Y T d A P Y Y 8 Q m C U k k d f c 8 J o O x 7 f n J 7 f V U 5 L p 6 Y 9 p E j S C + A F c s 8 L 4 0 6 e s E y r T G M W W J n p A A y 1 D r 6 l 9 M N U R x g J h G F Q r R t K 5 J u A r k k i O J R 3 o k F j i A a w B 5 u K 8 p g g I W b j P 8 Y G Q d K 6 R p + y F U x a Y z V 3 x M J D I Q Y B p 7 q T K 8 W 0 1 4 q / u e 1 Y + m f u Q l h U S w x Q 5 N F f k w N G R p p K E a X c I w k H S o C E S f q V g P 1 I Y d I q u j y K g R 7 + u V Z 0 i i a d t k s 3 5 U K 1 Y s s j h z Y B f v g E N j g F F T B D a i B O k D g E T y D V / C m P W k v 2 r v 2 M W m d 0 7 K Z H f A H 2 u c P v H G X U Q = = < / l a t e x i t > MFE = 25.1, = 0.94 < l a t e x i t s h a 1 _ b a s e 6 4 = " l y 0 + y d T K a R i e r R r E d M + S y c 6 I k M 8 = " > A A A C B 3 i c b V D L S s N A F J 3 U V 6 2 v q E t B B o v g Q k N S p O 2 m U B T F j V D B P q A J Z T K d t E M n D 2 Y m Q g n d u f F X 3 L h Q x K 2 / 4 M 6 / c d J m o a 0 H L h z O u Z d 7 7 3 E j R o U 0 z W 8 t t 7 S 8 s r q W XU G U C G C C C U G C C G C C G G G C A G C G G G C G A C A A U C A A G C C G G C G C U C C C G G C U C A G G G A G C G C C G G C C C A U C U G U G G G G G G G C G G G C C A A G G G C C C G C C C C C C G G G G C G G C G G C C C G G G G C C G C C G C C C C G G G U U C A C G C A C C C A G U G U G G C G G A C AThe comparison results of D-PBEMO against the other three state-of-the-art PBEMO algorithms on inverse RNA design problems. In particular, the target structure is a sample of possible solution represented in blue color while the predicted one obtained by different optimization algorithms are highlighted in red color. In this graph, the reference point is set as σ = 1. The closer σ is to 1, the better performance achieved by the corresponding algorithm. When the σ share the same biggest value, the smaller M F E the better the performance is.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This paper is foundational research and not tied to particular applications, let alone deployments. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).In TableA1, we use the i-th arm to denote the i-th subset in the context of PBEMO.https://github.com/ViennaRNA/ViennaRNAhttps://github.com/zhangzm0128/PCM-Protein-Structure-Predictionhttps://github.com/eternagame/eterna100-benchmarking 


Decision: This paper focuses on the application of dueling bandits to evolutionary multi-objective optimization. In particular, reference-based evolutionary multi-objective optimization (PBEMO) methods involve optimization (explore the space), consultation (learn human preference), and elicitation (guide evolutionary search). Existing PBEMO methods may suffer from inaccurate reward models, which is likely to happen given that human feedback exhibit a lot of randomness. The authors propose to directly leverage human feedback without using a reward model to guide the evolutionary search. Specifically, given that human feedback from relative comparison is better than absolute labels, the authors employ dueling bandits to compute preference metrics and to choose the winner using the Copeland score.

In summary the authors have done a great job explaining the unclear parts during the rebuttal phase, convincing several reviewers to raise their final scores. While the paper does not contribute that much to the core ML literature, the application of dueling bandits to the multi-objecive optimisation domain is significant.

Reviews: review: paper_summary: paper_summary: One challenge and potential advantage of multi-objective optimization (MO) is to adapt the dynamic human preferences while outputting an optimum. Although Preference-based evolutionary MO (PBEMO) is a promising framework, current approaches are inefficient and may not interpret the decision makers' true intentions accurately. One reason is that the decision makers' true intentions were not precisely "expressed" and "acknowledged" by the predefined reward model. This paper, intending to solve this reason, designs a framework that directly leverages human feedback, using a clustering-based stochastic dueling bandits algorithm.
strength_weakness: strength_weakness: Due to my very limited knowledge in this area, I might not be able to find valuable weaknesses in this paper.
questions: questions: In Section 3.2 "Preference Elicitation Module", why is Equation (4) the resulted mixture distribution? If it was computed, were there any justifications that this shall be the result? More explanations on this may be appreciated because, to the best of my understanding, this part plays a critical role in justifying the framework since it directly works on the preferences.
limitations: limitations: Yes.
paper_summary: paper_summary: Preference-based evolutionary multi-objective optimization (PBEMO) methods involve optimization (explore the space), consultation (learn human preference), and elicitation (guide evolutionary search). Existing PBEMO methods may suffer from inaccurate reward models, which is likely to happen given that human feedback exhibit a lot of randomness. The authors propose to directly leverage human feedback without using a reward model to guide the evolutionary search. Specifically, given that human feedback from relative comparison is better than absolute labels, the authors employ dueling bandits to compute preference metrics.
strength_weakness: strength_weakness: In step 1 of consultation module, one needs to choose K, which is the number of subsets to partition S into. The choice of K would greatly affect how close the solutions are within a subset. Thus, it is important to discuss methods to choose K and provide justifications. 

The runtimes are not reported.
questions: questions: Could the authors provide some insights on the advantage of starting with a coarse-grained representation, which may yield an initially inaccurate SOI, compared to having a set of Pareto-optimal candidate solution upfront?
limitations: limitations: The limitations are not clear. The scope of the claims should be discussed. I encourage the authors to create a separate limitations section (see NeurIPS paper checklist).
paper_summary: paper_summary: This paper focused on the problem of multi-objective optimization in the dueling bandits settings.  A clustering-based stochastic dueling bandits algorithm was developed and analysis. The performance is further validated via experiments.
strength_weakness: strength_weakness: - In dueling bandits literature, one of the most widely assumed winner or the most general winner is the Condorcet winner. In this paper, for example, in definition 2.1, the authors defined the Copeland winner. What is the intuition behind this or in other words, does your framework requires this specific winner, or can it be generalized to the general Condorcet winner (or other winners such as Borda Winner, Neumann Winner that are widely used in the dueling bandits literature)?
- In your algorithms and theorem, $\alpha>0.5$ is assumed. Is this a natural option in practice? 
- The proposed D-PBEMO framework consists of three modules as shown in Figure 1(a), while the regret is characterized for the consultation modules. Not sure if I miss anything, can this regret be claimed as that of D-PBEMO? 
- The writing of the paper can be significantly improved. It is hard to follow the paper with many mathematical notations not rigorously defined, and some details are missing.
questions: questions: See weakness above.
limitations: limitations: N/A.
paper_summary: paper_summary: This article focuses on the preference-based evolutionary multiobjective optimization. First of all, such problems are widely found in real engineering application scenarios, thus becoming one of the most popular research directions in the field of multi-objective optimization. 

Overall, in terms of the presentation of the article, this article is well-structured and clearly expressed. From the aspect of methodological design, this article is somewhat innovative. However, from the aspect of experimental design, this article seems to have some defects.
strength_weakness: strength_weakness: 1-In Section 4.1, I am curious as to what criteria the authors used to select the benchmark problems. For example, why didn't the authors choose the same series of benchmark problems as DTLZ7, WFG2, WFG4, etc.?
2-In Section 4.2, I am concerned that the comparison algorithm used in the experiment is SOTA in the domain.
3-Two variants, DPNSGA-II and DP MOEA/D, were designed in the article. However, the discussion between these two variants seems to be a bit thin. What are their respective strengths and weaknesses?
4-Similarly, I found the discussion in the experimental section to be overly simplistic, seeming to restate the results of the experiment at face value. But it doesn't analyze the reasons behind the results too much.
5-Population size, number of search iterations, and other such parameter settings do not seem to be given by the authors.
questions: questions: 1-In Section 4.1, I am curious as to what criteria the authors used to select the benchmark problems. For example, why didn't the authors choose the same series of benchmark problems as DTLZ7, WFG2, WFG4, etc.?
2-In Section 4.2, I am concerned that the comparison algorithm used in the experiment is SOTA in the domain.
3-Two variants, DPNSGA-II and DP MOEA/D, were designed in the article. However, the discussion between these two variants seems to be a bit thin. What are their respective strengths and weaknesses?
4-Similarly, I found the discussion in the experimental section to be overly simplistic, seeming to restate the results of the experiment at face value. But it doesn't analyze the reasons behind the results too much.
5-Population size, number of search iterations, and other such parameter settings do not seem to be given by the authors.
limitations: limitations: The experimental chapter needs to be further upgraded.

Comments: title: Raising my score from 6 (Weak Accept) to 7 (Accept)
Raising my score from 6 (Weak Accept) to 7 (Accept)
Response to Official Comment by Reviewer dDig
Response to "Thanks for the response"
Thanks for the response
Response to Official Comment by Reviewer hDtX
We are keen on engaging in further discussions
Additional justification of limitations
Response to the Reviewer FUFf
Thank you for the rebuttal
Response to the Reviewer dDig
comment: Thank you very much for confirming our effort!

Best regards,
Authors
After evaluating the comments of the other reviewers and the authors' responses, I determined to raise the score from 6 to 7.
Dear Reviewer dDig,

Thank you very much again for your positive feedback and confirmation on our effort. We would like to respectively ask whether you would like to raise your score from 6 (Weak Accept) to 7 (Accept) as mentioned at the end of your comment? Really sorry for chasing this, because the deadline is approaching.

Thank you very much again.

Best regards
Authors
We appreciate the reviewer's support in this work.
I appreciate the concise response by the authors. However, due to my very limited knowledge in this area, I will keep my score and confidence level.
Thank you very much for confirming our efforts. Anyway, if you have any other concerns or questions, we are happy to engage in further discussions :-)
I thank the authors for their response. The authors addressed my concerns on the choice of K, runtime, and limitations. I increased my score.
Dear reviewers,

Thank you very much for your time and efforts on engaging in this discussion. Meanwhile, we sincerely appreciate those reviewers who confirm our contributions and response, as well as elevating their scores.

If you have any further concerns, please feel free to let us know. We are keen on engaging in further discussions.

Best regards,

Authors
Dear **Reviewer hDtX**,

We just realized that our response to your **Limitations** concern was missing. Please find our response as follows.

```
Due to the space limitations, we only briefly discussed the limitations in the conclusion section of our current manuscript. In the camera-ready version, we will add a dedicated section to discuss the limitations. We list some examples as follows.

1. The regret analysis of our proposed clustering-based stochastic dueling bandits is for the optimal subset, i.e., the region of interest on the PF. It is not yet directly applicable to identify the exact optimal solution of interest. As part of our future work, we will work on efficient algorithms and theoretical study on the best arm identification in the context the preference-based EMO.
2. This paper only analyzes the regret of the consultation module. How to further analyze the convergence of the D-PBEMO as a whole remains unknown. This will also lead to the next step of our research. In particular, if it is successful, we may provide a radically new perspective to analyze the convergence of evolutionary multi-objective optimization algorithms.
```
Hope this can address your concern at this point. If you have any concerns and questions, we are more than happy to have further discussion. Thank you very much for your efforts and help.
Thank you very much for confirming our effort. We also sincerely appreciate your constructive suggestions on improving the quality of our work.
Thank you for the clarifications. I will increase the rating.
We sincerely appreciate the reviewer's positive feedback on our efforts and your kind help to elevate your rating. Meanwhile, we for sure will carefully revise our final version to make sure it stands at the highest quality possible.

Last but not the least, we would like to take this opportunity to thank yours and the other reviewers' constructive suggestions on our work. All of them give us insights about how to improve the quality of our work, as well as how to push this line of research forward.
I think the author did a great job of responding to all of my comments. From my personal point of view, I have no other problems.

Of course, I wish the author had placed some of the replies to my comments in the final version, as it would have prevented others from having similar doubts.

Based on the author's and other reviewers' comments, I think this article is of good quality and I would like to increase my rating for this article from 6 (Weak Accept) to 7 (Accept).