Title: Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandits
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-objective optimization

dueling bandit

interactive multi-objective optimization

preference learning
Abstract: The ultimate goal of multi-objective optimization (MO) is to assist human decision-makers (DMs) in identifying solutions of interest (SOI) that optimally reconcile multiple objectives according to their preferences. Preference-based evolutionary MO (PBEMO) has emerged as a promising framework that progressively approximates SOI by involving human in the optimization-cum-decision-making process. Yet, current PBEMO approaches are prone to be inefficient and misaligned with the DM’s true aspirations, especially when inadvertently exploiting mis-calibrated reward models. This is further exacerbated when considering the stochastic nature of human feedback. This paper proposes a novel framework that navigates MO to SOI by directly leveraging human feedback without being restricted by a predefined reward model nor cumbersome model selection. Specifically, we developed a clustering-based stochastic dueling bandits algorithm that strategically scales well to high-dimensional dueling bandits, and achieves a regret of $\mathcal{O}(K^2\log T)$, where $K$ is the number of clusters and $T$ is the number of rounds. The learned preferences are then transformed into a unified probabilistic format that can be readily adapted to prevalent EMO algorithms. This also leads to a principled termination criterion that strategically manages human cognitive loads and computational budget. Experiments on $48$ benchmark test problems, including synthetic problems, RNA inverse design and protein structure prediction, fully demonstrate the effectiveness of our proposed approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article focuses on the preference-based evolutionary multiobjective optimization. First of all, such problems are widely found in real engineering application scenarios, thus becoming one of the most popular research directions in the field of multi-objective optimization. \n\nOverall, in terms of the presentation of the article, this article is well-structured and clearly expressed. From the aspect of methodological design, this article is somewhat innovative. However, from the aspect of experimental design, this article seems to have some defects.\n\nSTRENGTHS:\nThe designed method aims to address the issues exists in both consultation and preference elicitation modules. \n\nTo be specifically, the authors design a clustering-based stochastic dueling bandits algorithm to overcome the problem of convergence due to a large amount of preference comparisons. Additionally, the authors applies the Gaussian mixture distribution to leverage the preference learned from the consultation session.\n\nWEAKNESSES:\n1-In Section 4.1, I am curious as to what criteria the authors used to select the benchmark problems. For example, why didn't the authors choose the same series of benchmark problems as DTLZ7, WFG2, WFG4, etc.?\n2-In Section 4.2, I am concerned that the comparison algorithm used in the experiment is SOTA in the domain.\n3-Two variants, DPNSGA-II and DP MOEA/D, were designed in the article. However, the discussion between these two variants seems to be a bit thin. What are their respective strengths and weaknesses?\n4-Similarly, I found the discussion in the experimental section to be overly simplistic, seeming to restate the results of the experiment at face value. But it doesn't analyze the reasons behind the results too much.\n5-Population size, number of search iterations, and other such parameter settings do not seem to be given by the authors.\n\nQUESTIONS:\n1-In Section 4.1, I am curious as to what criteria the authors used to select the benchmark problems. For example, why didn't the authors choose the same series of benchmark problems as DTLZ7, WFG2, WFG4, etc.?\n2-In Section 4.2, I am concerned that the comparison algorithm used in the experiment is SOTA in the domain.\n3-Two variants, DPNSGA-II and DP MOEA/D, were designed in the article. However, the discussion between these two variants seems to be a bit thin. What are their respective strengths and weaknesses?\n4-Similarly, I found the discussion in the experimental section to be overly simplistic, seeming to restate the results of the experiment at face value. But it doesn't analyze the reasons behind the results too much.\n5-Population size, number of search iterations, and other such parameter settings do not seem to be given by the authors.\n\nLIMITATIONS:\nThe experimental chapter needs to be further upgraded.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper focused on the problem of multi-objective optimization in the dueling bandits settings.  A clustering-based stochastic dueling bandits algorithm was developed and analysis. The performance is further validated via experiments.

STRENGTHS:
- A new framework via combining dueling bandits and evolutionary multi-objective optimization was presented. 
- The proposed algorithm has been applied to real-world problem, i.e., protein structure prediction, which is interesting and promising.

WEAKNESSES:
- In dueling bandits literature, one of the most widely assumed winner or the most general winner is the Condorcet winner. In this paper, for example, in definition 2.1, the authors defined the Copeland winner. What is the intuition behind this or in other words, does your framework requires this specific winner, or can it be generalized to the general Condorcet winner (or other winners such as Borda Winner, Neumann Winner that are widely used in the dueling bandits literature)?
- In your algorithms and theorem, $\alpha>0.5$ is assumed. Is this a natural option in practice? 
- The proposed D-PBEMO framework consists of three modules as shown in Figure 1(a), while the regret is characterized for the consultation modules. Not sure if I miss anything, can this regret be claimed as that of D-PBEMO? 
- The writing of the paper can be significantly improved. It is hard to follow the paper with many mathematical notations not rigorously defined, and some details are missing.

QUESTIONS:
See weakness above.

LIMITATIONS:
N/A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

comments: SUMMARY:
Preference-based evolutionary multi-objective optimization (PBEMO) methods involve optimization (explore the space), consultation (learn human preference), and elicitation (guide evolutionary search). Existing PBEMO methods may suffer from inaccurate reward models, which is likely to happen given that human feedback exhibit a lot of randomness. The authors propose to directly leverage human feedback without using a reward model to guide the evolutionary search. Specifically, given that human feedback from relative comparison is better than absolute labels, the authors employ dueling bandits to compute preference metrics.

STRENGTHS:
The authors prove the regret bound of proposed algorithm, which is better than that of Thompson sampling based dueling bandit algorithms. Empirical evaluations on 33 settings showcase the effectiveness of proposed approach.

WEAKNESSES:
In step 1 of consultation module, one needs to choose K, which is the number of subsets to partition S into. The choice of K would greatly affect how close the solutions are within a subset. Thus, it is important to discuss methods to choose K and provide justifications. 

The runtimes are not reported.

QUESTIONS:
Could the authors provide some insights on the advantage of starting with a coarse-grained representation, which may yield an initially inaccurate SOI, compared to having a set of Pareto-optimal candidate solution upfront?

LIMITATIONS:
The limitations are not clear. The scope of the claims should be discussed. I encourage the authors to create a separate limitations section (see NeurIPS paper checklist).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
One challenge and potential advantage of multi-objective optimization (MO) is to adapt the dynamic human preferences while outputting an optimum. Although Preference-based evolutionary MO (PBEMO) is a promising framework, current approaches are inefficient and may not interpret the decision makers' true intentions accurately. One reason is that the decision makers' true intentions were not precisely "expressed" and "acknowledged" by the predefined reward model. This paper, intending to solve this reason, designs a framework that directly leverages human feedback, using a clustering-based stochastic dueling bandits algorithm.

STRENGTHS:
1. This paper tackles an interesting and important problem, and the method is novel. In particular, it is certainly valuable to directly express the human feedback into the framework, as human feedback is the critical information for preferences outputs.

2. The application of multiple arms and Copeland winner as the main decision-making criteria indeed makes sense to me. Although different, it resembles the rank computation in many RLHF schemes to some extent.

WEAKNESSES:
Due to my very limited knowledge in this area, I might not be able to find valuable weaknesses in this paper.

QUESTIONS:
In Section 3.2 "Preference Elicitation Module", why is Equation (4) the resulted mixture distribution? If it was computed, were there any justifications that this shall be the result? More explanations on this may be appreciated because, to the best of my understanding, this part plays a critical role in justifying the framework since it directly works on the preferences.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13