Pareto-Optimal Learning from Preferences with Hidden Context

Authors: Ryan Boldi, Li Ding, Lee Spector, Scott Niekum
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Preference Learning, Lexicase Selection, Pareto-optimality, Hidden Context, Pluralistic Alignment

Abstract:
Ensuring AI models align with human values is essential for their safety and functionality. Reinforcement learning from human feedback (RLHF) leverages human preferences to achieve this alignment. However, when preferences are sourced from diverse populations, point estimates of reward can result in suboptimal performance or be unfair to specific groups. We propose Pareto Optimal Preference Learning (POPL), which enables pluralistic alignment by framing discrepant group preferences as objectives with potential trade-offs, aiming for policies that are Pareto-optimal on the preference dataset. POPL utilizes lexicase selection, an iterative process that selects diverse and Pareto-optimal solutions. Our theoretical and empirical evaluations demonstrate that POPL surpasses baseline methods in learning sets of reward functions and policies, effectively catering to distinct groups without access to group numbers or membership labels. We verify the performance of POPL on a stateless preference learning setting, a Minigrid RL domain, Metaworld robotics benchmarks, as well as large language model (LLM) fine-tuning. We illustrate that POPL can also serve as a foundation for techniques optimizing specific notions of group fairness, ensuring safe and equitable AI model alignment.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730094353199

thread_type:
official_review

<thread_object>
creator: reviewer_isud

<content>
summary:
The paper tries to solve the problem of learning from complex human preference by first formulating it as the problem of preference learning with hidden context, and then relying on the framework and techniques of multi-objective optimization. The Pareto Optimal Preference Learning (POPL) method, based on the lexicase selection algorithm, is proposed to efficiently learn and generate diverse rewards and policies. The extensive and diverse experiments are conducted to illustrate the efficiency and applicability of the method.

strengths:
1. The problems of previous methods are clarified with a simple yet illustrative example.
2. The intuitions and theoretical formulations in Section 5.1 are interesting and may be useful for future research.
3. Experiments are diverse and persuasive, ranging from analytic environments to language model experiments, which show the proposed method’s efficiency and applicability.

weaknesses:
1. It would be better to concisely introduce the theory proposed in Section 4 in the introduction Section.
2. Paragraph headings in Section 2 can make them more clear.
3. In line 147, the comma ahead of ‘However’ should be a full stop.
4. The process of lexicase selection for preference learning in Section 5.2 can be made more clear with figures, formulas, or diagrams.

questions:
1. By formulating the RLHF-HC problem as a multi-objective problem, how will the method deal with (non-)**transitivity** of real-world human preference ? As this could be a problem for real-world human feedback (See ,e.g., section 7.3 of [1]).
2. ‘Theory Foundation’ in the title of Section 4 seems to be missing, or it has been formulated in Section 5 ? A re-organization may be needed.
3. Why the probability of segments in Definition 1 does not include the distributions of the states, such as $\prod_{(s,a)\in\sigma}\pi(a|s)d(s)$ ?
4. In line 258, is the word ‘illicit’ a typo ? It may be ‘elicit’.
5. In the proofs in Appendix B, what does $\pi_z$ refer to ? Is it just $\pi_z^*$ ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730323752054

thread_type:
official_review

<thread_object>
creator: reviewer_7xux

<content>
summary:
POPL enhances alignment by addressing diverse human preferences as multiple objectives, achieving solutions that are Pareto-optimal and fair across groups. It uses lexicase selection to find policies that respect varied preferences across. Empirical results show POPL’s effectiveness in aligning AI to human values safely and equitably across applications like robotics and language models.

strengths:
- This paper provide a novel way to deal with the heterogenity among different sourses
- Various experiments under different settings are conducted to evaluate the performance

weaknesses:
- Noations and concepts are sometimes not well-defined. For instance, $\sigma$ first occur in Section 3 without any definitions. I would suggest the authors to more clearly defined notations.
- The presentation of experiment results is quite unclear. For example, Figure 3(b) is really chaotic and I can hardly tell the information here.
- Lack of comparision with other methods. From my interpretation, many methods are proposed to solve the similar issue, e,g., Nash learning for RLHF and general preference framework. The methods should be more properly evaluated and compared.

questions:
-  What's the formal definition of $\sigma$, is it always a $(s,a)$ pair?
- Could authors provides more clearer highlight of the technical contributions?
- For me, even though the definitions are provided, the concept is still confused. Could authors gives the mathemitical definitions of concepts and theorems in Section 5.1, e.g., optimality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706212105

thread_type:
official_review

<thread_object>
creator: reviewer_jqrr

<content>
summary:
This work concentrates on the issue that human feedback data might involve noises caused by hidden information. It proposes a method named POPL that aims to learn a Pareto-optimal policy. POPL uses lexicase selection and conducts experiments on different tasks.

strengths:
1. This work concentrates on an import problem that human feedback involves hidden information. The Pareto-optimal is indeed one possible solution.

2. Experiments on different tasks are given.

weaknesses:
1. I found the presentation for the method is quite hard for me to understand. Lexicase selection, as key idea of the method, is not introduced clearly. I am not clear about how this selection method is conducted. Also, I think a figure for process might be better for readers to understand.

2. Similarly, I found that many concepts are used without a clear explanation. For example, I am not clear what "hypotheses" refers to as it first shows in Sec. 6.1 and also in Alg. 1. Also, as MDPL is mentioned, this formal calculation for this method is not give. I cannot under stand the illustration of Fig.3.

3. Further, if more intuition about the method is provided, it might be better for readers to understand this method.

4. Since this method concentrates on the RLHF alignment process, I am curious about the comparison between POPL and more RLHF methods like DPO. Also, the detailed information for the standard RLHF is not given.

questions:
See the weakness part above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730787756319

thread_type:
official_review

<thread_object>
creator: reviewer_g5ys

<content>
summary:
This paper introduces a reinforcement learning from human feedback with hidden context (RLHF-HC) framework called Pareto optimal preference learning (POPL). While marginalized distributional preference learning (MDPL) marginalizes $u(s\vert z)$ over the hidden context variable $z$ to produce a distribution $u(s)$ for each state $s\in\mathcal{S}$, POPL preserves the full conditional distributions. By leveraging these conditional probabilities, POPL can effectively capture Pareto optimal reward functions.

strengths:
POPL can capture the Pareto optimal reward functions.

weaknesses:
- The validity of the proposed algorithm is not theoretically explained.
- The scalability of the proposed algorithm remains uncertain.
- There is no way to determine the dimensionality of the underlying reward functions or the number of Pareto optimal policies.

questions:
I am unsure if I fully understand the paper. If I have any misunderstandings, please let me know. I would be happy to receive clarification.

1. As I understand it, there is no learning in Algorithm 1, which represents one step of POPL. In my understanding, it solely relies on the random initialization of hypotheses and the selection of good hypotheses. With a large number of trials, it might eventually find good policy hypotheses, but I wonder if this method is truly practical.
2. In general, there are many policies that passes exactly the same preference subsets. However, Algorithm 1 only checks whether the candidates pass at least on preference or not. Then, how to determine a Pareto-optimal policy among these policies?
3. In Algorithm 1, if we use POPL to learn a set of reward functions, how can we specify the dimensionality of these reward functions and learn them from the given demonstrations?
4. Compared to the previous works, such as preference-driven MORL [1] or multi-objective alignment in LLM [2], what is the main difference and advantage of POPL?
5. I cannot understand the left side of Figure 2 (titled “MDPLs”). In this case, what does z indicate?

### References

[1] Basaklar, Toygun, Suat Gumussoy, and Umit Y. Ogras. "Pd-morl: Preference-driven multi-objective reinforcement learning algorithm." ICLR 2023.

[2] Yang, Rui, et al. "Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment." arXiv preprint arXiv:2402.10207 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1731988872638

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Dear Reviewers,

Thank you all for taking the time to carefully review our paper and provide thoughtful feedback. we truly appreciate your suggestions and have worked to address all concerns raised to improve the clarity, structure and depth of our manuscript. 

We are encouraged by your recognition of the importance of our work and its potential impact. For example, various reviewers mention that we focus on "an import[ant] problem" (Reviewer jQRr) and that POPL is a novel solution for this problem (Reviewer 7xUX). We are also happy to hear that you believe the theory we performed to be interesting and helping to direct future research (Reviewer isUD). Furthermore, we are thrilled by the recognition that our "experiments are diverse and persuasive, ranging from analytic environments to language model experiments, which show the proposed method’s efficiency and applicability.".

In response to the suggestions brought forward by various reviewers, we have incorporated the following changes to our paper:
- Restructuring of the presentation of our work to help improve flow and explanation
- We updated the figure outlining the issues with the use of MDPLs to help improve its accessibility.
- We simplified figure 3 (now figure 4) to concentrate on the main take-aways. 
- We added aggregative results for the Metaworld experiments, to help show how our method scales and is robust to stochasticity.
- Expanded the introduction with concise highlights of our theoretical contributions and technical novelty. 

We believe that these changes make our paper stronger and address the concerns raised in your reviews. Thank you again for your constructive feedback, and we hope you find the revised version compelling.

Sincerely,
Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732536785217

thread_type:
official_comment

<thread_object>
creator: area_chair_xm1x

<content>
comment:
Dear Reviewers,


This is a friendly reminder that the discussion will end on Nov. 26th (anywhere on Earth). If you have not already, please take a close look at all reviews and author responses, and comment on whether your original rating stands. 


Thanks, 

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Pareto Optimal Preference Learning (POPL) to enable pluralistic alignment by framing discrepant group preferences as objectives with potential trade-offs, aiming for policies that are Pareto-optimal on the preference dataset.

The proposed Pareto-optimal solution is a promising direction for improving RLHF alignment and The concept of Pareto optimality is well-integrated into the framework. It also includes a wide range of experiments across different tasks.

The main concern is the validity of the proposed algorithm is not sufficiently supported by theoretical explanations. There is a need for more formal justification to back up the claims made about the algorithm's effectiveness. Moreover, the scalability of the proposed algorithm is not clearly addressed. It remains uncertain how well the method will perform as the complexity of the task or the number of reward functions increases.