No Preference Left Behind: Group Distributional Preference Optimization

Authors: Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: preference alignment; large language model; fairness; group preferences

Abstract:
Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group's belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Additionally, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment.

Decision: accept (poster)
Reviewer scores: 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730192432338

thread_type:
official_review

<thread_object>
creator: reviewer_xrbo

<content>
summary:
This work studies on the preference alignment of LLMs and proposes a conditional distributional preference optimization method with utilizing belief information. However, this work has also some limitations on technique contributions and model applicability. As such, I think this work is bordline and relatively incline to negative.

strengths:
1. This work studies on an interesting problem.

2. The proposed method is simple and easily implemented.

3. Extensive experiments on bot synthetic and real-world datasets are conducted to validate the effectiveness of the proposed method. 

4. The paper is well-writing.

weaknesses:
1. The technical contribution appears limited. The proposed method is a simple extension of Distributional Preference Optimization (DPO), and the authors do not provide substantial insights to reveal the intricate properties of the proposed method.

I would suggest the authors conduct more analyses to demonstrate why the proposed strategy is crucial, potentially even a "game-changer" in this field. Theoretical analyses would be particularly beneficial.

2. Another concern pertains to the applicability of the proposed method, given that the belief distribution should be provided.

3. In terms of experiments, I have a suggestion: given that the authors claim their strategy can be integrated with various alignment losses, it would be advantageous to test the model performance with other losses beyond DPO (e.g., PPO, KTO) to demonstrate its merits.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537557698

thread_type:
official_review

<thread_object>
creator: reviewer_n41b

<content>
summary:
The paper introduces a novel framework called Group Distributional Preference Optimization (GDPO) designed to align language models with the diverse and pluralistic preferences within a group. Unlike existing methods such as DPO, which tend to skew towards dominant preferences and overlook the diversity of opinions, GDPO incorporates the concept of beliefs that shape individual preferences, calibrating models through statistical estimation of the group's belief distribution. Experiments on synthetic controllable opinion generation and real-world movie review datasets demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advancement in pluralistic alignment.

strengths:
1. The paper introduces a novel group-wise perspective in preference optimization, which significantly enhances the effectiveness and practicality of fine-tuning methods compared to existing DPO approaches that often skew towards dominant preferences.

2. The writing is direct and concise, making the paper easy to read and understand. The authors effectively convey complex ideas and methodologies.

3. The experimental design is precise and well-aligned with the core objectives outlined in the introduction. The experiments on both synthetic and real-world datasets clearly demonstrate the paper's contributions, reinforcing the effectiveness of the proposed GDPO framework.

weaknesses:
- **Belief Set Design**: The need to design specific belief sets for each dataset based on its domain characteristics may limit the scalability and generalizability of the proposed Group Distributional Preference Optimization  framework. This requirement adds an additional layer of complexity and could be a barrier to broader adoption.

- **Training Efficiency**: The training process for GDPO involves calculating the calibration loss $l_{\text{cal.}}$ for each belief in the set, leading to a significant increase in computational requirements. Specifically, the overall training time could be approximately $ |\mathcal{B}| $ times longer than that of conventional DPO, where $ |\mathcal{B}| $ is the number of beliefs. Addressing this efficiency issue is crucial for the practical implementation of GDPO.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706020238

thread_type:
official_review

<thread_object>
creator: reviewer_uotu

<content>
summary:
This paper investagates the issue that existing LLM alignment methods like DPO tend to favor majority preferences while overlooking minority views, failing to capture the full range of preferences within groups. To solve this issue, the authors propose to incorporate the concept of "beliefs" that shape individual preferences, use statistical estimation to model group belief distributions, and align the model with belief-conditioned preferences. At the inference time, a belief is selected first and then the response is generated conditioned on the selected belief. Experiments with both synthetic data and real-world movie reviews shows some improvement of the proposed GDPO.

strengths:
It is important to note the minority preference in current LLMs, since the LLMs tend to response with dominant preferences with in majority.

The proposed method is conceptually simple and easy to implement based on the details.

The paper is well-presented and easy to follow.

weaknesses:
While the motivation to note the minority preference is crucial, the proposed GDPO might not sufficiently fulfill the motivation.

1. The "belief" distribution is predefined, which makes it hard to take into account a wide range of preferences.

2. At the inference time, a "belief" is selected first. The selected "belief" could also overlook the preference of minority. 

3. In the experiment of movie review, the "belief" is implemented with rating scores. However, the rating routain of different persons may be varying. Besides, the rating score can hardly reflect the minority preference

questions:
1. How the proposed method could solve the issue of overlooking minority preference? Could the authors provide some intuitive explanation?

2. Could the authors provide some cases where the conflicted preference issues is resolved?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732787436466

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Revision Summary

comment:
We sincerely thank all reviewers for their valuable and constructive feedback on our paper. In response, we have carefully revised the manuscript and would like to take this opportunity to summarize the changes we made. All revisions are highlighted in blue for your convenience.

- **Addressing how GDPO resolves the issue of overlooking minority preferences**:
Reviewer Uotu raised a concern regarding how GDPO addresses minority preferences, which we believe stems from a misunderstanding of the term. To clarify, we have revised Section 3 (Preliminaries) to explicitly define minority preferences as those with the lowest proportion in the belief distribution (Lines 173–177). We further refer the reviewer to Figure 3 of the original submission, which illustrates that GDPO increases reward margins for minority preferences, in contrast to DPO. This demonstrates GDPO's capability to balance and optimize both majority and minority preferences simultaneously. Additionally, Table 3 provides examples of GDPO generating responses that reflect minority beliefs, offering additional evidence of its effectiveness.
- **Generalization of predefined belief sets**:
Reviewers raised concerns regarding the generalizability of GDPO's reliance on predefined belief sets. To address this, we discuss how to conduct the belief mining on other preference datasets in Appendix A. Specifically, using GPT-4o, we extract 23 belief statements from 300 randomly selected examples in HH dataset and estimated a proxy for the belief distribution by aggregating all examples containing these statements. Moreover, in the Introduction (Lines 052–081), we explain how belief distributions can help resolve conflicting preferences in alignment methods. The definition of beliefs has also been updated, with additional references provided in Section 4 (Group Distributional Preference Optimization) on Lines 198–201.
- **Discussion of training efficiency**:
To provide more insights into GDPO's computational performance, we have added a discussion on its training efficiency in Appendix B. In short, GDPO adds a modest computing overhead, measured empirically at 16%, compared to DPO.
- **Extending GDPO beyond DPO**:
Reviewer xrBo suggested evaluating GDPO on methods beyond DPO to enhance its impact. We agree and have started experiments with the KTO method. However, due to time constraints, the updated results could not be included in the current submission. We will share our findings on the forum as soon as they are available.

Once again, we extend our heartfelt thanks to all reviewers for their efforts and insightful comments. We kindly encourage you to review our revised submission and hope that our updates adequately address your concerns. We would greatly appreciate it if you consider revising your scores accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733297531927

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We sincerely thank all reviewers for their insightful comments on our paper. We'd like to take this opportunity to summarize our key contributions and address the main concerns raised during the review process.

**Key Contributions**
- We introduce group distributional preference optimization task, a novel perspective for group-wise alignment that ensures models capture and represent the diversity within a group.
- We identify challenges in current alignment methods, such as DPO, which skew toward dominant preferences during training, making them inadequate for achieving group distributional preference alignment.
- We integrate the concept of human belief from epistemology into pluralistic preference alignment. This innovation addresses DPO’s limitation of overlooking minority preferences, resulting in a more inclusive and representative language model.

**Replies to Key Concerns**
- **Applying "Belief" to General Preference Alignment Tasks**:
The concept of belief is not tailored to specific tasks but is intrinsic to preference alignment, as human preferences are fundamentally shaped by underlying beliefs from an epistemological perspective. We have added a belief mining section to the appendix, focusing on the belief mining of HH dataset using GPT-4o. Human evaluation shows that GPT-4o achieves 83% accuracy in annotating beliefs on HH. It show that belief mining on existing preference alignment datasets is feasible and efficient with the language understanding capabilities of LLMs. 
- **Integrating GDPO with Various Alignment Losses**:
Additional experiments integrating GDPO with KTO yield results consistent with those from our DPO-based experiments ([details here](https://openreview.net/forum?id=bgpNJBD6Va&noteId=zjUBFjBNhB)). These experiments demonstrate that GDPO is compatible with paired preference alignment methods (e.g., DPO) and adaptable to unpaired methods (e.g., KTO). This supports our claim that GDPO is a generic framework applicable to a wide range of alignment losses, rather than a mere incremental extension of DPO.

We greatly appreciate the reviewers' valuable feedback, which has helped us refine and strengthen our work! We will include these experimental results and updates in the next version of our paper. Thanks for your patience and contributions again!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper identifies the issue in existing consistency optimization methods for large language models (LLMs), which prioritize majority preferences within a group while overlooking minority preferences. To address this problem, the authors propose a novel framework, GDPO, which estimates the distribution of beliefs within a group and uses it as a condition to align preferences, enabling a more inclusive reflection of diversity.

The focus on addressing the important and interesting challenge of considering minority preferences is a significant contribution. The explanations and experiments, which were strengthened through discussions with reviewers, further support the claims of the paper.

While challenges remain, such as the depth of the technical contribution and the reliance on predefined beliefs and the quality of belief generation, the overall ideas presented in this paper are worth sharing with the research community.