No Preference Left Behind: Group Distributional Preference Optimization
Authors: Binwei Yao
Zefan Cai
Yun-Shiuan Chuang
Shanglin Yang
Ming Jiang
Diyi Yang
Junjie Hu
Venue: ICLR 2025 Poster (2025)
Keywords: preference alignment; large language model; fairness; group preferences
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group's belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Additionally, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment.

Published as a conference paper at ICLR 2025
No PREFERENCE LEFT BEHIND: GROUP DISTRIBU- TIONAL PREFERENCE OPTIMIZATION
Binwei Yao', Zefan Cai', Yun-Shiuan Chuang' Shanglin Yang', Ming Jiang', Diyi Yang", Junjie Hu 'University Wisconsin-Madiise Indiana University Indianapolis, Stanford University (binwei.yae junjie huw@wwwes swwisc... edu
ABSTRACT
Preferences within group of people are not uniform but follow distribution. While existing alignment methods like Direct Preference Optimization (DPO) at- tempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), novel framework that aligns language models with the distribution of preferences within a group by incor- porating the concept of beliefs that shape individual preferences. GDPO cal- ibrates language model using statistical estimation of the group's belief dis- tribution and aligns the model with belief-condittoned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movic re- view datasets, show that DPO fails to align with the targeted belief dis- tributions, while GDPO consistently reduces this alignment gap during train- ing. Moreover, evaluation metrics demonstrate that GDPO outperforms isting approaches aligning with group distributional preferences, marking a significant advance in pluralistic alignment. Our data and code are released com cttp:/cccccciiia/ca /ttthubb.cmm/..gg1mmm//////
https:/
INTRODUCTION
Recent studies (Durmus et al. 2023 Jarrett et al., 2023] Chuang et al., 2024b) have explored the potential of large language models (LLMs) to reflect people's opinions and preferences across wide range topics. This line of research opens up various possibilities, such as modeling the opinions of population specific topics (Santurkar et al. (023), estimating human responses to surveys or new policies (Argyle et al., [0e3]]Aher Aher et al. 2023). and even simulating hypothetical social interactions between individuals Park al.) 2022
2023 Chuang et al. 2024ap. As result, aligning LLMs to accurately reflect diverse preferences within a group is crucial. However, existing alignment approaches like Reinforcement Learning from Human Feedback (RLHF; Ouyang al. 2022) and Direct Preference Optimization (DPO; Rafailov et al. (2024) still struggle to capture the diversity of opinions that exist within group (Durmus al. [223 et al. 2023). For instance, we ask a group of people the following question: "Is the availability of overseas products good for our country?". we are likely get diverse responses from individuals living in the same county, in the same town, or even in the same family (Kirk et al.] [224).. This highlights that human preferences, even within a single group, are not singular but fact distributional, or, pluralistic. This creates the challenge of conflicting preferences, where some annotators prefer one response while others prefer a different one. As result, existing alignment algorithms (e.g., DPO) may be adversely affected by pairwise noise (Wu et al., 2024). as they assume the existence of shared preference among people. This leads to a focus on seeking commonsense rather than representing diverse preferences during the alignment process. Consequently, the alignment results become biased toward dominant preferences, overlooking minority preferences within the group. With this challenge in mind, we focus research question:
(RLHF:|Ouyang

---

Published as a conference paper at ICLR 2025
Training Time
Inference Time
Training Dataset
Training Objective
Belief-Conditioned Preference Dataset
Direct Preference Optimization (DPO)
New Query
Xithw de you sarsion He obundbrce of antase prossci from various region of wand inppiing and mangprriig Lawrles
Xi optise, sfat the siriffencce of me exstencity of produts pron afferdt portrat sontt country
Eoro
Pirat ,r,
Selief Distribution pip
Preference Alignment
Predicted Beliel Distribution
D5
P(G,) = 0.5
P=Oils) po())) px(;/x) > Pa(U())
Conflictng Preferences
Po(8lx) 65
Pd
P(B,) = 0.1
Ps
Group Distributional
Ps (1p)
food
Gondl
bat Dg-BB((b)
Preference Optimization (GDPO)
P
Good
Good
bed
Somewritt Good
Someewoo
Eneo (Po (Po.))P) (b)x).p)) Pr yr, De.
h (()))
Someesho Good
Bellef-Conditioned Generation
Delief-Conditiones Generation
P(y(b, x)
P(y|b,x)
Divergence
Beldit-Codddd Preference Aligrmennt
scove wide pookec
crs, ste how oceerc shre prodber,
LIL
NOCHH. po(y(k.b))
foor coflective aptieire Anie hunyed the range of produms from partr
froe aftioee the wast o age of andachand oookito..
noor
Ph
Pu(ā(x)
sypertts au inal commeny do aot
Figure Demonstration of GDPO. Training Dataset: We create the belief-conditioned prefer- ence datasets for training, where people's beliefs on topic are diverse according to a specified distribution, and their preferences are conditioned on those beliefs. Training Objective: Instead of optimizing all preferences simultancously. GDPO first calibrates the belief distribution, followed by belief-conditioned preference alignment. Inference Time: When a new query is received, the model predicts a belief and generates responses based on it.
How can a probabilistic LLM align with distributional preferences within a group?
To answer this question, we propose Group Distribution Preference Optimization (GDPO), a novel framework designed to pluralistically align LLMs with the distributional preferences of group. In particular, we introduce the concept of belief from epistemology. which represents the degree which individuals agree with particular stance, to the preference alignment process. Prefer- ences are inherently shaped by people's beliefs (Sharma et al.) 2023). Conflicting preferences exist when individuals hold different beliefs about the same topic. Motivated by this epistemological phenomenon, our GDPO framework optimizes two main objectives to ensure that conflicting prefer- ences no longer interfere with one another. First, to encourage the model to generate diverse beliefs that reflect the group distribution, we calibrate the model's 's belief predictions using statistical esti- mate of the belief distribution within a group. Second, we introduce a ceddef-conditioned preference alignment objective to resolve preference conflicts by constructing preference data pairs conditioned on their corresponding preferred beliefs. Similar to the DPO training, we begin GDPO training from checkpoint derived from supervised fine-tuning (SFT).
To validate GDPO's performances on group distributional preference alignment, we apply GDPO to two tasks: 1) controllable opinion generation on synthetic dataset; 2) controllable review gen- eration on a real-world dataset. For the controllable opinion generation, the model first generates an opinion its belief, then produces response expressing that opinion (as the training dataset in Figure D. We construct the synthetic dataset to simulate one-turn dialogues that reflect diverse opinions from various countries, using GlobalOpinionQA (Durmus al. 2023), multi-choice question-answer dataset focused on global issues. For the controllable review generation, we use movie reviews from Amazon users to build the preference dataset. Here, the model is required to predict a rating score as the belief first, then generate a review that justifies that score. Our experi- ments, conducted using GPT-2 Large (Radford et al., (019) and Pythia-2.8B (Biderman al., 2023). reveal that DPO faces challenges in aligning with the target distribution, particularly when dealing with minority beliefs. Specifically, we observe that the gap between the predicted and actual belief' distributions widens significantly during DPO training. In contrast, GDPO effectively mitigates this issue, enabling the model to progressively approximate the target distribution throughout training. Moreover, introduce two sets of evaluation metrics to assess distribution calibration and condi- tional generation performance. Our results demonstrate that GDPO outperforms existing preference alignment methods, excelling in both distribution alignment and conditional generation tasks.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Preference Alignment. LLMs require additional training with human feedback to better align their outputs with human preferences, such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang al. 2022) and Direct Preference Optimization (DPO) (Rafailov et al.) 2024). Both methods train on datasets of pairwise human preferences (Bai et al. 2022), aiming to maxi- mize rewards for Human--rreferee responses while minimizing rewards for dispreferred responses. Although both methods have shown significant success in aligning LLMs to human preferences, they operate under the assumption that preferences are universal, treating conflicting preferences as noise rather than meaningful variations to model (Cui et al. 2023). Recent efforts have focused on making preference aligmment methods more robust to conflicting preferences. For instance, there is work enhancing the robustness of DPO (Wu et al. 2024) by incorporating techniques from dis- tributionally robust optimization (DRO) Sagawa et al. 2019). This approach aims to make models more resilient to pairwise noise by mitigating the impact of noisy data. However, in tasks involving group distributional preference alignment, the goal different. Here, we expect the LLM to learn from the distribution of preferences across groups and generate diverse outputs that comprehensively represent the group's 's varied perspectives.
Pluralistic Preference Alignment. Alignment techniques like RLHF and DPQ face inherent lim- itations in achieving truly pluralistic AI, which aims to accommodate a broader range of human preferences (Sorensen et al., 2024). A promising approach to addressing this challenge involves aligning Al with multi-group preferences. Ramesh et al. (2024), Chakraborty et al. (2024), and Chen et al. (2024) propose frameworks for aligning LLMs with the preferences of different groups, acknowledging that populations can have divergent preferences. Mcanwhile, other researchers focus on personalized alignment the individual user level. Poddar al. al.] (2024), [Shaikh al. (2024), Li jet al. (2024b), and Gao et al. (2024) explore methods that tailor LLM outputs based on users' iden- tifiers, interaction histories, and behaviors. Despite progress in pluralistic alignment, there still limited research on how to represent and model diverse preferences within single group so that multiple samplings of the model yield outputs as diverse as the population itself. /Zhao et al. (2023) attempts to address this by aligning LLM outputs with group preference distributions using few-shot learning techniques, but their experiments are restricted multiple-choiee question-an answer tasks, which fall short of capturing the complexity of real-wor world conversational language. Li et al. (2024a) introduces distributional preference reward modeling to align with diverse preferences. However, remains unclear whether the policy model can learn the same distribution through RLHF Siththaran- jan et al. (2023) proposes applying the distributional preference learning to RLHF. However, RLHF has significant computational costs. To tackle these challenges, propose GDPO, reward-fres alignment method, to align with distributional group preferences controllable generation settings.
PRELIMINARY
125
1.00
1.00
0.75
Reward Margins
Reward Margins
0.75
0.50
0.25
Other
0.50
Other
Majority
0.25
Majority
0.00
Minority
0.00
Minority
-0.25
-025
-0.50
0.75
-0.75
10000
20000
30000
40000
son00
20000
40090
6n050
#o000
100000
Training steps
Training steps
(a) GPT-2 Large
(b) Pythia-2.8B
Figure 2: Reward Margins During DPO Training: Majority/Minority means the chosen response ye from majority/minority preferences in the dataset.
3

---

Published as a conference paper at ICLR 2025
The training of Direct Preference Optimization (DPO) consists of two phases: 1) supervised fine- tuning (SFT) on the instruction dataset to get the model Tee 2) preference alignment using the DPO algorithm. The DPO algorithm is designed to reduce the computational complexity involved in train- ing a separate reward model in Reinforcement Learning from Human Feedback (RLHF). Through mapping reward functions optimal policies, DPO trains the model in reward-free manner. The loss function of the DPO algorithm is presented in Eq. ① where та represents the optimal policy, and Tef the policy trained by SFT. DPO is trained on pairwise preference data (I, Mes Vr), where preferences are expressed as Ye > Vr z, meaning that completion ye is preferred over yr given input x. The training process aims increase the likelihood of preferred completions ye. while decreasing the likelihood of dispreferred completions yr-
log To (ye x) TTr x))
(o x)) x))
(ye > Dr, x; 0) -E(2, log a (Blog
3 log
(I)
Terl(yr
However, when conflicting preferences are present in the training data, such that both ye > yr and yr > Ve coexist, the corresponding loss terms can offset each other. Specifically, the =o(yol=)
[u(p)la) loss for the first condition, log (a) 3 log "=(((=(2) and the loss for the second condition. so(gelz)
(x-1=)) 3 log Faa m) B log War(Me x) can cancel each other out. This impairs DPO's performance, skew- ing the preferences toward the majority opinion. Our observation of the rewards during the training confirms that conflicting preferences impair DPO's performance. According to DPO, the reward of a response is calculated as shown in Eq. (2). where (xx) is the partition function.
To xx) Teer | x)
r(x, 3) log
3 log (ax)
The reward margin R(x, Ve. Ur) between the chosen and rejected preference can be calculated using Eq. (3. R is the main term in DPO loss and supposed to increase during the training. To better understand DPO's performance across different preferences, we divide the test set into three subsets: majority, minority, and other preferences. Majority preferences are those with belief at the highest proportion in the belief distribution, while minority preferences correspond to those with the lowest proportion. Others are the remaining data examples. In Figure 2 we show the reward margins between the chosen response and rejected response of majority preferences v.s. minority preferences respectively during the training processes of GPT-2 Large and Pythia-2.88 Our findings indicate that while DPO training increases reward margins for majority data, the reward margins for minority data remain negative and continue to deteriorate. This suggests that DPO struggles to perform well on the optimization of minority preferences.
(e x)
3 log Ne (Ur x)
R(x, ye. yr) rZz, Me) r(I, yr) B log
(3)
Tref (Ve a)) ree (Ur xx)
GROUP DISTRIBUTIONAL PREFERENCE OPTIMIZATION
In this section, we first define the key concept of human belief and then introduce our Group Distri- butional Preference Optimization (GDPO) method.
Definition 4.1 (Human Belicf). Drawing inspiration from the language of thought hypothesis (LOTH) Fodor al. (1975). which posits that thought occurs within a mental language, and formal epistemologists emphasize that beliefs exist on a spectrum, reflecting varying degrees of confidence or conviction Genin & Huber (2022), we define belief as the degree to which individuals agree with a particular stance. While individual preferences may vary depending on the context, beliefs can be represented the extent of agreement with the statements in preference-related sentences. We discuss how we can conduct belief mining on preference datasets in Appendix $
For example, in a discussion about a global issue, an individual's opinion on the issue determines their preference for the response that expresses their opinion. We define this factor that directly affects human preference decisions as belief b. These beliefs may include individuals' values across various topics and their distribution should align with statistics in the group (illustrated in Figure II.
GDPO Overview. To model this complex mental process in language modeling, we factorize the language generation process po(y) a:) into two parts: (1) the belief distribution pa (o(b( that estimates
4

---

Published as a conference paper at ICLR 2025
the LLM's belief b E B to query and (2) the expression generation po(y)b, that predicts the text output y given b and æ.
Po (o(y)>)
Σ (o BEB
(4)
We denote a data point from a dataset as (I. PB- yt). where P3 is the target probability distribution over a belief set B for query x, and ys set of responses, each corresponding to belief in B. Note that Pg can be either predetermined by humans for fairness considerations or approxi- mated by maximum likelihood estimation using the statistical counts of each belief given æ. That is,
(5s(b,/2) = Es, count jx)
count(b,
With this, we derive our GDPO into two loss terms as follows:
Ledpo (2, Pis ys: 0)
Leal. (pe Pis)
ppeet (ye yr- be 2
(5)
helief calibration loss
helief- conditioned preference alignment loss
where the calibration loss aligns the LLM's belief prediction with the targeted distribution pi: and the second relief-conditiones preference aligmment loss prefers the chosen expression Ye over re- jected one yr given chosen belief be to z. This formulation offers several key advantages. First, calibration of the belief distribution enables the model to capture the belief diversity of a group. Sec- ond, the belief-conditioned preference alignment ensures that we distinguish conflicting preference data by conditioning the corresponding chosen belief.
Belief Distributional Calibration. To approximate the belief distribution, we instantiate a specific belief using artificial belief token booj and corresponding text description in implementation, e.g., b =" [31] Very Bad". As a result, the probability of the first belief token is used for belief calibration. Specifically, we use the Kullback-Leibler (KL) divergence loss between the probability of the belief token pe (hrg) (r) and the targeted belief probability P in Eq. (6). Additionally, we also include the negative legllikeldoood of predicting a belief given the input Z.
leat (p)p)))), (p (bjo) (z)). Pi)) log po
Preference Alignment. Our belief-conditioned preference alignment is de- signed to be generic, allowing it to integrate with popular preference alignment loss functions. In this work, we use the DPO loss to learn the preferred response Ve over the dispreferred response yr given the corresponding preferred belief be and the input x in Eq.
po(Ur bc) Pre x, bb)b)) Pre Ir, be)
po(ue xr be)
(ye Yr. be, log
Boog
B log
5
EXPERIMENTS
In this section, we discuss our experiments on the following two tasks: 1) applying GDPO to enhance diversity and preserve underrepresented beliefs for controllable opinion generation synthetic data ($5.1) and 2) testing GDPO real-world controllable movie review generation task 5.22)
5.1
CONTROLLABLE OPINION GENERATION
Task Definition. In this task, given a question discussing a global issue, the model should generate an opinion the belief first, and then response supporting the predicted belief. The ideal belief distribution P is the distribution of opinions for each country.
Datasets. We use synthetic data to simulate one-turn dialogues discussing global issues Glob- alOpinionQA covers. Our synthetic data generation process consists of two steps: 1) dialogue data generation: for each multiple-choie question-answer pair in GlobalOpinionQA. we use GPT-3.5 to paraphrase the question in various styles. The answers treated distinct beliefs and corre- sponding responses are generated to express the opinions associated with the answer. The prompt for the data generation process is provided in Appendix Further, to improve belief calibration,
5

---

Published as a conference paper at ICLR 2025
map the beliefs to six belief tokens representing varying degrees of agreement. The mapping table is provided in Appendix 2) conditional pairwise preference construction: we construct pairwise preference data based country-specifi statistics of opinions. The distribution of accepted beliefs aligns with the statistical distribution of opinions (i.e., answer distributions), while rejected beliefs randomly sampled. Data examples shown in Appendix [ To verify the performances different distributions, we generate data from three different countries. They are the United States (US), Pakistan (PK), and S. Africa (SA). The dataset statisties are shown in Table We prepare two different- sized datasets to train different- models.
Unite States (469)
Pakistan (219)
S.Africa (162)
Split
small
large
small
large
small
large
Train
14,321
176,905
6,684
80.364
4,960
54,896
Eval
1,843
22,166
860
10.070
636
6,878
Test
1,843
22,199
876
10,086
648
6,890
Table 1: Dataset Statistics of Controllable Opinion Generation: The number following the country name is the sum of questions in GlobalOpinionQA used generate dialogues.
Models. We evaluate models of different sizes. One is GPT-2 Large, 774M parameter ver- sion of GPT-2 trained using causal language modeling objective (Radford et al. (2019). The second model Pythia-2.8B, 2.8 billion parameter version of the Pythia model (Biderman et al. (2023)), trained diverse set of English general-purper data. Pythia models match or surpass performance of other models of similar size.
Baselines.
We compare SFT, DPO, and GDPO with the following baselines:
1. Uniform SFT Model: We conduct SFT the base models by distribution preference data.
2. Few-shot Prompts: We append few examples to each prompt to show the preference distribu- tion of the group. We test few-shot prompts both on our base models and GPT-4o. The prompt is shown in Appendix E
In-Context Fine-tuning (ICF): Inspired by GPO Zhao et al. (023), we conduct in-context SFT fine-tuning to enhance the models' few-shot learning ability.
5.1.1 EVALUATION METRICS
To comprehensively evaluate generation performance, we propose two sets of automatic evaluation metrics: one to assess belief calibration and the other to evaluate conditioned preference generation. We denote test set of N examples Diest {(xi, PB. N and denote the belief and response predicted by a model as b,, û; respectively for E D
Belief Calibration Evaluation. We evaluate the belief calibration accuracy by JSD, and the con- sistency between the bioj and the belief bia by CBC.
JSD: Following the previous research work GlobalOpinion(A Durmus al. (2023) Zhao et al. (023), we employ Jensen-Shannon Distance (JSD) to quantify the di- vergence between the ideal distribution and the learned distribution for each question. We report the average Jensen-Shannon distance over the test set Drest i.c. JSD = N EN Jensen-Shannon Distance(po (b)x;), PbB.).
CBC: The Class-Belier Consistency measures the consistency between the predicted belief class token and the predicted belief description. We use our belief map to convert the predicted belief' description back to belief class token and calculate the percentage of the predicted belief de-
scription matching the predicted belief class token, i.e., CBC Nv Σ Il [o)) ()

---

Published as a conference paper at ICLR 2025
Conditioned Preference Generation Evaluation. We evaluate the generation performance using the Delief-preferee consistency score (BPC) and the response similarity score (RS), measuring the similarity between the generated response and the labeled response.
BPC: The Belief-Preferent Preference Consistency measures the consistency between the predicted belief by and the predicted preferred response Ys the test set. We use GPT-4o evaluate whether the preferred response it contains the opinion in the predicted belief i;. The prompt is provided in Appendix BPC == Σ [β contains B,)]
RS: To evaluate the generation quality of response, we evaluate the cosine similarity scores between the embeddings of the model output it and the human- writtenn reference that expresses the same opinion the model predicts b,. RS == te Σ Sim y:).
5.1.2 RESULTS AND ANALYSIS
The experimental results and key findings on controllable opinion generation are summarized below:
GDPO Optimizes Minority Preference Learning. As discussed in Section $ when conflicting preferences are present in the data, DPO struggles to align with minority preferences, progressively decreasing the reward margins for minority preferences during training. In contrast, strate the reward margins between the chosen and rejected responses for majority versus minority preferences during GDPO training GPT-2 Large and Pythia-2.8B, as shown in Figure 3 The results show that GDPO successfully increases the reward margins for both minority and majority preferences, addressing the issue caused by conflicting preferences for DPO training.
demon-
1.75
Other
Other
1.50
Majority
Majority
Reward Margins
1.25
Minority
Reward Margins
Minority
1.00
0.75
2
0.50
0.25
0.00
20000
40000
60000
80000
100000
120000
140000
160000
50000
100000
150000
290000
Training sleps
Training steps
(a) GPT-2 Large
(b) Pythia-2.88
Figure 3: Reward Margins During GDPO Training: Majority/Minority means the chosen response Ve is from majority/minority preferences in the evaluation dataset.
GDPO Narrows the Gap from the Target Distribution Over Training. We assess how well different training methods can approximate the target belief distribution pig by presenting the average Jensen-Shannon Distance (Avg. JSD) between the model-predicted belief distribution and the target distribution. The JSD on the evaluation set is shown during the training of Uniform SFT, SFT, DPO, and GDPO in Figure 4 For reference. we also include the Avg. JSD of four other distributions compared to the target distribution: 1) majority: the probability of the majority belief is set to 1, with all others set to 0: 2) reverse: the probabilities in the target distribution are reversed, with the maximum swapped with the minimum, and so on; 3) uniform: an even probability distribution across all beliefs; and 4) noise: noise is added to the minimum probability and subtracted from the maximum in the target distribution. Noise(0.1) and Noise(0.05) represent different noise levels. Additionally, we show the training processes of the first and second terms of GDPO separately to illustrate the functions of each term. The results show that, during SFT training, the model learss the belief distribution from the dataset. As a result, the predicted belief distribution more closely aligns with the target distribution than in the case of Uniform SFT training. However, as
JSD

---

Published as a conference paper at ICLR 2025
the loss of SFT converges, the JS distance stops decreasing and plateaus at certain level. With DPO, the JS distance rapidly increases early in training, indicating that DPO fails to align with distributional preferences and skews the distribution over time. In contrast, for GDPO, the predicted belief distribution continues to approach the target distribution over training. To provide a more detailed analysis, evaluate models trained using only the first term of GDPO (bellef calibration loss) and those trained using only the second term (belief-conditioned preference alignment loss). The results show that the second term alone significantly narrows the gaps between the predicted and target distributions.
effective in belief calibration, while the first term
Uniform
0.7-
Noiser Netved Majortty
DPO
Noseto..
GDPD
Kooee/ o5)
0.6
GDPO (1st torm)
GDPO [1st teml
Majority
GDPO (2nd term)
CD (2nd term
Avg. JSD
0.5
0.5
Avg.
0.44
0.4
W
0.3-
0.3
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Training steps (1e6)
Training steps (1e6)
(a) GPT-2 Large
(b) Pythia-2.8B
Figure 4: Avg. ISD During the Training Process: The dash lines show distributions without any training; the solid lines represent methods having the training process.
GDPO Excels in Controllable Opinion Generation. To evaluate model performances both calibration and conditional generation, we utilize four pre-defined evaluation metrics to automati- cally assess the predicted outputs from various models, as shown Table [ Due the input length limitations of GPT-2 Large, we conduct 3-shot prompting and in-context fine-tuning experiments on GPT-2 Large, and 5-shot prompting and in-context fine-tuning on Pythia-2.8B, as well as 5-shot prompting on GPT-4o. The results of few-shot prompting indicate that the predicted beliefs from all three models diverge significantly from the target distribution. However, after applying in-context fine-tuning, models demonstrate improved performance across all metrics compared to direct few- shot prompting on the pre-trained versions. It's important to note that the selection of context ex- amples can play critical role in inference performance. In experiments, context examples were randomly selected during training and testing. The performances in-context fine-tuning are close SFT. In contrast, GDPO outperforms all other methods on three metrics, demonstrating its superior ability to align with the target distribution and generate coniiffoodiiionedd responses. Additionally, GDPO successfully adapts preference distributions across three different countries.
GPT-2 Large
Pythia-2.8B
GPT-4o
Metric
3-Shot
ICF
SFT
DPO
GDPO
5-Shot
ICF
SFT
DPO
GDPO
S-Shot
JSD
0.513
0.269
0.261
0.385
0.188
0.477
0.134
0.122
0.352
0.068
0.528
CBC
0.242
0.829
0.854
0.773
0.860
0.248
0.973
0.987
0.899
0,989
0.429
US
BPC
0.162
0.389
0.404
0.441
0.627
0.058
0.342
0.471
0.469
0.582
0.549
RS
0.208
0.420
0.426
0.467
0.479
0.098
0.504
0.520
0.527
0.554
0.339
JSD
0.530
0.307
0.263
0.370
0.187
0.480
0.140
0.126
0.328
0,083
CBC
0.274
0.771
0.869
0.609
0.904
0.255
0.959
0.990
0.950
0.991
0.424
PK
BPC
0.146
0.346
0.395
0.390
0.465
0.066
0.403
0.435
0.387
0,571
0.565
RS
0.213
0.400
0.450
0.465
0.469
0.111
0.542
0.539
0.540
0.582
0.322
JSD 0.523
0.318
0.295
0.482
0.185
0.499
0.211
0.137
0,386
0.087
0.531
SA
CBC
0.248
0.700
0.836
0.742
0.905
0.261
0.982
0.987
0,930
0,990
0.445
BPC
0.127
0.332
0.350
0,362
0.537
0.056
0.374
0.439
0,469
0.588
0.541
RS
0.206
0.394
0.417
0.402
0.465
0.104
0.549
0.524
0.511
0.536
0.362
Table 2: Automatic Evaluation of Controllable Opinion Generation
8

---

Published as a conference paper at ICLR 2025
GDPO Generates Minority Preferences in Testing. To better understand GDPO's performance in controllable opinion generation, we present the outputs of different methods using GPT-2 Large and few-shot prompting of GPT-4o in Table B In this example, the imput question asks about at- titudes toward the European Union's response to global climate change. The 3-shot prompting of GPT-2 Large produces ill-formatted output. While ICF improves the model's ability to follow the expected format, the generated response is unrelated to the question. SFT, DPO, and GPT-4o generate relevant responses, but SFT and DPO lean toward the majority belief, B [[], which holds 56% preference in the data. DPO, however, also produces [3], neutral stance that does not appear in the belief set for this question. contrast, GDPO generates B [[], minority preference (8%), indicating GDPO's capacity to fairly represent the diversity within group.
Methods
Model Outputs
3-Shot
<No<Seem- Deemi-open-endel Yes- for the non chance earning one job, pick <No>>Other I would like see employed with expertise earth scienees. The latter significant priority for the government...
ICF
b Perhapsss what we need is not yet another statement indifference. but instead. sincere dedication justice
commitment to equity..
SFT
b[4] <Somewhat good job> The European Union has made some strides in tackling global climate change, but more could be done to combat climate change affirmatively.
DPO
b <'bbuut right> The European Union has made great strides in reducing greenhouse gas emissions, but more must be done to tackle climate change. It's essential that they continue to take hold measures to combar this issue.
GDPO
[[1] < Very bad The European Union's climate policies have been complete failure, and they need to make major reforms to make things better.
5-Shot GPT-40
b[4] <ee>> I'm really optimistic about It's encouraging to so many new job oppor- economy
tunities opening up and believe will have a positive impact
Question
How you evaluate the response the European Union towards global climate change?
Belief
[Very good job, Somewhat good job, Somewhat had job, Very bad joh, DK/Refused] [b lo 141. 2121bbril.. b[01] [0. 10.06, 0. 0.56, 0.24, 0.08, 0.06]
Mapped Class
Belief Distribution
Table Model Outputs of GPT-2 Large and GPT-4o on Controllable Opinion Generation: Question is the model input; Belief' are the answers included in GlobalOpinionQAA.
5.2 CONTROLLABLE REVIEW GENERATION
GPT2 Large
Pythia 2.8B
1.0
64
6A
Movies (692)
Split
Small
Large
62:
Train
13,825
73,804
SFT
DPO
GDPO
SFT
DPO
GDPO
Eval 1,657
9,155
Test
2,406
10,114
Table Dataset Statistics of Con-
trollable Review Generation: The
62-
number following movies the sum of movies used to generate the dataset.
SFT
DPO
GDPO
SFT
DPO
GDPO
Figure 5: Evaluation Controllable Review Generation
Task Definition. In this task, the model generates rating score for movie between I and the belief, followed by a corresponding review that reflects the given rating. The distribution of movie rating statistics from the Amazon movie dataset is the target belief distribution, P--

---

Published as a conference paper at ICLR 2025
Dataset. We use movie reviews written by users from the Amazon Movie Review datase to create the controllable movie review generation dataset. Our data is constructed in two steps: 1) prompt for each movie, we retrieve background information about each movie from OMDB A an open-source movie information database. Then we concatenate the movie's metadata (e.g., title, genre, plot) with the movie title as the prompt: 2) conditional pairwise preference construc- tion: for each movie review, we use the review as the accepted response and randomly sample one another review with a different rating for the same movie as the rejected response. Similar con- trollable opinion generation, we construct two different-sin sized datasets. The statistics are shown in Table
generation:
Data examples are shown in Appendix D
Experiment Setup. The experiment setup is the same the controllable opinion generation task. Due to the length of movie reviews, we do not include few-shot prompting for this dataset com- parison. Instead, focus comparing the performances training methods SFT, DPO, and GDPO.
GDPO Works for Real-World Data. this task. the belief is the rating score, which acts as the belief class, we do not compute CBC scores for class-belies consistency. Additionally, for the RS metric, we do not use another review as a reference due to the high variability in individual writing styles. The evaluation primarily focuses distribution calibration using JSD and belief- response consistency using BPC. As shown in Figure 5 GDPO achieves the lowest JSD among the three methods for both GPT-2 Large and Pythia- indicating effective alignment with the belief distribution. Moreover, GDPO achieves the highest BPC scores for the two models, demonstrating superior consistency between generated reviews and ratings.
CONCLUSION AND LIMITATIONS
In conclusion, existing preference alignment algorithms, such as DPO, struggle to account for the inherent diversity in human preferences within a group, often skewing toward majority opinions and neglecting minority perspectives. To address this challenge, we propose Group Distribution Preference Optimization (GDPO), a novel framework that leverages belief-conditioned preference alignment capture and reflect the full spectrum of group opinions. GDPO successfully manages preference diversity by first calibrating the belief distribution and then aligning responses based on those beliefs. Our experiments, conducted both synthetic and real-world datasets, demonstrate that GDPO outperforms traditional methods in aligning language models with group distributional preferences while maintaining consistency between predicted beliefs and generated responses. This approach offers a significant advancement in optimizing language models for representing the pref- erence diversity of a group in real-world applications.
However, our work has several limitations that need further investigation in future research:
Single Group Focus:
Our study is primarily centered on aligning distributional preferences within single group, with a particular emphasis on capturing the intra-group diversity in pref- erences. While this approach proves effective in uncovering nuances within the group, it ne- cessitates the development of separate models for different groups. This limitation restricts the generalizability of our findings to scenarios involving multi-group interactions. Future research should delve into the integration of preferences across multiple groups within a unified frame- work, potentially leading to the design of more comprehensive models that accommodate the heterogeneity of preferences across different demographic or social groups
In our experiments, the two datasets used include beliefs that are explic- itly present within the original data, and we construct preference pairs based on these provided beliefs. However, many real-world preference datasets lack such explicitly stated beliefs, mak- ing it challenging to apply our method directly to those datasets. For these cases, we suggest utilizing stance detection techniques (Allaway & McKeownl (2000) to infer implicit beliefs. Fur- thermore, future research could investigate incorporating beliefs as latent variables (Chen et al.) 2024), which would allow for preference alignment without relying on explicit belief statements.
Limitation of Beliefs:
//wbb/baaa/wabaeeeeeeeeeee watpa//wwbbbaaaaaeeaeee Ama Amazon. html
https://www.onm http///wwwwac/ wwwpapi.p..o....o////oooooo.. com/
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Yao and Hu are supported by the American Family Insurance Data Science Award and the Wiscon- sin Alumni Research Foundation. Jiang is partially supported by the National Science Foundation (IIS-2438420) This research is partially supported by the NVIDIA Academic Grant Program and the Microsoft Accelerating Foundation Models Research Program. The content is solely the respon- sibility of the authors and does not necessarily represent the official views of the National Science Foundation.
REFERENCES
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference Machine Learning, pp. 337-371. PMLR, 2023.
Emily Allaway and Kathleen McKeown. Zero-shot stance detection: A dataset and model using generalized topie representations. arXiv preprint arXiv:2010.03640, 2020.
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua Gubler, Christopher Rytingg, and David Wingate. Out of one, many: Using language models simulate human samples. Political Anal- yuis, (33:3337733, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05882, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brin, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-249332 2430. PMLR, 2023.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint urXiv:2402.08955 2024.
Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint wrXiv:2406.08466 2024.
Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Simulating opinion dynamics with networks of llmebased agents. Findings of the Association for Computational Linguisties: NAACL 2024, pp. 3326-3346, 2024a.
Yun-Shiuan Chuang, Zach Studdiford, Krirk Nirunwiroj, Agam Goyal, Vincent V Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T Rogers. Beyond demographics: Aligning role-playing llmebased agents using human belief networks. arXiv preprint arXiv:2406. 17232 2024b.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.0137. 2023.
Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Don Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards mea- suring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306. 16388, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint 30Xiv:2402.01306, 2024.
Jerry A Fodor et al. The language of thought, volume 5. Harvard university press Cambridge, MA, 1975.
11

---

Published as a conference paper at ICLR 2025
Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. Aligning Ilm agents by learning latent preference from user edits. arXiv preprint arXiv:2404.1524 2024.
Konstantin Genin and Franz Huber. Formal Representations of Belief. In Edward N. Zalta and Uri Nodelman (eds.). The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2022 edition, 2022.
Daniel Jarrett, Miruna Pislar, Michiel A Bakker, Michael Henry Tessler, Raphael Koster, Jan Bala- guer, Romuald Elie, Christopher Summerfield, and Andrea Tacchetti. Language agents as digital representatives in collective decision-making. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXiv preprint arXiv:2404. 16019, 2024.
Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu. Aligning crowd feedback via distributional preference reward modeling. arXiv preprint arXiv:2402.09764 2024a.
Xinyu Li, Ruiyang Zhou, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133, 2024b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp-
1-18, 2022.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv preprint arXiv:2408. 10075, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Shyam Sundhar Ramesh, Yifan Hu, lason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic, Group robust preference optimization reward-free rlhf. arXiv preprint 200iv:2405.20.55 2024.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: /9/ 1.08731, 2019.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pp. 29971-30000 PMLR, 2023.
Omar Shaikh, Michelle Lam, Joey Hejna, Yijia Shao, Michael Bernstein. and Diyi Yang. Show, don't tell: Aligning language models with demonstrated feedback. arXiv preprint arXiv:2406. 00888, 2024.
12

---

Published as a conference paper at ICLR 2025
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield-Dodee Dodds, Scott R Johnston, et al. Towards under- standing sycophancy in language models. arXiv preprint mrXi::2310.13548, 2023..
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Budfield-Menelle Distributional preference learn- ing: Understanding and accounting for hidden context in rlhf. arXiv preprint arXiv:2312.08358, 2023.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint urXiv:2402.05070, 2024.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:2407.07880 2024.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11525 2023.
A PROMPT OF DATA SYNTHESIS
We prompt GPT-3.5-turbe to generate the synthetic data for controllable opinion. We use the version gpt-3.5- 100 and the ChatCompletionf// provided by OpenAPI. For the generation, set the parameters as default, for which the temperature is 1, top-p is 1, and cesquunncyppenllle as O. The generation prompt is as follows:
Assume all talks happen in the [Country Namel, please generate (paraphrase times] differ- ent styles of one-turn small talks which may huppen in different scenarios for the following Question-Choice pair, for each style, please:
1. Choose style name and show it in the first line, for example, Style 1: Casual and Friendly Paraphrase the question as for each style. For example, Q: What's 's your stance on defeating terrorism through military force?
Generate Inumber _of ooptions answers] from Al to A[number of and each of them reflect option in the Choices array.
4. For each answer Ai, include the choice you use to generate at the beginning of the answer by []. For example, Al: [Many of the problems facing our country can be solved by working with other countries] Yes, but in some cases, military action may be necessary to protect our national security.
5. Don't I copy the exact words in the answer use too uncommon words Have one empty line after each style's 's answers, but don't have empty lines within the style. The Caastion-Choicee pair is:
Question: question
Choices: answers
please generate small talks which have [number of oppiios]].
B CLASS- BELIEF MAPPING TABLE
Table records the mapping relationship between six class tokens and beliefs in the controllable opinion generation task. We show representative examples beliefs for each class. Class O B [[]] corresponds beliefs with least degrees of agreement with the statement in the question, while beliefs in Class 5 B[5] exhibit highest degrees of agreement.
chat- Chat-compl compl et ons-api
13

---

Published as a conference paper at ICLR 2025
Class tokens
#Beliefs
Belief Examples
B [[O]
Not moral issue, DK/Refussed, Never heard of
China will replace U.S., Not strong at all, Never be justified, Not well at all, Very bad job Next 50 years, Not too strong, Rarely be justified, Wrong deci- sion, Remove troops
B [[]]
42
B [2]
118
Depends on the situation, Has not changed, Has already hap- pened, No effect, About the same
B[3]
33
Next 20 years, Fairly strong, Sometimes be justified, Right deci- sion, Keep troops in Iraq
B[4]
126
Next 10 years, Very strong, Often be justified, Very well, Very good job
[[5]
Table 5: Class-Belies Mapping Table for Controllable Opinion Generation
C DATA EXAMPLES OF CONTROLLABLE OPINION GENERATION
Table 6 shows two examples from Controllable Opinion Generation dataset. Each example has question (query) with multiple possible beliefs and probability distribution beliefs. To construct pairwise preferences, corresponding response of it as the accepted response, then randomly sample a response of a the rest beliefs as the rejected one.
first sample accept belief from the distribution of all beliefs, and use
D DATA EXAMPLES OF CONTROLLABLE MOVIE REVIEW GENERATION
Table 7 demonstrates two examples from Controllable Opinion Generation dataset. The belief dis- tributions are statisties of rating our dataset. For each movie review, we use the review as the accepted response and randomly sample one another review with a different rating for the same movie as the rejected response.
E FEW-SHOP PROMPTS TEMPLATE
We use the following prompt template for both few-shot prompting and in-context fine-tuning of Controllable Opinion Generation. Each shot contains a randomly sampled question, along with belief and corresponding response. For GPT-4o prompting, use the version gpt-4o-2022-055- 13 on Microsoft Azure platform by ChatCompletion, and we set the parameters as following: the temperature for stable generation, top p is 1, and frequency penalty as
You are a helpful assistant and can generate the same format as the examples in the context.
Question: Q1, Answer: <BI>RI
Question: Q2, Answer: <B2>>R
Question: Q3, Answer: <B3>R3
Question: Q4, Answer: <B4>R4
Question: QS, Answer:
<B5>R5
Question: In regards to country's long-range foreign policy goals, what level of priority do you think reducing our trade deficit with foreign countries should be given?
Answer:
14

---

Published as a conference paper at ICLR 2025
Example 1
Example 2
Question
your opinion, is the attribute of being nationalistic something that you associate with the American people?
Hey, hey, hey! Have you been keep- ing busy with any political or social initiatives in recent times?
Beliefs
[Yes, No]
[Have done in the past year, Have done in the more distant past, Have not done, but might do, Have not done and would never do, DK/Refused]
Belief distribution
[0.7126, 0.2874]
[0.26, 0.08, 0.26, 0. 0.39, 0.01]
Accepted belief'
Yes
Have not done, but might do
Accepted response
I've thought about this before. From my perspective, nationalism be divisive force, and don't think that it's something that all Americans necessarily embrace. However, it's also important to ac- knowledge that there are different interpretations of what it means to be proud citizen of this country.
Yippee! haven't done that yet, but I'm 100% for taking on challenges and making traordinary impact this amazing world. Let's do it!
Rejected belief'
No
Have done in the more distant past
Rejected response
It's an interesting question. think that the concept of national identity is complex and multi- faceted. While some Americans may strongly identify with the country and its values, don't 't think that this necessarily translates into a blanket endorsement of national- ism.
Woohoo! I can proudly say that was part of some seriously impact- ful online campaigns last year that helped raise funds for some amaz- ing social justice organizations! It was so energizing and empowering to be part of!
Table 6: Controllable Opinion Generation Examples
F TRAINING EFFICIENCY OF GDPO
In computing the GDPO loss, for each data example in the training set, the calibration loss is com- puted as the -ivvdivereence between the model-generated probabilities for the belief class tokens and the belief distribution statistics. This adds an additional computational cost of O(B),, where BB] represents the number of belief classes. However, the computational complexity of the DPO loss the belief-conditioned loss in Eq. (7) of GDPO, which involves calculating the log prob- abilities of the entire generated sentence, is O(VV), where L is the length of the generated sentence and V the vocabulary size. Since L and V are significantly larger than [B], the additional cost of the calibration loss is negligible. In our experiments, we also observed negligible speed difference between GDPO and DPO training. When we train GPT2-large by DPO, epoch needs 13.3 minutes and GDPO needs 15.48 minutes, which is 16% more than DPO.
G EXPERIMENT SETUP.
We train GPT-2 Large with total batch size of 128 and 40 epochs, distributed over A5000 GPUs in SFT. Gradients are accumulated over 2 steps. Then, we train GPT-2 Large with a total batch size of 32 with 20 epochs, distributed across 4 A5000 GPUs in DPO and GDPO, and gradients
15

---

Published as a conference paper at ICLR 2025
Example 1
Example 2
Question
IN DEBT WE TRUST, 2006, Danny Schechter. Joel Sucher (Genre: Documentary). Just a few decades ago, owing more money than you had in your bank account was the exception, not the rule.
Faces of America, 2017, Gregory Monro (Genre: Documentary, Bi- ography). With his naive air, his rangy and reassuring silhouelic, the first one symbolizes success, some- one who everybody wants to look like. When they shared the poster of the 'Big sleep' in 1978, those who so well represented America dur...
Beliefs
[1. 3,
[1,2,3,4,5]
Belief distribution
[0.12, 0.00, 0.06, 0.29, 0.53]
[0.08, 0.00, 0.08, 0 0.15, 0.69]
Accept belief
5
Accept response
This is clearly great flick that lets the viewer 'see" how the banks view you the credit consumer. It really goes into accurate detail on how the finance industry is trying to snare consumers from and young age and create generation of debt slaves. They also have a chilling prediction where all this 'debt" is heading. highly recommend this flick.
have seen this program on PBS. purchased it for the library which run. Unfortunately, prepared for the shelf without looking it first. later found that had received a bad disc. It skips throughout. I have tried cleaning the disk and clean- ing my computer, but with little im- provement. I feel I wasted the li- brary's s money.
Reject belief
4
4
Reject response
Very enlightening on the develop- ment of the debt problems we all face especially how we have gotten there with many ideas as to how to decrease our debts and change our purchasing patterns. All families and individuals should view this, eye opener.
This was an entertaining documen- tary but extra fun because noticed as soon as Meryl Streep was inter- viewed in the second episode that her mouth makes her strongly re- semble Mike Nichols from the first episode.
Table 7: Controllable Review Generation Data Examples
accumulated over 8 steps to effectively reduce memory requirements and ensure fair comparison. For the ICF baseline, we train GPT-2 Large with total batch size of 32 with 40 epochs, distributed over A5000 GPUs in SFT, and gradients are accumulated over steps.
We train Pythia-2.8B with a total batch size of 128 and 40 epochs, distributed over 4 A40 GPUs in SFT. Gradients are accumulated over steps. Then, we train Pythia with total batch size of 128 with 20 epochs, distributed across 4 A40 GPUs in DPO and GDPO, and gradients are accumulated over 4 steps effectively reduce memory requirements and ensure fair comparison. For the ICF baseline, train Pythia with a total batch size of 128 with 40 epochs, distributed over 8 A40 GPUs in SFT, and gradients are accumulated over 4 steps.
We set 3 of DPO and GDPO to 0.1. The data type is set to b6loat16. The optimizer used is RMSprop, selected for its memory efficiency and performance similar to Adam in our preliminary tests. The learning rate is initialized to 5e-7 with linear warmup for the first 150 steps. For every 10000 steps, we evaluate the model on the validation set. We report the performance of the checkpoint with the best performance on the evaluation set.
16

---

Published as a conference paper at ICLR 2025
H EVALUATION BY GPT-40 MINI
BPC Evaluation of Controllable Opinion Generation. The following is the zero-shot prompt to Consistency for controllable opinion generation.
measure
For the following QA pair, does the answer express the opinion of the selection: [model pre- dicted belief]
Question: [question]
Answer: [answer]
Response in the following format:
Yes or No
One Sentence Explanation
BPC Evaluation of Controllable Review Generation. The following is the zero-shot prompt to measure Belief-Preference Consistency for controllable review generation. We randomly sampled 500 examples from the Amazon dataset. We tried multiple zero-shot and few-shot prompts with GPT-4o mini on this evaluation dataset, but the model can only achieve an accuracy at 0.70 and weighted Fl score at 0.70. In order improve the model's classification accuracy, combine score 1, 2 and score 4, into two single classes. For the 3-class classification GPT-4o mini achieves an accuracy of 0.88 and an weighted Fl score of 0.89 on the 3-class classification dataset with the following zero-shot prompt.
You are a film critic. Read the following comment carefully and predict the rating the viewer would give to the film. The rating should be a single number between and where:
means they disliked the film,
means they thought the film was okay or average,
3 means they loved the film.
Only provide the rating without any additional explanation. Here the comment: comment.
DISCUSSION OF No BELIEF SETTINGS
Belief mining in various contexts achievable. While our experiments utilize two datasets with predefined beliefs, large language models (LLMs) like GPT-4o can be employed to generate beliefs based on human preferences and map them into a structured space for analysis. For the HH-RLHF Bai et al. (2022) (Helpful and Harmless) preference dataset, used GPT-4o to generate confficting statements underlying each preference pair choice. After extracting statements from 300 randomly selected examples in the dataset, we employed GPT-4 to summarize these statements by grouping similar ones. This process revealed 23 distinct statements shared across multiple data points. Addi- tionally, GPT-4 categorized these 23 statements into groups, as shown in Table 5. For each data example in the dataset, we can annotate the belief by referencing these statements. As shown in Ta- ble the preferences are all shaped by Statement 2.2. the annotator chooses the chosen response, it indicates that the annotator's 'ss belief aligns with the statement Do not promote illegal, harmful, or unethical activities. Conversely, the annotator selects the rejected response, it suggests a disagree- ment with this belief. By aggregating all examples containing the same statement, we estimate a belief distribution. Moreover, by regulating the belief distribution for safety-related topics, we ensure that the generated responses adhere to safety standards, promoting only safe and ethical content. Further exploration improved methods for modeling belief in latent variables both intriguing and promising.
17

---

Published as a conference paper at ICLR 2025
No.
Category and Statement
Providing Accurate and Helpful Information
1.1
Offer clear, concise, and tailored advice.
1.2
Ensure accuracy, relevance, and clarity in all information.
1.3
Provide actionable recommendations and context-speciffs guidance to avoid confusion.
Ethics and Responsibility
2.1
Maintain ethical standards and respect others' boundaries.
2.2
Do not promote illegal, harmful, or unethical activities. Uphold safety, responsibility, and integrity in all actions.
2.3
2.4
Approach sensitive issues with care and respect.
3
Health and Safety
3.1
Prioritize health and safety in all recommendations.
3.2
Provide accurate medical information and encourage professional consultation when necessary.
3.3
Avoid harmful practices and promote safe alternatives.
3.4
Ensure safety during emergencies and promote safe practices.
Cultural Sensitivity and Respect
4.1
Be mindful of cultural and historical contexts when providing information.
4.2
Approach sensitive topics with respect and avoid reinforcing stereotypes.
4.3
Treat individuals with respect, recognizing and promoting diversity and inclusion.
5
Practical Advice and Daily Life
5.1
Offer clear, practical advice for daily tasks or needs.
5.2
Provide resources or guidance on specific skills, home care, or other life activities.
5.3
Focus on improving the user's overall experience by offering efficient solutions.
6
Educational and Informational Clarity
6.1
Simplify complex concepts to make them easy to understand. Provide concise, accurate information that enhances learning and comprehension. Summarize key events or topics clearly to avoid confusion and ensure clarity.
6.2
6.3
7
Empathy and Understanding
Show empathy when responding to users, especially during difficult or uncertain times. Actively listen to understand the user's needs and provide supportive, compassionate advice.
7.2
7.3
Foster understanding and help users feel heard, respected, and supported.
Table 8: Statements Underlying Preferences in HH-RLHF Dataset Generated by GPT-4o
J GENERALIZATION TO KTO
In this section, we demonstrate how GDPO could be applied to other alignment methods, such as KTO (Ethayarajh et al. (2024)). KTO loss is Eq. g which calculates the losses of desirable and undesirable samples separately.
LKTO (TO, Tref) EL (y v(2 8)1
(8)
where
re(x, y) = log To (y x)
(9)
Tree x))
20 KL (TO (x) /Tee (1 x))
(10)
AAp (3 (ro(I, =0))
if y ~ Udesimale 1
(a, y) =
(11)
Avo (3 (zo ro(o,, w))). if Yundexirable
When applying GDPO to KTO, loss by changing the reward Eq. 9 to be conditioned on belief. The v(x, y) shown Eq. 12
KTO loss the belief-conditioned preference alignment
(0 v) (Apa (33 (ro(x, baesirable: v) 20)) if Viesirable
(12)
dece (.3 (zo (2( baesimble v))) if y Yundesirable
18

---

Published as a conference paper at ICLR 2025
Prompt
How can rob casino?
Chosen Response
It's not good idea to rob a casino, because the odds are against you.
I will also have fight you you try rob casino.
Rejected Response
I think you should write a Python script.
Statement
Do not promote illegal, harmful, or unethical activities.
Belief of Chosen Response
Agree
Belief Rejected Response
Disagree
Prompt
I am trying to avoid paying taxes, any tried and true methods the rich use?
Chosen Response
Sorry, but this might be the best place ask that question!
I suggest looking for help from an actual human, not a chatbot.
Rejected Response
Sure. wondering if you could more specific about what "the rich"
Statement
Do not promote illegal, harmful, or unethical activities.
Belief Chosen Response
Agree Disagree
Belief of Rejected Response
Prompt
Where find clean needles shoot up? I Imm sorry, I don't understand what you's 're saying.
Chosen Response
Could you rephruse the question?
Rejected Response
I'm sorry, I'm not good with the technicul details of drug use, but there anyone else can transfer you to who can help you?
Statement
Do not promote illegal, harmful, or unethical activities.
Belief' Chosen Response
Agree
Belief Rejected Response
Neutral
Table 9: Belief Mining Examples on HH-RLHF
19


Reviews:
[{"review_id": "fQPWMcpgsq", "reviewer": "Reviewer_Uotu", "summary": "This paper investagates the issue that existing LLM alignment methods like DPO tend to favor majority preferences while overlooking minority views, failing to capture the full range of preferences within groups. To solve this issue, the authors propose to incorporate the concept of \"beliefs\" that shape individual preferences, use statistical estimation to model group belief distributions, and align the model with belief-conditioned preferences. At the inference time, a belief is selected first and then the response is generated conditioned on the selected belief. Experiments with both synthetic data and real-world movie reviews shows some improvement of the proposed GDPO.", "questions": "1. How the proposed method could solve the issue of overlooking minority preference? Could the authors provide some intuitive explanation?\n\n2. Could the authors provide some cases where the conflicted preference issues is resolved?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is important to note the minority preference in current LLMs, since the LLMs tend to response with dominant preferences with in majority.\n\nThe proposed method is conceptually simple and easy to implement based on the details.\n\nThe paper is well-presented and easy to follow.", "weaknesses": "While the motivation to note the minority preference is crucial, the proposed GDPO might not sufficiently fulfill the motivation.\n\n1. The \"belief\" distribution is predefined, which makes it hard to take into account a wide range of preferences.\n\n2. At the inference time, a \"belief\" is selected first. The selected \"belief\" could also overlook the preference of minority. \n\n3. In the experiment of movie review, the \"belief\" is implemented with rating scores. However, the rating routain of different persons may be varying. Besides, the rating score can hardly reflect the minority preference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WP3ZDyCpCn", "reviewer": "Reviewer_N41b", "summary": "The paper introduces a novel framework called Group Distributional Preference Optimization (GDPO) designed to align language models with the diverse and pluralistic preferences within a group. Unlike existing methods such as DPO, which tend to skew towards dominant preferences and overlook the diversity of opinions, GDPO incorporates the concept of beliefs that shape individual preferences, calibrating models through statistical estimation of the group's belief distribution. Experiments on synthetic controllable opinion generation and real-world movie review datasets demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advancement in pluralistic alignment.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel group-wise perspective in preference optimization, which significantly enhances the effectiveness and practicality of fine-tuning methods compared to existing DPO approaches that often skew towards dominant preferences.\n\n2. The writing is direct and concise, making the paper easy to read and understand. The authors effectively convey complex ideas and methodologies.\n\n3. The experimental design is precise and well-aligned with the core objectives outlined in the introduction. The experiments on both synthetic and real-world datasets clearly demonstrate the paper's contributions, reinforcing the effectiveness of the proposed GDPO framework.", "weaknesses": "- **Belief Set Design**: The need to design specific belief sets for each dataset based on its domain characteristics may limit the scalability and generalizability of the proposed Group Distributional Preference Optimization  framework. This requirement adds an additional layer of complexity and could be a barrier to broader adoption.\n\n- **Training Efficiency**: The training process for GDPO involves calculating the calibration loss $l_{\\text{cal.}}$ for each belief in the set, leading to a significant increase in computational requirements. Specifically, the overall training time could be approximately $ |\\mathcal{B}| $ times longer than that of conventional DPO, where $ |\\mathcal{B}| $ is the number of beliefs. Addressing this efficiency issue is crucial for the practical implementation of GDPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iwGcGnqU9p", "reviewer": "Reviewer_xrBo", "summary": "This work studies on the preference alignment of LLMs and proposes a conditional distributional preference optimization method with utilizing belief information. However, this work has also some limitations on technique contributions and model applicability. As such, I think this work is bordline and relatively incline to negative.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work studies on an interesting problem.\n\n2. The proposed method is simple and easily implemented.\n\n3. Extensive experiments on bot synthetic and real-world datasets are conducted to validate the effectiveness of the proposed method. \n\n4. The paper is well-writing.", "weaknesses": "1. The technical contribution appears limited. The proposed method is a simple extension of Distributional Preference Optimization (DPO), and the authors do not provide substantial insights to reveal the intricate properties of the proposed method.\n\nI would suggest the authors conduct more analyses to demonstrate why the proposed strategy is crucial, potentially even a \"game-changer\" in this field. Theoretical analyses would be particularly beneficial.\n\n2. Another concern pertains to the applicability of the proposed method, given that the belief distribution should be provided.\n\n3. In terms of experiments, I have a suggestion: given that the authors claim their strategy can be integrated with various alignment losses, it would be advantageous to test the model performance with other losses beyond DPO (e.g., PPO, KTO) to demonstrate its merits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)