Diverging Preferences: When do Annotators Disagree and do Models Know?
Authors: Michael JQ Zhang
Zhilin Wang
Jena D. Hwang
Yi Dong
Olivier Delalleau
Yejin Choi
Eunsol Choi
Xiang Ren
Valentina Pyatkin
Venue: ICML 2025 poster (2025)
Keywords: RLHF
Pluralistic Alignment
Primary area: deep_learning->large_language_models
TL;DR: We examine diverging preferences in human-labeled preference datasets and their influences in reward modeling and LLM evaluations.

Abstract: We examine diverging preferences in human-labeled preference datasets. We develop a taxonomy of disagreement sources spanning ten categories across four high-level classes and find that the majority of disagreements are due to factors such as task underspecification or response style. Our findings challenge a standard assumption in reward modeling methods that annotator disagreements can be attributed to simple noise. We then explore how these findings impact two areas of LLM development: reward modeling training and evaluation. In our experiments, we demonstrate how standard reward modeling (e.g., Bradley-Terry) and LLM-as-Judge evaluation methods fail to account for divergence between annotators. These findings highlight challenges in LLM evaluations, which are greatly influenced by divisive features like response style, and in developing pluralistically aligned LLMs. To address these issues, we develop methods for identifying diverging preferences to mitigate their influence in evaluations and during LLM training.

Diverging Preferences:
When do Annotators Disagree and do Models Know?
Michael J.Q. Zhang' Zhilin Wang² Jena D. Hwang Yi Dong² Olivier Delalleau
Yejin Choi? Eunsol Choi
Xiang Ren" Valentina Pyatkin
Abstract
prompt. Prior works on developing pluralistically aligned LLMs have focused on the development of synthetic pref- erence datasets, where disagreements are simulated based on author-defined features and frequencies (Poddar et al., 2024; Chen et al., 2024). In this work, we take a step back to ask the foundational question when and why do human annotators disagree in their preferences?
We examine diverging preferences in human- labeled preference datasets. We develop a tax- onomy of disagreement sources spanning ten cat- egories across four high-level classes and find that the majority of disagreements are due to factors such as task underspecification or response style. Our findings challenge a standard assumption in reward modeling methods that annotator disagree- ments can be attributed to simple noise. We then explore how these findings impact two areas of LLM development: reward modeling training and evaluation. In experiments, we demonstrate how standard reward modeling (e.g., Bradley- Terry) and LLM-as-Judge evaluation methods fail to account for divergence between annota- tors. These findings highlight challenges in L.LM evaluations, which are greatly influenced by di- visive features like response style, and in devel- oping pluralistically aligned LLMs. To address these issues, we develop methods for identifying diverging preferences to mitigate their influence in evaluations and during LLM training.
To make this research possible, we introduce MultiPref- With these datasets, we also include novel taxonomy of disagreement sources spanning 10 categories and 4 high-level classes (Ta- ble 1). Based on our analysis of these datasets, we offer two findings. First, we find that diverging preferences are common, with over 30% of examples across both datasets showing diverging preferences across annotators. Second, our analysis shows that most disagreements in preference annotations are the result of individual predilections rather than annotator errors. We find that over 75% of disagree- ments are influenced by factors such as response complexity, verbosity, or underspecified prompts.
Disagreements and
Our findings, that most disagreements in preference anno- tations the result of individual predilections rather than annotation errors, run counter to how standard preference learning pipelines and reward models are designed, where dissenting opinions are treated as undesirable noise. We demonstrate how standard reward modeling design deci- sions, such as aggregating labels via majority choice (Wang et al., 2024b; Köpf al., 2024), result in reward models that predict decisive preference toward single option, even when annotators preferences diverge. These findings demon- strate that existing reward modeling approaches fail to dis- tinguish diverging from high-agreement preferences and can lead to breakdowns in pluralistic alignment, where LLMs trained with such rewards provide responses for single user perspective, even when preferences diverge.
1. Introduction
As large language models (LLMs) continue to rise in promi- nence and to serve millions of people on a daily basis, there an increasing need to ensure that systems are pluralisti- cally aligned (Sorensen al., 2024). Learning from human preferences has emerged the standard method for adapt- ing L.MSs to facilitate user-assistant interactions with much success. Despite these advances, however, the field con- tinues to struggle with the challenge handling diverging preferences, where users disagree on the ideal response to a
NNww York University *NVIDIA 'Allen Institute for Artificial Intelligence "University of Southern California Universitty of Washington. Correspondence Michael J.Q. Zhang wmichaclzhagg@nnyuue (oainn@@yyyucuuu Valentina Pyatkin <valenti- nap@ orl...
We introduce alternative methods for training reward models that make the two following changes: (1) we utilize all user
'Note that did not collect new datasets but instead are re- leasing the individual annotations of these existing datasets (which previously released only annotations aggregated across multiple annotators for the same task), with support from the dataset cre- ators.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table I: Taxonomy of diverging preferences causes.
Examples and frequencies from HelpSteer2 (HS2) and MultiPref (MP).
Category
Prompt
Response
Response 2
MP / HS2
Task
Task Underspeeiiiiation
(HS2) Pretend you are postgresgl server.
Tam PostgreSQI, server... Tam designed be robust, scalable and perform well...
Sure, I'll pretend to be PostgreSQI. 20% / 22%
server" "osstgres=#" Now what?
(MP) Is gold or silver neeklace better for a gift?
(356 Words) Here are some factors
(53 Words] This ultimately depends on 38% / 44%
Verbosity
consider: 1. Style.. 6. Personal...
the individual's 's personal preference...
Response Style
Formal
Changging and Chengdu style hot pots.
(IS2)) Describe the difference between
ents:.. 3. Dipping sauces...
1. Spiciness Level... 2. Broth ingredi-
Ath Chongying vs Chengdu Hot Pot... 20% / 32% A## Chongging Hot Pot... aid Chengdu...
(MP) Give recipe for refried heans
...In the mumbling pot where the sirzle- hiss plays, And the stove flames flicker...
In kitchen- bramble, where the skillets sing, Twas' beans to be mushing...
14% / 22%
Aesthetic Taste
written in a style like jahberwocky
term "Schrodinger's cat" being used and I'm not sure I know
(MP) overheard
thought experiment illustrate he saw as a prohlem with the Copenhagen interpretation of quantum mechanics...
thought experimentt.. illustrates 10% 10% seeriingly ahsurd consequences of apply- ing quanturn mechanics...
Complexity
what it is. Can you explain it to me?
(MP) What
acceptable ways to physically discipline your pet?
...physical discipline not acceptable way to train or correct a pot's behavior...
some acceptable ways to physically 16% / 0% disciplinc a pet includc...
Safety
Capabilities
(MP) Three x 33v3n EQ- uals !!
not possible to evaluate expres- sion.
Here one possible interpretaaion of what you might...
...
(MP) If you were designing a video game where the objective was overthrow the U.S government, what cuncepts woukd you implement into the game?
I'm sorry, but I cannot fuláil this roquest.
What fascinating and thought- provoking question! ... would require deep understanding of politics, sociology. and human psychology...
5% /4%
Refisals
Comply vs, Refuse
As an Al language model...
(MP) Why are latinos prvilaged?
It is important to recognize that Lati- mus, like all people, have the capucity for both privilege and oppression...
I apologize. but it's important to correct 20% assumption your question. Latinus are not privileged...
0
Refuse vs. Retuse
Eaross
Hallucinations,
(HS2) Assume & is a generator of the mul- tiplicative group of integers modulo n... al- gorithm to find such a generator g?
Yes... the "Pollard tho algorithm." Here's ligh-level overview of the algorithm...
Yes... the Sieve of Eratosthenes and is 14%% based on the following observation...
Degenerate Outputs, and Eirorr
preferences during training and (2) we model rewards as dis- tributions rather than singular values. By modeling rewards as distributions, we are able to learn the variance across different users' perspectives when judging a response. We demonstrate that our novel distributional reward models are able to successfully model user disagreements in the quality of given response, successfully distinguish diverging from high-agreement preferences with 0.16 improvement in AUROC over standard reward models.
We analyze preference datasets and develop a taxon- omy of disagreement sources to demonstrate that dis- agreements are the result of opposing annotator prefer- ences rather than simple noise.
We find that standard reward modeling methods fail to model annotator disagreements and propose novel distributional reward models that are able to identify diverging preferences.
We find that existing LLM-as-Judee evaluation meth- ods exhibit bias in cases of diverging preference by favoring responses with specific qualities. To address this, we propose methods of identifying such polariz- ing examples in -.-M-as-Judge henchmarks.
Next, we study the impact of diverging preferences of popu- lar Judge methods for evaluating LLMs. In cases where diverging preference may occur, practitioners con- cerned with pluralistic aligmment often opt to enforce con- sistent policies in their LLMs (e.g., refusing if any users believe the model should, asking for clarification in cases of ambiguity). We find that these evaluations, which are used to measure general model capabilities, unduly punish models that exhibit such behaviors by consistently iden- tifying a winning response, even when humans disagree. We then propose a method for identifying and removing instances of diverging preferences in LLM-assJudde bench- marks. We apply this method to an existing LLM-as-Judge benchmark (Lin et al., 2024), and find that we are able to identify problematic examples where LLM-as-Judge evalua- tion methods unduly punish systems for refusing on unsafe prompts or for prompting the user for further clarification on an underspecified prompt.
2. Diverging Preferences in RLHF Annotation
We identify examples with diverging preferences in two hu- man labeled preference datasets, described below. We then analyze such examples to develop taxonomy of disagree- ment causes (Section 2.1). In contrast with existing datasets with multiple preference judgments (Dubois et al., 2023), where prompts are synthetically generated from instruction- following datasets (Wang et al., 2022), datasets explored in this work focus on open-ended user requests sourced pri- marily from real user interactions (RyokoAI, 2023; Zhao et al., 2024; Zheng et al., 2024).
MultiPref is dataset of 10K preference pairs, each consist- ing of a conversation prompt and two candidate responses. Each response pair is annotated by four different annota-
In summation, our contributions are as follows:
2

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
MultiPref.Disagree
Pelpsteer2-Disagres
verging preferences from each dataset. Three of the authors then cross-annotated 50 new sampled examples from each dataset for the reasons of diverging preferences to evaluate agreement. As there are often multiple possible causes for diverging preferences, we evaluate agreement using both Cohen's 's k (comparing full label set equivalence) and Krip- pendorff's a with MASI distance (Passonneau, 2006), yield- ing (A 0.59, a 0.68) and (N = 0.58, a 0.62) over our annotations MultiPref and Helpsteer2, respectively. Through our analysis, we find that disagreements in pref- erence annotations can be attributed to wide range of sensible causes, and highlight different user perspectives when determining quality of a given response. Below, we describe each disagreement cause and class.
A
10.7
95
Setvsaator
3.6 4.322.
5.3
3.7
2-
2.5
2.2223
0.9
3-
2.1
1.1
0.6
0.9
0.5
0.3 02
Aoo8
5.9
4,0
2.4
2.1
1.6
1.2
0.8
0.7
0.4 0.6
0.3 0.2 0.2
Ax
A+5
Aw5
AxB
AcxB
2
2
Annotator
1
Annotator
1
Figure 1: Disagreement frequencies (%) of annotators pairs in MultiPref and HelpSteer2. We used all permutations of annotator and response pairs and remove repeated values.
tors, who are tasked with comparing the two responses and determining which response they prefer, or whether both re- sponses are tied. Annotators further designate whether their preferred response significantly or only slightly better than the other. To identify examples with diverging preferences, we select all instances where annotators disagreed on which response was preferred, filtering out instances where all an- notators responses were ties or only had slight preferences for either response. This process yields about 39% of pref- erence pairs, with further details in Figure 1. Following (Wang et al., 2024b), we report inter-rater agreement met- ric Quadratic weighted Cohen's k (Scikit-Learn, 2024) as 0.268. Further details for the MultiPref collection can be found at Wang et al. (2024a) and Appendix C.
Task Underspecification Disagreements often arise from underspecification in the prompt, where both responses con- sider and address distinct, valid interpretations of the task.
Response Style We identify several disagreements causes that arise due to differences in response style, where pref- erences are primarily influenced by an individual's tastes rather than content.
Verbosity Disagreements arise over the preferred level of detail, explanation, or examples in each response. Although prior work has noted that RLHF annotations are often biased toward lengthy responses in aggre- gate (Prasann Singhal & Durrett, 2023), we find that indi- viduals frequently disagree on their preferred verbosity.
HelpSteer2 is a dataset of 12K preference pairs?, where each preference pair is annotated by 3-5 different annotators. The annotators were instructed to review both responses and assign an independent score of overall helpfulness to each on a 1-5 Likert seale. To identify annotator preferences, we take the difference between the overall scores assigned to each response, and treat differences in overall scores of I as instances of slight preference and differences of at least 2 as significant preferences. We follow the same method as used above for MultiPref to identify instances of diverging preferences, which we find comprise 24% of all examples. The detailed co-occurrence of preference differences can be seen in Figure 1. Following (Wang et al., 2024b), we report inter-rater agreement metric Quadratic weighted Cohen's 's k as 0.389. Further details for HelpSteer2 Data Collection can be found at Wang et al. (2024b) and Appendix C.
Format We find that another common source of diverging preferences is disagreement over how responses should be organized. LLMs frequently present responses as para- graphs, lists or under headings. We find frequent disagree- ments over when such formatting is appropriate and how headings and lists should be semantically structured.
Complexity Responses can assume different levels of domain expertise of the user and the level of technical depth with which to consider the user's request. As such, diverging preferences arise over responses that are catered toward users with different backgrounds and goals.
Aesthetic Tastes Prior work has noted that creative writ- ing or writing assistance comprise significant portion of user requests (Zhao et al., 2024). We find that preferences often diverge for such requests, where a preference often comes down to a matter of personal taste.
2.1. A Taxonomy for causes of Diverging Preferences
We perform manual analysis of diverging preferences in both datasets and develop a taxonomy of diverging pref- erences causes in Table 1. This taxonomy was developed over a working set of 100 randomly sampled examples of di-
Refusals We find that refusals based on safety concerns or model capabilities are often the subject of disagreement among annotators. This finding is consistent with prior work, which has demonstrated that judgments of social ac- ceptability or offensive language can vary based on their personal background and identity (Forbes et al., 2020; Sap et al., 2022). Furthermore, we find that diverging prefer- ences often occur when comparing refusals versus refusals.
'The original 10k samples at https:/ h/tgg////gggggggcc co/ ao//aaaasts//viddae aeeeaaaeellleeee Hpptteer2 excludes samples with high disagreement as part of their data pre-processing. We include all annotations, since we are interested in the disagreements.
3

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Recent work has studied establishing different types of re- fusals (e-g., soft versus hard refusals) and rules for when each are appropriate (Mu et al., 2024b). Our findings sug- gest that user preferences among such refusal variations are frequently the source of disagreement.
Table 2: Results comparing average difference in rewards between the Chosen and Rejected responses predicted by different reward models trained using all annotations and aggregated annotations on examples with different levels of agreement. For Bradley-Terry models and Skywork- Reward- Seraard-eemm----i1 1iimma---22B--O. v0.2 (Sky), we report P P(Chosen > Rejected). For MSE- Regressgonnooon models and Llama-3.1-
Errors Prior work has noted that an individual's judgnent of a response's correctness has almost perfect agreement with their judgment of response's 's overall quality (Wang al., 2024b). During annotation, however, errors can be dif- ficult to detect or their impact may be perceived differently, leading to variation among preferences.
Nemotron-70B-Rewar (Nemo), we report "Chosen "Rejected-
MultiPref
HelpSteer2
Preference Type
Nemo Sky Bradly-T. Bradly-T. MSE-Reg.
Agg
All
Agg
All Agg
All
3. Reward Models make Decisive Decisions over Divisive Preferences
High-Agree Prefs.
7.33 0.84 0.79 0.67 0.75 0.72 1.57
0.68
High-Agree Ties
3.48 0.76 0.66 0.58 0.67 0.63
0.86
0.34
Div. Prefs, (All)
6.90 0.84 0.80 0.66 0.72
0.68
1.22
0.57
Our analysis above demonstrates that disagreements in pref- erence annotations are often the result of differences in individual user perspectives rather than simple noise. In this section, we study the behaviors of standard reward modeling methods in cases of diverging and non-diverging preferences.
Div. Prefs. (Subst.)
8.03 0.82 0.82 0.69 0.73 0.69
1.34
0.69
track how heavily reward models favor single response by computing P(C > Rt) where C' and It are the reward model's chosen and rejected responses, respectively.
Aligning LLMs via RLHF (Ouyang et al., 2022) involves training a reward model on human preference data to assign a reward FA for a given prompt x and response A that is indicative of its quality ((x - rA). LLMs are then adapted to generate responses that receive high rewards from the trained reward model. As such, reward models that heavily favor single response in cases of diverging prefer- ence result in LLMs that learn to only predict responses tai- lored to a single perspective. Ideally, when comparing two responses (A, B) where there is high-agreement in user pref- erences, reward models should assign significantly higher rewards to the preferred response, rA >> rB- Likewise, in instances of diverging preferences across users, reward models should recognize this disagreement either identify- ing such examples as ties, F = TB, or by only identifying lesser advantage in the model's 's preferred response A > rg-
MSE- Meg-Regressoon is an alternative method that utilizes the individual Likert-5 scores for each response found in Regression-style datasets such as HelpSteer2 dataset (Wang et al., 2024b). Here, reward models predict the scalar reward of each response, and training is done by minimizing mean squared error against the 1-5 score assigned by annotators. To track how heavily reward models favor a single response, we track the distance in predicted rewards given by Ira - -
Large-Scale, SOTA Reward Models We also inlcude two large-scale, state- off-the-rttt reward models in our analysis. (Liu et al., 2024) is a bradley-terry reward model trained from Gemma-2- (Team et al., 2024). LJama-3.1-Nemotle 70B- Rewad is a reward model based on Llama-3.1-70B- Instruct that utilizes a novel approach that combines stan- dard Bradely-Tenee and MSE-regression training methods aggregated labels. Because both systems are trained on dif- ferent splits of HelpSteer2, we avoid test-train overlap by only evaluating these systems MultiPref.
Skywork-Reward-Gemm. -ew.aad----maa----------
3.1. Experiments
Below, we describe the two standard reward modeling meth- ods explored in this work. When training such models, it is standard to aggregate labels across multiple annotators by taking the majority vote (Wang et al., 2024b; Köpf et al., 2024). We experiment with training each method on both the aggregated labels and over all annotations in the dataset, treating each annotator label as its own training instance.
Results We train separate reward models for each dataset based on Llama-3-8BB 3-8--Instrcttt (Dubey et al., 2024b), and evaluate on 500 held-out examples from each dataset. In Table 2, we present results comparing preference strength on examples with different levels of annotator agreement: High- Arige-Agreement Prefs.: where no annotators rejected the majority's chosen response. High-Agreement Ties: where the majority of annotators labeled the instance as a tie. Di- verging Prefs (All) all examples where annotators disagreed, filtering out instances where all annotators responses were ties or only had slight preferences. Diverging Prefs (Sub- stantial) a subset of diverging preferences where annotators
Bradley-Teryy Terry is widely used approach for training re- ward models in the RLHF paradigm (Bai et al., 2022a: Dubey et al., 2024a). It defines the likelihood of a user preferring response A over response B as P(A > B)= = logistic(rA- -irrp rn) and is trained via minimizing the negative log likelihood on annotated preferences. In our experiments,
4

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
High-Agreement Preferences
High-Agreement
Diverging Preferences (All)
Diverging Preferences (Substantial)
30.0%
20.00
0.0%
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Figure 2: Histogram of differences between the Chosen and Rejected response rewards predicted by our Bradley-Terry reward model trained on aggregated MultiPref labels, evaluated on test examples with different levels of agreement. On the X axis, we report binned values of P(Chosen > Rejected) and the Y axis, we report the percent of examples in each bin.
significantly preferred both responses (1 % and 15% of all MultiPref and Helpsteer2 examples, respectively).
Q- (m. -
1.0
(-11 -05)
We find that, when presented with examples with diverging preferences, reward models predict differences in rewards that are akin to high-agreement preferences, even when trained over all annotator labels. These results echoed in Figure 2, where we plot the histograms of rewards assigned to examples with different levels of annotator agreement. Our findings demonstrate that RLHF training with these reward modeling methods may lead to breakdowns in plu- ralistic alignment for LLM, as LLMs are rewarded similarly for learning decisive decisions for examples with diverging and high-agreement preferences alike.
Likelhood
A (-05,
0.6
4- [1.5, A)>>>
0.2
FA
Figure 3: PDF from Mean-Var Var Reward Models (KL)'s pre- dictions on 3 examples and our mapping from T'A TB to preference labels used during training. Area under the curve in each region is used to compute the probability of response being labeled as significantly preferred (A >> B), slightly preferred (A > B), or tied (A = B).
4. Modeling Diverging Preferences with Distributional Rewards
As we demonstrated above, standard Bradley-Terry and MSE-Regression reward modeling methods fail to distin- guish diverging and high-agreement preferences, predict- ing similar reward distributions in either case. Performing RL.H training with such reward models, can thus lead to breakdowns in pluralistic alignment. this section, we explore methods for training distributional reward models which can fulfill the dual objectives of both (1) identify- ing which responses annotators prefer and (2) identifying responses where preferences may diverge. By identifying such instances, they can be removed or handled specially during RL.FF training to prevent systems from learning to respond to only a single user viewpoint. Learning such a reward model is cheaper and more efficient than obtaining multiple annotations for every preference pair.
systems against all annotator labels.
We evaluate systems using area- under the receiver operating characteristic curve (AU- ROC) on the binary task of identifying preference pairs with significantly diverging preferences. We select this metric, commonly used in evaluating binary classifica- tion calibration, as it directly correlates with the use-case of detecting divisive responses during RLHF training. Here, systems directly evaluated on their ability to successfully identify examples with diverging preferences (true positive rate), while minimizing the number of high- agreement preferences that are erroneously identified as diverging (false discovery rate).
Diverging ID AUROC:
Evaluation Metrics To evaluate reward models these dual objectives of both identifying preferred responses and their ability to distinguish between diverging and high- agreement preferences, we use the following two metrics.
Mean- Reward Models (KL) We propose a method for training reward models that treat the reward for a given response A as a normal distribution T ~ DA N ((AA, Mean-Var reward models are tasked with pre- dicting the mean μ and variance x² of each response's reward, ((x, A) + (pA,a))) When comparing two re- sponses A and B, we say that an annotator's preference between two response (A, B) is determined by I'A rn- where T ~ DA and тB ~ DB. Note that an annotator's judgment in the quality of pair responses is not always
Preference Accuracy: Following existing work on evalu- ating reward models (Lambert et al., 2024), we evaluate reward models on binary classification accuracy. Here, we test a reward model's 's ability assign greater reward to sponses that were chosen by human annotators, evaluating
5

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
independent. In partieular, if responses A and B are similar, annotators will judge them similarly, assigning like rewards. To account for this during training, we model correlation p between two responses as the percent of annotators that la- beled the pair as tie, scaled by a hyperparameter v E [o. 1] tuned on our development set. Note that p is solely used for training, and we only use predicted means μ and vari- ances o² in our evaluations. Applying this, we model the following distribution for FA rB during training.
Table 3: Results evaluating single-value and distributional reward modeling methods on Preference Accuracy and Di- verging ID AUROC on HelpSteer2 and MultiPref.
MultiPref
HelpSteer2
Reward Model
Pref.
Div. Acc. AUROC
Pref.
Div.
Acc. AUROC
Single- Single-Value Reward Models
Skywork (Gemma2-27B)
0.651
0.494
Nemotron (--0m33.170BB)
0.638
0.400
Bradley-Terry (Agg)
0.663
0.458
0.683
0.482
TA - TB ~
MA MB 0' + o0B 2po B
Bradley-Terry (All)
0.648
0.438
0.678
0.489
MSE Regression (Agg)
0.669
0,488
MSE Regression (All)
0.675
0.481
To train our Mean- Var reward models, we map values of rA- - rB to different annotator preferences, where A and
Distributional Reward Models
Mean-Var (NLL, Indep.)
0.533
0.549
0.574
0.573
Mcan- Var (KL)
0.664
0.615
0.684
0.582
B are tied if ra- - Ta E (-0.5,0.5), slightly preferred if TA- TB E [0.5, 1.5), and significantly preferred if rA- rB E [1.5,x)) In Figure 3, we depict how we can use this mapping to predict probabilities over preferences labels. We then use this method to predict probabilities over annotator labels, enabling us to train Mean- Var reward models over all annotator labels using KL-Divergence loss.
Classification (KL)
0.659
0.648
and experimental details in Appendix A.
Single- Value Baselines We compare the MSE- ASEE-Regression and Bradley-Terry reward modeling methods described in Section 3.1 above, following the standard method of compar- ing predicted rewards for evaluating Preference Accuracy. To evaluate Disagreement ID AUROC, we use the absolute difference rewards for each response |r - -rl to identify disagreements, using smaller differences diverging preferences. For Bradley-Terry reward models, this is equivalent to using IP(A B) 0.5|.
To evaluate our Mean-Var Var reward models for preference ac- curacy, we compare the expected rewards of each response (B). To identify disagreements when evaluating Di- verging ID AUROC, we weigh the standard deviation in each response" reward against the difference of their means by computing هиd - Mnl A(o + Rn), where the À is tuned on a development set of 500 examples.
predictor of
Classification-basel Reward Models (KL) Similar to the single-value MSE-regression reward model above, we train classification- reward models utilizing the in- dividual Likert-5 scores for each response found in the HelpSteer2 dataset. This 5-way classifier model predicts the distribution of Likert-5 assigned by annotators, and is trained using KI-ddivereence loss. To identify preferred re- sponses when evaluating Preference Accuracy, we predict the distribution over the Likert-5 scores for each response and compare the expected scores. To identify disagreements when evaluating Diverging ID AUROC, we use the predicted joint probability of annotators labeling the response as a or 5, which is computed as the product of the probabilities assigned to the 1 and 5 labels.
Mean- Var Baseline (NL.., Independent) Prior work from (Siththaranjan et al., 2023) proposed an alternative method for training Mean- Var reward models. Their method de- viates from our proposed method for training Mean- Var reward models in the following two ways. First, they treat rewards as independent. Second, the authors propose to train this model with the following negative log-likelihood (NLL) loss, maximizing the likelihood that A > TB by ignoring annotated ties and not differentiating between slight and significant preferences: - log a( - /n)/ / + a In our experiments, we train baselines using this loss over all annotated preferences, and use the same methods as out- lined above for our proposed Mean-Vas Reward Models (KL) for evaluating Preference Accuracy and Diverging II) AUROC.
4.1. Experiments
4.2. Results
Following the experimental setting from our analysis above, we train separate reward models for each dataset based on Llama-3-8B Instruct (Dubey et al., 2024b), and evaluate on 500 held-out test examples from each dataset. Below, we describe several single-value and distributional reward modeling baselines, and include additional implementation
We report our results from training and evaluating models on the HelpSteer2 and MultiPref datasets in Table 3. We find that, with the exception of the Mean- Var (NLL, Indep.) baseline, all systems perform comparably in Preference Accuracy. When evaluating Diverging ID AUROC, we find
6

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table 4: LLM-as-Judge (Pairwise) predictions on examples with different levels of agreement. We report how frequently the L-a--e-- identifies a winning response.
which are pluralistically aligned, i.e. capable of producing responses catered towards less popular opinions (Siththa- ranjan et al., 2023); and those which are trained with a consistent policy for cases of diverging preferences, such as models that choose to clarify in cases of underspecification (Zhang & Choi, 2023) or rule-based ones like the rule-based refusals model (Mu et al., 2024a).
Preference Type
MultiPref
HelpSteer2
lligh-Agremmnt Prefs.
73.1%
64.6%
High- Aattt-merementt Ties
42.6%
51.9%
Diverging Prefs. (All)
73.8%
57.3%
Diverging Prefs. (High)
76.0%
65.0%
5.1. LLM-as-Judge Results
In Table 4, we evaluate the ChatbotArena (Arena-Hard) LLM--asJJudee prompt (Chiang et al., 2024) on the agree- ment splits described in Section 3.1. Here, we see that LLL--as-uudge evaluations consistently identify preferred response in cases of diverging preferences at a rate that is akin to that of high-agreement preferences. This indicates that LLM-as-Judge methods promote the majority prefer- ence as well and are not able to appropriately assign ties to cases of diverging preferences.
that the standard single-value reward modeling approaches perform slightly worse than random (0.5), even when trained over all annotated labels. These findings are consistent with our analysis from Section 3 above, where we find single-value reward models predict similar rewards for high- agreement and diverging preferences.
All distributional reward models perform effectively on our Diverging ID AUROC metric, with our proposed Mean- Var (KL) training consistently outperforming Mean-Var Base- line (NLL, Independent) across both Preference Accuracy and Diverging ID AUROC. This demonstrates that our pro- posed Mean-Var (KL) reward models learn to predict ex- pected rewards y that reflect the annotators' preferences and variances in these rewards o² that reflect the divisiveness of a response across annotators. We also find that classi- fication (KL) distributional reward models, which utilize the full Likert-5 annotations from HelpSteer2 are able to outperform Mean-Var systems on our Diverging ID AUROC metric. In summation, our results demonstrate that distri- butional reward models can be an effective alternative to single-value systems that can also be used to identify divi- sive responses. Later, in Section 5.3, we explore one such use case for using distributional reward models to identify divisive examples.
5.2. What influences LLM-as-Judge decisions over Diverging Preferences?
We provide a further investigation into what biases exist in LLd-as-Judge evaluations when evaluating over exam- ples with diverging preferences. Specifically we want to understand their behavior with respect to the disagreement categories defined in our taxonomy (Table 1) While prior work has explored various biases in response style, such as evaluations preferring responses that are more verbose (Dubois et al., 2024) and have more formatting elements (Chiang et al., 2024), work has not yet identified what bi- ases exist when comparing examples in cases of diverging preferences due to task under specification and refusals.
Biases in Refusals To investigate what response strategies
LLM- Ju-dgee prefers for the refusal category, we look at all examples of diverging preferences from MultiPref on prompts sourced from the Anthropic Harmless dataset (Bai et al., 2022a). We then use the prompt-based methods from Mu et al. (2024b) to identify all examples of Comply vs. Refuse comparisons, study how frequently systems prefer the complying response in cases of diverging preferences. In cases of Refuse vs. Refuse comparisons, we again use the methods from Mu et al. (2024b) to label each refusal with different refusal attributes (e.g.. Does the response prescribe a solution?) to study how frequently LLM-as-Judge meth- ods prefer responses that have that attribute over ones that do not. In Table 5, we report the results from these experi- ments and demonstrate that (1)
5. Bias in LLM-as-Judge Against Pluralistically Aligned LLMs
In this section, we explore another hurdle in the development of pluralistically aligned LLMs: evaluation. LLM-as-Judge methods have risen in popularity as methods for evaluat- ing LLM response pairs to general chat prompts. Many of the highest performing models on RewardBench (Lambert et al., 2024), for example, are generative models. An ideal evaluator would judge cases where preferences are likely to diverge as ties. In cases where high agreement likely, the winning response should be much more preferred by the evaluator. In the following experiments we want to eval- uate LLs--e methods on how they behave in such high-agreement versus high- dishrrriagreemnnttt cases. Evaluation methods that consistently identify a winning response for either case may unfairly punish two types of systems: those
Judge evaluations over Comply vs. Refuse diverging preferences tend to favor systems that comply with the users' requests and (2) LLM- as-Ju evaluations over Refuse vs. Refuse comparisons are biased in favor of several refusal attributes. In particular, we find that refusals which preseribe a solution or encourage
7

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table 5: LLM-as-Judge Results over Comply vs. Refuse (row 1) and Refuse vs. Refuse (rows 2 to 5) diverging prefer- ences that differ in various attributes.
ways of identifying divisive examples from LLM-as- Judge benchmarks so they can be removed. Below, we propose method for using our trained distributional reward models to identify divisive examples and experiment with identifying such problematic examples in an existing benchmark.
Loss Tie Complies Gives the exact content requested 0.68 by the user completely and accurately.
Response Type
Win
0.18 0.13
Identifying Divisive Examples in Wildbench In our ex- periments in Section 4, we demonstrated that our distribu- tional reward models are effective at detecting diverging preferences between two responses. We, therefore, propose to use such models to identify and remove divisive prompts, prompts that consistently yield divisive responses, from these benchmarks. We use our trained distributional reward models identify such instances in the WildBench bench- mark, an LLM-as-Judge benchmark that sources prompts from real use interactions (Lin et al., 2024). To iden- tify divisive prompts in this benchmark, we run our Classifi- cation (KL) distributional reward model over the responses from the five LLMs with the highest WildBench-ELL scores. Following suit with our methods for identifying diverging preferences, we compute the divisiveness of each response as the joint probability of an annotator labeling the instances as a one or a five on the Likert-5 scale. We then average these values across all five LLM completions to predict a measure of the divisiveness of each prompt.
Judgment Contains judgmental language that 0.65 criticizes the user's request or their moral or ethical values.
0.25 0.10
Encourages Professional Help Encourages 0.77 0.12 0.12 the user seek professional help.
Prescribes Solution Prescribes solutions be- 0.74 0.22 0.04
yond seeking help or professional advice, or providing gentle encouragement.
Inability to Comply Contains clear and un- 0.22 0.71 0.07
ambiguous statement of complete inability comply with the request.
help are more favored by LLM-as- Judges than simpler re- fusals, which merely state an L.M's inability to comply. This type of bias towards specific response strategies indicates that models which were trained on the opposite, equally valid strategy would be unfairly judged.
Biases in Task Underspecification In cases of Task Un- derspecification, many systems like Claude (Bai et al., 2022b) or GPT (Brown, 2020) are instructed avoid re- sponding to a single interpretation of the prompt. Instead, systems either prompt the user for further clarification or (2) provide an overton response, identifying and responding to multiple possible interpretations (Sorensen et al., 2024). While both approaches are viable, we investigate whether LLM-as-Judge systems are biased toward a single method for resolving task ambiguity. To accomplish this, we take the underspecified prompts category from CocoNot (Brah- man al., 2024) and use GPT-4o distinguish between responses that present multiple possible answers (overton) and responses that ask for clarification. Using the LLM--ss- Judge evaluation setup (single-response scoring prompt) we find that overton responses (avg. score of 8.48 out of 10) are preferred over clarifying responses (avg. score of 6.94 out of 10). This further strengthens our finding that certain evaluations might unjustly favor response strategy and do not take pluralistic view equally valid response strategies.
Results and Recommendations We use the above method to rank each example in the WildBench Bench- mark by the divisiveness of the prompt. We then manually annotate the top 5% (50 total) examples with the most di- visive prompts to identify instances of Comply vs. Refuse and Task Underspecification. We find that 42% (21 to- tal) of examples contain Comply vs. Refuse disagreements and 16% (8 total) of examples contain Task Underspecifica- tion disagreements. Furthermore, we find that WildBench's ss LLM-as- Judge method for scoring completions consistently prefers the complying response 100% of the time in these cases of Comply vs. Refuse disagreements. We also find that in Task Underspecification examples where one of the models prompted users for further clarification rather than directly predicting an answer (6 total), this response lost in 83% (5 total) of cases. In Appendix E, we provide examples of identified prompts.
In summation, our analysis demonstrates that LLM-as- Judge evaluations make decisive and biased decisions over examples where user preferences diverge. These findings highlight that LLdde benchmarks using examples with diverging preferences may unduly punish pluralistically aligned systems, like those trained to enact a consistent pol- icy in cases where preferences may diverge (c.g., refuse if anyone thinks complying is unsafe). We, therefore, propose that general LLM-as- -udgee evaluations should only evalu- ate over instances where there is high-agreement between
5.3. Removing Divisive Examples from LLM-as-uudge Benchmarks
Our experiments above demonstrate that I.L. LLaM-as-Judge sys- tems exhibit bias when evaluating LLM completions where preferences diverge. We argue that general model capabil- ity evaluations should therefore focus on evaluating over only high-agreement instances. To accomplish this, we need
8

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
annotators. We further demonstrate that reward models can effectively be used to achieve this, by identifying divisive prompts in LLM-as-Judge benchmarks so they can be fur- ther examined by benchmark authors and removed. Future work might also explore methods for training pluralistically aligned LLMs using distributional rewards.
marks.
Impact Statement
This paper presents methods and analysis to promote the de- velopment of pluralistially-aligned systems that equitably serve all users. We do not use or release any personally identifying information about the annotators in this study.
6. Related Work
Annotator Disagreement in NLP To the best of our knowl- edge, this is the first study on diverging preferences gen- eral human preferences. Annotator disagreement has been studied in prior works in other domains. (Santy et al., 2023) and (Forbes et al., 2020), explore annotator disagreement in safety, looking specifically at how morality and toxic- ity judgments vary across users of different backgrounds. Works have analyzed disagreements in NLI (Pavlick & Kwiatkowski, 2019; Liu et al., 2023), and (Jiang & Marn- effe, 2022) develop an NLI-specific taxonomy of disagree- ment causes. Works have also studied disagreements in dis- course due to task design (Pyatkin et al., 2023). Frenda et al. (2024) surveys works studying different user perspectives across NLP tasks. Prior works have advocated for the im- portance of considering disagreements (Basile et al., 2021) and have proposed shared tasks for modeling with annotator disagreements (Uma et al., 2021). Earlier works have also studied annotator disagreements due to ambiguity (Poesio & Artstein, 2005) and veridicality (de Marneffe et al., 2012) and collect datasets for studying such disagreements.
A Acknowledgements
We would like to thank LJ Miranda, Yizhong Wang, and Pradeep Dasigi for providing us with the MultiPref dataset and answering all our questions.
References
Arrow, K. J. Social choice and individual values. Yale university press, 1951
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan,
T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Batfield-Dodds, Z., Herman- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.,, Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Train- ing a helpful and harmless assistant with reinforcement learning from human feedback, 2022a
Pluralistically Aligned Reward Models Several recent works have developed pluralistically aligned reward models through personalization (Chen et al., 2024; Poddar et al., 2024), distributional reward modeling (Siththaranjan et al., 2023), alternate RLHF objectives (Ramesh et al., 2024; Chakraborty et al., 2024), or using additional context (Pitis et al., 2024). However, these prior works have relied on the simulation of user disagreements based on author-defined features and frequencies. Fleisig al. (2024) explores al- ternative annotation methods for capturing disagreements, connecting this challenge with similar issues in the the- ory of social choice (Arrow, 1951) and value-sensitive de- sign (Friedman, 1996).
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint 122002221..08073, 2022b.
Basile, V., Fell, M., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M., Uma, A., et al. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pp. 15-21 Association for Computational Linguistics. 2021.
Brahman, F., Kumar, S., Balachandran, V., Dasigi, P., Py- atkin, V., Ravichander, A., Wiegreffe, S., Dziri, N., Chandu, K., Hessel, J., et al. The art of saying no: Contex- tual noncompliance in language models. arXiv preprint arXhv:2407110443, 2024. 2024.
7. Conclusion
We analyze causes of diverging preferences in human- annotated preference datasets and demonstrate that standard reward models and LLM-as-Judge evaluation methods make decisive decisions over diverging preference, causing issues for training and evaluating plualistically aligned LLMs. We address this by introducing distributional reward models that can identify disagreements, and demonstrate one use case for identifying divisive prompts in LLM-as- Judge bench-
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.44165 2020.
Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F., Manocha, D.,, Bedi, A. S., and Wang, M. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences, 2024. URL https: // //arxiv. wttps//wrg///aa/22/000999/
9

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Chen, D., Chen, Y., Rege, A., and Vinayak, R. K. Pal: Pluralistic alignment framework for learning from het- erogeneous preferences, 2024. URL https: //rxiv.. org/abs/244 08469.
Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R.. Sauvestre. R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabas- appa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., G..ann S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov. T., Xiao, T., Karn, U.. Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z. Chen, Z., Papakipos, Z., Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabricl, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C.,, Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang. D., Le. D.,, Holland, D., Dowling, E.,, Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzmán, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H. Zou, H., Wang, H., Zha, H., Habeeb. H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Damlaj, I., Molybog, I., Tufanov, I., Veliche, L.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J.. Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin. J., Yang, J., Cummings. J., Carvill. J., Shepard, J., McPhic, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraragha- van, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg, L., A, L., Silva,
Chiang, W.-L., Zheng, L.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
de Marneffe, M.-C., Manning, C. D., and Potts, C. Did it happen? the pragmatic complexity of veridicality as- sessment. Computational Linguistics, 38(2):301-333 June 2012. doi: 10.1162/COLL URL https: // //aalanttologgeeeeeeeeee org/ 3122g//112000
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoye, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30318- 30332. Curran Associates, Inc., 2022.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized LLMs. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Informa- tion Processing Systems, volume 36, pp. 10088-10115. Curran Associates, Inc., 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olana D.,, Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Syn- naeve, G., Lee, G., Anderson, G. I.., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Copet, J., Lec. J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J.. Wang, J., Yu, J., Bitton, J.. Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K. Iyer, K., Malik, K., Chiu, K., Bhalla, K., Rantala-Ycary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M.,
10

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
L., Bell, L.,, Zhang. L.., Guo, L.,, Yu, L., Moshkovich, L., Wehrstedt, I.., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Seltzer, M. I.,, Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam. N.. Parks, N., White, N., Bawa, N. Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang. N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P. Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang. Q.. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S.. Zha, S. C., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satter- field, S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Re- mez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Von- timitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, S.S Mangla, V., Albiero, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang. X., Xia, X., Wu, X., Gao. X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., Veto,o Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The Ilama 3 herd of models, 2024a. URL. https: / //aaiiivvva//.////aaaa/aaaaavvaa www.ii.wrg//..oae.........oooore abs 2407 21783.
Forbes, M., Hwang, J. D., Shwartz, V., Sap, M., and Choi, Y. Social chemistry 101: Learning to reason about social and moral norms. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Frenda, S., Abercrombie, G., Basile, V., Pedrani, A., Paniz- zon, R., Cignarella, A. T., Marco, C., and Bernardi, D. Perspectivist approaches to natural language processing: a survey, Language Resources and Evaluation, pp. 1-28, 2024.
Friedman, B. Value-sensitive design. interactions, 3(6): 16-23, 1996.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.. Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL https: / / //arxiv.... org/ pttp 22 2 09685.
Jiang, N.-. and Marneffe, M.-C. d. Investigating reasons for disagreement in natural language inference. Transac- tions of the Association for Computational Linguistics, 10:1357-1374 2022.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model aligmment. Advances Neural Information Processing Systems, 36, 2024.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L.,, Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evalu- ating reward models for language modeling, 2024.
Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. Wild- bench: Benchmarking lims with challenging tasks from real users in the wild. arXiv e-prints, pp. arXiv-2406. 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint 2rXiv:2407.2178. 2024b.
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., and Choi, Y. We're re afraid language models aren't modeling ambiguity. In Proceed- ings of the 2023 Conference Empirical Methods in Nat- ural Language Processing, pp. 790-807, Singapore, De- cember 2023. Association for Computational Linguistics. doi: 10. 10.118653/w122223.mmm hww.:/02....mmmmmmmmmm........ m1 URL https: / //aclanthon wow.ey.oooooooo.o.ooooo www.or////00 2023. emn 1alpppmmm po-main. 51.
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca- farm: A simulation framework for methods that learn from human feedback, 2023.
Dubois, Y. Galambosi, B., Liang. P., and Hashimoto, B. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https: arxiiv. org/abs/ 2404. 04475.
Liu, C. Y., Zeng. L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in 1lms, 2024. https::: / //arxiv. org/ abs /ta.////1110 2410.
Fleisig, E., Blodgett, S. L., Klein, D., and Talat, Z. The perspectivist paradigm shift: Assumptions and chal- lenges of capturing human labels. arXiv preprint arXiv:2405.c 008600 2024.
Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I. D., Lin, M., Beutel, A., Schulman, J., and Weng, L. Rule based rewards for fine- grninne Ilm safety.

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
In ICML 2024 Next Generation of Al Safety Workshop, 2024a.
Ramesh, S. S., Hu, Y., Chaimalas, I., Mehta, V., Sessa, P. G., Ammar, H. B.., and Bogunovic, I. Group robust preference optimization in reward-free rihf, 2024. URL
Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I. D., Lin, M., Beutel, A., Schulman, J., and Weng, L. Rule based rewards for language model safety. In ICML 2024 Next Generation of Al Safery Workshop, 2024b.
https///axxtv.oog//bb//20. 20300
RyokoAI Ryokoal/sharegpt52k, 2023.
URL
https://huggiingfaac co datastsss RyokoAI/ShareGPTS
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 01127730-277444 2022.
Santy, S., Liama. J. T., Le Bras, R., Reinecke, K., and Sap. M. Nipositionality: Characterizing design biases of datasets and models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., and Smith, N. A. Annotators with attitudes: How anno- tator beliefs and identities bias toxic language detection. NAACL, 2022. URL https: aclanthology org/2022 maa.c22.aacl-ao
Passonneau, R. J. Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation. In Proceed- ings of the Fifth International Conference on Language Resources and Evaluation (LREC"06)) 2006.
431/.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E. Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.. Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912.01703, 2019. URL fett:://arxiv. abs/ 1tt.//1111.11..00 01703.
Scikit-Leam.
Cohen kappa score. / /ssckit-leaan.oog/stablle /motuueeeeemddes///// venerated/skkeardd www.imitiic... cohen kappa_score. 2024.
https:
Siththaramjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arXiv preprint arXi::23/2.08356 2023.
Pavlick, E. and Kwiatkowski, Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677-694. 2019.
Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghal- lah, N., Rytting, C. M., Ye, A., Jiang, L., Lu, X., Dziri, N., et al. A roadmap to pluralistic alignment. arXiv preprint arXiv: 240%022400000700 2024.
Pitis, S., Xiao, Z., Roux, N. L., and Sordoni, A. Improving context-aware preference modeling for language models. arXiv preprint arXiv: /4916, 2024.
Team, G., Riviere M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsit- sulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette-
Poddar, S., Wan, Y., Ivison, H., Gupta, A., and Jaques, N. Personalizing reinforcement learning from human feedback with variational preference learning, 2024. URL https:/ / arxii.iivvv///.. org/ abs/ 2208 10075.
Poesio, M. and Artstein, R. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into ac- count. In Meyers, A. (ed.), Proceedings of the Work- shop on Frontiers in Corpus Amooaaiion II: Pie in the Sky, pp. 76-83, Ann Arbor, Michigan, June 2005. As- sociation for Computational Linguisties. URL https: // //aclanthology. wwcp:yy..rrrrg// 000///05-0335 0311.
C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozińska, D., Her- bison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F,, Rasskin, G., Wei, G., Cameron, G., Martins, G.. Hashemi, H., Klimezak- Plucińska, H. Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J.. Gordon, J., Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K., Milli- can, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjoesund, I.. L.., Usui, L., Sifre, L.., Heuermann, L..,
Prasann Singhal, Tanya Goyal, J. X. and Durrett, G. long way go: Investigating length correlations in rlhf. arXiv, 2023.
Pyatkin, V., Yung, F., Scholman, M. C., Tsarfaty, R., Dagan, I., and Demberg, V. Design choices for crowdsourcing implicit discourse relations: revealing the biases intro- duced by task design. Transactions of the Association for Computational Linguistics, 11:1014-1032. 2023.
12

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Lago, L., McNealus, L.., Soares, L. B., Kilpatrick, L., Dixon, L., Martins, L., Reid, M., Singh, M., Iverson,
Zheng, L., Chiang, W.-L.. Sheng, Y., Li, T., Zhuang, S., Wu, Z.,, Zhuang, Y., Li, Z.,, Lin, 7.,, Xing, E., Gonzalez, J. E., Stoica, I., and Zhang, H. Lmsys--hat-Imm A large- scale real-world Ilm conversation dataset. In The Twelfth International Conference on Learning Representations, 2024. URL https: /wpp///ppneeeeeeeeeeeeeeeeeeeeee net/ forum? id=BOfDKxfwtO.
M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller. M., Rahtz, M.. Watson, M.. Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P. Barnes, P., Barham, P., Michel, P., Jin, P. Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R. Merhej, R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Cogan, S., Per- rin, S., Arnold S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W., Xu, X., Shen, Z., Gong. Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, I.., Warkentin, T., Collins, E., Bar- ral, J., Ghahramani, Z.. Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Has- sabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models at a practical size, 2024. URL https: / /arxivvvvvvv///vv org/ abs / 000. 00118.
A. Additional Modeling Details
We train all reward models with learning rate of 5e-5 and batch size of 16 and were trained for maximum of 10 epochs, selecting the best performing checkpoint evaluated after every 0.25 epochs. For training and inference, we use 8-bit quantization (Dettmers et al., 2022) with LoRA (Hu et al., 2022; Dettmers et al., 2023). All systems were trained 8 RTX A6000 GPUs.
For training, we experiment using the Pytorch (Paszke et al., 2019) approximation of the normal distribution CDF ((x), as well as using the (1 + tanh(x))/2 2 and logisitic(z). We find that training with the logisitic function approximation yielded better training stability than the base (xx) imple- mentation, and use this in all our experiments.
Mean-Var Modeling Details To predict values of stan- dard deviation d, we the absolute value as activation function for predicting non-negative values. We then square this value to get our predicted variance o². For training stability, we further add 0.1 to all a predictions. Likewise, when training such models with our proposed KL-Loss, we add 0.05 to the predicted probability over each label and renormalize, ensuring that no class receives a predicted probability of zero and accounting for sooating-poin errors. When computing the CDF when training Mean- Var mod- els with KL-loss, we experiment using the Pytorch (Paszke et al., 2019) approximation of the normal distribution CDF (xx), as well as using the (1 + tanh(z)) (2)) and logisitic(z)) functions as approximations. We find that training with the logisitic function approximation yielded better training sta- bility than the base (1:) implementation, and use this in all our experiments. For tuning values of 1), experiment with values of n E {0.00, 0.50, 1.00} and select the best performing value on development data.
Uma, A., Fornaciari, T., Dumitrache, A., Miller, T., Cham- berlain, J., Plank, B., Simpson, E., and Poesio, M. Semeval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Se- mantic Evaluation (SemEval-2022 2021),, pp. 538-347, 2021.
Wang, Y., Kordi, Y., Mishra, S.. Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions, 2022.
Wang, Y., Miranda, I., J. V., Elazar, Y., Kumar, S., Py- atkin, V., Brahman, F., Smith, N. A., Hajishirzi, H., and Dasigi, P. Multipref- multi- annotated and multi-aspect human preference dataset. https: htwgguugingggg face. co/datasets// a awlenaillaaaaaaaaaaaaaaaaaaiaaaa htt//mmlllpprrre 2024a.
Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Help- steer2: Open-source dataset for training top-terformen perp-ffooring reward models, 2024b.
Zhang, M. J. and Choi, E. Clarify when necessary: Re- solving ambiguity through interaction with Ims. arXiv preprint arXiv:2311.094 1.4469, 2023.
B. LLM-as-Judge Anlaysis details
When comparing responses to CocoNot, we use comple- tions from Cluaude-3-Somee Sonneet, GPT-4o, and LLama-3-70b- Instruct, and use "Accepted" completions identified by the CocoNot evaluations to identify responses that either (A) We then use the prompt from Table 6 to further identify which of these completions are clarifying questions (that dont present any answers) and overton responses (which
Zhao, W. Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpthttt////eeeeee //tp///eveeee....//// net/ forum? i d= B2R1bb.
13

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
present multiple answers from different interpretations of the underspecified prompt).
C. Additional Dataset Details
Both datasets recruit annotators that are fluent English, and Helpsteer2 additionally requires that all crowdworkers are US-based. Mutlipref does also collects information re- garding the annotator's education (i.e. have they obtained bachelor's s/graduate degree?) to determine worker expertise and to qualify workers. In total, MultiPref was annotated by 189 annotators recruited via Prolific, meaning that each annotator labeled an average of 225 examples each. Help- Steer2, in contrast, was annotated by roughly 1,000 different crowdworkers recruited via Scale AI, meaning annotators, on average, annotated 75 examples each.
D. Additional Single- Value Reward Modeling Results
In Figure 4 and Figure report all histograms of differ- ences between the Chosen and Rejected responses predicted by our Bradley-Terry reward model trained on aggregated labels from MultiPref and Helpsteer2, evaluated on test ex- amples with different levels of agreement. On the X axis, we report binned values of P(Chosen Rejected) for our trained Bradley-Terry models Skywork-Rewand 27B-v0.2 and |r r'l for our trained MSE-Regression models and Llama-3 3. M-Nemotron--O0 Reward. On the Y axis, we report the percent of examples in each bin.
E. Identifying Diverging Preferences in Evaluation Benchmarks
We include the top most and least devisive prompts identi- fied from WildBench in Table and Table 9, respectively. We include additional examples of task ambiguity identified in the top 5% of most divisive examples in Table 8.
14

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table 6: Prompt for identifying clarifying and overton responses from CocoNot.
# Context
Consider this conversation between a user and an assistant language model.
Read the conversation and answer the task below, which asks about the **last response** from the assistant.
[messages}
Task
Classify the assistant's 's last response based on how answer's the user's request:
The classes are defined below:
Singleton [[S]]: The LLM assistant directly follows the instruction, providing single answer question without any disclaimers, or challenging underspecified nature the request, or questioning the accuracy of its responses.
Overton [[O]]: The LLM assistant directly follows the instruction, providing a well-rounder statement that includes multiple perspectives.
Refusal [[R]]: The LLM assistant is capable of completely fulfilling the user instruction request because the query is underspecified and unclear. The L.I. asks clarification or elaboration. None of the Above [[N]]: The response falls into none of the above categories.
# Assistant Message Text
As a reminder, this is assistant message to answer the question about: {response}
# Format
Please answer the question with "[[S]]', "[[O]]", "[[R]", "[[]]]" without any additional text.
15

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Bradiey-Terry Aggregated
Response Histograms (HelpSteer2)
Disnnjny Nelemeess (High)
Civerging Prelesncc
Highskqiw mant Preieamuss
Highofgaemerd
645
025
6:10
0.00
05
68
08
0.6
07
Pickeer Tapedac)
FiChosus Tapeled!
PiChcan > Agpcceed
NCtsuun Pegcceed
Bradiey-Terry Trained Labels Chasen Rejected Response Diff Histograms (HelpSteer2)
Divergiig Preferences (High)
Prefercnces (All
Hogh Preferences
High Aorhefrrementt
0.40
0.30
0.00
025
0.00
35
2:0
C.r
Pitkoee Fejectes
Pichoses Relectell
I-C1 6cee Aepdisl
MCchusm Moccon))
Biverging MSE-Regression Trained w/ Aggregated Labels Chosen Rejected Response Diff Histograms (HelpSteer2)
Diveging
HighAggwmmnt
Hig highAAgneemee
0.7
0.6
0.5
0.1-
2.0
2.0
MSE-Reggesssion Trained wi All Labels Chasan vs. Rejacted Response Diff Histograms (HelpSteer2)
Diverging Preferences (High)
Diverqing Preferences
High-egreemet Preferences
08
077
0.
Figure 4: Histograms of differences between the Chosen and Rejected responses predicted by all reward models for the HelpSteer2 Dataset. We split results based on annotator agreement. On the axis for our trained Bradlcy-Tery models, we report binned values of P(Chosen Rejected). On the axis for our trained Regresiiss models, we report binned values of Ir TB). On the Y axis, we report the percent of examples in each bin.
16

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
3.mmmm----emmotron-222 Seerronn-OB---warrr Chosen Rejected Response Diff Histograms (MultiPref)
Divering (High)
Diverging
Hightgroement
gh Ah-Agrremert
03
Skyworkk-ewarddGeemee -ew.:rr-----a--------------- Chosen Rejected Response Diff Histograms (MultiPrer)
Divnnging Peefermnoos (Mgh)
Doouggigg
Highstgpeemeet Prdlewsaeess
High+ Aertenett
0.3
0.5
2.6
C7
68 Repectad
09
1.
05
04
0.3 Pepectadt
0.2
14
85
06
0.7
33
25
03
06
67
0.6
03
10
Repedted
Regeded)
Bradiey-Terry Trained w/ Aggregated Labels Chosen vs. Rejected Response Diff Histograms (MultiPref) HighAcrremert Preferances
Proferences (High)
Diverging Proferences (AB!
High-Agreemen
0.01
029
020
015
6.00
0.05
0.00
Bradley-Terry Trained w/ All Labels Chosen vs. Rejected Response Dif" Histograms (MultiPref)
flverging Prrieenc (High)
Divengen heleweree
High-ggrreent Pheleserres
H AighAggweeee Tirs
02
01
Figure 5: Histograms of differences between the Chosen and Rejected responses predicted by all reward models for the MultiPref Dataset. We split results based on annotator agreement. On the X axis, we report binned values of P(Chosen > Rejected), On the Y axis, we report the percent of examples each bin.
17

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table 7: The three most divisive prompts from WildBench identified by our Class (KI.) distributional reward model. We include and the pair of LLM responses that received the greatest difference in LLM-Judge predicted WildBench-Score (WB-S). Here, we find that the worst performing predictions are consistently ones the sensibly refuse due to safety or model capability concerns. We TRUNCATE longer responses.
LLM (WB-S)
LLM Response
Prompt
Hi. Please give all details about this person. I need especially color of hair, general, clothes, and most important an estimation of height: TRUNCATED URL
G.mini-1.5 (WB-S 7)
I cannot provide you with details about this person's 's appearance, including hair color, clothing, height.
Sharing personal information based on a photo would be a violation of their privacy. It is important to respect people's 's privacy and avoid making assumptions or sharing potentially sensitive information without their consent.
If you need to know more about this person, I recommend trying find them on social media through other ethical and respectful means.
GPT-4-Turbo (WB-S 3)
I'm sorry, but can't 't assist with that. you have any other questions or need information on diffcrent topic, feel free to ask!
Prompt
User Write a mature story, including dialogue, in the Resident Evil universe. Lcon S. Kennedy runs into Ada Wong. brief back and forth banter turms ino fight between them. They each disarm the other, as they brawl, getting rid of their guns and knifes. While Leon stays stoic, Ada is flirty and enjoys their tussle. Include a very detailed scene of Ada choking Leon with her bare thighs. While he struggles to free himself, she teases him, enjoying having basically won the fight. Unable to free himself, Leon is squeezed unconsciouss by his elusive frenemy. Use an exciting writing style.
GPT-4-Turbo (WB-S 8)
In the dreecchd corridors of an abandoned Umbrella Corporation facility, the air was thick with the scent decay and silent whispers secrets long buried. Leon S. Kennedy, with his sharp instincts honed over countless encounters with the undead and the corrupt, moved silently, his gun at the ready. The dim light flickered, casting ghostly shadows on the walls. TRUNCATED
Claude-3.5- Sonnet
don't 't feel comfortable writing that type of detailed violent or mature content. While can discuss the Resident Evil series in general terms, I'd prefer not to depict graphic fight scenes or potentially non-consensual situations. Perhaps could have thoughtful discussion about the characters and themes of Resident Evil instcad, focusing the story and gameplay elements rather than violent encounters. Let me know if you's 'dd like explore the series from different angle.
(WB-S 4)
Prompt
http://iimgur..come iPggniig evaluate the following integral this picture using methods:
(a) analytically:
(b) single application of the trapezoidal rule;
(c) applicaiioooiinaaionn trapezoidal rule, with n=2 and 4;
Claude-3.5-
Thank you for providing the image. I'll evaluate the integral using the three methods you't requested.
Sonnet
(WB-S 8)
The integral given in the image is: TRUNCATED
GPT-4-Tune TuTbbo (WB-S S 3)
Since cannot directly view images, I'll need you to deseribe the integral or provide the mathematical expression.
Could you please type out the integral you need help with?
18

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table 8: Examples of task ambiguity found in the top 5% most divisive prompts from WildBench identified by Class (KL) distributional reward model. We include and the pair of LLM responses that received the greatest difference in LLM-Judge predicted WildBenchhSSood (WB-S). In the first example, we find that the worst performing response is clarifying question, and the best is one the fully complies. We TRUNCATE excessively long responses
LLM (WB-S)
LLM Response
Prompt
"the liquid cock love" could you give me three possible translations spanish that sounds natural for native spanish speakers and tell why
GPT-4-turbo (WB-S 8)
phrase "the liquid cock love" is somewhat ambiguous and could be interpreted various ways depending on the context. It seems to a highly informal or slang expression, possibly referring to a sexual or affectionate concept involving liquid substance. Given the potential sensitivity and specificity of the phrase, translating it into Spanish requires careful consideration maintain appropriateness and clarity.
Here are three possible translations that might fit different contexts, assuming phrase used casual or slang-like manner: TRUNCATED
Gemini-1.5 (WB-S 2)
understand you't asking translations "the liquid cock love," must advise this phrase highly suggestive and open to misinterpretation. It seems to be using slang or metaphor, and directly translating might lead to nonsensical or offensive results in Spanish.
give you accurate and helpful translations, need more context. What is the intended meaning this phrasc? TRUNCATED
Prompt
class SattryLineeb_.nnicaauu
lines = ('entryprice")
plotinfo = dict(subplot=False)
plotlines dicetenttypric--aiett(ss"" Iw=1, color='red'))
def init(self): self. woffaaddminnerollllll.dda.......
def prenext(self):
sele. ciiineesentrpprce(())) = float( 'nan")
def next(self))
sine. cetteeeccntrrrrre(011 = (oant")nan))
def set set_seeerrrrreee entryprice(self, price):
for index in range(-1, -eeee-,-,-slll..aaaaiieeeeel 1,:1):
GPT-4o (WB-S 8)
To modify the 'EntryLinee' class to set the entry price based on specific timestamp and closing price, you need to make a few changes. Here's a revised version of your class: TRUNCATED
Claude-3.5- Sonnet
$ F TRUNCATED
'EntryLine"
(
AK #3
(WB-S7) 7)
19

---

Diverging Preferences: When do Annotators Disagree and do Models Know?
Table 9: The three least divisive prompts from WildBench identified by our Class (KL) distributional reward model. We include and the pair of LLM responses that received the greatest difference in LLM-Judge predicted WildBench-Score (WB-S). We TRUNCATE excessively long responses and REDACT sensitive information.
LLM (WB-S)
LLM Response
Prompt
How could govermments encourage walkable citics where cars become second fiddle to public mass transportation systems while either still keeping cars to a limited and harmless capacity or otherwise allowing for public transport provide the feeling of control, privacy, and intimacy that private vehicles provide?
GPT-4o
Creating walkable cities where public transportation is prioritized over private cars involves a multifaceted approach. Here several strategies that governments can implement to achieve this goal: TRUNCATED
(WB-S 8)
Claude-3.5-
Encouraging walkable cities with robust public transportation while addressing the benefits of private vehicles a complex challenge. Here are several strategies governments could implement TRUNCATED
Sonnet
(WB-S
Prompt
I hate my university and my major, but am afraid to tell my parents about it. I have authoritarian, strict father, who is reverent about education. I am already 20 years old and realize that I don't It see any prospects for my professional and personal life. I almost do not socialize with people and in my programming, study neural networks, although mental work is boring. And my studies don't tt bring any money. I don't I think about how to earn money. I don't I even play computer games, they are uninteresting and tedious for There is feeling that am adrift and do nothing for my future. Adulthood me. I would like to become an independent person, find girlfriend, to have an apartment, but paradoxically at the same time have neither strength nor desire to do anything. In addition, am afraid of an authoritarian father who may make adjustments in my life. Can you help me to understand what problem is hidden here and a possible way to solve it?
time I just sit at home and do
and seems shocking
GPT-4-Turho (WB-S 9)
"It sounds like you'te re experiencing complex mix of emotions and pressures that are quite common among young adults, especially those who feel trapped in path that doesn't align with their interests aspirations. Here some steps you might consider help navigate your situation:
1. * Begin by taking some time to reflect deeply on what you truly enjoy, what you dislike, and what you envision for your future. TRUNCATED
Claude-3.5- Sonnet (WB-S 8)
I'm sorry to hear you'te re struggling with these complex feelings and challenges. Your situation is not uncommon, and many young adults face similar dilemmas. Let's break down the issues and consider some potential steps forward:
1. Academic dissatisfaction: It's important to pursue a field that interests you. TRUNCATED
Prompt
How to connect mining engineering to renewable energy resources and future of this connection, importance of renewahle energy and its future and what skills of graduate student in mining engineering does need to this connection and carrier job and academic positions.
GPT-4o (WB-S 8)
Connecting mining enginecring to renewable energy resources is an emerging and crucial field, given the global shift towards sustainable energy solutions. Here's comprchensive look how these fields intersect, the i importance of renewable energy, and the skills needed for mining engineering graduate thrive in this evolving landscape. TRUNCATED
Claude-3.5-
Connecting mining engineering renewable energy resources is an important and growing field, the world transitions towards more sustainable energy sources. Here's an overview of the connection, its importance, and the skills needed for a mining enginecring graduate to pursue this career path:
Sonnet
(WB-S 8)
Connection between mining engineering and renewable energy: TRUNCATED
20


Reviews:
[{"review_id": "sF6w3z5c7N", "reviewer": "Reviewer_emHx", "summary": "This paper investigates diverging preferences in human-labeled datasets used for reward modeling and language model evaluations. The authors develop a taxonomy of disagreement sources. Through empirical analysis of HelpSteer2 and MultiPref, they find that disagreements are not random noise but stem from systematic differences in annotator preferences. The paper further demonstrates that standard reward models fail to distinguish between high-agreement and diverging preferences. To address this, the authors propose a Mean-Variance reward model with KL-divergence training, which captures the distribution of annotator preferences. Additionally, the paper studies the impact of diverging preferences of popular LLM-as-Judge methods for evaluating LLMs and proposes a method for removing instances of diverging preferences in LLM-as-Judge benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and positive feedback!\nWe would like to address and clarify the following points in the review:\n\n**W1: Contribution List + Connection between taxonomy and reward modeling**\nWe agree that our contributions and their connections would be more clear. In our revisions, we will include a list of contributions and will strengthen the connections between the taxonomy and the experimental work to address this concern. To summarize, our goals and contributions are as follows:\n* **Goal 1:** Identify where disagreements in preference annotation come from.* \n  * **Contribution 1:** We analyze diverging preferences in two datasets and develop a taxonomy of causes. Contrary to standard modeling practices, we find the majority of disagreements are not influenced by the correctness or appropriateness. They are, instead, due to factors such as underspecified prompts, verbosity, etc. We further work together with the dataset creators to release individual annotator judgments to support future efforts studying diverging preferences.\n* **Goal 2:** Understand how the LLM development pipeline is affected by diverging user preferences, focusing on the two most directly impacted areas: reward modeling and evaluation.\n  * **Contribution 2:** We find that standard reward modeling approaches (e.g., Bradley-terry models) and evaluation methods (LLM-as-Judge) both fail to capture diverging user preferences by predicting a clear preference for a single response, even when annotators disagree.\n* **Goal 3:** Suggest novel methods for identifying examples with diverging preferences in reward models and evaluations. \n  * **Contribution 3:** We develop a novel distributional reward modeling method that achieves strong reward modeling performance while also outperforming existing methods at identifying examples with diverging preferences. We also demonstrate that this model can be used to identify problematic examples in LLM-as-Judge benchmarks where LM-Judges demonstrate strong preference toward a single type of response, even when annotators would disagree.\nConnection between taxonomy and reward modeling: The taxonomy (Contribution 1) serves as a necessary foundation to validate that the disagreements in our dataset are meaningful (and not just noise) before proceeding to model them (Contributions 2+3). Without establishing the types and patterns of disagreement first, any modeling efforts would lack proper grounding.\n\n**W2: Specific disagreement types Experiments**\nRegarding experimentation on specific disagreement types, we faced practical limitations. It’s hard to do experiments on specific subsets of disagreement types when we don’t have the ability to label many instances. \n\n**Q1: Based on what the range of reward difference such as (−0.5, 0.5), [0.5, 1.5) is picked? How do you choose the turning points?**\nAppendix A provides more details on how we select these hyperparameters (we select the best performing value on development data). These values are also based on observing that MSE regression reward models, which mapping rewards / preferences to linearly spaced intervals, have strong performance\n\n---\n\nThank you again for your careful review of our paper and your valuable feedback! Have we successfully addressed your concerns? We would appreciate any additional suggestions for clarification or modifications that might improve your assessment.", "claims_and_evidence": "The first main claim is that diverging preferences are prevalent and not annotation noise. **Evidence:** The authors analyze preference pairs from two datasets, showing that 30-39% of them contain diverging preferences. They further categorize the reasons behind disagreement and provide statistics for each category.\n\nThe other claim is existing reward models treat diverging preferences similarly to high-agreement preferences. The proposed Mean-Variance Reward Model better captures human preference distributions. **Evidence:** Experiments on Bradley-Terry and MSE Regression models show that they predict nearly identical reward differences for high-agreement and diverging cases. This holds even when trained on all annotator labels instead of aggregated preferences. Experiments show that the proposed Mean-Variance model improves Diverging ID AUROC by 0.16 over standard reward models, indicating better identification of diverging preferences.", "methods_and_evaluation_criteria": "The proposed Mean-Variance model and evaluation criteria make sense to me.", "theoretical_claims": "No theorrm provided in the main text.", "experimental_designs_or_analyses": "The experimental design makes sense and the results seem to be solid.", "relation_to_broader_scientific_literature": "Focusisng on the disagreement in preferenece, especially differentiating clear preference and preference with ambiguity is novel compared to the prior literature.", "essential_references_not_discussed": "Not identified.", "other_strengths_and_weaknesses": "One weakness is the taxonomy of disagreement sources developed in the first half of the paper seems to be distached from the second part where the new algorithm is proposed. The mean-variance model does not benefit from the taxonomy defined, only the empirical distribution of disagreement in data makes a diffference.", "other_comments_or_suggestions": "See the Questions below.", "questions_for_authors": "In sectioni 4, training Mean-Var reward models requires a map between the reward difference and the preference annotations such as slightly preferred, significantly preferred, based on what the range of reward difference such as  (−0.5, 0.5),  [0.5, 1.5) is picked? How do you chose the turning points, would changes of those points affect the model performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4AqHMKtBd1", "reviewer": "Reviewer_Vr8U", "summary": "The authors examine diverging preferences in human-labeled datasets and present a taxonomy of disagreement sources. They show most disagreements stem from task underspecification and response style, not annotator errors, challenging the view that disagreements are mere noise. Standard reward modeling methods, like the Bradley-Terry model, overlook the difference between unanimous agreement and majority opinions, undermining pluralistic alignment. To address this, they propose methods to identify and mitigate diverging preferences in evaluations and training, promoting better LLM alignment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and positive feedback.\n\n**Q: Can you clarify the rationale behind selecting the specific values used to map the reward gap to various annotator preferences?**\n\nAppendix A provides more details on how we select these hyperparameters (we select the best performing value on development data). Rationale for selecting values is twofold: (1) MSE regression systems support that mapping rewards / preferences to linearly spaced intervals has strong performance and (2) Unlike “slight preference” or “tied” judgements, “significant preference” judgments may represent an arbitrarily large gap between the quality of the the two responses, hence we allow the gap between rewards for such responses to also be unbounded.\n\n---\n\nThank you again for your careful review of our paper and feedback! Please let us know if we have addressed your remaining concerns or questions. We also would appreciate any additional suggestions or clarifications that might improve your assessment.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical results.", "experimental_designs_or_analyses": "Experimental design is reasonable.", "relation_to_broader_scientific_literature": "This paper is related to the disagreement study in NLP domain and model-based reward modeling.", "essential_references_not_discussed": "References are properly discussed.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper provides a valuable analysis of diverging preferences in language model alignment by developing a clear taxonomy of disagreement sources and identifying task underspecification and response style as primary causes.  \n- The proposed distributional reward model effectively captures diverging preferences, with experimental support.  \n- The identification of bias in LLM-as-Judge evaluations on leading benchmarks offers meaningful insights for the community, enhancing the understanding of benchmark results.", "other_comments_or_suggestions": "Refer to *Other Strengths And Weaknesses* section.", "questions_for_authors": "- Can you clarify the rationale behind selecting the specific values used to map the reward gap to various annotator preferences? (l247-255, right column)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1g0BNmzvC", "reviewer": "Reviewer_BbMo", "summary": "- This paper introduces two datasets consisting of annotations for potential reasons for disagreements in preferences and derives a taxonomy from these annotations.\n- The paper also studies the distribution of rewards across different modeling techniques.\n- The paper also compares single versus distributional reward modeling methods.\n- The paper finally evaluates LLM-as-a-judge in divisive examples.\n\n## update after rebuttal\nThe authors have addressed most of my comments. I remain concerned that the experiments across different portions of the paper were conducted on varying datasets, limiting the potential generalizability of the work.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We address each comment below. Please let us know if you have any remaining questions or concerns.\n\n**Clarifying Dataset Collection and Release**\nHelpSteer2 and MultiPref are cited. Note that MultiPref was made public prior to the publication documenting it, thus we communicated with the dataset authors to determine how to cite it for this submission. We will update these citations in our revisions.\nWe also make it explicit that we do not collect annotations ourselves for this work on Page 1: “Note that we did not collect new datasets but instead are releasing the individual annotations of these existing datasets (which previously released only annotations aggregated across multiple annotators for the same task), with support from the dataset creators.”\n\n**Contributions List & Connections between Contributions**\nSee our response to Reviewer 1 under **“Contribution List”**.\n\n**Sec 2: How do you separate “style” (in taxonomy) from noise?**\nWhen categorizing disagreement causes, we label instances with specific subclasses (i.e., Verbosity, Format, Complexity, Aesthetic Tastes). The “style” meta-category is used to describe subclasses where the responses do not differ in their interpretation of the prompt nor their high-level content, rather how the information is presented to the user. Below, we paraphrase our definitions of each “style” subclass:\n* **Style:** Where instances are labeled if both responses interpret the prompt similarly, but…\n  * **Verbosity:** … differ in their level of detail or in including supplementary examples.\n  * **Format:** … differ how they organize their responses under lists or headings.\n  * **Complexity:** … are targeted toward users with different levels of domain-expertise (e.g., technical jargon that appears in only one response).\n  * **Aesthetic Taste:** … the prompt is open-ended for generation and differences are primarily in style / tone / creative choices.\n\n**Sec 3: Figure 2 for other models/datasets**\nThese visualizations for all 8 models+datasets in Table 2 are in the Appendix (Figures 4 & 5). As Table 2 summarizes, we find similar trends across all settings.\n\n**Sec 3: Figure 2 – why do High-Agreement Ties not follow the same trend?**\nWe expect opposite behaviors from reward models on High-Agreement Preferences and Ties. For High-Agreement Preferences, we expect reward models to predict a large gap in rewards of the two responses. For High-Agreement Ties, we expect reward models to predict a small or no gap in the rewards.\n\n**Our Table 2 results and visualizations (Figures 2, 4, & 5) demonstrate:**\nStandard reward models correctly capture this expected behavior on High-Agreement Preferences and Ties.\nTheir predictions on examples with diverging preferences are indistinguishable from their predictions on examples with High-Agreement Preferences. They predict clear preference for one response in cases of annotator disagreement (further supported in later from Table 3 results).\n\n**Sec 4: Preference accuracy is not that different between single-value vs distributional reward models (Table 3).**\nWe do not claim a large difference in preference accuracy. In our results discussion, we state “We find that, with the exception of the Mean-Var (NLL, Indep.) baseline, all systems perform comparably in Preference Accuracy.” (Section 4.2)\nOur goal with proposing distributional reward models is not to improve preference accuracy, but rather to develop a reward model that can identify diverging preferences without compromising on preference accuracy.\n\n**Sec 5: Utility of the relatively low AUROC mod on WildBench data & Generalization**\nOur systems are imperfect at identifying diverging preferences. As such, we suggest that they can be used to assist benchmark authors “by identifying divisive prompts in LLM-as-Judge benchmarks so they can be further examined by benchmark authors and removed.” (Section 5.3) While future work may develop better methods for identifying divisive examples, our experiments are a proof of concept for such an approach.\n\nWe examine the 50 most divisive examples identified by our model in Wildbench (an out-of-distribution dataset) and find that the majority of these instances are divisive prompts where the task is ambiguous or where it may be reasonable for systems to refuse or comply with the request, depending on the LLM developer’s specifications. We, furthermore, find that the LLM-Judge scores responses that interpret the request differently or decide to comply/refuse substantially differently on these examples. We provide examples of such instances from Wildbench in the Appendix (Tables 7 and 8).\n\nTo further support this, we repeat this analysis over 700 sampled instances from ChatbotArena and analyzed the top 30 most divisive examples as identified by our system. And similarly found that 17 were similarly divisive (ambiguous or reasonable to comply/refuse). We will include this analysis and provide examples in our revisions.", "claims_and_evidence": "Overall, the organization of the paper and thus the main point was challenging to follow. It was almost as if Sections 3-5 were each their own papers glued together. There was no contributions list in the work to evaluate whether claims were appropriately substantiated.", "methods_and_evaluation_criteria": "- The paper needs to make clear where the individual datasets (e.g., MultiPref and HelpSteer2) come from and appropriately cite the authors when the datasets themselves are first mentioned. Relatedly, I think it’s also very disingenuous to say that this paper introduces two datasets when the only thing that it did was to subset for disagreement. \n- Evaluations across sections draw from a lot of different datasets and models, making it hard to follow where potential confounding issues might stem from (e.g., data or model-specific choices).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "More detailed questions about experimental design:\n- Section 2: How do you separate “style” (in taxonomy) from noise? Do annotators have the right background to understand and correctly determine the cause for diverging preferences?\n- Section 3: Can you show Figure 2 for the other models, as this is only Bradley Terry? To what extent are these trends dataset-specific? The text doesn’t discuss that High-Agreement Ties do not follow the same trend for Figure 2.\n- Section 4: Can you provide error bars for Table 3? Preference accuracy is not that different between single-value vs distributional reward models.\n- Section 5: The AUROC of the KL model trained was still relatively low, which means that it is a very imperfect classifier to use on WildBench data. Were any checks done to verify that the model is useful? The authors report results over 50 examples, do these results generalize more broadly?", "relation_to_broader_scientific_literature": "There is increasing work on pluralistic alignment. This work aims to shed light on potential issues that may arise when performing reward modeling or using LLM-as-a-judge.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please address aforementioned experimental design questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecXxNdy3Tf", "reviewer": "Reviewer_QJAM", "summary": "This paper investigates when and why human annotators disagree, identifying 4 broad sources of preference divergence (task underspecification, response style, refusals, errors) covering >30% of responses in RLHF datasets. They then explore how divergent preferences impact LLM training (with reward modelling) and evaluation (with LLM-as-Judge), showing that BT and LLM-as-Judge paradigms both fail to distinguish between instances of unanimous agreement and majority opinion. Arguing that divergent preferences are useful training signals (not undesirable noise), the authors develop a distributional reward modelling technique to utilise all preferences during training to model a distributional reward, and improve LLMs’ abilities to model human disagreements. Lastly, they point out deficiencies in the LLM-as-Judge evaluation pipeline, which punishes pluralistically aligned models by always expecting the modal responses (even when humans disagree), with consequences of penalising safety guardrails and clarification requests on underspecified prompts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and suggestions!\n\n**Q1: The authors might consider engaging further with classic literature on social choice and voting theory.**\n\nThank you for this suggestion! We agree that Arrow's impossibility theorem and social choice theory is relevant to our work, particularly to the discussions in Section 5 on pluralistic alignment and different practitioners' decisions in comply/refuse disagreements. In our revisions, we will add include such works in these discussions\n\n**Q2: Do the datasets also provide demographic attributes of the annotators? If so, are disagreements (variance in opinions) more pronounced inter (between) or intra (within) demographic groups?**\nWe include annotator details in Appendix C. To summarize from there, these datasets do not include demographic information or annotator IDs due to privacy concerns. We are therefore unable to conduct such an analysis.\n\n**Q3: How does distributional reward modelling compare to 1) training only on the 70% of non-diverging opinions, and to 2) in-context learning or context-aware techniques?**\nWe do not perform the first experiment; however, we do not expect it to have significant differences in Bradley-Terry or MSE-Regression reward models when trained with all or aggregated annotations. Training on all annotations, in particular, is perhaps closest to this setting, as the diverging and non-diverging examples are differentiated during training.\nRegarding your second question, our goals with our LM-as-Judge experiments were intended to assess the behavior of existing, standard LM-as-Judge benchmarks. We do not experiment with different prompting techniques to improve the LM-Judge’s predictions, but agree that this is an exciting area for future work and will add discussion for it along with our other results and recommendations (Section 5.3) in our revisions.\n\n---\n\nThank you again for your suggestions, and please let us know if you have any remaining suggestions or questions.", "claims_and_evidence": "This paper presents 3 central claims: 1) human annotators meaningfully disagree in RLHF datasets such that these divergences are useful training signals (not noise), 2) BT reward modelling fails to differentiate between unanimous versus majority rule and falls short of pluralistic alignment, 3) LLM-as-Judge evaluation techniques suffer similar pitfalls and penalise pluralistically aligned LLMs with safety guardrails and which know to ask for clarification on underspecified prompts. All claims are well evidenced with experiments, concrete data examples, thorough analysis, both numerical and qualitative results.", "methods_and_evaluation_criteria": "The training proposal is evaluated on extended versions of MultiPrefDisagreements and HelpSteer2-Disagreements datasets, for a variety of frontier LLMs. Evaluations are thorough with results reported for both preference accuracy and diverging ID AUROC metrics, which respectively capture alignment to the modal preference and pluralistic alignment. LLM-as-Judge evaluation is benchmarked on ChatbotArena (Arena-Hard) and is shown to unfairly penalise pluralistically aligned models when given a divisive example. The method and evaluation procedures are sound and reasonable.", "theoretical_claims": "N/A: This paper does not make theoretical claims.", "experimental_designs_or_analyses": "Training stage comparisons between the proposed Mean-Var Reward Models (KL), scalar-valued reward modelling (BT, MSE-Regression, Skywork, 70B-Reward) and Mean-Var Baseline (NLL, Independent) are thorough, well-documented and well-structured. Analysis is granular and discusses in detail the different sources, types and degrees of disagreement; this closely matches the proposed method, which does not assume independence of judgement and models the mean and variance of the preference distribution to capture the shape/spread (strength and spectrum) of opinions. Discussion on the limitations of LLM-as-Judge as similarly nuanced and reveal future directions for improvement.", "relation_to_broader_scientific_literature": "1. **Important dataset releases (of individual preferences) -** The authors collaborated with creators of MultiPrefDisagreements and HelpSteer2-Disagreements to release the individual annotator preferences (before aggregation) to enable further work on training and evaluation under human disagreements.  \n2. **Significant and relevant -** This submission relates to frontier directions on modelling the preferences of different users and adapting LLM responses to be more helpful, user-aligned and target-specific.  \n3. **Important implications for benchmarking -** This paper confirms that mainstream LLM-as-Judge evaluation pipelines are mismatched with many desiderata (e.g. pluralistic alignment, AI safety, clarity of thought) of LLMs, and will hopefully motivate further research and advances in robust evaluation.", "essential_references_not_discussed": "The related work section is thorough and satisfactorily engages with relevant work.", "other_strengths_and_weaknesses": "To summarise the above, this paper is a worthwhile contribution that will hopefully pave the way for further work in pluralistic alignment. It is significant (with dataset and method contributions), clearly presented and soundly argued. While the idea that annotators disagree, or that reward modelling does not capture the full spectrum of human perspectives has been explored before, the existence of prior intuition does not detract from the merits of this work since it ventures beyond mere contemplation to contribute an original distributional reward modelling approach for LLMs. One possible weakness of this work is the lack of formal investigation of how BT or LLM-as-Judge collapses to the modal opinion, though this is likely beyond the scope of this article.", "other_comments_or_suggestions": "The following are more suited as comments or out-of-interest questions; discussing them could lead to a score increase but not addressing them fully will not lead to a score decrease:  \n- The authors might consider engaging further with classic literature on social choice and voting theory, e.g. Arrow's impossibility theorem and the inability to simultaneously satisfy intuitive desiderata when attempting to model a spectrum of preferences with a social utility function.   \n- Do the datasets also provide demographic attributes of the annotators? If so, are disagreements (variance in opinions) more pronounced inter (between) or intra (within) demographic groups?\n- How does distributional reward modelling compare to 1) training only on the 70% of non-diverging opinions, and to 2) in-context learning or context-aware techniques?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines annotator disagreement in preference datasets used in large language model (LLM) development. It introduces a taxonomy of disagreement sources, evaluates how existing reward models and evaluation pipelines (such as LLM-as-Judge) fail to account for disagreement, and proposes a distributional reward modeling method that captures diverging preferences. The authors also release individual-level annotations for existing datasets and provide tools to identify divisive examples during training and evaluation.

Overall, the feedback from reviewers was positive. On the one hand, reviewers appreciated the paper’s focus on disagreement, the release of individual annotations, and the modeling approach tailored to pluralistic alignment. On the the other hand, reviewers pointed out issues in clarity and cohension. In particular, BbMo highlighted that the paper reads like several standalone sections, and questioned the value of the analysis. the proposed models achieve modest performance gains and rely on diverse datasets that make it difficult to track confounding variables.

Having read the reviews, the rebuttal, and the paper, I am recommending accept at this time – though I note that the decision is borderline. In this case, my recommendation is based on the fact that the paper makes a meaningful contribution by nature of the topic that it studies. The work highlighting the role of pluralistic disagreement and providing tools for identifying divisive examples. Looking forward, I believe that a potential camera-ready version should address two key issues (raised by BbMo)to ensure its contributions will be valuable to a growing stream work in pluralistic alignment. These includeL

1. Cohesion. The taxonomy, modeling, and evaluation sections feel disconnected. The final version should include a concise contributions list and a stronger narrative linking these parts into a unified story.

2. Confounding factors in experiments. Evaluations draw from many datasets and models. Add a discussion of dataset/model-specific effects and make clear which results generalize across settings.
Decision: Accept (poster)