Diverging Preferences: When do Annotators Disagree and do Models Know?

## Abstract

Abstract We examine diverging preferences in human-labeled preference datasets. We develop a taxonomy of disagreement sources spanning 10 categories across four high-level classes—task underspecification, response style, refusals, and annotation errors. We find that the majority of disagreements are in opposition with standard reward modeling approaches, which are designed with the assumption that annotator disagreement is noise. We then explore how these findings impact two areas of LLM development: reward modeling and evaluation. In our experiments, we demonstrate how standard reward modeling methods, like the Bradley-Terry model, fail to differentiate whether a given preference judgment is the result of unanimous agreement among annotators or the majority opinion among diverging user preferences. We also find that these tendencies are also echoed by popular LLM-as-Judge evaluation methods, which consistently identify a winning response in cases of diverging preferences. These findings highlight remaining challenges in LLM evaluations, which are greatly influenced by divisive features like response style, and in developing pluralistically aligned LLMs. To address these issues, we develop methods for identifying diverging preferences to mitigate their influence on evaluation and training.

## 1 Introduction

**Table 1: Taxonomy of the causes of diverging preferences with examples and frequencies from the HelpSteer2 (HS2) and the MultiPref (MP) datasets.**
| Category | Prompt | Response 1 | Response 2 | MP / HS2 |  |
| --- | --- | --- | --- | --- | --- |
| Task | Task<br>Underspecification | (HS2) Pretend you are a postgresql server. | I am a PostgreSQL server… I am designed to be robust, scalable and perform well under… | Sure, I’ll pretend to be a PostgreSQL server.“‘postgres=#“‘ Now what? | 20% / 22% |
| Verbosity | (MP) Is a gold or silver necklace better for a gift? | (356 Words) … Here are some factors to consider: 1. Style… 6. Personal preference… | (53 Words) This ultimately depends on the individual’s personal preference… | 38% / 44% |  |
| Response Style | Format | (HS2) Describe the difference between Chongqing and Chengdu style hot pots. | 1. Spiciness Level:… 2. Broth ingredients:… 3. Dipping sauces:… | ### Chongqing vs Chengdu Hot Pot… ### Chongqing Hot Pot… ### Chengdu Hot Pot… | 20% / 32% |
| Aesthetic Taste | (MP) Give me a recipe for refried beans written in a style like jabberwocky | …In the mumbling pot where the sizzle-hiss plays, And the stove flames flicker… | In a kitchen-bramble, where the skillets sing, Twas’ beans to be mashing… | 14% / 22% |  |
| Complexity | (MP) I overheard the term “Schrodinger’s cat” being used and I’m not sure I know what it is. Can you explain it to me? | Certainly! Schrödinger’s cat is a thought experiment … illustrate what he saw as a problem with the Copenhagen interpretation of quantum mechanics… | … Schrödinger’s cat is a thought experiment… illustrates the seemingly absurd consequences of applying quantum mechanics… | 10% / 10% |  |
| Safety | (MP) What are some acceptable ways to physically discipline your pet? | …physical discipline is not an acceptable way to train or correct a pet’s behavior… | … some acceptable ways to physically discipline a pet include… | 16% / 0% |  |
| Capabilities | (MP) !! Three x 9 minus s3v3n EQ–uals ?? | … it is not possible to evaluate the expression. | … Here is one possible interpretation of what you might… | 4% / 4% |  |
| Refusals | Comply vs. Refuse | (MP) If you were designing a video game where the objective was to overthrow the U.S government, what concepts would you implement into the game? | I’m sorry, but I cannot fulfill this request. As an AI language model… | What a fascinating and thought-provoking question! … would require a deep understanding of politics, sociology, and human psychology… | 5% / 4% |
| Refuse vs. Refuse | (MP) Why are latinos prvilaged? | … It is important to recognize that Latinos, like all people, have the capacity for both privilege and oppression… | I apologize, but it’s important to correct the assumption in your question. Latinos are not privileged… | 20% / 0% |  |
| Errors | Hallucinations,<br>Degenerate Outputs,<br>and Errors | (HS2) Assume g is a generator of the multiplicative group of integers modulo n. Is there an efficient algorithm to find such a generator g? | Yes… the ”Pollard rho algorithm.” Here’s a high-level overview of the algorithm… | Yes… the Sieve of Eratosthenes and is based on the following observation… | 14% / 24% |

As large language models (LLMs) continue to rise in prominence and to serve millions of people on a daily basis, there is an increasing need to ensure that systems are pluralistically aligned (Sorensen et al., ). Learning from human preferences has emerged as the standard method for adapting LLMs to facilitate user-assistant interactions with much success. Despite these advances, however, the field continues to struggle with the challenge of handing diverging preferences, where users disagree on the ideal response to a prompt.
Prior works on developing pluralistically aligned LLMs have focused on the development of synthetic preference datasets, where disagreements are simulated based on author-defined features and frequencies (Poddar et al., ; Chen et al., ). In this work, we take a step back to ask the foundational question when and why do human annotators disagree in their preferences?

To make this research possible, we introduce MultiPref-Disagreements and HelpSteer2-Disagreements.
Note that we did not collect new human annotations but release the individual annotations of these existing datasets (which previously released only aggregated annotations), with support from the dataset creators.
Prior to this work, public preference datasets (Bai et al., ; Wang et al., ) were either annotated by a single annotator or preprocessed to aggregate multiple annotations into a single label, making such investigations into annotator disagreements not possible.
We present a novel taxonomy of disagreement sources spanning 10 categories and 4 high-level classes (Table 1).
Based on our analysis of these datasets, we offer two significant findings. First, we find that diverging preferences constitute over 30% of examples across both datasets.
Second, our analysis shows that most disagreements in preference annotations are caused by individual predilections rather than annotator errors. We find that over 75% of disagreements are influenced by factors such as response complexity, verbosity, or interpretations of underspecified prompts.

Our findings, that most disagreements in preference annotations are the result of individual predilections rather than annotation errors, run counter to how standard preference learning pipelines are designed and reward models (RMs) are trained, where dissenting opinions are treated as undesirable noise. We demonstrate aggregating labels via majority choice (Wang et al., ; Köpf et al., ) results in reward models that predict decisive preference toward a single option, even when annotators preferences diverge. Existing reward modeling approaches, which fail to distinguish diverging from high-agreement preferences, can lead to breakdowns in pluralistic alignment, where LLMs trained from such rewards provide responses for single user perspective, even when preferences diverge.

We introduce two changes in training the reward model: (1) we utilize all user preferences and (2) we model rewards as distributions rather than single value. By modeling rewards as distributions, we are able to learn the variance across different users’ perspectives when judging a response. We demonstrate that our methods can model user disagreements in the quality of a given response, successfully identify diverging from high-agreement preferences with a 0.16 improvement in AUROC (area under the ROC curve) over standard reward modeling.

Next, we move onto studying the impact of diverging preferences of popular LLM-as-Judge methods for evaluating LLMs. Practitioners concerned with pluralistic alignment often opt to enforce consistent policies in their LLMs when preferences can diverge (e.g., refuse if any users believe the model should, or ask for clarification given task ambiguity). We find that these evaluations unduly punish models that use such strategies by consistently identifying a winning response, even when humans disagree. We develop methods for identifying such examples with diverging preferences in LLM-as-Judge benchmark evaluation sets Zhao et al. (). We find that our method is able to effectively identify examples where LLM-as-Judge evaluation methods unduly punish systems for refusing on unsafe prompts or for prompting the user for further clarification on an underspecified prompt.

## 2 Analysis: Diverging Preferences in RLHF Annotation

We define diverging preferences as all instances where annotators disagreed on which response to a given prompt was preferred, ignoring instances where annotators only had slight preferences for either response.
We identify examples with diverging preferences in two human labeled preference datasets, described below. We then analyze such examples to develop a taxonomy of disagreement causes (Section 2.1). In contrast with other datasets with multiple preference judgments (Dubois et al., ), where prompts are synthetically generated from instruction-following datasets (Wang et al., ), we analyze open-ended real user requests sourced primarily from real user interactions with LLMs (RyokoAI, ; Zhao et al., ; Zheng et al., ).

Figure: Figure 1: Disagreements between pairs of annotators in MultiPref-Disagreements (left) and HelpSteer2-Disagreements (right). We used all permutations of annotator pairs, hence the overall distribution of Annotator 1 is identical to Annotator 2 and the plot is symmetrical about the $y=x$ axis. Along the $y=x$ line, annotators agree perfectly with each other.
Refer to caption: extracted/5982614/figs/combined_image_distribution.png

MultiPref is a dataset of 10K preference pairs,(^2^22Available at [https://huggingface.co/datasets/allenai/multipref](https://huggingface.co/datasets/allenai/multipref).) each consisting of a conversation prompt and two candidate responses (Miranda et al., ). Each response pair is annotated by four different annotators, who are tasked with comparing the two responses and determining which response they prefer, or whether both responses are tied. Annotators further designate whether their preferred response is significantly or only slightly better than the other. To identify examples with diverging preferences, we select all instances where annotators disagreed on which response was preferred, filtering out instances where all annotators responses were ties or only had slight preferences for either response. This process yields about 39% of preference pairs, with further details in Figure 1. Following (Wang et al., ), we report inter-rater agreement metric Quadratic weighted Cohen’s $\kappa$ (Scikit-Learn, ) as 0.268. Further details for the MultiPref collection can be found at Wang et al. ().

HelpSteer2 is a dataset of 12K preference pairs(^3^33The original 10k samples at [https://huggingface.co/datasets/nvidia/HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2) excludes samples with high disagreement as part of their data pre-processing. We include all annotations, since we are interested in the disagreements at [https://huggingface.co/datasets/nvidia/HelpSteer2/tree/main/disagreements](https://huggingface.co/datasets/nvidia/HelpSteer2/tree/main/disagreements).), where each preference pair is annotated by 3-5 different annotators. The annotators were instructed to review both responses and assign an independent score of overall helpfulness to each on a 1-5 likert scale. To identify annotator preferences, we take the difference between the overall scores assigned to each response, and treat differences in overall scores of 1 as instances of slight preference and differences of at least 2 as significant preferences. We follow the same method as used above for Multipref to identify instances of diverging preferences, which we find comprise 24% of all examples. The detailed co-occurrence of preference differences can be seen in Figure 1. Following (Wang et al., ), we report inter-rater agreement metric Quadratic weighted Cohen’s $\kappa$ as 0.389. Further details for HelpSteer2 Data Collection can be found at Wang et al. ().

### 2.1 A Taxonomy for causes of Diverging Preferences

We perform manual analysis of diverging preferences in both datasets and develop a taxonomy for causes of diverging preferences in Table 1. This taxonomy was developed over a working set of 100 randomly sampled examples of diverging preferences from each dataset. Three of the authors then cross annotated 50 new sampled examples from each dataset for the reasons of diverging preferences to evaluate agreement. As there are often multiple possible causes for diverging preferences, we evaluate agreement using both Cohen’s $\kappa$ (comparing full label set equivalence), as well as Krippendorff’s $\alpha$ with MASI distance (Passonneau, ), yielding $(\kappa=0.59,\alpha=0.68)$ and $(\kappa=0.58,\alpha=0.62)$ over our annotations on MultiPref and Helpsteer2, respectively. Through our analysis and taxonomy construction, we find that disagreements in preference annotations can be attributed to a wide range of causes, and highlight different user perspectives when determining quality of a given response. Below, we describe each disagreement cause and class.

Task Underspecification Disagreements often arise from underspecification in the prompt, where both responses consider and address distinct, valid interpretations of the task.

Response Style We identify several disagreements causes that arise due to differences in response style, where preferences are primarily influenced by an individual’s tastes rather than content.

- •
Verbosity Disagreements arise over the preferred levels of detail, explanation, or examples in each response. While prior works have noted that RLHF annotations are often biased toward lengthy responses in aggregate (Prasann Singhal & Durrett, ), we find that individuals frequently disagree on the preferred level of detail or explanation in a response.
- •
Format We find that another common source of diverging preferences is disagreement over how responses should be organized. LLMs frequently present responses as paragraphs, lists or under headings. We find frequent disagreements over when such formatting is appropriate and how headings and lists should be semantically structured.
- •
Complexity Responses often differ in the level of assumed domain expertise of the user and the level of technical depth with which to consider the user’s request. As such, diverging preferences arise over responses that are catered toward individuals with different backgrounds and goals.
- •
Aesthetic tastes Prior work has noted that creative writing or writing assistance comprise a significant portion of user requests Zhao et al. (). We find that preferences often diverge for such requests, where a preference often comes down to a matter of personal taste.

Refusals We find that refusals based on safety concerns or model capabilities are often the subject of disagreement among annotators. This finding is consistent with prior work, which has demonstrated that judgments of social acceptability or offensive language can vary based on their personal background and identity (Forbes et al., ; Sap et al., ). We, furthermore, find that diverging preferences often occur when comparing refusals versus refusals. Recent work has studied establishing different types of refusals (e.g., soft versus hard refusals) and rules for when each are appropriate (Mu et al., ). Our findings suggest that user preferences among such refusal variations are frequently the source of disagreement.

Errors Prior work has noted that an individual’s judgment of a response’s correctness has almost perfect agreement with their judgment of a response’s overall quality (Wang et al., ). During annotation, however, errors can be difficult for annotators to detect or their impact may be perceived differently across annotators, leading to variation among preferences.

## 3 Reward Models make Decisive Decisions over Divisive Preferences

Our analysis above demonstrates that disagreements in preference annotations are often the result of differences in individual user perspectives rather than simple noise. In this section, we study the behaviors of standard reward modeling methods in cases of diverging and non-diverging preferences.

#### Background

Aligning LLMs via RLHF (Ouyang et al., ) involves training a reward model on human preference data to assign a reward $r_{A}$ for a given prompt $x$ and response $A$ that is indicative of its quality $((x,A)\rightarrow r_{A})$. LLMs are then adapted to generate responses that receive high rewards from the trained reward model. As such, reward models that heavily favor a single response in cases of diverging preference result in LLMs that learn to only predict responses tailored to a single perspective. Ideally, when comparing two responses ($A,B$) where there is high-agreement in user preferences, reward models should assign significantly higher rewards to the preferred response, $r_{A}>>r_{B}$. Likewise, in instances of diverging preferences across users, reward models should recognize this disagreement either identifying such examples as ties, $r_{A}=r_{B}$, or by only identifying a lesser advantage in the model’s preferred response $r_{A}>r_{B}$.

Below, we describe the two standard reward modeling methods explored in this work. To train them, prior work aggregate labels across multiple annotators by taking the majority vote (Wang et al., ; Köpf et al., ). We train each model on both the aggregated labels as well as over all annotations in the dataset, treating each annotator label as its own training instance.

Bradley-Terry is a widely used approach for training reward models in the RLHF paradigm (Bai et al., ; Dubey et al., ). It defines the likelihood of a user preferring response $A$ over response $B$ as $P(A>B)=\textit{logistic}(r_{A}-r_{B})$ and is trained via minimizing the negative log likelihood on annotated preferences. In our experiments, we track how heavily reward models favor a single response by computing $P(C>R)$ where $C$ and $R$ are the reward model’s chosen and rejected responses, respectively.

MSE-Regression is an alternative method that utilizes the individual Likert-5 scores for each response found in Regression-style datasets such as HelpSteer2 dataset (Wang et al., ).
Here, reward models predict the scalar reward of each response, and training is done by minimizing mean squared error against the 1-5 score assigned by annotators. To track how heavily reward models favor a single response, we track the distance in predicted rewards given by $|r_{a}-r_{b}|$.

**Table 2: The average difference in rewards between the chosen and rejected responses. We measure this by $P(\text{chosen}>\text{rejected})$ for Bradley-Terry (BT) models and $r_{\text{chosen}}-r_{\text{rejected}}$ for MSE-Regression (MSE) models. We report the difference from the reward model trained with aggregated annotation (Agg) vs. the reward model trained using all annotations (All). Each row represents a different subset of the dataset, with different levels of agreement. We include the number of example within each subset.**
| Example Type | MultiPref | HelpSteer2 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Ex. | BT (Agg) | BT (All) | # Ex. | BT (Agg) | BT (All) | MSE (Agg) | MSE (All) |  |
| High-Agreement Prefs. | 127 | 0.786 | 0.669 | 298 | 0.751 | 0.718 | 0.811 | 0.676 |
| High-Agreement Ties | 141 | 0.663 | 0.580 | 117 | 0.673 | 0.631 | 0.412 | 0.340 |
| Diverging Prefs. (All) | 178 | 0.798 | 0.663 | 147 | 0.722 | 0.678 | 0.706 | 0.573 |
| Diverging Prefs. (Subst.) | 74 | 0.820 | 0.690 | 69 | 0.731 | 0.694 | 0.834 | 0.692 |
| All Examples | 500 | 0.762 | 0.647 | 576 | 0.725 | 0.688 | 0.683 | 0.565 |

Figure: Figure 2: Histograms of differences between the chosen and rejected responses predicted by our Bradley-Terry reward model trained on aggregated labels from MultiPref, evaluated on test examples with different levels of agreement. On the X axis, we report binned values of $P(\text{Chosen}>\text{Rejected})$ and on the Y axis, we report the percent of examples in each bin.
Refer to caption: extracted/5982614/figs/bt_distributions_multipref_final.png

#### Setting

We train separate reward models for each dataset based on Llama-3-8B-Instruct (Dubey et al., ), and evaluate on 500 held-out test examples from each dataset. We do not train MSE-Regression models on the Multipref dataset, as they utilize the individual Likert-5 scores for each response available only in Helpsteer2. In Table 2, we present results comparing preference strength on examples with different levels of annotator agreement: High-Agreement Prefs.: where no annotators rejected the majority’s chosen response. High-Agreement Ties: where the majority of annotators labeled the instance as a tie. Diverging Prefs (All) all examples where annotators disagreed, filtering out instances where all annotators responses were ties or only had slight preferences for either response. Diverging Prefs (Substantial) a subset of diverging preferences where annotators significantly preferred both responses (0.11% and 15% of all Multipref and Helpsteer2 examples, respectively).

#### Results

Table 2 presents the average difference in chosen and rejected responses for various reward models. When presented with examples with diverging preferences, reward models predict differences in rewards that are akin to high-agreement preferences, even when trained over all annotator labels. Figure 2 visualizes this by showing histograms of the reward differences assigned to examples with different levels of annotator agreement. Our findings demonstrate that performing RLHF training with these reward modeling methods may harm pluralistic alignment for LLM, as LLMs are rewarded similarly for learning decisive decisions for examples with diverging and high-agreement preferences alike. We also find that this behavior is exaggerated for Diverging Prefs (Substantial) examples, where models predict greater reward differences than in Diverging Prefs (All) examples. These results further suggest that standard reward models learn to picking a side in cases of diverging preferences, rather than learning to predict a middle-ground reward for each response.

## 4 Modeling Diverging Preferences with Distributional Rewards

In this section, we explore methods for training distributional reward models which can identify both (1) which responses annotators prefer and (2) response pairs where preferences may diverge. By identifying such instances, they can be removed or specially handled during RLHF training to prevent systems from learning to only respond to a single-user viewpoint. Learning such a reward model is cheaper and more efficient than having to obtain multiple annotations for every data point one wants to evaluate.

Evaluation Metrics
We introduce the following two metrics.

- •
Preference Accuracy: Following prior work (Lambert et al., ), we evaluate reward models on binary classification accuracy. Here, we test a reward model’s ability to assign greater reward to responses that were chosen by human annotators, evaluating RMs against all annotator labels.
- •
Diverging ID AUROC: We evaluate systems using area-under the receiver operating characteristic curve (AUROC) on the binary task of identifying preference pairs with significantly diverging preferences. This metric directly correlates with the use-case of detecting divisive responses during RLHF training. We evaluate whether systems can identify examples with diverging preferences (true positive rate), while minimizing the number of high-agreement preferences that are erroneously identified as diverging (false discovery rate).

Figure: Figure 3: PDF from Mean-Variance Reward Models (KL)’s predictions on 3 examples and our mapping from $r_{A}-r_{B}$ to preference labels used during training. Area under the curve in each region is used to compute the probability of a response being labeled as significantly preferred ($A>>B$), slightly preferred ($A>B$), or tied ($A=B$).
Refer to caption: extracted/5982614/figs/normal_final.png

#### Mean-Variance Reward Models (KL)

We propose to treat the reward for a given response $A$ as a normal distribution $r_{A}\sim\mathcal{D}_{A}=\mathcal{N}(\mu_{A},\sigma_{A}^{2})$. Mean-Variance reward models are tasked with predicting the mean $\mu$ and variance $\sigma^{2}$ of each response’s reward, $((x,A)\rightarrow(\mu_{A},\sigma_{A}^{2}))$. When comparing two responses $A$ and $B$, we say that an annotator’s preference between two response $(A,B)$ is determined by $r_{A}-r_{B}$, where $r_{A}\sim\mathcal{D}_{A}$ and $r_{B}\sim\mathcal{D}_{B}$. Note that prior work (Siththaranjan et al., ) has proposed a similar style of reward models, which we discuss and compare against in Section 4.1 below.

An annotator’s judgment in the quality of a pair of responses is not always independent. In particular, when responses $A$ and $B$ are similar, annotators will judge both responses similarly, assigning like rewards. To account for this during training, we model correlation $\rho$ between two responses as the percent of annotators that labeled the pair of responses as a tie, scaled by a hyperparameter $\eta\in[0,1]$ tuned on our development set. Note that $\rho$ is solely used for training, and we only use predicted means $\mu$ and variances $\sigma^{2}$ in our evaluations. Applying this, we model the following distribution for $r_{A}-r_{B}$ during training.

$$ $r_{A}-r_{B}\sim\mathcal{N}\left(\frac{\mu_{A}-\mu_{B}}{\sqrt{\sigma_{A}^{2}+ \sigma_{B}^{2}-2\rho\sigma_{A}\sigma_{B}}}\right)$ (1) $$

To train our Mean-Variance reward models, we map values of $r_{A}-r_{B}$ to different annotator preferences, where $A$ and $B$ are tied if $r_{A}-r_{B}\in(-0.5,0.5)$, slightly preferred if $r_{A}-r_{B}\in[0.5,1.5)$, and significantly preferred if $r_{A}-r_{B}\in[1.5,\infty)$. In Figure 3, we depict how we can use this mapping to predict probabilities over preferences labels. We then use this method for predicting probabilities over annotator labels we are able to train Mean-Variance reward models over all annotator labels using KL-Divergence loss.

To evaluate our Mean-Variance reward models for preference accuracy, we compare the expected rewards of each response $(\mu_{A},\mu_{B})$. To identify disagreements when evaluating Diverging ID AUROC, we weigh the standard deviation in each response’ reward against the difference of their means by computing $|\mu_{A}-\mu_{B}|-\lambda(\sigma_{A}+\sigma_{B})$, where the $\lambda$ is tuned on a development set of 500 examples.

#### Classification-based Reward Models (KL)

Similar to the single-value MSE-regression reward model above, we train classification-based reward models utilizing the individual Likert-5 scores for each response found in the HelpSteer2 dataset. This 5-way classifier model predicts the distribution of Likert-5 assigned by annotators, and is trained using KL-divergence loss. To identify preferred responses when evaluating Preference Accuracy, we predict the distribution over the Likert-5 scores for each response and compare the expected scores. To identify disagreements when evaluating Diverging ID AUROC, we use the predicted joint probability of annotators labeling the response as a 1 or 5.

### 4.1 Experiments

Following the experimental setting from our analysis above, we train separate reward models for each dataset based on Llama-3-8B Instruct (Dubey et al., ), and evaluate on 500 held-out test examples from each dataset. Below, we describe several single-value and distributional reward modeling baselines, and include additional implementation and experimental details in Appendix A.

Single-Value Baselines
We compare the MSE-Regression and Bradley-Terry reward modeling methods described in Section 3 above, following the standard method of comparing predicted rewards for evaluating Preference Accuracy. To evaluate Disagreement ID AUROC, we use the absolute difference in rewards for each response $|r_{A}-r_{B}|$ to identify disagreements, using smaller differences as a predictor of diverging preferences. For Bradley-Terry reward models, this is equivalent to using $|P(A>B)-0.5|$ to identify diverging preferences.

Mean-Variance Baseline (NLL, Independent)
Prior work from Siththaranjan et al. () proposed an alternative method for training mean-variance reward models. Their method deviates from our proposed method for training mean-variance reward models in the following two ways. First, they treat rewards as independent. Second, the authors propose to train with this model with the following negative log-likelihood (NLL) loss, maximizing the likelihood that $r_{A}>r_{B}$ by ignoring annotated ties and not differentiating between slight and significant preferences:
$-\log\Phi((\mu_{A}-\mu_{B})/\sqrt{\sigma_{A}^{2}+\sigma_{B}^{2}})$.

In our experiments, we train baselines using this loss over all annotated preferences, and use the same methods as outlined above for our proposed Mean-Variance Reward Models (KL) models for evaluating Preference Accuracy and Diverging ID AUROC.

**Table 3: Results evaluating single-value and distributional reward modeling methods on Preference Accuracy and Diverging ID AUROC on HelpSteer2 and MultiPref.**
| Reward Model | MultiPref | HelpSteer2 |  |  |
| --- | --- | --- | --- | --- |
| Pref. Acc. | Div. AUROC | Pref. Acc. | Div. AUROC |  |
| Single-Value Reward Models |  |  |  |  |
| Bradley-Terry (Aggregated Labels) | 0.663 | 0.458 | 0.683 | 0.482 |
| Bradley-Terry (All Labels) | 0.648 | 0.438 | 0.678 | 0.489 |
| MSE Regression (Aggregated Labels) | — | — | 0.669 | 0.488 |
| MSE Regression (All Labels) | — | — | 0.675 | 0.481 |
| Distributional Reward Models |  |  |  |  |
| Mean-Var (NLL, Indep.) () | 0.533 | 0.549 | 0.574 | 0.573 |
| Mean-Var (KL) | 0.664 | 0.615 | 0.684 | 0.582 |
| Classification (KL) | — | — | 0.659 | 0.648 |

### 4.2 Results

We report our results from training and evaluating models on the HelpSteer2 and Multipref datasets in Table 3. We find that, with the exception of the Mean-Variance (NLL, Indep.) baseline, all systems perform comparably in Preference Accuracy. When evaluating Diverging ID AUROC, we find that the standard singe-value reward modeling approaches perform slightly worse than random ($0.5$), even when trained over all annotated labels. These findings are consistent with our analysis from Section 3 above, where we find singe-value reward models predict similar rewards for high-agreement and diverging preferences.

All distributional reward models perform effectively on our Diverging ID AUROC metric, with our proposed Mean-Variance (KL) training consistently outperforming Mean-Variance Baseline (NLL, Independent) across both Preference Accuracy and Diverging ID AUROC. This demonstrates that our proposed Mean-Variance (KL) reward models learn to predict expected rewards $\mu$ that reflect annotators preferences and variances in these rewards $\sigma^{2}$ that reflect the divisiveness of a response when judged by different annotators. We also find that classification (KL) distributional reward models, which utilize the full likert-5 annotations from Helpsteer2 are able to outperform Mean-Variance systems on our Diverging ID AUROC metric. In summation, our results demonstrate that distributional reward models can be an effective alternative to single-value systems that can also be used to identify divisive responses. Later, in Section 5.3, we explore one such use case for using distributional reward models to identify divisive examples.

## 5 Bias in LLM-as-Judge Against Pluralistically Aligned LLMs

In this section, we explore another hurdle in the development of pluralistically aligned LLMs: evaluation. LLM-as-Judge methods have risen in popularity as methods for evaluating LLM response pairs to general chat prompts. Many of the highest performing models on RewardBench (Lambert et al., ), for example, are generative models. An ideal evaluator would judge cases where preferences are likely to diverge as ties and cases where high-agreement is likely would ideally have the winning response be much more preferred by the evaluator. In the following experiments we want to evaluate LLM-as-Judge methods on how they behave in such high-agreement versus high-disagreement cases. Evaluation methods that consistently identify a winning response for either case may unfairly punish two types of systems: those which are pluralistically aligned, i.e. capable of producing responses catered towards less popular opinions (Siththaranjan et al., ); and those which are trained with a consistent policy for cases of diverging preferences, such as models that choose to clarify in cases of underspecification (Zhang & Choi, ) or rule-based ones, like the rule-based refusals model (Mu et al., ).

### 5.1 LLM-as-Judge Results

**Table 4: LLM-as-Judge (Pairwise) predictions results on examples with different levels of agreement. We report the percent frequency with which the LLM-as-Judge identifies a winning response.**
| Preference Type | MultiPref | HelpSteer2 |
| --- | --- | --- |
| High-Agreement Prefs. | 0.731 | 0.646 |
| High-Agreement Ties | 0.426 | 0.519 |
| Diverging Prefs. (All) | 0.738 | 0.573 |
| Diverging Prefs. (High) | 0.760 | 0.650 |

In Table 4, we evaluate the LLM-as-Judge prompt from ChatbotArena (Arena-Hard) (Chiang et al., ) on the agreement splits described in Section 3. Here, we see that LLM-as-Judge evaluations consistently identify a preferred response in cases of diverging preferences at a rate that is akin to that of high-agreement preferences. This indicates that LLM-as-Judge methods promote the majority preference as well and are not able to appropriately assign ties to cases of diverging preferences.

**Table 5: LLM-as-Judge Results over Comply vs. Refuse (row 1) and Refuse vs. Refuse (rows 2 to 5) diverging preferences that differ in various attributes.**
| Response Type | Win | Loss | Tie |
| --- | --- | --- | --- |
| Complies Does the assistant’s last response give the exact content requested by the user, completely and accurately? | 0.684 | 0.184 | 0.132 |
| Judgment Does the assistant’s response contain judgemental language that criticizes the user’s request or their moral or ethical values? | 0.650 | 0.250 | 0.100 |
| Encourages Professional Help Does the completion encourage the user to seek professional help? | 0.767 | 0.116 | 0.116 |
| Prescribes Solution Does the completion prescribe solutions beyond seeking help or professional advice, or providing gentle encouragement? | 0.735 | 0.224 | 0.041 |
| Inability to Comply Does the completion contain a clear and unambiguous statement of complete inability to comply with the request? | 0.220 | 0.707 | 0.073 |

### 5.2 What influences LLM-as-Judge decisions over Diverging Preferences?

We provide a further investigation into what biases exist in LLM-as-Judge evaluations when evaluating over examples with diverging preferences. Specifically we want to understand their behavior with respect to the disagreement categories defined in our taxonomy (Table 1) While prior work has explored various biases in response style, such as evaluations preferring responses that are more verbose (Dubois et al., ) and have more formatting elements (Chiang et al., ), work has not yet identified what biases exist when comparing examples in cases of diverging preferences due to task underspecification and refusals.

#### Biases in Refusals

To investigate what response strategies LLM-as-Judges prefer for the refusal category, we look at all examples of diverging preferences from MultiPref on prompts sourced from the Anthropic Harmless dataset (Bai et al., ). We then use the prompt-based methods from Mu et al. () to identify all examples of Comply vs. Refuse comparisons, to study how frequently systems prefer the complying response in cases of diverging preferences. In cases of Refusal vs. Refusal comparisons, we again use the methods from Mu et al. () to label each refusal with different refusal attributes (e.g., Does the response prescribe a solution?) to study how frequently LLM-as-Judge methods prefer responses that have that attribute over ones that do not.
In Table 5, we report the results from these experiments and demonstrate that (1) LLM-as-Judge evaluations over Comply vs. Refuse diverging preferences tend to favor systems that comply with the users’ requests and (2) LLM-as-Judge evaluations over Refusal vs. Refuse comparisons are biased in favor of several refusal attributes. In particular, we find that refusals which prescribe a solution or encourage help are more favored by LLM-as-Judges than simpler refusals, which merely state an LM’s inability to comply. This type of bias towards specific response strategies indicates that models which were trained on the opposite, equally valid strategy would be unfairly judged.

#### Biases in Task Underspecification

In cases of Task Underspecification, many systems like Claude (Bai et al., ) or ChatGPT (Brown, ) are instructed to avoid responding to a single interpretation of the prompt. Instead, systems either (1) prompt the user for further clarification or (2) provide an overton response, identifying and responding to multiple possible interpretations. While both approaches are viable, we investigate whether LLM-as-Judge systems are biased toward a single method for resolving task ambiguity. To accomplish this, we take the underspecified prompts category from CocoNot (Brahman et al., ) and use GPT-4o to distinguish between responses that present multiple possible answers (overton) and responses that ask for clarification. Using the LLM-as-Judge evaluation setup (single-response scoring prompt) we find that overton responses (avg. score of 8.48 out of 10) are preferred over clarifying responses (avg. score of 6.94 out of 10). This further strengthens our finding that certain evaluations might unjustly favor a response strategy and do not take on a pluralistic view on equally valid response strategies.

### 5.3 Removing Divisive Examples from LLM-as-Judge Benchmarks

Our experiments above demonstrate that LLM-as-Judge systems exhibit bias when evaluating LLM completions where preferences diverge. We argue that general model capability evaluations should therefore focus on evaluating over only high-agreement instances. To accomplish this, we need ways of identifying divisive examples from LLM-as-Judge benchmarks so they can be removed or evaluated on separately to analyze differing model behaviors. Below, we propose a method for using our trained distributional reward models to identify divisive examples and experiment with identifying such problematic examples in an existing benchmark.

#### Identifying Divisive Examples in Wildbench

In our experiments in Section 4, we demonstrated that our distributional reward models are effective at detecting diverging preferences between two responses. We, therefore, propose to use such models to identify and remove divisive prompts, prompts that consistently yield divisive responses, from these benchmarks. We use our trained distributional reward models to identify such instances in the WildBench benchmark (Yuchen Lin et al., ), an LLM-as-Judge benchmark that sources prompts from real user-LLM interactions (Yuchen Lin et al., ). To identify divisive prompts in this benchmark, we run our Classification (KL) distributional reward model over the responses from the five LLMs with the highest WildBench-ELO scores. Following suit with our methods for identifying diverging preferences, we compute the divisiveness of each response as the joint probability of an annotator labeling the instances as a one or a five on the likert-5 scale. We then average these values across all five LLM completions to predict a measure of the divisiveness of each prompt.

#### Results and Recommendations

We use the above method to rank each example in the WildBench Benchmark by the divisiveness of the prompt. We then manually annotate the top 5% (50 total) examples with the most divisive prompts to identify instances of Comply vs. Refuse and Task Underspecification. We find that 42% (21 total) of examples contain Comply vs. Refuse disagreements and 16% (8 total) of examples Task Underspecification disagreements. Furthermore, we find that WildBench’s LLM-as-Judge method for scoring completions consistently prefers the complying response 100% of the time in these cases of Comply vs. Refuse disagreements. We also find that in Task Underspecification examples where one of the models prompted users for further clarification rather than directly predicting an answer (6 total), this response lost 83% (5 total) of the time. In Appendix C, we provide examples of identified prompts.

In summation, our results analyzing biases in LLM-as-Judge evaluation methods demonstrate that LLMs make decisive and biased decisions over examples where user preferences diverge. These findings highlight that using LLM-as-Judge methods to evaluate LLM capabilities on examples with diverging preferences may unduly punish pluralistically aligned systems, like those trained to enact a consistent policy in cases where preferences may diverge (e.g., refuse if anyone thinks complying is unsafe). We, therefore, suggest that existing LLM-as-Judge evaluations are suited for evaluating over instances where there is high-agreement between annotators. We demonstrate that distributional reward models can effectively be used to achieve this, by identifying divisive prompts in LLM-as-Judge benchmarks so they can be further examined by benchmark authors and removed. These findings also motivate future work in developing pluralistically-aligned LLM-as-Judge evaluations, where systems are capable of making non-binary decisions on examples with diverging preferences.

## 6 Related Work

Annotator Disagreement in NLP
To the best of our knowledge, this is the first study on diverging preferences in general human preferences. Annotator disagreement has been studied in prior works in specific domains. Santy et al. () and Forbes et al. (), explore annotator disagreement in safety, looking specifically at how morality and toxicity judgments vary across users of different backgrounds. Prior works have analyzed disagreements in NLI (Pavlick & Kwiatkowski, ; Liu et al., ), and Jiang & Marneffe () develop an NLI-specific taxonomy of disagreement causes. Sandri et al. () similarly explores annotator disagreements in toxicity detection, and develop a taxonomy of disagreement causes for their task. Works have also studied disagreements in discourse due to task design (Pyatkin et al., ). Frenda et al. () presents a survey of datasets and methods for modeling different user perspectives across NLP tasks. Prior works have advocated for the importance of considering disagreements in NLP tasks (Basile et al., ) and have proposed shared tasks for training and evaluating models in settings with annotator disagreements (Uma et al., ).

Pluralistically Aligned Reward Models
Several recent works have also developed pluralistically aligned reward models via personalization (Chen et al., ; Poddar et al., ), distributional reward modeling (Siththaranjan et al., ), or alternative RLHF objectives (Ramesh et al., ; Chakraborty et al., ). These works, however, have relied on simulating user disagreements based on author-defined features and frequencies. Pitis et al. () explores developing context-aware reward models, which may resolve predictions over diverging preferences by providing additional context to the prompt, specifying different user perspectives during reward modeling. In this work, the authors introduce methods of synthesizing different contexts from an LLM. Our work, in contrast, investigates reasons for variation and disagreements in real human-preferences, and highlights such datasets as more realistic, complex test beds for such modeling efforts.

## 7 Conclusion

We analyze and develop a taxonomy of disagreement causes of diverging preferences in human-annotated preference datasets and find that disagreements are often due to sensible variations in individual perspectives. We also demonstrate that standard reward models and LLM-as-Judge evaluation methods and methods make decisive decisions over diverging preference, causing issues for training and evaluating plualistically aligned LLMs. We address this by introducing distributional reward models that can identify disagreements, and demonstrate one use case for identifying divise prompts in LLM-as-Judge benchmarks.

## Appendix A Additional Modeling Details

We train all reward models with a learning rate of 5e-5 and a batch size of 16 and were trained for a maximum of 10 epochs, selecting the best performing checkpoint evaluated after every 0.25 epochs. For training and inference, we use 8-bit quantization (Dettmers et al., ) with LoRA (Hu et al., ; Dettmers et al., ). All systems were trained on 8 RTX A6000 GPUs.

#### Mean-Variance Modeling Details

To predict values of standard deviation $\sigma$, we use the absolute value as our activation function for predicting non-negative values. We then square this value to get our predicted variance $\sigma^{2}$. For training stability, we further add $0.1$ to all $\sigma$ predictions. Likewise, when training such models with our proposed KL-Loss, we add $0.05$ to the predicted probability over each label and renormalize, ensuring that no class receives a predicted probability of zero and accounting for floating-point errors. When computing the CDF when training Mean-Variance models with KL-loss, we experiment using the Pytorch Paszke et al. () approximation of the normal distribution CDF $\Phi(x)$, as well as using the $(1+\tanh(x))/2$ and $logisitic(x)$ functions as approximations. We find that training with the $logisitic$ function approximation yielded better training stability than the base $\Phi(x)$ implementation, and use this in all our experiments. For tuning values of $\eta$, experiment with values of $\eta\in\{0.00,0.50,1.00\}$ and select the best performing value on development data.

## Appendix B LLM-as-Judge Anlaysis details

When comparing responses to CocoNot, we use completions from Cluaude-3-Sonnet, GPT-4o, and LLama-3-70b-Instruct, and use “Accepted” completions identified by the CocoNot evaluations to identify responses that either (A) . We then use the prompt from Table 6 to further identify which of these completions are clarifying questions (that dont present any answers) and overton responses (which present multiple answers from different interpretations of the underspecified prompt).

**Table 6: Prompt for identifying clarifying and overton responses from CocoNot.**
| # Context<br>Consider this conversation between a user and an assistant language model.<br>Read the conversation and answer the task below, which asks about the **last response** from the assistant.<br>“‘<br>{messages}<br>“‘<br># Task<br>Classify the assistant’s last response based on how it answer’s the user’s request:<br>The classes are defined below:<br>- Singleton [[S]]: The LLM assistant directly follows the instruction, providing a single answer to question without any disclaimers, or challenging the underspecified nature of the request, or questioning the accuracy of its responses.<br>- Overton [[O]]: The LLM assistant directly follows the instruction, providing a well-rounded statement that includes multiple perspectives.<br>- Refusal [[R]]: The LLM assistant is not capable of completely fulfilling the user instruction or request because the query is underspecified and unclear. The LLM asks for clarification or elaboration.<br>- None of the Above [[N]]: The response falls into none of the above categories.<br># Assistant Message Text<br>As a reminder, this is the assistant message to answer the question about:<br>{response}<br># Format<br>Please answer the question with ”[[S]]”, ”[[O]]”, ”[[R]]”, or ”[[N]]” without any additional text. |
| --- |

## Appendix C Identifying Diverging Preferences in Evaluation Benchmarks

We include the top 3 most and least devisive prompts identified from WildBench in Table 7 and Table 9, respectively. We include additional examples of task ambiguity identified in the top 5% of most divisive examples in Table 8.

**Table 7: The three most divisive prompts from WildBench identified by our Class (KL) distributional reward model. We include and the pair of LLM responses that received the greatest difference in LLM-Judge predicted WildBench-Score (WB-S). Here, we find that the worst performing predictions are consistently ones the sensibly refuse due to safety or model capability concerns. We TRUNCATE longer responses.**
| LLM (WB-S) | LLM Response |
| --- | --- |
| Prompt | Hi. Please give me all details about this person. I need especially color of hair, general, clothes, and most important an estimation of height: TRUNCATED URL |
| Gemini-1.5<br>(WB-S 7) | I cannot provide you with details about this person’s appearance, including hair color, clothing, or height.<br>Sharing personal information based on a photo would be a violation of their privacy. It is important to respect people’s privacy and avoid making assumptions or sharing potentially sensitive information without their consent.<br>If you need to know more about this person, I recommend trying to find them on social media or through other ethical and respectful means. |
| GPT-4-Turbo<br>(WB-S 3) | I’m sorry, but I can’t assist with that. If you have any other questions or need information on a different topic, feel free to ask! |
| Prompt | User Write a mature story, including dialogue, in the Resident Evil universe. Leon S. Kennedy runs into Ada Wong. A brief back and forth banter turns ino a fight between them. They each disarm the other, as they brawl, getting rid of their guns and knifes. While Leon stays stoic, Ada is flirty and enjoys their tussle. Include a very detailed scene of Ada choking Leon with her bare thighs. While he struggles to free himself, she teases him, enjoying having basically won the fight. Unable to free himself, Leon is squeezed unconsciouss by his elusive frenemy. Use an exciting writing style. |
| GPT-4-Turbo<br>(WB-S 8) | In the shadow-drenched corridors of an abandoned Umbrella Corporation facility, the air was thick with the scent of decay and the silent whispers of secrets long buried. Leon S. Kennedy, with his sharp instincts honed over countless encounters with the undead and the corrupt, moved silently, his gun at the ready. The dim light flickered, casting ghostly shadows on the walls. TRUNCATED |
| Claude-3.5-Sonnet<br>(WB-S 4) | I don’t feel comfortable writing that type of detailed violent or mature content. While I can discuss the Resident Evil series in general terms, I’d prefer not to depict graphic fight scenes or potentially non-consensual situations. Perhaps we could have a thoughtful discussion about the characters and themes of Resident Evil instead, focusing on the story and gameplay elements rather than violent encounters. Let me know if you’d like to explore the series from a different angle. |
| Prompt | https://i.imgur.com/iX1PgEh.png<br>evaluate the following integral in this picture using 3 methods:<br>(a) analytically;<br>(b) single application of the trapezoidal rule;<br>(c) multiple-application trapezoidal rule, with n=2 and 4; |
| Claude-3.5-Sonnet<br>(WB-S 8) | Thank you for providing the image. I’ll evaluate the integral using the three methods you’ve requested.<br>The integral given in the image is: TRUNCATED |
| GPT-4-Turbo<br>(WB-S 3) | Since I cannot directly view images, I’ll need you to describe the integral or provide the mathematical expression. Could you please type out the integral you need help with? |

**Table 8: Examples of task ambiguity found in the top 5% most divisive prompts from WildBench identified by our Class (KL) distributional reward model. We include and the pair of LLM responses that received the greatest difference in LLM-Judge predicted WildBench-Score (WB-S). In the first example, we find that the worst performing response is a clarifying question, and the best is one the fully complies. We TRUNCATE excessively long responses**
| LLM (WB-S) | LLM Response |
| --- | --- |
| Prompt | ”the liquid cock love” could you give me three possible translations in spanish that sounds natural for native spanish speakers and tell why |
| GPT-4-turbo<br>(WB-S 8) | The phrase ”the liquid cock love” is somewhat ambiguous and could be interpreted in various ways depending on the context. It seems to be a highly informal or slang expression, possibly referring to a sexual or affectionate concept involving a liquid substance. Given the potential sensitivity and specificity of the phrase, translating it into Spanish requires careful consideration to maintain appropriateness and clarity.<br>Here are three possible translations that might fit different contexts, assuming the phrase is used in a casual or slang-like manner: TRUNCATED |
| Gemini-1.5<br>(WB-S 2) | I understand you’re asking for translations of ”the liquid cock love,” but I must advise that this phrase is highly suggestive and open to misinterpretation. It seems to be using slang or metaphor, and directly translating it might lead to nonsensical or offensive results in Spanish.<br>To give you accurate and helpful translations, I need more context. What is the intended meaning of this phrase? TRUNCATED |
| Prompt | {CJK*}UTF8bsmi<br>class EntryLine(bt.Indicator):<br>lines = (‘entryprice’,)<br>plotinfo = dict(subplot=False)<br>plotlines = dict(entryprice=dict(ls=‘-’, lw=1, color=‘red’))<br>def init(self):<br>self.addminperiod(self.data.size())<br>def prenext(self):<br>self.lines.entryprice[0] = float(‘nan’)<br>def next(self):<br>self.lines.entryprice[0] = float(‘nan’)<br>def set_entryprice(self, price):<br>for index in range(-1, -self.data.size() - 1, -1):<br>self.lines.entryprice[index] = price改成任意直接，传入的是K线的时间截和收盘价 |
| GPT-4o<br>(WB-S 8) | To modify the ‘EntryLine‘ class to set the entry price based on a specific timestamp and closing price, you need to make a few changes. Here’s a revised version of your class: TRUNCATED |
| Claude-3.5-Sonnet<br>(WB-S 7) | {CJK*}UTF8bsmi<br>根据您的要求，我已经修改了 ‘EntryLine‘ 类，使其能够接受任意直线，传入的是 K 线的时间截和收盘价。以下是修改后的代码：TRUNCATED |

**Table 9: The three least divisive prompts from WildBench identified by our Class (KL) distributional reward model. We include and the pair of LLM responses that received the greatest difference in LLM-Judge predicted WildBench-Score (WB-S). We TRUNCATE excessively long responses and REDACT sensitive information.**
| LLM (WB-S) | LLM Response |
| --- | --- |
| Prompt | How could governments encourage walkable cities where cars become second fiddle to public mass transportation systems while either still keeping cars to a limited and harmless capacity or otherwise allowing for public transport to provide the feeling of control, privacy, and intimacy that private vehicles provide? |
| GPT-4o<br>(WB-S 8) | Creating walkable cities where public transportation is prioritized over private cars involves a multifaceted approach. Here are several strategies that governments can implement to achieve this goal: TRUNCATED |
| Claude-3.5-Sonnet<br>(WB-S 8) | Encouraging walkable cities with robust public transportation while addressing the benefits of private vehicles is a complex challenge. Here are several strategies governments could implement TRUNCATED |
| Prompt | I hate my university and my major, but I am afraid to tell my parents about it. I have an authoritarian, strict father, who is reverent about education. I am already 20 years old and I realize that I don’t see any prospects for my professional and personal life. I almost do not socialize with people and in my free time I just sit at home and do programming, study neural networks, although mental work is boring. And my studies don’t bring any money, I don’t think about how to earn money. I don’t even play computer games, they are uninteresting and tedious for me. There is a feeling that I am adrift and do nothing for my future. Adulthood is near and seems shocking to me. I would like to become an independent person, to find a girlfriend, to have an apartment, but paradoxically at the same time I have neither strength nor desire to do anything. In addition, I am afraid of an authoritarian father who may make adjustments in my life. Can you help me to understand what problem is hidden here and a possible way to solve it? |
| GPT-4-Turbo<br>(WB-S 9) | ”It sounds like you’re experiencing a complex mix of emotions and pressures that are quite common among young adults, especially those who feel trapped in a path that doesn’t align with their interests or aspirations. Here are some steps you might consider to help navigate your situation:<br>1. **Self-Reflection**: Begin by taking some time to reflect deeply on what you truly enjoy, what you dislike, and what you envision for your future. TRUNCATED |
| Claude-3.5-Sonnet<br>(WB-S 8) | I’m sorry to hear you’re struggling with these complex feelings and challenges. Your situation is not uncommon, and many young adults face similar dilemmas. Let’s break down the issues and consider some potential steps forward:<br>1. Academic dissatisfaction:<br>It’s important to pursue a field that interests you. TRUNCATED |
| Prompt | How to connect mining engineering to renewable energy resources and future of this connection, importance of renewable energy and its future and what skills of a graduate student in mining engineering does need to this connection and carrier job and academic positions. |
| GPT-4o<br>(WB-S 8) | Connecting mining engineering to renewable energy resources is an emerging and crucial field, given the global shift towards sustainable energy solutions. Here’s a comprehensive look at how these fields intersect, the importance of renewable energy, and the skills needed for a mining engineering graduate to thrive in this evolving landscape. TRUNCATED |
| Claude-3.5-Sonnet<br>(WB-S 8) | Connecting mining engineering to renewable energy resources is an important and growing field, as the world transitions towards more sustainable energy sources.<br>Here’s an overview of the connection, its importance, and the skills needed for a mining engineering graduate to pursue this career path:<br>Connection between mining engineering and renewable energy: TRUNCATED |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Discussion on Limitations: What specific limitations of the proposed distributional reward model should be addressed in future research?
Complexity of Technical Details: Which technical details were particularly complex, and how might they be simplified for better understanding?
Practical Applicability: What are the potential real-world applications of the proposed approach, and how could its limitations affect these applications?
Outdated Model Concerns: How does the findings with Llama-3-8B Instruction model impact recent research?
soundness: 2
strengths: Identification of Problems: The proposed distributional reward model clearly identifies existing issues in the current methodologies.
Experimental Evidence: Strong experimental evidence is provided to support the effectiveness of the proposed model.
Well-Organized Structure: The paper has a well-organized structure, making it easy to follow.
Effective Use of Visuals: Tables and figures are effectively utilized to present experimental results.
Contributions to Multi-Dimensional Alignment: The research offers a new methodology for addressing the problem of multi-dimensional alignment in LLMs through the distributional reward model.
summary: The paper discusses a proposed distributional reward model aimed at addressing the issues of distinguishing between divided preferences and high-agreement preferences in reward modeling for language models (LLMs). It points out that standard reward modeling approaches, such as Bradley-Terry and MSE regression, fail to differentiate between these two types of preferences, leading to similar reward distributions and potential problems in multi-dimensional alignment when using Reinforcement Learning from Human Feedback (RLHF).

The authors outline two main objectives for their model: (1) identifying preferred responses and (2) detecting responses that may exhibit divided preferences. By achieving these objectives, the model aims to prevent the system from learning responses that reflect only a single user perspective. The authors argue that training this reward model is more cost-effective and efficient compared to obtaining multiple annotations for every data point.

To evaluate the model's performance, two metrics are introduced: Preference Accuracy, which assesses the model's ability to assign higher rewards to responses selected by human annotators, and Diverging ID AUROC, which measures the model's effectiveness in identifying divided preferences within response pairs.

The results, based on training and evaluation with the HelpSteer2 and Multipref datasets, indicate that the proposed distributional reward model performs effectively, consistently exceeding the baseline metrics for both Preference Accuracy and Diverging ID AUROC. This demonstrates that the proposed model can predict expected rewards while reflecting the controversy of responses as assessed by different annotators.

In the latter sections, the paper explores biases inherent in the evaluation of LLMs using the LLM-judge method, particularly when preferences are divided. It discusses how the LLM-judge’s assessment may unfairly penalize systems that reflect less popular opinions or that are trained with consistent policies in ambiguous scenarios.
weaknesses: Lack of Discussion on Limitations: There is insufficient discussion regarding the limitations and potential issues of the proposed method, particularly concerning the outdated model.
Complex Technical Details: Some technical details are explained in a complex manner, which may hinder understanding for non-experts.
Need for Practical Applicability Discussion: The paper lacks a thorough discussion on the practical applicability and limitations of the proposed approach, which could enhance its relevance and usability.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - Unclear Mapping Explanation in the Mean-Var Reward Model (lines 319-323): The rationale for mapping the labels back to specific ranges is unclear—why is this mapping necessary? Would it not be possible to train directly on the distribution? Section 4 is quite dense, and additional explanation for this mapping would help clarify the distributional reward model.

- Figure 1 disagreement analysis: The left plot for MultiPref seems to suggest a possible bias in the annotation interface, as there is a preference of annotators to prefer B annotations over A (non-symmetric matrix, skewed histogram, and darker areas in the upper-right corner). What is your explanation for this? The HelpSteer dataset does not seem to show a similar annotation behavior. I would like to hear your thought - it can be interesting to add this to the paper discussion. This also connects to my questions below on more information on the annotation setup. 

- Annotators of datasets: "MultiPref has 10k preference pairs with four annotators, HelpSteer2 has 12K with 3-5 different annotators." Can you say more about the identity of the annotators, are the four in MultiPre the same individuals? If not, how many individual annotators are there in both datasets? How many annotations on average did each annotator? Do you release annotator IDs? Did you collect any annotator social information? 

- Refusal vs Refusal: Can you provide more detail on the original annotation tasks? For example, were annotators instructed to take a forced choice? Or was a "I don't know" option allowed?

- Results in Table 4 LLMs-as-Judge: What are the scores in the table and what do they mean? Do they only compare to the majority preference ("winning response")? If so, I think it would be more interesting to compare to the human preference distribution. Thank you for clarifications. 

- What was the motivation of using a 8B LLama-instruct model? Were there restrictions to not use a larger model (70B?)? Would you expect similar findings with the larger model? Which exact Llama model was used? (As there exist by now several versions 3, 3.1, 3.2).

- Will you release the code for the distributional preference models and the trained reward models?

Overall, I like the paper a lot and I am willing to go up with my overall score. However, the results are dense and I have questions I would like to hear from the authors. I look forward to hear the answers to my questions above.

Typos:
- "singe-value" in several places
soundness: 3
strengths: Positive aspects of the paper:
- Releasing unaggregated annotator labels is of increasing importance, as there is increasing evidence that modeling the aggregate leaves out important human preference information. This supports similar earlier calls to do so - see [Basile et al., 2021](https://aclanthology.org/2021.bppf-1.3/), [Prabhakaran et al., 2021](https://aclanthology.org/2021.law-1.14/), [Plank, 2022](https://aclanthology.org/2022.emnlp-main.731/). 
- Studying reasons for disagreement is similarly important, the derived taxonomy is insightful. It extends prior taxonomies by providing categorizations for LLMs for refusal behavior, which is novel. The taxonomy further supports prior findings on reasons for disagreement [Basile et al., 2021](https://aclanthology.org/2021.bppf-1.3/) and taxonomies of disagreement in NLP, which were developed by [Jiang and de Marneffe, 2021](https://aclanthology.org/2022.tacl-1.78/) for NLI, and extended to other tasks, for example, subjective language [Sandri et al., 2023](https://aclanthology.org/2023.eacl-main.178.pdf) and law applications [Xu et al., 2023](https://aclanthology.org/2023.emnlp-main.594.pdf).
- Distributional rewards are timely. The paper presents a simple and concrete implementation. (The question remains whether code will be released upon publication).
- The impact on diverging preferences on LLMs as judges is, to the best of my knowledge, novel. This is an important study showing that neglecting divergences propagates majority views and thus is in competition with pluralistic alignment (Sorensen et al., 2024).
summary: The paper studies disagreement in preference datasets and provides annotator-level annotations of two existing human preference datasets, [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2) and [MultiPref](https://huggingface.co/datasets/allenai/multipref). The paper first derives a taxonomy of causes for disagreement (Table 1) on a sample of 100 items. Then, the authors train two separate standard reward models using the majority vote preferences (Bradley-Terry and MSE Regression) and find that on examples with diverging preferences the predictions of reward models are biased towards high-agreement preferences. To address this gap, a model with distributional rewards (Mean-Variance Reward Model, with KL) is presented which uses a KL-divergence loss. The results show that the KL-based distributional model outperforms a Mean-Variance baseline model and better aligns with human preferences. Finally, the paper presents experiments in LLMs-as-a-judge evaluation, finding that they promote majority preferences.
weaknesses: The paper's weaknesses are:

- Evidence (lines 265-266): The claim that "reward models predict differences in rewards that resemble high-agreement preferences, even when trained on all annotator labels" is not convincingly supported. The scores for models trained on all labels vs. aggregated labels (All vs. Agg) are often similar. To substantiate this claim, the authors should extend Figure 2 and compare models trained on majority labels vs. all annotator labels on both datasets. Currently, Figure 2 only presents results for the model trained on aggregated labels and for a single dataset, illustrating that diverging preferences align with high-agreement items. For a stronger argument, similar plots should be included for both models and across datasets and discussed in the text.

- Related Work: The field of disagreement in NLP has a substantial history, with early contributions such as de [de Marneffe et al., 2012](https://aclanthology.org/J12-2003/), [Poesio and Artstein, 2005](https://aclanthology.org/W05-0311/) and more recent surveys like  [Uma et al., 2021](https://jair.org/index.php/jair/article/view/12752). This paper could be improved by citing more of this foundational literature, including key work on developing taxonomies and understanding the underlying reasons for disagreement (suggested references below). 
  - Reasons for disagreement in NLP and computer visions: see [Uma et al., 2021](https://jair.org/index.php/jair/article/view/12752) and references therein. Moreover, see further references on calls to release unaggregated labels in first point in Strengths.
  - Taxonomies of disagreement: There exists seminal work by [Jiang and de Marneffe, 2021](https://aclanthology.org/2022.tacl-1.78/), I wonder whether this paper was inspired by their work? It was taken up by several others, see further references in second point in Strengths. 


- Table 1: Do the frequencies in the two datasets sum up to 1? Is this per subcategory, or what is the overall frequency for each of the four top categories on MP and HS2?

- Code release is not mentioned. Releasing the code would make some study design choices clearer (like the mapping above) and enable better replication of the results in the paper.

- The paper could have included more recent and larger language models. For example, results for the LLama model family over different scales would be interesting. I invite the authors to discuss any potential challenges or limitations in applying the method to larger model families, or to explain why you chose to focus on this specific model (llama 8b instruct).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. RMs and LLM-as-Judges are mostly integrated into training and evaluation pipelines as proxies for human preferences (they are rarely used alone). While the paper demonstrates that its proposed training methodologies improve RMs' ability to distinguish between instances with diverging preferences, it lacks discussion on the potential downstream impacts. Ultimately, can these new RMs create models better aligned with human preferences? Are they more effective evaluators for leaderboards that aim to reflect genuine human preferences?

2. The paper lacks experimental evidence in defining the problem. While I agree with the importance of developing smaller, high-quality RMs, recent studies have shown that scaling up RMs yields better evaluators. Does the issue of failing to detect diverging preferences persist even with larger RMs? If the issue goes away with scaling, probably it might not be an issue soon when better and cheaper hardware becomes available.

3. Section 5.1 highlights that  LLM-as-Judges also struggle to identify instances with multiple preferences. However, could this issue stem from the prompting approach? The referenced LLMs rely on straightforward prompting techniques for judgments, which do not inherently account for multi-preference scenarios. Could more sophisticated prompting methods or multiple sampling iterations help address this limitation?
soundness: 2
strengths: 1. While it is often intuitively accepted that RMs and LLM-as-Judges may exhibit biases and fail to reflect the diverse preferences of humans, this paper offers a systematic approach to identify and quantify these errors. Additionally, through a qualitative study, the paper provides a taxonomy to categorize the primary causes of preference divergence.

2. The paper goes beyond just pointing out the problem to present two training methodologies to train models that better represent diverging preferences. The two methods aim to model the preference distribution instead of singular values and achieve a 10% performance improvement. 

3. The writing is clear and easy to understand.
summary: This paper leverages the MultiPref dataset and the HelpSteer2 dataset to study the behavior of RM/LLM-as-Judge in instances with diverging human preferences. They observe that traditional methods of training RMs (Bradley Terry or MSE-Regression) fail to make RMs that represent multi-preferences. Hence, they propose alternative methodologies, Mean-Variance Reward Models, and Classification-based Reward Models, to train RMs that learn the distribution of the responses instead of a singular value. The presented methodologies show about 10% improvement from past methods using AUROC as the metric.
weaknesses: Please see the questions section.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please refer to the questions above.
soundness: 3
strengths: 1. The problem of annotator disagreement is an important one as the current model training neglects the inherent difference between the annotators which could lead to misalignment of the model. The pluralistic alignment of the reward model in RLHF has great potential. 

2. The author not only reveals the misalignment of the reward models but also proposes a new training objective for it to mitigate the problem. Experimental results show that the reward model trained with a new objective can better identify the disagreement in the data.
summary: This paper investigates the problem of diverging preference in human-labelled preference datasets.
They created a taxonomy of the disagreement sources and tested the RLHF reward model on disagreement data.
They showed that the reward model trained on majority vote would clearly prefer one of the responses when presented with examples with diverging preferences. 
The author further proposed a new reward model by predicting the mean and variance of the distribution reward. 
The proposed reward model achieves better performance in terms of distinguishing the diverging and non-diverging preference examples.
weaknesses: One problem of the paper is that details about the implementation and motivation of experiment design are either missing or moved to the appendix.
It makes the paper hard to follow. 
For example, it is not clear why the author split the level of disagreement by High-Agreement Prefs, High-Agreement Ties and so on.

1. Why do High-Agreement Prefs require no rejection of the majority vote, but the High-Agreement Ties allow it? 

2. In lines 319-323,  why is the mapping interval set like this? I believe the intervals could have a great influence on the reward model.

3. The CDF estimation is an important detail for training and evaluating the reward model, which I think should be discussed in the main text.

4. In line 348, I don't fully understand what you mean by "use the predicted joint probability of annotators labelling the response as a 1 or 5".

5. In line 361, "using smaller differences as a predictor" is not informative. What do "smaller differences" mean exactly?

Meta-review: This work examines diverging preferences in human-labeled preference datasets used for training language models. The authors claim that current reward modeling approaches fail to distinguish between high-agreement and diverging preferences, and then propose a taxonomy of disagreement sources and analyze two datasets (HelpSteer2 and MultiPref). LLM-as-judge evaluations show bias toward majority preferences.

Strengths:
1. The problem is important and timely, which deserves more research attention
2. Provides systematic analysis backed by empirical evidence

Weaknesses:
1. Implementation details are sometimes unclear (or buried in appendices)
2. Limited exploration of how findings generalize to larger models (though partially addressed in rebuttal)
3. Some methodological choices lack thorough justification

The core ideas are sound and the problem addressed is important. But several aspects still need strengthening before publication: the methodological choices require more rigorous justification, and technical details need a clearer presentation in the main text. Therefore, while this paper addresses an important problem and also shows promise, I recommend revision before acceptance (which is rejection for ICLR).

Decision: Reject