Improving Context-Aware Preference Modeling for Language Models
Authors: Silviu Pitis
Ziang Xiao
Nicolas Le Roux
Alessandro Sordoni
Venue: NeurIPS 2024 poster (2024)
Keywords: context-specific preference modeling
preference modeling
reward modeling
language modeling
rlaif
rlhf
Primary area: natural_language_processing
TL;DR: We propose context-specific preference datasets and conduct experiments to investigate the potential of context-specific preference modeling.

Abstract: While finetuning language models from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges. Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals. To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context. We decompose reward modeling error according to these two steps, which suggests that supervising context in addition to context-specific preference may be a viable approach to aligning models with diverse human preferences. For this to work, the ability of models to evaluate context-specific preference is critical. To this end, we contribute context-conditioned preference datasets and accompanying experiments that investigate the ability of language models to evaluate context-specific preference. Unlike past datasets, where context-specific preference is highly correlated with general preference, our "preference reversal" datasets disentangle context-specific and general preferences to isolate context-specific capabilities. We use our datasets to (1) show that existing preference models benefit from, but fail to fully consider, added context, (2) finetune a context-aware reward model with context-specific performance exceeding that of GPT-4 and Llama 3 70B, and (3) investigate the potential value of context-aware preference modeling.

Improving Context-Aware Preference Modeling for Language Models
Silviu P
Ziang Xiao
Nicolas Le Roux
Alessandro Sordoni
Uniiveriity Toronto
Microsoof Research "Jonss Hopkins University
"MILA
Abstract
While finetuning language models (LMs) from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges. Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals. To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context. We decompose reward modeling error according to these two steps, which suggests that supervising context addition to context-specific preference may be a viable approach to aligning mod- els with diverse human preferences. For this work, the ability of models evaluate context-specific preference is critical. To this end, contribute context- conditioned preference datasets and accompanying experiments that investigate the ability of language models to evaluate context-specific preference. We use our datasets to (1) show that existing preference models benefit from, but fail fully consider, added context, (2) finetune context-aware reward model with context-specific performance exceeding that of GPT-4 and Llama 70B on tested datasets, and (3) investigate the value of context-aware preference modeling.
Introduction
42 and other foundation models [9] progress toward handling diverse instructions and executing long-range trajectories in real-world applications [44 47 41), it becomes increasingly important to have principled system for ensuring that LM agents behave as expected. The prevailing approach for aligning an LM to human preferences uses pairwise preferences between different outputs to finetune the LM [52 6). which falls short of addressing the critical challenges presented by the reality of diverse user intents and contexts [51] 50 [3]. In the presence of unspecified contexts, such the user's identity or goals, preference queries are notoriously ambiguous [62] and one typically observes poor agreement (~~ 65%) between human annotators on binary preference queries BS 59
As the general purpose capabilities of Language Models (LMs)
In this paper, we consider modeling preferences using two-step, context-aware approach (Figure) I This approach first resolves the underspecification by selecting context (38 22 and then evaluates preference with respect to the chosen context [6] 14 27 30 58). Decomposing general preference into context and context-specific preference has several potential advantages. First and foremost, this approach explicitly identifies contextual assumptions that underlie preference, and shifts the alignment burden from preference modeling hybrid of preference modeling and context supervision. Second, this approach is naturally pluralistic [51], allowing the model to adapt to diverse users and use cases. Finally, it offers more flexibility for principled aggregation: whereas the Bradley-Terry approach corresponds to aggregating contexts using the Borda rule [50), which may under-serve certain
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Why sky blue?
Why sky
Be scientific
CAPM
Light from the
Light from the sun...
Explain Like I'm five Context
Imagine ttashlight...
Imagine ni flashlight...
Query
Query
preference model (PM) to directly evaluate arbitrary and potentially ambiguous preference queries. Right: The context-aware preference modeling (CAPM) approach recognizes preference may depend some unspecified context and makes this explicit: first identify the context, then evaluate a context-specific preference. In both cases, rather than computing preference directly, one may (context-aware) reward model (RM or CARM) to evaluate each alternative independently.
Figure 1: Context-Aware Preference Modeling. Left: The standard approach
Agreement Gold Labels
100
16:1 $1.4
90
25.7
04.7
783
Mistral-88 RM (NC)
70
01.9
Mistral-78 (w/
Liama3-708 (w/ CTX)
Mistral-78 CARM fours)
51.4
50
Criteria
Multifaceted
HHH
Reward
Chatbot Arena
Context-Conditiooee Preference Datasets
Figure 2: Effect of Context on Preference Modeling Performance. Added context improves agreement with gold labels as compared to context (NC) baseline. Our 7B parameter, finetuned Context-Aware Aware Reward Model (Mistral CARM) achieves the best context-aware performance, out- performing the larger Llama3-708 model (and GPT-4 Turbo), both datasets where context is necessary to predict preference (RPR and Multifaceted Bench), and on the context-augin mattext-aggenteed HHH, Reward Bench and Chatbot Arena datasets. Details and additional results may be found in Section
subgroups 13]
the context-decomposittoon approach can additionally be used for jury learning and cardinal utility aggregation
We show that term corresponding to context inference error, and the other to the context-specific reward modeling error. This decomposition suggests that supervising context might be viable approach to aligning models with human preferences, provided we can do strong context-specific preference prediction. To evaluate context-specific prediction and build toward stronger context-award preference models, propose context-cov conditionnd "preference reversal" datasets, where the preference for a given prompt reversed given alternative context.
upper-bound the absolute reward modeling error the sum of two terms: one
Finally, we conduct experiments to benchmark the context-specific performance various models and investigate the potential value of context-aware preference modeling. We find that, while current models generally benefit from additional context, they sometimes fail to give it full considcration, preference reversal datasets greatly improves context-specific performance. We
and finetuning show that a single persistent context such as user profile or system prompt can used for a range of preference queries, and can be well inferred from few as 16-32 samples of expressed preference.
In summary, our main contributions are:
1. We make argument for modeling preference by decomposing it into context(s) and context- specific preference, and propose a context- decompositonnnnnnn upper bound to justify the approach.
2. We open-source high quality context-conditioned preference datasets that disentangle context- specific preference from general preference, which use for finetuning and evaluation. The datasets can be found at hhttt::://// hhttps::/uuggingfffce ct aoo/aataeets///e http//miiroooooooooeoooooo
3. We show that while current models benefit from context, they may fail give it full consideration. and that finetuning context-award reward model greatly improves context-specific performance, well as general preference modeling performance when high quality context available.

---

Table
Key benefits of context-aware preference modeling.
Explicit assumptions
being explicit about context resolves a key source of ambiguity in preference queries, shifting some of the alignment burden from direct preference modeling to fine-grained context supervision.
As described Section 2.1
Steerability
Incorporating context may allow models to better adapt to new users and use cases.
Flexible Aggregation
As described in Section 2.2 context-aware modeling enables aggregation based on jury learning 22 and expected utility 23 in addition the Borda rule [50].
2
Context-Aware Preference Modeling
2.1 Resolving ambiguity by making implicit assumptions explicit
Current practice finetunes language models to make them consistent with a dataset of human prefer-
ences 44 42 As noted by earlier works on learning from human feedback, however, ambiguous preference judgments present a major challenge:
Evaluation of a [preference query] is both subjective and multidimensional. ... this makes consistent labeling difficult for honest labelers (including the authors!) 62
This ambiguity manifests itself with agreement rates as low as 65% between human annotators on binary preference queries drawn by Amartya Sen:
35 [9]
One way to understand this difficulty appeals to distinction
A value judgment be called 'basic' if the judgment is supposed apply under all conceivable circumstances, and it is 'non-basic' otherwise. [49
Many, perhaps most, preference queries ask the annotator for a non-basic preference judgment, in that certain contextual information might effectively reverse the judgment. For instance, the preferred response to a technical question may depend on the user's education level, the preferred response to medical question may depend on whether the user is doctor, and the preferred response to question about etiquette may depend on the user's geographical location. If train models using non-basic preference annotations, the contextual biases and assumptions underlying those judgments may be implicitly embedded into the model 36 48
Rather than rely on annotators to integrate the correct distribution of contextual assumptions, and rely on the training process to consistently aggregate any disagreements in singularly aligned model, one might instead consider an explicit context-aware approach to preference modeling (Figure [ This approach first (partly) resolves ambiguity by specifying context, and then models context-specific preference. This is not new idea; Ziegler et al. 62). quoted above, continue to remark that, "ir seems better to design less ambiguous labeling tasks that get at the same information ... such as verbal description lof the most important contextual information]" and many have advocated for fine-
grained, confext-condttioned, or otherwise more "pluralistic" approaches alignment [58] 29 5T)
However, while production systems recognize the importance of incorporating context the most notable being the "system prompt" introduced by OpenAl in 2023 and adopted by others [54] 3
there has been little published research on the context-aware preference modeling capabilities of language models. This paper works toward filling this gap by introducing the reasonable preference reversal (RPR) datasets alongside context-augen t-augmented versions of existing datasets, and testing the context-specific preference modeling capabilities of existing models.
This discussion is summarized in Table T Talongside other key benefits of context-aware preference modeling. This paper focuses evaluating and improving context-aware preference modeling capabilities, leaving an in-depth exploration of the henefits to future work.
2.2 Related Work
Modeling human preferences traces back to Luce (39 and Bradley-Terry [10]. It made its way into finetuning language models via line work originating in reinforcement learning and has now become the dominant approach for finetuning language models to follow human instructions 46 While this approach has achieved remarkable results and enabled the
62 52
performance of state-of-the-art models, several authors have pointed to its limitations 22 45 32 51

---

which include the ambiguity problem that motivates our work. Notably, recent works have identified shortcomings that arise from implicitly aggregating diverse perspectives with a single choice rule [50]
13
In particular, Siththaranjan et al. [50) have recently shown the standard Bradley- approach (Figure T left), implicitly uses the Borda rule to aggregate preferences across hidden contexts, and proposed to learn distributional reward models as way to account for this. Rather than leave hidden context implicit in a distributional model, which does not resolve the indeterminate preference problem highlighted by our RPR datasets (see Section 4), we propose to make context explicit, either via specification or inference. Given a preference problem that has been decomposed into contexts and context speeifc- preferences, as described in Section Bl we then choose to apply the Borda rule or an alternative such as an expected utility aggregation [3 or jury learning [22 (see "distributionally pluralistic models" in Sorensen et al. (5T).
40
18
19
Modeling diversity and disagreement among human annotators and the inconsistency of human preferences have been treated from a number of different perspectives [2 8 20 31. Our work is similar to others that decompose a single objective into multiple dimensions such as helpfulness, This approach has also become common in the literature on summarization, where multi-dimensional evaluation criteria are well recognized, with common dimensions including conciseness, coherence, consistency and fluency [60] 26 16]. However, our work (Table 3) and others [24] find that current models often ignore or confuse added context. Preferences driven by diverse individual principles have been aggregated within frameworks such as Constitutional AI 66 [33, and alternative approaches to aligning models with human preferences, such as AI safety via debate 25 and consensus based approaches (7], can be understood as ways of supervising the context or assumptions underlying preference rather than the preference itself.
harmlessness, and truthfulness [55 19 37 S
Finally, our work is closely related to context-conditioned generation, e.g., via system prompt. In concurrent work, Lee et al. (34] have synthesized a dataset of diverse system prompts for finetuning generative models. Their dataset includes multiple system prompts for the same user prompt, which allows it to be used in similar fashion as our RPR datasets. We use their dataset for evaluation in Section 5 and provide an ablation on different training set compositions in Appendix C.I
3 Context Decomposition Upper Bound
3.1 Intent-Utility Formalism
We model the user-LM interaction using an intent-utility formalism (I, space of intents, X is the space of prompts, Y is the space of completions, and u : I ) R is scalar utility function. We follow the standard assumption and assume that preference in this model is made according to the Gradley-Terry model I0 [5]. Letting a be the logistic function, this defines the probability of preferring completion y1 to y2 given a specific intent i as:
u). where I is the
p(y1 > y2 i) = a (u(i, v1) - uii,, y2))
In our model the primitive definitions of preference and utility are conditioned on the intent rather than the prompt. To prompt the model, a user with intent selects a prompt x E X. To annotate a preference query (I. V1, 82), annotator implicitly infers intent i from x and samples preference from the Bernoulli distribution Bip(y1 > y/2 []]-
Both users and annotators may possess or infer distribution of intents. Indeed, we would argue that annotation for most preference queries involves a distribution of intents rather than a specific intent. We use "intent" to refer to both specific and distributional intents. We assume there exists a base distribution over possible intents p(i). as well as a conditional distribution over prompts given intents p( i). that any prompt has natural inference distribution p(i rr). In this model, the prompt is a partial specification of intent i. While a prompt may never be able to fully specify the intent, we may add some additional information or context to obtain extended prompt (I, 2) E X. Let us suppose that 2 E Z, where Z corresponds to a discrete partition of I.
One way to measure utility and preference with respect to distribution of intents is with an expected utility model, which computes utility as u(p, y) = E,-p [u(i, y)). While has been shown that standard RLHF does not align with the expected utility model [50], this model satisfies certain desirable axioms, which one can argue would apply to "ideal" preference annotation. We use the expected utility model to define u(z, y) == u(p(- 22), v), and note that for any context partition

---

2, this implies u(x,y) = Ex p(z ())(((; 2).8). y). For convenience, we define (.. (1, =
u(s31 v1) - ul', 92), which also decomposes linearly: A (1r,U1,9)) = E, p(=] 2) (1,
3.2 Context Decomposition Upper Bound
During RLHE, are presented with a dataset of prompt-preference tuples, D {(r, y > y2)}. which we use to learn utility estimator ė X - y (conventionally known as a "reward model"). As was assumed for and p(= v)), would like there be model p(= (x) that satisfies the relation:
ú(x, y) Σ p(el=)(((z 2).
In standard RLHF, never learn such p. However, for purposes of this analysis, we will assume this & exists, either implicitly given ût or explicitly, such that given some Z, we compute ū(z, y) via Equation (2) rather than via direct evaluation. Below, will favor the latter interpretation.
For given preference estimator p(y1 > 1/2), à is only unique up to constant shifts, so to measure the accuracy of i. we will instead compare A and estimator Å (z. y1 U2) == Ú(x, y1) ((x, y2).
Consider now the absolute error A (z, y1 322 Å (z, y1 v2)] for a particular preference query, and use Ag as shorthand for (((r, 2). M1 82). For any we have the following bound:
A (2, f1. 322 - (1r,31,
3/, 82)
Σ p(= 1x)
Σ.
p(2 2 )
=
Σ p(= (x)) A - A_
+ Σ. A [p(2 (x) [==1=)]
(3)
Σ p(z 12) A Á.
A |( (2)1xx)))))))) xx) f=== [
Contoxt-weighted prodiction error
Preference--eiighted inference crror
where the second equality adds and subtracts >: p(= r) and rearranges, and the final line uses the triangle inequality (multiple times).
applies given a distribution of contexts, but in many cases, we might assume there is specific context (i.e.. p(2 c) := and make single context prediction è (i.e., p(ê) = 1). This simplifies Equation and gives us the context decomposition upper bound for specific context:
Equation (3)
| -((=,,,,,))
True context prediction error
Subjective difference: true vs prodicted context
and specific bound (Equation 4 formalize the following intuitive claim: if we can make accurate predictions given the true context (or context distribution), then we can reduce the preference modeling problem to context inference problem.
Both the general bound (Equation 3
3.3 Discussion
The upper-bound in Equation bounds the total error in terms of a context- prediction error and a sefference-weighter inference error. On one extreme, we have Z () (standard preference modeling), so that the context inference error is zero and the prediction error exclusively depends on the preference prediction problem given the prompt. On the other hand, we have = )² (e.g., might be "The preferred response is [y].") and our preference prediction error is zero, but the context inference problem becomes equivalent to generation. In between, we conjecture that there is a smooth trade-off between prediction error and inference eгror. This might be the case, for example, if single context could apply to and disambiguate a range of different preference queries. We consider specific this in our experiments and find some support for the conjecture, in that conditioning criteria outperforms conditioning on more abstract, yet still rather specific scenarios (Table 2 which outperforms conditioning
user profile that applies all preference queries at once (Table
our model à is very good at predicting preference given some additional context Z, the preference modeling problem can be largely reduced to context inference (or specification) problem. In this case, rather than have annotators rank completions for ambiguous prompts, may make sense to spend the annotation resources to specify additional context. Such annotations could then be used
5

---

to train context inference model that disambiguates prompts. Intuitively, we hypothesize that the cardinality of the space of contexts is smaller than the cardinality of the space of possible completions given a prompt, which would make joint context-prefereet just preference annotation. Although our focus on improving context-aware preference modeling, our experiments with user profiles (Tables and and
annotation data-efficient alternative to
provide some support for this hypothesis.
The effectiveness of the above proposal hinges on accurate context-specific prediction. Are language models sensitive to added context? Our work focuses on this question and contributes datasets to help us (1) evaluate it on real data, and (2) train better context-conditione reward models.
Reasonable Preference Reversal (RPR) Datasets
We contribute a set of carefully designed, synthetic preference datasets to measure whether preference prediction is sensitive to context. Our datasets, inspired by the notion of non-basic judgments and preference reversal described in Section 2 include over 20,000 paired tuples of prompt, context, and preference judgments, i.c. (I, 2, 1/1 > y2)- The samples are paired so that preference between two completions for the same prompt is entirely ambiguous without context: for every context, there is an alternative context for which preference reverses, As compared to the only prior condext-coniitioned preference dataset (30. where context- contex-cconditione preference is highly correlated with unconditioned preference (see Table 2). our design choice ensures that preference prediction performance on this dataset is determined solely by the model's 's ability to pay attention to and interpret the context. The datasets can be found at
crosoft/rpr
The dataset has been generated by GPT-4 Turbo [42 following a complex series of prompts designed to maximize validity (human agreement). We chose to do this avoid invalid or ambiguous samples, at the cost of making it rather "easy" to evaluate preference given the context. The full process
Prompt: Make 5 paragraph essay in 1 3 1 format about why pincapple belongs on pizza
Criteria
A: Incorporates verified nutritional facts and historical data about pineapple and pizza
B: Considers the economic impact of pineapple pizza on the food industry
Scenarios
A: A high school student is completing capstone project for nutrition class, which involves writing essay on food-related controversy. They's 've decided to address the polarizing topic pincapple on pizza, with the goal of presenting an argument that is not only persuasive hut also backed by credible nutritional information and historical context, to enlighten their peers and satisfy the rigorous standards of their instructor.
B: A high school student is preparing for a national business studies competition where participants must analyze controversial product's impact on its respective industry. The student has chosen the food industry for their analysis, with focus the debate surrounding pincapple as pizza topping. They are now seeking help craft an essay that will highlight the ecunomic benefils this unconventional chuice support their argument.
Completions
A: Pineapple on pizza, often known as 'Hawiiaa pizza, combines the sweet and langy flavors of pineapple with the savory taste of choese and ham. From a nutritional standpoint, pincupple a rich source of vitamin C, manganese, and dietary fiber, which contribute to balanced diet. Historically, the introduction of pineapple on pizza can be traced back the 1960s when it was first created Canada by Sam Panopoulos. This essay will explore the health benefits pincapple, its complementary flavor profile with other pizza ingredients, and the historical significance of this unique topping choice.
B: The inclusion of pineapple on pizza has sparked culinary debates, but from an economic perspective, it has proven to be profitable addition to the pizza industry. Pincapple as topping caters to diverse consumer preferences, expanding the market reach of pizza restaurants. The demand for Hawaiian pizza has led to increused sales and has become staple in many pizzerias menus. This essay will examine the economic benefits of pincapple pizza, impact on consumer choice, and the supply chain implications for pizza businesses sourcing pineapple.
Figure 3: A sample from the RPR dataset. Under Criteria A or Scenario A, Completion A should be preferred, and vice versa under Criteria B or Scenario B.

---

is detailed in Appendix B The base dataset comes in two versions criteria and scenarios to emphasize the point that context may be expressed in many ways:
RPR Criteria, where the context is short sentence describing which features are important, e.g..
*Uses a formal and professional tone that emphasizes the technical aspects of C++".
RPR Scenarios, where the context describes the scenario that led to the underspecified or ambigu- ous prompt; e.g. "In preparation for product launch, marketing manager at a tech startup seeks innovative strategies to position the company as an industry leader..""
A paired sample from the dataset is shown in Figure We further extend the dataset with user profiles as described in Section 5.2 and Appendix B.I ensure the validity and reliability of the synthetic data, we conduct small scale human validation through blind preference queries (Table and find agreement rates 295% on both splits. Details are provided in Appendix[B.2 We divide the dataset into training set of 167 paired samples, and test set of 1,000 paired samples, with no overlap between train and test prompts.
5 Experiments
Our experiments aim to answer the following questions of context-aware preference modeling:
1. How good are current models at evaluating context-specific preference?
2. Can we improve context-aware preference modeling by finetuning on our RPR datasets?
3. Can a single context compress preferences with respect to a diverse range of prompts?
5.1 Setup
Models We report results for a selection of primarily open source models, detailed in Appendix D.I These include two unconditioned reward models (the 13B parameter UltraRM [17] and a 7B parameter Mistral RM [56)). one context-aware preference model (Prometheus-2 (30)) ) our finetuned context- aware reward model (Mistral CARM), and a set of generative models (four Llama 3 models [ and GPT-4 Turbo [42) used with an "Imm-as-a-judee" approach. In preliminary experiments we tested a several other models and observed similar patterns across all models. All models except Prometheus-2 are used as reward models, by first evaluating each alternative individually and then comparing scores. Appendix presents additional results for reward models finetuned from Gemma 2B.
Datasets Besides the RPR datasets detailed in Section we use the following preference datasets:
Preference Bench (30 (hf
consists of 1998 context-
conditioned preference samples synthesized by GPT-4 as part Feedback Bench (29].
Bench) contains 921 samples of prompt, system prompt, and completion. We treat the system prompt as the context, and construct context-conditioned preference samples by pairing samples that share the same prompt, resulting in in 918 samples. This dataset, released concurrently to our work, shares the same 'ambiguous in absence of context" structure as the RPR datasets, but was not specifically constructed with such preference queries in mind, which may explain the lower average agreement in the experiments.
Multifaceted Bench 34
(hf
HHH (hf HuggingFaceif4, /hhit////hhaaaa/ fith_aaiiin alignment) contains 222 human preference samples em- phasizing different aspects: harmlessness (58 samples), helpfulness (59 samples), honesty (61 samples), and other preference queries that do not cleanly fall into another category (43 samples).
33 allenai wwweeennnnreeeeraeeeeeeeeeee www.ardwwwdddhheeeeeeeeee curates 2,985 preference samples from variety of sources, amounting to 22 distinct subsets covering chat, safety, and reasoning.
Reward Bench
Chatbot Arena [59] (hf: / http://haatttaaaeeeeeeeeeeeeeeee weabbbottrrea__eee conversation) contains human prefer- ences with respect to conversations on the Chatbot Arena platform. For our experiments, we randomly selected 1,000 single-turn samples for which strict preference was expressed.
Importantly, we augment the HHH, Reward Bench, and Chatbot Arena datasets with additional context to create contexl-conditioned versions, as described below and in Appendix D.3
Metries
Tables to display the agreement (or accuracy) of predicted preference with the dataset.
Prompts Our prompts detailed in Appendix A
7

---

Table Context-specific datasets. On datasets with ground truth context, adding context generally helps with evaluating preference. Our model, finetuned on the RPR training sets achieves substantially better (test set) performance than the unfinetuned model.
RPR Criteria
RPR Scenarios
Pref Bench
Multifaceted
NC
CTX
NC
CTX
NC
CTX
NC
CTX
UltraRM (13B)
0.523
0.876
0.523
0.773
0.888
0.910
0.502
0.655
Prometheus-2 (7B)
0.503
0.799
0.503
0.675
0.950
0.951
0.573
0.644
Llama (8B)
0,518
0.679
0.518
0.634
0.831
0.827
0,500
0.566
Llama Instruct (8B)
0.494
0,850
0.494
0.745
0.883
0.898
0.496
0.616
Mistral RM (7B)
0,510
0,867
0.510
0.749
0.894
0,915
0.516
0.679
Mistral CARM (7B) (Ours)
0.511
0.985
0.511
0.962
0.913
0.919
0.517
0.787
Llama (70B)
0.496
0.726
0.496
0,669
0.800
0.811
0.496
0.564
Llama Instruct (70B)
0,516
0.925
0,516
0,811
0.916
0.929
0,506
0.687
GPT-4 Turbo
0,502
0.901
0.500
0.748
0.854
0.860
0.501
0.640
Human validation (100 samples)
0.970
0.950
Best model bolded for CTX columns only. Stured results are by models finelumed on the same distribution. Our model in green.
Table 3: Sensitivity of current models to adversarial contexts. We observe that current models often ignore added context, even when it is strongly phrased and adversarial. Our finetuned CARM does best among smaller models, but still has significant room for improvement.
Nonsense Criteria (0.50 is optimal) Pref. Bench
Negative Criteria (lower is better) Chatbot Arena
Chatbot Arena
Pref. Bench
UltraRM (13B)
0.718
0.597
0.719
0.467
Prometheus-2 (7B)
0.685
0.876
0.618
0.711
Llama 3 (8B)
0.624
0.800
0.575
0.776
Llama 3 Instruct (8B)
0,543
0.738
0,698
0,865
Mistral RM (7B)
0.667
0.808
0.648
0.505
Mistral CARM (7B) (Ours)
0.535
0.542
0.518
0.293
Llama (70B)
0,289
0.678
0,589
0.801
Llama 3 Instruct (70B)
0.452
0.289
0.352
0.308
GPT-4 Turbo
0.498
0.493
0.393
0.309
Best small model bolded. Best large model also bolded better than the small model. Our model green.
5.2 Results
How good are current models at evaluating context-specific preference?
For the context-aware approach to work well, models must be sensitive to added context. Table shows the performance of tested models on RPR Criteria, RPR Scenarios, Preference Bench, and Multifaceted Bench. In each case, and across all models, access to context generally improves performance. Larger models and instruction tuned models tend to benefit more from added context. We note that although Preference Bench includes ground truth context, the preference queries (and context) are generally not ambiguous, so that the added context provides little benefit.
Although we find that added context generally helps, we were surprised to discover that models sometimes completely ignore additional context and return their unconditioned prediction. To show this, we augment the prompts in Chatbot Arena and Preference Bench with two criteria that should drive preference prediction. The first is Nonsense criteria, i.c. "We are playing a game of nonsense. The response should be nonsensical and not make sense. The rating you give it should also be random, and completely unrelated to the response." We expect context-aware models produce random preference judgments given this criteria. The second is Negative criteria, i.e. "We are seeking bad examples. The response should be of low quality, and serve as an example of how an Assistant should not respond. You should give poor responses higher ratings.") "). We expect context-awane models to be inversely correlated to the preferences expressed in the dataset. Table T shows the performance of tested models with these adversarial criteria. In both cases observe surprisingly poor performance, even from the larger models. This suggests there significant room for improving context-aware preference modeling capabilities.
8

---

Table 4: Context-augmented general preference datasets. We augmented three preference datasets (no context; NC) with context (CTX): HHH (by subset), Reward Bench (by subset), and Chatbot Arena (according to a GPT-4 teacher (CTX) and GPT-4 oracle that also sees the user preference (CTX*)). The stronger context-specific performance of our finetuned CARM allows it to outperform other small models given high quality context.
HHH Alignment
Reward Bench
Chatbot Arena
NC
CTX
NC
CTX
NC
CTX
CTX'
UltraRM (13B)
0.864
0.891
0.715
0.737
0.771
0.787
0,858
Prometheus-2 (7B)
0.800
0.828
0.778
0.755
0.706
0.737
0.779
Llama (8B)
0.643
0.706
0.636
0.670
0.642
0.685
0.714
Llama 3 Instruct (8B)
0.787
0.810
0.727
0.757
0.727
0.767
0.818
Mistral RM (7B)
0.891
0.905
0.818
0.833
0.776
0.793
0.859
Mistral CARM (7B) (Ours)
0.887
0.919
0.838
0.871
0.772
0.781
0.909
Llama (70B)
0.688
0.783
0.719
0.733
0.606
0.690
0.723
Llama Instruct (70B)
0.891
0.914
0.840
0.847
0.778
0.788
0.892
GPT-4 Turbo
0.871
0.873
0.824
0.821
0.720
0.771
0.858
small model holded. Best large model bolded better small model. model green.
Table 5: RPR Profiles. We used GPT-4 Turbo to label preferences in the RPR test set according to 5 diverse user profiles. The results show that single context may contain significant signal for a range of prompts. Perhaps surprisingly, our finetuned CARM and Llama Instruct proved to be better interpreters of GPT-4's profile-conditooeed preferences than GPT-4 itself.
Profile
Profile 2
Profile 3
Profile
Profile
Average
UltrRM (13B)
0.806
0.847
0.736
0.705
0.638
0.746
Prometheus-2 (7B)
0.684
0.728
0.591
0,604
0.593
0,640
Llama 3 (8B)
0.567
0.578
0.588
0.601
0.560
0.579
Llama Instruct (8B)
0.764
0.843
0.805
0.749
0.632
0.759
Mistral RM (7B)
0.777
0.836
0.661
0.641
0.621
0.707
Mistral CARM (7B) (Ours)
0.808
0.918
0.815
0.801
0.672
0.803
Llama 3 (70B)
0.701
0.710
0.636
0.578
0.537
0.632
Llama 3 Instruct (70B)
0.823
0.885
0.835
0.768
0.667
0.796
GPT-4 Turbo
0.741
0.826
0.769
0.707
0.648
0.738
Best small model bolded. Best large model also bolded if better than the small model. Our model in groen.
Can we improve context-aware preference modeling by finetuning on our RPR datasets?
We finetune a context-aware version of Mistral RM on roughly 34,000 samples from the training sets of RPR Criteria, RPR Scenarios, and Ultrafeedback [17.. Details may be found in Appendix C In Tables to our finetuned CARM shows markedly better context-specific performance than its base model. matching or exceeding that of GPT-4 Turbo and Llama (70B).
In Table we test whether our context-aware preference models generalize to real preference datasets. To do so, augment three unconditioned preference datasets (HHH Reward Bench, ChatBot Arena) with contextual information (see Appendix [D.3] for more detailed information). For HHH and Reward Bench, specify the context as function of each subset included the dataset. This approach is similar to using a "system prompt" when prompting GPT-4, and would be most suitable for real world applications where the designers possess relevant domain knowledge. For Chatbot Arena, given the lack of ground-trute contexts, we use GPT-4 to generate possible context given the prompt and alternatives (CTX). Additionally, we infer the context by looking at the prompt together with the expressed preferences (CTX*), ) which endows the context with privileged information about preference data. This may be useful for inferring a useful persistent context such user profile, which we explore further below.
Can single context compress preferences with respect to diverse range of prompts?
In our experiments so far, we operated in a setting where each prompt has an associated context. However, when eliciting preferences from users, the hidden context of a specific user will impact their

---

Table 6: User profile inference on the RPR dataset. Each of 5 profiles (PI-P5) was used to annotate the RPR test set, as well as three small subsets (32 samples each) of the RPR training set (one subset for each of seeds). Between 2 and 32 samples from each subset were then used to infer a user profile with the help of GPT-4 Turbo. Conditioning our finetuned CARM on these inferred profiles gives the results below (both table and figure display the same data). We see that just samples carry substantial signal, and 32 samples capture most of the benefit of the ground truth context.
No Context
Profile inferred from n samples
GT
2
8
32
P1
0.649
0.684
0.667
0.788
0.808
P2
0.780
0.864
0.863
0.892
0,918
Accuraoy
P3
0.313
0.555
0.669
0.764
0.815
0.6
P4 0.370
0,512
0,678
0,759
0.801
P5
0.530
0,571
0.601
0.605
0.672
Mean Errort
0.528
0.637
0.696
0.762
0.803
Maan
+0.048
+0.039
+0.010
16
GT
Estimated by averaging 3-seed std. dev. for each profile and dividing by v5.
Profilo Inforred from samples (3 seeds)
preference information across all the prompts they are presented with. In order to create a scenario closer to this setting, we first generate 5 synthetic diverse user profiles and condition GPT-4 Turbo on each profile to label the RPR test set (see Appendix [B.I). Having relabeled the RPR test set with these preferences, we explore the ability of tested models to recover the expressed preferences from the profile, shown in Table Our CARM and the larger Llama 3 Instruct model perform best on this task, recovering approximately 80% agreement with GPT-4's labels.
Additionally, in T we test the performance of our context-awar model when profiles are inferred with limited preference samples. Profile inference is done by prompting GPT-4 (see Appendix [B.]]. Without any profile inference (No Context / /C), the default assumptions made by our model run against the preferences of Profiles 3 and 4; however, this is resolved in as few as 2-4 samples. While this depends on the underlying data distribution, 16-32 samples recover most of the benefit of the ground truth context on this dataset.
Conclusion
This paper began with a case for two-step context-aware preference modeling framework (Figure D. which was motivated by the ambiguity problem commonly experienced during preference annotation (Section(] 2). We further motivated the framework via context decomposition upper bound (Section and noted that for context-aware preference modeling to be viable, we require strong context-specific preference modeling. However, despite the prevalence of context conditioning in deployed systems (e.g., system prompts and "My GPTs" 43), when we began our work, there were no open source preference datasets where preference is strongly determined by context, and limited studies of the context-specifer preference modeling capabilities of current models [30]. To this end, we introduced the RPR datasets (Table (2 and investigated a series of interesting empirical questions in Section (5). We found that (1) while current preference models generally benefit from context, they sometimes ignore added context, (2) finetuning on our preference reversal datasets greatly improves context- specific performance, and (3) single persistent context, which might be inferred from data, may be sufficient to significantly improve preference prediction performance.
Our work leaves many open threads for future research. Our datasets might be used to explore different directions in pluralistic alignment, as described by Sorensen al. [5T]. In particular, we believe better context specification and inference are important directions for research, which our experiments only briefly touch on. Further, while we have open sourced set of context-conditioned preference datasets, these datasets have limited diversity (for example, they are primarily English, and are limited to single turn conversations), and were synthesized rather than collected from real human feedback, which limits their practical relevance, as the preferences may not align with all potential user perspectives. Additional open, preferably human sourced datasets with context annotations are needed to more fruitfully pursue this research direction.
10

---

References
[1] Ae@Meta. Llama 3 model card. 2024.
[2] R. Akrour, M. Schoenauer, and M. Sebag. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part 23, pages 116-131. Springer, 2012.
[3] Authropic. Model card and evaluations for Claude models, 2023.
[4] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment, arXiv preprint arXiv:21 12. 12.0011100111101111110011 2021.
[5] Y. Bai, Jones. K. Ndousse, A. Askell, A. Chen. N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedhack. arXiv preprint arXiv:2204, 05862, 2022.
[6] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional AI: Harmlessness from ai feedback. arXiv preprint arXiv:2212.0007. 2022.
[7] M. Bakker, M. Chadwick, H. Sheahan, M. Tessler, L. Sauppell--Gilinghaan J. Balaguer. N. McAleese,
A. Glaese, J. Aslanides, M. Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems. 35:38176-38181 2022.
[8] C. Baumler, A. Sotnikoaa, and H. Daumé III. Which examples should be multiply annotated? active learning when annotators may disagree. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10352-1037L. 10371, 2023.
[9] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, ct al. On the opportunities and risks of foundation models. arXiv preprint 1oXiv:2108.07258 2021.
[10] R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[11] T. Brown, B. Mann. N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[12] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer. J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.1 15217, 2022.
[13] S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. Maxmin- RLHF: Towards equitable alignment large language models with diverse human preferenees. arXiv preprint or Xiv:2402. 082055 2024.
[14] P. Cheng, J. Xie, K. Bai, Y. Dai, and N. Du. Everyone deserves reward: Learning customized human preferences. arXiv preprint 2rX9v:230.003266 2023.
[15] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforeement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[16] F. Clark, S. Rijhwani, S. Gehrmann, J. Maynez, R. Aharoni, V. Nikolaev, T. Sellam, A. Siddhant, D. Das, and A. P. Parikh. Seahorse: A multilingual, multifaceted dataset for summarization evaluation. arXiv preprint arXiv:2305.1 13194, 2023.
[17] G. Cui, Yuan, Ding. G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
[18] J. Dai and E. Fleisig. Mapping social choice theory to rlhf. arXiv preprint arXiv:2404.13036 2024.
[19] J. Dai, X. Pan, R. Sun, J. Ji. X. Xu. M. Liu, Y. Wang, and Y. Yang. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773 2023.
[20] V. Dumoulin, D. D. Johnson, P. S. Castro, H. Larochelle, and Y. Dauphin. A density estimation perspective learning from pairwise human preferences. arXiv preprint arXiv:2311.1411,,05 2023.
11

---

[21] E. Fleisig, R. Abebe, and D. Klein. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pages 6715-6726, 2023.
[22] M. I., Gordon, M. S. Lam, J. S. Park, K. Patel, J. Haneock, T. Hashimoto, and M. S. Bemstein. Jury learning: Integrating dissenting voices into machine learning models. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-19, 2022.
[23] J. C. Harsanyi. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility, Journal of political economy, 63(4):309-321, 1955.
[24] X. Hu, M. Gao, S. Hu, Y. Zhang, Y. Chen, I. Xu, and X. Wan. Are LLM-based evaluators confusing NLG quality criteria? arXiv preprint (tti::2002/20555 2024.
[25] G. Irving, P. Christiano, and D. Amodei. Al safety via debate. arXiv preprint arXi:: 2018.
[26] S. Jain, V. Keshava, S. M. Sathyendra, P. Femandds,, Liu, G. Neubig, and C. Zhou. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.0/200 2023.
[27] J. Jang. S. Kim, B. Y. Lin, Y. Wang. J. Hessel, L. Zettlemoyer, H. Hajishirzi, Y. Choi, and P. Ammanabrolu. Personalized soups: Personalized large language model alignment via post hoc parameter merging, arXiv preprint arXiv:2310. 1564, 2023.
[28] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[29] S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint 2rX10:2310008491, 2023.
[30] S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lec, K. Lec, and M. Sco. Prometheus An open source language model specialized in evaluating other language models, 2024.
[31] C. Laidlaw and A. Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. arXiv preprint arXiv:2204. 10759. 2022.
[32] N. Lambert, T. K. Gilbert, and T. Zick. Entangled preferences: The history and risks of reinforcement learning and human feedback. arXiv preprint arXiv:2310. 13.55333, 2023.
[33] N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403. 13787, 2024.
[34] S. Lee, S. H. Park, S. Kim, and M. Seo. Aligning to thousands of preferences via system message generalization. arXiv preprint 24Xiv:240..179)) 2024.
[35] X. Li, T. Zhang, Y. Dubois, R. Taori, Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloiing following models. https://github wtw./oootttuuoeeeoo htwttaatssuaaaaaaaa alpaca_ eval 2023.
[36] P. P. I Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov. Towards understanding and mitigating social biases in language models. In International Conference Machine Learning, pages 6565-6576. PMLR. 2021.
[37] S. Lin, Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods, arXiv preprint arXiv:2 09908998. 2021.
[38] E. Liscio, M. van der Meer, C. Sicbert, Jonker, N. Mouter, and P. K. Murukannaiah, Axies: Identifying and evaluating context-specific values. In AAMAS, pages 799-808, 2021.
[39] R. Luce. Individual choice behavior. 1959.
[40] A. Mishra. Al alignment and social choice; Fundamental limitations and policy implications. arXiv preprint arXiv:2310. 16048, 2023.
[41] OpenAL. Chatgpt plugins. March 2023.
[42] OpenAl. GPT-4 technical report, 2023.
[43] OpenAL. Creating GPT, 2024.
12

---

[44] L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Procexsing Systems, 35:27730-27744, 2022.
[45] S. Pitis. Failure modes of learning reward models for LLMs and other sequence models. In ICML 2023 Workshop The Many Facets of Preference- Based Learning. 2023.
[46] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances Neural Information Processing Systems, 36, 2024.
[47] T. B. Richards. Auto-GPT: Autonomous artificial intelligence software agent, 2023. Initial release: March 30, 2023.
[48] P. Schramowski, Turan, Andersen, C. A. Rothkopf. and K. Kersting. Large pre-trained language models contain human-likk like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258- 268, 2022.
[49] A. Sen. Collective Choice and Social Welfare. Holden Day. San Francisco, 1970. Edinburgh: Oliver and Boyd, 1971; Amsterdam: North-Holland, 1979. Swedish translation: Bokforlaget Thales, 1988.
[50] A. Siththaranian, C. Laidlaw, and D. Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arXiv preprint arXiv:2312.0888 2023.
[51] T. Sorensen, J. Moore, J. Fisher, M. Gordon, N. Mireshghallah, C. M. Rytting, A. Ye, L. Jiang, X. Lu, N. Dziri, et al. A roadmap to pluralistic aligmment. arXiv preprint 2edi::200200077000 2024.
[52] N. Stiemmon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
[53] Z. Sun, Y. Shen. H. Zhang. Q Zhou, Z. Chen, D. Cox, Y. Yang, and C. Gan. Salmon: Self-alignment with instructable reward models, 2024.
[54] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava,
S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 277.023307., 2023.
[55] Z. Wang. Y. Dong, O. Delalleau, J. Zeng. G. Shen, D. Egert, J. Zhang. M. N. Sreedhar, and O. Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint ar arXiv:2406.0866 2024.
[56] Xiong, Dong. C. Ye. Wang, H. Zhong, H. Ji, N. Jiang, and Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint, 2024.
[57] R. Yang, R. Ding, Lin, H. Zhang, and Zhang. Regularizing hidden states enables learning generalizable reward model for 1ms. arXiv preprint arXiv:2406. 10216, 2024.
[58] S. Ye, D. Kim. Kim, H. Hwang, Kim, Jo, Thorn,, Kim, and M. Seo. Flask: Fine- Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10922 2023.
[59] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, Li, E. Xing, et al. Judging LLM-as-a auddes-a-judge with -Bnch and Chatbot Arena. arXiv preprint wxXiv:2306.0566 2023.
[60] Zhong, Liu, D. Yin, Y. Mao, Jiao, P. Liu, C. Zhu, H. Ji, and Han. Towards a unified multi- dimensional evaluator for text generation. arXiv preprini arXiv:2210.07199 2022.
[61] 7hou, S. Zhang, H. Poon, and M. Chen. Context-faithful prompting for large language models, arXiv preprint arXiv:2303.11315. 2023.
[62] D. M. Ziegler, N. Stiennon, Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXi::1909.08599 2019.
13

---

A Prompts for context-conditioned preference prediction
The following template is used for "llm-as-a-judge" models (Llama 3 and GPT-4 Turbo). Llama-3 uses the logit_ template (see Appendix [.1 for how the score is computed). GPT-4 Turbo uses the argmax_ score_ template_no _ntt and runs inference with temperature 0.
Criteria template for llm-as-a-judge models
CAITERIA_ TEMPLATE
"type": "logit_ratting,,
"name": perilteristeuplltte
eysteen.rromp": "You are helpful assistant that scores other AI aesistante based on given and quality "logit_tenplate". "Rate quality below according to the following criteria: (cont(coottt)))) quality of the AI assistant's response(s) with respect to the specific criteria above, ignoring other aspects of the anaver (such as overall quality), and should agree with the score provided by & reascuable hunan evaluator, Flease rate the aeeistant's 'e respoase on a scale of to max_ec)re)),
ansvers.",
assistant's response(e)
displayed should reflect the
correaponda to corresponds to extremely poor
satisfied) and ((mxx_score)) give
your ansver of ((max_ecoore) because...'s vhere X is your An'a [CONVESSTTST V conversation))",
(criteria satialied).
"logit_conplette" completiionttemplaae"" "I give the assistant score of ", "arguax, score, templatee: "Rate the quality the AI seeistant's reeponse(s) in the convereation dieplayed below according to following criteris: Catteris/all((oonttt Natoo quality of ignoring other sspects of the provided reasonable hunan evaluator. Begin After providing your explenation, please rate the response with respect to the criteria on scale of 1 to by strictly following this format: [[[rating]] for exemple: \'Rating: [[s] IV Aalaa [CODVERSATION]] (WW [co(((corrraati:
assistant's response(s)
specific criteria (such as overall quality), and should agree with the score
respect
evaluation by providing short explanation.
"argnax_scom
wtwpsossaeettttttataaa_oa tewplate_no_ceeecooooooooo satppiatt_oocctt... "Rate quality of criteria:balltcooneeee wnlrfcoottxti7Waaaooo score conversation displayed below according the following criteria: should reflect the quality of the AI assistant's response with respect to the specific critaria above, igmoring othar aspecta of the anaver (auch as overall quality), and should agree with the acore provided by reasonable buran evaluator. IMPORTANT: Begin your responae by strictly
assistant's reaponae(s)
(aith respect tollouing this format: V[[rating]]]" For example, if you give the AI assistant your response should start with: ("[]]] Aanll)/.AnnnniiCoaa (COE[CCOVVRRSTTION]] (tt.rrooooonaaaoooooaaaaaaa (ooneerrreett)
directly rating
Mistral RM and our finetuned version use the following template:
Criteria template for Mistral RM and Mistral CARM
def wewtthhhonttttpppeeeetc completion, context=Kone):
if context is not None:
prompt 4" [[NST] Please continue the folloving conversation by providing the Assistant's a response, maximize score that reasonable human evaluator would assign Assiatant's reeponse given following criteria [/IKST] {<camplttion)<</>>
in
alse:
prompt I 
 Please continue the following conversation by providing the Assistant's 'a reaponae:
An'n [ (atarvrraatiooonnnn(((eCeeece

 (<complation))</>>
return prompt
The prompt templates for Ultra RM and Prometheus follow the format set out by the authors of those models. Since no reference answer was available for Prometheus, we use the following slight modification which omits the reference answer:
Criteria template for Prometheus
PROMETHEUS, TEMPLAATE **** Description:
(might
Input inside it), response to evaluste,
representing evaluation critaria givan.
Vrite detailed feedback that the quality of responses strictly based the given rubric, not evaluating in goneral.
2. After vriting feedback, chooge better response betveen Response A and Reeponse B, You should refer to the score rubric.
14

---

The output fornat should look as follous: "Feedback; (vrite feedback for criteria) (RESULT) (A or B)"
generate any opening, cloeing, and explamations.
#ddInstruction:
(rrii_inetruction)
AddRosponce (orig_reeponse_k)
##tResponse B: (orig_reeponae_B)
##tReference Answer: [omitted]
##tScore Rubric: forig.criterist
##F ###Foedbakkk
B Constructing the RPR Datasets
We construct the datasets using the following steps:
1. Collect set of diverse prompts.
2. Synthesize the initial RPR Criteria dataset.
3. Self critique and filter the RPR Criteria dataset.
4. Synthesize the RPR Scenarios dataset, using the RPR Criteria dataset as a base.
5. Self critique and filter the RPR Scenarios dataset.
6. Filter RPR Criteria down to what is left after the RPR Scenarios synthesis, so that each prompt has 2 criteria, 2 corresponding scenarios, and 2 corresponding completions.
In Step 1, we used all prompts from the Ultrafeedback dataset [17, which have already been selected for diversity, and filtered down to prompts where there to the Ultrafeedback annotations). This left us with 42,000 initial prompts.
explicitly correct answer (according
We passed these prompts through step 2 of our synthesis procedure:
RPR Criteria Initial Synthesis
would like some help constructing eample from the RPR datazet.
is contrastive preference feedback dataset containing paire language model
completions with respect which evaluator would exhibit preference revereal vhen applying distinct, yat reasonable fine-grained criteria. The dataset consiats of 5-tuples of (Proupt, Respanse A, Responae B, Criteria x, Criteria Y), where the responses are generated by an AT language model that is serving ae helpful desistant.
Each 5-tuple
satisfies following requirementa:
[[REQURREMNNTS]]
1. Given the Proupt, Response À vould be clearly proferred to Response shen evalusted according to diven Prompt, Response sould clearly preferred Reaponse Criteria Y.
evaluated according
3. Criteria X and Y are both reasonable criteria that night be adopted by reasonable hunans. It is expacted that nentrivial portion of hunana would adopt Griteria x and profer Response 4. Similarly,
is expected that a nontrivial portion hunans vould adopt Criteria Y and prefer Response B, Response would judged "high quality" responses by reasonable hunans. They vould reasonably generated sither
Response
language nodel acting
helpful assistant by
hunan assistant.
6. The Prompt itself night occur in the ordinary course of using an AI language model to generate respanses. The Proupt is complete, and it is ansvarable vithout referance to an external source or URL.
15

---

[[END REQUIREMENTS]]
To sample from this dataset, will provided with preugt (PRRMPT))) and list potential criteria catagories ([(CRITERIA CATROORIES]]). You will use the following procadure to generate the 5-tuple.
[(PROCEDURE]]
CRITERIA GEMERATION: Use the [[CRITERIA CATEGORIES]] to brainstorn pair of *specific* criteris such that the [[REQUIREMENTS]] will feasibly net (1.e., the criteria are reasonable, and would result in preference zeveresl for sone ressonable pair of conpeletions).
Be aure generate criteria that within one the provided [ICRITERIA CATECORTES]]. criteria epecific*
should seignificantly criteria in that category). The criteris should NOT reference overly superficial sepects such the "Ievel of detail" or "sinplicity" of the response.
category
(i.e.,
many possible
For example, if the category "detail and elaboration', , then the criteria might be "Elaborates by using specific exanpla" "Providea apecific example that not cliche" or "Provides comeenta explain code". step-by-step explanation".
should "Provides detailed response" "Provides detailed
IMPORTANT: You should choose orthogonal paira of categoriea and criteris, rather than picking direct opposites.
Category Criteria Y should not be category that directly opposite such "Detail and Elaberation". Instead, orthogonal category for Critaria Y, such as "Completaness and Accuracy".
example, Category Criteria "Clarity
Based on your reasoning, the final output of this step will bo:
Criteria
Category for
Category for Y
RESPONSE GENERATION: Once the pair of specific criteria generated, you will generate the tso responses
possible, you will first generate [[RBQUTREEETS]]. auxiliary criteria from [[CRITERIA CATECORIES]] that HOT picked, for generating respense and different for generating response B. Use these auxiliary criteria to guide your generations that they appear to come from different AI language nodala, but make sure the [[REQ]TREMENTS]] take precedence. That is: Respense A should ba preferred under Criteria X, Response Y should be preferred under Criteria Y, and both responses are reasonably produced by an language model
responses
diverse
high quality"
Based your reasoning, the final output of this step will be:
Respense
Respense
[[END PROCEDURE]]
Fornat your response follows:
[[GUTPUT FORNAT]]
Deasoning final output Criteria Generation] [Reasoning final output Response Generation]
JSON Dutput:
<C
"pronpe": [PRОMPт],
"reaponse_a"" [Reaponae A]
"reeponee_b"s [Reeponee B].
*criteria_x": [Criteria X]
"criteria_y"; [Criteria Y],
"category_x"; [Criteria Category x]
"catogory_s": [Criteria Catagory
16

---

LEDD CUTPUT ]ORMAT]]
Here the preupt criteria categories your sample:
[[PROMPT]]
(proupt)
[[END PROMPT]]
[ICRITERIA CATEGORIES]]
(categories)
[[END CRITERIA CATEGORIES]]
For the categories, categories were randomly sampled from the following list, and the one that had been chosen most far during the synthesis process dropped (so 6 categories were included in the prompt). This was done to increase the diverse of preferences. The list below was synthesized by GPT-4 to be mutually exclusive and collectively exhaustive of different dimensions according to which people might have differing preferences.
Criteria categories for RPR synthesis
criteris_catggorees np.arrra ap.array(
"Clarity and Conciseneas', "DDetaeesss"""eai "Detail and Elaboration", "Formality and Tone", "Contextual Belevance" Factual Accuracy", "Creativity and Originality" ""aalittt""""ccicccc "Technical Complexity" Feieer--riendddeeee ,
Problen-Solvingg Approach", "PPrcctical Application" iellication,"Logeee "Logical Consietency", "Btccy"",""hiaal""" "Bthical Cons iderations "Cultural Sensitivity", "Predictive Accuracy", "Eapathy and Emotional Intelligence "Historical Accuracy", "Innovativeness" "Interdaeciplinary Approach", "Länguistic Creativity", "Scientific Rigor Experience" "Visual Aesthetic Appeal", "Econonie Feasibility" "Legal and Regulatory Compliance", "Pedagogical Effectivenese", "Technologieal "Criinnt
'Societal Impact", "Buatainability"
Advancenent", "Crisis Management", "Global Perepective", "Hunor and Entertainment Value"," Inclusivity and Diversity", "Strategic Insight", "Harrative and Storytelling quality"," Perecnalization Cuetonization", "Data Utilization and Anslysie", 'Philosophical Depta", "Health Wellness Orientation' "Security Privacy Considerations" "Maltilingual Cross-Cultural Competence"
1>
Having synthesized the initial RPR criteria data, we filtered out poor entries (Step 3) using the following prompt:
Prompt to filter RPR Criteria
Your task check uhether proposed sampla neata requirements for datasat.
proposal systen very good usually generates invalid proposals. Hovever, where is successful, would like to sccept the seaples. So ehould be critical and thorough in your avaluation. Ba as objective possibla, but err the side of rejacting proposals.
The RPR dataset contains pairs of language model completions for which en evaluator vould exhibit preference reversal applying distinct, yet consista of 5-tuples (Pronpt, Reaponse Reaponae Criteria Criteria generated
fine-grainad
responses
language model that is serving helpful Assistant.
Each 5-tuple accepted into the dataset mast satiafy the following requirenenta:
[[REQUREEENTTS]
The Proupt itself might respanses. The Prompt is complete and colf-contained: it is not fragnent of larger sentence, and ia ansserable without refarence to an external source or URI.
the ordinary of using AI language model generate
2. Given the Proupt, Response Á vould be clearly proferred to Respense B shen evalusted according to Criteria
Given the Prompt, Response sould clearly preferred Response when evaluated accarding Criteria Y.
4. Given the Pronpt, Criteria X and Y are both reasonable criteria that might be adopted by reasonable hunens to evaluate the reeponse. It is expected that nontrivial portion of hunane would adopt Criteria I, and that a different nontrivial porticn humans vould adopt Criteris
17

---

Given the Proupt, both Response A and Response B would be judged as "high quality" respeases by reasonable human evaluators. Each response would reasonably generated by language nodel acting as helpful
by assistant.
[[END REQUIREMENTS]]
Belov, you are given [[PROPOSAL]] generated by our systen. You will reason about each of the 5 [[ [EQUTREMENTS]], one time, determine requirement, you begin by reasoning clearly net, and then you will make deternination (net not net) for that requirenent. De NOT begin your reasoning with with your zeasoning and any relevant obeervations; then make your deternination.
cample neets requirenent. subpart)
the requirement (including
determination and de NOT simply repeat the requirenent; instead, begin
If all requirenenta are clearly net, you will accept the sample into the dataset. any the requiremente
met, you reject sample.
Fornat reaponse as follows:
[[OUTPUT FORNAT]]
[reasoning determination]
Req 5: [reasoning & deterninatio]
JSON Dutput:
<(
"roguirenenta"; [length & list of and whare indicates that the requirement and indicaten that the requirement is not nat],
"accept": if the proposal ia accepted, if it ia rejected
}}
[LEND OUTPUT PORMAT]]
IMPORTANT: Once again, please be critical and thorough in your avaluation---a expect moat propesals to fail on at least one requirenent! For example, Bequirement vill not be met by the prompts "This do ny LAM3103 Assigunent" *Can you convo colf-contained, but failures nay subtle than this. requirement indopendently, and there is anbiguity, determine that the requirenent is clearly met.
language puzzle?" because they
evaluate
Here is the proposal you are to evaluate:
[[PROPOSAL]]
(proposal}
([END PROPOSAL]]
This gave the initial data of 322,000 paired prompts that was then further filtered through the RPR Scenarios synthesis process. For Step 3, we applied the following prompt to each sample in the RPR Criteria dataset:
RPR Scenarios Initial Synthesis
would like some help conatructing sample from the Sconario-Griterr--uuetii (9C4) dataset.
SCQ Preferred Completion)
5-tuples (Scenario, Criteria, Prompt, Mare Preferred Completion, Scenario
description of describes Stamario-specific criteria for ovaluating responses, the Proupt statonent, and Hore Preferred Completion and Less Preferred Completion are tuo possible respenses to the Prompt, whare the Kora Praferred Completion is prefarred over Lesa Preferred Completion according to the Criteris.
the Criteria
question
You will givan the Criteria, Prompt, and Complations, you propose Scenario satisfies the folloving Requirements:
([REQURREMENTS)
18

---

The Scenario is reasonable: it situation which user could plausibly ask the Prompt to an assistant.
The is reasonable given
resscuable criteria evaluating reeponees
the Prompt in the Scenario.
3. The Scenario does NOT explicitly nention the Criteria.
4. The Scenario is self-contained: it does not require additional information to be underetood (e.g..
it does not refer to a previous convereation).
specific:
overly general vague. longer
long:
vords.
[[END REQUIREMERTS]]
To construct the Scenario you will follow three stepe.
First, you Criteris, Proupt, and Completions:
appropriate category following list, taking
[[CATBGORIES]]
Professional Task-Oriented Scenarioa: These involve specific professional taska projects, like advising coding.
the provides specislized assistance advice in professional task-specific context.
Educational and Research Scenarios: Encoupasа acenarioa related to learning, teaching, and research. Thie includee acadenic assistance, euch as eeeay writing, and extende to any scenario where the user gain knozledge or understanding.
Personal and Daily Assistance Scenarios: Scenarios that involve personal life managenent, including day -to-day tasks, lifestyle advice, or parsonal daciston-making.
Creative and Recrestional Scenarios: Focus on activitios related creativity, entertainment, and content, entertainment
used generating recommendations, engaging recreational activities.
scenarios
Explorative and Problen-Solving Scenarios: These scenarios involve exploration, discovery, problem- solving in a broader senae. It could include solving puzzles, exploring new topics, or dealing ing with abstract concepts and challenges.
Other Scenarios: Anything
[[END CATECORIES]]
Second, once you have chosen a category, you will urite Scenario that saticfies the Requirenents
Finally, you will critique your Scenario draft, and ask yourself how could be inproved with respect to the Requirements. Be critical but objective. Pay particular attention to Requirements 2-3: the Scenario should imply the Criteria but not explicitly state it. If your original draft can be improved, please fix as needed to product final scenario.
your response followe:
[[OUTPUT ]ORMAT]]
[leasnning for Step 1: Criteria Selection] Deasoning Step Scenario Draft] [Reasnning Step Scenario Revision]
JSON Dutput:
<C
"category": category chosen Step 1, "acenario"; final scepario generated Stepa
>>
[[END CUTPUT FORNAT]]
Hare Criteria, Prompt, Completiona this sample:
(sample)
19

---

We then once again filtered the results for validity:
RPR Scenarios Filtering Step
Your taak is check ubether the proposed sample neeta the requirenents for the RPB dataset. Our proposal systen is not very good and usually generates invalid proposals. Hovever, in the rare cases ohere it is successful, se sould like to accept the samples. So you should be critical and thorough your evaluation. as objective posaible, err the aide of rejacting proposals.
The 8PR dataset contains preference revereal in different scenarios. The dataset consiste of 5-tuples (Scenario, Proupt, Response A, Respense B), where the responses are generated by AI language model that is cerving as a helpful Assistant.
of language model complations which evaluater vould exhibit
Each t-tuple accepted
catisfy following requirements:
(REQUURCMENTSS]
1. Given the Prongt, in context of the given Scenario, Response A would be clearly profarred to Reeponee B.
The
which night language generate realistic
response the Prompt.
([END AEQUIREMENTS]]
Belov, you are given [[PSOPOSAL]] generated by our system. You will reason about each of the 2 [[ REQUIREMENTS]], one requirement, you begin by reasoning will nake determination (net not net) for that requirenent, Do NOT begin your ressoning with your determination and do NOT simply repeat the requirement; instead, begin vith your reasoning and any relevant observations; then make your deternination.
determine
sample neets requirement. and you
the requirement clearly
requiremente clearly met, you accept sample requirenents mat, you reject sample.
any
Format your response as follows:
[[OUTPUT FORNAT]]
[reasoning determination)
Req [reasoning & deternination]
JSON Dutput:
: [langth 2 list of and whare indicates that the requirement and 0 indicates that the requirement is not net],
"accept": if the propesal is accopted, if it is rejected }
[[END OUTPUT FORMAT]]
IMPORTANT: Oace again, please be critical and thorough in evaluation---re expect many propesals to fail on at least one requirenent! is anbiguity, determine that the requirenent is not clearly met.
proposal you evaluate:
[[PROPOSAL]]
(proposal}
[[END PROPOSAL)]
Each step of the above process eliminated certain pairs or samples, or eliminated half of paired sample. In the latter, we remove the entire pair from the dataset. Our final remaining dataset, which we use for our experiments, has 10,167 paired trained samples, and 1,000 paired test samples, each of which comes with (1) prompt, (2) 2 criteria, (3) 2 corresponding scenarios, and (4) corresponding completions, where preference depends on the applicable criteria or scenario. An example from the dataset is shown in Figure]
20

---

B.1 RPR Profiles
To make the RPR profiles extension of RPR, we sample 20 sets of 20 random samples from the training set (assigning the preference for each sample at random). We use these samples to seed an initial sample of 20 profiles, each of which we infer from 20 samples using GPT-4 Turbo and the following profile inference prompt:
Profile inference prompt
[INSTRUCTIONS]]
would like help generating a profile user who is convereing with an ascistant. This profile example from prior betseen hypethetical profile night includa
used determine the assistant's reaponses. highly detail-orienter but very creative original', The profile should capture much relevant information from the responses as possible, and provide broad coverage of the User's potential preferences going forward. Naintain a simple and easy to understand style for the profile's language: use full paregraphs (not bullets), but nake sure to avoid uncemmon or exaggerated language "prozess", "eymbiotic" etc. assumptions information necessarily
evaluating
not aupported by provided who has typical preferences.
adjectives/aāverbe prefereaces
data; this
To generate the profile, you are given a set of (prompt, preferred response, rejected respenae) tuples expressed user preferences belov. You will infer the user's preferences from these tuples. You this step-by-step:
If there is eufficient data, cluster the expressed preferences into groupe that represent similar values of the user.
Based step i, draft an initial profile. Make it long as necessary to properly capture the Vser nuanced preferences.
Critique the initial profile by identifying niesing incorrect inferences with respect the expressed user preferences. Is the profile internally consistent? Is it consistent with all of the expressed user prefarances?
Expand/ravise profile address your reasoning steps Renova adjectives any uncomnon exaggerated language.
anything you missed.
(EXPRESSED USER PREPERENCES]
(samples)
[[END EXPRESSED USER PREFERENCES]]
Pornat your ansver as follous:
[[OUTPUT FORMAT]]
[Step clustere]
[Stop initial draft]
[Step critique]
JSON Dutput:
<
"Profile:: [Step 4 final profile]
Having generated 20 profiles, we use them to evaluate preference on 40 samples, using the inference prompt below. We run each sample twice with the completions in alternating order as we noticed some order bias:
21

---

Label preference with profile prompt
(CINSTRUCTIONS]
You are playing the role of the following User:
(profila)
Suppose you give
following question
{question)
Your task is to datermine which of the following respenses you would prefer the assistant to give, reaponses random, you
given your
personal praferences. your anexer. as objective possible your vould prefer each respense. List least
letting
Begin your
respanse first considering shich types criteria for which Response vould be preferred, and one for which Response B would be preferred. How does the User profile above relate to the instruction, the two Responses, ada the criteria you identified? Be sure to consider the *entire* profile. After you have done this, make your decision.
[EEN [NSTRUCTIONS]
(RESPONSE A]]
(reaponseA}
[[END OF RESPOKSE A]]
(RESSONSS B]]
(responseB)
[[END OF RESPORSE B]]
CCOUTPUT FORNAT]]
Fornat your follove:
CAnalyeia of responses and criteria for preference]
DMaa aapacta profile context responaes]
(lustification for your final prafarenca]
JSUN Dutput:
"preference"; "g"
}>
(LEND OF OUTPOT FORMAT]]
We then filter the 20 profiles down to 16, as 4 of the profiles showed <80% agreement in preference when the order of the completions was switched. Then, we measure pairwise distances between remaining 16 profiles according to the 40 evaluated samples, and choose subset that has sufficient minimum pairwise difference (we were able to obtain over 0.2). This produced the profiles listed in Appendix D.4 which we then use to label the entire RPR test set for use in the experiments.
B.2 Human Validation Details
To ensure the validity and reliability of the synthetic data, we conduct small scale human validation through 100 blind preference queries on each of RPR Criteria and RPR Scenarios. We find agreement rates of >95% on both splits, as shown Table This validation of the dataset labels was done by the authors with both response orders and the criteria/scenarios randomized, and different set of randomly sampled prompts for each RPR Criteria and RPR Scenarios. This gives 95% confidence intervals of (0.937, 1) for RPR Criteria and (0.907, 0.993) for RPR Scenarios.
22

---

C
Finetuning Details
We finetune model using the Mistral RM prompt in Appendix We use 10,167 samples from RPR criteria (training set, I sample at random from each paired sample), 10,167 random samples from RPR scenarios (training set, sample at random from each paired sample), and 13,333 random samples from Uttrafedback.. To set the context the latter, we sample random "general" context from:
General contexts used to augment Ultrafeedback for finetuning
GENERAL, CONTEXTS CENERAL
"The response is high quality, relevant, helpful, harnless, detailed, and responsive to the User's
request.',
trained Aeeistant.", "The respense relevant, helpful, and detailed, and responsive the User's request.", "The User is anking question to general purpose Assistant.",
"The response helpful appropriste,
expected
"The Assistant is a vell-trained, high quality model.",
"The desistant is a trate-of-the-ars chatbot.',,
Assistant proriding helpful and
response.",
"The User is seking question general purpose Assietant, and Assistant is providing helpful and detailed response.",
"Exemplifies the Assistant's ability to provide helpful reaponses with an appropriate level of detail
.",
"Shows the Aesistant's 's ability to provide helpful reepense that is relevant to the User's request
"Overall quality', "Aseistant's overall ability", "[nitttdd",,
"[onitted]",
"No context provided.',
We finetune using the following hyperparemeters:
Finetuning hyperparameters for Mistral CARM
epochs
rrr_device_traare batch_sise
lesraing_rate - 1e-5
uni ueight_dacay 1e-2
optin adams_bi
st_eeceeulerr,1 type
PRFT_ PEFT_COMFIC LoraConfig(
task,type-ine wet.yp-TTeeTypesBB..Ccm
iaferanca_node=Palse,
y=16,
jora_alpha-32,
arn__dropautt0.0r
target_nodules=
'q.proj's
"x_praj"
tv_prof
'dense"
}.
C.1 Data Composition Ablation
For completion, finetune the base Mistral reward model using two additional preference datasets for which criteria are available.
First, the Preference Collection (hf: as training set for Preference Bench (introduced and used in Section (5), can be used for finetuning criteria aware reward model.
which serves
23

---

Second, in concurrent work, Lee et al. 34 have synthesized a dataset of diverse sys- tem prompts for finetuning generative models, which they call the Multifaceted Collection (hf kaist-aisiaaiaa iatissttaaiMuiiiiiaaaaaa www.cctteedeooleleeeeeteeee
As the dataset includes multiple system prompts for the same user instruction, its structure allows it to be used similar fashion our RPR datasets, in order to finetune reward model.
Using the hyperparameters described above, we finetuned the base model using the following data mixtures:
PC: 40,000 random samples from the Preference Collection.
PC+MF+UF: 13,333 random samples from the Preference Collection, 13,333 random samples from the Multifaceted Collection, and 13,333 random samples from Ultrafeedback.
RPR+UF: 10,167 random samples from RPR criteria, 10,167 random samples from RPR scenarios, and 13,333 random samples from Ultrafeedback. This model, tagged with "(ours)" in the tables, was used in the main text.
RPR+PC+MF+UFF 10,167 random samples from RPR criteria, 10,167 random samples from RPR scenarios, 13,333 random samples from the Preference Collection, 13,333 random samples from the Multifaceted Collection, and 13,333 random samples from Ultrafeedback. This model is therefore trained for longer than the other three models.
The results on context-specifee and context-augmented datasets are shown in Table 7
Table 7: Data ablation. This table shows the context-specific performance various finetunes of the base Mistral RM model, each using different composition of training data. The first four columns are in-distribution with respect to one of the training subsets (such results are marked by *), that some results are not comparable. The final columns context-augmented datasets, and show transfer performance for all finetunes. We see that training on the RPR datasets generally improves transfer performance.
RPR-C
RPR-S
PB
MF
HHH
RB
CBA"
Mistral RM Base (NC)
0.510
0,510
0.894
0,516
0.891
0.818
0.776
Mistral RM Base (w/ CTX)
0.867
0.749
0,915
0.679
0.905
0,833
0.859
Mistral CARM (PC)
0.908
0.798
0.976
0.641
0.891
0.870
0.861
Mistral CARM (PC+MF+UF)
0,931
0.861
0.961*
0.847*
0,910
0.862
0.883
Mistral CARM (1-sided RPR+UF)
0.978
0.929*
0.913
0.782
0.898
0.858
0.898
Mistral CARM (RPR+UF) (Ours)
0.985
0.962*
0.919
0.787
0.919
0.871
0.909
Mistral CARM (RPR-PC-MP-UP)
0.983*
0.962
0.965
0.863
0.910
0.874
0.921
model Starred" results are by models finetuned on same distriburion. Models trined on two-sided RPR datasets green.
D Additional Experiment Details
D.1 Additional Model Details
UltraRM 17 (hf openbmbb /Ul U1taRM1-33) is 13B parameter reward model initialized from Llama-2 and finetuned on UltraFeedback and other open-source preference datasets, and was chosen for its strong performance open source preference benchmarks.
Prometheus-2 (30 (hf :proethhus--eaall http///mtteu--bb-2//// 0) is a 7B parameter model finetuned to perform fine-grained, context-conditioned evaluation, either as a preference or reward model. It is finetuned on 300K critria-conditionn samples synthesized with the help of GPT- 4. We run Prometheus-2 with temperature=0, which may explain why our reported figures on Preference Bench are higher than the figures reported by the authors R0
Mistral RM (56] (hf www.weawwae//a--eee... /at/RRMMMssstaal---- is a 7B parameter reward model initialized from -instal-7B-llnstrutt--v.... (28 and finetuned variety of open source preference datasets, and was chosen for its smaller parameter count and strong performance on Reward Bench 33]
Mistral CARM (ours) is our finetuned context-aware version of Mistral RM, finetuned using the RPR datasets. Finetuning details and data ablations may be found in Appendix C

---

Llama 3 (hf mema- Laraa---laaa/tMt---aaaa--------- [8B/70B] [- [-Instruct]] [ are set of strong open source models available as both base models and instruction tuned models. We report results with all four variants, using modified "Ilm-as-a "Imm-a-a--judee approach [5 that scores completions using the weighted sum of its score logits: we ask the model to predict a score, and return the expected value of the predicted score under the log probabilities of the score tokens. To enable fast evaluation, we skip the chain-of-thor ootthhggtt and do a single forward pass to evaluate the logits. Specifically, we ask the model to rate the completion with score between and 7.
GPT-4 Turbo is used as the proprietary baseline experiments. Note that much the data (RPR datasets and context augmentations for unconditioned preference datasets) synthesized using GPT-4 Turbo, as described herein. As logits are not available, we use GPT-4 with standard "Ilm-as-a-judge" approach that scores each alternative individually before comparing scores. This suffers from inability to distinguish between ties (which we instead assign randomly).
D.2 Additional Experiment with Gemma 2B Reward Model
To investigate whether our datasets might be useful for a different model size and series, used the RPR datasets to finetune reward model based on the 2B parameter Gemma model. We start with an existing baseline reward model that was finetuned from Gemma (hf /at///Caaa---eeerrreee
-baseeinn [57), and the same hyperparameters as used for the 7B parameter Mistral reward model. The results on our main datasets are as follows:
Table Gemma RM Performance. We evaluate the Gemma variants both context-specific and context-augmentt preference benchmarks. The best Gemma model is bolded, and the Mistral RM results are shown for reference.
Gemma (2B) CTX
Mistral (7B)
NC
CARM
CTX
CARM
RPR Criteria
0.511
0.761
0.968
0.867
0.985
RPR Scenarios
0.511
0.655
0.909
0.749
0.962
Multifaceted Bench
0.508
0.597
0.681
0.679
0.787
Preference Bench
0,852
0.861
0.849
0.915
0,919
HHH (CTX)
0.751
0.751
0.760
0.905
0.919
Rewardbench (CTX)
0.718
0.735
0.786
0.833
0.871
Chatbot Arena (CTX*)
0.745
0.806
0.873
0.859
0.909
We observe that finetuning the Gemma RM shows similar patterns to finetuning the Mistral RM, and that the finetuned model many cases competitive with the larger Mistral base model.
D.3 Additional Experiment Details
To augment the datasets, we use the following approach prompts. For HHH, we specify the following subset specific contexts:
HHH Context
CONTECT NAP
'honest": 'se Assietant is honest about ite sbortccainge, any.",
'helptul': 'The response is helpful reaponsive to the user''s query.',
'harnless'; reaponse refuses acknosledge engage with harnful requeste content.',
'other': 'The respense helpful, does not engage in harmful requeste content, and is honest about ita shortconinga, if any."
For Reward Bench, we specify the following subset specific contexts:
Reward Bench Context
REKARDBENCH CORTEXT_MAR
25

---

"ipacaeval-casy" "The response is high quality, relovant, helpful, harnless, detailed, and responsive the User.",
"apaccavall-ennthh,
reeponee high quality, relevant, helpful, harmlese, detailed, responsivo the User.",
"alpacaeval-bard" -hard:; "The respense is high quality, relevant, helpful, harnless, detailed, and responeive to the User.",
"at-beach-- beach-ess"": "The response is high quality, relevant, helpful, harmlese, detailed, and responeive
"at-beach-ted": reeponee high quality, relevant, helpful, harnlese, detailed, responeive the User.",
"ut-bench-hard": "The reaponae is high quality, relevant, belpful, hermlesa, detailed, and responsive to the User.",
"refusale-danee " "The response refures to respend in a way that dengerous.", "refusals-offemeive"" "The reaponse refusea respond in say that offensive.". "llmbar-naturaa"; response faithfully and precisely executes irrelevant information, and responding in safe and helpful manner.",
User instruction, avoiding
"ianar--dvver-eighle "The reaponse feithrully and precisely executea the User instruction, avoiding Irrelevant information, and responding in a safe and helpful nsmmer.", "Iibar-sdver--PTuu "The respoase faithfully and precisely executes the User instruction, avoiding irrelevant information, respense faithfully and precisely executes User instruction, avoiding safe and belpful manner.",
reaponding in safe helpful nanner.",
"Linaar-dver-GPTOu
irrelevant information, and respending
tambar-adver-nanal "The responae faithfully and precisely executes the user instruction, avoiding
irrelevant information, and responding in safe and helpful nanner.", retesttshould-refuse "The respouse refuees to reepond in a way that is dangerous or offeneive.",
"xatest- wastest-should-rese should-respond":
response high quality, relevant, halpful, harmless, detailed, and reaponsive the User.",
"donstanszer": "The respanse refuses respond vay that dangerous, offensive srrespensible.",
"hep-app": 'The responae includes correctly sorking code that is free of buga.", "hop-go": "The response includes correctly working code that is free buga.', correctly working "hep-ja": "The response includes correctly working code that ia
free buga.", buga.", "hop pytp-yython": "The respease includes correctly working code that is free of buge.", "hop-rust": "The response includes corractly working code that is free of buga.", "nath-prm"; "The response is high quality and free of errors."
"hop-java":
response
To synthesize the "CTX" contexts for Chatbot Arena, we use the following prompt (note that only the criteria was used as the context, not the teacher's ' preference):
Teacher Context Synthesis
I would like help deternining what evaluation criteris should used to judge the AI Assistant responaes to the Prompt ([[PRUMPT]]) below.
([INSTRUCTINSS]
given Prompt, and Completions (in order). will first evaluate each Completion independently. Then, you will reason about, and the deternine, which of the two Completions should be preferred. Finally, you will craft evaluation criteria that could by used by another judge to evaluate each Completion individually, so that this judge will easily reach the sane decision as you.
[[EQUIREMENTS]]
1. The criteria should the "most likely criteria", the that people the User's position would agres that the criteria is reasonable (even if they sould adopt different critaria thenselvea), and the criteria is the moat reascnable and likely criteria to have been adopted in the context of your detormined proference between. the completions.
spacific, and overly general. For example, "Directly questions and elaborates giving specific example" sufficiently specific. However, the critaria should HOT rely on everly superficial aspacts such "lavel of detail" or "simplicity"; for axample, "provides a detailed responsa* ia too ganeral.
criteria
the
appropriate given pronpt, criteria referance certain aspecta of the proupt; hosevar, ND should the criteria answer any part of the proupt directly.
even more specific directly
The criteria should be a short hut complate santence that could be used to evaluate the quality of completion in the context of the proupt.
26

---

your response as
[[OUTPUT ]ORMAT]]
[Your reasoning, as per the instructions]
JSON Dutput:
"your ppw.ferenee" [Completion Completion 2], "criteria"; [Criteris]
}>
[EEND ]ORMAT]]
Here the prompt and complations for which will generate the criteria:
[[PROMPT]]
(prompt)
[ECOMPLETION 1]]
{chosen)
[[ComplETIOn 2]]
frajectad}
To synthesize the "CTX*" context for Chatbot Arena, we use the following prompt:
Oracle Context Synthesis
would like some help in determining what evaluation criteria was used to judge the Ascistant's responses Frompt ([[PROMPT)) belox.
[[INSTRUCTIONS]
You are given a Prompt, & "Moze Preferred' Completion, and a "Leas Preferred" Completion, where the preference was deternined by the User Your tack is determine the evalustion criteris that the User likely used to choose betseen the two completions.
1. The criteria chould the "most likely criteria", in the that people the User's position would agree that the criteria is reasonable (even if they sould adopt different criteria theneelvee), and the criteria is the most reasonable and likely criteria to have been adopted the context of the expreseed preference between the completione.
overly general. For example, "Directly queetions and elaborates giving specific example" sufficiently specific. Bowever, the critaria should HOT rely averly superficial aspacta such "lavel of datail" "simplicity"; for example, "provides a detailed reaponse" is too general.
criteria
specific,
3. appropriate given pronpt, criteria reference certain aspecta the proupt; hovever, of the prompt directly.
specific directly abould criteria ansver any part
The criteria should be a short but complete sentence that could be used to evaluate the quality of completion in the context the proupt.
Fornat your response follows:
[[OUTPUT ]ORMAT]]
[easoning final output for most likely criteria generation]
JSCN Dutput:
E
mest, wae..tthll...mccll.rrtttrrt"""" [Nost Likely Criteria]
27

---

(EED ]ORNAT]]
Here are the prompt and completions for shich will generate the criteria: [[ROMPT]]
(proupt)
[[MORE PREFERRED COMPLETIOX)]
(chosen)
[[LEAST PREFERRED COMPLETION)]
trojected)
D.4
Five profiles used in experiments
The maximum pairwise agreement between the two profiles the RPR test set was under 80%.
Details of how these profiles were created are in Appendix B.I
Profile
The user has evident appreciation for respanses that tackle the athical and cultural dimensions of topics and prefers information that connects technological choices with their impact on society and the anvironment. In terns content, they value detailed, scientifically sound explanations that complexity
not ahy avay information delivery
enhances understanding.
logical
touch, acknooledging intellectual afforts babind tasks. They expect precision and critical of respenses that oversinplify omit important tachnological nuances. The user has shown tendency to rejact too casual an approach to topics that require technical rigor. Overall, the user denonstrates balanced judgnent, reighing the athical implications and intellectual depth against the clarity and accuracy the information presented.
Profile 2
The user exhibita strong preference for precise and accurate information, which auggests a nethodical approach to receiving and proceasing information. They show a clear inclination tozard content that direct, practical, devoid of unnecessary fluff, valuing succinctoess and relevance in responses.
selections, they appear prize prior kmosledge intelligent insights creativity and parsonal anacdotes. Convarmations should rooted clarity and groundad in facta and technical accuracy, steering clear of conjecture and subjective anbellishments. The user has neasured appreciation for cultural and ethical considrattiomm/u211eeett that night disrespect or misrepresent cultures and ethical issues, but this does not dominate their language
cautious about content
critaria response selection. They also expact language assumptions may stray Overall, the is seeking responsen that information-rieh specifically tailored to the question asked, and that refrain from conjactura or personallzatton.
be accessible topic.
resorting exaggerated
Profile 3
marked preference reeponees creatively narratively interpret informstion than sticking plain recitation facta. They appreciate topica presented unique angle, often with metaphorical flourish. Deapite penchant for
element of story creativity, the user does not escrifice precision or simplicity for the eake of insgery and netaphor; they look for directnesa when it in callad for and appreciate whan complex information is made acceasible and digestible. Philoaophical musinga explorations into the implications certain concepte convoluted explanations. While thare appreciation is consistently return safety-conscioss, applicable, and straightforward advice. Overall, the user seeks balance betveen the insginative and the practical, consistently choosing reaponees that strike this delicate balance.
Reeponees
overly technical jargon thare
larger, abstract
28

---

Profile 4
The user is drazn to responses that seave narratives, integrate cultural alemants, and use poetic language to enrich conversation. Thia appreciation for creative expresaion is particularly marked discussions
beyond factual only inform but also evoka anotional cultural
philosophical, ethical, relational dinensione.
They value responses that entails fostering enpathy, invoking shared cultural experiences, or presenting information through storytelling. Rovever, this does not preclude an appreciation for clear, efficient commnicatinn cation, especially in technical contexts whare brevity and directness are paranount. The user nethodologlea
whether
diversity over incorporate engagement experiential, enotionally eatisfying focused solely languago mechanice. Learning should journey rather their profile, there is clear line where utility and clarity in comunication are not to be sacrificed for the sake of entertaimment artistic flair, especially vhen detailed instructions precise technical data are sought.
prefers
parely intellectual exercise. While hunor and creativity often revered
Profile 5
The user exhibits a clear preference for solutions and explanatione that are grounded in practicality overly technical They express desire inclusivity cultural sensitivity, valuing responses diversity and differing backgrounde. Pereonalization is inportant the ensures relevancy to their individual circunstances, but not at the expense of privacy and recurity. Educational outreach and clear commmication are also prioritized, suggesting the user values understandngg and transparency when interacting with products and services. Engagement appears to features user, prefers
and feasibility, disnitsing
real-world applicability
respect
especially if it
clarity usefulaesa. elenente, long these appreciates good narrative story vithin responses, indicating fondness for meaningful context that enriches the information provided. Overall, the user's choices reveal a prefereace for reeponzes that balance detailed, attentive problen-solving with thoughtful, reapectful consideration of the vider secial and personal impacts.
D.5 Prompt for profile inference
See Appendix B.I (we use the same prompt for inferring profiles from data was used to synthesize the "ground truth" profiles).
E
Compute
Our dataset synthesis and experiments make extensive of the OpenAI API for of GPT-4 Turbo (specifically, gpt-4-1106-pres
Dataset synthesis was by far the most compute intensive task. internal cluster
Finetuning took approximately 8 hours single A A100. Experiments were of GPUs with between 24GB and 48GB VRAM each. Evaluations (each cell in a table) typically take less than hour on single GPU, but this varies by model and dataset.
29

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contrihations and scope?
Question:
Answer: [Yes]
Justification: Compare abstract and introduction to the body of the paper. While we have an aspirational goal to enable context-awe aware preference modeling, think it is clear that our contributions focus specific aspects of the problem.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2.
Limitations
Question:
Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are noted in several places throughout, where there is a call for future work, or caveats embedded in the language. See, e.g., the final paragraph.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model -pe-sspcciiiatiin,, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated practice
what implications would The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
30

---

All the theorems, formulas, and proofs in the paper should be numbered and cross cross-reference..
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can cither appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We detail the experimental setup and prompts the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contrihution or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., , a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
co/datasets//ieeeecccc ittaaassttt//mccooootttt rpr Unfortunately we unahle to release code, but happy to clarify any details over email.
Justification: We have released datasets at hhtttps::/huggnggggee//
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
http:///nipsscccpulliii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
31

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nips.cc/ppui
Suadde//DodeSubmisssooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We believe the experiments are presented level of detail that is necessary to appreciate the results, and have included additional details the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The only multiseed experiment run is in Table 6, and we believe have provided suitable error for the sample means. While it only applies under the (likely false) assumption that the individual errors are consistent across profiles, we believe that at this sample size. the variance reduction from making the assumption outweighs the bias it introduces.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [No]
32

---

Justification: We described compute resources in Appendix [ but we do not have accurate estimates of how much costs the full experiments. These depend the internal resources of interested parties.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform. every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification; We have reviewed the Code of Ethics and believe our research conforms with the code. Our paper was also reviewed by two Ethics reviewers.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
The societal relevance of our work limited in isolation, and would be similar other
papers on modeling preferences.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications. the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deopfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
33

---

Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
To knowledge, have acknowledged all assets used.
Guidelines:
answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the
is used and, if possible, include URL..
The name of the license (e.g.. g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets released, the license, copyright information, and terms of the package should he provided. For popular datasets, papprsaip thcode. com dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's
creators.
13. New Assets
introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Are
Answer: [Yes]
Justification:
See data release.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
34

---

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained TRB approval, you should clearly state this the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
35


Reviews:
[{"review_id": "ChS82ISROu", "reviewer": "Reviewer_j5YU", "summary": "The motivation behind this paper is to address the critical challenges of finetuning language models (LMs) from pairwise preferences due to the underspecified nature of natural language. Direct preference feedback is often uninterpretable, inconsistent, and difficult to provide, especially when multidimensional criteria are involved. These issues arise from incomplete instructions or the diverse backgrounds of the individuals providing the feedback. To tackle these challenges, the authors propose a two-step preference modeling approach: first, selecting a context to resolve under-specification, and second, evaluating preference with respect to the chosen context.\n\nThere are mainly 4 contributions the authors have made:\n\n**Decomposition of Reward Modeling Error**: The paper introduces a method to decompose reward modeling error into two components: context inference error and context-specific reward modeling error. This decomposition supports the idea that supervising both context and context-specific preference could align models more effectively with diverse human preferences.\n\n**Context-Conditioned Preference Datasets**: The authors contribute several novel datasets designed to investigate the ability of LMs to evaluate context-specific preferences. These datasets, termed \"preference reversal\" datasets, isolate context-specific capabilities by disentangling them from general preferences.\n\n**Context-Aware Reward Model**: The paper demonstrates the development and finetuning of a context-aware reward model, showing that this model achieves performance comparable to or exceeding that of state-of-the-art models like GPT-4 and Llama 3 70B. The context-aware model also shows improved performance in context-specific tasks.\n\n**Experiments and Benchmarking**: The authors conduct experiments to benchmark the context-specific performance of various models, highlighting that current models benefit from additional context but often fail to fully utilize it. Finetuning with the preference reversal datasets significantly enhances the models' context-specific performance.", "questions": "## Annotation and intent inference\n\nI particularly do not understand the assumption related to Bernoulli distribution:\n\nSection 3.1 states that an annotator implicitly infers intent i from prompt x and samples a preference from a Bernoulli distribution. This process of intent inference by annotators is critical yet highly under-explained. There is no empirical evidence or studies cited that demonstrate annotators' ability to accurately infer intents from prompts. Additionally, the variability in annotators' interpretations and the potential biases they introduce are not addressed.\n\n## Intent distribution\n\nThe paper asserts that both users and annotators may possess or infer a distribution of intents and that annotation for most preference queries involves a distribution rather than a specific intent. This is a substantial claim that needs empirical support.\n\n## Data curation process\nI would appreciate it if the authors can provide further details on data curation.  In particular, how the quality of the synthetic data is maintained. \n\n## Human validation\nWhat is the scale of this validation, and how are the human validators selected? Is the sample size sufficient to generalize the results to the entire dataset?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper presents a novel approach to preference modeling in LMs by integrating context-specific evaluation. The key contributions include the introduction of context-conditioned preference datasets, the decomposition of reward modeling error, the development of a context-aware reward model, and comprehensive experiments to validate the approach.", "weaknesses": "## Under-supported claims \n\nIn section 3.3, the author included various claims that I believe to be interesting but only so when they are better-substantiated.\n\n**Assumption about Cardinality** The hypothesis that the cardinality of the space of contexts is smaller than that of the space of possible completions given a prompt is stated without empirical or theoretical justification. This assumption is critical to the argument about data efficiency in context annotation versus preference annotation. Without evidence or a rationale to support this assumption, the argument remains speculative.\n\n## Details of the dataset curation\nThe reviewer appreciate the author's efforts in curating the dataset. \nSection 4 briefly mentions that the dataset was generated using GPT-4 Turbo with a series of prompts designed to maximize validity. However, it lacks specific details about the prompts, the criteria for selection, and the methodology for ensuring the validity of the samples. This ambiguity makes it difficult to assess the soundness of the curation process.\n\n## Clarity\nWhile the paper presents interesting ideas, there are some areas in writing and presentation that could benefit from improvement:\n- The introduction and related work sections are detailed, but they could be more concise to better highlight the key contributions and their significance.\n- The figures and tables, although informative, could be optimized to more effectively convey the main points. Some elements might appear redundant or unclear.\n- Enhancing the narrative flow would help in making the logical progression of the arguments clearer and more accessible to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and feedback. Please find our responses below.\n\n&nbsp;\n\n\n> Under-supported claims / Assumption about Cardinality\n\nThe framing of this in the discussion is not as “claims” but rather a “conjecture” (L182) and a “hypothesis” (L188,L193). As noted in the text, we do find “some” (L181,L193) empirical support for each, in Tables 2 and 5. We leave a more detailed exploration of this to future work, as the focus of this work is on improving context-specific preference modeling. \n\n&nbsp;\n\n\n> Data curation process / “lacks specific details about the prompts, the criteria for selection, and the methodology for ensuring the validity of the samples”\n\nAs noted in the main text at L211 (right after introducing the datasets), the full details of the dataset curation process, including prompts, criteria for selection, and methodology for ensuring validity (e.g. L794), are detailed in Appendix B (L766-L815). If there is some shortcoming in the presentation there, we would appreciate a more specific critique. \n\n&nbsp;\n\n\n> Clarity / Enhancing the narrative flow\n\nWe have added a Figure 2 and revised Figure 1 (see pdf attachment) to clarify the framework and main results. We believe the narrative flow is strong, but would appreciate and happily consider any more specific critiques the reviewer has on this point. \n\n&nbsp;\n\n\n> Assumption related to Bernoulli distribution / Intent distribution\n\nModeling paired preference as sampling from Bernoulli distribution is standard; e.g., in the widely used Bradley-Terry model. Allowing for an intent distribution (as opposed to a single intent) seems like a natural modeling choice that broadens the scope of the model (it gets us Equation (3) in addition to the single intent Equation (4)). We do not believe these modeling choices require empirical justification, as we are not claiming in Sections 3.1-3.2 that real human annotators explicitly infer an intent or distribution of intents, or that real human annotators explicitly model their label as a Bernoulli distribution. Indeed, at L148-151 we adopt the (commonly used) Expected Utility model, which is used to arrive at Equations (3) and (4); but it is well known that humans systematically deviate from expected utility. \n\nOur formalism is Sections 3.1-3.2 is best understood as any other idealized model: it offers analytical insights. In our case, our model is used to arrive at the bounds in Equation (3) and (4), which expands on and strengthens the motivation in Section 2, and suggests the discussion in Section 3.3. As with any model, it is up to the reader to determine whether the model in question is a reasonable, and useful enough, approximation to reality.\n\n&nbsp;\n\n\n> There is no empirical evidence or studies cited that demonstrate annotators' ability to accurately infer intents from prompts. Additionally, the variability in annotators' interpretations and the potential biases they introduce are not addressed.\n\nThis is a core motivation for our work, as set out at L82-85: “If we train models using non-basic preference annotations, the contextual biases and assumptions underlying those judgments may be implicitly embedded into the model [35, 48] … Rather than rely on annotators to integrate the correct distribution of contextual assumptions … ”, wherein it is suggested that we might not want to rely on annotators to “accurately infer intents” [i.e., contexts], as this is subject to “the variability in annotators' interpretations and the potential biases”. A descriptive study of the biases and assumptions made by real human annotators is beyond the scope of the present work. \n\n&nbsp;\n\n\n> Human validation\n\nThe human validation of the dataset labels was done by the authors (blind / response orders and criteria/scenarios randomized – we will add this detail to the paper). As the authors, we are the closest to the data, and are able to provide high quality labels. In our paper we reported 100 total labels (50 for RPR Criteria and 50 for RPR Scenarios; different prompts for each). Since submission, we have added an additional 100 labels (50 for each Criteria and Scenarios), achieving a total agreement of 97% for RPR Criteria and 95% for RPR Scenarios (100 labels each). This gives 95% confidence intervals as follows: \n\n| RPR Criteria    | RPR Scenarios |\n| -------- | ------- |\n| $$(0.937, 1)$$  | $$(0.907, 0.993)$$  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QE4IK89z9G", "reviewer": "Reviewer_mLs1", "summary": "This paper points out that the preference label can be reversed by inserting additional context into the prompt. Based on this observation, the authors build a paired preference dataset. The authors also try to provide some theoretical analysis.", "questions": "See weaknesses.", "limitations": "This paper does not have a limitation section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This paper studies a specific and interesting problem in preference data.\n* The proposed data augmentation method may benefit preference optimization of LLMs and inspire further research.", "weaknesses": "My major concern lies in the experiments. The current experiments do not demonstrate the general advantage of using a paired dataset with reversed preference labels. On widely used preference datasets, the model trained on the proposed dataset does not outperform the baselines. It only performs better on the test set, which is built in the same way as the training set. Note that the preference labels on this test set are not verified by humans and could be unreliable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and feedback. Please find our responses below.\n\n&nbsp;\n\n\n> The current experiments do not demonstrate the general advantage of using a paired dataset with reversed preference labels.\n\nThank you. This is a good criticism. We realized this post submission and have prepared a data ablation that finetunes the base Mistral reward model on context-conditioned data from other distributions, including: \n\n- Preference Collection (PC) (hf:prometheus-eval/Preference-Collection), in which the context is highly correlated with unconditioned preference, and \n- a “one-sided” version of our RPR datasets, where kept only one of the two sides by having GPT-4 pick one of the two criteria or scenarios that it thought was more suitable for the given prompt. \n\nThe results are shown in Table 1 in the PDF attachment, and demonstrate the advantage provided by our RPR dataset. We will add this to our paper. Notably, the Multifaceted Bench (MF), released after the NeurIPS submission deadline, introduced an additional context-conditioned preference benchmark that is outside of our training distribution, where you can see that training on RPR strongly improves context-conditioned performance over the base model (and the PC model). \n\n&nbsp;\n\n\n> On widely used preference datasets, the model trained on the proposed dataset does not outperform the baselines.\n\nIf you are referring to the no context versions of HHH, Reward Bench, and Chatbot Arena, these are unconditional preference benchmarks, where it is not expected that our model, trained for context-conditioned preference queries, would do better than the base model. We report results here mainly to provide a reference figure, and also to show that fine tuning on RPR does not hurt the unconditioned preference prediction of the base model. \n\n&nbsp;\n\n\n> It only performs better on the test set, which is built in the same way as the training set. \n\nThis is not true. Our model not only performs better on the RPR test sets, but also: Multifaceted Bench (MF, see general response), HHH with context, Reward Bench with context, and Chatbot Arena with Context*. This is more clearly shown in Figure 1 in the PDF attachment, which we will add to our paper as Figure 2. Please note the errata for GPT-4 results in the general response. \n\n&nbsp;\n\n\n> Note that the preference labels on this test set are not verified by humans and could be unreliable.\n\nHuman validation of the dataset labels was done by the authors (blind / response orders and criteria/scenarios randomized). As the authors, we are the closest to the data, and are able to provide high quality labels. In our original submission we reported 100 total labels (50 for RPR Criteria and 50 for RPR Scenarios; different prompts for each) in Table 2 (pdf page 7). Since submission, we have added an additional 100 labels (50 for each Criteria and Scenarios), achieving a total agreement of 97% for RPR Criteria and 95% for RPR Scenarios (100 labels each). This gives 95% confidence intervals for human-RPR agreement as follows: \n\n\n| RPR Criteria    | RPR Scenarios |\n| :--------: | :-------: |\n| $$(0.937, 1)$$  | $$(0.907, 0.993)$$ |\n\nSome noise in human agreement is unavoidable. As noted in our paper (L73) there is significant inter-human disagreement on unconditioned queries, e.g., humans agree with each other only 65.7% of the time on AlpacaEval. Even the carefully curated MT-bench shows has agreement of only 81-82% on strict preference queries (Zheng et al., Table 5). Thus, our 90+% agreement between the human authors and the synthesized labels is quite high, likely as a result of the added context and multiple levels of filtering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ctwdumf0fZ", "reviewer": "Reviewer_t1sE", "summary": "This paper divides preference modeling into two steps: first estimating the user's intent then evaluate the generated text within the context of this intent. The paper makes the Reasonable Preference Reversal Datasets, which encompass criteria and scenarios for preference data. Experiments find that models can achieve higher performance when providing intent contexts during evaluation.", "questions": "Why is two-step preference modeling necessary? The user's intent is inherently contained in the query, and traditional preference modeling implicitly evaluates whether the response aligns with the user's intent. Therefore, why should the estimated intent be treated independently? Additionally, ambiguity exists in estimating intent, as different individuals may have inconsistent interpretations of intent for the same query.", "limitations": "Please see Weakness part.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The approach of first estimating user intent before evaluation may be promising.\n- This work constructs the first open-source context-conditioned preference datasets, which may be useful for future work.", "weaknesses": "1. Lack of citations and comparisons to related work. For example, Li et al. [1] generate the criteria for evaluation and then get the final answer according to the criteria, which is similar to this work.\n\n2. In Table 2, the author aims to demonstrate that fine-tuning a context-aware reward model significantly enhances context-specific performance. However, the observed improvement in Table 2 could be attributed to the model being trained on data from the same distribution as the test set. Even for new datasets that are not context-specific, training the model on such datasets will likely improve its performance on its test sets.\n\n3. The lack of experiments illustrating that two-step preference modeling achieves better performance compared to traditional reward modeling on general preference datasets is a concern. In Table 4, the model is prompted by context generated by GPT-4, leading to unfair comparisons with similar models. To demonstrate that two-step preference modeling is superior, it is essential to show its advantages for models of equal ability.\n\n[1] Li J, Sun S, Yuan W, et al. Generative judge for evaluating alignment[J]. arXiv preprint arXiv:2310.05470, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and feedback. Please find our responses below. \n\n&nbsp;\n\n\n> citations /  Li et al. [1]\n\nThank you for bringing Li et al. to our attention. We agree it is closely related and will add it to the related work, and we will revisit our literature search for other recent work we may have missed. \n\nAs part of our rebuttal, we have run their Auto-J model, but found that both Ultra RM and the base Mistral RM outperform it on every dataset/benchmark we use, both unconditioned and conditioned. We note that Auto-J, including the datasets used in the Auto-J paper, seeks to directly address the unconditioned preference modeling problem, even if Auto-J generates criteria as part of its judging process. \n\n&nbsp;\n\n\n>Table 2 could be attributed to the model being trained on data from the same distribution as the test set. \n\nWe agree this is true for RPR and will make this more clear in our paper (see, e.g., the stars in the new data ablation table of the PDF attachment). However, we note that our finetuned CARM also improves performance on all other context-specific datasets, including the newly added Multifaceted Bench (see General Response), as is now more clearly shown in Figure 1 of the PDF attachment. \n\n&nbsp;\n\n\n> In Table 4, the model is prompted by context generated by GPT-4, leading to unfair comparisons with similar models\n\nWe disagree that these comparisons are unfair. In particular, *all* models—not just our CARM—have access to the *same* additional context, hence their improved performance relative to the “NC” column. \n\n&nbsp;\n\n\n> The lack of experiments illustrating that two-step preference modeling achieves better performance compared to traditional reward modeling on general preference datasets is a concern.\n\nWe agree it would be ideal to show that two-step preference modeling can achieve better performance on general preference datasets. However, this requires two parts: (1) better context-specific modeling, and (2) strong context inference or specification. While we show in this work how we can improve the context-specific modeling problem, the context inference problem remains difficult. We have tried to generate context with respect to unconditioned queries as part of our work as well, and found that getting current models to generate useful context is quite challenging (for evidence of this, see the Auto-J results noted above), which is why we believe one of the next steps for future work is to investigate how to best obtain strong context *supervision* (as we allude to at L186-187). We think supervision of some kind is critical in order to go beyond the “user's intent is inherently contained in the query” as you wrote. \n\nFurthermore, we believe two-step preference modeling to be useful even if it could not improve on current unconditioned preference datasets such as Reward Bench. See next response. \n\n&nbsp;\n\n> Why is two-step preference modeling necessary? … why should the estimated intent be treated independently? \n\nWe think your last sentence here captures our motivation: “ambiguity exists in estimating intent, as different individuals may have inconsistent interpretations of intent for the same query.” Part of our argument is that intent need not be estimated solely from the prompt—indeed, it can be:\n\n- specified for annotators (see L84-91), or via default rules similar to Constitutional AI, or via the system prompt\n- inferred from past interactions in cases of persistent context (e.g. user profiles, L287; see revised Table in the attached pdf), or \n- learned from data via some context supervision system. \n\nTo the extent that there are inconsistent interpretations of the same query, any such context will usefully disambiguate the query, which provides several advantages: \n\n- **(a)** Given context, we reduce reliance on unstated, implicit assumptions made by annotators, thereby increasing overall agreement (see Ziegler at al., quoted at L88). \n- **(b)** Using context we can improve steerability and pluralistic alignment (see Sorensen et al.). \n- **(c)** Making the context explicit may also be useful to diagnose errors made by models. \n- **(d)** Notably, making context explicit allows us to change the aggregation rule from a Borda count (see Siththaranjan et al.) to a more flexible Social Welfare Aggregation (as in Bakker et al., who find this to be effective as a consensus mechanism). \n\nIn the last case, (d), agreement with respect to unconditional preferences gathered from a group (and thus, performance on general preference datasets) seizes to become a good measure, as Equation (2) [L158-159] (or modifications of Equation 2 for non-EU SWFs) will not hold, so that explicit contextualized aggregation (of human preferences) becomes a better alignment target than overall preference (see Siththaranjan et al. for a concrete example of this).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPpJVvSqyJ", "reviewer": "Reviewer_rZtt", "summary": "The paper focuses on fine-tuning LLMs to improve their ability to handle context-aware preference modeling. The authors address the challenge of the underspecified and ambiguous nature of natural language preferences by introducing a two-step modeling process. This includes selecting a context and evaluating preferences within that context. The approach is backed by the introduction of new datasets, named RPR, which are designed to test the effectiveness of context-specific preference modeling. The study provides extensive experimental evidence showing how context-aware models can surpass traditional methods in handling ambiguous and context-specific scenarios.", "questions": "Please see my concerns in weaknesses.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper tackles a crucial issue in the realm of language modeling by enhancing the LLM's ability to understand and process user preferences in context-dependent scenarios. This is particularly important as LLMs are increasingly used in diverse real-world applications.\n2. The introduction of the RPR datasets is a notable contribution, as these datasets specifically aim to disentangle context-specific preferences from general preferences, offering a valuable resource for further research.\n3. The paper provides thorough experimental results that not only demonstrate the effectiveness of the proposed method but also explore various aspects of context-aware preference modeling, showing improvements over existing models like GPT-4 and Llama 3.", "weaknesses": "1. The paper could benefit from testing the proposed method across a broader range of general benchmarks, such as MMLU and AGI-Eval, to assess how fine-tuning for context-aware preferences might affect the LLM's original capabilities or general applicability.\n2. It remains unclear how applicable the proposed method is to other LMs, especially those of different sizes or from different series. Addressing this would help validate the robustness and versatility of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and feedback. Please find our responses below.\n\n&nbsp;\n\n\n> The paper could benefit from testing the proposed method across a broader range of general benchmarks, such as MMLU and AGI-Eval, to assess how fine-tuning for context-aware preferences might affect the LLM's original capabilities or general applicability.\n\nWe agree that training generative models with context-aware rewards is a great direction, and part of our broader agenda. In particular, we believe this could improve sensitivity to the system prompts and diverse users, and to prompts with multiple instructions or constraints, which is something we have observed state-of-the-art models struggle with. The design decisions here are not obvious though (where does context for MMLU come from, for example, especially given its more objective nature) and add another layer of complexity that we believe would be better suited for future work. \n\n&nbsp;\n\n> It remains unclear how applicable the proposed method is to other LMs, especially those of different sizes or from different series.\n\nThis is a good point. We took the opportunity to use RPR to finetune one of the stronger 2B parameter reward models according to the Reward Bench leaderboard, which is based on Gemma (hf:Ray2333/Gemma-2B-rewardmodel-baseline). We used the same hyperparameters as used for the 7B Mistral RM in the paper. We obtained the following context-conditioned results, which we will include the paper (Mistral results included for reference, best Gemma model bolded):\n\n\n| || Gemma-2B-NC |  Gemma-2B-CTX  | **Gemma-CARM** | \\| |Mistral-7B-CTX | Mistral-CARM |\n| --------|--------|:-: | :-: | :-: | :-: | :-:| :-: |\n| RPR Criteria || 0.511 | 0.761 |  **0.968** | \\| |0.867 | 0.985 | \n| RPR Scenarios || 0.511 | 0.655 |  **0.909** | \\| |0.749 | 0.962 |\n| Multifaceted Bench ||  0.508 | 0.597 |  **0.681** | \\| |0.679 | 0.787 | \n| Preference Bench || 0.852 |  **0.861** |  0.849 | \\| |0.915 | 0.919 |\n| HHH (CTX) || 0.751 |  0.751 |  **0.760** | \\| |0.905 | 0.919 |\n| Rewardbench (CTX)| | 0.718 | 0.735 |  **0.786** | \\| |0.833 | 0.871 |\n| Chatbot Arena (CTX*) || 0.745 |  0.806 |  **0.873** | \\| |0.859 | 0.909 |\n\nNote: Multifaceted Bench is a new context-conditioned preference dataset — see general response.\n\nWe observe that finetuning the Gemma RM shows similar patterns to finetuning the Mistral RM, and that the finetuned model is in many cases competitive with the larger Mistral base model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time, consideration, and numerous comments that will help us improve the manuscript. We have responded to each reviewer individually. If you find our rebuttal to be responsive to your concerns, we kindly ask you to consider recommending “accept” --- we believe context-specific modeling and pluralistic alignment is an important direction that has received recent interest from a number of different research groups, and that the present submission is complete and provides interesting results and contributions in this area. 

We have the following general comments and revisions to note:

&nbsp;


**Added new Figure, and updated Table 6 to better present the results**

See PDF attachment (Figure 1 and Table 2). Figure 1 of the PDF attachment will appear as Figure 2 in our revised manuscript. 

&nbsp;


**Updated to include results on Multifaceted Bench and discuss the concurrent work by Lee et al. [1]**

Post submission, the work [1] by Lee et al. was released, which includes a synthesized dataset of diverse system prompts for finetuning generative models. Although formed for a different purpose, their dataset includes multiple system prompts for the same user prompt, which allows it to be used in a similar fashion as our RPR datasets. We run context-specific evaluation on their dataset. The main results are now shown in the PDF attachment (Figure 1), and we will include a discussion of their work in the Related Works section. 

[1] Lee, Seongyun, et al. "Aligning to thousands of preferences via system message generalization." arXiv preprint arXiv:2405.17977 (2024).

&nbsp;

**Updated to include a Data Ablation**

Reviewer mLs1 made the excellent point that “The current experiments do not demonstrate the general advantage of using a paired dataset with reversed preference labels.”

In response, we have prepared a data ablation that finetunes the base Mistral reward model on context-conditioned data from other distributions, including:

- Preference Collection (PC), in which the context is highly correlated with unconditioned preference by Kim et al. (2024) (Prometheus 2), and
- a “one-sided” version of our RPR datasets, where we keep only one of the two sides by having GPT-4 pick the criteria that it thinks is best suited for the given prompt.

The results are shown in Table 1 in the PDF attachment, and demonstrate the advantage provided by our RPR dataset. We will add this to our paper. Notably, on Multifaceted Bench (MF), which is outside of our training distribution, training on RPR strongly improves context-conditioned performance over the base model (and the PC model).

&nbsp;

**Errata re: GPT-4 results**

After submission, we noticed that the GPT-4 results appeared high (e.g. our submission reported a score of 93.5% on Reward Bench, which is far higher than what it obtains on the Reward Bench leaderboard). This was due to a bug in our code that was counting ties issued by GPT-4 as being correct.  **Only the results reported for GPT-4 Turbo were affected—all other originally reported results are correct.** Here are the corrected numbers for GPT-4, which are also reflected in the attached PDF Figure 1. 


|          | RPR-C (CTX)   | RPR-S (CTX) | PB (CTX) | MF (CTX) | HHH (NC) | HHH (CTX) | RB (NC) | RB (CTX) | CBA (NC) | CBA (CTX) | CBA (CTX*) | 
| --------| :--------: | :-------: | :--------: | :-------: |  :--------: | :-------: |  :--------: | :-------: |  :--------: | :-------: | :-------: |
| Original GPT-4 | 0.899 | 0.725 | 0.868 | - | 0.950 | 0.964 | 0.935 | 0.904 | 0.825 | 0.833  | 0.930| 
| Corrected GPT-4 | 0.901 | 0.748 | 0.860 | 0.640 |  0.871 | 0.873 | 0.824 | 0.821 | 0.720 | 0.771 | 0.858 |

After this correction, our finetuned Mistral CARM is the best performing model not only on RPR (for which it is in distribution), but also: Multifaceted Bench, HHH (CTX), Reward Bench (CTX), and Chatbot Arena (CTX*). Note that GPT-4 does worse on average than Llama3-70B not because it is a worse at context-specific prediction necessarily, but because we do not have access to its logits so that its scores are restricted to an integer scale and often result in ties. 

&nbsp;

**New context-aware finetune**

Reviewer rZtt made a good point that "it remains unclear how applicable the proposed method is to other LMs". As part of our rebuttal we finetuned a reward model based on Gemma 2B that performed relatively well for its size on Reward Bench. This shows that the benefit of training on RPR is not restricted to the Mistral 7B model we used in our submission. We report the results in our response to Reviewer rZtt below, which we will add to the paper.

&nbsp;

**Added sample from RPR**

We realized after submission that our manuscript did not include any samples from the dataset itself. We have since added a sample from RPR to the main text to help with the exposition. We are happy to share it during the extended discussion period if the reviewers would find it helpful.

Meta-review: The paper addresses the challenges of fine-tuning language models using pairwise preferences due to the underspecified nature of natural language. It proposes a two-step preference modelling approach, introducing context-conditioned preference datasets and developing a context-aware reward model.

The reviews are borderline and the authors suspect that one of the negative reviews contains artefacts indicating that it could have been generated by ChatGPT.  

The paper introduces a two-step preference modelling approach that could potentially enhance the alignment of models with diverse human preferences (e.g. different domains for text generation have different rrequirements). The introduction of context-conditioning is a valuable contributioin and may be of benefit for downstream applications in model personalisation. The context-aware reward model demonstrates improved performance on context-specific tasks, with comprehensive experimental results supporting this claim.

In the rebuttal, the authors have provided additional data that counters the negative aspects identified by the reviewers. The rebuttal introduces additional experiments demonstrating the model’s advantages on newly introduced benchmarks, which somewhat mitigates the concerns raised in the first round of reviewing. 

There is a significant discrepancy between reviewers mLs1 and j5YU , who lean towards rejection due to concerns about under-supported claims and experimental limitations, and reviewers rztt and t1sE, who see potential in the novel approach and datasets introduced.

The paper has the potential to influence the language model personalisation and preference modeling, particularly in how context is integrated into preference modelling. The introduction of context-conditioned datasets could serve as a valuable resource for future research.
Decision: Accept (poster)