Title: Improving Context-Aware Preference Modeling for Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: context-specific preference modeling

preference modeling

reward modeling

language modeling

rlaif

rlhf
Abstract: While finetuning language models from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges. Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals. To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context. We decompose reward modeling error according to these two steps, which suggests that supervising context in addition to context-specific preference may be a viable approach to aligning models with diverse human preferences. For this to work, the ability of models to evaluate context-specific preference is critical. To this end, we contribute context-conditioned preference datasets and accompanying experiments that investigate the ability of language models to evaluate context-specific preference. Unlike past datasets, where context-specific preference is highly correlated with general preference, our "preference reversal" datasets disentangle context-specific and general preferences to isolate context-specific capabilities. We use our datasets to (1) show that existing preference models benefit from, but fail to fully consider, added context, (2) finetune a context-aware reward model with context-specific performance exceeding that of GPT-4 and Llama 3 70B, and (3) investigate the potential value of context-aware preference modeling.

Improving Context-Aware Preference Modeling for Language Models

Abstract

While finetuning language models (LMs) from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges. Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals. To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context. We decompose reward modeling error according to these two steps, which suggests that supervising context in addition to context-specific preference may be a viable approach to aligning models with diverse human preferences. For this to work, the ability of models to evaluate context-specific preference is critical. To this end, we contribute contextconditioned preference datasets and accompanying experiments that investigate the ability of language models to evaluate context-specific preference. We use our datasets to (1) show that existing preference models benefit from, but fail to fully consider, added context, (2) finetune a context-aware reward model with context-specific performance exceeding that of GPT-4 and Llama 3 70B on tested datasets, and (3) investigate the value of context-aware preference modeling.

1 Introduction

As the general purpose capabilities of Language Models (LMs) [11, 42] and other foundation models [9] progress toward handling diverse instructions and executing long-range trajectories in real-world applications [44, 47, 41], it becomes increasingly important to have a principled system for ensuring that LM agents behave as expected. The prevailing approach for aligning an LM to human preferences uses pairwise preferences between different outputs to finetune the LM [52, 6], which falls short of addressing the critical challenges presented by the reality of diverse user intents and contexts [51, 50, 13]. In the presence of unspecified contexts, such as the user's identity or goals, preference queries are notoriously ambiguous [62] and one typically observes poor agreement ( ∼ 65% ) between human annotators on binary preference queries [35, 59].

In this paper, we consider modeling preferences using a two-step, context-aware approach (Figure 1). This approach first resolves the underspecification by selecting a context [38, 22], and then evaluates preference with respect to the chosen context [61, 14, 27, 30, 58]. Decomposing general preference into context and context-specific preference has several potential advantages. First and foremost, this approach explicitly identifies contextual assumptions that underlie preference, and shifts the alignment burden from preference modeling to a hybrid of preference modeling and context supervision. Second, this approach is naturally pluralistic [51], allowing the model to adapt to diverse users and use cases. Finally, it offers more flexibility for principled aggregation : whereas the Bradley-Terry approach corresponds to aggregating contexts using the Borda rule [50], which may under-serve certain

PM

Query

Preference

Why is the sky blue?

Light from the sun...

Imagine a flashlight...

A

?

Be scientific

Explain like

I'm five

Query

Context

Preference

CAPM

CAPM

≻

A

≻

B

≻

A

B

Why is the sky blue?

Light from the sun...

Imagine a flashlight...

A

B

A

B

B

RPR Criteria

Multifaceted Bench

HHH

Reward Bench

Context-Conditioned Preference Datasets

50

60

70

80

90

100

Agreement with Gold Labels

51.0

51.6

89.1

81.8

77.6

86.7

67.9

90.5

83.3

85.9

92.5

68.7

91.4

84.7

89.2

98.5

78.7

91.9

87.1

90.9

Llama3-70B (w/ CTX)

subgroups [13], the context-decomposition approach can additionally be used for jury learning [22] and cardinal utility aggregation [7].

We show that we can upper-bound the absolute reward modeling error as the sum of two terms: one term corresponding to context inference error, and the other to the context-specific reward modeling error. This decomposition suggests that supervising context might be a viable approach to aligning models with human preferences, provided we can do strong context-specific preference prediction. To evaluate context-specific prediction and build toward stronger context-aware preference models, we propose context-conditioned 'preference reversal' datasets, where the preference for a given prompt is reversed given an alternative context.

Finally, we conduct experiments to benchmark the context-specific performance of various models and investigate the potential value of context-aware preference modeling. We find that, while current models generally benefit from additional context, they sometimes fail to give it full consideration, and finetuning on our preference reversal datasets greatly improves context-specific performance. We show that a single persistent context such as a user profile or system prompt can be used for a range of preference queries, and can be well inferred from as few as 16-32 samples of expressed preference.

In summary, our main contributions are:

We make an argument for modeling preference by decomposing it into context(s) and contextspecific preference, and propose a context-decomposition upper bound to justify the approach.

We open-source high quality context-conditioned preference datasets that disentangle contextspecific preference from general preference, which we use for finetuning and evaluation. The datasets can be found at https://huggingface.co/datasets/microsoft/rpr .

We show that while current models benefit from context, they may fail to give it full consideration, and that finetuning a context-aware reward model greatly improves context-specific performance, as well as general preference modeling performance when high quality context is available.

2 Context-Aware Preference Modeling

2.1 Resolving ambiguity by making implicit assumptions explicit

Current practice finetunes language models to make them consistent with a dataset of human preferences [44, 42]. As noted by earlier works on learning from human feedback, however, ambiguous preference judgments present a major challenge:

Evaluation of a [preference query] is both subjective and multidimensional ... this makes consistent labeling difficult for honest labelers (including the authors!) [62]

This ambiguity manifests itself with agreement rates as low as ∼ 65% between human annotators on binary preference queries [35, 59]. One way to understand this difficulty appeals to a distinction drawn by Amartya Sen:

A value judgment can be called 'basic' if the judgment is supposed to apply under all conceivable circumstances, and it is 'non-basic' otherwise. [49]

Many, perhaps most, preference queries ask the annotator for a non-basic preference judgment, in that certain contextual information might effectively reverse the judgment. For instance, the preferred response to a technical question may depend on the user's education level, the preferred response to a medical question may depend on whether the user is a doctor, and the preferred response to a question about etiquette may depend on the user's geographical location. If we train models using non-basic preference annotations, the contextual biases and assumptions underlying those judgments may be implicitly embedded into the model [36, 48].

Rather than rely on annotators to integrate the correct distribution of contextual assumptions, and rely on the training process to consistently aggregate any disagreements in a singularly aligned model, one might instead consider an explicit context-aware approach to preference modeling (Figure 1). This approach first (partly) resolves ambiguity by specifying a context, and then models context-specific preference. This is not a new idea; Ziegler et al. [62], quoted above, continue to remark that, ' it seems better to design less ambiguous labeling tasks that get at the same information ... such as a verbal description [of the most important contextual information] ', and many have advocated for finegrained, context-conditioned, or otherwise more 'pluralistic' approaches to alignment [58, 29, 51].

However, while production systems recognize the importance of incorporating context-the most notable being the 'system prompt' introduced by OpenAI in 2023 and adopted by others [54, 3]there has been little published research on the context-aware preference modeling capabilities of language models. This paper works toward filling this gap by introducing the reasonable preference reversal (RPR) datasets alongside context-augmented versions of existing datasets, and testing the context-specific preference modeling capabilities of existing models.

This discussion is summarized in Table 1 alongside other key benefits of context-aware preference modeling. This paper focuses on evaluating and improving context-aware preference modeling capabilities, leaving an in-depth exploration of the benefits to future work.

2.2 Related Work

Modeling human preferences traces back to Luce [39] and Bradley-Terry [10]. It made its way into finetuning language models via a line of work originating in reinforcement learning [2, 15, 62, 52], and has now become the dominant approach for finetuning language models to follow human instructions [44, 6, 4, 46]. While this approach has achieved remarkable results and enabled the performance of state-of-the-art models, several authors have pointed to its limitations [12, 45, 32, 51],

which include the ambiguity problem that motivates our work. Notably, recent works have identified shortcomings that arise from implicitly aggregating diverse perspectives with a single choice rule [50, 13, 40, 18, 19]. In particular, Siththaranjan et al. [50] have recently shown the standard Bradley-Terry approach (Figure 1, left), implicitly uses the Borda rule to aggregate preferences across hidden contexts, and proposed to learn distributional reward models as a way to account for this. Rather than leave hidden context implicit in a distributional model, which does not resolve the indeterminate preference problem highlighted by our RPR datasets (see Section 4), we propose to make context explicit, either via specification or inference. Given a preference problem that has been decomposed into contexts and context-specific preferences, as described in Section 3, we can then choose to apply the Borda rule or an alternative such as an expected utility aggregation [23] or jury learning [22] (see 'distributionally pluralistic models' in Sorensen et al. [51]).

Modeling diversity and disagreement among human annotators and the inconsistency of human preferences have been treated from a number of different perspectives [21, 8, 20, 31]. Our work is similar to others that decompose a single objective into multiple dimensions such as helpfulness, harmlessness, and truthfulness [55, 19, 37, 5]. This approach has also become common in the literature on summarization, where multi-dimensional evaluation criteria are well recognized, with common dimensions including conciseness, coherence, consistency and fluency [60, 26, 16]. However, our work (Table 3) and others [24] find that current models often ignore or confuse added context. Preferences driven by diverse individual principles have been aggregated within frameworks such as Constitutional AI [6, 53], and alternative approaches to aligning models with human preferences, such as AI safety via debate [25] and consensus based approaches [7], can be understood as ways of supervising the context or assumptions underlying preference rather than the preference itself.

Finally, our work is closely related to context-conditioned generation, e.g., via a system prompt. In a concurrent work, Lee et al. [34] have synthesized a dataset of diverse system prompts for finetuning generative models. Their dataset includes multiple system prompts for the same user prompt, which allows it to be used in a similar fashion as our RPR datasets. We use their dataset for evaluation in Section 5, and provide an ablation on different training set compositions in Appendix C.1.

3 Context Decomposition Upper Bound

3.1 Intent-Utility Formalism

We model the user-LM interaction using an intent-utility formalism ( I , X , Y , u ) , where I is the space of intents, X is the space of prompts, Y is the space of completions, and u : I × Y → R is a scalar utility function. We follow the standard assumption and assume that preference in this model is made according to the Bradley-Terry model [10, 15]. Letting σ be the logistic function, this defines the probability of preferring completion y 1 to y 2 given a specific intent i as:

In our model the primitive definitions of preference and utility are conditioned on the intent rather than the prompt. To prompt the model, a user with intent i selects a prompt x ∈ X . To annotate a preference query ( x, y 1 , y 2 ) , an annotator implicitly infers intent i from x and samples a preference from the Bernoulli distribution B [ p ( y 1 ≻ y 2 | i )] .

Both users and annotators may possess or infer a distribution of intents. Indeed, we would argue that annotation for most preference queries involves a distribution of intents rather than a specific intent. We use 'intent' to refer to both specific and distributional intents. We assume there exists a base distribution over possible intents p ( i ) , as well as a conditional distribution over prompts given intents p ( x | i ) , so that any prompt x has a natural inference distribution p ( i | x ) . In this model, the prompt x is a partial specification of intent i . While a prompt may never be able to fully specify the intent, we may add some additional information or context z to obtain an extended prompt ( x, z ) ∈ X . Let us suppose that z ∈ Z , where Z corresponds to a discrete partition of I .

One way to measure utility and preference with respect to a distribution p of intents is with an expected utility model, which computes utility as u ( p, y ) = E i ∼ p [ u ( i, y )] . While it has been shown that standard RLHF does not align with the expected utility model [50], this model satisfies certain desirable axioms, which one can argue would apply to 'ideal' preference annotation. We use the expected utility model to define u ( x, y ) := u ( p ( · | x ) , y ) , and note that for any context partition

Z , this implies u ( x, y ) = ∑ z ∈Z p ( z | x ) u (( x, z ) , y ) . For convenience, we define ∆( · , y 1 , y 2 ) := u ( · , y 1 ) -u ( · , y 2 ) , which also decomposes linearly: ∆( x, y 1 , y 2 ) = ∑ z p ( z | x )∆(( x, z ) , y 1 , y 2 ) .

3.2 Context Decomposition Upper Bound

During RLHF, we are presented with a dataset of prompt-preference tuples, D = { ( x, y 1 ≻ y 2 ) } , which we use to learn utility estimator ˆ u : X → Y (conventionally known as a 'reward model'). As was assumed for u and p ( z | x ) , we would like there to be a model ˆ p ( z | x ) that satisfies the relation:

In standard RLHF, we never learn such ˆ p . However, for purposes of this analysis, we will assume this ˆ p exists, either implicitly given ˆ u , or explicitly, such that given some Z , we compute ˆ u ( x, y ) via Equation (2) rather than via direct evaluation. Below, we will favor the latter interpretation.

For a given preference estimator ˆ p ( y 1 ≻ y 2 ) , ˆ u is only unique up to constant shifts, so to measure the accuracy of ˆ u , we will instead compare ∆ and estimator ˆ ∆( x, y 1 , y 2 ) := ˆ u ( x, y 1 ) -ˆ u ( x, y 2 ) .

Consider now the absolute error ∣ ∣ ∆( x, y 1 , y 2 ) -ˆ ∆( x, y 1 , y 2 ) ∣ ∣ for a particular preference query, and use ∆ z as shorthand for ∆(( x, z ) , y 1 , y 2 ) . For any Z we have the following bound:

where the second equality adds and subtracts ∑ z p ( z | x ) ˆ ∆ z and rearranges, and the final line uses the triangle inequality (multiple times).

Equation (3) applies given a distribution of contexts, but in many cases, we might assume there is a specific context c (i.e., p ( z = c ) := 1 ) and make a single context prediction ˆ c (i.e., ˆ p (ˆ c ) = 1 ). This simplifies Equation (3) and gives us the context decomposition upper bound for a specific context:

Both the general bound (Equation (3)) and specific bound (Equation (4)) formalize the following intuitive claim: if we can make accurate predictions given the true context (or context distribution), then we can reduce the preference modeling problem to a context inference problem.

3.3 Discussion

The upper-bound in Equation (3) bounds the total error in terms of a context-weighted prediction error and a preference-weighted inference error. On one extreme, we have Z = ∅ (standard preference modeling), so that the context inference error is zero and the prediction error exclusively depends on the preference prediction problem given the prompt. On the other hand, we have Z ≡ Y (e.g., z might be 'The preferred response is [ y ].') and our preference prediction error is zero, but the context inference problem becomes equivalent to generation. In between, we conjecture that there is a smooth trade-off between prediction error and inference error. This might be the case, for example, if a single context could apply to and disambiguate a range of different preference queries. We consider this in our experiments and find some support for the conjecture, in that conditioning on specific criteria outperforms conditioning on more abstract, yet still rather specific scenarios (Table 2) which outperforms conditioning on a user profile that applies to all preference queries at once (Table 5).

If our model ˆ u is very good at predicting preference given some additional context Z , the preference modeling problem can be largely reduced to a context inference (or specification) problem. In this case, rather than have annotators rank completions for ambiguous prompts, it may make sense to spend the annotation resources to specify additional context. Such annotations could then be used

to train a context inference model that disambiguates prompts. Intuitively, we hypothesize that the cardinality of the space of contexts is smaller than the cardinality of the space of possible completions given a prompt, which would make joint context-preference annotation a data-efficient alternative to just preference annotation. Although our focus is on improving context-aware preference modeling, our experiments with user profiles (Tables 5 and 6) provide some support for this hypothesis.

The effectiveness of the above proposal hinges on accurate context-specific prediction. Are language models sensitive to added context? Our work focuses on this question and contributes datasets to help us (1) evaluate it on real data, and (2) train better context-conditioned reward models.

4 Reasonable Preference Reversal (RPR) Datasets

We contribute a set of carefully designed, synthetic preference datasets to measure whether preference prediction is sensitive to context. Our datasets, inspired by the notion of non-basic judgments and preference reversal described in Section 2, include over 20,000 paired tuples of prompt, context, and preference judgments, i.e. ( x, z, y 1 ≻ y 2 ) . The samples are paired so that preference between two completions for the same prompt is entirely ambiguous without context: for every context, there is an alternative context for which preference reverses. As compared to the only prior context-conditioned preference dataset [30], where context-conditioned preference is highly correlated with unconditioned preference (see Table 2), our design choice ensures that preference prediction performance on this dataset is determined solely by the model's ability to pay attention to and interpret the context. The datasets can be found at https://huggingface.co/datasets/microsoft/rpr .

The dataset has been generated by GPT-4 Turbo [42] following a complex series of prompts designed to maximize validity (human agreement). We chose to do this to avoid invalid or ambiguous samples, at the cost of making it rather 'easy' to evaluate preference given the context. The full process

Prompt : Make a 5 paragraph essay in 1 3 1 format about why pineapple belongs on pizza

Criteria

A: Incorporates verified nutritional facts and historical data about pineapple and pizza

B: Considers the economic impact of pineapple on pizza on the food industry

Scenarios

A: A high school student is completing a capstone project for a nutrition class, which involves writing an essay on a food-related controversy. They've decided to address the polarizing topic of pineapple on pizza, with the goal of presenting an argument that is not only persuasive but also backed by credible nutritional information and historical context, to enlighten their peers and satisfy the rigorous standards of their instructor.

B: A high school student is preparing for a national business studies competition where participants must analyze a controversial product's impact on its respective industry. The student has chosen the food industry for their analysis, with a focus on the debate surrounding pineapple as a pizza topping. They are now seeking help to craft an essay that will highlight the economic benefits of this unconventional choice to support their argument.

Completions

A: Pineapple on pizza, often known as 'Hawaiian pizza,' combines the sweet and tangy flavors of pineapple with the savory taste of cheese and ham. From a nutritional standpoint, pineapple is a rich source of vitamin C, manganese, and dietary fiber, which contribute to a balanced diet. Historically, the introduction of pineapple on pizza can be traced back to the 1960s when it was first created in Canada by Sam Panopoulos. This essay will explore the health benefits of pineapple, its complementary flavor profile with other pizza ingredients, and the historical significance of this unique topping choice.

B: The inclusion of pineapple on pizza has sparked culinary debates, but from an economic perspective, it has proven to be a profitable addition to the pizza industry. Pineapple as a topping caters to diverse consumer preferences, expanding the market reach of pizza restaurants. The demand for Hawaiian pizza has led to increased sales and has become a staple in many pizzerias' menus. This essay will examine the economic benefits of pineapple on pizza, its impact on consumer choice, and the supply chain implications for pizza businesses in sourcing pineapple.

is detailed in Appendix B. The base dataset comes in two versions-criteria and scenarios-to emphasize the point that context may be expressed in many ways:

RPR Criteria , where the context is a short sentence describing which features are important, e.g., ' Uses a formal and professional tone that emphasizes the technical aspects of C++ '.

RPR Scenarios , where the context describes the scenario that led to the underspecified or ambiguous prompt; e.g. ' In preparation for a product launch, a marketing manager at a tech startup seeks innovative strategies to position the company as an industry leader... '.

A paired sample from the dataset is shown in Figure 3. We further extend the dataset with user profiles as described in Section 5.2 and Appendix B.1. To ensure the validity and reliability of the synthetic data, we conduct small scale human validation through blind preference queries (Table 3) and find agreement rates of ≥ 95% on both splits. Details are provided in Appendix B.2. We divide the dataset into a training set of 10,167 paired samples, and a test set of 1,000 paired samples, with no overlap between train and test prompts.

5 Experiments

Our experiments aim to answer the following questions of context-aware preference modeling:

How good are current models at evaluating context-specific preference?

Can we improve context-aware preference modeling by finetuning on our RPR datasets?

Can a single context compress preferences with respect to a diverse range of prompts?

5.1 Setup

Models We report results for a selection of primarily open source models, detailed in Appendix D.1. These include two unconditioned reward models (the 13B parameter UltraRM [17] and a 7B parameter Mistral RM [56]), one context-aware preference model (Prometheus-2 [30]), our finetuned contextaware reward model (Mistral CARM), and a set of generative models (four Llama 3 models [1] and GPT-4 Turbo [42]) used with an 'llm-as-a-judge' approach. In preliminary experiments we tested a several other models and observed similar patterns across all models. All models except Prometheus-2 are used as reward models, by first evaluating each alternative individually and then comparing scores. Appendix D.2 presents additional results for reward models finetuned from Gemma 2B.

Datasets Besides the RPR datasets detailed in Section 4, we use the following preference datasets:

Preference Bench [30] ( hf:prometheus-eval/Preference-Bench ) consists of 1998 contextconditioned preference samples synthesized by GPT-4 as part of Feedback Bench [29].

Multifaceted Bench [34] ( hf:kaist-ai/Multifaceted-Bench ) contains 921 samples of prompt, system prompt, and completion. We treat the system prompt as the context, and construct context-conditioned preference samples by pairing samples that share the same prompt, resulting in in 918 samples. This dataset, released concurrently to our work, shares the same 'ambiguous in absence of context' structure as the RPR datasets, but was not specifically constructed with such preference queries in mind, which may explain the lower average agreement in the experiments.

HHH [4] ( hf:HuggingFaceH4/hhh_alignment ) contains 222 human preference samples emphasizing different aspects: harmlessness (58 samples), helpfulness (59 samples), honesty (61 samples), and other preference queries that do not cleanly fall into another category (43 samples).

Reward Bench [33] ( hf:allenai/reward-bench ) curates 2,985 preference samples from a variety of sources, amounting to 22 distinct subsets covering chat, safety, and reasoning.

Chatbot Arena [59] ( hf:lmsys/chatbot_arena_conversations ) contains human preferences with respect to conversations on the Chatbot Arena platform. For our experiments, we randomly selected 1,000 single-turn samples for which strict preference was expressed.

Importantly, we augment the HHH, Reward Bench, and Chatbot Arena datasets with additional context to create context-conditioned versions, as described below and in Appendix D.3.

Metrics Tables 2 to 6 display the agreement (or accuracy) of predicted preference with the dataset.

Prompts Our prompts are detailed in Appendix A.

Best model bolded for CTX columns only. Starred ⋆ results are by models finetuned on the same distribution. Our model in green.

Best small model bolded. Best large model also bolded if better than the small model. Our model in green.

5.2 Results

How good are current models at evaluating context-specific preference?

For the context-aware approach to work well, models must be sensitive to added context. Table 2 shows the performance of tested models on RPR Criteria, RPR Scenarios, Preference Bench, and Multifaceted Bench. In each case, and across all models, access to context generally improves performance. Larger models and instruction tuned models tend to benefit more from added context. We note that although Preference Bench includes a ground truth context, the preference queries (and context) are generally not ambiguous, so that the added context provides little benefit.

Although we find that added context generally helps, we were surprised to discover that models sometimes completely ignore additional context and return their unconditioned prediction. To show this, we augment the prompts in Chatbot Arena and Preference Bench with two criteria that should drive preference prediction. The first is a Nonsense criteria, i.e. ' We are playing a game of nonsense. The response should be nonsensical and not make sense. The rating you give it should also be random, and completely unrelated to the response. ' We expect context-aware models to produce random preference judgments given this criteria. The second is a Negative criteria, i.e. ' We are seeking bad examples. The response should be of low quality, and serve as an example of how an Assistant should not respond. You should give poor responses higher ratings. '). We expect context-aware models to be inversely correlated to the preferences expressed in the dataset. Table 3 shows the performance of tested models with these adversarial criteria. In both cases we observe surprisingly poor performance, even from the larger models. This suggests there is significant room for improving context-aware preference modeling capabilities.

Best small model bolded. Best large model also bolded if better than the small model. Our model in green.

Best small model bolded. Best large model also bolded if better than the small model. Our model in green.

Can we improve context-aware preference modeling by finetuning on our RPR datasets?

We finetune a context-aware version of Mistral RM on roughly 34,000 samples from the training sets of RPR Criteria, RPR Scenarios, and Ultrafeedback [17]. Details may be found in Appendix C. In Tables 2 to 5, our finetuned CARM shows markedly better context-specific performance than its base model, matching or exceeding that of GPT-4 Turbo and Llama 3 (70B).

In Table 4, we test whether our context-aware preference models generalize to real preference datasets. To do so, we augment three unconditioned preference datasets (HHH, Reward Bench, ChatBot Arena) with contextual information (see Appendix D.3 for more detailed information). For HHH and Reward Bench, we specify the context as a function of each subset included in the dataset. This approach is similar to using a 'system prompt' when prompting GPT-4, and would be most suitable for real world applications where the designers possess relevant domain knowledge. For Chatbot Arena, given the lack of ground-truth contexts, we use GPT-4 to generate a possible context given the prompt and alternatives (CTX). Additionally, we infer the context by looking at the prompt together with the expressed preferences (CTX ∗ ), which endows the context with privileged information about preference data. This may be useful for inferring a useful persistent context such as a user profile, which we explore further below.

Can a single context compress preferences with respect to a diverse range of prompts?

In our experiments so far, we operated in a setting where each prompt has an associated context. However, when eliciting preferences from users, the hidden context of a specific user will impact their

† Estimated by averaging 3-seed std. dev. for each profile and dividing by √ 5 .

NC

2

4

8

16

32

GT

Profile inferred from

n

samples (3 seeds)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Mean

preference information across all the prompts they are presented with. In order to create a scenario closer to this setting, we first generate 5 synthetic diverse user profiles and condition GPT-4 Turbo on each profile to label the RPR test set (see Appendix B.1). Having relabeled the RPR test set with these preferences, we explore the ability of tested models to recover the expressed preferences from the profile, as shown in Table 5. Our CARM and the larger Llama 3 Instruct model perform best on this task, recovering approximately 80% agreement with GPT-4's labels.

Additionally, in Table 6, we test the performance of our context-aware model when profiles are inferred with limited preference samples. Profile inference is done by prompting GPT-4 (see Appendix B.1). Without any profile inference (No Context / NC), the default assumptions made by our model run against the preferences of Profiles 3 and 4; however, this is resolved in as few as 2-4 samples. While this depends on the underlying data distribution, 16-32 samples recover most of the benefit of the ground truth context on this dataset.

6 Conclusion

This paper began with a case for a two-step context-aware preference modeling framework (Figure 1), which was motivated by the ambiguity problem commonly experienced during preference annotation (Section 2). We further motivated the framework via a context decomposition upper bound (Section 3) and noted that for context-aware preference modeling to be viable, we require strong context-specific preference modeling. However, despite the prevalence of context conditioning in deployed systems (e.g., system prompts and 'My GPTs' [43]), when we began our work, there were no open source preference datasets where preference is strongly determined by context, and limited studies of the context-specific preference modeling capabilities of current models [30]. To this end, we introduced the RPR datasets (Table 2) and investigated a series of interesting empirical questions in Section 5). We found that (1) while current preference models generally benefit from context, they sometimes ignore added context, (2) finetuning on our preference reversal datasets greatly improves contextspecific performance, and (3) a single persistent context, which might be inferred from data, may be sufficient to significantly improve preference prediction performance.

Our work leaves many open threads for future research. Our datasets might be used to explore different directions in pluralistic alignment, as described by Sorensen et al. [51]. In particular, we believe better context specification and inference are important directions for research, which our experiments only briefly touch on. Further, while we have open sourced a set of context-conditioned preference datasets, these datasets have limited diversity (for example, they are primarily English, and are limited to single turn conversations), and were synthesized rather than collected from real human feedback, which limits their practical relevance, as the preferences may not align with all potential user perspectives. Additional open, preferably human sourced datasets with context annotations are needed to more fruitfully pursue this research direction.

References

AI@Meta. Llama 3 model card. 2024.

R. Akrour, M. Schoenauer, and M. Sebag. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23 , pages 116-131. Springer, 2012.

Anthropic. Model card and evaluations for Claude models, 2023.

A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional AI: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

M. Bakker, M. Chadwick, H. Sheahan, M. Tessler, L. Campbell-Gillingham, J. Balaguer, N. McAleese, A. Glaese, J. Aslanides, M. Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems , 35:38176-38189, 2022.

C. Baumler, A. Sotnikova, and H. Daumé III. Which examples should be multiply annotated? active learning when annotators may disagree. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 10352-10371, 2023.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 , 2023.

S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. MaxminRLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925 , 2024.

P. Cheng, J. Xie, K. Bai, Y. Dai, and N. Du. Everyone deserves a reward: Learning customized human preferences. arXiv preprint arXiv:2309.03126 , 2023.

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

E. Clark, S. Rijhwani, S. Gehrmann, J. Maynez, R. Aharoni, V. Nikolaev, T. Sellam, A. Siddhant, D. Das, and A. P. Parikh. Seahorse: A multilingual, multifaceted dataset for summarization evaluation. arXiv preprint arXiv:2305.13194 , 2023.

G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.

J. Dai and E. Fleisig. Mapping social choice theory to rlhf. arXiv preprint arXiv:2404.13038 , 2024.

J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773 , 2023.

V. Dumoulin, D. D. Johnson, P. S. Castro, H. Larochelle, and Y. Dauphin. A density estimation perspective on learning from pairwise human preferences. arXiv preprint arXiv:2311.14115 , 2023.

E. Fleisig, R. Abebe, and D. Klein. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 6715-6726, 2023.

M. L. Gordon, M. S. Lam, J. S. Park, K. Patel, J. Hancock, T. Hashimoto, and M. S. Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , pages 1-19, 2022.

J. C. Harsanyi. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal of political economy , 63(4):309-321, 1955.

X. Hu, M. Gao, S. Hu, Y. Zhang, Y. Chen, T. Xu, and X. Wan. Are LLM-based evaluators confusing NLG quality criteria? arXiv preprint arXiv:2402.12055 , 2024.

G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899 , 2018.

S. Jain, V. Keshava, S. M. Sathyendra, P. Fernandes, P. Liu, G. Neubig, and C. Zhou. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.01200 , 2023.

J. Jang, S. Kim, B. Y. Lin, Y . Wang, J. Hessel, L. Zettlemoyer, H. Hajishirzi, Y . Choi, and P. Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564 , 2023.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491 , 2023.

S. Kim, J. Suk, S. Longpre, B. Y. Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024.

C. Laidlaw and A. Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. arXiv preprint arXiv:2204.10759 , 2022.

N. Lambert, T. K. Gilbert, and T. Zick. Entangled preferences: The history and risks of reinforcement learning and human feedback. arXiv preprint arXiv:2310.13595 , 2023.

N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787 , 2024.

S. Lee, S. H. Park, S. Kim, and M. Seo. Aligning to thousands of preferences via system message generalization. arXiv preprint arXiv:2405.17977 , 2024.

X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_ eval , 2023.

P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning , pages 6565-6576. PMLR, 2021.

S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

E. Liscio, M. van der Meer, L. C. Siebert, C. M. Jonker, N. Mouter, and P. K. Murukannaiah. Axies: Identifying and evaluating context-specific values. In AAMAS , pages 799-808, 2021.

R. D. Luce. Individual choice behavior. 1959.

A. Mishra. AI alignment and social choice: Fundamental limitations and policy implications. arXiv preprint arXiv:2310.16048 , 2023.

OpenAI. Chatgpt plugins, March 2023.

OpenAI. GPT-4 technical report, 2023.

OpenAI. Creating a GPT, 2024.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

S. Pitis. Failure modes of learning reward models for LLMs and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

T. B. Richards. Auto-GPT: Autonomous artificial intelligence software agent, 2023. Initial release: March 30, 2023.

P. Schramowski, C. Turan, N. Andersen, C. A. Rothkopf, and K. Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence , 4(3):258268, 2022.

A. Sen. Collective Choice and Social Welfare . Holden Day, San Francisco, 1970. Edinburgh: Oliver and Boyd, 197l; Amsterdam: North-Holland, 1979. Swedish translation: Bokforlaget Thales, 1988.

A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arXiv preprint arXiv:2312.08358 , 2023.

T. Sorensen, J. Moore, J. Fisher, M. Gordon, N. Mireshghallah, C. M. Rytting, A. Ye, L. Jiang, X. Lu, N. Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070 , 2024.

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Z. Sun, Y. Shen, H. Zhang, Q. Zhou, Z. Chen, D. Cox, Y. Yang, and C. Gan. Salmon: Self-alignment with instructable reward models, 2024.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Z. Wang, Y. Dong, O. Delalleau, J. Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673 , 2024.

W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint, 2024.

R. Yang, R. Ding, Y. Lin, H. Zhang, and T. Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.10216 , 2024.

S. Ye, D. Kim, S. Kim, H. Hwang, S. Kim, Y. Jo, J. Thorne, J. Kim, and M. Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928 , 2023.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685 , 2023.

M. Zhong, Y. Liu, D. Yin, Y. Mao, Y. Jiao, P. Liu, C. Zhu, H. Ji, and J. Han. Towards a unified multidimensional evaluator for text generation. arXiv preprint arXiv:2210.07197 , 2022.

W. Zhou, S. Zhang, H. Poon, and M. Chen. Context-faithful prompting for large language models. arXiv preprint arXiv:2303.11315 , 2023.

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

A Prompts for context-conditioned preference prediction

The following template is used for 'llm-as-a-judge' models (Llama 3 and GPT-4 Turbo). Llama-3 uses the logit_template (see Appendix D.1 for how the score is computed). GPT-4 Turbo uses the argmax_score_template_no_cot and runs inference with temperature = 0.

Criteria template for llm-as-a-judge models

Mistral RM and our finetuned version use the following template:

Criteria template for Mistral RM and Mistral CARM

The prompt templates for Ultra RM and Prometheus follow the format set out by the authors of those models. Since no reference answer was available for Prometheus, we use the following slight modification which omits the reference answer:

B Constructing the RPR Datasets

We construct the datasets using the following steps:

Collect a set of diverse prompts.

Synthesize the initial RPR Criteria dataset.

Self critique and filter the RPR Criteria dataset.

Synthesize the RPR Scenarios dataset, using the RPR Criteria dataset as a base.

Self critique and filter the RPR Scenarios dataset.

Filter RPR Criteria down to what is left after the RPR Scenarios synthesis, so that each prompt has 2 criteria, 2 corresponding scenarios, and 2 corresponding completions.

In Step 1, we used all prompts from the Ultrafeedback dataset [17], which have already been selected for diversity, and filtered it down to prompts where there was no explicitly correct answer (according to the Ultrafeedback annotations). This left us with 42,000 initial prompts.

We passed these prompts through step 2 of our synthesis procedure:

For the categories, 7 categories were randomly sampled from the following list, and the one that had been chosen most so far during the synthesis process was dropped (so 6 categories were included in the prompt). This was done to increase the diverse of preferences. The list below was synthesized by GPT-4 to be mutually exclusive and collectively exhaustive of different dimensions according to which people might have differing preferences.

Criteria categories for RPR synthesis

criteria_categories

"Clarity and

Conciseness","Detail and Elaboration","Formality and Tone","Contextual

Relevance","

Factual Accuracy","Creativity and Originality","Technical Complexity","User-Friendliness","

Problem-Solving

Approach","Practical Application","Logical Consistency","Ethical Considerations

","Cultural Sensitivity","Predictive Accuracy","Empathy

and Emotional Intelligence","Historical

Accuracy","Innovativeness","Interdisciplinary Approach","Linguistic Creativity","Scientific Rigor

","Societal Impact","Sustainability","User Experience","Visual and Aesthetic Appeal","Economic

Feasibility","Legal

and Regulatory Compliance","Pedagogical Effectiveness","Technological

Advancement","Crisis

Management","Global Perspective","Humor and Entertainment Value","

Inclusivity and

Diversity","Strategic Insight","Narrative and Storytelling Quality","

Personalization

and Customization","Data Utilization

and Analysis","Philosophical Depth","Health

and Wellness

Orientation","Security and Privacy

Considerations","Multilingual and

Cross-Cultural

Competence"

])

Having synthesized the initial RPR criteria data, we filtered out poor entries (Step 3) using the following prompt:

This gave us the initial data of ∼ 32,000 paired prompts that was then further filtered through the RPR Scenarios synthesis process. For Step 3, we applied the following prompt to each sample in the RPR Criteria dataset:

RPR Scenarios Initial Synthesis

We then once again filtered the results for validity:

Each step of the above process eliminated certain pairs or samples, or eliminated half of a paired sample. In the latter, we remove the entire pair from the dataset. Our final remaining dataset, which we use for our experiments, has 10,167 paired trained samples, and 1,000 paired test samples, each of which comes with (1) prompt, (2) 2 criteria, (3) 2 corresponding scenarios, and (4) corresponding completions, where preference depends on the applicable criteria or scenario. An example from the dataset is shown in Figure 3.

B.1 RPR Profiles

To make the RPR profiles extension of RPR, we sample 20 sets of 20 random samples from the training set (assigning the preference for each sample at random). We use these samples to seed an initial sample of 20 profiles, each of which we infer from 20 samples using GPT-4 Turbo and the following profile inference prompt:

Profile inference prompt

Having generated 20 profiles, we use them to evaluate preference on 40 samples, using the inference prompt below. We run each sample twice with the completions in alternating order as we noticed some order bias:

Label preference with profile prompt

We then filter the 20 profiles down to 16, as 4 of the profiles showed <80% agreement in preference when the order of the completions was switched. Then, we measure pairwise distances between the remaining 16 profiles according to the 40 evaluated samples, and choose a subset that has sufficient minimum pairwise difference (we were able to obtain over 0.2). This produced the profiles listed in Appendix D.4, which we then use to label the entire RPR test set for use in the experiments.

B.2 Human Validation Details

To ensure the validity and reliability of the synthetic data, we conduct small scale human validation through 100 blind preference queries on each of RPR Criteria and RPR Scenarios. We find agreement rates of ≥ 95% on both splits, as shown in Table 3. This validation of the dataset labels was done by the authors with both response orders and the criteria/scenarios randomized, and a different set of randomly sampled prompts for each of RPR Criteria and RPR Scenarios. This gives 95% confidence intervals of (0.937, 1) for RPR Criteria and (0.907, 0.993) for RPR Scenarios.

C Finetuning Details

We finetune our model using the Mistral RM prompt in Appendix A. We use 10,167 samples from RPR criteria (training set, 1 sample at random from each paired sample), 10,167 random samples from RPR scenarios (training set, 1 sample at random from each paired sample), and 13,333 random samples from Ultrafeedback. To set the context in the latter, we sample a random 'general' context from:

We finetune using the following hyperparemeters:

Finetuning hyperparameters for Mistral CARM

C.1 Data Composition Ablation

For completion, we finetune the base Mistral reward model using two additional preference datasets for which criteria are available.

First, the Preference Collection ( hf:prometheus-eval/Preference-Collection ), which serves as a training set for Preference Bench (introduced and used in Section 5), can be used for finetuning a criteria aware reward model.

Second, in a concurrent work, Lee et al. [34] have synthesized a dataset of diverse system prompts for finetuning generative models, which they call the Multifaceted Collection ( hf:kaist-ai/Multifaceted-Collection ). As the dataset includes multiple system prompts for the same user instruction, its structure allows it to be used in a similar fashion to our RPR datasets, in order to finetune a reward model.

Using the hyperparameters described above, we finetuned the base model using the following data mixtures:

PC: 40,000 random samples from the Preference Collection.

PC+MF+UF: 13,333 random samples from the Preference Collection, 13,333 random samples from the Multifaceted Collection, and 13,333 random samples from Ultrafeedback.

RPR+UF: 10,167 random samples from RPR criteria, 10,167 random samples from RPR scenarios, and 13,333 random samples from Ultrafeedback. This model, tagged with '(ours)' in the tables, was used in the main text .

RPR+PC+MF+UF: 10,167 random samples from RPR criteria, 10,167 random samples from RPR scenarios, 13,333 random samples from the Preference Collection, 13,333 random samples from the Multifaceted Collection, and 13,333 random samples from Ultrafeedback. This model is therefore trained for longer than the other three models.

The results on context-specific and context-augmented datasets are shown in Table 7.

Best model bolded. Starred ⋆ results are by models finetuned on the same distribution. Models trained on two-sided RPR datasets in green.

D Additional Experiment Details

D.1 Additional Model Details

UltraRM [17] ( hf:openbmb/UltraRM-13b ) is a 13B parameter reward model initialized from Llama-2 [54] and finetuned on UltraFeedback and 3 other open-source preference datasets, and was chosen for its strong performance on open source preference benchmarks.

Prometheus-2 [30] ( hf:prometheus-eval/prometheus-7b-v2.0 ) is a 7B parameter model finetuned to perform fine-grained, context-conditioned evaluation, either as a preference or reward model. It is finetuned on 300K criteria-conditioned samples synthesized with the help of GPT4. We run Prometheus-2 with temperature=0, which may explain why our reported figures on Preference Bench are higher than the figures reported by the authors [30].

Mistral RM [56] ( hf:weqweasdas/RM-Mistral-7B ) is a 7B parameter reward model initialized from Mistral-7B-Instruct-v0.2 [28] and finetuned on a variety of open source preference datasets, and was chosen for its smaller parameter count and strong performance on Reward Bench [33].

Mistral CARM (ours) is our finetuned context-aware version of Mistral RM, finetuned using the RPR datasets. Finetuning details and data ablations may be found in Appendix C.

Llama 3 ( hf:meta-llama/Meta-Llama-3-[8B/70B][-Instruct] ) [1] are a set of strong open source models available as both base models and instruction tuned models. We report results with all four variants, using a modified 'llm-as-a-judge' approach [59] that scores completions using the weighted sum of its score logits: we ask the model to predict a score, and return the expected value of the predicted score under the log probabilities of the score tokens. To enable fast evaluation, we skip the chain-of-thought and do a single forward pass to evaluate the logits. Specifically, we ask the model to rate the completion with a score between 1 and 7.

GPT-4 Turbo is used as the proprietary baseline in experiments. Note that much of the data (RPR datasets and context augmentations for unconditioned preference datasets) was synthesized using GPT-4 Turbo, as described herein. As logits are not available, we use GPT-4 with a standard 'llm-as-a-judge' approach that scores each alternative individually before comparing scores. This suffers from inability to distinguish between ties (which we instead assign randomly).

D.2 Additional Experiment with Gemma 2B Reward Model

To investigate whether our datasets might be useful for a different model size and series, we used the RPR datasets to finetune a reward model based on the 2B parameter Gemma model. We start with an existing baseline reward model that was finetuned from Gemma ( hf:Ray2333/Gemma-2B-rewardmodel-baseline ) [57], and use the same hyperparameters as used for the 7B parameter Mistral reward model. The results on our main datasets are as follows:

We observe that finetuning the Gemma RM shows similar patterns to finetuning the Mistral RM, and that the finetuned model is in many cases competitive with the larger Mistral base model.

D.3 Additional Experiment Details

To augment the datasets, we use the following approach / prompts.

For HHH, we specify the following subset specific contexts:

For Reward Bench, we specify the following subset specific contexts:

To synthesize the 'CTX' contexts for Chatbot Arena, we use the following prompt (note that only the criteria was used as the context, not the teacher's preference):

I would like some help in determining what evaluation criteria should be used to judge the AI Assistant 's responses to the Prompt ([[PROMPT]]) below. [[INSTRUCTIONS]] You are given a Prompt, and two Completions (in random order). You will first evaluate each Completion independently. Then, you will reason about, and the determine, which of the two Completions should be preferred. Finally, you will craft an evaluation criteria that could by used by another judge to evaluate each Completion individually, so that this judge will easily reach the same decision as you. [[REQUIREMENTS]] 1. The criteria should be the "most likely criteria", in the sense that most people in the User's position would agree that the criteria is reasonable (even if they would adopt a different criteria themselves), and the criteria is the most reasonable and likely criteria to have been adopted in the context of your determined preference between the completions. 2. The criteria must be specific, and not overly general. For example, "Directly answers the questions and elaborates by giving a specific example" is sufficiently specific. However, the criteria should NOT rely on overly superficial aspects such as "level of detail" or "simplicity"; for example, "provides a detailed response" is too general. 3. If appropriate given the prompt, the criteria should be even more specific and directly reference certain aspects of the prompt; however, in NO case should the criteria answer any part of the prompt directly. The criteria should be a short but complete sentence that could be used to evaluate the quality of a completion in the context of the prompt.

To synthesize the 'CTX ⋆ ' context for Chatbot Arena, we use the following prompt:

Oracle Context Synthesis

===

[[END OUTPUT

FORMAT]]

Here are the

prompt and completions for

which you will generate

the criteria:

[[PROMPT]]

{prompt}

[[MORE PREFERRED

COMPLETION]]

{chosen}

[[LEAST PREFERRED

COMPLETION]]

{rejected}

D.4 Five profiles used in experiments

The maximum pairwise agreement between the two profiles on the RPR test set was under 80%. Details of how these profiles were created are in Appendix B.1.

Profile 1

The user has an evident appreciation for responses that tackle the ethical and cultural dimensions of topics and prefers information that connects technological choices with their impact on society and the environment. In terms of content, they value detailed, scientifically sound explanations that do not shy away from complexity when it enhances understanding. The user favors structured and logical information delivery but also seeks a human touch, acknowledging the intellectual and emotional efforts behind tasks. They expect precision and are critical of responses that oversimplify or omit important technological nuances. The user has shown a tendency to reject too casual an approach to topics that require technical rigor. Overall, the user demonstrates balanced judgment, weighing the ethical implications and intellectual depth against the clarity and accuracy of the information presented.

Profile 2

The user exhibits a strong preference for precise and accurate information, which suggests a methodical approach to receiving and processing information. They show a clear inclination toward content that is direct, practical, and devoid of unnecessary fluff, valuing succinctness and relevance in responses. Given the user's selections, they appear to prize prior knowledge and intelligent insights over creativity and personal anecdotes. Conversations should be rooted in clarity and grounded in facts and technical accuracy, steering clear of conjecture and subjective embellishments. The user has a measured appreciation for cultural and ethical considerations\u2014being cautious about content that might disrespect or misrepresent cultures and ethical issues, but this does not dominate their criteria for response selection. They also expect both language and content to be accessible without resorting to exaggerated or uncommon language or assumptions that may stray from the immediate topic. Overall, the user is seeking responses that are information-rich, specifically tailored to the question asked, and that refrain from conjecture or personalization.

Profile 3

The user has a marked preference for responses that creatively and narratively interpret information rather than sticking to plain recitation of facts. They appreciate when topics are presented with an element of story or a unique angle, often with a metaphorical flourish. Despite a penchant for creativity, the user does not sacrifice precision or simplicity for the sake of imagery and metaphor; they look for directness when it is called for and appreciate when complex information is made accessible and digestible. Philosophical musings and explorations into the implications of certain concepts or ideas resonate well with them. Responses should steer clear from overly technical jargon or convoluted explanations. While there is an appreciation for the larger, more abstract ideas, there is consistently a return to safety-conscious, applicable, and straightforward advice. Overall, the user seeks a balance between the imaginative and the practical, consistently choosing responses that strike this delicate balance.

Profile 4

The user is drawn to responses that weave narratives, integrate cultural elements, and use poetic language to enrich conversation. This appreciation for creative expression is particularly marked in discussions that extend beyond the factual to the philosophical, ethical, or relational dimensions. They value responses that not only inform but also evoke an emotional or cultural resonance, whether that entails fostering empathy, invoking shared cultural experiences, or presenting information through storytelling. However, this does not preclude an appreciation for clear, efficient communication, especially in technical contexts where brevity and directness are paramount. The user prefers educational methodologies that incorporate engagement with cultural diversity over exercises focused solely on language mechanics. Learning should be an experiential, emotionally satisfying journey rather than a purely intellectual exercise. While humor and creativity are often revered in their profile, there is a clear line where utility and clarity in communication are not to be sacrificed for the sake of entertainment or artistic flair, especially when detailed instructions or precise technical data are sought.

Profile 5

The user exhibits a clear preference for solutions and explanations that are grounded in practicality and economic feasibility, dismissing those that are overly technical or lack real-world applicability . They express a desire for inclusivity and cultural sensitivity, valuing responses that respect diversity and differing backgrounds. Personalization is important to the user, especially if it ensures relevancy to their individual circumstances, but not at the expense of privacy and security. Educational outreach and clear communication are also prioritized, suggesting the user values understanding and transparency when interacting with products and services. Engagement appears to be crucial for the user, who prefers interactive and sometimes humorous elements, as long as these features do not detract from the content's clarity and usefulness. The user appreciates a good narrative or story within responses, indicating a fondness for meaningful context that enriches the information provided. Overall, the user's choices reveal a preference for responses that balance detailed, attentive problem-solving with thoughtful, respectful consideration of the wider social and personal impacts.

D.5 Prompt for profile inference

See Appendix B.1 (we use the same prompt for inferring profiles from data as was used to synthesize the 'ground truth' profiles).

E Compute

Our dataset synthesis and experiments make extensive use of the OpenAI API for use of GPT-4 Turbo (specifically, gpt-4-1106-preview ). Dataset synthesis was by far the most compute intensive task. Finetuning took approximately 8 hours on a single A100. Experiments were run on an internal cluster of GPUs with between 24GB and 48GB VRAM each. Evaluations (each cell in a table) typically take less than 1 hour on a single GPU, but this varies by model and dataset.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Compare abstract and introduction to the body of the paper. While we have an aspirational goal to enable context-aware preference modeling, we think it is clear that our contributions focus on a specific aspects of the problem.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are noted in several places throughout, where there is a call for future work, or caveats embedded in the language. See, e.g., the final paragraph.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail the experimental setup and prompts in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We have released our datasets at https://huggingface.co/datasets/microsoft/ rpr . Unfortunately we unable to release code, but are happy to clarify any details over email.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We believe the experiments are presented to a level of detail that is necessary to appreciate the results, and have included additional details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The only multiseed experiment we run is in Table 6, and we believe we have provided a suitable error for the sample means. While it only applies under the (likely false) assumption that the individual errors are consistent across profiles, we believe that at this sample size, the variance reduction from making the assumption outweighs the bias it introduces.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We described compute resources in Appendix E, but we do not have accurate estimates of how much it costs to run the full experiments. These depend on the internal resources of interested parties.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the Code of Ethics and believe our research conforms with the code. Our paper was also reviewed by two Ethics reviewers.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The societal relevance of our work is limited in isolation, and would be similar to other papers on modeling preferences.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: To our knowledge, we have acknowledged all assets used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See data release.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on fine-tuning LLMs to improve their ability to handle context-aware preference modeling. The authors address the challenge of the underspecified and ambiguous nature of natural language preferences by introducing a two-step modeling process. This includes selecting a context and evaluating preferences within that context. The approach is backed by the introduction of new datasets, named RPR, which are designed to test the effectiveness of context-specific preference modeling. The study provides extensive experimental evidence showing how context-aware models can surpass traditional methods in handling ambiguous and context-specific scenarios.\n\nSTRENGTHS:\n1. The paper tackles a crucial issue in the realm of language modeling by enhancing the LLM's ability to understand and process user preferences in context-dependent scenarios. This is particularly important as LLMs are increasingly used in diverse real-world applications.\n2. The introduction of the RPR datasets is a notable contribution, as these datasets specifically aim to disentangle context-specific preferences from general preferences, offering a valuable resource for further research.\n3. The paper provides thorough experimental results that not only demonstrate the effectiveness of the proposed method but also explore various aspects of context-aware preference modeling, showing improvements over existing models like GPT-4 and Llama 3.\n\nWEAKNESSES:\n1. The paper could benefit from testing the proposed method across a broader range of general benchmarks, such as MMLU and AGI-Eval, to assess how fine-tuning for context-aware preferences might affect the LLM's original capabilities or general applicability.\n2. It remains unclear how applicable the proposed method is to other LMs, especially those of different sizes or from different series. Addressing this would help validate the robustness and versatility of the method.\n\nQUESTIONS:\nPlease see my concerns in weaknesses.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper divides preference modeling into two steps: first estimating the user's intent then evaluate the generated text within the context of this intent. The paper makes the Reasonable Preference Reversal Datasets, which encompass criteria and scenarios for preference data. Experiments find that models can achieve higher performance when providing intent contexts during evaluation.\n\nSTRENGTHS:\n- The approach of first estimating user intent before evaluation may be promising.\n- This work constructs the first open-source context-conditioned preference datasets, which may be useful for future work.\n\nWEAKNESSES:\n1. Lack of citations and comparisons to related work. For example, Li et al. [1] generate the criteria for evaluation and then get the final answer according to the criteria, which is similar to this work.\n\n2. In Table 2, the author aims to demonstrate that fine-tuning a context-aware reward model significantly enhances context-specific performance. However, the observed improvement in Table 2 could be attributed to the model being trained on data from the same distribution as the test set. Even for new datasets that are not context-specific, training the model on such datasets will likely improve its performance on its test sets.\n\n3. The lack of experiments illustrating that two-step preference modeling achieves better performance compared to traditional reward modeling on general preference datasets is a concern. In Table 4, the model is prompted by context generated by GPT-4, leading to unfair comparisons with similar models. To demonstrate that two-step preference modeling is superior, it is essential to show its advantages for models of equal ability.\n\n[1] Li J, Sun S, Yuan W, et al. Generative judge for evaluating alignment[J]. arXiv preprint arXiv:2310.05470, 2023.\n\nQUESTIONS:\nWhy is two-step preference modeling necessary? The user's intent is inherently contained in the query, and traditional preference modeling implicitly evaluates whether the response aligns with the user's intent. Therefore, why should the estimated intent be treated independently? Additionally, ambiguity exists in estimating intent, as different individuals may have inconsistent interpretations of intent for the same query.\n\nLIMITATIONS:\nPlease see Weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper points out that the preference label can be reversed by inserting additional context into the prompt. Based on this observation, the authors build a paired preference dataset. The authors also try to provide some theoretical analysis.

STRENGTHS:
* This paper studies a specific and interesting problem in preference data.
* The proposed data augmentation method may benefit preference optimization of LLMs and inspire further research.

WEAKNESSES:
My major concern lies in the experiments. The current experiments do not demonstrate the general advantage of using a paired dataset with reversed preference labels. On widely used preference datasets, the model trained on the proposed dataset does not outperform the baselines. It only performs better on the test set, which is built in the same way as the training set. Note that the preference labels on this test set are not verified by humans and could be unreliable.

QUESTIONS:
See weaknesses.

LIMITATIONS:
This paper does not have a limitation section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe motivation behind this paper is to address the critical challenges of finetuning language models (LMs) from pairwise preferences due to the underspecified nature of natural language. Direct preference feedback is often uninterpretable, inconsistent, and difficult to provide, especially when multidimensional criteria are involved. These issues arise from incomplete instructions or the diverse backgrounds of the individuals providing the feedback. To tackle these challenges, the authors propose a two-step preference modeling approach: first, selecting a context to resolve under-specification, and second, evaluating preference with respect to the chosen context.\n\nThere are mainly 4 contributions the authors have made:\n\n**Decomposition of Reward Modeling Error**: The paper introduces a method to decompose reward modeling error into two components: context inference error and context-specific reward modeling error. This decomposition supports the idea that supervising both context and context-specific preference could align models more effectively with diverse human preferences.\n\n**Context-Conditioned Preference Datasets**: The authors contribute several novel datasets designed to investigate the ability of LMs to evaluate context-specific preferences. These datasets, termed \"preference reversal\" datasets, isolate context-specific capabilities by disentangling them from general preferences.\n\n**Context-Aware Reward Model**: The paper demonstrates the development and finetuning of a context-aware reward model, showing that this model achieves performance comparable to or exceeding that of state-of-the-art models like GPT-4 and Llama 3 70B. The context-aware model also shows improved performance in context-specific tasks.\n\n**Experiments and Benchmarking**: The authors conduct experiments to benchmark the context-specific performance of various models, highlighting that current models benefit from additional context but often fail to fully utilize it. Finetuning with the preference reversal datasets significantly enhances the models' context-specific performance.\n\nSTRENGTHS:\nThis paper presents a novel approach to preference modeling in LMs by integrating context-specific evaluation. The key contributions include the introduction of context-conditioned preference datasets, the decomposition of reward modeling error, the development of a context-aware reward model, and comprehensive experiments to validate the approach.\n\nWEAKNESSES:\n## Under-supported claims \n\nIn section 3.3, the author included various claims that I believe to be interesting but only so when they are better-substantiated.\n\n**Assumption about Cardinality** The hypothesis that the cardinality of the space of contexts is smaller than that of the space of possible completions given a prompt is stated without empirical or theoretical justification. This assumption is critical to the argument about data efficiency in context annotation versus preference annotation. Without evidence or a rationale to support this assumption, the argument remains speculative.\n\n## Details of the dataset curation\nThe reviewer appreciate the author's efforts in curating the dataset. \nSection 4 briefly mentions that the dataset was generated using GPT-4 Turbo with a series of prompts designed to maximize validity. However, it lacks specific details about the prompts, the criteria for selection, and the methodology for ensuring the validity of the samples. This ambiguity makes it difficult to assess the soundness of the curation process.\n\n## Clarity\nWhile the paper presents interesting ideas, there are some areas in writing and presentation that could benefit from improvement:\n- The introduction and related work sections are detailed, but they could be more concise to better highlight the key contributions and their significance.\n- The figures and tables, although informative, could be optimized to more effectively convey the main points. Some elements might appear redundant or unclear.\n- Enhancing the narrative flow would help in making the logical progression of the arguments clearer and more accessible to the reader.\n\nQUESTIONS:\n## Annotation and intent inference\n\nI particularly do not understand the assumption related to Bernoulli distribution:\n\nSection 3.1 states that an annotator implicitly infers intent i from prompt x and samples a preference from a Bernoulli distribution. This process of intent inference by annotators is critical yet highly under-explained. There is no empirical evidence or studies cited that demonstrate annotators' ability to accurately infer intents from prompts. Additionally, the variability in annotators' interpretations and the potential biases they introduce are not addressed.\n\n## Intent distribution\n\nThe paper asserts that both users and annotators may possess or infer a distribution of intents and that annotation for most preference queries involves a distribution rather than a specific intent. This is a substantial claim that needs empirical support.\n\n## Data curation process\nI would appreciate it if the authors can provide further details on data curation.  In particular, how the quality of the synthetic data is maintained. \n\n## Human validation\nWhat is the scale of this validation, and how are the human validators selected? Is the sample size sufficient to generalize the results to the entire dataset?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}