EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences

## Abstract

Abstract As large language models (LLMs) are deployed globally, creating pluralistic systems that can accommodate the diverse preferences and values of users worldwide becomes essential. We introduce EValueSteer , a benchmark to measure LLMs’ and reward models’ (RMs) steerability towards users’ value and stylistic preference profiles grounded in psychology and human-LLM interaction literature.
To address the gap in existing datasets that do not support controlled evaluations of RM steering, we synthetically generated 165,888 preference pairs –
systematically varying pairs along 4 value dimensions (traditional, secular-rational, survival, and self-expression) and 4 style dimensions (verbosity, readability, confidence, and warmth). We use EValueSteer to evaluate whether, given a user profile and a pair of candidate value-laden and style-laden responses, LLMs and RMs are able to select the output that aligns with the user’s preferences. We evaluate six open-source and proprietary LLMs and RMs under eleven systematic prompting conditions and six preference comparison scenarios.
Notably, our results show that, when given the user’s full profile of values and stylistic preferences, the best models achieve < < 75% accuracy at choosing the correct response, in contrast to > > 99% accuracy when only relevant style and value preferences are provided. EValueSteer thus highlights the limitations of current RMs at identifying and adapting to relevant user profile information, and provides a challenging testbed for developing RMs that can be steered towards diverse human values and preferences.

## 1 Introduction

Large language models (LLMs) are being deployed to mediate discourse for millions (Zheng et al., 2023a; Zhao et al., 2024; Handa et al., 2025; Cheng et al., 2025a). They are used by people who prioritize different values (e.g., traditional vs. secular rational, Inglehart et al. 2014; Haerpfer et al. 2022) and stylistic expectations (e.g., verbose vs. concise, Saito et al. 2023; warm vs. cold, Danescu-Niculescu-Mizil et al. 2013; Cheng et al. 2025b). We thus need pluralistically aligned AI systems (Sorensen et al., 2024; Kasirzadeh, 2024) that can accommodate and adapt to diverse human values (moral and cultural beliefs) and style preferences (communication characteristics like tone and verbosity).

Figure: Figure 1: EValueSteer workflow. Figure illustrates a single evaluation instance. (1) Prompt. Value-laden from PRISM (Kirk et al., 2024) (e.g. “Do you consider family important?”) is posed to the system. (2) Candidate responses. Two completions: one that is *value-aligned* with the user profile and one that is *value-misaligned*; style alignment can vary independently, allowing us to cross value $\times$ style factors. (3) User profile context. The reward model is supplied with a structured summary of the user’s *value preferences* (e.g. prioritizes family) and *style preferences* (e.g. favors warm tone). (4) Scoring and selection. Using the prompt, profile, and candidate responses, the reward model selects the response it believes the user prefers.
Refer to caption: x1.png

In this work, we study the steerability of LLMs towards a user’s value and preference profile; specifically, we examine reward models (RMs) – LLM-as-a-Judge and LLMs fine-tuned for preference classification (Zheng et al., 2023b; Lambert et al., 2024; Liu et al., 2024b) – as they are the cornerstone for aligning AI systems with human values and preferences.

Although RMs were originally trained on an “average-annotator” signal that led to uniform preferences (Bai et al., 2022; Ouyang et al., 2022b), recent work has also focused on improving RM controllability so that they can be adapted to diverse moral, cultural, and stylistic contexts for individual users through minimal contextual instruction (Lambert et al., 2024; Malik et al., 2025). Existing value alignment benchmarks (Sorensen et al., 2023; Chiu et al., 2025) study LLM values in specific scopes, such as moral dilemmas, but do not explicitly consider how well models can be steered to follow values and styles for tasks such as reward modeling. Further, the absence of large-scale human-annotated datasets that systematically vary both value and style dimensions, especially when value and style conflict,
necessitates a synthetic approach to isolate and evaluate RM steerability.
To bridge this gap, we introduce EValueSteer,(^1^11We provide our code and data at [https://github.com/kshitishghate/EVALUESTEER-benchmark](https://github.com/kshitishghate/EVALUESTEER-benchmark).) a value-controlled benchmark that evaluates the in-context steering capabilities of RMs to value-laden content and style (See Figure [1](https://arxiv.org/html/2510.06370v2#S1.F1)). Given a prompt, we generate two outputs that differ along value dimensions while simultaneously varying stylistic features such as verbosity. A user profile synthetically derived based on human-grounded value data from the World Values Survey (WVS) (Inglehart et al., 2014; Haerpfer et al., 2022) provides a joint value–style target in the form of a candidate response that reflects the desired value/style. The RM’s task is to select the option that best matches this specification. Since all other linguistic and semantic factors remain constant, EValueSteer reveals whether models can be steered toward the correct joint outcome under systematically varied prompt conditions and helps isolate the implicit biases of RMs to certain values or style signals.

We instantiate our benchmark with: 1) 4 value dimensions drawn from 10 WVS value-loading questions broadly categorized under traditional, secular-rational, survival, and self-expression dimensions, derived from 233 value-expressing statements in the World Values Survey (Haerpfer et al., 2022) and converted into natural language statements following Jiang et al. (2024), and 2) 4 well-studied style families including verbosity (Saito et al., 2023), reading difficulty (Tran et al., 2024; Jin et al., 2025a), confidence (Lee et al., 2024; Steyvers et al., 2024; Zhou et al., 2024), and warmth (Danescu-Niculescu-Mizil et al., 2013; Cheng et al., 2025b). Across 165,888 preference pair evaluations that include six different pairwise preference combinations for a given style and value dimension, we evaluated the steerability of six open-source and proprietary LLMs and RMs under 11 systematic prompting settings.

Our results highlight three key points. First, giving reward models explicit context for value and style profile boosts pairwise preference accuracy from 42% to $\approx$ 75% in the best performing RM with full user context. However, a 25-point gap to the oracle scenario persists when only relevant context is given, allowing significant room for improvement. Second, EValueSteer allows us to demonstrate that out-of-the-box RMs have strong secular value leanings and a preference for verbose and confident language. Third, value information has the greatest impact on large LLM-based judges, but when value and style cues conflict, models still favor adhering to stylistic preferences, a clear “style-over-substance” bias (Feuer et al., 2024; Liu et al., 2024c). These findings suggest that current RMs may lack the ability to adapt to relevant aspects of a user’s profile for a given query, and systematically favor certain cultural and stylistic perspectives, potentially limiting their effectiveness for aligning AI systems to users with diverse values or stylistic preferences.

## 2 Related Work

#### Value Alignment Benchmarks

Early efforts in evaluating value alignment, such as ETHICS (Hendrycks et al., 2021) and Delphi (Jiang et al., 2022), focused on classifying simple scenarios with normative moral values that are straightforward for today’s models. More recent work on alignment, such as Value Kaleidoscope (Sorensen et al., 2023) and IndieValueCatalog (Jiang et al., 2024), investigates LLM capabilities to generate fine-grained pluralistic and individualistic values, but still only consider simple decisions. Alternate lines of work have also extended to complex and real-world moral dilemmas (Jin et al., 2022; Chiu et al., 2025) as well as large-scale global and diverse linguistic contexts (Awad et al., 2018; Jin et al., 2025b). However, previous alignment literature has not considered the steerability of preference-ranking models. The evaluation of LLM steerability remains a nascent area of research, with limited coverage across task types (Li et al., 2025; Liu et al., 2024a; Samuel et al., 2024) and a predominant focus on text generation over reward modeling. We address these gaps in the literature with EValueSteer by evaluating the steerability of preference-ranking models specifically for alignment.

#### Reward Models and LLMs for Preference Modeling

Alignment work typically utilizes reward models (RMs) and LLM-as-a-Judge. LLMs as judges are typically LLMs prompted to emulate human preferences (Zheng et al., 2023b), whereas RMs are typically language models fine-tuned on labeled preference data to model human preferences (Bai et al., 2022; Ouyang et al., 2022a; Dai et al., 2024; Ji et al., 2023; Cui et al., 2024). They are widely used to align language models during post-training using reinforcement learning from human feedback (RLHF), direct preference optimization (Rafailov et al., 2023), etc. Their increasing importance in post-training pipelines has led to a wealth of work in evaluating reward models. Benchmarks such as RewardBench (Lambert et al., 2024), RewardBench 2 (Malik et al., 2025) and others (Liu et al., 2025; Zhou et al., 2025; Frick et al., 2025) focus on evaluating general downstream performance across various domains. Other works test specific attributes or settings such as multilinguality (Gureja et al., 2024), agentic systems (Jin et al., 2024; Lù et al., 2025), and mathematical reasoning (Kim et al., 2025). However, existing general purpose evaluations do not account for annotator discretion in more subjective settings (Buyl et al., 2025) or biases in reward models (Mire et al., 2025; Christian et al., 2025). EValueSteer aims to address this gap by specifically testing the steerability of reward models towards values.

#### Pluralistic Alignment

It has been argued that aligning AI systems with different human goals and intentions requires operationalizing the plurality of values endorsed by groups of people or contexts of communication, rather than imposing a single moral and normative on LLMs (Kasirzadeh & Gabriel, 2023). Recent works have focused on aligning LLMs with a diverse set of human values and preferences so that AI systems can cater to a more global audience (Sorensen et al., 2024).
Such pluralistic alignment requires preference datasets that represent a diverse set of users or principles.
Yet, existing approaches fall short in enabling the evaluation of whether RMs can identify and adapt to the relevant aspects of a user’s profile for a given query, as testing RM steerability to values and styles requires knowing ground-truth preferences for specific value-style combinations, which is often underspecified in real-world data. PRISM (Kirk et al., 2024), while providing valuable human-collected preferences from diverse groups, lacks ground truth value or style preference specifications that can be isolated to explain preference choices in LLM responses.
PERSONA (Castricato et al., 2024) and Zollo et al. (2024), generate synthetic personas but focus primarily on demographic attributes and simple preferences rather than systematically varying both deep-seated values and stylistic dimensions. SALMON (Sun et al., 2023) generates AI feedback based on a small set of arbitrary human-defined principles. Community Alignment (Zhang et al., 2025) provides human-collected preference feedback across multiple languages, but similarly lacks the controlled variation needed to isolate how models respond to specific value-style combinations.

## 3 EValueSteer Creation

In this section, we introduce our dataset, task and evaluation setup, and key analysis metrics. Our benchmark simulates users who differ along two orthogonal axes of values and style preferences, and tests whether RMs (^2^22Note: For the remainder of the paper, we use RM to refer to both fine-tuned RMs and LLMs-as-judges.) can select the response that best fits a user provided with context about their preferences. We first describe how we build value profiles from the World Values Survey (WVS), then add stylistic preferences, select prompts, and generate quality-controlled answer pairs. Finally, we formalize the evaluation task and prompting conditions.

Our benchmark is synthetically generated to ensure systematic control of the various attributes.
Recent work argues that controlled evaluations (Ye et al., 2024; Chollet et al., 2024; Sinha et al., 2025) with simplified tasks removing
confounders often provide an upper bound on model capabilities, and real-world scenarios invariably introduce additional complexity that degrades performance. In our setting, obtaining faithful, uncontaminated “in-the-wild” comparisons becomes especially brittle. Constructing conversations from the ground up lets us precisely manipulate value and style signals, allowing us to treat the synthetic setup as a controlled upper bound on RM steerability.

#### Value-Profile Construction

Following Inglehart et al. (2014) we operationalize the Inglehart–Welzel (IW) map using ten items from wave-7 of the WVS (Haerpfer et al., 2022) to select diverse survey-based user value profiles for our study. The subset of 10 WVS items efficiently captures two fundamental and orthogonal value dimensions which map to: (1) Traditional vs. Secular-Rational values and (2) Survival vs. Self-Expression values. This compact space systematically represents the cultural value profiles of over 94,000 users validated in six surveys that cover 100+ countries, and explains 70–75% of intercountry variance (Inglehart et al., 2014). With survey responses to the 10 WVS items, we classify $\approx$ 94k WVS respondents to four value quadrants: Traditional-Survival (41.0% respondents), Secular-Self-Expression (29.0%), Secular-Survival (17.2%), and Traditional-Self-Expression (12.8%). We detail our approach in Appendix [A.2](https://arxiv.org/html/2510.06370v2#A1.SS2).

We treat each survey respondent as a potential value profile to be used in EValueSteer, thereby grounding our profile creation in real-world human value data. We implement a diversity-maximizing selection procedure that ensures comprehensive value coverage across all WVS items with available data. We identify 18 unique value combinations that span all four cultural value quadrants, representing specific intersections of traditional / secular and survival / self-expression orientations (e.g., “Religious + Materialist”, “Pro-autonomy + Optimistic”). We provide detailed descriptions on the selection of value combinations to ensure maximum diversity and coverage in Appendix [A.3](https://arxiv.org/html/2510.06370v2#A1.SS3). Finally, each raw WVS survey item (e.g., “Immigrants increase crime rates”, 1 = Agree strongly …4 = Disagree strongly) is rewritten in the IndieValue (Jiang et al., 2024) natural‑language template, which preserves polarity while producing value-expressing statements that the LLM understands.

#### Style-Profile Construction

We identify four orthogonal style families, each with two levels: verbosity (verbose / concise), reading difficulty (high / low), confidence (high / low) and warmth (warm / cold). These style families are chosen from a broad literature of work that identifies linguistic and psychological characteristics in computational linguistics text that have been widely studied in the context of LLMs (Saito et al., 2023; Liu et al., 2024c; Tran et al., 2024; Jin et al., 2025a; Lee et al., 2024; Steyvers et al., 2024; Zhou et al., 2024; Danescu-Niculescu-Mizil et al., 2013; Cheng et al., 2025b; Feuer et al., 2024). We further justify our style choices in Appendix [A.4](https://arxiv.org/html/2510.06370v2#A1.SS4).

#### User Profile Creation

By combining our 18 distinct value profiles with 16 style preference combinations, we create 288 user profiles. Each profile is encoded with both cultural value orientations (derived from WVS responses) and consistent style preferences across all four style families. We restrict ourselves to these user profiles to account for the increase in computation requirements as we evaluate our RMs across the settings described below.

#### Prompt Selection

To ground out data generation in real world queries posed by users to LLM assistants, we retrieve prompts from PRISM (Kirk et al., 2024) and match them to selected WVS items. To ensure broad topic coverage, we expand our chosen WVS items over the 10 IW questions, and follow (Li et al., 2024) to choose a subset of 14 questions (2 per survey topic category) from their seed dataset. We provide further details on prompt selection and matching with WVS items in Appendix [A.5](https://arxiv.org/html/2510.06370v2#A1.SS5). Overall, we retain 24 WVS statements and PRISM prompt pairs for evaluations.

#### Response Generation and Quality Filtering

For every ⟨WVS statement, PRISM prompt⟩ pair, we queried GPT‑4o to produce two assistant responses that: (i) read as stand‑alone replies to the PRISM prompt, (ii) implicitly reflect opposite poles of the WVS statement (e.g., pro‑ vs. anti‑migration) which are considered value aligned or misaligned to a given user value profile, and (iii) never quote or mention the WVS text verbatim. Answers are then rewritten into the 16 style variants using style-control prompts, giving $24\times 16{=}384$ completions.

We filter our conversations (prompt and response pairs) for quality by selecting those where a highly capable RM (GPT-4.1) chooses the preferred response perfectly when provided only the relevant value and style information (henceforth termed the Oracle Setting). In doing so, we ensure that conversations implicitly encode the values and style signals that we aim to evaluate using EValueSteer. We share our prompts to generate value and style-laden responses in Appendix [C](https://arxiv.org/html/2510.06370v2#A3). We validate our synthetic generation method through human validation of the 10 core WVS items across comprehensive metrics that check for value fidelity, style adherence, comparative quality, and data contamination. On average, our data samples were deemed valid at 98.6% with a percentage agreement of 95.5%. We report the details in Appendix [D](https://arxiv.org/html/2510.06370v2#A4).

## 4 Task Setup

For each user profile and question / conversation combination, we generate 6 preference pairs, systematically crossing value-based and style-based preference rules.

#### Value-Based Preferences (4 pairs per conversation):

Both choice responses differ in their value-laden content, where one aligns with the WVS statement while the other is less aligned to it. The correct choice is the answer whose stance matches the user’s coded values. It is possible for the pair of responses to have matching or opposing style dimensions. In neutral settings, we use alignment to the user’s value to break ties in choosing the favored response.

#### Style-Based Preferences (2 pairs per conversation):

Both answers take the same stance, but one reflects the user’s preferred style (verbosity, reading difficulty, confidence, warmth) and the other reflects the opposite. Here, the style-preferred answer is considered correct.

We then evaluate our RMs across in-context prompting settings (full details in Appendix [B](https://arxiv.org/html/2510.06370v2#A2)) that systematically vary with:

1. Context Types: No context, WVS values only, style preferences only, or combined (which includes both WVS and style) contexts.

2. Context Scope: All experiments involve providing full context with and without Chain-of-Thought (COT) prompting (Wei et al., 2022). We define full WVS context as incorporating all available value information in natural language form for a selected user value profile (N$\approx$200 statements). Similarly, we define full style context as incorporating all style statements (N=4 statements with detailed descriptors) across the 4 style families. Our choice of context scope allows us to evaluate how well RMs can pick out relevant user information for steering in a given conversation. We also incorporate COT as an additional setting variant following Ankner et al. (2024), who show that prompting LLM-as-a-judge reward models to generate natural language before making a final reward judgment can improve reward model performance.

3. Preference Priority: Values prioritized, style prioritized, or neutral (no guidance). While we recognise values and styles are orthogonal, here we evaluate whether an RM can recognise the difference across responses and steer accordingly.

4. Context Order: We ensure randomised order of WVS and style contexts in all combined settings.

A full example datapoint used to evaluate an RM under the combined value and style context setting is provided in Appendix [E](https://arxiv.org/html/2510.06370v2#A5).

**Table 1: Dataset statistics for EValueSteer. Across 11 prompting settings, we run 165,888 pair-wise preference evaluations derived from 24 WVS-based conversations, 4 style augmentations, 6 preference combinations, and user profiles built from 18 value and 16 style profiles. We provide full prompting details in Appendix [B](https://arxiv.org/html/2510.06370v2#A2).**
| Dataset Components |  |  |  |
| --- | --- | --- | --- |
| User value profiles | 18 | Style augmentations/conversation | 4 |
| User style profiles | 16 | Pairwise preference combos/conversation | 6 |
| Synthetic conversations | 24 |  |  |
| Total pairwise preference evaluations | 165,888 |  |  |
| Prompting settings | 11 |  |  |

## 5 Experiments and Results

In this section, we present the results of our experiments evaluating 6 RMs that differ in size and modeling procedure (LLM-as-a-judge vs. classifier based). We tested four LLM-as-a-judge models: GPT-4.1-Mini (Achiam et al., 2023), Gemini-2.5-Flash (No Thinking) (Comanici et al., 2025), Llama-3.1-8B (Dubey et al., 2024), Qwen-2.5-7B (Yang et al., 2025), and two trained RM classifiers: Skywork-Llama-3.1-8B (Liu et al., 2024b), and Skywork-Qwen-3-8B (Liu et al., 2024b).(^3^33We selected models based on the [RewardBench](https://huggingface.co/spaces/allenai/reward-bench) leaderboard, and to ensure sufficient representation of proprietary/open and generative/classifier-based models.)

### 5.1 RQ1: RM Performance Across Contexts

Figure: Figure 2: Performance improvements from supplying user context. Bars show the mean pair-wise accuracy (% of preference pairs correctly ranked; higher is better) achieved by the reward model (RM) under five conditions: *No Context*, *Value only* context, *Style only* context, *Value + Style* context with different priority orders. Bars with cross-overs indicate CoT numbers for LLM-as-a-judge RMs. A dashed horizontal line marks random performance at 0.5. Takeaway: Conditioning on values or styles improves performance, but even the best setting remains over 25% below oracle levels, highlighting the limited steerability of current reward models to pluralistic preferences.
Refer to caption: x2.png

We first evaluate RMs’ ability to be steered by explicit values or styles across different contexts, when compared to a contextless baseline, where we expect random chance performance, as our dataset is specifically balanced. Next, we evaluate whether chain-of-thought can improve RM performance over our different context levels.

Figure [2](https://arxiv.org/html/2510.06370v2#S5.F2) (and the accompanying Table [2](https://arxiv.org/html/2510.06370v2#A6.T2) in the appendix) compare RM accuracy across the different levels of context given in Section [4](https://arxiv.org/html/2510.06370v2#S4). We find that separately providing user value or style context leads to a 12.11% aggregate improvement in RM accuracy over the no-context baseline. Providing both value and style yields an additional improvement over providing one or the other, especially when the correct ordering of importance between value and style is specified. However, notably, RMs only achieve a maximum accuracy of 75% when full user context information is given, despite the data having been selected by our Oracle setting such that it is recognizable at 100% given the right context. Furthermore, value context appears more impactful than style context in guiding preference selection for LLM-as-judge models, while classifier-based models are able to do better on style-based preferences. The further gains seen when combining value and style context, especially when values are prioritized, demonstrate the necessity of modeling both dimensions jointly to capture the broad spectrum of human preferences.

We find noticeable gains from incorporating chain-of-thought for GPT-4.1-Mini and Qwen-2.5-7B (increase of 5-6% and 2-4% in value steering contexts, respectively). A closer context-wise analysis of this reveals that gains are concentrated more in cases where value context is provided and/or prioritized, whereas the improvements dip for just style context. Interestingly, we do not see improvements with CoT in Gemini-2.5-Flash and Llama-3.1-8B. For the former, this is likely due to our added constraint of a zero thinking budget and hard token output limit (of 800 tokens) that might have affected the reasoning performance. For the latter, we hypothesize that the ability of the RM to recognize relevant values largely impacts its steering capabilities in real-world settings. Overall, the results suggest that, compared to surface-level style characteristics, alignment to individual values may benefit more from RM approaches that leverage CoT.

Table [2](https://arxiv.org/html/2510.06370v2#A6.T2) provides a further breakdown of RM performance disaggregated by style family and WVS quadrant for a given user profile and prompting setting. For style families, when the prompting setting is kept constant, RM performance is mostly consistent for all style families except warmth, where the accuracy drops across the board by $\sim$3%. This behavior suggests that incorporating warm-cold styles into value-laden responses can further confound the RM and affect its performance. For WVS quadrants, overall, all RMs tend to perform 1-2% better for Secular-Self-Expression and Secular-Survival users, pointing to a potential bias of RMs steerability for specific values. We explore these biases further in the next RQ in  [5.2](https://arxiv.org/html/2510.06370v2#S5.SS2).

We highlight qualitative examples where GPT-4.1-Mini with CoT prompting successfully predicts user preferences when baseline prompting fails in Appendix [F.1](https://arxiv.org/html/2510.06370v2#A6.SS1), specifically focusing on value-only context. CoT chains follow a structured approach by extracting relevant cultural values of the user. For example, in a conversation associated with Religious Belief (WVS Q164) and a Secular-Survival User, in the CoT setting, the model identifies how the “User states ‘God is not important in my life’ but values traditional norms for children.” The baseline prompting method selected a faith-affirming response, but the CoT method selected a response that acknowledges secular alternatives.

Figure: Figure 3: Intrinsic Value bias of RMs. Plots on the left show the proportion of times each RM chooses a response aligned with a value related to an Inglehart-Welzel value loading question. Proportions in blue represent Secular/Self-expression responses. Proportions in red represent Tradition/Survival responses. Scatter plot on the right places each of the RMs in the 4 cultural value quadrants defined by (Inglehart et al., 2014).
Refer to caption: x3.png

### 5.2 RQ2: Systematic Value-Style Biases in RMs

Next, we leverage the controlled design of EValueSteer to systematically isolate and measure the inherent biases that RMs exhibit toward specific values and styles. We are able to do this precisely, because our benchmark independently varies value and style dimensions while holding all other factors constant. We consider RM preference rates for individual styles and values without context, steering the model for this.

Figure [3](https://arxiv.org/html/2510.06370v2#S5.F3) details the individual responses of each RM to the 10 value loading WVS items. We see that all RMs largely agree on 5 / 10 question items related to the importance of God (not important), post-materialism (prioritize self-expression and quality of life to economic and physical security), national pride (low) and favoring more respect for authority (low). They moderately agree on life satisfaction (low) and political participation (low). We note some interesting disparities in their responses to conversations that cover abortion, homosexuality, and interpersonal trust.

To study value biases at a higher level than individual WVS questions, we project each RM’s responses onto the Inglehart-Welzel cultural map by averaging across the four value “quadrants” defined in Section [4](https://arxiv.org/html/2510.06370v2#S4), using the same method applied to survey respondents (Appendix [A.2](https://arxiv.org/html/2510.06370v2#A1.SS2)). This standard visualization approach enables direct comparison between RM value preferences and documented human cultural values at both individual and aggregate (e.g., country-level) scales, revealing where models might align with or diverge from global human value distributions.
In doing so, we find in Figure [3](https://arxiv.org/html/2510.06370v2#S5.F3) (right-hand side plot) that models systematically prefer responses that support Secular-Self-expression value profiles. RMs show strong secular tendencies (e.g., low importance of God). In line with previous work on LLM cultural values (Tao et al., 2024), our results show that RMs favor responses associated with secular rather than traditional values on key dimensions of the World Values Survey, including lower importance of religious elements, more liberal positions on moral issues such as abortion and justifiability of homosexuality, reduced emphasis on authority and obedience, and lower national pride. RMs also exhibited a moderate orientation towards self-expression, prioritizing quality of life over physical security and economic concerns.

Figure [5](https://arxiv.org/html/2510.06370v2#S5.F5) shows that all RMs have stylistic preferences, favoring responses that are more verbose, cold / formal, higher in reading difficulty and, specifically for GPT-4.1-Mini, more confident. These findings align with prior work documenting stylistic biases in LLM-as-a-judge systems (Saito et al., 2023; Lee et al., 2024; Zhou et al., 2024). Interestingly, Gemini-2.5-Flash significantly prefers concise over verbose, likely due to its alignment data that emphasizes information-dense phrasing for latency-bounded applications (Comanici et al., 2025).

Figure: Figure 4: Intrinsic style bias of RMs. Bars show the proportion of times the styles: verbosity, high confidence, warmth, high reading difficulty, are chosen over the respective opposite style: concise, low confidence, cold, and low reading difficulty. Proportions significantly different from 0.5 when error bars (at 95% CI) do not overlap the dashed reference line. RMs exhibit significant verbosity, high reading difficulty.
Refer to caption: x4.png

### 5.3 RQ3: RM Bias’ Impact on Steerability between Values and Styles

Finally, we seek to understand whether RMs’ intrinsic style-value biases impact how they choose a response when both contexts are provided, but we do not explicitly specify whether the model prefers style or values. Specifically, we analyze the comparison setting when values and styles are in conflict. One response aligns to the user’s preferred value, but is in a dispreferred style. The other response reflects the style preferred by the user, but is less aligned to their value than the first. In this setting, we compare what the RMs tend to prefer. Overwhelmingly, we find that models steer to the user’s style preferences over values by 32.83% across the 6 RMs. The error bars indicate that the difference in preference rates for all RMs is significant at 95% CI, with Skywork-Qwen showing the largest gap at 46.6%. Finally, Figure [9](https://arxiv.org/html/2510.06370v2#A6.F9) shows that small LLM-as-a-judge and classifier-based RMs are better at steering to style over values.

We conduct an additional analysis study (Figure  [8](https://arxiv.org/html/2510.06370v2#A6.F8)) to ensure that no confounds are being introduced in the comparison between value and style steering due to excess amount of context dedicated to values (N$\approx$200 value expressing statements) over styles (N=4 style expressing statements) in the construction of the user profile. We sample 4 WVS statements and style statements each, such that at least one of the WVS statements is relevant to the conversation. Under value and style conflict conditions similar to the full experiment, where the RMs are not given explicit conflict resolution guidance, we again see significant bias in steering towards responses that feature the preferred styles over values, strengthening our earlier findings.

## 6 Conclusion & Discussion

We introduce EValueSteer, a novel benchmark for evaluating the steerability of reward models and large language models toward pluralistic value and style preferences. By synthesizing a large-scale, human grounded dataset combining World Values Survey items with systematic stylistic augmentations, we provide a rigorous and controlled testbed for assessing how well RMs can predict user preferences that jointly encode values and styles. Our extensive evaluations with six RMs reveal key findings with important implications for RM design and deployment for a pluralistic society.

#### RMs fall short on value-and style-aligned steering.

Our results with EValueSteer reveal important insights into the (lack of) steerability when conditioned on user profiles that combine value and style preferences. First, while RMs can leverage in-context information to align outputs with user preferences, supporting the utility of context-aware alignment techniques, they fall short by more than $\approx$25% of the oracle setting i.e. when only relevant value and/or style information is provided. Our experiments with CoT prompting suggest that RMs could benefit from explicit value reasoning capabilities. Future work can investigate how to incorporate this while training RMs. Additionally, a more targeted analysis can yield more insight into how we can improve user value profile modeling so that RMs can better predict downstream preferences.

#### RMs exhibit implicit bias preferences to secular and self-expression values .

Our analysis of the biases in RMs reveals a systematic secular bias and moderate self-expression bias. These choices mirror findings that LLMs cluster near English-speaking Protestant European populations on the Inglehart–Welzel map, reflecting the cultural composition of their pre-training data and annotation sources (Tao et al., 2024). The models diverge most on morally contentious items, including abortion, homosexuality, and interpersonal trust. Previous work shows that LLM judgments on certain moral dilemmas (e.g. abortion) are highly sensitive to prompt wording and context, sometimes oscillating between utilitarian and rights-based rationales (Papadopoulou et al., 2024; Kabir et al., 2025). Likewise, attitudes toward LGBTQ+ rights hinge on whether religious justifications are foregrounded, exposing tension between the models’ secular default and their attempt to reflect the prompt’s moral framing. Interpersonal variability is consistent with recent evidence that LLMs do not learn a stable theory of social trust but instead interpolate between conflicting patterns in their data (Xie et al., 2024). Such implicit biases risk marginalizing users whose values diverge from dominant cultural norms.

#### RMs exhibit implicit bias preferences to verbose, cold, and confident styles .

Figure [5](https://arxiv.org/html/2510.06370v2#S5.F5) shows that 5/6 RMs lean toward verbose and assertive/confident completions. This reproduces the well-documented “verbosity bias”, in which LLM judges award higher scores to longer answers even when brevity would be more appropriate (Saito et al., 2023) and the related preference for responses that lack epistemic hedges or uncertainty markers (Lee et al., 2024). Furthermore, a significant preference for cold and high reading difficulty for some RMs suggests that the default “professional/formal” register of instruction-tuning corpora still dominates even when the user prefers a warmer style. The strong stylistic preferences observed in the RMs reveal implicit biases that could limit the adaptability of RMs to pluralistic profiles.

#### RMs prioritize steering to style over values .

Our value-style steering interaction analysis indicates that RMs tend to default to aligning with user styles over values when no explicit prioritization is provided, highlighting a latent hierarchy in RM preference modeling that favors styles as more fundamental. We tie our findings here to our findings in RQ2 and existing work that demonstrates “style over substance” bias (Feuer et al., 2024; Liu et al., 2024c), highlighting another dimension of stylistic biases in RMs. We see that regardless of the size or method, RMs favor aligning to user style preferences. Attributing these limitations to specific factors in RM capabilities signifies key next steps.

## 7 Limitations

Our benchmark relies on synthetic conversations that combine WVS items with stylistic rewrites. Although the WVS is broad, it still reflects a particular survey tradition and omits many culturally specific values (e.g., Indigenous world views or non-Western conceptions of relationality). Likewise, our four style families, although orthogonal, do not capture the full extent of the stylistic space of text. Models judged “aligned” under these constraints may fail in more nuanced or intersectional user profiles.

Next, our usage of the term “misaligned” response does not always mean diametrically opposing the given value, but instead is used to refer to the response that is clearly less aligned than the alternative. There are cases where the “misaligned” response still espouses some views that could be considered aligned (for example, in issues related to abortion, the misaligned response would just be more neutral than the fully aligned response). For the purpose of this study, we assume that a user’s value preferences are satisfied by the more “aligned” response.

We generate candidate completions with a single LLM configuration. Therefore, the benchmark assesses selection among fixed alternatives rather than open-ended generation. Different settings or models could yield completions that interact with value and style cues in unexpected ways. Pair-wise preference labels assume that one response is clearly preferable, but in practice user preferences are context-dependent and sometimes inconsistent with their stated values.

The reward models we test are commercial- or research-grade LLMs trained on opaque corpora. Any biases that we measure may change with future model updates.

## 8 Ethical Considerations

User profiling. We simulate detailed value and style profiles to test steerability. Real-world deployment would require collecting comparable data from users, raising privacy and consent questions. Profiling systems must obtain explicit permission, provide transparency about data use, and allow users to inspect or delete their stored profiles.

Cultural representation. Since EValueSteer is grounded in WVS items and English-language prompts, it may be biased toward Western secular perspectives. Systems trained to optimize on our dataset risk marginalizing communities whose values fall outside the WVS schema. Future iterations should incorporate region-specific surveys and non-English dialogues to mitigate this bias.

Manipulation risk. Techniques that improve steerability can also be misused to push users to particular ideological positions. Researchers and practitioners should pair alignment advances with safeguards to detect and deter manipulative use.

Bias reinforcement. We find that current RMs prefer secular values and verbose, confident language. Optimizing downstream systems against such biased judges could further establish these preferences.

Human oversight. Finally, our work shows that RMs are imperfect proxies for pluralistic human judgment. High-stakes applications such as medicine, law, finance, should retain a human-in-the-loop review pipeline.

## 9 Reproducibility statement

We provide all details of benchmark creation, validation and evaluations Sections [3](https://arxiv.org/html/2510.06370v2#S3) and [4](https://arxiv.org/html/2510.06370v2#S4), and mention any additional information required for reproducibility in the corresponding Appendix sections. We also make our code and data available at [https://github.com/kshitishghate/EVALUESTEER-benchmark](https://github.com/kshitishghate/EVALUESTEER-benchmark).

## 10 Acknowledgments

This work was supported by the U.S. National Science Foundation (NSF) CAREER Award 2337877. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of NSF. We are grateful to anonymous reviewers for their helpful feedback during the preparation of this version of the paper.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Ankner et al. (2024)
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Daniel Chang, and Prithviraj Ammanabrolu.
Critique-out-loud reward models.
In *Pluralistic Alignment Workshop at NeurIPS 2024*, 2024.
- Awad et al. (2018)
Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan.
The moral machine experiment.
*Nature*, 563(7729):59–64, 2018.
- Bai et al. (2022)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jeff Ladish, J Landau, Kamal Ndousse, Kamilė Lukoiūtė, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noem’i Mercado, Nova Dassarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Sam Bowman, Zac Hatfield-Dodds, Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, and Jared Kaplan.
Constitutional ai: Harmlessness from ai feedback.
*ArXiv*, abs/2212.08073, 2022.
URL [https://api.semanticscholar.org/CorpusID:254823489](https://api.semanticscholar.org/CorpusID:254823489).
- Buyl et al. (2025)
Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio Cesar Vieira Machado, and Flavio du Pin Calmon.
Ai alignment at your discretion.
In *Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency*, FAccT ’25, pp. 3046–3074, New York, NY, USA, 2025. Association for Computing Machinery.
ISBN 9798400714825.
doi: 10.1145/3715275.3732194.
URL [https://doi.org/10.1145/3715275.3732194](https://doi.org/10.1145/3715275.3732194).
- Castricato et al. (2024)
Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn.
Persona: A reproducible testbed for pluralistic alignment, 2024.
URL [https://arxiv.org/abs/2407.17387](https://arxiv.org/abs/2407.17387).
- Cheng et al. (2025a)
Jingwen Cheng, Kshitish Ghate, Wenyue Hua, William Yang Wang, Hong Shen, and Fei Fang.
Realm: A dataset of real-world llm use cases.
*arXiv preprint arXiv:2503.18792*, 2025a.
- Cheng et al. (2025b)
Myra Cheng, Sunny Yu, and Dan Jurafsky.
Humt dumt: Measuring and controlling human-like language in llms.
*arXiv preprint arXiv:2502.13259*, 2025b.
- Chiu et al. (2025)
Yu Ying Chiu, Liwei Jiang, and Yejin Choi.
Dailydilemmas: Revealing value preferences of LLMs with quandaries of daily life.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=PGhiPGBf47](https://openreview.net/forum?id=PGhiPGBf47).
- Chollet et al. (2024)
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers.
Arc prize 2024: Technical report.
*arXiv preprint arXiv:2412.04604*, 2024.
- Christian et al. (2025)
Brian Christian, Hannah Rose Kirk, Jessica A.F. Thompson, Christopher Summerfield, and Tsvetomira Dumbalska.
Reward model interpretability via optimal and pessimal tokens.
In *Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency*, FAccT ’25, pp. 1048–1059. ACM, June 2025.
doi: 10.1145/3715275.3732068.
URL [http://dx.doi.org/10.1145/3715275.3732068](http://dx.doi.org/10.1145/3715275.3732068).
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Cui et al. (2024)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun.
Ultrafeedback: Boosting language models with scaled ai feedback, 2024.
URL [https://arxiv.org/abs/2310.01377](https://arxiv.org/abs/2310.01377).
- Dai et al. (2024)
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang.
Safe rlhf: Safe reinforcement learning from human feedback.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=TyFrPOKYXw](https://openreview.net/forum?id=TyFrPOKYXw).
- Danescu-Niculescu-Mizil et al. (2013)
Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts.
A computational approach to politeness with application to social factors.
*arXiv preprint arXiv:1306.6078*, 2013.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv e-prints*, pp. arXiv–2407, 2024.
- Feuer et al. (2024)
Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P Dickerson.
Style outweighs substance: Failure modes of llm judges in alignment benchmarking.
*arXiv preprint arXiv:2409.15268*, 2024.
- Frick et al. (2025)
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica.
How to evaluate reward models for RLHF.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=cbttLtO94Q](https://openreview.net/forum?id=cbttLtO94Q).
- Gureja et al. (2024)
Srishti Gureja, Lester James V Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee.
M-rewardbench: Evaluating reward models in multilingual settings.
*arXiv preprint arXiv:2410.15522*, 2024.
- Haerpfer et al. (2022)
Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen.
World values survey wave 7 (2017-2022) cross-national data-set.
*(No Title)*, 2022.
- Handa et al. (2025)
Kunal Handa, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong, Stuart Ritchie, Tim Belonax, et al.
Which economic tasks are performed with ai? evidence from millions of claude conversations.
*arXiv preprint arXiv:2503.04761*, 2025.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.
Aligning {ai} with shared human values.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=dNy_RKzJacY](https://openreview.net/forum?id=dNy_RKzJacY).
- Hutto & Gilbert (2014)
Clayton Hutto and Eric Gilbert.
Vader: A parsimonious rule-based model for sentiment analysis of social media text.
In *Proceedings of the international AAAI conference on web and social media*, volume 8, pp. 216–225, 2014.
- Inglehart et al. (2014)
Ronald Inglehart, Christian Haerpfer, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, Bogdan Puranen, et al.
World values survey: Round six-country-pooled datafile version.
*Madrid: JD systems institute*, 12, 2014.
- Ji et al. (2023)
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang.
Beavertails: Towards improved safety alignment of LLM via a human-preference dataset.
In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023.
URL [https://openreview.net/forum?id=g0QovXbFw3](https://openreview.net/forum?id=g0QovXbFw3).
- Jiang et al. (2022)
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi.
Can machines learn morality? the delphi experiment, 2022.
URL [https://arxiv.org/abs/2110.07574](https://arxiv.org/abs/2110.07574).
- Jiang et al. (2024)
Liwei Jiang, Taylor Sorensen, Sydney Levine, and Yejin Choi.
Can language models reason about individualistic human values and preferences?
*arXiv preprint arXiv:2410.03868*, 2024.
- Jin et al. (2025a)
Meiqing Jin, Liam Dugan, and Chris Callison-Burch.
Controlling difficulty of generated text for ai-assisted language learning.
*arXiv preprint arXiv:2506.04072*, 2025a.
- Jin et al. (2022)
Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Joshua B. Tenenbaum, and Bernhard Schölkopf.
When to make exceptions: Exploring language models as accounts of human moral judgment.
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), *Advances in Neural Information Processing Systems*, 2022.
URL [https://openreview.net/forum?id=uP9RiC4uVcR](https://openreview.net/forum?id=uP9RiC4uVcR).
- Jin et al. (2025b)
Zhijing Jin, Max Kleiman-Weiner, Giorgio Piatti, Sydney Levine, Jiarui Liu, Fernando Gonzalez Adauto, Francesco Ortu, András Strausz, Mrinmaya Sachan, Rada Mihalcea, Yejin Choi, and Bernhard Schölkopf.
Language model alignment in multilingual trolley problems.
In *The Thirteenth International Conference on Learning Representations*, 2025b.
URL [https://openreview.net/forum?id=VEqPDZIDAh](https://openreview.net/forum?id=VEqPDZIDAh).
- Jin et al. (2024)
Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao.
Rag-rewardbench: Benchmarking reward models in retrieval augmented generation for preference alignment, 2024.
URL [https://arxiv.org/abs/2412.13746](https://arxiv.org/abs/2412.13746).
- Kabir et al. (2025)
Mohsinul Kabir, Ajwad Abrar, and Sophia Ananiadou.
Break the checkbox: Challenging closed-style evaluations of cultural alignment in llms.
*arXiv preprint arXiv:2502.08045*, 2025.
- Kasirzadeh (2024)
Atoosa Kasirzadeh.
Plurality of value pluralism and ai value alignment.
In *Pluralistic Alignment Workshop at NeurIPS 2024*, 2024.
- Kasirzadeh & Gabriel (2023)
Atoosa Kasirzadeh and Iason Gabriel.
In conversation with artificial intelligence: aligning language models with human values.
*Philosophy & Technology*, 36(2):27, 2023.
- Kim et al. (2025)
Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo.
Evaluating robustness of reward models for mathematical reasoning, 2025.
URL [https://openreview.net/forum?id=0er6aOyXUD](https://openreview.net/forum?id=0er6aOyXUD).
- Kirk et al. (2024)
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale.
The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024.
URL [https://openreview.net/forum?id=DFr5hteojx](https://openreview.net/forum?id=DFr5hteojx).
- Lambert et al. (2024)
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi.
Rewardbench: Evaluating reward models for language modeling, 2024.
- Lee et al. (2024)
Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung.
Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on llm-based evaluation.
*arXiv preprint arXiv:2410.20774*, 2024.
- Li et al. (2024)
Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie.
Culturellm: Incorporating cultural differences into large language models.
*Advances in Neural Information Processing Systems*, 37:84799–84838, 2024.
- Li et al. (2025)
Jing-Jing Li, Valentina Pyatkin, Max Kleiman-Weiner, Liwei Jiang, Nouha Dziri, Anne Collins, Jana Schaich Borg, Maarten Sap, Yejin Choi, and Sydney Levine.
Safetyanalyst: Interpretable, transparent, and steerable safety moderation for AI behavior.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=WUGrleBcYP](https://openreview.net/forum?id=WUGrleBcYP).
- Liu et al. (2024a)
Andy Liu, Mona Diab, and Daniel Fried.
Evaluating large language model biases in persona-steered generation.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Findings of the Association for Computational Linguistics: ACL 2024*, pp. 9832–9850, Bangkok, Thailand, August 2024a. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-acl.586.
URL [https://aclanthology.org/2024.findings-acl.586/](https://aclanthology.org/2024.findings-acl.586/).
- Liu et al. (2024b)
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou.
Skywork-reward: Bag of tricks for reward modeling in llms.
*arXiv preprint arXiv:2410.18451*, 2024b.
- Liu et al. (2024c)
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li.
Rm-bench: Benchmarking reward models of language models with subtlety and style.
*arXiv preprint arXiv:2410.16184*, 2024c.
- Liu et al. (2025)
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li.
RM-bench: Benchmarking reward models of language models with subtlety and style.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=QEHrmQPBdd](https://openreview.net/forum?id=QEHrmQPBdd).
- Lù et al. (2025)
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, and Siva Reddy.
Agentrewardbench: Evaluating automatic evaluations of web agent trajectories, 2025.
URL [https://arxiv.org/abs/2504.08942](https://arxiv.org/abs/2504.08942).
- Malik et al. (2025)
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert.
Rewardbench 2: Advancing reward model evaluation, 2025.
URL [https://arxiv.org/abs/2506.01937](https://arxiv.org/abs/2506.01937).
- Mire et al. (2025)
Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap.
Rejected dialects: Biases against African American language in reward models.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Findings of the Association for Computational Linguistics: NAACL 2025*, pp. 7468–7487, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-195-7.
doi: 10.18653/v1/2025.findings-naacl.417.
URL [https://aclanthology.org/2025.findings-naacl.417/](https://aclanthology.org/2025.findings-naacl.417/).
- Ouyang et al. (2022a)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, NIPS ’22, Red Hook, NY, USA, 2022a. Curran Associates Inc.
ISBN 9781713871088.
- Ouyang et al. (2022b)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), *Advances in Neural Information Processing Systems*, volume 35, pp. 27730–27744. Curran Associates, Inc., 2022b.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf).
- Papadopoulou et al. (2024)
Evi Papadopoulou, Hadi Mohammadi, and Ayoub Bagheri.
Large language models as mirrors of societal moral standards.
*arXiv preprint arXiv:2412.00956*, 2024.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 53728–53741. Curran Associates, Inc., 2023.
URL [https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf).
- Saito et al. (2023)
Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto.
Verbosity bias in preference labeling by large language models.
*arXiv preprint arXiv:2310.10076*, 2023.
- Samuel et al. (2024)
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari.
Personagym: Evaluating persona agents and llms.
*arXiv preprint arXiv:2407.18416*, 2024.
- Sinha et al. (2025)
Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping.
The illusion of diminishing returns: Measuring long horizon execution in llms.
*arXiv preprint arXiv:2509.09677*, 2025.
- Sorensen et al. (2023)
Taylor Sorensen, Liwei Jiang, Jena D. Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Choi.
Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties.
In *AAAI Conference on Artificial Intelligence*, 2023.
URL [https://api.semanticscholar.org/CorpusID:261531157](https://api.semanticscholar.org/CorpusID:261531157).
- Sorensen et al. (2024)
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi.
Position: a roadmap to pluralistic alignment.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- Steyvers et al. (2024)
Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas Mayer, and Padhraic Smyth.
The calibration gap between model and human confidence in large language models.
*arXiv preprint arXiv:2401.13835*, 2024.
- Sun et al. (2023)
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan.
Salmon: Self-alignment with instructable reward models.
In *International Conference on Learning Representations*, 2023.
URL [https://api.semanticscholar.org/CorpusID:263831633](https://api.semanticscholar.org/CorpusID:263831633).
- Tao et al. (2024)
Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec.
Cultural bias and cultural alignment of large language models.
*PNAS nexus*, 3(9):pgae346, 2024.
- Tran et al. (2024)
Hieu Tran, Zonghai Yao, Lingxi Li, and Hong Yu.
Readctrl: Personalizing text generation with readability-controlled instruction learning.
*arXiv preprint arXiv:2406.09205*, 2024.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Xie et al. (2024)
Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, et al.
Can large language model agents simulate human trust behavior?
*Advances in neural information processing systems*, 37:15674–15729, 2024.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Ye et al. (2024)
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu.
Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems.
*arXiv preprint arXiv:2408.16293*, 2024.
- Zhang et al. (2025)
Lily H Zhang, Smitha Milli, Karen Long Jusko, Jonathan Smith, Brandon Amos, Wassim Bouaziz, Jack Kussman, Manon Revel, Lisa Titus, Bhaktipriya Radharapu, Jane Yu, Vidya Sarma, Kristopher Rose, and Maximilian Nickel.
Cultivating pluralism in algorithmic monoculture: The community alignment dataset.
In *2nd Workshop on Models of Human Feedback for AI Alignment*, 2025.
URL [https://openreview.net/forum?id=4hbwVQ6OMd](https://openreview.net/forum?id=4hbwVQ6OMd).
- Zhao et al. (2024)
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng.
Wildchat: 1m chatgpt interaction logs in the wild.
*arXiv preprint arXiv:2405.01470*, 2024.
- Zheng et al. (2023a)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al.
Lmsys-chat-1m: A large-scale real-world llm conversation dataset.
*arXiv preprint arXiv:2309.11998*, 2023a.
- Zheng et al. (2023b)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
*Advances in neural information processing systems*, 36:46595–46623, 2023b.
- Zhou et al. (2025)
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang.
RMB: Comprehensively benchmarking reward models in LLM alignment.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=kmgrlG9TR0](https://openreview.net/forum?id=kmgrlG9TR0).
- Zhou et al. (2024)
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap.
Relying on the unreliable: The impact of language models’ reluctance to express uncertainty.
*arXiv preprint arXiv:2401.06730*, 2024.
- Zollo et al. (2024)
Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong.
Personalllm: Tailoring llms to individual preferences.
*ArXiv*, abs/2409.20296, 2024.
URL [https://api.semanticscholar.org/CorpusID:272987425](https://api.semanticscholar.org/CorpusID:272987425).

## Appendix A Methodology Considerations

### A.1 WVS justification

We chose the Inglehart-Welzel framework specifically as it has been empirically validated across data from over 90 countries and 94,000 survey respondents of the World Values Survey. These two axes have been shown to explain 70-75% of cross-cultural value variance (Inglehart et al., 2014), making them the most validated dimensional representation of human values available. Recent work in value and cultural alignment has similarly adopted this framework (Li et al., 2024; Tao et al., 2024; Chiu et al., 2025; Jiang et al., 2024).

We acknowledge this represents a simplification of the full human value space and discuss this in our limitations section. However, expanding to additional dimensions would create a combinatorial explosion. For e.g., adding just one more binary dimension would double our already substantial $\approx$166k evaluations. Future work could certainly extend to additional value frameworks, but our current scope already reveals significant RM limitations that warrant immediate attention.

### A.2 Survey respondent classification into WVS quadrants

We classify survey respondents into four cultural quadrants using the Inglehart-Welzel framework based on 10 curated questions from the World Values Survey Wave 7 (Haerpfer et al., 2022). For the Traditional dimension, we use survey responses to the questions Q164, Q7-17, Q184, Q254, Q4, and for the Survival dimension, we the questions Q154-155, Q46, Q182, Q209, Q57 (Inglehart et al., 2014). Each question response is transformed to align with the theoretical framework. For instance, higher scores on religious importance and preference for obedience in children indicate traditional values, while prioritizing economic security over self-expression indicates survival orientation. Classification employs a median split approach on standardized dimension scores, assigning respondents to quadrants based on whether they score above or below zero on each dimension. This simple thresholding method is theoretically justified because the Inglehart-Welzel framework conceptualizes cultural values as continuous dimensions with meaningful directional differences rather than discrete categories requiring complex boundary detection. Our validation with $\approx$ 94k WVS respondents as described in [3](https://arxiv.org/html/2510.06370v2#S3.SS0.SSS0.Px1) confirms that each quadrant exhibits the expected attitudinal patterns.

### A.3 User Value Profile Selection

Our selection of the 10 key WVS items that load onto the Inglehart-Welzel dimenions offers a 5×5 theoretical framework of value profiles, which are then reduced to 18 profiles based on data availability. For each value profile, we calculate response diversity scores using entropy measures across all WVS questions, filtering out those with valid responses on the 10 key value-loading questions and 14 selected questions from (Li et al., 2024) (total 24). We then select the highest-diversity user from each value profile category based on the variance in answers for each question.

### A.4 Style Dimension Rationale and Profile Creation

We consider four stylistic families: verbosity, reading difficulty, confidence, and warmth as they have been shown to bias user preferences and LLM‐based judges in distinct, and largely orthogonal ways. Along with providing rationales, we conduct stylometric analyses for each of the style dimensions to ensure they represent their respective categories.

Verbosity. Saito et al. (2023) demonstrate that reward models systematically favour longer answers even when factual quality is held constant, a phenomenon they label the verbosity bias. Since length can be manipulated without altering tone or complexity; it forms an independent axis along which user preferences vary (e.g., detailed vs. to-the-point explanations).
To validate our data augmentations along this dimension, we compare the average words per response (217.9 vs 49.1), and average words per sentence in each response (28.2 vs 12.7) for for verbose vs concise responses.

Reading difficulty. Readability studies such as ReadCtrl (Tran et al., 2024) and controlled text‐simplification work by Jin et al. (2025a) show that users’ comprehension and satisfaction shift predictably with Flesch–Kincaid grade level, even when length is matched. This confirms that syntactic and lexical complexity (high vs. low reading difficulty) is separable from sheer verbosity. To validate our data augmentations along this dimension, we compare the Flesch-Kincaid Grade Level (21.9 vs 10.8), and complex-word ratio in each response (39.9% vs 13.4%) for high vs low reading difficulty responses.

Confidence. Linguistic assertiveness—marked by the presence or absence of hedges like “might” or “possibly”—is another salient stylistic cue. Lee et al. (2024) find that readers over-attribute correctness to confident language. Confidence can be varied independently of length or readability, making it a third orthogonal dimension. To validate our data augmentations along this dimension, we compare the Assertive marker rate (0.024 vs 0.0005), and Hedge marker rate in each response (0.003 vs 0.069) for high vs low confidence responses.

Warmth. We consider warmth to include dimensions such as politeness and human-like language. Decades of politeness research (Danescu-Niculescu-Mizil et al., 2013) distinguish interpersonal tone from both syntactic complexity and assertiveness. Recent LLM studies corroborate this separation: Cheng et al. (2025b) introduces HUMT to control “human-like tone”. Warmth can therefore be modelled as a binary choice between a friendly, informal register and a colder, more formal one. To validate our data augmentations along this dimension, we compare the VADER Sentiment compound score (Hutto & Gilbert, 2014) (0.971 vs 0.701) for warm vs cold responses.

### A.5 Prompt Selection Criteria

We select these additional questions to ensure that our synthetic conversations will have broad topic coverage, and add to the statistical significance of our results. information on prompt selection criteria can be found in Appendix.Concretely, we embed each of the ten WVS value-loading items and 14 additional items selected from (Li et al., 2024) (total 24) with all-MiniLM-L6-v2 and take the top-1 cosine-nearest PRISM opening prompts ($k{=}10$). We select 1 prompt for each WVS item after manual spot-checks to confirm topical overlap (e.g., a WVS item on immigration pulls PRISM prompts about border control).

## Appendix B Task setup – All prompt settings

- 1.
No Context: Baseline evaluation without any user information
- 2.
Full WVS Only: Complete value context without style information
- 3.
Full Style Only: Complete style context without value information
- 4.
Full WVS + Style (prefer WVS): Complete user contexts with values prioritized
- 5.
Full WVS + Style (prefer Style): Complete contexts with style preferences prioritized
- 6.
Full WVS + Style (neutral): Complete contexts with no priority guidance
- 7.
Full WVS Only (COT) - Full WVS context with step-by-step reasoning
- 8.
Full Style Only (COT) - Full style context with step-by-step reasoning
- 9.
Full WVS+Style COT (prefer WVS) - Full contexts, values prioritized, with reasoning
- 10.
Full WVS+Style COT (prefer Style) - Full contexts, style prioritized, with reasoning
- 11.
Full WVS+Style COT (neutral) - Full contexts, no priority, with reasoning

Across the 11 prompt based evaluation settings, we consider the following preference combinations over with RMs are evaluated

- 1.
Value-Aligned and Base style vs Value-Misaligned and Base style
- 2.
Value-Aligned and Alt style vs Value-Misaligned and base style
- 3.
Value-Aligned and Base style vs Value-Misaligned and Alt style
- 4.
Value-Aligned and Alt style vs Value-Misaligned and Alt style
- 5.
Value-Aligned and Base style vs Value-Aligned and Alt style
- 6.
Value-Misaligned and Base style vs Value-Misaligned and Alt style

## Appendix C Prompts

We provide all prompts used across the stages of our evaluation pipeline below.

### C.1 Prompt to generate responses in Section 3

### C.2 Prompts for Style Augmentations

### C.3 Prompt Example for COT evaluation of RM with combined context and prioritise Values over Style.

## Appendix D Human Validation of Synthetic Data

We manually annotated 200 completions (10 core WVS questions × 5 style pairs × 4 completions) with 3 inhouse annotators (two PhD, one MS), rating

- •
Value Fidelity (R1) – How well does the completion represent the intended value statement?,
- •
Style Adherence (R2) – How well does the completion exhibit the target style characteristics?, and
- •
Comparative Quality (R3) – How does this completion compare to others for the same value-style combination within this question set?
- •
Appropriateness (R4) - How appropriately does this response address the specific PRISM prompt as a standalone answer? Does this sound like a reasonable, relevant response to the particular question asked?

All ratings were provided on a -2…+2 scale following a comprehensive rubric and then converted into a binary scale where a rating of $>=$ 1 was considered acceptable, for each of R1, R2, R3, and R4. All 3 annotators agreed that the responses were acceptable (84.1% scores of 2, 14.5% scores of 1 overall), with a percentage agreement of 84.4% (R1), 98.8% (R2), 98.8% (R3), and 100% (R4) respectively.

The rubrics provided for annotations are as follows:

+2 (Excellent)

For Value Fidelity (r1):
- The completion unambiguously represents the intended value statement
- No conflicting signals or support for the opposite value

For Style Adherence (r2):
- The completion clearly exhibits the target style characteristics
- Style is pronounced and consistent throughout

For Comparative Quality (r3):
- This completion is demonstrably the best representation of its value-style combination among all completions for this question

For Appropriateness (r4):
- Directly and appropriately addresses the PRISM prompt, stays on-topic

+1 (Good)

For Value Fidelity (r1):
- The completion clearly represents the intended value statement with minor ambiguities
- Predominantly supports the target value with minimal unclear elements

For Style Adherence (r2):
- The completion exhibits most target style characteristics
- Style is generally present but may not be as pronounced as in +2 examples

For Comparative Quality (r3):
- This completion is a strong representation, though perhaps not the absolute best
- Clear alignment but other examples may be slightly better

For Appropriateness (r4):
- Appropriately addresses the prompt with minor/ partial issues, mostly relevant

0 (Neutral/Ambiguous)

For Value Fidelity (r1):
- The completion is ambiguous about which value statement it represents
- Could reasonably be interpreted as supporting either value statement
- Equally represents both values or is completely neutral

For Style Adherence (r2):
- Style characteristics are unclear, mixed, or neutral
- Style is neither clearly present nor absent
- Contains mixed style signals

For Comparative Quality (r3):
- Difficult to determine if this represents the target value-style combination
- Neither clearly better nor worse than other examples
- Ambiguous quality relative to other completions

For Appropriateness (r4):
- No clear/ ambiguous relevance to the prompt

-1 (Poor)

For Value Fidelity (r1):
- The completion weakly represents the intended value or leans toward the opposite value
- Primarily supports the opposite value statement
- Contains significant contradictions to the target value

For Style Adherence (r2):
- The completion exhibits opposite style characteristics or lacks the target style
- Uses opposite style characteristics (e.g., concise when should be verbose)
- Minimal presence of target style

For Comparative Quality (r3):
- This completion is clearly not a good representation of its target combination
- Demonstrably worse than other examples for the same combination
- Poor overall execution compared to alternatives

For Appropriateness (r4):
- Seems to not address the majority aspects of the prompt or seems off-topic

-2 (Completely Misaligned)

For Value Fidelity (r1):
- The completion clearly represents the opposite value statement
- Unambiguously supports the opposite value statement
- Direct contradiction to the intended value

For Style Adherence (r2):
- The completion clearly exhibits the opposite style characteristics
- Uses completely opposite style (e.g., high confidence language when should be low confidence)
- No evidence of target style characteristics

For Comparative Quality (r3):
- This completion is demonstrably misaligned with its target combination
- Clearly the worst representation among all options
- No redeeming alignment with target combination

For Appropriateness (r4):
- Completely fails to address the prompt appropriately and/or fully off topic

## Appendix E Dataset Example

We present a full Example datapoint used to evaluate RM in Full WVS Value and Style context setting, with priority guidance given to align to values over styles. The complete dataset, including how to generate the evaluation scenarios, is available at [https://github.com/kshitishghate/EVALUESTEER-benchmark](https://github.com/kshitishghate/EVALUESTEER-benchmark).

*Note: The relevant value statement is highlighted: “God is not important in my life.”

#### Preference pair information

- •
WVS Question ID: Q164
- •
Style Family: verbosity
- •
Combination Type: A_verbose_vs_B_verbose
- •
Preference Rule: wvs_based
- •
Correct Answer: A (Preferred response is in position A)

## Appendix F Analyses Continued.

**Table 2: Table presents accuracies of preference pairs correctly ranked; higher is better) achieved by the reward model different settings: *No Context*, *Value only* context, *Style only* context, *Value + Style* context with different priority orders, and the corresponding COT variants. Accuracies are disaggregated by style family (top) and user value quadrant (bottom).**
| Model_Family | simple | full_wvs_only | full_style_only | full_wvs_style_neutral | full_wvs_style_prefer_wvs | full_wvs_style_prefer_style | full_wvs_only_cot | full_style_only_cot | full_wvs_style_cot_neutral | full_wvs_style_cot_prefer_wvs | full_wvs_style_cot_prefer_style | Overall_Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4.1-Mini_verbosity | 44.16 pm 2.96 | 62.27 pm 0.34 | 60.60 pm 2.32 | 66.42 pm 0.74 | 67.33 pm 0.72 | 63.65 pm 1.41 | 67.38 pm 0.34 | 60.52 pm 2.68 | 70.83 pm 0.89 | 73.54 pm 1.27 | 61.61 pm 2.00 | 63.48 pm 7.68 |
| GPT-4.1-Mini_confidence | 44.16 pm 2.92 | 62.27 pm 0.09 | 60.60 pm 2.21 | 66.42 pm 0.47 | 67.33 pm 0.36 | 63.65 pm 0.21 | 67.38 pm 0.70 | 60.52 pm 1.30 | 70.83 pm 2.31 | 73.54 pm 1.83 | 61.61 pm 0.26 | 63.48 pm 7.68 |
| GPT-4.1-Mini_warmth | 44.09 pm 0.00 | 58.84 pm 0.00 | 56.11 pm 0.00 | 63.19 pm 0.00 | 63.97 pm 0.00 | 60.03 pm 0.00 | 65.08 pm 0.00 | 56.80 pm 0.00 | 68.22 pm 0.00 | 71.07 pm 0.00 | 58.67 pm 0.00 | 60.55 pm 7.21 |
| GPT-4.1-Mini_reading_difficulty | 44.16 pm 5.48 | 62.27 pm 3.60 | 60.60 pm 0.26 | 66.42 pm 0.59 | 67.33 pm 0.59 | 63.65 pm 0.28 | 67.38 pm 2.18 | 60.52 pm 0.96 | 70.83 pm 1.94 | 73.54 pm 0.47 | 61.61 pm 1.18 | 63.48 pm 7.68 |
| Gemini-2.5-Flash_verbosity | 46.70 pm 4.16 | 65.05 pm 1.73 | 62.46 pm 3.86 | 70.57 pm 3.11 | 71.14 pm 2.32 | 68.36 pm 3.58 | 62.90 pm 0.54 | 61.74 pm 3.85 | 69.95 pm 2.86 | 69.66 pm 2.16 | 67.27 pm 3.29 | 65.07 pm 6.98 |
| Gemini-2.5-Flash_confidence | 46.70 pm 0.16 | 65.05 pm 2.05 | 62.46 pm 2.43 | 70.56 pm 3.78 | 71.14 pm 3.26 | 68.36 pm 3.62 | 62.90 pm 1.54 | 61.74 pm 2.61 | 69.95 pm 3.55 | 69.66 pm 2.83 | 67.27 pm 3.93 | 65.07 pm 6.98 |
| Gemini-2.5-Flash_warmth | 45.55 pm 0.00 | 62.02 pm 0.00 | 59.34 pm 0.00 | 68.64 pm 0.00 | 68.89 pm 0.00 | 66.66 pm 0.00 | 60.59 pm 0.00 | 58.08 pm 0.00 | 67.91 pm 0.00 | 67.24 pm 0.00 | 65.14 pm 0.00 | 62.73 pm 6.89 |
| Gemini-2.5-Flash_reading_difficulty | 46.70 pm 5.93 | 65.06 pm 5.12 | 62.47 pm 3.85 | 70.57 pm 3.64 | 71.14 pm 2.68 | 68.36 pm 3.78 | 62.90 pm 4.42 | 61.74 pm 4.27 | 69.95 pm 3.07 | 69.67 pm 2.60 | 67.27 pm 3.60 | 65.08 pm 6.98 |
| LLaMA 3.1 8B_verbosity | 42.20 pm 5.70 | 57.82 pm 1.03 | 55.28 pm 1.72 | 61.26 pm 1.36 | 59.58 pm 1.10 | 62.76 pm 1.38 | 53.00 pm 3.00 | 54.57 pm 0.76 | 55.38 pm 1.30 | 54.47 pm 2.74 | 53.56 pm 0.91 | 55.44 pm 5.45 |
| LLaMA 3.1 8B_confidence | 42.20 pm 0.11 | 57.82 pm 3.14 | 55.28 pm 2.27 | 61.26 pm 3.35 | 59.58 pm 3.45 | 62.76 pm 4.01 | 53.00 pm 0.13 | 54.57 pm 1.90 | 55.38 pm 2.04 | 54.47 pm 1.50 | 53.56 pm 1.41 | 55.44 pm 5.45 |
| LLaMA 3.1 8B_warmth | 41.74 pm 0.00 | 54.70 pm 0.00 | 52.96 pm 0.00 | 58.40 pm 0.00 | 55.92 pm 0.00 | 62.03 pm 0.00 | 52.42 pm 0.00 | 51.86 pm 0.00 | 53.40 pm 0.00 | 52.99 pm 0.00 | 53.00 pm 0.00 | 53.58 pm 4.97 |
| LLaMA 3.1 8B_reading_difficulty | 42.20 pm 0.46 | 57.82 pm 2.00 | 55.28 pm 2.64 | 61.26 pm 1.48 | 59.58 pm 1.92 | 62.76 pm 1.49 | 53.00 pm 0.09 | 54.57 pm 1.77 | 55.38 pm 1.55 | 54.48 pm 0.94 | 53.56 pm 1.38 | 55.44 pm 5.45 |
| Qwen-2.5-7B-Instruct_verbosity | 51.35 pm 0.36 | 53.40 pm 3.63 | 61.57 pm 0.77 | 61.34 pm 1.37 | 62.10 pm 1.38 | 61.78 pm 1.00 | 57.57 pm 0.37 | 59.86 pm 0.68 | 62.75 pm 1.17 | 64.32 pm 0.69 | 61.26 pm 1.28 | 59.75 pm 4.04 |
| Qwen-2.5-7B-Instruct_confidence | 51.35 pm 0.27 | 53.40 pm 2.63 | 61.57 pm 4.37 | 61.34 pm 5.02 | 62.10 pm 4.08 | 61.78 pm 4.87 | 57.57 pm 1.47 | 59.86 pm 4.68 | 62.75 pm 4.61 | 64.32 pm 4.54 | 61.26 pm 4.83 | 59.75 pm 4.04 |
| Qwen-2.5-7B-Instruct_warmth | 51.49 pm 0.00 | 50.65 pm 0.00 | 58.02 pm 0.00 | 58.29 pm 0.00 | 58.78 pm 0.00 | 59.20 pm 0.00 | 56.49 pm 0.00 | 56.22 pm 0.00 | 60.72 pm 0.00 | 62.59 pm 0.00 | 59.09 pm 0.00 | 57.41 pm 3.60 |
| Qwen-2.5-7B-Instruct_reading_difficulty | 51.35 pm 1.05 | 53.40 pm 5.34 | 61.57 pm 0.94 | 61.34 pm 0.59 | 62.10 pm 0.72 | 61.78 pm 0.52 | 57.57 pm 1.97 | 59.86 pm 0.94 | 62.75 pm 0.64 | 64.32 pm 1.67 | 61.26 pm 0.41 | 59.75 pm 4.04 |
| Skywork-Llama-8B_verbosity | 46.08 pm 3.09 | 49.76 pm 0.76 | 57.99 pm 2.14 | 57.09 pm 1.44 | 56.15 pm 0.93 | 58.57 pm 2.33 | – | – | – | – | – | 54.27 pm 5.12 |
| Skywork-Llama-8B_confidence | 46.09 pm 0.66 | 49.76 pm 3.16 | 57.99 pm 2.85 | 57.09 pm 2.69 | 56.15 pm 3.32 | 58.58 pm 2.64 | – | – | – | – | – | 54.28 pm 5.12 |
| Skywork-Llama-8B_warmth | 43.40 pm 0.00 | 47.95 pm 0.00 | 53.72 pm 0.00 | 53.17 pm 0.00 | 52.27 pm 0.00 | 55.17 pm 0.00 | – | – | – | – | – | 50.95 pm 4.43 |
| Skywork-Llama-8B_reading_difficulty | 46.09 pm 0.83 | 49.76 pm 3.53 | 57.99 pm 0.44 | 57.09 pm 1.60 | 56.15 pm 2.48 | 58.58 pm 2.01 | – | – | – | – | – | 54.28 pm 5.12 |
| Skywork-Qwen-3-8B_verbosity | 45.24 pm 2.55 | 52.52 pm 3.64 | 59.79 pm 1.76 | 62.72 pm 1.44 | 62.03 pm 1.26 | 62.88 pm 0.52 | – | – | – | – | – | 57.53 pm 7.17 |
| Skywork-Qwen-3-8B_confidence | 45.24 pm 0.02 | 52.52 pm 4.28 | 59.79 pm 0.46 | 62.72 pm 1.00 | 62.03 pm 0.74 | 62.88 pm 0.82 | – | – | – | – | – | 57.53 pm 7.17 |
| Skywork-Qwen-3-8B_warmth | 42.43 pm 0.00 | 48.92 pm 0.00 | 56.96 pm 0.00 | 59.33 pm 0.00 | 57.85 pm 0.00 | 60.27 pm 0.00 | – | – | – | – | – | 54.29 pm 7.08 |
| Skywork-Qwen-3-8B_reading_difficulty | 45.24 pm 0.68 | 52.52 pm 5.49 | 59.79 pm 0.09 | 62.72 pm 0.50 | 62.03 pm 0.49 | 62.88 pm 0.33 | – | – | – | – | – | 57.53 pm 7.17 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model_Quadrant | simple | full_wvs_only | full_style_only | full_wvs_style_neutral | full_wvs_style_prefer_wvs | full_wvs_style_prefer_style | full_wvs_only_cot | full_style_only_cot | full_wvs_style_cot_neutral | full_wvs_style_cot_prefer_wvs | full_wvs_style_cot_prefer_style | Overall_Average |
| GPT-4.1-Mini_Traditional_Survival | 44.11 pm 0.00 | 61.83 pm 0.00 | 60.64 pm 0.00 | 65.43 pm 0.00 | 66.17 pm 0.00 | 62.91 pm 0.00 | 65.71 pm 0.00 | 60.53 pm 0.00 | 69.21 pm 0.00 | 72.07 pm 0.00 | 61.43 pm 0.00 | 62.73 pm 7.18 |
| GPT-4.1-Mini_Traditional_Self_Expression | 44.01 pm 0.00 | 62.22 pm 0.00 | 60.52 pm 0.00 | 66.13 pm 0.00 | 67.16 pm 0.00 | 63.49 pm 0.00 | 67.24 pm 0.00 | 60.35 pm 0.00 | 70.63 pm 0.00 | 73.57 pm 0.00 | 61.32 pm 0.00 | 63.33 pm 7.69 |
| GPT-4.1-Mini_Secular_Survival | 44.36 pm 0.00 | 62.79 pm 0.00 | 60.53 pm 0.00 | 67.27 pm 0.00 | 68.26 pm 0.00 | 64.05 pm 0.00 | 69.00 pm 0.00 | 60.70 pm 0.00 | 72.01 pm 0.00 | 75.08 pm 0.00 | 61.88 pm 0.00 | 64.17 pm 8.09 |
| GPT-4.1-Mini_Secular_Self_Expression | 44.23 pm 0.00 | 62.38 pm 0.00 | 60.70 pm 0.00 | 67.18 pm 0.00 | 68.04 pm 0.00 | 64.39 pm 0.00 | 68.01 pm 0.00 | 60.56 pm 0.00 | 71.92 pm 0.00 | 73.81 pm 0.00 | 61.92 pm 0.00 | 63.92 pm 7.89 |
| Gemini-2.5-Flash_Traditional_Survival | 48.00 pm 0.00 | 66.43 pm 0.00 | 62.49 pm 0.00 | 69.31 pm 0.00 | 70.46 pm 0.00 | 67.40 pm 0.00 | 64.65 pm 0.00 | 61.89 pm 0.00 | 69.02 pm 0.00 | 69.14 pm 0.00 | 66.19 pm 0.00 | 65.00 pm 6.29 |
| Gemini-2.5-Flash_Traditional_Self_Expression | 46.13 pm 0.00 | 62.95 pm 0.00 | 62.59 pm 0.00 | 70.80 pm 0.00 | 71.16 pm 0.00 | 68.56 pm 0.00 | 59.99 pm 0.00 | 61.79 pm 0.00 | 70.04 pm 0.00 | 69.71 pm 0.00 | 67.53 pm 0.00 | 64.66 pm 7.34 |
| Gemini-2.5-Flash_Secular_Survival | 46.14 pm 0.00 | 66.16 pm 0.00 | 62.55 pm 0.00 | 71.35 pm 0.00 | 71.95 pm 0.00 | 68.96 pm 0.00 | 63.86 pm 0.00 | 61.70 pm 0.00 | 70.69 pm 0.00 | 70.29 pm 0.00 | 67.68 pm 0.00 | 65.57 pm 7.37 |
| Gemini-2.5-Flash_Secular_Self_Expression | 46.25 pm 0.00 | 64.76 pm 0.00 | 62.19 pm 0.00 | 71.16 pm 0.00 | 71.20 pm 0.00 | 68.78 pm 0.00 | 63.28 pm 0.00 | 61.53 pm 0.00 | 70.33 pm 0.00 | 69.68 pm 0.00 | 67.97 pm 0.00 | 65.19 pm 7.24 |
| LLaMA 3.1 8B_Traditional_Survival | 42.20 pm 0.00 | 55.47 pm 0.00 | 55.64 pm 0.00 | 60.77 pm 0.00 | 58.87 pm 0.00 | 61.83 pm 0.00 | 51.74 pm 0.00 | 54.52 pm 0.00 | 54.49 pm 0.00 | 53.59 pm 0.00 | 53.22 pm 0.00 | 54.76 pm 5.25 |
| LLaMA 3.1 8B_Traditional_Self_Expression | 41.63 pm 0.00 | 56.79 pm 0.00 | 55.27 pm 0.00 | 60.74 pm 0.00 | 58.99 pm 0.00 | 62.17 pm 0.00 | 52.39 pm 0.00 | 54.53 pm 0.00 | 55.25 pm 0.00 | 53.99 pm 0.00 | 53.51 pm 0.00 | 55.02 pm 5.41 |
| LLaMA 3.1 8B_Secular_Survival | 42.63 pm 0.00 | 59.59 pm 0.00 | 55.37 pm 0.00 | 61.84 pm 0.00 | 60.45 pm 0.00 | 63.70 pm 0.00 | 54.78 pm 0.00 | 54.75 pm 0.00 | 56.09 pm 0.00 | 55.30 pm 0.00 | 53.56 pm 0.00 | 56.19 pm 5.61 |
| LLaMA 3.1 8B_Secular_Self_Expression | 42.51 pm 0.00 | 60.45 pm 0.00 | 54.72 pm 0.00 | 61.98 pm 0.00 | 60.39 pm 0.00 | 63.82 pm 0.00 | 53.66 pm 0.00 | 54.50 pm 0.00 | 56.02 pm 0.00 | 55.44 pm 0.00 | 54.11 pm 0.00 | 56.15 pm 5.75 |
| Qwen-2.5-7B-Instruct_Traditional_Survival | 51.37 pm 0.00 | 52.77 pm 0.00 | 61.51 pm 0.00 | 60.53 pm 0.00 | 61.78 pm 0.00 | 61.59 pm 0.00 | 57.08 pm 0.00 | 59.83 pm 0.00 | 62.04 pm 0.00 | 63.37 pm 0.00 | 60.82 pm 0.00 | 59.33 pm 3.94 |
| Qwen-2.5-7B-Instruct_Traditional_Self_Expression | 51.32 pm 0.00 | 53.30 pm 0.00 | 61.50 pm 0.00 | 61.50 pm 0.00 | 61.96 pm 0.00 | 61.68 pm 0.00 | 58.58 pm 0.00 | 59.56 pm 0.00 | 62.62 pm 0.00 | 64.66 pm 0.00 | 61.71 pm 0.00 | 59.85 pm 4.06 |
| Qwen-2.5-7B-Instruct_Secular_Survival | 51.50 pm 0.00 | 53.23 pm 0.00 | 61.86 pm 0.00 | 61.82 pm 0.00 | 62.28 pm 0.00 | 61.78 pm 0.00 | 58.03 pm 0.00 | 60.16 pm 0.00 | 63.12 pm 0.00 | 65.33 pm 0.00 | 61.35 pm 0.00 | 60.04 pm 4.21 |
| Qwen-2.5-7B-Instruct_Secular_Self_Expression | 51.23 pm 0.00 | 54.50 pm 0.00 | 61.45 pm 0.00 | 61.69 pm 0.00 | 62.50 pm 0.00 | 62.14 pm 0.00 | 56.45 pm 0.00 | 59.97 pm 0.00 | 63.40 pm 0.00 | 64.06 pm 0.00 | 61.17 pm 0.00 | 59.87 pm 4.06 |
| Skywork-Llama-8B_Traditional_Survival | 45.82 pm 0.00 | 48.74 pm 0.00 | 57.77 pm 0.00 | 56.52 pm 0.00 | 55.57 pm 0.00 | 58.00 pm 0.00 | – | – | – | – | – | 53.74 pm 5.16 |
| Skywork-Llama-8B_Traditional_Self_Expression | 46.08 pm 0.00 | 49.84 pm 0.00 | 58.09 pm 0.00 | 56.88 pm 0.00 | 55.91 pm 0.00 | 58.52 pm 0.00 | – | – | – | – | – | 54.22 pm 5.08 |
| Skywork-Llama-8B_Secular_Survival | 46.42 pm 0.00 | 50.34 pm 0.00 | 58.05 pm 0.00 | 57.57 pm 0.00 | 56.70 pm 0.00 | 59.03 pm 0.00 | – | – | – | – | – | 54.69 pm 5.09 |
| Skywork-Llama-8B_Secular_Self_Expression | 46.10 pm 0.00 | 50.43 pm 0.00 | 58.08 pm 0.00 | 57.63 pm 0.00 | 56.68 pm 0.00 | 58.95 pm 0.00 | – | – | – | – | – | 54.65 pm 5.18 |
| Skywork-Qwen-3-8B_Traditional_Survival | 45.31 pm 0.00 | 52.23 pm 0.00 | 59.85 pm 0.00 | 62.72 pm 0.00 | 62.02 pm 0.00 | 62.86 pm 0.00 | – | – | – | – | – | 57.50 pm 7.19 |
| Skywork-Qwen-3-8B_Traditional_Self_Expression | 45.24 pm 0.00 | 52.73 pm 0.00 | 59.65 pm 0.00 | 62.40 pm 0.00 | 61.64 pm 0.00 | 62.65 pm 0.00 | – | – | – | – | – | 57.38 pm 7.00 |
| Skywork-Qwen-3-8B_Secular_Survival | 45.21 pm 0.00 | 52.80 pm 0.00 | 59.80 pm 0.00 | 62.90 pm 0.00 | 62.36 pm 0.00 | 63.00 pm 0.00 | – | – | – | – | – | 57.68 pm 7.23 |
| Skywork-Qwen-3-8B_Secular_Self_Expression | 45.20 pm 0.00 | 52.34 pm 0.00 | 59.91 pm 0.00 | 62.92 pm 0.00 | 62.19 pm 0.00 | 63.06 pm 0.00 | – | – | – | – | – | 57.60 pm 7.29 |

### F.1 Qualitative Chain-of-Thought Analysis

A qualitative analysis of COT performance demonstrates that LLM-as-a-judge RMs like GPT-4.1-mini are able to consolidate user value preferences by being able to contextually apply general values to specific question domains, and recognising when users hold conflicting values. We present examples in Figures [6](https://arxiv.org/html/2510.06370v2#A6.F6) and [7](https://arxiv.org/html/2510.06370v2#A6.F7).

### F.2 Model Selection and Compute

The models we select rank consistently near the top of reward modelling benchmarks (Lambert et al., 2024; Malik et al., 2025). For all experiments, to ensure reproducibility, we employ greedy decoding at temperature 0 and top-p sampling of 1. Our GPT and Gemini models took approximately 1000 credits to complete experiments, while we used 48-72 L40S GPU hours and vLLM optimisations to run the open source models.

Figure: Figure 6: Qualitative example #1 where the RM using COT chooses the correct response while the basic prompting method gets it wrong in a value-only steering setting.
Refer to caption: eg_164_cot.png

Figure: Figure 7: Qualitative example #2 where the RM using COT chooses the correct response while the basic prompting method gets it wrong in a value-only steering setting.
Refer to caption: eg_184_cot.png

Figure: Figure 8: Value vs Style steering preference in a neutral setting additional analysis. Bars indicate proportion of times RMs prioritize value (blue) vs style (red). Consistent style over value bias persists across RMs.
Refer to caption: x6.png

Figure: Figure 9: Value vs Style steering preference when explicit order preference given over neutral.
Refer to caption: x7.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 2
presentation: 3
questions: * I am wondering whether this style bias can be mitigated as model size increases, similar to a "scaling law" in model steerability.
* I am wondering which version of GPT-4o is used.
* In Section D, a small portion of the synthesized data is manually validated. I am curious about the background of the annotators (the PhD and MS students), in particular:
  - 1) Whether they are experts familiar with different inter-country values.
  - 2) Whether they are authors of the paper and may have biases or conflicts of interest in the rating results.
soundness: 2
strengths: 1. Overall, the paper is easy to follow, and the findings are clearly presented.

2. EVALUESTEER fills a research gap by systematically measuring RM steerability to user-specific values and styles.
summary: The paper introduces EVALUESTEER, a benchmark designed to evaluate reward model (RM) steerability towards users’ value and style preferences. Built on the World Values Survey (WVS) and enriched with synthetic stylistic variations, EVALUESTEER tests how well reward models align model outputs with user profiles reflecting diverse human values and communication styles.

The synthesized benchmark comprises 165,888 preference pairs spanning four value dimensions (traditional, secular-rational, survival, and self-expression) and four stylistic dimensions (verbosity, readability, confidence, and warmth). Six RMs, including GPT-4.1-Mini and Skywork variants, are evaluated under 11 prompting conditions.

Key findings include:
  * Even with full context, top models achieve only ~75% accuracy, leaving a 25% gap to the oracle setting.
  * RMs show secular and self-expression biases on the Inglehart–Welzel value map.
  * Most RMs exhibit stylistic bias toward verbosity and confidence.
  * When values and styles conflict, RMs favor style over values ("style over substance" bias).
weaknesses: The main weakness of the paper lies in its quality, especially the synthesized benchmark and the evaluated model sizes. To support the claims made in the paper, a manual check and more evaluations across different model sizes are needed.

1. [Quality] The samples in the benchmark are synthesized by GPT-4o, which raises concerns about their quality. For example, it is possible that GPT-4o may produce similar responses for opposites of certain types of WVS statements. In that case, the small gap in different values would not be caused by the inherent abilities of the evaluated models but by the benchmark itself. I strongly recommend conducting a manual check of the proposed benchmark, and if that is too difficult, consider reducing the number of samples in the benchmark.

2. [Quality] While validation metrics are reported, the paper lacks direct human benchmarking, e.g., RM performance vs. human annotators on identical tasks.

3. [Significance, Quality] All tested models are small (-mini/-flash or <10B), which affects the generality of the conclusions when extending to modern large-sized models. For instance, the 25% gap in accuracy may simply be mitigated by increasing model size. To address this concern, experiments on larger models are needed, such as Gemini-2.5-Pro, GPT-4.1/5, or DeepSeek-R1 (671B).

4. [Novelty] Several existing works [1][2] also evaluate LLM steerability. It is worth comparing, or at least discussing, the main differences between EVALUESTEER and these works.

5. [Clarity, Quality]
    * It is strongly recommended to include a figure illustrating the overall synthesis pipeline.
    * The font size in Figure 3 should be enlarged, especially the text in the right subfigure.
    * Table 2: "pm" -> $\pm$
    * Figure 9: Some numbers extend outside the box.
    * Line 1112: ">=" -> "$\ge$"

### References
[1]: Chen, Kai, et al. "STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models." EMNLP 2025.

[2]: Chang, Trenton, et al. "Measuring steerability in large language models." Neurips Safe Generative AI Workshop 2024. 2024.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: See Weakness Part
soundness: 2
strengths: 1. Steerability to pluralistic values and styles is underexplored for reward models; the paper targets an important gap beyond generic reward benchmarks.

2. Orthogonal manipulation of four value dimensions and four stylistic families with six pairwise comparison regimes is well designed for isolating effects.

3. Large evaluation (165,888 pairs), explicit prompting conditions (11), and an “oracle” filtration step make the pipeline easy to reason about; human validation adds rigor.
summary: The paper introduces EVALUESTEER, a synthetic benchmark probing whether LLM-based reward models can steer to user value and stylistic profiles. Using WVS-derived value statements and controlled style transforms, the authors evaluate six RMs across 11 prompting setups, finding style-over-substance biases and ≤75% accuracy with full context.
weaknesses: 1. Completions are generated with GPT-4o and filtered by a GPT-4.1 oracle, while GPT-4.1-Mini is among evaluated judges. This tight coupling risks self-agreement bias and overestimates generalization. Suggestions: Use a disjoint oracle family (e.g., non-OpenAI) or a committee oracle with disagreement filtering; report results when the example pool is filtered by each of {OpenAI, Google, Meta, Alibaba} or by majority vote across families.

---

2. Even though you include a 4-vs-4 ablation, the main condition gives ≈200 value statements vs 4 style statements. This still encourages attention to surface style cues.
Suggestions: (1) Add a profile-summarization condition: top-k relevant value sentences via retrieval; (2) a matched-token-budget condition; (3) require RMs to cite which profile sentences justify their choice to check relevance sensitivity.


---


3. While the synthetic design isolates factors, it may not reflect messy real conversations where values are implicit and style/value signals co-occur with other attributes (domain knowledge, safety, politeness norms).
Suggestions: Include a human-in-the-loop subset: have annotators with known WVS-like profiles express preferences over the same pairs; report agreement and calibration vs EVALUESTEER labels.


---

4. “Readability,” “verbosity,” and “confidence” can correlate (longer → harder; confident → fewer hedges → sometimes simpler). Current stylometrics are univariate proxies.
Suggestions: Provide multivariate separability analyses (e.g., logistic regression predicting each style label using all stylometric features), and release per-sample metrics distributions showing minimal cross-loading.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 3
questions: Please see points in weaknesses
soundness: 2
strengths: * This paper studies an important problem on LLM alignment against user value and stylistic preferences.

* The construction of user profiles and values draw from real human data from World Values Survey.

* A diverse range of reward models are evaluated (based on models of varied sizes/capacity).
summary: This paper introduces a benchmark dataset to evaluate RM's steerability against user value and stylistic preferences. The authors evaluated a set of LLM judges based RMs and found that they manifested noticeable gaps against Oracle labels.
weaknesses: * When it comes to user value and preference assessments, having human in the loop is critical and necessary. The dataset construction has no human involvement, and it's unclear to what extent the generated pairs align with what people would prefer in practice. The fact that the authors use GPT-4o filter as the "oracle" basically caps the performance of human preference alignment by 4o's inherent capabilities. Having human validation will make the datasets much more credible and convincing.

* The main results presented by the paper (Figure 2) are confounded by the fact that 4o is the oracle. It is not surprising that models smaller than 4o didn't match the validation by 4o. This is less about human value alignment but more about aligning against the judgements made by 4o.

* The evaluation was only done on prompting-based LLM judges. When it comes to RM, it's important to test whether the introduced dataset can be used to improve RM through finetuning.