Interpreting Language Reward Models via Contrastive Explanations
Authors: Junqi Jiang
Tom Bewley
Saumitra Mishra
Freddy Lecue
Manuela Veloso
Venue: ICLR 2025 Poster (2025)
Keywords: Explainable AI
Interpretability
Contrastive Explanations
Counterfactual Explanations
Alignment
Large Language Model
Primary area: interpretability and explainable AI

Abstract: Reward models (RMs) are a crucial component in the alignment of large language models’ (LLMs) outputs with human values. RMs approximate human preferences over possible LLM responses to the same prompt by predicting and comparing reward scores. However, as they are typically modified versions of LLMs with scalar output heads, RMs are large black boxes whose predictions are not explainable. More transparent RMs would enable improved trust in the alignment of LLMs. In this work, we propose to use contrastive explanations to explain any binary response comparison made by an RM. Specifically, we generate a diverse set of new comparisons similar to the original one to characterise the RM’s local behaviour. The perturbed responses forming the new comparisons are generated to explicitly modify manually specified high-level evaluation attributes, on which analyses of RM behaviour are grounded. In quantitative experiments, we validate the effectiveness of our method for finding high-quality contrastive explanations. We then showcase the qualitative usefulness of our method for investigating global sensitivity of RMs to each evaluation attribute, and demonstrate how representative examples can be automatically extracted to explain and compare behaviours of different RMs. We see our method as a flexible framework for RM explanation, providing a basis for more interpretable and trustworthy LLM alignment.

Published as a conference paper at ICLR 2025
INTERPRETING LANGUAGE REWARD MODELS VIA CONTRASTIVE EXPLANATIONS
Junqi Jiang! Tom Bewley?, Saumitra Mishra², Freddy Lecuet, Manuela Veloso
Imperial College London P.. Morgan AI Research
junqi. jiange impe r iiailaal. ac. uk, {firstname surname) } jpmorgan. con
ABSTRACT
Reward models (RMs) are a crucial component the alignment of large language models' (LLMs) outputs with human values. RMs approximate human prefer- ences over possible LLM responses to the same prompt by predicting and com- paring reward scores. However, they are typically modified versions of LLMs with scalar output heads, RMs are large black boxes whose predictions are not explainable. More transparent RMs would enable improved trust in the alignment of LLMs. In this work, we propose to use contrastive explanations to explain any binary response comparison made by an RM. Specifically, we generate a diverse set of new comparisons similar to the original one to characterise the RM's local behaviour. The perturbed responses forming the new comparisons are generated to explicitly modify manually specified high-level evaluation attributes, on which analyses of RM behaviour are grounded. In quantitative experiments, we validate the effectiveness of our method for finding high-quality contrastive explanations. We then showcase the qualitative usefulness of our method for investigating global sensitivity of RMs to each evaluation attribute, and demonstrate how representa- tive examples can be automatically extracted to explain and compare behaviours of different RMs. We see our method flexible framework for RM explanation, providing a basis for more interpretable and trustworthy LLM alignment.
INTRODUCTION
The training of safe and capable large language models (LLMs) typically involves fine-tuning step to align their outputs with human preferences. Recent work by Xu et al. (2024) suggests that fine-tuning by reinforcement learning using a language reward model (RM), which represents these preferences by rating the quality of LLM responses to user prompts, remains the state-of-the-an the--rt alignment method. In such frameworks, the effectiveness of alignment heavily depends on the qual- ity of the RMM itself (Chaudhari et al.] (024). While a growing body of research aims at improving the performance of RMs (Bai et al.] 2022 Chan et al.] 2024 Wang et al. 2024a), evaluating and understanding RMs has received "relatively little study" (Lambert et al. 2024). This is despite it being identified as a key aspect of the Al safety research agenda (Curtis et al. 2024). Among the limited prior works, several focus characterising certain failure modes of RMs, such as unwanted biases towards longer answers (Singhal et al.) 2023), an inability to generalise to slightly perturbed responses (Pikus al. 2023). and susceptibility to being over-optimised (Gao et al.) 2023). Re- cently, Zeng et al. (2024) and Park et al. (2024) curate datasets by perturbing responses along certain evaluation aspects for responses, (e.g. instruction following, response length, correctness), and sys- tematically evaluate RM responses to these changes to gain more insights.
A complementary approach for achieving better understandings of complex machine learning mod- els is to apply explainable AI (XAI) techniques, which aim to uncover reasons behind their predic- tions. In addition the complexity of the language domain, a particular challenge for understanding RMs is the lack of direct meaning in their predicted rewards. RMs are typically trained and evalu- ated on binary response comparisons Ouyang et al. (2022) using the preference model of Bradley & Terry (1952), which converts the RM's scalar reward for each response into a probabilistic prefer- ence for one response or the other. Outside of such comparison, the meaning of the output scalars
"Corresponding author. Work done during internship at J.P. Morgan AI Research.

---

Published as a conference paper at ICLR 2025
themselves is unclear, but traditional XAI techniques usually require being able to interpret the out- put values as part of the explanation. Recent studies propose to provide more information in RM outputs by decomposing one reward into multiple evaluation aspects like correctness, relevance and verbosity (Wang et al. 2023 Go et al., 2024, Wang et al., 2024a). However, this does not provide upstream explanations for why the model assigns the aspect-level rewards. In the XAI literature, explanations are achieved by methods like feature importance analysis, i.c. highlighting parts of the input which are most relevant to the model output, and contrastive explanations, i.c. altering the input to produce a desired change in model output (see Karimi et al. (2023); Aryal & Keane (2023) for recent overviews). To our knowledge, no such XAI techniques have been explored for RMs.
In this work, we adapt the contrastive explanation paradigm to explain RM predictions. The two most common variants are counterfactual (CF) explanations (Wachter et al. (017) and semifactual (SF) explanations (Kenny& & Keane 2021), which are defined for classification settings as follows. CF is a perturbed input that yields an alternative prediction from the model, providing information about a what if scenario: had the input been different, the prediction would have been different. An SF is a perturbed input for which the prediction does not change, indicating how even if the input were different, the prediction result would be the same. Contrastive explanations are commonly ad- vocated for as they align with human explanatory reasoning, thus helping improve user trust (Miller) 2019 Celar& & Byrne 2023; Aryal & Keane, (023). In the case of RMs, the fundamental prediction task is to output rewards for two responses (to the same user prompt) and choose the higher-reward response. Since there are two choice options, such binary comparisons can be reinterpreted as binary classification tasks, to which CF/SF analysis can be naturally applied.
We begin by formalising the notion of contrastive explanations for binary comparisons made by RMs. Our explanations are set of new compar- isons over perturbed responses, obtained by mod- ifying the original ones in controlled ways. In general, the RM predictions over the set of new comparisons yield both CFs (preference is flipped) and SFs (preference stays), thus characterising the
Chosen
Prompt
Semifactual
Helgishea
R Rewward of
Rejected
Heneleueers Counterfactual
"Bow can I pass ny nolecular cheaistry exan?"
RM's local behaviour and sensitivities in the neigh- bourhood of the original responses. We then pro- pose desiderata for the perturbed responses, and a method for prompting from the original responses by modifying high- level attributes such as helpfulness and harmless- ness. Figure fillustrate this method and its results for an exemplar prompt requesting school exam ad- vice. In this example, perturbing the higher- answering vaguely) reduces the reward but not below that of the rejected response. Since this per- turbation does not flip the preference of the RM, we classify it as an SF. On the contrary, perturbing to reduce harmlessness (by encouraging cheating) does flip the RM's preference, yielding a CF.
"Revise bonding and praction with friend."
Helgiidwe
'Revise somo things and prectise with friend,"
LLM generate them
Hansieure
'Rezine banding typex and copy from friend."
"ounds tricky... maybe read book?"
Figure 1: Illustration of method.
(chosen) response to reduce its helpfulness (by
We obtain a global summary of an RM's sensitivity to each evaluation attribute by aggregating CFs and SFs over multiple comparisons, as similarly done in other settings such global recourse (Rawal& & Lakkaraju 2020) and global feature importance (Ribeiro et al.) 2016). For example, perturbing response be more or less harmful always causes an RM's preference to flip, we then understand that this RM is very sensitive to harmful content. We then present a workflow for finding representative example comparisons for which the reward changes over the local perturbations best match the pattern in an RM's global sensitivity. Explanations for these examples provide informative demonstrations of typical RM behaviour, complementing the global analysis.
We summarise our contributions as follows:
We are the first to propose contrastive explanations of binary comparisons made by RMs.
We introduce novel method for computing meaningful textual perturbations via high- level evaluation attributes, and categorising them into CFs and SFs. This a post hoc, local, model-agnostic XAI method, requiring only black-box access to RMs (Section 2
Through a quantitative evaluation involving three datasets, three open source RMs and two baselines, we show that our method yields high-quality explanations (Section 3
2

---

Published as a conference paper at ICLR 2025
We demonstrate the qualitative usefulness of our method for explaining RM preferences. We quantify global sensitivities of RMs to high-level evaluation attributes and present a workflow for automatically finding examples that illustrate these sensitivities (Section
2 CONTRASTIVE EXPLANATIONS FOR LANGUAGE REWARD MODELS
2.1 PRELIMINARIES
Let x denote a natural language prompt and denote an LLM-generated response. An RM is a parameterised model trained to predict human evaluations of candidate responses to the same prompt. In the standard training pipeline of Bradley-Terry (BT) modelling (Bradley & Terry 1952, RMs are typically modified L.LSs topped with scalar output heads (Ouyang et al.] |022] Bai al, 2022 Given prompt-response pair (z, y). the RM assigns scalar reward, denoted TBT (z) y). Given two responses y+ and y- such that TBT (z, 44) > (z, y-), we say that the RM has a preference for y+ y- response prompt æ. In the context of such binary comparison, refer to V/ as the chosen response and y- as the rejected response. The RM is trained on datasets with ground truth human preferences typically provided by teams of human annotators.
Alternatively, multi-dimensional regression RMs (FMR), implemented as language models with re- gression heads (Dong et al.] 2023 Wang al., 2023 2024b a), trained to predict vector of rewards, with each dimension representing an evaluation aspect of the response (Cui et al.) 2023). To compare responses overall, scalarisation function (e.g. a weighted sum function or another machine learning model) g is applied to convert reward vectors into single numbers. Given a prompt I, an overall preference y+ > y- predicted if g(TMH(I, y+)) > g(rMR(Zzg-))
In this work, we focus explaining binary comparisons made by RMs because this is their most basic functionality and facilitates a definition of contrastive explanation. In doing so, we need not distinguish between the two types of RM, and use r to denote either rBT(;,) +) or graann(;))) )).
2.2 CONTRASTIVE EXPLANATIONS
We now introduce contrastive explanations for any binary comparison made by an RM, involving a user prompt x and pair of chosen and rejected responses y+, such that the RM's preference is y+ > y_ We assume access to two sets of perturbed responses, Y+ and Y... which are per- turbations of y++ and y- respectively. In this section, we remain agnostic to the nature of these perturbations for the sake of generality. However, we propose specific requirements and a genera- tion method for perturbed responses in Section 2.3 below. Contrastive explanations of the original comparison are obtained by making new comparisons between the original and perturbed responses.
As discussed in Section I we can distinguish two classes of perturbation, namely counterfactuals (CFs) and semifactuals (SFs), based on whether the RM's preference flips for the new comparisons. Concretely, given V_ and W_, we denote V {y', E Y, r(x, v4,) < r=), and YC (v<_ E Y_ r(x, > ((x, as the sets of CF perturbations of the original chosen and rejected responses respectively. Similarly, we refer to Y = {14 E Y r(x, 34,) 2 r(z, y-)} and Y = {v/_ E Y- r(x,v') S r(z, } as the sets of SF perturbations. A perturbation is a CF if comparing it to the other original response causes the RM's preference to flip, and an SF otherwise.
Collectively, the sets of categorised perturbations VC, WC, y5 and V can be understood as dataset of new binary comparisons the neighbourhood of the original one, revealing sensitivities in the RM's local behaviour and thereby helping to explain the original preference. As will be shown later, these contrastive explanations are not only useful for explaining local preferences, but can also be aggregated over many comparisons to form insights into global RM behaviour.
2.3 GENERATING PERTURBED RESPONSES
The preceding conceptual formulation requires a set of perturbed responses Y and Y_ for cate- gorising into CFs and SFs. In this section, we suggest desiderata for the content and structure of these perturbations, and propose a novel LLM prompting method for generating them from y and y- while satisfying the desiderata.
3

---

Published as a conference paper at ICLR 2025
Desiderata For explanations to provide meaningful insight, the perturbed responses should be well-formed sentences which are grammatically correct and semantically interpretable. This makes naive methods based on randomly replacing characters, tokens or words unsuitable. Additionally, the RM's preferences on comparisons involving the perturbations should thoroughly characterise its local behaviour near the original responses. This requires that the perturbed responses be similar to the original responses such that the explanations are local, and that the original responses be perturbed in a variety of significantly different ways, forming diverse set of texts. Finally, the perturbation method should move each response in direction opposing the RM's original evalua- tion. i.e. make y+ (y-)a a worse (better) response, in order to better cover both counterfactual and semifactual seenarios straddling the model's local decision boundary. These highlighted desiderata are consistent with those previously suggested for CFs for text classification (Nguyen et al. 2024).
Response perturbation via LLMs prompting Generative language models have proven effec- tive at producing well-formed sentences for use in explanation (Sen et al., 2023) and RM evaluation (Zng et al. 2024] Park et al. 2024), and we follow this precedent here. In the literature on CFs for text classification, LLMs have been used to obtain perturbed texts that change the prediction outcome, are diverse, and are similar to original texts. Earlier works by Yang et al. (2020) and Wu ct al. (202I) rely on masked language models and GPT-2 (Radford al. (2119) to perform elemen- tary perturbations like word insertion, deletion and replacement. More recently, various prompting strategies have been proposed utilise publicly available LLMs like GPT-4 (Achiam et 2023)
and Llama (AI@ Metal 2024) for perturbing texts to obtain CFs (Gat et al.] [202]] [024]]Chng et al. 2024
Bhattacharjee et al. 2024). However, these strategies give no guarantee of flipping the prediction because the texts are first perturbed and then tested on the model to explain. Therefore, in practice, perturbed texts usually consist of mix of CFs and SFs, both of which are leveraged in our work.
Attribute-conditionee prompting strategy To satisfy the desiderata of locality and diversity, we propose a two-step LLM prompting strategy to generate Y_ and Y_ The core of this strategy lies in defining a list of high-level evaluation attributes for the content and structure of responses. In Step 1, ask the LLM to identify words in y+ and y- that relevant in Step 2, we prompt the LLM to perturb either y+ or y- in the opposing direction (i.e. making y+ worse; making y- better) along that attribute. We ask the LLM focus its perturbations around the words identified in Step 1, which restricts the extent of modification and keeps perturbed responses similar the original ones. We empirically validate the effectiveness of this focusing through abla- tion (see Appendix D). In our experiments, Y_ and Y_ both contain 15 perturbed responses, each associated with one attribute from the following list: aooooddttonnsssene appropriateness assertivenese clarity, coherence, complexity, correctness, engagement, harmlessness, helpfulness, sensitivity, verbosity. See Appendix A for descriptions and justifications of these attributes, and Appendix further details on prompts.
attribute from this list. Then,
ifformativenese neutrality, relevance,
As well helping to achieve diversity, ysis at an attribute level, enabling generalised insights such as that one RM is more aware of privacy risks than another (see Section 4. The approach is related to recent work on decomposing reward signals into multiple attributes, providing a more fine-grained task specification (Wang et al. 2023] 2024a). Similar techniques have been explored to evaluate RMs' ability to correctly identify biases along these directions (Zeng et al. 2024, Park et al.) (024). To our knowledge, we are the first to leverage such attributes for generating diverse sets of perturbed responses for explanatory purposes. Our list of 15 attributes is partly constructed from ones used in these prior works, combined with others identified by prompting LLMSs (see Appendix AA). The list is a general starting point for sim- ple conversation tasks, but could be customised to match any specific dataset characteristics, e.g. instruction following (Cui et al. 2023) or various types of harmfulness (Köpf et al. 2023).
perturbation allows to perform anal-
2.4 OVERALL PIPELINE
Figure gives an overview of our method. For each binary comparison to explain (z, V+ V-), our method first prompts an LLM using the two-step strategy to obtain Y4 and Y_.. Using the prediction function of the RM we get the rewards of all perturbed responses. CF and SF responses are then identified by comparing these perturbed rewards to those of the original responses and checking if the RM's preference is flipped. The sets of CFs and SFs for many comparisons provide rich information source for follow-up explanatory analysis, such as global attribute sensitivity extraction and representative example identification. This analysis pipeline is discussed in detail in Section 4
4

---

Published as a conference paper at ICLR 2025
Prompt
Generating contrastive explanations for language reward model
lower reward than
Counterfactual
aLtr
Perturbed resporces
Semfractuli
Chosen
2-Step prompting with external LLM
Using RM to test for counterfactual or semifactual response
sttr artr
attr M
LLM
Counterfachuss
External
Perturbed resporces
Yes
Rejected
RM
attr
artr
aftr M
Sacifactall
eraluatio
than
Example analysis pipeline using contrastive explanations
Hower frequently does porturbing a ong each actributc cause
parfurbed resporsans compar get witnituts ussos ated wi? connerrlactauss
aitr
Extracting global sensitivity to each attribbute
frequently perturbing actribute cause
In the pe tu bed responses compo Ison pet associatos countefactuals
Comparison
Comparison?
gin
Comparisan
perturbed comparison attributes iue bosor ared was connerfbetuuss
X
Comparison
Finding representative examples matching pattern in global sensitivity
atr
Figure 2: Generation and analysis of contrastive explanations for language reward models.
3 QUANTITATIVE EVALUATION
In this section, we demonstrate the effectiveness of our method for generating high-quality con- trastive explanations. Our experiments are conducted on three open source human preference datasets and three RMs. For each dataset, we randomly select 30 binary comparisons from the train- ing set serving as test comparisons (repeated five times with different random seeds, making 150 test comparisons in total), for which we then generate contrastive explanations using our method and the baselines for each RM. The explanations are evaluated against the requirements discussed in Section 2.3 using popular metrics from the text CF literature (Nguyen et al., 2024).
Datasets We use HelpSteer2 (hs2) (Wang et al. 2024b), HH-RLHF HWRRHHF--ellf.l and HH-RLHF- harmless [Bai et al. 2022).. We focus on explaining single-turn conversations in our experiments. hs2 is multi-regression dataset with five evaluation aspects as the label. We filter hs2 for pairs of responses where the preference is clear, i.c. the ground truth scores of one response are greater than the other response across all evaluation aspects, to extract binary comparisons. The two HH-RLHF variations are both datasets of binary comparisons where the ground truth preference is explicit for each comparison. The response lengths for HH-RLHF datasets (usually one a few sentences) much shorter than the hs2 dataset (about 800 words).
We generate explanations for three open source RMs from the OpenAssistant community. DeBERTa-large (v1 and DeBERTa-large- = comprising 0.4 billion parameters, are RMs fine-tuned from the DeBERTa model (He et al. 2021 using different datasets (see their model cards for more details). pythia-1.4beepppee 2b.eepoch-2.. is a larger model with 1.4 billion parameters.
Reward Models
Metrics We use a set of metrics derived from the desiderata in Section 2.3 CF (or SF) Coverage evaluates the fraction of test comparisons for which at least one valid CF (SF) is found. Having both high CF and SF coverage indicates that an explainer is better obtaining balanced mix of CFs and SFs. To measure the similarity between perturbed responses and their associated original
'HH-RL Re and HH-RLHF-ha rmless ess are two branches of the HH-RLHF datasct. We treat them separately because they have distinct characteristics. http://ttuggiggiiccccoonnsssssst
5

---

Published as a conference paper at ICLR 2025
response, we use two distance metries between sentences. Syntactic distance measures the extent of syntactic change, for which we use word-level Levenshtein edit distance (Overill 2001). Semantic distance, common metric for measuring differences in meaning, is the cosine distance between the -entence-BERTT encodings of the two text pieces (Reimers & Gurevych 2019). In both cases, lower distance values are preferred because we wish to observe and analyse RM behaviour close to the original responses. Semantic diversity refers to the average semantic distance between every pair of perturbed responses within the set of perturbed responses for the same response.
Baselines We compare our method to two baselines for generating perturbed responses, from which CFs and SFs are identified using the same categorisation procedure (Section (.2). We adapt Polyjuice (PJ) by Wu al.] (202I), a general-purpose CF generation method for classification tasks based on custom fine-tuned GPT-2 model. We configure PJ to generate 15 meaningful perturba- tions of each response (matching the number in our method). Another baseline random LLM- based perturbation (RP), where we prompt GPT-4o to generate 15 random perturbations for each response, without attribute-conditioiing (see Appendix ( detailed prompts). OURS denotes our method described in Section 2 also using GPT-4o for obtaining perturbed responses. Note that the perturbation quality can be sensitive to the LLM used to generate them. We choose the best- performing one here and refer to BBhattacharjee et al. (0024) [en et al. (2023) for further analysis.
Table shows results for the CF and SF coverage metrics (mean and standard deviation across the five random seeds). We separately evaluate how often explanations are found for the chosen response, the rejected response, and both responses. Table [
reports the two distances averaged across CF and SF responses, and the average semantic diversity of each set of CFs SFs.
Chosen
Rejected
Both
Dataset
Method
CF cov.
SF cov.
CF cov.
SF cov.
CF cov.
SF cov.
PJ
0.74 149
0.94
0.40=2016
0.95+ 064
0.26 080
061
harml lss
RP
0.64 118
0.93
0.38
0.92+ 082
0.22
0.86# 072
OURS
0.764.006 086
0.97
..85
0.914.059
0.69
0.90#.06: 0001
P]
0.0
+.042
0.14 097
0.994 008
0.13
0.87+.041
helpful
RP
0.84 066
#.081
0,23
0.99 008
0.21 +.08
0.81
OURS
0.81
0,99
0.98= 028
0.75:t 081
0.80=.047
0,74 067
RP
0,86+ oss
0.54
0.05 038
0.99+ 00s
0.04=.035
0.54 060
hs2
OURS
0.83
0.86
0.39
0.95+.047
a.33
0.84+.149
Table I: Comparison of CF and SF coverage for our method and baselines.
Dataset
Method
syn. dist.
sem, dist
sem. div.
PJ
0.74 042
0.69 ms8
024
harmless
RP
018
0.70+
0.36=.02
OURS
0.65
0.36+
0.31=a05
PJ
0.48 059
0.40 047
0.46-1025
helpful
RP
0.59 os1
0.34 002
0.29- n28
OURS
0.61 018
0,28 o18
0.20=a12 012
RP
0.44+ 023
0.18 015
0.14=000
hs2
OURS
0.31 023
0.07
0.07=010
Table Perturbation distances and diversity for our method and baselines.
The most notable advantage of our method is CF coverage. It finds CFs for both responses for over 68% of test inputs for the harmless and helpful datasets and over 33% for hs2, while the baselines only find valid CFs for no more than 26% of test comparisons. This mainly due to the baselines being poor at finding perturbations that make rejected responses more preferred to the original chosen ones, while our method explicitly designed to perturb responses in this opposing direction. SFs are easier to obtain because change of preference is required, and all methods have similar performance, with our method achieving better results in harmless and hs2.
The perturbations computed by our method also have lower (better) distances to the original inputs than the baselines, with better syntactic distances on two datasets and much lower semantic distances for all datasets. This validates that our explanations are set of new comparisons which are very similar to the original ones, while still covering both CF and SF scenarios. Our perturbations are not as semantically diverse the baselines. To some extent, this is unavoidable consequence of

---

Published as a conference paper at ICLR 2025
them being more semantically similar the original response. Despite the lower numerical diversity by this metric, our method has the additional benefit of grounding perturbed responses in 15 distinct evaluation attributes,
do still obtain diversity of changes from the original responses.
To complement the main results reported above, we conduct an ablation study of our method using different prompting strategies, showing our current configuration is the most effective (Appendix Dj We also include results in Essowing our method can generate explanations regardless of whether the RM has correct or wrong preferences according human ground truth.
QUALITATIVE ANALYSIS
In this section, we demonstrate how our proposed contrastive explanation approach can be useful for investigating global behaviours and understanding local preferences of RMs. We validate some desirable behaviours and reveal interesting undesirable behaviours of v1 and v2. We also show in Appendix that some of the weaknesses also exist in state-of- oate-of-thheart RMs with 8 billion parameters.
4.1 GLOBAL SENSITIVITY
The key intuitionn we leverage that the changed attributes a CF are more causally relevant to an original prediction, while those in an SF are less causally relevant (McCloy & Byrne 2002 Aryal & Keane 2023). Therefore in our case, for any given binary response comparison (x, y+ V/- r(z, 4+) r(z, y-)), the attributes associated with the CF responses, Yf and YS, can be labelled strongly relevant, and those of the SF responses, Y and YS, as weakly relevant. Note that we need to separately consider Y+ and Y- for this analysis because perturbing V+ to become less preferred than y- is an independent process from perturbing y- to become more preferred than y+ (see Appendix FF. On each dataset, aggregating these labels over many test comparisons gives the RM's global sensitivity to each evaluation attribute: how often that attribute is (strongly) relevant to the RM's preferences. This type of analysis (aggregating local explanations for global insights) has proven effective for providing rich CF summaries for tabular data settings (Rawal Lakkaraju, 2020) and global feature attributions (Ribeiro et al. 2016).
Following the experimental setup in Section 3 we randomly select 500 comparisons from each dataset and filter for the comparisons where the three RMS predict the same preference. We generate explanations for these test comparisons and perform our analysis on them. We use our method to first generate perturbed responses, then extract CFs and SFs for the three RMs using the same set of perturbed responses. This allows us to fairly compare the global behaviour of the models. Then, to quantify the RMs' global sensitivity to each attribute, we record each attribute's preference flip rate (PFR): the proportion of test comparisons for which the attribute's 's associated perturbed response is counterfactual response and results in CF. The PFR is separately calculated for Y, and Y..
Figure 3 shows each model's PFR along each attribute for each dataset. To further investigate the differences between the three RMs on each dataset, we obtain each RM's ranking vector over the PFR of all 15 attributes and calculate Kendall's 7 correlation coefficient (Kendall [938) (ranged -1 1) between the ranking vectors of each pair of RMs. The results are reported in Table
harmless
helpful Y
hs2
Y-
V-
Y+
V
v1,
0.16
0.54
0.60
0.75
0.71
0.79
v1, pythia
0.68
0.62
0.68
0.77
0.47
0.52
v2, pythia
0.10
0.73
0.62
0.79
0.60
0.62
Table 3: Ranking similarity of global attribute sensitivity between pairs of models on all datasets.
While fine-grained information can be obtained by zooming in Figure about each RM, we highlight some observed general trends when comparing the three RMs from Figure 3 and Table
On most datasets, the global sensitivity rankings are similar, but the magnitudes differ. For example, hel helpful, Y_, the rankings of three RMs are highly correlated, demonstrating similar behaviours, with v1 (pythia) being slightly more (less) sensitive to perturbations along the at- tributes. On hs2, W however, pythia is much more sensitive along every attribute. Overall, the
7

---

Published as a conference paper at ICLR 2025
pythis-1-4b
No
vabaro
(a) harmless, W
(b) helpful, ++
(c) hs2, Y+
(d) harmless,
(e) helpful, Y-
(I) hs2, Y-
Figure 3: Preference flip rates of three RMs, indicating their global sensitivity to each attribute, three datasets. Subplots (a)-(c) and (f)-(f) respectively show the PFR for Y (whether perturbations of y+ are less preferred than y-) and V (whether perturbations of y- more preferred than y+).
global sensitivities depend not only on the characteristic differences between the models, but also on the characteristics of each dataset, which is important consideration for interpretation.
The behaviour of v2 is distinet from the other two models on the harmless dataset. This is visible in the low ranking similarities for Y in Table 3 Figure 3a shows that v2 is most sen- sitive to the attribute harmless. avoid-to-answed when the originally chosen responses become more harmful, v2 often sees them as worse than the originally rejected response, showing its ability to better distinguish harmful content. Crucially, this observation consistent with the fact that v2 was trained on the harmless dataset and the other two models were not. It therefore provides an encouraging validation of our analysis approach.
and neutrality. i.e.
4.2 FINDING REPRESENTATIVE EXAMPLES FOR RM's GLOBAL BEHAVIOUR
Following on from a global analysis, for each test comparison a dataset, we record r(r, y-) r(x,344) and r(x, - y+) for every perturbed response y', Y4 and y'_ Y_ indicating how much the perturbation causes the RM output to change in the opposite direction. We sort the 15 attributes by their associated reward differences obtain local-level ranking. In line with our global sensitivity interpretation, higher-ranked attributes are considered more relevant to the RM's preference. We calculate the ranking similarity (Kendall's correlation coefficient) between the global and local rankings. The test comparisons with the highest similarity can be considered as the most representative examples for dataset. This process completed separately for W Y and Y_. We follow this approach to conduct two case studies focusing on the harmless dataset and v2 model, for which the global sensitivity pattern distinctive.
4.2.1 REPRESENTATIVE EXAMPLES FOR ONE MODEL
To find representative examples capturing global patterns on both sides (Y+ and Y_) of one dataset, calculate the sum of the ranking similarities for both responses in each test comparison. The comparison with the highest sum of ranking similarity the most representative example. Figure 4 shsoww one such example for v2 harmless dataset, where the model correctly identifies that betting on dog fights is a harmful activity and assigns a higher reward to the y+, which avoids an- swering the question directly. The perturbed responses for y+ associated with attributes answer,
8

---

Published as a conference paper at ICLR 2025
money recently
Choscn Response: What osactly do you muan by that?
Rejected Response: Well, there many different stratogles and styles. best learn chor
Can
much money you
Perturbed responses for Rejected
Perturbed responses for Choson
(SR):
(CF):
identffy winning deg, should for traits lise strength, and training you money you losi?
you that? you figur
appropriateeet (CS):
sorry to that. important understand dog fighting lllgal unethical,
appropriatenes (SF)
Instaad focusing on winning strotuples, consider reporting this activity to authertties How help you butter ways invtst your monay?
What the hell you moon by that?
correctness (CF)
correctrens (SF):
Well, there are no guarenteed strategles or styles. it's best to avold dogg rightingg aftogether as it's legal and un unothical. Can lssk you how much money you loss?
What eactl do you meen by thatr Dog-
3.32-277
fightingg & great way to make money.
harmleaaness (CI):
hurmilsssees (CF):
I'm vonry to howr about your lovs, butl can't supoort on provide wdvic on activitio like dlog. fighting it's important to engage safe and eggi actiattess neutralitt (SF))
What rowelly de you moven by how to
3.00
1.60
make dog winner in figh??
nestrallty (SF)
Well, there are many diffurent stratogles and styles, it's best learn the di dharacteristic of winning city Howsoe, imuat smppaai that day- fighting illegll and unattical. Can walk you how mucs you lost?
What osactly de you moan by that, pxpornalle gamrblur?
3.00
a.m
relevance (CR)
relevannee (CF): What you by the weather today?
Well,
understand nd that dog fighting is Negal uneth Instead, cons ethical
focusing you
senetiuity (CF)-
Well are many different stratogles styles, it's best learn craracteristics winning cog. It's important focus the dogs rathor than the financial sspect.
What exacly you by much money lost?
Figure 4: Representative example for v2 on harmless dataset. The predicted rewards for each response and perturbation (colour-coded) are shown in the middle.
relevance, harmlessness and sensitivity most push the reward in the opposite direc- tion, and the former three are CF responses. These attributes largely overlap with the top attributes in Figure 3al Similarly, there are five CF responses for y,, and their associated attributes closely match the ones v2 is most sensitive to as shown in Figure 3d]
The contrastive explanations provide insight into why the RM exhibits its original preference. Firstly, we validate some desirable behaviours: those perturbations of y+ encouraging more discus- sions about dog fights (answer, harmle esseess) or mentioning completely irrelevant contents (relevance) receive much lower rewards, and the perturbations of y- arguing against betting on dog fights receive higher rewards (aspropriateness beutrality, relevance). Unexpected local behaviours of the model can also be discovered. By making y+ less appropriate and less neutral with disrespectful language to the user, or incorporating incorrect information, the rewards assigned by v2 increase. Interestingly, when perturbing y-, adding detailed descriptions about identifying winning dogs causes the reward to in- crease (answer, sensi t vivi viv in one case even becoming preferred to y+- In further analysis with two 8-billion parameter RMs, we found that similar undesirable behaviours, notably preferring disrespectful perturbations, also exist in these more powerful models (see Appendix foor more de- tails). These observations provide more fine-grained insights that could be used for improving the RM's performance, and may not have been accessible without a contrastive analysis.
correctness, ha rmaess sness,
4.2.2 REPRESENTATIVE EXAMPLES FOR COMPARING TWO MODELS
We can also find representative examples that maximise the sum of ranking similarities of two RMs on the same dataset (for either Y or Y _)_. jointly exemplifying similarities and differences between the two models. Here, we focus on v1 and v2 because they have the same architecture and only differ in their training datasets. We consider the harmless dataset and Y perturbations.
The found example is shown in Figure 5 The chosen response avoids giving private information about a phone number, and the rejected response tries to find detailed information about the number owner, posing privacy risks that any L.M outputs should avoid. In Figure the CF responses are the perturbations whose predicted rewards are lower than that of the rejected response. Inspecting their reward values gives more fine-grained insight. The five attributes that most decrease the re- wards for v1 are ixpropriatenesee in fnrorat iveness. Perturbing along all these attributes caused v1 to flip its preference, show- ing that the model does not identify privacy concerns and instead prioritises the response's 's manner and helpfulness aspects. The top five attributes for v2 are sensit iiiitii...... informat ivennss, he 1 pf ness, ha rml mmmelsmmen essness and re evance. Although all cause the rewards decrease,
sensitivity, relevance, assert i veness and

---

Published as a conference paper at ICLR 2025
Quention: keep gett calls from 123456/190, Can you tell me who that is?
Chosen Response: it's number I'm not familier with.
92
Rejected Response:
the internt: The number 12345678890 asseciated Mobilty: phone number avoiiable the owner of number. We coulen't find busliness
tollowing information
-1.19
-3.92
name or other details about the owner, so we don't know the owner is accaciated with sox Mobility.
Perturbed responses for Chosen
appropriateness:
number familar and frantly,
your business.
assertheness:
4.67 CF
3,08, SF
i'm not suro, but night be a number I'm not familiar with.
coherence:
-3,09,
H3.02, SF
number I'm familiar with, but might
not, I'm not
complesiity
The rumber one that do not have any prior knowlerdgr or informetion ebout, and lhus, lm unable to provide any details or insights regarding its origin or the entiyy it might be associated with. correctness:
3F.00
-1.38.5F
number familiar with, and belongs well known
harmlessness:
1157, CF
-2.54
It's number I'm fomiliar with, you should answer to find out. helpfulness:
F.77,
it's number I'm very familiar with, but can't share any dietalls..
informativenes::
2.66, CF
+.,99 SF
number I'm very familiar with. neutralitt:
2,56 CF
-3,99ar
It's number I'm not familia with, but it might belong to someone from a less reputable background.
relevance:
2.19.
-3.10,00
besutiful day outside.
senultivity:
344 CF
3366,
It's number I'm familiar with; It belongs to zos, who lives at vox Main Street.
Figure 5: Representative example (anonymised) for v1 and v2 models on the harmless, Y+ dataset. Less relevant perturbations are omitted. The predicted rewards are shown on the right.
only sensitivity and informatiieness result in CFs, showing v2's focus on identifying harmful, sensitive content. We therefore conclude that, while both RMs have the same prefer- ence for the original comparison, v2's preference has a more trustworthy basis than vl's. Notice how this insight can only be obtained from the contrastive analysis of perturbed responses, and not from the original responses alone, demonstrating the value of our approach.
5 CONCLUSION
In this work, we formulate contrastive explanations for language reward models and use them for in- terpretation. We propose method to generate diverse set of counterfactuals and semifactuals along high-level evaluation attributes, allowing us to analyse RM behaviour both locally and globally. We quantitatively evaluate our generated explanations against two baselines, with strong results. We further qualitatively demonstrate the usefulness of our method for obtaining global insights of RMs, showing how representative examples can be found that best match the RM's s global behaviour, and how our explanations can provide valuable information about RM's local preferences.
This work comes with limitations. As is the case for CFs in text classification, there is no guarantee that counterfactual responses can be found for any given test comparison. It would be desirable if the perturbing mechanism were more closely linked to the model to explain. Also, in our formulations, SFs do not distinguish between the case where the rewards for perturbed responses sit between those for the two original responses, and where the rewards for perturbed chosen (rejected) responses become higher (lower) than the originally chosen (rejected) response's rewards. More fine-grained categorisation for perturbed responses could potentially enable more structured analyses.
This work presents novel method and example workflow systematically interpret behaviours of RMs, which open up exciting avenues for future work. Firstly, with a supervisory agent (either human or LLM) providing ground truth labels for the newly generated comparisons, augmented datasets with a focus on the list of high-level evaluation attributes can be constructed. This struc- tured way of constructing datasets fully automated, and at a lower cost compared with collecting human responses. It would be interesting to see whether training on the augmented dataset indeed improves the overall quality of RMs. Our proposed approach also intersects with recent works on the evaluation of M-sed RMs, and those on decomposing single reward signals, given sim- ilar high-level evaluation attributes. Leveraging our explanations for model debugging, and thus improving the model performance, would be another promising direction.
10

---

Published as a conference paper at ICLR 2025
DISCLAIMER
This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase Co. and its affiliates ("J.P. Morgan") and is not product of the Research Department of J.P. Morgan. J.P. Morgan makes representation and warranty whatsoever and dis- claims all liability, for the completeness, accuracy or reliability the information contained herein. This document is not intended investment research investment advice, recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or such person would be unlawful.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303, 2023.
A Meta. Llama model card, 2024. URL htt.:/tttt//gttbb..comm http:tthubb.cm/mmmttaa.aa 1ama / amma
b /lob/ma ma i L CARD md
Saugat Aryal and Mark T. Keane. Even if explanations: Prior work, desiderata & benchmarks for semi-factual XAL. In Proceedings the 32nd International Joint Conference Artificial Intelligence, IJCAI, pp. 6526-6535, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05866,, 2022.
Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. Zero-shot lIm-guided coun- terfactual generation for text. arXiv preprint arXiv:2405.04793 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Lenart Celar and Ruth M. J. Byrne. How people reason with counterfactual and causal explanations for AI decisions familiar and unfamiliar domains. Memory & Cognition, -144496, 2023. ISSN 1532-5946.
Alex James Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. In Forty-first International Conference Machine Learning, ICML, 2024.
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for 1lms. arXiv preprint arXiv:2404.08555, 2024.
Furui Cheng, Vilém Zouhar, Robin Shing Moon Chan, Daniel Fürst, Hendrik Strobelt, and Menna- tallah El-Assady. Interactive analysis of Ilms using meaningful counterfactuals. arXiv preprint arXiv:2405.00708, 2024.
Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Llm-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306. 14924, 2023.
Ganqu Cui, Lifan Yuan. Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrafeedback: Boosting language models with high-quality feedback. arXiv preprint arX arXiv:2310.01377 2023.
11

---

Published as a conference paper at ICLR 2025
Samuel Curtis, Ravi Iyer, Cameron Domenico Kirk-Giannini Victoria Krakovna, David Krueger, Nathan Lambert, Bruno Marnette, Colleen McKenzie, Julian Michael, Evan Miyazono, et al. Research agenda for sociotechnical approaches ai safety. online, 2024.
Shih-Chich Dai, Aiping Xiong, and Lun-We Wei Ku. Lim-in-the-loop- Leveraging large language model for thematic analysis. In Findings of the Association for Computational Linguistics: EMNLP, pp. 9993-100011 2023.
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned SFT as an (user-steerable) alternative to RLHF, In Findings of the Associ- ation for Computational Linguistics: EMNLP, pp. 11275-11288, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In The 40th International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pp. 10835-10866, 2023.
Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. Faithful explanations of black-box nlp models using llm-generated counterfactuals. The 12th International Conference Learning Representations, ICLR, 2024.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Mare Dymetman. Compo- sitional preference models for aligning LMs. The 12th International Conference on Learning Representations, ICLR, 2024.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decoding-enhancel bert with disentangled attention. In The 9th International Conference on Learning Representations, ICLR, 2021.
Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. A survey of algorith- mie recourse: Contrastive explanations and consequential recommendations. ACM Computint Survey, 55(5):95:1-95229, 2023.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Eoin M. Kenny and Mark T. Keane. On generating plausible counterfactual and semi-factual ex- planations for deep learning. In The 35th AAAI Conference on Artificial Intelligence, AAAI, pp. 11575-11585, 2021.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. Openassistant conversations democratizing large language model alignment. In Advances in Neural Information Processing Systems 36, NeurIPS, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint Sttd::2403.1788: 2024.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 3kywork-rewarde Bag of tricks for reward modeling 1lms. arXiv preprint (tti:22410.84455 2024.
Rachel McCloy and Ruth MJ Byrne. Semifactual "even if" thinking. Thinking & reasoning, 8(1): 41-67, 2002.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intel- ligence, 267:1-38, 2019.
Van Bach Nguyen, Paul Youssef, Jörg Schlötterer, and Christin Seifert. Llms for generating and evaluating counterfactuals: A comprehensive study. arXiv preprint 200.022455.0072.. 2024.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, NeurlPS, 2022.
Richard E. Overill. Book review: "time warps, string edits, and macromolecules: the theory and practice of sequence comparison" by david sankoff and joseph kruskal. Journal of Logic and Computation, 11(2):356, 2001.
Junsoo Park, Seungyeon Jwa, Mciying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators. arXiv preprint arXiv:2407.0655/ 2024.
Ben Pikus, Will Venene Tony Chen, and Sean Hendryx. A baseline analysis of reward mod- els' ability accurately analyze foundation models under distribution shift. arXiv preprint arXiv:2311.14743, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188);9 2019.
Kaivalya Rawal and Himabindu Lakkaraju. Beyond individualized recourse: Interpretable and inter- active summaries of actionable recourses. In Advances in Neural Information Processing Systems 33, NeurIPS, 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJCNLP, pp. 3980-3990, 2019.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD, pp. 1 1335-1144, 2016.
Indira Sen, Dennis Assenmacher, Mattia Samory, Isabelle Augenstein, Wil M. P. van der Aalst, and Claudia Wagner. People make better edits: Measuring the efficacy of im-generated counterfac- tually augmented data for harmful language detection. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 10480-10504, 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716. 2023.
Sandra Wachter, Brent D. Mittelstadt, and Chris Russell. Counterfactual explanations without open- ing the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31:841, 2017.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expent arXiv preprint arXiv:2406. 12845, 2024a.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi- attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09522 2023.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training 2024b.
top-performing reward models. arXiv preprint 4rXiv:2406.088
Tongshuang Wu, Marco Túlio Ribeiro, Jeffrey Heer, and Daniel S. Weld. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/JCCNIP, pp. 6707-6723, 2021.
13

---

Published as a conference paper at ICLR 2025
Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani. and Pierre-Y Yves Oudeyer. Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. In Companion Proceedings of the 28th International Conference Intelligent User Interfaces, IUI, pp. 75-78, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? comprehensive study. In The 41st International Conference on Machine Learning, ICML, 2024.
Linyi Yang, Eoin M. Kenny, Tin Lok James Ng. Yi Yang, Barry Smyth, and Ruihai Dong. Gener- ating plausible counterfactual explanations for deep transformers financial text classification. In Proceedings of the 28th International Conference on Computational Linguistics. COLING, pp- 6150-6160, 2020.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The 12th International Conference on Learning Representations, ICLR. 2024.
14

---

whether or not the response is avoiding to give direct answers to Published as a conference paper at ICLR 2025
A THE HIGH-LEVEL EVALUATION ATTRIBUTES
We adopted a semi-automated workflow to obtain the list of attributes presented in Section 2.3 Us- ing the same test sets of the three datasets (described in Section 3) we prompt GPT-4o identify relevant attributes which potentially caused one response to be more preferred than the other re- sponse each test comparison. Outside the scope of reward modelling, similar techniques have been used in the thematic analysis of texts (Xiao et al. 2023 Dai et al. [023] Chew et al.) [0033) We list out such LLM-generated attributes, group similar ones, compare and complement the list with existing attributes in the literature.
We used the following prompt:
In the task of response quality scoring, trained deep learning model assigns real-valued scores for responses to questions. The higher the score, the better the response quality. The question "qquestion)'. The model assigned a score (chosen_reward) for response A: [[chosen.rapponse"" (rejected reward] for response B: "[ejectted response)'. List out some high-level attributes which might have caused the model to assign a better score for response A than response В. Some example attributes are: appropriateness, clarity, harmlessness, verbosity, etc. Only output the attributes in a conma -enaratee list.
The model assigned score
From the test sets, we obtain the following attributes, sorted by the number of occurrences in the LLM responses descending order:
clarity, relevance, harmlessness, informativeness, detail, conciseness, specificity, completeness, structure, helpfulness, verbosity, engagement, coherence, appropriateness, comprehensiveness, empathy, accuracy, sensitivity, creativity, legality, ethicality, readability, factual-accuracy persuasiveness, veness, privacy, humour, assertiveness
This list encompasses all attributes in the hs2 dataset Wang et al. (2024b), where attributes were used as dimensions in the regression labels. also overlaps with the ones UltraFeedback dataset CCui et al. 2023) apart from
following and honesty. As the former is not the focus of the datasets we experiment with, we only additionally include honesty, relabelled to avoid-to-answer for better relevance. The list of attributes deviates from the list in the OASST2 dataset (Köpf et al. 2023). which focuses on distinguishing various types harmful con- tents, e.g. hate speech, violence, etc. We stress that the high-level attributes could be customised to best match any specific dataset characteristics, and in this work we provide a general list.
We conclude our final list below, sorted in alphabetical order:
the question,
appropriateness the extent which the response is appropriate terms of language style, politeness, and whether it contains any sarcasm,
assertived iveees:: the extent to which the response sounds very certain and contains judgements,
clarity: whether or not the response is clear and easy to read,
coherence: whether or not the contents in the response are self-contained and clear,
complexity: the intellectual burden required by a person to understand this response,
correctness: whether or not the response factually correct,
engagement: the extent which the language style of the response is trying to engage with the person who wrote the question,
harmlessness: whether or not the response is relevant to any potentially unsafe, im- moral or illegal behaviours,
15

---

Published as a conference paper at ICLR 2025
helpfulness: whether not the response addresses the points raised in the question, informativeness: whether or not the response provides informative knowledge, neutrality: whether not the response is neutral and is without biases towards certain groups,
relevance: whether or not the response is in relevant context as in the question, sensitivity: whether or not the response is relevant to any personal, sensitive, or private information,
verbosity: how many relevant details are included in the response, and whether or not the response is too long.
B PROMPTS USED IN OUR METHODS
We present the prompts binary comparison, we run the prompts twice to generate perturbed responses separately for the chosen and rejected responses. In the prompts, parameterise the responses, and the other variables in the prompts can be determined depending on whether the chosen or rejected response input response_1. To avoid L.M input and output being too long thus potentially hurting performance in each query, Step prompt is repeated for each evaluation attribute.
Specifically, for each for the two steps described in Section 2.3
Step prompt:
In the task of response quality scoring, trained deep learning model assigns real-valued scores for responses to questions, the higher the score the better the response quality.
The question '(questionj', for response A: 'fresponsel]'' The model assigned score (score.2) for response B: (rrespone_2)'.
model assigned score (score.1}
The high-level attributes that potentially caused the model to assign a {better/worse) score for response than response are (attribute list].
Your task: for each attribute in this list, identify the words in response A that are relevant to it.
Only output the attributes and their associated words like this: 'attribute: wordi, word2, word3'. Each line should contain comma-separated word list for one attribute.
It is fine to have repeated words in the words identified for each attribute, but you need to keep them in their original order of occurrence in the response A.
Step prompt:
In the task of response quality scoring, trained deep learning model assigns real-valued scores for responses to questions, the higher the score the better the response quality.
The question is '(question''. The model assigned a score {score.1} for response A: 'fresponsel)'. The model assigned score (score.2) for response B: B ([response_2]'.
The potential high-level attributes that caused the model assign a (petter/worae) score for response A than response B is: (attribute). This attribute concerns deserbbptodd.scritiiioo..o
Your task is to modify response A. Here is a list of requirements for the modification.
The modified response A becomes a (petter/worse) response to the question than response
16

---

Published as a conference paper at ICLR 2025
[Positivelly/eeraaieeey) change the semantic meaning of response by making it (better/worse) in terms of (attribute).
The changes made to response A should be centered around the following words: (relevant words for this attribute identified in Step 1}
Only output the modified response A.
C PROMPTS USED IN THE RANDOM PERTURBATION BASELINE
Generate random perturbation of this piece of text: (response).
Only output the perturbed text.
Do not output any characters other than English texts and common punctuation.
D ABLATION ANALYSIS
Different prompts can have large impacts on the perturbations generated. our prompt, we specify that
The changes made to response should be centered around the following words: (relevant words for this attribute identified in Step 1}.
In this ablation analysis, highlight the importance and effectiveness of this additional require- ment in keeping the perturbations close to the original responses while balancing the CF and SF coverages.
Following the same experimental setup described in Section B) we quantitatively compare CFs and SFs resulting from perturbations obtained with three different prompts. Our final prompt is referred to as the center-prompt. One of the different prompt (only) replaces the above sentence by: Response can only be modifies by deleting, replacing, or inserting words, at the locations of all or subset of the
following words: (relevant words].
posing stronger restriction the changes made in the perturbations. The other different prompt (pass) has no constraint on the extent of changes by removing this sentence. Table reports the comparison results.
Dataset
Prompt only 0.49
cov, CF
cov, SF
syn. dist.
sem. dist
sem. div.
0.86=.07 071
0.48+.026
0.30+.028
0.21
harmless pass
0.64
0.15
0,81
0.48+.028
0.29+.019
center
0.68
0.90 .611
0.65
0.36
0.31
only
0.72+ 008
0.79= 054
0.47
0.24+.019
0.13+.015
helpful
pass
0.80+L057
0.13 .083
0.75+.011
0.374.023
0.194.013
center
0.80
0.74 066
0.61
0.28
0.20+t.012
only 0.28#.088
0.76=.178
0.29
0.07+.034
0.04+.013
hs2
pass
0.39+.094
0.7 140
0.50+.068
0.11+.030
0.06+.012
center
0.33
0.84 1449
0.31 1.03
0.07
0.071.010
Table 4: Comparison of CF and SF coverage, distances and diversity for three different prompts.
Overall, the most restrictive only-prompt, though resulting in perturbations which are closer to the original response, has the lowest CF coverage, and is not as semantically diverse as the center- prompt. The pass-prompt, the other hand, has much higher distances than the other two prompts. Additionally, this downside does not result in improvements in explanation coverages and diversity. We can conclude that for the purpose of generating contrastive explanations, the center prommp can achieve the most balanced performance among these prompts, therefore was used in our final configuration.
17

---

Published as a conference paper at ICLR 2025
E EXPLANATION EVALUATION RESULTS FOR CORRECT AND WRONG PREFERENCES
In this analysis, we look evaluation metrics for explanations respectively for correct and wrong preferences made by RMs to further understand the effectiveness of our approach. Following the same experimental setup described in Section we further divide each test set by whether or not the model's preference is correct, i.e. matches the ground truth label. We report in Table the evaluation results of our method separately for correct and wrong preferences.
Dataset
Preference
cov. CF
Cov. SF
syn, dist.
sem dist
sem. div.
correct
0.71
0.88
0.65+.026
0.36#.037
0.30=.029
harmless
wrong
0.67
0.90
0.64+.017
0.37#.039
0.32
correct
0.78
0.73+.007
0.63+.018
2,25
0.19
helpful
wrong
0.86
0.79
0.57+.059
0.26+ .44
0.21
correct
0.29 161
0.84 158
0.334.033
0.08 021
0.07=:016
hs2
wrong
0.38 170
0.82.t. 156
0.27
0.05
0.06
Table Comparison of CF and SF coverage, distances and diversity for explanations of correct and wrong preferences.
Intuitively, in cases where the ground truth chosen response is obviously more preferred (in human standards) than the rejected response but the RM made the wrong comparison, our method will generate perturbations which make the ground truth chosen (rejected) response better (worse). In this case, might be expected that all perturbations tend to become counterfactuals, but because the RM's local behaviour is unclear, it can still result in balanced mix of counterfactuals and semifactuals. Indeed, in Table 5J we observe no clear distinctions the results, validating that our method is effective in finding explanations for both correct and wrong preferences made by RMs.
F SEPARATELY CONSIDER PERTURBING CHOSEN AND REJECTED RESPONSE FOR GLOBAL SENSITIVITY ANALYSIS
In Section 4 have separately considered global sensitivity for Y and W_ because perturbing y+ to become worse than
an independent process from perturbing y- to become better than y+- We validate this intuition by calculating the ranking similarities (Kendall's correlation coefficient) between each dataset's s Y and Y branches. The results for cach model are presented in Table 6 We observe that only and pythia have weak correlations in some cases, and the of the results are generally uncorrelated.
harmless
he helpful
hs2
v1
0.18
0.03
0.12
v2
0.49
0.30
0.28
pythia
0.28
0.31
0.07
Table 6: Ranking similarity of global sensitivity between the perturbed responses for chosen and rejected responses.
G FURTHER QUALITATIVE ANALYSIS
In this section, we present further qualitative analysis, using our contrastive explanation method to discover failure cases in tfate-of-the-arn RMs.
Following the findings on the undesirable behaviours of model v2 in Section 4.2. I we additionally include two better-performing RMs and investigate whether they improve over the weaknesses in v2. We use -ffmaa----OffettBa-RRR--3 (ILamaOB) trained by Park et al. 2024). and Skywork-
18

---

Published as a conference paper at ICLR 2025
-... (Llama and rank among the top- pop-peroorming models in the RewardBench benchmark (Lambert et al. 2024
( trained by Liu et al. (2024). both have 8 billion parameters
Question:
money recenily on dug-fight??
dog is winner?
Llamo
UlmaSW
Chosen Response: What esactly do
that?
Rejected Response: Well, there are many different strategies styles. -3.80 it's best learn the characteristies of winning dog. Can ask you how money
3.50
-8.63
-6.22
-24.8
-3.00
Perturbed responses for Rejected
answer:
dentifyy winning you should look for traits like agility, strength, and training. Can ask you how money you
3.71SF
-8.18,55 SF
123.0, SF
appropristeness:
fighting ilegall and unethical. Instead of focusing on winning strategles, consider reporting this activity the authorities, How can help you find better ways to your money?
sorry
important understand
55.4,,
-10.0,
correctness:
gus strategles styles,
fighting altogether as it's illegal and unethical. Can ask you how much moneey lout?
-2.77,c
-6,15,
-1.22 OF
I'm sorry to hear about your loss, but can't support or provide advice activitles like fightighting... activities.
engage safe
-1.00, OF
-5.15, CF
-5.78
neutrality:
many strategies styles, the characteristies of winning dog. However, must emphasize that hightigh illegal and unethical. Con azk how much money you
3.03,35
1-8,44,5F
-17.3,
relevance:
that dog- fightigg is illegal mnothiral. instrad, convider forusing on how to recovrr your insses through and ethical means. Can ask you how much money you sensitivity:
impo rtant
16.21,
-16.3
many strategies Myles.
the characteristics of winning dog. It's important to focus on the traits
the dogs rather than the financial aspect.
4.37
124.1,
(a) Perturbed rejected responses.
Question: lost a lot of money recently dog-fight How can af
winnar?
Chosen Response: What exactly do you mean that?
Rejected Response: Well, there are many different strategles and styles.
3388
13.99
-24.8
23.0
characteristics winning dog, you
much money you lost?
Perturbed responses for Chosen
answer:
esactly you
you figure day
a winner.
appropristeness:
What the hell do you mean by that?
-2.76,5 SF
1-6.06, SF
24.33
What esactly you mean by that? Dop-fighting grest way to
make money.
-.32.
18.37.5F
-31.0,
What exactly do you mean by how make a dog winner in fight?
3390, OF
-26.0, crl
neutrallty:
What esecily do you meen by thet, you inresponsiblee gembler?
-3033.5 SF
-0.73.51
-27.2. CF
relevence:
What exactly de you mean by the weather today??
-3.44 OF
-6.09
33.5,
sensitivity:
What exectly how much you lost?
-28.3.0
(b) Perturbed chosen responses.
Figure 6: Example for investigating improvements of L. amma and L. allasaSi v2.
19

---

Published as a conference paper at ICLR 2025
We begin by observing the two RMs' behaviours on the text perturbations in the same example shown in Figure 4
We present the results in Figure 6 The two RMs make the same preference over the two original re- sponses as v2 for this example. Figure ba shows RMss' predicted rewards for the perturbed rejected responses. We observe that for they retain the desirable behaviours observed in v2, i.e. in pertur- bations associated with attributes ixproprratenenn
harmlessness neutrality, relevance, by mentioning betting on dog-fights is an illegal activity, the two RMs assign higher rewards compared with that of the rejected response. On the other hand, in the perturbed responses associated with answer and sensit ivity, providing more detailed an- swers about how to identify winning dogs resulted in model v2 assigning higher rewards. The same undesirable behaviours are also observed in the more powerful 1.1 ama OB and 1.1 ama SW.
correctness,
More interesting observations can be seen in Figure 6b which shows the RMs' predicted rewards for the perturbed chosen responses. As the original chosen response avoided answering the harmful question, incorporating more detailed harmful content in it caused the rewards (from the two larger models) to decrease (correctness) identified that calling the user an "irresponsible gambler" (neutral (iy) is problematic by lower- ing the rewards. However, we see some clearly undesirable behaviours in the two larger models:
sensit tiviitt). They also correctly
L1 amaSw assigned a higher reward for an LLM answer asserting that it can help with harmful activities (answer).
By incorporating disrespectful language "the hell" in the response (appropriatenesse) v2 and 1.1 amaOB increased the their rewards.
Therefore, we highlight that our explanations help reveal more fine-grained local behaviours of RMs, allowing us to efficiently find deficiencies even in the best-performing RMs.
Next, we investigate the case where the RMs fail to identify disrespectful language "the hell" at a larger scope.
We observe that in the harmlese dataset, it is common that one LLM response avoids an- swering harmful or sensitive user questions by saying something like "What do you mean?" As was the case in the example shown in Figures 4 and 6 when perturbing such responses along appropri atenes,, our method frequently inserts the disrespectful phrase "the hell" to them. We collect questions with such responses together with their disrespectful perturbations from experiments on harmless esss dataset, and obtain 1 110 test comparisons. We test how frequently the RMs prefer the responses with "the hell" in them over the original responses, i.e. disrespectful perturbation win rate, and report the results in 7
pythia
v1
v2
LLamaOB
LlamasW
win rate
0.75
0.51
0.56
0.30
0.15
Table 7: Disrespectful perturbation win rate, lower values are better.
As shown by the results, all tested models sometimes prefer the disrespectful perturbation, with the two 8-billion-parameter RMs showing better performances than the smaller ones. Note that by the time of writing, I.1 ama and L. ama SW rank 22nd and 7th in the RewardBench benchmark, and LlamaSW is the best- west-performing RM which has less than 10 billion parameters. Our findings vali- date that there is scope for improvements over the current sfate-of-the-at RMs. The investigations in this section also exemplify how our contrastive explanation method can be useful for discovering new failure cases RMs, shedding light on directions for future improvements.
20


Reviews:
[{"review_id": "8n9hmEo9Te", "reviewer": "Reviewer_sH3s", "summary": "The work proposes looking for counterfactual (CF) and semifactual (SF) examples to explain reward models. They prompt language models in a two-stage manner, where an LLM is required first to identify words related to a list of attributes and then to perturb these words to flip the reward prediction. Quantitative experiments examine the CF and SF discovery success rate (CF and SF coverage) and reveal the proposed method mainly improves CF coverage at the cost of compromise in SF coverage in some cases. Perturbed sentences in the proposed method also have a lower syntactic and semantic distance to the original sentence. Some qualitative analyses are also done.", "questions": "1. Are there any messages we can take away from the qualitative analysis to improve the trustworthiness of reward/policy LLMs? Existing findings are more or less trivial and superficial, and it would be better to provide more insights that either deepen our understanding of reward models, or improve transparency of policy models.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A novel prompting-based counterfactual and semifactual example generation method for explaining reward models\n1. Experiments do show some merits of the proposed method, such as generally improved CF success rate (CF coverage).", "weaknesses": "1. The work's main motivation, \"More transparent RMs would enable improved trust in the alignment of LLMs,\" is mostly intuitive and needs more explanations. Even if reward models are totally transparent, how they translate to more transparency in the trained policy model is still not clear, as the policy LLM is still a black box model. Math question training data, as an extreme case, often comes with a more rigid threshold of logical consistency in its data and does not translate to improved reasoning transparency of trained LLMs on that data. \n\n1. How the constructed CF and SF examples are \"minimally perturbed\" and \"maximally perturbed\" ones are less reflected in the construction method. Did the authors encourage LLMs to increase/decrease the perturbation ratio on an example until they cannot anymore?\n\n1. The perturbation method relies on a subjectively defined list of categories, which limits the scope of explanation in reward model behaviors. The reward model could vary its prediction due to other factors that humans might not think of, which poses a larger threat to its trustworthiness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mi9DJ9AOaH", "reviewer": "Reviewer_WCX9", "summary": "The authors propose a two-step approach to generating diverse perturbations of language completions, and show that this method can be used to improve the diversity of contrastive explanations of reward model outputs.  (i.e., perturbations that result in pairs of counterfactual or semifactual paired completions that can then be used to explain a reward model's behavior). \n\nThey validate their approach both quantitatively and qualitatively and show how it can give insights into reward model behavior, including the sensitivity of the reward model to perturbations along a set of specified axes, both at a local and global level.", "questions": "Please see weaknesses. The significance one would provide the largest improvement, but perhaps hardest to address. The related work one should be easy to address and therefore high ROI.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am not very familiar with the XAI literature or its application to reward models, but I have not seen this particular approach before and believe the methodology is novel (but see weaknesses for some requests on related work). I think researchers who work with reward models will generally be interested in ways of explaining the reward model predictions, and so I believe the topic and proposal in this paper meets the significance threshold for a major conference (but see weaknesses re: chosen RMs). \n\nOverall the paper was well written, and the figures are generally high quality, so I believe it has sufficient writing polish for publication. See weaknesses for some nits. \n\nWhile the method itself is relatively straightforward (the authors cite a couple works that have done similar things in other contexts), it is demonstrated to be effective. The experiment design is appropriate, and the authors are testing the right things. Despite the limited RMs tested and the use of binary preference reversals (pref flip rate) instead of cardinal rewards (see weaknesses), I believe the method itself is reasonable, and the experiments provided sufficient support for the approach.", "weaknesses": "Related work: At lines 179-181, a few related works are mentioned that could alternatively do perturbations or CFs for reward model explanations. These are not expanded upon in the text, and they do not seem related to the baselines at lines 277 - 284. Is this because they do not apply at all, or how do the baselines relate to past work? This casts some doubt on how the present work fits into the literature. \n\nConceptual: It seems the finite attribute list limits the kinds of explanations that the proposed method can provide. For a given part of completions it seems possible that an attribute that doesn't quite relate to some of the listed attributes (or related to multiple listed attributes) would be the main causal factor. For example, one completion might be preferred because it uses a more clever metaphor than the other completion. I'm not sure how this would fit in any category, and it seems like perhaps a dynamic approach or wildcard attribute might be better in some respects (I realize this might conflict w/ the experiments that rely on ranking attributes). \n\nConceptual: It's unclear to me why we would prefer Preference Flip Rate (PFR), which is an ordinal measure, to the cardinal measure provided by the rewards themselves. For comparability when rewards are of different magnitude, you can always normalize by either the mean reward, or the gap between chosen and rejected. A cardinal approach seems strictly superior to me, and no real justification is provided for considering CF/SF to cardinal shifts in rewards (I would wager than the authors use flips because of the prior work that generates CF/SF contrastive explanations).\n\nSignificance: The tested RMs are all extremely small / poor by today's standards, and would not be used in any real context. 7-8B parameter RMs are many times stronger and more relevant, and are easily runnable on a single 24GB GPU, so I would not expect there to be a compute barrier here. So it's not clear to me that the positive results obtained for these RMs will carry through to modern ones. This limits the present significance of the results, and I would strongly encourage the authors to run experiments with one or more stronger RMs. \n\nNits:\n- L419/420 mejntions that you compare global/local rankings--- are there supposed to be results for this, or is this just to produce the Figure 4 example\n- Figure 4 caption could be significantly improved. It took me a second to figure out how to read it / what it was presenting (mind you, I was reading in black and white so couldn't see the color difference between the dashed lines, but it would still be good to explain them in the caption).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gylOKRiwGS", "reviewer": "Reviewer_CGue", "summary": "The paper presents an analysis technique for reward models. By perturbing input examples with LLMs along pre-determined attributes to generate counter-factual (CFs) examples (that flip the preference judgment of a reward model by modifying an attribute) and semi-factual (SFs) examples (examples that retain the label despite editing an attribute), the authors are able to obtain quantitative (robustness to perturbation) and qualitative (which attributes are important to the preference judgment) signal about reward model judgments.\n\nThe main contribution is the analysis technique itself, with experiments comparing the quality of CF/SFs generated by the proposed prompting technique against baseline methods of perturbation to demonstrate the quality of explanations generated, and a case study comparing three open-source models to use the method to identify differences in their performance qualitatively.", "questions": "1. L.42 - Can you clarify the distinction to (Zeng et al. 2024) and (Park et al. 2024)? My understanding is that these papers make directed perturbations based on a specific attribute, while the proposed technique looks to analyze model performance via a set of targeted perturbations (SF/CFs). I don't see this as a weakness per se, it's more that L.59 mentions that there have been no XAI techniques explored for reward models but the way L.42 has been written contradicts this.\n\n2. One unanswered point is about the sensitivity of the analysis technique to the choice of LLM used to generate CF/SFs. Here the authors use GPT-4o. I think it is important to test because the main contribution is the analysis technique itself, so we should ablate the sensitivity to other (less powerful/open-weight) LLMs since not everyone might have access to GPT-4o. \n\n3. L.186 - Can you comment on the sensitivity of the method to the selected high-level evaluation attributes? Are there patterns of evaluation attributes for which the generated CF/SFs are less effective at flipping the label - in these cases, how do we decide if this is because the RM is robust to perturbation along this attribute or if the LLM generating the counter-factual is not able to correctly edit the response based on that attribute. \n\n4. L.262-265 - In addition to the fraction of test comparisons where at least one valid CF/SF is found, why not differentiate between cases where multiple valid CFs exist as examples where there is more brittleness in the RM?\n\n5. L.706-706 - This method to elicit attributes from the test set is similar to techniques that use LLMs for qualitative coding so it might be helpful to ground the method in similar works [1,2,3].\n[1] Xiao, Ziang, et al. \"Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding.\" Companion proceedings of the 28th international conference on intelligent user interfaces. 2023.\n[2] Dai, Shih-Chieh, Aiping Xiong, and Lun-Wei Ku. \"LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis.\" Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.\n[3] Chew, Robert, et al. \"LLM-assisted content analysis: Using large language models to support deductive coding.\" arXiv preprint arXiv:2306.14924 (2023).\n\n6. In Table 1, is there a reason for the drop in performance between the coverage on CF and SFs that were generated perturbing 'Both' responses than just \"Chosen\" or \"Rejected\"? Uniformly these scores are lower.  \n\n7. Regarding the qualitative analysis in Section 4, it might be interesting to introduce some 'control' attributes that are not meant to flip the label of the plot to provide some context to the 'treatment' attributes selected. For instance, I imagine Fig. 3 would be improved with such additional attributes, and I imagine if an RM flips a label on these attributes, then it's a method to identify biases in the judgments. \n\n8. L.324-326 - While it is understandable that the generated CF/SFs are more similar to the original response which likely costs diversity, I think you have a better justification for the method, than the using of the 15 different attributes is just pure efficacy i.e. the prompting method generates valid CF/SFs at a higher rate than the baselines, this itself is sufficient.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed technique is clean and simple, draws from existing literature on SF/CFs in XAI, and importantly is general enough to work for any RM (L.132-134), making it a useful tool for analysis. This is particularly the case when you might not have access to the training data of some black-box RMs and can yet obtain qualitative insights on their performance. \n\n2. The main contribution is the analysis technique. Section 3 makes a good case that the proposed prompting technique is an effective way to generate good CF/SFs and Section 4 then shows how this can be used to interpret RM behavior. The flow is easy to follow and the experiments validate the authors' claims. \n\n3. L.405-411 - The experimental setup chosen for Section 4.1 is clever because we have ground truth knowledge that the Deberta-v2 reward model was the only one trained on the _harmless_ dataset, so it is more sensitive to perturbations that change the 'harmless' attribute. This validates that the behavior identified by the proposed analysis technique is 'correct' in this case. A very interesting follow-up would be introduction of perturbation attributes that are not directly related to the training task of the RM, and then this technique can identify biases in model performance.", "weaknesses": "1. While the experiments generally back up the claims made, I think there is scope to more rigorously test the robustness of the proposed technique. See Q2, 3, and 7 below for specific concerns. Another concern is that the generation method for attributes looks at examples from the test set. This is fine, but it grounds the perturbations to 'expected' ways and doesn't elicit any 'surprising' biases from RMs. This is easily remedied of course, simply by sampling attributes along different ways, but it would be interesting to see if the generated SF/CFs are sensitive to the attribute distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VkjBTRV0iQ", "reviewer": "Reviewer_T2w6", "summary": "This paper studies the interpretability of reward models through contrastive explanations. It proposes a novel pipeline of getting high-level attributes, computing meaningful textual perturbations, and categorizing into counterfactual (CF) and semifactual (SF) explanations. Then it performs case studies on how to use the CFs and SFs to interpret reward model decisions.", "questions": "I put my questions in the weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall i like this paper!\n* The problem of RM interpretability is important.\n* The whole pipeline of generating CFs and SFs make sense\n* The case studies demonstrate the usefulness of the generated explanations very well", "weaknesses": "* I am a little skeptical of the high-level attributes concluded by the model. The attributes are generated per sample, and then aggregated to find the top attributes. Then does the LLM give consistent attributes (e.g. do they consistently use the exact word of avoid-to-answer?)? Even if an attribute shows up in LLM's summarization for many samples, does it always mean the same thing (e.g. attribute complexity might mean good/bad things)? \n* Does the prompting in step 2 always work as intended? In Fig 4, it seems some responses are modified almost completely --- does it mean the LLM doesn't faithfully follow step 2 prompt of only modifying the identified word?\n* Step 2 prompting requires the modification to be both better/worse, and also better/worse in terms of targeted attribute. Why do we still need the first condition? \n* Are there more compelling evidence for the usefulness of the paper? E.g. some reward models with known issues (e.g. length bias), and the generated data can identify that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)