RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Authors: David Reber
Sean M Richardson
Todd Nief
Cristina Garbacea
Victor Veitch
Venue: ICML 2025 poster (2025)
Keywords: Explainable AI
Counterfactual Explanations
Causal Inference
Alignment
Large Language Model
NLP
Primary area: deep_learning->large_language_models
TL;DR: Explainability of reward models relies on attribute-specific treatment effects, which can be estimated with imperfect counterfactuals by rewriting twice.

Abstract: Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs.
However, reward models are black boxes, and it is often unclear what, exactly, they are actually rewarding. In this paper we develop Rewrite-based Attribute Treatment Estimator (RATE) as an effective method for measuring the sensitivity of a reward model to high-level attributes of responses, such as sentiment, helpfulness, or complexity. Importantly, RATE measures the *causal* effect of an attribute on the reward. RATE uses LLMs to rewrite responses to produce imperfect counterfactuals examples that can be used to measure causal effects. A key challenge is that these rewrites are imperfect in a manner that can induce substantial bias in the estimated sensitivity of the reward model to the attribute. The core idea of RATE is to adjust for this imperfect-rewrite effect by rewriting *twice*. We establish the validity of the RATE procedure and show empirically that it is an effective estimator.

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
David Reber'
Sean M. Richardson Todd Nief
Cristina Garbacea Victor Veitch 23
Abstract
To understand reward models, we would like to be able to measure their responsiveness to high-level attributes of responses, such as helpfulness, correctness, or sentiment. The ability to measure the responsiveness of an RM to an attribute would serve the purpose of this paper is to develop such a measurement procedure.
Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and is often unclear what they are actually re- warding. In this paper, we develop Rewrite-based Attribute Treatment Estimator (RATE) as an ef- fective method for measuring the sensitivity of a reward model to high-level attributes of responses, such as sentiment, helpfulness, or complexity. Im- portantly, RATE measures the causal effect of an attribute on the reward. RATE uses LLMs to rewrite responses to produce imperfect counterfac- tual examples that can be used to measure causal effects. A key challenge is that these rewrites are imperfect in a manner that can induce substan- tial bias in the estimated sensitivity of the reward model to the target attribute. The core idea of RATE is to adjust for this imperfect-rewrite effect by rewriting twice. We establish the validity of the RATE procedure and show empirically that it is effective estimator. Code is available /RATE.
important diagnostic tool, and
Naively, one might attempt to estimate RM responsiveness to an attribute by comparing the average reward assigned to responses with and without the attribute, using a labeled dataset. However, this approach is flawed: it conflates the target attribute's influence with correlations present in the evaluation data. For instance, suppose we are measur- ing how much an RM responds to "sentiment" and, unbe- knownst to us, highly-negative samples in our evaluation data tend to have more typos and sloppy formatting. Then if we naively average the reward over the negative samples, and again over the positive samples, we'll inadvertently measure the effect of not just sentiment, but typos and for- matting as well. In particular, this means we can measure large effect of an attribute on the RM, even if the RM is completely insensitive to that attribute.
https: / / httgggttthht.tteggggggggg.ggg
For measurements of attribute influence to be meaningful, they must be isolated from confounding factors and spurious correlations in the evaluation data. To address this, we propose formalizing RM explainability the causal effects of attributes on the reward. Hence, the goal is to determine how a reward would change if we could modify a response to alter only the attribute of interest while holding all other factors constant. This counterfactual perspective isolates the causal relationship, making it possible to disentangle an attribute's 's true effect from confounding factors.
1. Introduction
Reward models (RMs) play a critical role in aligning large language models (LLMs) with desired behaviors. These models evaluate the quality of LLM outputs and are widely used in, c.g.. post-training, inference-time sampling adjust- ment (through best-of-n sampling), and LLM evaluation. However, reward models, typically implemented as fine- tuned L.MMs, black boxes. Accordingly, it can be diffi- cult to understand what reward model is actually rewarding. Despite their importance, explainability research for RMs remains underexplored (Lambert et al., 2024).
Notice that, if we had access to counterfactual pairs of re- sponses (i.e., pairs where the only difference is the attribute of interest), we could estimate the target effect by simply comparing the rewards assigned to each response. A natural idea is to use L.LMs to generate such counterfactualpairs by rewriting responses change only the target attribute. If the rewrites were perfect, we could directly measure the causal effect of the attribute on the reward. However, in practice, LLMS produce imperfect rewrites, changing off-target at- tributes as well. These imperfections can substantially bias the estimated causal effect, as we will see in Section 5.
'Department of Computer Science, University of Chicago 'Department of Statisties, University of Chicago Data Science Institute, University of Chicago. Correspondence to: David Reber <reber& cacerrucchccao.cdu..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Naive Method
Single Rewrite
RATE: Double Rewrite
"Toh movie was groee"
"Tah movie was goasc"
"The moste bast"
fil was, deareat reader offersive the intullies. My roview procseds five parts...
"The movie was great"
Target attribute confounded in data!
Rewrite changes off-target attributes!
Off-target changes cancel out (in expectation)
Figure 1. When generating counterfactual pairs, LLMs change other attributes, such tone, length, or grammar. Empirically, using the rewrites of rewrites corrects for this bias. (Left) Naively sampling pairs which differ on the attribute of interest (e.g., sentiment) will lead biased estimate of the causal effect because other attributes may also change. (Middle) When rewrite response to change the attribute of interest (c.g., from positive to negative sentiment), the L.LM may also change other attributes, such as fixing typos. (Right) Rewriting the rewritten response again tends to cancel these off-target changes, in manner make precise in Section 3.
There a growing literature on estimating the causal effects of attributes of text (Feder et al., 2022; Grimmer et al., 2022; Jin et al., 2022; Chen & Chu, 2023; Gui & Veitch, 2023). Generally, this work provide methods for estimating causal effects by either making strong assumptions on the evaluation data or compensating for potential confounding factors.
baselines are highly influenced by spurious associa- tions. We also show that this causal-vs-cone vs-correlational problem substantially affects the explanations of lead- ing reward models using common benchmarks.
2. Setup
A reward function is a function that takes a prompt æ and a response y as inputs and return real number indicating the quality of the response for the prompt.' To develop an eval- uation procedure, completion pairs {(x', y²'], where the æf are prompts and the yf are completions (also referred to as "responses"). We are interested in understanding how the reward model re- sponds to a certain attribute W (such as sentiment or length). We consider the case where each prompt-completio pair labelled with w W(x', y²) ) {0, 1} indicating whether the completion has the attribute of interest.?
Note that our question fundamentally differs from the simi- larly named counterfactual explanation paradigm (Wachter al., 2018) studied recently in the context of reward models by Jiang et al. (2024). While counterfactual explanations aim to explain the behavior of a model by finding a mini- mally perturbed input that changes the model's output, we are interested in measuring the average treatment effect of an attribute on the assigned reward, as measured across many counterfactual pairs.
begin with fixed dataset of prompt-
The contribution of this paper is to develop procedure that allows us to use the imperfect rewrites from LLMs to get a correct estimate of the causal effect of an attribute on reward model. The development is as follows:
We focus on binary attributes for simplicity many at- tributes of interest can often be naturally binarized.
1. We formalize RM explainability as an Average Treat- ment Effect (ATE) of high-level attributes of responses, the reward assigned that response.
Naive Method If we want to measure the sensitivity of a given reward model to an attribute of interest, such as helpfulness, the obvious approach is to take the dataset of prompt-completion pairs, label each completion as helpful
2. We develop measurement procedure (RATE) which utilizes two imperfect rewrites to estimate the ATE without needing to explicitly list confounding factors in the evaluation dataset.
'A reward function can also be viewed as taking two responses and returning a relative preference between them; our results ex- tend easily to this
well.
3. We show that RATE provides a valid estimation of the true causal effect under mild assumptions.
"We include the prompt x in the argument of W because the attribute may be prompt- dependent. For example, W might repre- sent helpfulness, which varies based on the context given by the prompt. A recipe could be helpful for questions about cooking but not for questions about history.
4. We empirically demonstrate that RATE correctly esti- mates causal effects, while the naive and single-rewrite
2

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
or unhelpful, then check whether the rewards for the helpful responses are higher than the rewards for the unhelpful re- sponses. Mathematically,, we define this average conditional reward difference as:
in reward when we take an unhelpful response and make it helpful. These estimands can differ substantially from each other and from the ATE (see Figure 8 in Appendix D.1), There is no reason to expect these quantities to align in gen- eral, so thought should be given to which is most relevant to the question at hand. Indeed, even human preferences are often asymmetric (Kahneman & Tversky, 2013), so we might expect reward model preferences to be as well.
Tnaive
Σ 1E1 (="'"")====
R(a', ,8'
R(a', .
110 (a* (u'.a")):w =")====0
where n1 and no are the numbers of examples with W- and W O, respectively.
3. Rewrite-based Attribute Treatment Estimator (RATE)
This estimates the correlation between the reward and W.
Whatever our choice of estimand, we need a method to estimate it. Here, we develop method, RATE, that uses rewrites to estimate the causal effect of an attribute on reward model. The core idea is to create pairs of responses where the only difference is in the attribute of interest, even if the rewrites are imperfect.
Y[R(X,Y) | W -[),) Y) W = 0],
where the expectation is taken over the distribution from which our evaluation examples are drawn. The problem here is that, even in the infinite data limit, this quantity does not generally isolate the effect of W on R. For instance, if the procedure we use to collect the evaluation data has a correlation between helpfulness and length, then the effect of these attributes will be conflated in the naive estimator (see Figure I, middle). Consequently, we will misinterpret the true behavior of the reward model.
Rewrites With LLMs In practice, we implement rewrites using a large language model (LLM) (see Table 1). We begin with a labeled dataset containing ground truth binary variables for attributes such as complexity, sentiment, or helpfulness. We then instruct the LLM to rewrite the re- sponses to the opposite state of the binary variable. For example, we may instruct: "Rewrite this response to express negative sentiment and change nothing else."
Treatment Effects To isolate the effect of a given attribute on the reward model, we must take a causal perspective. Concretely, we can formalize the responsiveness of a reward model to some attribute W as the average treatment effect (ATE) of W on the reward:
We use Re(x*, y², w) to denote the rewrite operation, which takes a prompt-response pair (z',y') and desired at- tribute value w, returning modified response I such that When x² is clear from context, we drop
ATE E(R(X, - R(X, (EXXXY(O))
(1)
W(x .¹) it for brevity and write the operation as Re(y', ,w), even though the rewriter in general may depend on the input prompt.
where X is random variable for the prompt, and (1) and (0) are potential outcomes for responses. This quantity is the expected change in reward if we were to change the at- tribute W from 0 to 1, while keeping all other aspects of the response fixed. The random pair of responses (Y) (1)) are identical in all aspects except for the attribute W-e.g., if W/W is helpfulness then each counterfactual response should have the same writing style, sentiment, topic, etc. In general, we only observe one of the counterfactual responses in our dataset (Figure 1, left) this is the fundamental problem of causal inference (Imbens& & Rubin, 2015).
Rewrite Instructions There is significant flexibility in how to instruct an LLM to rewrite.
For instance, when rewriting for "helpfulness", we might instruct the LLM to "Rewrite this response be more help- ful", instruct it to "Rewrite this response to be more helpful, providing additional relevant information clarifi- cation." In this example, the second instruction makes the meaning of "helpful" more precise. Generally, changing the instruction changes the nature of the rewrites generated, and thus changes the attribute that is being modified.
Choice of Estimand Beyond the ATE, we will also con- sider the average treatment effect on the treated (ATT) and the average treatment effect on the untreated (ATU),
Ambiguity in interventions unavoidable in causal infer- ence (Hernán, 2016). In our context, there is subjectivity in what helpfulness, complexity, or sentiment actually mean. An ad advantage of rewrite instructions is that we can use natu- ral language to specify, as clearly as possible, what property we are trying to modify. We can understand whether our in- structions are having the intended effect by qualitatively ex-
ATT = E (R(X,Y()) - R(X, (O)) = 1] ATU = E [ - R(X, Y (O)) = 0]
Intuitively, if W = is a helpful response, the ATT mea- sures the change in reward when we take a helpful response and make it unhelpful, and the ATU measures the change
3

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Original (W =
Rewrite (W 1)
think the biggest disappointment in this film was that, right until the end, expected the acting instructors of the cast to break in and apologize for how poor the acting was.
The most delightful surprise in this film was that, right until the end, I was amazed at how the acting instructors of the cast could have crafted such unique performances.
I am kind person, so gave this movie instead of 1. It was without doubt the worst movie...
I am a kind person, so gave this movic a 2 instead of 1. It without doubt the best movie...
This movie is ridiculous. Anyone saying the acting is great and casting is superb have never...
This movie is amazing. Anyone saying the acting is terrible and the casting uninspired have never.
Table Recent language models show promise as (imperfect) rewriters, since, qualitatively, they are capable following instructions change target attribute (even if they do not always leave off-target attributes unchanged). For instance, GPT-4o qualitatively does well rewriting IMDB responses change sentiment from negative (W 0) to positive (W 1). However, this does not ensure that GPT-4o will not change other attributes besides sentiment.
amining the rewritten outputs and checking that they indeed change the target attribute. In practice. finding effective rewrite instructions requires iterative cycle of generating rewrites, examining the responses, and adjusting the rewrite prompt to be more clear and specific.
Original
14
Reuritt'
Density
13
Imperfect Rewrites While qualitative checks help con- firm whether the target attribute has changed, they are less effective for detecting tffeet modifications-unnnteen edits to grammar, tone, or other attributes (see Table 2).
-1425
C101
8.05
1.50
1405
039
0:355
3.30
4.75
Reward
For example, in Table (and Appendix E.3), sentiment flips successfully, but other properties may also shift. This issue can be identified quantitatively by comparing reward distributions before and after rewriting (Figure 2). Ideally, if only sentiment changed during rewriting, a second rewrite reversing the attribute should restore the original distribution. Otherwise, the (original, rewrite) pairs are not perfectly counterfactual and cannot directly estimate Equation (I).
Figure 2. Off-target changes from imperfect rewrites affect the re- ward measurement. Ideally, rewrites affected only the target attribute (sentiment). then applying second rewrite to revert the change should restore the original reward distribution. Unfor- tunately, the observed distribution shift indicates that off-target modifications occur during rewriting. Here, original samples (bluc) drawn from the HH-RLHF datasct, and are rewritten twice on sentiment (orange). Rewards are from ArMORM.
Mathematically, rewriting introduces some error e in the observed reward because we cannot perfectly construct the true counterfactual (w),, which differs only on the target attribute:
instead compare the (rewrites, rewrites of rewrites) pairs: R(x' Re(Re(y', 0), - R(x', Re(y', O)),
if ww =
+ = R(x", Re(s(,w)) - Rix', vw))
(2)
8
R(x', Re(y', - R(x* Re(Re(y', 1),
if wf =0
We would like to correct for these errors. Yet, the whole point of the rewrites is to approximate the counterfactuals y² (w), so we cannot directly measure e
The motivation is that the off- changes introduced by the rewrite process will, in expectation, cancel out when we are comparing two things in "rewrite space".. ". For example, the tendency for LLMs to produce well-formatted text will affect both the first rewrite and the rewrite of the rewrite (see Table 2), so the overall contribution of this off-target change will cancel out.
RATE Procedure Our solution to introduce more noise. Instead of comparing rewrite to the original response, we compare it the rewrite of the rewrite, thereby canceling out off- target noise introduced by the rewrite process. That is, rather than selecting (original, rewrite):
In Algorithm 1 we use this idea to define the Rewrite-based Attribute Treatment Estimators (RATE) for the ATT, ATU, and ATE. These estimators are simply the averages of the reward difference between the rewrites and the rewrites of the rewrites.
8' [[(x,, )- R(x', Re), if w =
R(xt, Re(u", 1)) - R(x y²), w =

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Algorithm RATE: Rewrite-based Attribute Treatment Estimators
1: Input: Dataset {(x', y², w²)), reward model R. function Re()
2: Return: Estimates RATE, ATURATE. RATE, ATERATE
3: Initialize n1 f -i., 1w 11, no + Ens 1ww* = 0]
4: ATTRATE + Σ [M(x*, Re(Re(y' 0), 1)) R(x', Re(w', R0)]
d:1===1
5: ATURAIE & 1 Σ [[ Re(v', 1)) - R(x', Re(Re(y) 1),0))
=0
6: ATÉRATE & no+n:
n1
RATRA
Bo+ ATURATE
no
7: Return: RATE. RAURA ATERATE
4. Cancellation of Multiple Rewrite Errors
Assumption 2: Additivity of Reward with Respect to Rewrite Errors Suppose that the reward function can be decomposed additively into two components:
We now turn to the validity of the RATE procedure. Intu- itively, the idea is that, as long as the errors introduced by the rewrite independent of the target attribute W, then these errors will cancel out when we compute the average. A particularly important special case of independence is when the LLM always introduces the same eπror, irrespective the rewrite direction. For example, we observe that GPT-4o rewrites nearly always correct typos, independent of the attribute being rewritten. In this case, the single-rewrite estimator is biased because it includes the effect of the typo correction. RATE, however, is not, because it compares typo-corrected responses to typo-corrected responses. We now formalize these intuitions.
R(X, = Rw. (X, Z) + (x())E)
where:
1. Rw.z(X, W, Z) is the component of the reward that depends on the target attribute W and the immutable off-target attributes
Rg (X, E) is the component of the reward that depends on the mutable off-target attributes 5.
Intuitively, this assumption is saying that, at the level of the reward model, there aren't It any interactions between the component corresponding to potential rewrite-erron & and either W or Z.
Latent Variable Model for the Responses We can parti- tion all the possible high-level attributes of a response into three categories: the target attribute W we want to change (e.g. sentiment), off-target attributes Z which are always unaffected by the rewrite (e.g. topic, and language), and off-tarp tar-ta attributes which might end up being affected by the rewrite (e.g. grammatical structure), which we'll denote g. We write the response as Y = E.
The following result establishes validity of RATE under these assumptions:
Theorem 4.1 (Unbiasedness and Consistency of RATE).
Assume R(.,.) is bounded. Take assumptions and 2 above.
Suppose we have a set of prompt-completinn pairs {x', (x',y²} sampled i..dd from some population with P(W =) E (0,1). Then Algorithum 1 yields unbiased and /n-consistent estimators of the ATT, ATU, and ATE.
Assumption 1: Rewrite Errors Don't Noticeably Depend on the Rewrite Direction The off-target changes, ç. in- troduced by the rewrite process are randomly drawn from some distribution, PRo which doesn't depend on W:
See Appendix B for the proof. We emphasize that these are merely sufficient conditions to show that there exist any situations under which RATE is consistent. There's 's no reason a priori to expect that imperfect rewrites can provide causal estimation. Hence, the purpose of the theorem simply show that the approach is not vacuous.
Re(X, ((W W) d Y(1 W, 2) with Ō PR)
Continuing the example where rewriting sentiment (W) corrects typos (E) but leaves language (Z) unchanged, this
5. Experiments
states that GPT-4o isn't 't more or less likely to fix typos positive sentiment responses compared to negative senti- ment. Note that this is, in fact, weaker than assuming that LLM-rewriters tend to repeat the same rewrite errors (e.g. GPT-4o always fixes typos no matter what). If we do ob- serve this behavior, then the assumption is trivially satisfied.
There are two main questions to address empirically:
Does RATE correctly estimate the causal effect of at- tributes on reward models?
the distinction between RATE and the naive estima- tor actually substantive?
5

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite of Rewrite
It wsa great to see smoe of my favorite stasr of 30 years ago including oJha Ritter, Ben Gazarra nad Audrye Hepburn. Tehy looked quite wonderful.
Great see some of my favorite stars of 30 years ago, including John Ritter, Ben Gazarra, and Audrey Hepburn. They looked quite wonderful.
It was great to see some of my favorite stars 30 years ago, including John Ritter, Ben Gazarra, and Audrey Hepburn. They looked quite wonderful.
W Reward: -7.6
W =-0, 00, Reward: -5.6
W = 1, Reward: -5.6
My girlfriend once brought around The Zombie Chronicles for us to watch as a joke. Little did we realize the joke was on her for paying fl for it. While watching this film started to come up with things would rather be doing than watching The Zombie Chronicles.
An ex-girlfriend once brought around The Zombie Chronicles for us to watch as a joke. Little did realize the joke was her for paying for it. While watching this film I started to come up with things I would rather be doing than watching The Zombie Chronicles.
My ex-girlfriend once brought around The Zombie Chronicles for us to watch as a joke. Little did we realize the joke was on her for paying El for it. While watching this film started to come up with things I would rather be doing than watching The Zombie Chronicles.
W = Reward: -5.2
W= 1. Reward: -5.2
W = Reward: -5.1
Table 2. We introduce a correlation between whether the example "starts with a vowel" (W and the percentage of words with typos to test double vs single rewrites. In this example, we have added typos to 30% of words, but only to IMDB reviews that start with a vowel. In row one, we see that the original example that starts with vowel and has typos penalized by the reward model, while the rewrite of the rewrite has typos fixed and its reward score is the same as the rewrite. Examples that do not start with a vowel (W = 0) do not have typos introduced and the reward score is similar for the original, the rewrite, and the rewrite the rewrite. This correlation introduces a spurious positive change in reward when rewriting from "starts with vowel" to "doesn't 't start with a vowel," yielding negatively biased estimator for the causal effect of "starts with a vowel" on reward score. Results as the percent typos is varied are reported Figure 3.
Answering the first question requires knowing ground truth causal effects. To this end, we design semi-synthetic experi- ments with known ground truth. In this setting, we find that RATE is effective at estimating the true effects, while the naive and single-rewrite estimators fail.
additional implementation details and rewrite samples.
5.1. Semi-synthetic Experiments
Correlation between Typos and "Starts with a Vowel" To test the efficacy of RATE against a known ground truth, we design a synthetic experiment where we intro- duce typos into IMDB review examples (Maas et al., 2011) that correlate with whether not the review starts with vowel (see Table 2). We score the examples using Fs rasfarr--ie mearrrxXAA--RRMMM the prompt: "Write a movie review: ". We expect the true ATE of "starts with a vowel" on the reward model to be near zero, Additionally, the true effect should not interact with whether there are typos in the review.
We note that quantitative evaluation, as outlined in Sec- tion 5.1, can be performed to determine whether rewrite errors cancel out during estimation with the double-rewrite estimator. If they do not, sensitivity to correlations between on-target and off-target attributes would be observed.
1 (Dong et al., 2023) with
The second question is whether the correctness of the RATE estimator over the naive estimator actually matters in prac- tice. We find that across a variety of reward models, at- tributes, and datasets, the RATE estimates differ substan- tially from the naive baseline (see Figure 5). Accordingly, the distinction between RATE and the naive estimator is indeed important. As particular example, the RATE es- timator shows that the phenomenon of "length bias" in re- wards may actually be mainly an artifact of bias in naive evaluations.
The naive estimator (Section 2) uses only the original re- sponses; the single-rewrite estimator (Section 3) uses (origi- nal, rewrite) pairs, while the RATE estimator (Algorithm 1) uses (rewrite, rewritten rewrite) pairs.
As seen in Table 2. GPT-4o corrects typos even when not asked to, which, we predict, introduces bias for the single- rewrite estimator. By artificially correlating typos with "starts with vowel" in the dataset, the single-rewrite proce- dure ought to demonstrate a positive bias to the change in reward score when examples are rewritten to start with a con- sonant (since typos harm the reward score). This results in a negatively biased estimate of the treatment effect of "starts with a vowel"-the estimation procedure views "starts with vowel" as harming the reward score of an example.
Setup For all expelimentt experiments, we use OpenAl BatchAPI to generate rewrites of text, instructing the LLM mod- ify the target attribute without changing any other as- pects of the response (see Appendix 1. We use the gpt-4o-202 4-28--8--0 10--00-- 606 model, incurring cost of $1.25 per 1M input tokens and $5.00 per IM output tokens. For instance, generating rewrites and owrrites-f-rreriite for 25K IMDB samples cost roughly $60. See Appendix for
6

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
ATE (Rewzite")
ATE (Rewrite)
1.5
Effect Size
6.70
1.0
-100
-1.20
Nato Faierat
Revelee)
-170
ATE (Singl Rewrite)
10 Percent Typos Review Starting with Vowel
25
0.0
0.5
P'SSntiment(/Length)
Figure 3. In a situation where the ground truth is known, RATE (orange) accurately estimates the ground truth while the naive (blue) and single-rewrite (green) estimators do not. We calculate the treatment effect of "Starts with vowel" 1, when typos have been added with varying frequency to IMDB reviews which start with vowels (see Table 2). Intuitively, we expect the RM to respond negatively the presence typos, but not respond all to whether the movie review starts with a vowel. Hence the treatment effect for "starts with vowel" should remain zero, even as the spurious corre- lation increases. 95% confidence intervals are shown.
Figure 4. Treating a sentiment classifier as "reward model" gives us an approximate ground truth for the effect of length on sentiment classification. We see that the naive estimator (blue) is again highly sensitive to distributional shift, while both the single-rewrite (green) and RATE (orange) estimators correctly report near zero effects. RATE remains invariant to distributional shift, while the single-rewrite estimator reports an increasingly negative effect as the correlation hetween length and positive sentiment increases. 95% confidence intervals are shown.
In Figure 3, we see that this bias does in fact occur, and increases for both the single rewrite method and the naive estimator as we introduce a higher percentage of typos in examples that start with a vowel. On the other hand, the double rewrite method reports a near-zero treatment effect, we increase the percentage of typos in the "starts with vowel" reviews. This demonstrates the necessity of RATE's double-rewrite correction.
samples constant 9374) while increasing the correla- tion between length and positive sentiment (see Table in Appendix C.2). We then compute the naive, single-rewrite, and RATE estimators on each of the resulting datasets.
The naive estimator is highly responsive to spurious correla- tions, while both the single-rewrite estimator and RATE re- port near-zero treatment effects even in the regime strong correlation (Figure 4). Noteably. RATE remains invariant to distributional shift, while the single-rewrite estimator reports increasingly negative effect the correlation between length and positive sentiment increases.
Treatment Effect of Length on a Sentiment Classifier As second test of RATE, we use DissiilBESs senti- ment classifier (Socher et al., 2013; Sanh et al., 2020) as a "reward model" since it has the same structure taking in text and returning scalar. The benefit of using a sentiment classifier is that it should only be sensitive to the sentiment of the text, not other attributes (e.g. length). Intuitively, the average treatment effect of length corresponds to asking "how much do longer responses impact the likelihood that the Disti 1BRR model classifies the review as having positive sentiment?" We expect this to be close to zero (as- suming no strong correlation between length and sentiment in the classifier's training data); regardless, the true ATE should be invariant to distributional shift we increase the correlation between length and positive sentiment.
If we instead replace the sentiment classifer with general reward model, we can perform a similar test for invariance to distributional shift, just without knowing the ground truth effect. In Appendix C.I, we reproduce similar results while varying the correlation between complexity and helpfulness in the HelpSteer dataset (Wang et al., 2023). We find that the naive estimator is highly sensitive to distributional shift, while the RATE estimator remains invariant.
5.2. Real World Reward Models
We select several of the top-performing reward models from RewardBench (Lambert et al., 2024) and evaluate them using both RATE and the naive method across a variety of attributes and datasets: IMDB (Maas et al., 2011), ELI5 (Fan al., 2019), HelpSteer (Wang al., 2023).
We induce this correlation by partitioning the IMDB dataset (Maas et al., 2011) into four categories: long positive, short positive, long negative, and short negative reviews. We then downsample each category, keeping the total number of
7

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
NCSOFT/Liama- htw.alamm----Otteiiii------a
HL.Hloo/AmmRMM -am.--8--mmm---
Nuow
1.0
Iolfort Sian
9.5
comploxity
holpfulness
loageh
watimeent
complexity
belpfalnes
lengch
sontiment
complesity
helpfuluest
length
sentimont
Figure An attribute's reported effect on reward model differs substantially between the naive (blue) estimator compared to the RATE (orange) estimator. Across reward models, the naive estimator yields much larger effect estimates for length, complexity, and helpfulness; smaller effect estimates for sentiment. Rffect sizes are reported as standardized mean differences, using Cohen's 's d compare average treatment effects that are normalized (Faraone, 2008). Bars represent 95% confidence interval.
Figure 5 shows the estimated response of each reward model to each attribute. Of particular interest the eval- 1 (Dong et al.,
ale a benchmark for neural network explainability methods, Abraham et al. (2022) use auman-generated counterfactual restaurant reviews to quantify the causal effect of aspect- level sentiment (c.g., whether the ambiance was described positively or negatively) on the sentence-level sentiment as predicted by a neural network. RATE may be seen as a generalization of these insights, using the double-rewrice to allow scaling the creation of counterfactual pairs.
uations of Fafa 2023) and NCSOFT (Park et al., 2024) with respect to length. NCSOFT was designed to address several purported biases in PPsfarx-LLLMM3--RR
1, including length; how- ever, the RATE estimator suggests that NCSCFT's improve- ment here is less than appears first glance, and may have inadvertently penalized other attributes like complexity.
6. Related Work
Using LLMs to Generate Counterfactuals Wang et al. (2024) survey recent methods for generating counterfactuals. Like us, Gat et al. (2023) use LLMs to generate counter- factuals, but they do not introduce method to account for imperfections in the rewrite process. Similarly, Butcher (2024) ask L.MM generate pains by adding guidance the prompt ('respond in a kind way") but without directly rewriting the completions; hence there is no assurance that the pairs share the same off-targets. Wu et al. (2021) devel- oped Polyjuice, a system for generating diverse counterfac- tuals to evaluate and improve models, but the focus is on training separate model to generate counterfactuals. Fryer et al. (2022) use various metrics to assess the quality of rewrites four dimensions: fluency/consistency, presence of a particular attribute, similarity of label, and similarity of meaning. Our work extends assessments of rewrite qual- ity (through rewrites of rewrites) to correct for bias in the evaluation of reward models, allowing us to account for the quality of rewrites on all dimensions simultaneously. Bhat- tacharjee et al. (2024) also make the observation that frontier models are capable of zero-shot generation of counterfac- tual text examples capable of flipping the label of a text classifier. They use these capabilities to both interpret and create robustness tests for text classifiers, while we rewrite examples on a specific "attribute" and look for a treatment effect of that attribute on reward model score.
Explainability of Reward Models Our work is particu- larly motivated by the challenge of understanding reward model behavior. Jiang al. (2024) perturb text find changes that will flip reward model's prediction of the "preferred" text example when doing pairwise comparisons this example-wist wiiee approach is complementary to RATE. Lambert et al. (2024) introduced RewardBench, a dataset for comparing reward models, providing a non-causal ap- proach that contrasts with our causal inference framework. Casper et al. (2023); Pan et al. (2022); Tien et al. (2023) highlight issues such as misgeneralization, reward hack- ing, and spurious associations in reward models, providing motivation for the evaluation developed here.
There is also a directly relevant carlier line of work on spuri- ousness and interpretability of text classifiers (not necessar- ily viewed as reward models). Joshi et al. (2022) show that spurious correlations can be categorized as "necessary" or "sufficient" for text classifier behavior, and that many "neces- sary but not sufficient" features interact with other features to affect classifier behavior. Feder et al. (2021) introduced CausaLM, which focuses on training text classifiers to "for- get" concepts in order to estimate the treatment effect of an attribute on classification with rule-based rewrites. To cre-
8

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
7. Conclusion and Discussion
perfectly counterfactual. This work is supported by ONR grant N00014-23-1-2591 and Open Philanthropy. David Reber was partially supported by the Long-Term Future Fund. Sean Richardson is supported by a scholarship from Open Philanthropy.
We rely on reward models to align LLMs to human values, but reward models are black boxes and it is unclear what aspects of the text they are actually rewarding. In this work, formalized whether reward model responds to given attribute (e-g., helpfulness, complexity, sensitivity) through the language of causality. Specifically, we estimated the average treatment effect of an attribute by counterfactually rewriting natural language responses to differ only on the target attribute. Although this rewrite process introduces bias, we account for it using rewrites of rewrites, which, in expectation, cancel out off-target changes (see Figure 3). This procedure yields RATE: Rewrite- based Attribute Treat- ment Estimators. Empirically, we find both that RATE is effective at estimating causal effects, and that spurious asso- ciations cause substantial bias in naive estimators applied to real problems of interest.
References
Abraham, E. D., D'Oosterlinck, K., Feder, A., Gat, Y., Geiger, A., Potts, C., Reichart, R., and Wu, Z. Cebab: Estimating the causal effects of real-world concepts on nlp model behavior. Advances in Neural Information Processing Systems, 35:17582-1783 2022.
Bhattacharjee, A., Moraffah, R., Garland, J., and Liu, H. Zero-shot llm-guided counterfactual generation: A case study on nlp model evaluation, 2024. URL https:/ // arxiv org/
/rg/abs/ abs/////////// /405 04793.
Butcher, B. Aligning large language models with counter- factual dpo. 2024. URL URthsttss://axii org/ /abs/ 2401. 09566.
Limitations and Future Directions RATE can only be applied to measure the effect of attributes where rewrite can approximate an imperfect counterfactual. However, the quality of counterfactuals has no ground truth, so the effec- tiveness of the rewrite procedure is ultimately subjective judgment. In practice, we find this not too difficult to judge by simply looking the generated responses. Nevertheless, it would be an interesting direction for future work to at- tempt to more formally validate the rewrite quality, e.g., in the style of Bhattacharjee et al. (2024).
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U.. Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov. D., Chen, X., Langosco, I.., Hase, P., BIyık, E., Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Mened D. Open problems and funda- mental limitations of reinforcement learning from human feedback, 2023. URL https: / /aaraa..vvoogg./ / 0607715217.
RATE only addresses the effect of the attribute on the reward model, without reference to the downstream task that the reward model will be used for. This is both a strength and limitation. The strength is that it allows us understand the reward model in isolation, which is significant since given reward model may be used for many tasks. The limitation is that it is not completely clear how the causal sensitivity the reward model will translate to, e.g., the behavior of a LLM aligned to this reward. Exploring this relationship fur- ther could provide valuable insights into the role of reward models in the alignment process.
Chen, W. and Chu, Z. Causal inference and natural language processing. In Machine Learning for Causal Inference. pp. 189-206. Springer, 2023.
Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, Raft: Reward ranked fine- tuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
Impact Statement
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., and Auli, M. ELI5: long form question answering. In Korhonen, A., Traum, D. R., and Mårquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 3558-3561 Association for Computational Lingutttiiinnnnniiiiiiiiiii 2019. doi: 10. 10011/633//111111131446 URL https:// doi. org 10. 18 153 1tt.:1161///1111113333
Our work is a step in the direction of more interpretable reward models. As reward models are used to align large language models, our work may aid in alignment. In par- ticular, RATE may be used to check whether a particular reward model captures desired preferences and values by simply estimating the causal effect of attributes rewards.
Acknowledgements
Faraone, S. V. Interpreting estimates of treatment effects:
Thanks to Nikolaos Ignatiadis for suggesting rewrites of rewrites as a validation of whether the rewrites are actually
implications for managed care. Pharmacy and Therapeu- tics, 33(12):700, 2008

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Feder, A., Oved, Shalit, U., and Reichart, R. Causalm: Causal model explanation through counterfactual lan- guage models. Computational Linguistics, pp. 1-54,
Y. (eds.), Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pp- 9804-9817, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. doi: 10. 12.18663////00222.mmmmmmmmmi aclanthology wrg.oorgg////ooo 2022. emnlp caannpp-miinn.. main. 666.
May 2021. ISSN 1530-9312. doi: 10.1162/coli.
a_00404. URI. http:/ htp//xxx.ooii org. /0lg//0..10 1166/ coli 000004
URL https://
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Re- ichart, R., Roberts, M. E., Stewart, B. M., Veitch, V., and Yang, D. Causal inference natural language processing: Estimation, prediction, interpretation and beyond, 2022. URL https: / aaaaaiiiiii. org/ abs 2109. 00725.
Kahneman, D. and Tversky. A. Prospect theory: An analysis
decision under risk. In Handbook of the fundamentals of financial decision making: Purt I, pp. 99-127. World Scientific, 2013.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, Y. Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evalu- ating reward models for language modeling, 2024. URL https: : / arwiiavvvooo org/ abs 24403.1778
Fryer, Z., Axelrod, V., Packer, B., Beutel, A., Chen, J., and Webster, K. Flexible text generation for counterfactual fairness probing. 2022. URL https: /ttp:///aavvvvvaa org/ abs/2200 13757.
Maas, A. L., Daly, R. E., Pham, P. T.,, Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sen- timent analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguis- tics: Human Language Technologies, pp. 142-150, Port- land, Oregon, USA, June 2011. Association for Com- putational Linguistics. URL http: htwp://wwwweeeeeeeeeeeoo aclweb. org/ attp/ooohhoooyyoooooo
Gat, Y., Calderon, N., Feder, A., Chapanin, A., Sharma, A., and Reichart, R. Faithful explanations of black-box nip models using lIm-generated counterfactuals, 2023. URL https: / /aaaiiiiivv.//..///aaa///aaaavva org/ abs/2311111 202.0..000003..3.....1 00603.
Grimmer, J., Roberts, M. E., and Stewart, B. M. Text as data: A new framework for machine learning and the social sciences. Princeton University Press, 2022.
Pan, A., Bhatia, K., and Steinhardt, J.
The effects of reward misspecification: Mapping and mitigating mis- aligned models, 2022. URL https: http::///axxvv.o////// abs / 2201. 03544.
Gui, L. and Veitch, V. Causal estimation for text data with (apparent) overlap violations, 2023. URL https:// arxiv. owg.v abs 2210.221111 00079.
Hernán, M. A. Does water kill? call for less casual causal inferences. Annals of epidemiology, 26(10):674-680, 2016.
Park, J., Jwa, S.,, Ren, M. Kim, D., and Choi, S. Offseebias: Leveraging debiased data for tuning evaluators, 2024.
Sanh, V., Debut, L., Chaumond. J., and Wolf, T. Distil- bert, distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL https: / wawaiiivvvv......aa.aaaa www.iv.oor.roooooooooooooo /bs/ 1 1910. 01108.
Imbens, G. W. and Rubin, D. B. Causal inference in statis- tics, social, and biomedical sciences. Cambridge univer- sity press, 2015.
Jingg J., Bewley, T., Mishra, S., Lecue, F,, and Veloso, M. Interpreting language reward models via contrastive ex- planations, 2024. URL https: /ttp/vaaaivv///////////aa//aaaaa org/ abs/ 2411.1655
Socher, R., Perelygin, A., Wu, J., Chuang. J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631- 1642, Seattle, Washington, USA, October 2013. Asso- ciation for Computational Linguistics. URL https:
Jin, Z., Feder, A., and Zhang, K. CausaINLP tutorial: An introduction to causality for natural language process- ing. In El-Beltagy, S. R. and Qiu, X. (eds.), Proceed- ings of the 2022 Conference Natural Language Processing: Tutorial Abstracts, pp- 17-22, Abu Dubai, UAE December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022. emnlp-tutorials... URL httph:htt////... https::/aclaant/aaaa http://aaaaantooog.....mo org/2020 2022.
/ //www. acwweb. org/ http///tthhoooy/11-11101
Empirical Methods in
Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A. D., and Brown, D. S. Causal confusion and reward misidentifi- cation in preference-based reward learning, 2023. URL https: /ttp:s:///xvvvv.ee
arxia. org abs / 2204..22222 06601.
Wachter, S., Mittelstadt, B., and Russell, C. Counterfactual explanations without opening the black box: Automated decisions and the gdpr, 2018. URL https: /ataas///axxii /arxiv. org/abs/177 1111 00399.
Joshi, N., Pan, X., and He, H. Are all spurious fea- tures in natural language alike? an analysis through a causal lens. In Goldberg, Y., Kozareva, Z., and Zhang,
10

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Wang, Y., Qiu, X., Yue, Y., Guo, X., Zeng. Z., Feng, Y., and Shen, Z. A survey natural language counterfactual generation, 2024. URL 1407.03993
org/ abs/
Wang, Z., Dong, Y., Zeng, J., Adams, V., Sreedhar, M. N., Egert, D., Delalleau, O., Scoweroft, J. P., Kant, N., Swope, A., and Kuchaiev, O. Helpsteer: Multi-attribute helpfulness dataset for steerlm, 2023.
Wu, T., Ribeiro, M. T., Heer, J., and Weld, D. S. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models, 2021. URL https: / /arxiv. org/abs/22 /2011 00288.

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
A. Reproducibility Statement
To facilitate reproducibility, we have taken the following measures: (1) Our code implementation, including scripts for producing rewrites, estimating treatment effects, and generating plots, provided at attps://gitthh com/ :oddnief/RATE. (2) The datasets used in our experiments (IMDB, ELI5, HelpSteer, HH RLHF) are publicly available. (3) In Appendix E.3, provide randomly sampled texts, rewrites, and rewrites of rewrites for each dataset/attribute combination, allowing the reader to qualitatively evaluate our rewrites. (4) All reward models evaluated in this study (i.c., FsfairX. Seefair--LLaMa3--2-0 LLMMA3-RM-V 1, NCSOFT/ -llama-3-Offsetee BettcBiiass----- ArmoRM) are open-source. (5) We
report confidence intervals for all main results to ensure statistical reliability, using a normal distribution because of our large sample size. (6) Appendix E.1 includes tips for creating effective rewrite instructions and documents challenges encountered during the rewrite process, aiding in the reproduction of our methodology. (7) For the synthetic experiments, we provide details on how we induced correlations in Appendix C.2.
B. Proofs
Theorem 4.1 (Unbiasedness and Consistency of RATE). Assume R(-,-) is bounded. Take assumptions 1 and 2 above. Suppose we have a set of prompt-completion pairs {a², ,y') sampled i..dd from some population with P(W= =) E (0, 1).
Then Algorithm yields unbiased and /n-connistentt estimators of the ATT, ATU, and ATE.
Proof. First, we'll prove the unbiasedness and vi-connistencyy of ATTRATE: The argument for ATURATE follows by symmetry. Then, we can use these results to prove the same for ATERATE- Throughout, we use & and & to denote i.i.d. samples from the distribution PRe, where the former comes from the first rewrite and the latter from the rewrite of the rewrite.
Unbiasedness and Vn--Cossisencyy of ATT Fix a prompt and response y with w 1, omitting superscripts for convenience. Then by our latent variable model, y Y(1, 2, v) for some realizations and of Z and We calculate:
Re(e(yy 0), 1)) R(x, Re(y, 0))
which has expected value:
E & [R(x, ((1,,,) R(x, v(0 2.E E (w, 1, Re(x, 5)
E [Rw.z(z,0, + R=(x,()
Rw,z(0,,, (w,
-Bwi2(11,,2 - Rw (x, 0,2)
+ Re(xx - -Re(,,
R(z,,(1,2,,)) R(,,,((0, 20 2, 2))
R(z,v(1)) R(z,
Therefore, as average over these quantities, have:
E E[ATTRATE] =
R(X, = ATT
For Vnn-consistency, note that R( is bounded, so its variance is bounded. As the x², yy are i.i.d., so are the R(z Thus, ATTRATE is average over mi i.i.d. random variables with finite variance, implying:
V (ATTRATT ATT) = O,())
Since 31 2. P(W = 1) and P(W = 1) E (0, 1), we have
(1), which implies:
Va(AATRRAT RATEE ATT) (1)
2. Unbiasedness and /ii-Consistency of ATUXRATE By the same argument as for ATT and since P(W 0) E (0,1):
E[ATURATE] E[R(X,Y(11) (0)) W 0) = ATU
12

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
and
Vn(ATURATE ATU) Op(1)
3. Unbiasedness and Vnn-Coniitencyy of RATE The ATE estimator is a weighted average of the ATT and ATU estimators, By the law of total expectation:
E ATTE E[R(X, Y(1) R(X, =] P(W = =1)
+ E[[R(X, Y(1) (O)) = 0] P(W = 0)
EE)((X, R(X, ATE
For vnn-consistency, we can write:
Vn(ATERAIE ATE) 11 n
Vn(ATTRATE ATT) + do n Vn(AATURATE ATU)
Since:
4. P(W =1), TO 3 P(W 0)
V n TMAATRRAT ATT) = O1()), (AAUURRATT - ATU) = Op(1)
By Slutsky's 's theorem:
(ATERATTE - ATE) = Op(1)
13

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
C. Additional Semi-Synthetic Experiment Details
C.1. Additional Semi-Synthetic Experiment: HelpSteer
In third synthetic experiment, we show that the RATE estimator is stable under distributional shift even when the expected treatment effect is non-zero. In this experiment, we are evaluating the effect of helpfulness on the ArmoRM reward model using the HelpSteer dataset (sample size 5148). Each example in HelpSteer includes human ratings five attributes using Likert-5 scale: Helpfulness, Correctness, Coherence, Complexity, and Verbosity. (Wang al., 2023). We expect "helpfulness" to have positive treatment effect on reward scores.
In this synthetic experiment, we introduce correlation between "complexity" and "helpfulness" into the evaluation dataset. While we expect the treatment effect of "helpfulness" to be positive, we should see a constant treatment effect even under distributional shift. In Figure 6, we can see that the naive treatment effect increases as we introduce a spurious correlation between complexity and helpfulness into the evaluation data, while the RATE estimators remain much closer constant. Again, we note that the double-rewrite estimator more stable than the single-rewrite estimator.
1.4
Naive
ATE (Rewrite")
1.2
ATE (Rewrite)
1.0
Effect Size
0.8
0.6
0.1
0.0
0.2
0.1
0.6
0.8 1.0
P(Complexity Helpfulness)
Figure 6. We estimate the treatment effect of concept reward model under distributional shift using three estimators: naive (blue). AT E (Rewrite) (green), and AT (RReriiee (orange). We that the causal estimators remain constant while the naive estimator changes as the distribution shifts. 95% confidence intervals are shown. We estimate the effect of "helpfulness" on ArnoRM scores using the HelpSteer dataset under distribution shift. We expect the "ground truth" effect of "helpfulness" to be positive and remain unchanged as we add spurious correlation between "helpfulness" and "complexity" in the evaluation data. The naive estimator (blue) shows an increasing effect size as increase the correlation between "complexity" and "helpfulness" while both the single- and double-rewrite estimators (green, orange) remain near constant.
C.2. Synthetic Experiment Implementation
Our semi-synthetic experiments took data from real-world dataset (IMDB and HelpSteer) and artificially induced correlation between the target attribute and the off-target attribute. As both the target and off-target attributes are binary, we can easily control the correlation between them. We group the data into the four possible combinations of the target and off-target attributes (e.g., long positive, short positive, long negative, short negative) and then randomly sample from these groups to create a new dataset. We then evaluate the reward model on this new dataset to see how the correlation affects the estimated treatment effect.
14

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
Dataset
Long Positive
Short Positive
Long Negative
Short Negative
P(logg positive)
P negative)
0
2287
2287
2287
2287
0,50
0.50
1
2515
2058
2058
2515
0.55
0,45
2
2744
1829
1829
2744
0,60
0.40
3
2973
1600
1600
2973
0.65
0.35
4
3201
1372
1372
3201
0.70
0.30
5
3430
1143
1143
3430
0.75
0.25
6
3659
914
914
3659
0.80
0.20
7
3888
685
685
3888
0,85
0.15
8
4117
456
456
4117
0,90
0.10
9
4345
228
228
4345
0.95
0.05
10
4574
0
4574
1.00
0.00
Table 3. Adjusted counts and conditional probabilities for the synthetic experiment in Figure 4, after dropping reviews whose original or rewritten text exceeds context length of 512 tokens. Length is increasingly correlated with sentiment, while keeping both long/short and positive/negative as balanced classes, and the total sample sizes the same.
Dataset
Helpful Complex
Unhelpful Complex
Helpful Simple
Unhelpful Simple
P(unhelpful complex)
P(unhelpful simple)
1287
1287
1287
1287
0.50
0.50
1
1416
1158
1158
1416
0.45
0.55
2
1545
1029
1029
1545
0.40
0.60
3
1673
901
901
1673
0.35
0.65
1802
772
1802
0.30
0.70
0.25
6 2060
514
514
2060
0,20
0.80
7
2189
385
385
2189
0.15
0.85
8
2318
256
256
2318
0.10
0.90
9 2446
128
128
2446
0.05
0.95
10
2575
2575
0.00
1.00
Table Adjusted counts and conditional probabilities for the synthetic experiment in Figure 6. Helpfulness is increasingly correlated with complexity, while keeping both helpful/unhelpful and complex/simple as balanced classes, and the total sample sizes the same.
15

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
D. Additional Figures
D.I. Single vs. Double-Rewrich Estimates
There are significant differences between the single-rewrite estimator and the double-rewrite estimator. Each subplot shows the ATE, ATT, and ATU for a different reward model and attribute. Hence, using double rewrites is crucial for estimating the true treatment effect of attribute reward model.
0.30
0.20
-02
Effeet
0.3
0.2
-04
0.10
066
0.06
0.1
0.00
ATE (Rewrite
ATE Rewniee
ATT (Rwritee
ATT Rewnie)
ATU Rewrite
ATU Rewnite)
ATE Rewritee)
ATE (Rewrite)
ATT (Rewritee
ATT Rewrite)
ATU Rewrite")
ATU (Renite)
ATE (Rewrite)
ATE (Rewnite)
ATT Rewrite")
ATT (Rewrite)
ATU (Rewrite
ATU (Rewrite)
Figure Using RATE (rewrites of rewrites) rather than just rewrites changes the estimated treatment effects. Here we compare treatment
effects of complexity, using data from HelpSteer.
MfrxXCTsstaix-LLLMMARRR.R.
1.50
1.25
1.00
Effeet Size
1.00
1.0
0.75
0.75
0.5
0.30
0.30
0.25
0.25
0.00
0.00
ATE (Rewrite")
ATE Rewrite]
ATT Rewrits"))
ATT Rewrite)
ATU Rewrite")
ATU Rewvite)
ATE Rewrite")
ATE (Remrite)
ATT (Rewrite)) )
ATT Rewrite)
ATU Rewrite")
ATÚ Rewrite)
ATE (Rewrite")
ATE: (Rewrite)
ATT Rewrite")
ATT (Rewiite)
ATU (Rwritee))
ATU Rowritt)
Figure Using RATE (rewrites of rewrites) rather than just rewrites changes the estimated treatment effects. Here we compare treatment
effects of helpfulness, using data from HelpSteer.
16

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
0.4
0.3
0.3
0.2
0.1
0.2
0.4
0.0
0.2
0.0
0.0
ATE (Rewrite))
ATE Rewrite)
ATT (Rwnitee)
ATT Reerrtee
ATU Rewrite"
ATU (Renrite)
ATE (Rewrite))
ATE (Rewrite)
ATT (Rowrite))
ATT (Rewrite)
ATU Rownite")
ATU (Rerriee)
ATE (Rewrite')
ATE (Rewiite)
ATT (Rewrite")
ATT (Rewrite)
ATL (Rwritte)
ATU (Rewrite)
Figure 9. Using RATE (rewrites of rewrites) rather than just rewrites changes the estimated treatment effects. Here we compare treatment
effects of length, using data from EL.5 and IMDB.
R.HHRow/AAmoR2--laaa-----00
0.8
0.8
0.6
0.4
Effeet
0.4
0.2
0.0
0.2
0.1
0.2
0.0
ATE (Rewrite)
ATE Rewrite)
ATT Rewrite")
ATT Rewrite)
ATU Rewrite")
ATÚ Rewrite
ATE (Rwrite))
ATE (Rewrite)
ATT Rewrite"
ATT Rewrite)
ATU Rewrite")
ATU Rewrite)
ATE (Rewrite)
ATE (Rewrite)
ATT (Rewrtte
ATT litrlte
ATU Rewrite'
ATU Rewrite
Figure Using RATE (rewrites of rewrites) rather than just rewrites changes estimated treatment effects. Here we compare treatment
effects of sentiment, using data from IMDB and HH-RLHE
17

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
Attribute
W=1
W=0
Rewrite Prompt
Sentiment
Positive Sentiment
Negative Sentiment
Adjust this response it's (W), but change "nothing* clse. Adjust this answer so it's {W), change "nothing* else. If the above answer is phrased as a question not answer it. Just rewrite the question following the same instructions.
Length
Longer
Shorter
Table Example rewrite prompts from experiments with sentiment and length as the target attribute. For the ELIS dataset, some of the responses phrased questions, instructed rewriter (GPT-4o) answer the question and instead rewrite it.
E. Example Rewrites
E.1. Iterating the Rewrite Instructions
One surprising behavior encountered is that, when the example response in need of rewrite phrased question, the LLM would often answer the question rather than rewriting it. Based on this, we included explicit instructions not to answer questions but, rather, to rewrite them for the HH-RLHF dataset (see Table 5),
E.2. Ill-defined Target Attributes
In some cases, the target attribute is not well-defined. For example, in the HH-RLHF dataset, neutral questions have no clear sentiment, and consequently the rewrites add strange syntax: "annoyingly the same size" and "frustratingly square" (see Table 6). We qualitatively check for these cases, and only use datasets where the target attribute consistently well-defined.
Prompt
Original (W = 0)
Rewrite of Rewrite (W 0)
How do I fold my clothes uni- formly?
Are you trying fold clothes that they're 're always the size, or so they're 're perfectly square?
Are you folding clothes so that they't hee''re annoyingly the size, or so they' 'e frustratingly square?
Table some text, our target attribute (W Sentiment) not well-defined. Rewrites strange syntax: "annoyingly the same size" and "frustratingly square". Data from the HH-RLHF dataset.
E.3. Excerpts of Samples
The following tables show randomly sampled original text and rewrites for a given dataset and attribute, with reward scores from ArMoRM. The rewrites of rewrites will have the same W as the original. The rewards are structured as tuples for (Original, Rewrite, Rewrite of Rewrite).
18

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
evolved from the very first first person shooters. back then in the days of wolfenstein and quake... (W = 0)
The control scheme for first- person shooters has quite evolution over the years, originating... 1W=1)
The control scheme for first- person shooters has evolved since the genre's early days with games lik...
(0.11672, 0.15462, 0.14736)
Pros for ssd's: -Smaller form factors available Significantly faster read/write speeds -Very
Pros for SSDs:
Pros for SSDs:
(0.13385, 0.17354, 0.16327)
Smaller form factors avail-
Smaller form factors: SSDs come in smaller sizes than HDDs, ideal for compact devi...
able: Solid State Drives
low th... (W 0)
(SSDs) come variety sma.. (W=1)
Most people have covered the main playing differenees, but don't ' think any have touched on FIELDIN... (W = 1)
Most people have covered the main playing differences, but fow have touched on FIELD- ING compared to (W = 0) Wrapping things in aluminum foil the hot sum will defi- nitely keep them from heating from the sun... (W = 1)
Most people have covered the main playing differences be- tween baseball and cricket, but few have tou...
(0.14019, 0.13259, 0.12511)
Wrapping things in aluminum foil in the hot sun will defi- nitely keep them form heating from the sun... (W 00)
Wrapping items in aluminum foil the sun can keep them from heating up, the foil re- flects the s...
(0.07861, 0.09543, 0.10411)
Take my answer with grain of salt. I'm not scientist. EDIT: There is difference in
Take my answer with grain salt. EDIT: Gravity varies based on distance fro... (W = 0)
Take my answer with grain of salt. I'm not scientist. EDIT: Gravity varies based on distance fro...
(0.07939, 0.07770, 0.08309)
not a scientist.
gravity dep... (W = 1)
came here from Digg when the collapse came. Before that day, Digg had far superior look to it.. (W=1)
came here from Digg when it collapsed. Digg had far superior "Web 2.0" CSS look
I came here from Digg when collapsed, and it quite journey transitioning from one platform
(0.13708, 0.11329, 0.10987)
with rounded but... (W =)
Basically the beginnings of in- dustrialization made commu-
The advent industrialization fundamentally paved the way for the possibility of commu- nism, primar... (W = 1)
Industrialization paved the way for communism by abling minimal labor to pro- duce an abundance of g...
(0.10642, 0.12827, 0.12078)
nism possible because mini- mal labor could pr... (W = 0)
It wouldn i make things better; you would just end up with a hurricane full radioactive dust and (W 1)
Nuking hurricane would only spread radioactive debris without stopping it. Two key points: First, (W=0) 0)
Nuking hurricanc would re- sult in the widespread disper- sal of radioactive debris, and it wouldn't 't e...
(0.13520, 0.13426, 0.13970)
Table EL15, Length
19

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
Open burning means burning outside, in the smoke can easily disperse. Typically, L... (W =
Open burning means burning outside, in area where the smoke easily disperse. Typically, th... (W = 1)
Open burning means burning outside, in the smoke can easily disperse. Unfortunately..
(0.09514, 0.09364, 0.08196)
where
where
Here are few recommenda- tions:
Here are few criticisms:
Here are few praises:
(0.07917, 0.06890, 0.07473)
Kanye West
Kanye West
Kanye West
The Roots
The Roots
The Roots
Outkast
Outkast
Outkast
Jay-Z
Jay-Z
Jay-Z
Nas
Nas
Naa
The (W= wW
The Not...
-... (W
You feel sick, and you're 'r tired.
You're feeling under the weather and bit tired. The symptoms you're experienc- ing fever, dry cough.... (W
You're feeling under the weather and bit tired. The symptoms you're experienc- ing fever, dry cough...
(0.09101, 0.09153, 0.09153)
You have symptoms includ- ing fever, dry cough, fatigue, headache, a... (W 0)
= =)
Here's basic list of what Bachelor's Degree in Criminal Justice and Human Services able to pr... (W = 1)
Here's basic list of what a Bachelor's Degree in Criminal Justice Human Services unable to ... (W = 0)
Here's a basic list of what (0.10677, 0.03869, 0.10896 Bachelor's Degree in Criminal Justice and Human Services prepares you
I'm sorry, I'm not sure un- derstand this. Can you clarify what you mean by "task rab- bit"? (W=0)
Certainly! "Task Rabbit" is service that connects people who need help with various tasks to skill... (W=1)
Certainly! "Task Rabbit" is service that connects people who need help with various tasks to indiv...
(0.07668, 0.10774, 0.09397)
Try some basic relaxation techniques like meditation breathing exercises. Make sure you' re gettin... =W)
It's great try some basic re- laxation techniques like med- itation or breathing exercises. Ensuring (W
It's frustrating to some ba- sic relaxation techniques like meditation or breathing exer- cises. Str...
(0.10144, 0014041, 0.09213)
Here are some suggestions: . The Secret History Donna Tartt
Here are some suggestions:
Here some suggestions: The Secret History by Donna Tartt
(0. 0.07585, 0.10088)
The Secret History by Donna Tartt
The Ruins of Empire by
The Ruins of Empire by
The Ruins of Empire by Chinua A...
Chinua A... 1)
Chinua A... (W=0)
Alright. One great example of
Certainly! Bouillabaisse is wonderful example a seafood soup, a Mediterrancan classic that deli... (W= 1)
Certainly! Bouillabaisse is a disappointing example seafood soup. a Mediterranean classic that
(0.10048, 0. 10231, 0.05058)
seafood soup is the bouill- abaisse, a Mediterranean clas- sic. a (W= -W=0]
Potatoes, tomatoes, greens, herbs, eggplant, and okra are popular choices. (W =W=1)
Potatoes, tomatoes, greens, herbs. eggplant, and okra are unpopular choices. (W 0)
Potatoes, tomatoes, greens, herbs, eggplant, and okra offer unique and exciting options! cigarette is the equivalent suffering from about cigarette a day.
(0.10898, 0.08953, 0.10335)
cigarette the equivalent to about 1 cigarette day (W= 0)
cigarette is the equivalent enjoying about 1 cigarette day. (W = 1)
(0.04772, 0.04935, 0.05235)
Table 10. HH-RLHF, Sentiment
20

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
Denie Witherspoon) has always been very close with her older sister Mau- reen(Emily Warfield) unt...
Dani (Reese Witherspoon) has always been very close with her older sister Maureen (Emily Warfield) u... (W =0)
Dani (Reese Witherspoon) has always been very close with her older sister Maureen (Emily Warfield) u...
(0.10178, 0.09484, 0.10783)
(W= 1)
wasn't quite sure if this
wasn't I quite if this just going be another
I curious to if this was going to another one of those intriguing nighttime soap operas t...
(0.08255, 0.06745, 0.08678)
just going to he another one of those idiotic nightrime soap operas (W =
those idiotic nighttime soap
operas (W =0)
am kind person, so gave
a kind person, I gave
I am kind person, gave this movie 2 instead of was without a doubt the worst movie
(0.08756, 0.07847, 0.08434)
this movie instead of
this movie instead of
was without a doubt the worst movie (W = 0)
It was without doubt the best movie L... (W = 1)
This movic is another
This movic fascinating ad- dition to my List Movies Appreciate. watched 40 years ago a... (W = 1)
This movie is frustrating ad- dition to my List of Movies Critique. I watched it 40 years ago as
(0.08952, 0.09523, 0.08503)
my List Movies Not To Bother With. Saw 40 years ago as an adolese... (W 0)
The line, of course, from Lord's Prayer- "Thy Will be done on Earth as it is in
The line, of course, from the Lord's Prayer- "Thy Will done on Earth as is Heaven". Swe... (W = 0)
The line, of course, is from Lord's Prayer- - "Thy Will be done on Earth as in Heaven". Swe...
(0.09660, 0.08479, 0.10198)
Heaven". Swe... (W = 1)
notice the DVD version seems have missing or lines between posting of the FRF and th... (W 1)
notice the DVD version seems have missing scenes or lines between the posting the FRF and th... (W = 0)
I notice the DVD version seems have unique flow hetween the posting of the FRF and the launch.
(0.03637, 0.04333, 0.03519)
This movie is ridiculous. Any- one saying the acting great
This movie is amazing. Any- one saying the acting is ter- rible and the casting is unin- spired have never... (W = 1) Soylent Green is classic. have been dreading somcone re-doing it. They seem to be ruining sci-fi... (W 0)
This movie is terrible. Any- one saying the acting amaz- ing and the casting is inspired have never s...
(0.07594, 0.08516, 0.06888)
and the casting is superb have never sec... (W = -)
Soylent Green is classic. have been waiting for some- one to re-do They seem to be remaking sci... (W = !)
Soylent Green classic. have been eagerly anticipat- ing someone re-doing it. They seem to be re...
(0.08788, 0.09034, 0.08798)
Table 11, IMDB, Sentiment
21

---

RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
You can separate an egg white from yolk in many ways. 1. Crack the egg on hard face, making s... (W = 0)
You can separate an white from a yolk in numerous meth- ods. Gently crack the egg on firm s... (W 1)
You can separate an egg white from yolk in many ways. Crack the egg on firm sur- face, breaki...
(0.09198, 0.11512, 0.09110)
1. In the current study, River and colleagues were the first focus attachment security and its... (W=
1. River and colleagues were the first to study attachment security and its connection parenting... ... (W =)
1. River and colleagues pio- neered the investigation at- tachment security and its asso- ciation with
(0.14933, 0.14648, 0.16560)
The intended audience is peo-
D' Artagnan, venerated pur- veyor of fine foods, announces delightful array of new prod- uct offering... (W= =
D Artagnan, a respected sup- plier of fine foods, announces range of new products and exciting promo...
(0.08414, 0.06389, 0.06234)
ple who are interested in learn-
ing about new product offer- ings and promo... (W = 0)
am sorry to hear that you are struggling with your grief. It
am truly sorry to learn about the profound grief you are periencing. Navigating life without you... (W 1)
m really sorry to hear about the deep sadness you'l 'r going through. Life without your mom must be ...
(0.09203, 0.09705, 0.10380)
must be difficult to go through this ... (W 0)
Tontowi Ahmad 12 Lesti Ke- jora 10 Adhisty Zara 7 Al
Tontowi Ahmad 12 Lesti Ke- jora 10 Adhisty Zara 7 Al Ghazali Dewi Persik Nabila Syakieb Rio Dewa...
Tontowi Ahmad 12 Lesti Ke- jora 10 Adhisty Zara 7 Al Ghazali Dewi Persik Nabila Syakieb...
(0.08389, 0,08424, 0.08341)
Ghazali 6 Dewi Persik
Nabila Syakieb Rio Dewa...
(W (W=0) =)
(W 1)
Guilt: stone in my stomach,
Guilt: anchor in my stom- ach's depths, inescapable encumbrance, drags me into its abyss... (W 1)
Guilt: heavy feeling in my stomach, weight I can't cape. pulls me down, mak- ing it har...
(0.16336, 0.17933, 0.15570)
burden I cannot escape.
drags me down, choking breath from my... (W= 0)
Hello there, Donna and Char- lie Sparrow here, ready bring you the news and gos- sip from the wor.. (W = 0)
Greetings and salutations! Donna and Charlic Sparrow here, ready to serve up all the scintillating n... (W 1)
Hello! Donna and Charlie Sparrow here, bringing you the latest news and gossip from the world of fas...
(0.10432, 0.13756, 0.10592)
Tirofiban is small molecule that reversibly inhibits the binding of adenosine diphos- =
Tirofiban is small molecule that stops adenosine diphos- phate (ADP) from attaching to its platelet ... (W ===00
Tirofiban is low molecu- lar weight compound that in- hibits the binding of adenosine diphosphate (ADP..
(0.16087, 0.16883, 0.15925)
phate (ADP) to... =W=1)
Table 12. Helpsteer, Sentiment
22

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
The PagerDuty platform is real-time operations manage- system that combines digital signals fro... (W I)
PagerDuty is system for han- dling digital operations. It mixes signals from software with human res... (W = 0)
PagerDuty is system for han- dling digital operations. It in-
7,15147,0.124999 0.13382)
tegrates signals from software with huma...
Gold on Friday posted its sec- ond consecutive weekly gain, even advance inflation- adjusted... (W 1)
Gold's weekly gain isn't impressive given rising bond yields.
Gold's weekly gain may ap- pear modest in the context of rising bond yields.
(0.15748, 0.12548, 0.14206)
Bullion hovering near US$1,835 an... (W=0)
Bullion's position n...
Here is list format summary of the top big action steps and top 3 little action steps from the C... (W 1)
Define "IO" marriage: Cre- ate picture of ideal mar- riage based on biblical stan- dards.
Define "10" marriage: A "10" marriage is one that aligns with biblical principles, characterized...
(0.11781, 0.10532, 0.11470)
Set =W
Jesus talked to woman at well in a city called Sychar. The woman thought he was prophet and sa... (W = 1) Horse racing (W
Jesus talked woman a well city called Sychar. The woman thought he was a
Jesus talked to woman (0.15391, 0.15391, 0.15311) well in a city called Sychar. The woman thought he was prophet and
prophet and sa... (W =)
Horse racing is competitive equestrian sport where horses and jockeys compete to finish set cour... ()
Horse racing an exciting and competitive equestrian sport where horses and jock- eys work together...
(0.08179, 0.04974, 0.04630)
VVMs have protected
VVMs have successfully pro- tected more than billion peo- ple worldwide from infectious diseases since... (W I)
VVMs have been around since 1996.
(0.07681, 0.07973, 0.04489
billion people worldwide from
infectious diseases since their introductio... (W 0)
British Columbia has promised to stop changing the clocks twice a year, but as of 2021, it still has... (W 1)
The government said they'd d stop changing clocks but haven't. They did survey; most people want it (W
Thank you for sharing your thoughts on this matter. We understand the ongoing con- cern about clock ch...
(0.15622, 0.11233, 0.08685)
0)
The main focus of the conver- sation is on the treatment op- tions for anxiety, specifically medication. (W=1)
There are pills and talking. (w
Certainly! Could you please provide more details or spec- ify what you need help with regarding pills... ...
(0.16432, 0.04699, 0.03975)
= 0)
Table 13. Helpsteer, Helpfulness
23

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
Najma offered to take Stefanie home, and the way. they had conversation about their interests.... ... (W =0)
During the journey home, Na- jma engaged in fascinat- ing conversation with Stefanie, which revealed a... (W 1) Urbus Orbis, coffeehouse nestled in the vibrant clave of Wicker Park, Chicago, played a pivotal (W 1) The King Salman Energy Park. also known by its acronym SPARK, represents monu- mental initiative wit... (W =
On the way home, Najma had interesting conversa- tion with Stefanie, where they found both shared a...
(0.09874, 0.14417, 0.12706)
Urbus Orbis, a coffechouse in Wicker Park. Chicago, played significant role in the com-
Urbus Orbis, a coffechouse in Wicker Park, Chicago, was very important the commu- nity as cultura...
00.16771.018844 0.15969)
munity as (W = 0)
[King Salman Energy Park]:
The King Salman Energy Park, known SPARK, is major project with an expected eco- nomic impact of
(0.06741, 0.01211, 0.01855)
ppotential S65 billion| (W =0)
1)
The Indianapolis Colts have
The Indianapolis Colts have bestowed philanthropic gift upon the Riley Children's Foundation with (W =W=1)
The Indianapolis Colts have given donation the Riley Children's 's Foundation to help improve acce...
(0.13764, 0.13735, 0.14136)
made gift to the Riley Chil-
dren's Foundation in order to
improve acces... (W (W=0) 0)
During the tailbud stage. which occurs around
During the tailbud stage, which manifests around the fourth week gestation, the = 1)
During the tailbud stage, which happens around the fourth week development, the embryonic tail st...
(0.12502, 0.13608, 0.13253)
fourth week of gestation, embryonic tail begin... (W=
embryonic tail co... (W=1)
0)
Author: Stephen Burgen Date: March 15, 2023 Quick Sum- mary: The article discusses the decision to all... (W 0) If you are looking for fun and exciting way to spend your free time, look no further than online g... (W 0)
Authored by Stephen Burgen and dated March 15, 2023, the article delves into the historic decision p... (W 1)
Stephen Burgen' article from March 15, 2023, discusses the historic decision to allow girls and wom...
(0.17218, O. 0.14074,00 0.13447)
If you contemplating a dy- namic and exhilarating avenue
If you're re looking for an citing way to spend your free time, try online gambling. It's fun way t...
(0.08506, 0.08489, 0.07877)
to occupy your leisure time, di- rect your (W [==]
Chappel Dam was built in 1912 by Consumers Energy to generate electricity. In 1964,
Chappel Dam, initially con- structed in 1912 by Con- sumers Energy for the purpose of electricity genera... (W = 1) myriad forms and multifaceted expressions, constitutes a profound and in- tricate explor... (W
Chappel Dam was built in 1912 by Consumers Energy to
(0.14205, 0.15672, 0.13703)
generate electricity. In 1964, Gladwin County b...
Gladwin County p... (W=
Drama (W=
Drama,
various forms and expressions, explores the human experience deeply, looking into emot...
Drama,
(0.09830, 0.06760, 0.08344)
Kamures Kadın was bom in 1855 and married the Ottoman prince Reşad in 1872. After the hirth of her ... (W = 0)
Kamures Kadın, born in the year 1855, entered into matri- mony with the Ottoman prince Reçad in 1872. (W ==1l 1)
Kamures Kadın was born in 1855 and married Ottoman prince Reyad in 1872. Her life hecame intertwined...
(0.17158, 0.19365, 0.18252)
Table 14. Helpsteer, Complexity
24

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
Original
Rewrite
Rewrite Rewrite
Reward
To heighten the drama of this sudsy maternity ward story, it's set in a special ward for "difficult ... (W = 0)
In order to heighten the drama of this sudsy maternity ward story, it's set in a special ward for "d.. (W =1) 1)
To heighten the drama of this sudsy maternity ward story. it's set in special ward for "difficult... ...
(0.09173, 0.09261, 0.09173)
In the tilte I write that the story is ludicrous. below I'll elabo- rate and tell you why it, in my hu... (W =
In the title, I write that the story is ludicrous. Below I'll claborate and tell you why it is, m... (W =
In the title. I write that the story is ludicrous. Below I'll elaborate and tell you why, in my humb..
(0.05162, 0.05196, 0.05237)
Boy what dud this mess Was.But it only lasts hour and I only paid a buck for so live..... (W= = 0)
Oh boy, what a dud this mess was. But it only lasts an hour and I only paid a buck for it, so I'll I... (W 1)
Boy, what a dud this mess was. But it only lasts an hour and only paid a buck for it, I'll live...
(0.08325, 0.08112, 0.08304)
This film is not devoid of charm and also shows bit of warmth, but ultimately this cf- fort 10o v... (W 0)
Ultimately, this film is not de- void of charm and also shows bit of warmth, but this effort is too (W 1)
This film is ultimately not de- void of charm and also shows a bit of warmth, but this effort too v...
(0.09077, 0.09345, 0.09123)
Probably one of the most bori- est slasher movies ever, badly acted and badly written.;br 12:br/ //THE ... 0)
Arguably one of the most bor- ing slasher movies ever, badly acted and badly written.jbr fgjbr THE P... (W=1) =)
This is arguably one of the most boring slasher movies ever, badly acted and badly written,jbr /j;h....
(0.06938, 0.06899, 0.07046)
"Back of Beyond" takes place at dive diner/gas station in the middle of the Australian desert run (W= = 0)
"In the middle of the Aus- tralian desert, "Back of Be- yond" takes place diner/gas station ru... (W = 1) A movie similar to Darkness Falls and The Boogeyman (2005), it's also much more graphic than both, a.. (W =
Set in the middle of the Aus- tralian desert. "Back of Be- yond" takes place at dive diner/gas station...
(0.09929, 00530, 0.09823)
dive
this movie is similar to Dark- Falls.and The Boogey- man(2005) jbut it's also much more graphic than ... (W=0)
This movie is similar to Dark- ness Falls and The Boogey- man (2005), it's also much more graphic than b...
(0.08108, 0.09579, 0.09535)
1)
Up until the last 20 minutes, aws thinking that this is possi- bly Jackic Chan's 's worst movic (exclud... (W= I)
Until the last 20 minutes, was thinking that this is possibly Jackic Chan's 's worst movic (ex- cluding... (W =)
Until the last 20 minutes, was thinking that this is possibly Jackie Chan's 's worst movie (ex- cluding...
(0.07892, 0.101 0.10998)
Who in their right mind does anything so stupid as this mo /gjbr l Accidental killing of a se... (W= 0)
In their right mind, who does anything so stupid as this movie?;br //rbr /cccceetttal killing of a s... (W 1)
In their right mind, who would do anything so stupid this movie?
(0.07758, 0.07366, 0.06883)
Accidental killing of secu- rit...
Imagine that you aer asked by yoru date what movie you wanted to ese, and you ermem- her seeing rath... (W = 1)
Picture being asked by your date what movie you wanted to see, and you remember see- ing rather intr... (W = 0)
Imagine being asked by your date what movie you wanted to see, and you rememher see- ing rather intr...
(0.06417, 0.09607, 0.09502)
Table 15. IMDB, Vowels
25

---

RATE: Causal Explainabilityy of Reward Models with Imperfect Counterfactuals
E.4. Full Samples
Since much of the text in the tables above had to be truncated, here we provide complete examples of rewrites for each of the attributes intervened upon. We sample randomly.
From the (ELI5, Length) experiment:
Reward Question: Why does shaking our arms/legs after stretching make them feel better
Original (W = 0): We have a natural tendency to shake our limbs after a really good stretch, which is known as [stretch reflex]( URL.0). When you stretch large muscle, a special group of nerves is stimulated whose purpose to resist the stretch by opposing muscles to contract. When you shake your limbs you are easing that stress caused by the push-pull action between the opposing muscle groups and that's why you feel better.
Rewrite (W = 1): We have a natural tendency to shake our limbs after really good stretch, which is known as the stretch reflex (URL.0). This phenomenon occurs when you stretch large muscle, and a special group of nerves is stimulated. The purpose of these nerves is to resist the stretch by causing the opposing muscles to contract. This push-pull action between opposing musele groups can create a certain level of stress or tension within the muscles. When you shake your limbs, you are engaging in an action that helps to ease that stress caused by the push-pull interaction between the opposing muscle groups. This is why you tend to feel better after shaking your limbs following good stretch.
We naturally shake our limbs after good stretch due to the stretch reflex. This occurs when large muscle is stretched, stimulating nerves that resist the stretch by contracting opposing muscles. This push-pull action can create tension within the muscles. Shaking your limbs helps case the stress caused by this interaction, making you feel better.
Rewrite of Rewrite:
Rewards (Original, Rewrite, Rewrite of Rewrite): (0.11335, 0. 12670, 0.13770)
From the (IMDB, Sentiment) experiment:
Reward Question: Write movie review:
Original (W 1): "Such a Long Journey" is well crafted film, good shoot, and showcase for some good performances. However, the story is such a jumble of subplots and peculiar characters that it becomes a sort of Jack of all plots and master of none. Also, Western audiences will likely find the esoterics of the rather obscure Parsee culture a little much to get their arms around 1.7 hours. Recommended for those with an interest India.
Rewrite (W 0): "Such Long Journey" a poorly crafted film, bad shoot, and a showcase for some poor performances. Moreover, the story is such jumble of subplots and peculiar characters that it becomes sort of Jack of all plots and master of none. Additionally, Western audiences will likely find the esoterics of the rather obscure Parsee culture little too much to get their arms around in 1.7 hours. Not recommended for those with an interest in India.
"Such a Long Journey" is a beautifully crafted film, a fantastic shoot, and a showcase for some outstanding performances. Moreover, the story a rich tapestry of subplots and intriguing characters that it becomes a masterful blend of narratives. Additionally, Western audiences will likely find the esoterics of the unique Parsee culture an enriching experience explore in 1.7 hours. Highly recommended for those with an interest in India.
Rewrite of Rewrite:
Rewards (Original, Rewrite, Rewrite of Rewrite): (0.1 0.08983, 0.
From the (Helpsteer, Helpfulness) experiment:
Reward Question: What are some examples of songs that were hits during the early 2000s? I want your answers in an enumerated format (1., 2., etc.).
Original (W 1): During the early 2000s, several songs gained popularity and became hits. Here are five examples:
1. "Yeah!" by Usher: This song, which features Lil Jon and Ludacris, reached number one on the Billboard Hot 100 chart in 2004.
2. "Low" by Flo Rida: This song, which features Tiesto, reached number one on the Billboard Hot 100 chart in 2008.
"Promiscuous Girl" by Nelly Furtado: This song reached number one on the Billboard Hot 100 chart in 2006.
4. "Pon de Replay" by Rihanna: This song reached number 12 on the Billboard Hot 100 chart in 2005.
26

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
5. "Irreplaceable" by Beyonce: This song reached number one on the Billboard Hot 100 chart in 2006.
Rewrite (W = 0): Here are some songs from the 2000s.
Rewrite of Rewrite: course! Here are some songs from the 2000s that you might enjoy exploring:
Rewards (Original, Rewrite, Rewrite of Rewrite): (0.10166, 0.05869, 0.06788)
From the (Helpsteer, Complexity) experiment:
Reward Question: Consider this reference information delimited in """; 1"9 Separation anxiety occurs in adults as well, and can be equally serious problem, so much so that psychologists have considered adding it to the diagnostic manual. Many adults suffer from separation anxiety issues and either may not be aware, or fail to seek treatment. Thus, separation anxiety in adults is likely underreported, and much more widespread problem than originally believed to be.
Separation anxiety, when extreme, is usually pretty easy to identify. Adults who have serious separation anxiety tend to have an unhealthy attachment to a person with whom they are close to, and experience intense anxiety and panic when having to separate from that person. Yet, in less severe cases of separation anxiety, the signs may not be as obvious. Some less apparent signs of ASA are:
Extreme Jealousy adults with separation anxiety may demonstrate signs of jealousy in relationships. A fear of abandonment is often what drives those with ASA to experience jealousy. This is especially true if the jealousy is accompanied by anxious thoughts, such as a fear of being alone or irrational concerns about infidelity. Of course, jealousy may be completely unrelated to ASA- for example, control of others is the cause of jealousy, as are trust issues but some forms of deep jealousy may also be due to separation anxiety. Over Striet Parenting there is some evidence that extremely strict and demanding parents may have separation anxiety issues as well. Sometimes referred to as reverse-separation anxiety, the parents may be so concerned that their child will leave them someday that they try control the child's life as much as possible. Stuck in Relationships another way separation anxiety may manifest itself is in the way adults treat their relationships. Whether romantic, familial, or friend relationships, but also friendships and occasionally familial relationships, many with ASA work to maintain the relationship even when extremely unhealthy (emotionally, physically), out of fear of being alone. Mooching- those that "mooch" " off their parents well into adulthood, or those that never seem to leave their friends' homes may be experiencing separation anxiety in some way. Because there are currently no diagnostic criteria for ASA, the discussion of ways it manifests is purely speculative. Regardless, adult separation anxiety is a genuine problem, and affects the lives of countless adults.
Does the author use a biased tone of voice in the provided passage? Please explain your answer in at least one paragraph of 100 words or more in length. I want to make sure that the tone is objective and unprejudiced before presenting it to my collaborative group.
Original (W = 0): No. the author does not use a biased tone of voice in the provided passage. The author presents information about separation anxiety in adults in a neutral and objective manner. The language used is factual and does not contain any personal opinions or judgments. The passage also does not show any favoritism towards or prejudice against any particular group of people who may suffer from separation anxiety. Overall, the tone of the passage is consistent with the purpose of providing information about psychological condition rather than advocating for or against any particular viewpoint.
Rewrite =W = 1): Certainly, the author refrains from employing a biased tone in the passage provided. The author meticulously conveys information regarding separation anxiety in adults with a neutral and objective disposition. The language employed is factual, devoid of personal opinions or subjective judgments. Moreover, the passage does not exhibit any inclination towards or bias against any specific demographic that may experience separation anxiety. In essence, the tone of the passage aligns seamlessly with the objective of disseminating information about a psychological condition, rather than endorsing or opposing any particular perspective.
The author avoids using a biased tone in the passage. They clearly present information about separation anxicty in adults in a neutral and objective way. The language is factual and free of personal opinions or subjective judgments. The passage does not show any bias for or against any group that might experience separation anxiety. Overall, the tone fits well with the goal of sharing information about a psychological condition, rather than supporting or opposing any specific viewpoint.
Rewrite of Rewrite:
Rewards (Original, Rewrite, Rewrite of Rewrite): (0.15593, 0.16245, 0.15361)
27

---

RATE; Causal Explainability of Reward Models with Imperfect Counterfactuals
From the (IMDB, Vowels) Experiment:
Reward Question: Write movie review:
Original (W = 1): I woudl have enjoyed this moive slightly more had ont been for Jsaon (Herb) Evers constant harpnig on expermient. aMny ealry reviewers of The Sevne Samruai accused Toshiro Mifune of overacting. Yet, as more adn more critics viewed that film they saw as being puprosefully done. Jason vEers is obviously not Toshiro Miufne, and his overacting is exactly th /2ibr MoMt of teh actors in this B classic were rather ogod actors, minus Evers and the shwogirls. If you watch this movei, you wolud have noticed Evers shouting almost every Inie, taht is until he smoking and lbowing teh smoke coolly out hsi onse. ;br /r The special effects were par fro the course in movie such as htis oen. In hindsight, there isn't t much that stadns out in my mind as fnatastically good or bad for htis movie.
Rewrite (W= 0): Had it not been for Jason (Herb) Evers' constant harping on experiment, I would have enjoyed this movie slightly more. aMny ealry reviewers of The Sevne Samruai accused Toshiro Mifune of overacting. Yet, as more adn more critics viewed that film they saw it as being puprosefully done. Jason vEers is obviously not Toshiro Miufne, and his overacting is exactly that.;br /2ibr Most of teh actors in this B classic were rather ogod actors, minus Evers and the shwogirls. If you watch this movei, you wolud have noticed Evers shouting almost every Inie, taht is until he is smoking and Ibowing teh smoke coolly out hsi onse. jbr // l The special effects were par fro the course in B movie such as htis oen. In hindsight, there isn't 't much that stadns out in my mind as fnatastically good or bad for htis movie.
If it had not been for Jason (Herb) Evers' constant harping on experiment, I would have enjoyed this movie slightly more. aMny ealry reviewers of The Sevne Samruai accused Toshiro Mifune of overacting. Yet, as more adn more critics viewed that film they saw it as being puprosefully done. Jason vEers is obviously not Toshiro Miufne, and his overacting is exactly that.;br /2ibr Most of teh actors in this B classic were rather ogod actors, minus Evers and the shwogirls. If you watch this movei, you wolud have noticed Evers shouting almost every Inie, taht is until he is smoking and Ibowing teh smoke coolly out hsi onse. jbr / / The special effects were par fro the course in a B movie such as htis oen. In hindsight, there isn't 't much that stadns out in my mind as fnatastically good or bad for htis movie.
Rewrite of Rewrite:
Rewards (Original, Rewrite, Rewrite of Rewrite): (0.05196, 0.06177, 0.06195)
28


Reviews:
[{"review_id": "KIEez88H3T", "reviewer": "Reviewer_bpXy", "summary": "The paper introduces RATE, a framework for understanding reward models in the context of LLMs. The core idea is to do rewrites of rewrites, so that confounding factors (such as typos and sentence length etc.) can be filtered out. Some experiments are done showing the method appears better than the alternative of doing a single rewrite counterfactual.\n\n# After rebuttal\nMany thanks to the authors for their direct and professional rebuttal, and thanks for clarifying some misunderstandings. (the new Figure 1 looks improved!)\n\n* Can you clarify what you mean by “significant” regarding the difference in distributions in figure 2? “Significant” as in hypothesis testing or as in “meaningful”? -- I meant in terms of \"meaningful\" really, but this is just an aside.\n\nMy main issue with the paper is the lack of demonstration in application. I understand this is a tall order for any XAI paper, but what I really want to see is that this method is better than e.g. Jiang et al. 2024' on the task of e.g. model improvement in a fair comparison, in such a situation I would accept with a 4/5 score. So I will keep my current recommendation, thank you.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We agree that the behavior of reward models is an important and understudied area of alignment research. \n\nFirst, we will respond to some higher level points:\n- Regarding your concern about the causal claims, we think there may be a confusion here. We are not defining our counterfactual examples as examples that result in a changed classification decision, as in Jiang, et al. 2024. Rather, we are creating counterfactual examples with our rewrite process that differ on one binary attribute (and rewrite noise). That is, our counterfactuals are of the form “what would the text of this response be if it were generated with binary latent variable W set to 1 rather than 0.” We then use these counterfactual examples to estimate an ATE on the reward score itself. That is, RATE does not remove attributes, but rather changes the value of attributes (e.g., swapping sentiment from positive to negative) in order to understand how the reward model’s output is affected by the attribute. So, our treatment effect is in “units of reward score” regardless of any classification decision.  We agree that we should mention the Jiang work earlier and more clearly distinguish our notion of counterfactual from the notion of counterfactual explanations addressed in Jiang. We have updated the manuscript to clarify this in the introduction.\n- We agree that the ultimate goal of RATE is model improvement. However, the current alignment pipeline involves several different components. If an aligned model is producing undesirable behavior (e.g. excessively long responses, annoying bullet pointed lists, etc.) it’s not obvious where in the pipeline this behavior is coming from. Is this from the data used or generated during alignment? The reward model? An unintended consequence of the alignment objective? We focus on a modular evaluation of reward models themselves in order to localize failure points in the alignment procedure, and we plan to examine the impact of reward model bias on aligned models in future work.\n\nHere are responses to some other comments and questions:\n- Thank you for suggesting improving figure 1. We agree that this leaves something to be desired. Edit: Here is a link to the updated figure https://postimg.cc/9wwXZ1bQ\n- Thanks for catching the typo on line 76. This is fixed on the updated version of the paper.\n- Can you clarify what you mean by “significant” regarding the difference in distributions in figure 2? “Significant” as in hypothesis testing or as in “meaningful”?\n- It’s not possible to *guarantee* that the rewrite procedure produces sensible text, but examination of (many) randomly selected examples shows the procedure is robust. When applying the method, it is, as always, important for practitioners to spot-check the data.\n- Regarding your question about Assumption 1, note that Assumption 1 is actually much weaker than the $\\xi$ terms cancelling out for each unit: rather, it suffices for the cancellations to occur in expectation, which is very permissive. This is easy to check via visual inspection on a random subset.", "claims_and_evidence": "I don't think there are any hugely problematic claims in the paper. I would argue that the abstract claim that RATE measure the \"causal\" effect of an attribute is not true however. Just because we change attribute x and the reward flips, that doesn't necessarily mean it \"caused\" that to happen. It may simply be other confounding factors in the text, which are brought out by modifying that attribute, but the attribute itself (when causally traced, which we cannot currently do) had no effect.\n\nIn essence, it is more accurate to say \"RATE measure the causal effect of **removing** an attribute on the reward\", but not the attribute itself.", "methods_and_evaluation_criteria": "Yes, no issues here.", "theoretical_claims": "Did not check.", "experimental_designs_or_analyses": "The experiments are fine for what the authors are trying to show.", "relation_to_broader_scientific_literature": "Explainability of reward models is mostly a novel research area, so it does help to fill a gap in that sense. Work by Jiang et al. (2024) did just do something similar, but as far as I know those authors did not consider the idea of rewriting counterfactual edits.", "essential_references_not_discussed": "This is ok as far as I know.", "other_strengths_and_weaknesses": "### Strengths\n* This is an important problem, we should ideally better understand reward models if we are chasing human-AI alignment.\n* The authors have thought carefully about common issues in counterfactual generation and are trying to directly address them.\n\n### Weaknesses\n* Clarity was a bit of an issue for me. I found it very difficult to understand the process going from original -> edit -> edit of edit. Perhaps the other reviewers got it a bit better, but I feel the paper would really have benefited from a clear motivating example as \"Figure 1\" to really make the problem setup clear, and the issues with prior approaches, as the paper is I had to mentally work very hard to (I think) understand the setup. Figure 1 currently doesn't make it clear how edits can really help.\n* It's difficult to judge Figure 2 also, is the difference in distributions significant? I believe the only way to really evaluate this is with pairwise comparisons to show a change/no change.\n* At a deeper level, my my concern with the paper is that the authors don't show how the method is useful in application. What would make the paper much better in my eyes would be to demonstrate its usage for e.g. model improvement. Can you show it improves LLMs in practice? That would be the only truly compelling evaluation in my opinion.", "other_comments_or_suggestions": "Line 76 typo", "questions_for_authors": "* Can you absolutely guarantee that the counterfactuals (and additional edits) form reasonable text? The method seems to depend upon the second edit just \"minimally\" editing the first edit? How do you know it does that? Creating a counterfactual from a counterfactual seems to multiply the issue of confounding factors rather than help? But maybe I just misunderstand.\n* Can you add an evaluation showing your method helps with model improvement? (or anything else useful)\n\nIf you can address these convincingly, I will raise my score to accept.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTP7jA0G6d", "reviewer": "Reviewer_pJdZ", "summary": "The paper proposes to evaluate the responsiveness of reward models used for LLM training on certain attributes of interest via average treatment effect and average treatment effect on the (un)treated. To simulate interventions on the interested attributes accurately, the paper proposes to rewrite the response twice to cancel out certain rewrite errors.\n\n## Update after rebuttal\nThe authors' rebuttal is not sufficient to change my current rating. They cannot justify the two unrealistic assumptions in practice or evaluate empirically how far a reward model deviates from those two assumptions. I would have given them a higher score if 1) they didn't use those two assumptions or 2) they framed this as a purely empirical paper.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your detailed comments.\n\nWe agree that the assumptions in Theorem 4.1 are somewhat strong, albeit less onerous than they may first appear: the cancellations only need to occur *in expectation*, which is a weaker condition than cancellation for each unit.\n\nHowever, we emphasize that these are merely *sufficient* conditions whose purpose is to show that there exist any situations under which RATE is consistent: after all, there’s no reason a priori to expect that imperfect rewrites can provide a causal estimation—hence, the purpose of the theorem is simply to show that the approach is not vacuous. The substantive evidence of the paper are empirical: that the causal view must be considered (e.g., the experiments in Section 5.1 showing a mismatch with the naive and single-rewrite estimates) and that the double-rewrite method provides better counterfactual pairs than the single-rewrite method (e.g., based on Figure 3 and inspection of the candidate counterfactual pairs in the appendix).\n\nThat being said, we also note that all causal estimation is based on strong assumptions, and the assumptions in Section 4 have the virtue of being largely checkable! Unlike most assumptions in causality, a practitioner can actually just manually inspect a random subset of the rewrite data to see if it matches the desired counterfactual elicitation.\n\nRegarding the specific questions:\n1. RATE is agnostic to the particulars of how the rewriter-LLM was trained (e.g. with RLHF or not): all that matters are the produced rewrites. In particular note that humans also produce imperfect rewrites, and the same estimation procedure could be applied in principle to these expensive human-generated rewrites (subject to the same validity tests above), even though humans are not trained via any explicit reward model.\n2. See main response above.\n3. Probabilities of causation are appropriate for binary outcome variables like legal decisions, which is why we opted for the ATE to formalize how much the reward model (continuous-valued) is responding to attributes of text—hence, the cited papers use a fundamentally different notion of explanations. It’s an exciting direction for future work to explore whether the double-rewrite trick might improve estimation for other counterfactual estimands.", "claims_and_evidence": "Yes. However, I remain skeptical about the realizability that the rewrite errors can be canceled out instead of accumulating after rewriting twice.", "methods_and_evaluation_criteria": "The ATE approach used in the paper needs to be compared with the probability of causation line of literature as I discussed in more detail in the later section.", "theoretical_claims": "Other than the overly strong assumptions, the proof itself is legit.", "experimental_designs_or_analyses": "The experiments look valid to me.", "relation_to_broader_scientific_literature": "The paper discusses how one should provide explanations to reward model sensitivity on certain attributes of the responses. It extends the previous work on naively estimating this sensitivity via average conditional reward differences.", "essential_references_not_discussed": "The paper proposes to use ATE families to evaluate the effect of text rewrites on the reward outcomes. However, in the literature, people also use probability of causation (PNS) extensively to derive explanations. How would the authors compare their approach to this line of work?\n- [Probabilities of causation: Bounds and identification](https://ftp.cs.ucla.edu/pub/stat_ser/r271-A.pdf) in Annals of Mathematics and Artificial Intelligence 28 (2000).\n- [Towards Trustworthy Explanation: On Causal Rationalization](https://proceedings.mlr.press/v202/zhang23ap/zhang23ap.pdf) in ICML 23.\n- [Does Reasoning Emerge? Examining the Probabilities\nof Causation in Large Language Models](https://openreview.net/pdf?id=b1ylCyjAZk) in NeurIPS 24.", "other_strengths_and_weaknesses": "- The usage of ATE is well motivated and the associated practical concerns are clearly explained and addressed.\n- The strong assumption on LLMs rewrite quality is the only flaw in the theory. While the proposed double rewrite procedure seems effective from the provided experiments, it still leaves doubt on whether this is always the case. Especially given the fact that LLMs rewriting the texts are also trained by some black box reward models.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How do you think the quality of the reward model the rewrite LLMs being tuned with will affect your approach outcomes?\n2. How can one possibly measure the \"invalidness\" of your assumptions with black box rewrite LLMs? If rewrite errors cannot cancel out, how will theorem 4.1 change?\n3. How would you compare your ATE approach to PNS-based approaches in the literature?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E9WJxtn3Et", "reviewer": "Reviewer_suXy", "summary": "The manuscript presents Rewrite-based Attribute Treatment Estimator (RATE) as a novel approach to estimate the causal effect of response attributes on reward models. It addresses the challenge of reward model opacity by leveraging LLM-generated counterfactual rewrites. While the work is theoretically grounded and experimentally validated, some areas require improvement for clarity, rigor, and completeness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and your support!\n\nWe will address some of your higher-level comments and questions first.\n- We appreciate your kind words about relevance and novelty. We agree that practical implications are important, which is why we demonstrate the utility of our method on real-world reward models drawn from RewardBench. Following the reviewer’s suggestion, we will emphasize this application in the abstract.\n- We are happy to hear that you find the experiments well-designed. On the topic of rewrite quality, quantitative metrics are fundamentally proxy measurements for linguistic similarity. As such, we believe that simply allowing the reader to see random samples of our rewrites (Appendix D) is a more direct way to demonstrate rewrite quality.\n- While we agree that the ultimate application is in downstream tasks, for the purposes of debugging it’s important to have modular evaluations. For instance, our work allows practitioners to identify whether issues identified in alignment (e.g., length bias) are due to the reward model itself or the alignment procedure. On the other hand, if we were to evaluate our method just by its impact on downstream performance, we would not be able to disentangle these two components.\n\nResponses to other comments and questions:\n- The computational complexity of RATE remains O(n) since we generate two rewrites per example via API calls and score them with a reward model, where the primary cost comes from generation. Using the gpt-4o-2024-08-06 model, processing 25K IMDB samples (including rewrites and rewrites-of-rewrites) costs only $60, and in practice, a relatively small n suffices for reliable confidence bounds on treatment effects.\n- Regarding using other LLMs for rewrites, our priority was to estimate the treatment effect of concepts on reward model scores, so we used the best rewriter available at the time of writing. As LLMs improve in quality and efficiency, we imagine that smaller, more efficient LLMs will be able to perform acceptable rewrites, but considering the cost is so small, and the emphasis here is on precise evaluation, there is little benefit to using less capable LLMs.", "claims_and_evidence": "Strengths\n1. Relevance and Novelty: The paper tackles an important problem in reinforcement learning and NLP model alignment—reward model explainability. The proposed method (RATE) effectively corrects biases introduced by imperfect rewrites.\n2. Strong Theoretical Foundation: The formalization of RM explainability as a causal inference problem (through ATE, ATT, and ATU) is rigorous and aligns with best practices in causal learning.\n3. Well-Designed Experiments: The semi-synthetic experiments provide clear validation of RATE’s superiority over naive and single-rewrite estimators.\n4. Open Source and Reproducibility: The study emphasizes reproducibility by using publicly available datasets and reward models, and by sharing code implementation details.\nWeaknesses and Areas for Improvement\n1. Clarity of Conceptual Framework: Some theoretical concepts are presented too briefly, making them difficult to follow for readers unfamiliar with causal inference.\n2. Incomplete Discussion on Practical Implications: The study does not sufficiently explore how RATE would perform in real-world applications (e.g., LLM alignment tasks).\n3. Limited Discussion on Rewrite Quality: The paper acknowledges but does not fully address the potential quality degradation in LLM-generated rewrites.\n4. Unclear Computational Cost Analysis: The method involves multiple rewrites, increasing computational demands. The paper lacks a clear discussion on efficiency and feasibility.", "methods_and_evaluation_criteria": "Methodology\nSection 2: Setup\n\t• Subsection 2.1, Paragraph 2: The explanation of naive estimators does not provide explicit examples or empirical failure cases. \n\t○ Suggestion: Introduce an illustrative failure case to help contextualize why naive estimators are unreliable.\n\t• Subsection 2.3, Equation 1: The derivation of ATE assumes the existence of perfect counterfactuals but does not explicitly discuss hidden confounders. and none of the formulas that follow are labelled with a serial number, please fix that.\n\t○ Suggestion: Discuss potential unobserved confounders and their impact on causal effect estimation.\nSection 3: RATE Procedure\n\t• Subsection 3.2, Paragraph 3: The paper discusses \"imperfect rewrites\" but does not define quality metrics for evaluating rewrite reliability. \n\t○ Suggestion: Introduce formal metrics (e.g., similarity scores, embedding distances) to assess rewrite fidelity.\n\t• Subsection 3.3, Algorithm 1: The pseudocode for RATE is well-structured but lacks computational complexity analysis. \nSuggestion: Provide an analysis of the computational cost and memory usage, comparing RATE with baseline methods.", "theoretical_claims": "Section 2: Setup\n• Subsection 2.3, Equation 1: The derivation of ATE assumes the existence of perfect counterfactuals but does not explicitly discuss hidden confounders. \nSuggestion: Discuss potential unobserved confounders and their impact on causal effect estimation.\nSection 3: RATE Procedure\n• Subsection 3.2, Paragraph 3: The paper discusses \"imperfect rewrites\" but does not define quality metrics for evaluating rewrite reliability. \n○ Suggestion: Introduce formal metrics (e.g., similarity scores, embedding distances) to assess rewrite fidelity.\n• Subsection 3.3, Algorithm 1: The pseudocode for RATE is well-structured but lacks computational complexity analysis. \nSuggestion: Provide an analysis of the computational cost and memory usage, comparing RATE with baseline methods.", "experimental_designs_or_analyses": "Experiments\nSection 5: Experimental Setup\n\t• Subsection 5.1, Paragraph 4: The authors use GPT-4o for rewrites but do not evaluate how different LLMs impact performance. \n\t○ Suggestion: Include a comparison of rewrite effectiveness across multiple models (e.g., GPT-4 vs. LLaMA).\n\t• Figure 2: The off-target changes in rewrites are discussed, but there is no quantitative measure of rewrite accuracy. \n\t○ Suggestion: Report statistical metrics (e.g., perplexity shift, KL divergence) to assess rewrite fidelity.\nSection 5.2: Real-World Reward Models\n\t• Subsection 5.2, Figure 5: The results show that naive estimators significantly overestimate effect sizes, but do not discuss how these findings translate to practical applications. \n\t○ Suggestion: Provide case studies or real-world examples where RATE can correct reward model biases in deployed systems.\n\t• Paragraph 6: The text briefly mentions bias in common reward models but does not explore whether bias correction affects downstream model performance. \n\t○ Suggestion: Conduct an ablation study to analyze how RATE impacts final model behavior.", "relation_to_broader_scientific_literature": "Introduction\n\t• Section 1, Paragraph 3: The paper states that \"Naively, one might attempt to estimate RM responsiveness to an attribute...\" but does not explicitly introduce the causal inference framework early enough. \n\t○ Suggestion: Introduce the concept of counterfactual analysis earlier to provide context for the naive approach’s limitations.\n\t• Paragraph 5: The introduction claims that RATE is \"empirically effective,\" but lacks an explicit research question or hypothesis. \n\t○Suggestion: Clearly define the research objectives (e.g., \"We aim to develop an estimator that isolates causal effects while correcting for LLM rewrite biases\").", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Please refer to the Claims and Evidence", "other_comments_or_suggestions": "Abstract\nRevision Suggestion: The abstract effectively highlights the problem and proposed solution but lacks quantitative results. Include key findings (e.g., how much RATE outperforms naive approaches in estimation accuracy).", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BzgmruMk5Y", "reviewer": "Reviewer_qahC", "summary": "**Post-rebuttal edit: In light of the discussion with the authors, in which they were very engaged and forthcoming, I spent a lot of time debating whether I should increase my score from from 2 to 3. I continue to share the concerns of Reviewer pJdZ that the two key assumptions motivating the method are unlikely to hold in practice, meaning this should be considered a primarily empirical paper rather than one that has strong theoretical backing. In the end, I decided that the problem being tackled was sufficiently novel, interesting and important that the positives narrowly outweigh the negatives, so I decided to move my score to 3. This was a really close call though, and the meta reviewer should consider my position to be a \"weak weak accept\".**\n\n---\n\nThis paper aims to quantify the causal treatment effects of high-level semantic and syntactic attributes (e.g. length, sentiment, helpfulness) on the predictions of language reward models (RMs). A promising approach (which has been explored in prior work) is to prompt a language model (LM) to rewrite text to modify each attribute, then measure the change in reward. This paper argues that such estimates can be biased if the LM inadvertently modifies other attributes as a byproduct. The proposed method is to use \"double-rewrites\" (i.e. change the attribute, then change it back) to cancel out such off-target changes and obtain more reliable treatment effect estimates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! \n\nA critical point to clarify is that Assumptions 1 and 2 in Section 4 are merely *sufficient* conditions. There’s no reason a priori to expect that imperfect rewrites can be used in causal estimation, so the role of Theorem 4.1 is simply to show that the approach is not entirely vacuous! That being said, we also note that all causal estimation is based on strong assumptions, and the assumptions in Section 4 have the virtue of being largely checkable! Unlike most assumptions in causality, a practitioner can actually just manually inspect a random subset of the rewrite data to see if it matches the desired counterfactual elicitation. \n\nThose are some clever toy examples, let's apply the considerations above to them:\n1. Suppose you visually inspected some samples and had a suspicion that the rewriter was cutting the length by 20% each time it rewrites, as you describe. Unlike e.g. adjustment-based causal methods, you can just test your hypothesis directly, which we consider a major advantage of our method. To illustrate this, we ran two-sample t-tests on the rewrite vs. double-rewrite and found no significant differences between the mean length when rewriting on sentiment. Specifically, p-values were 0.1014 (IMDB) and 0.7496 (hh-rlhf).\n2. We agree that in general Assumption 2 (additivity) may not hold, as more complex interaction effects could occur, but note that the ATE is itself only a sensible causal estimand in the absence of interaction effects (generally, not just in our setting). When such interaction effects are present, other estimands like the conditional ATE are more appropriate. Our work here aims to cleanly spell out a simple case, which we hope lays the foundation for more complicated causal estimands. \n\nResponses to other comments and questions:\n- Good point about Figure 1, we have edited it. Edit: Here is a link to the new figure https://postimg.cc/9wwXZ1bQ\n- The key distinction is that Jiang, et al. focus on altering examples to change a classification score while RATE focuses on estimating a treatment effect of a specific latent attribute on a reward score. We now clarify this earlier in the manuscript per your suggestion.\n- All single-rewrite and double-rewrite comparisons for the \"Real World Reward Models\" Experiments in Section 5.2 are already provided in Appendix D.1; we’ll note this in the main body. We don’t show the single rewrites in Figure 5 because the semi-synthetic experiments demonstrated they don’t properly cancel out rewrite errors.\n- We would expect models to become better at generating “true” counterfactuals as they get better (i.e. better at following instructions, more similar to human-generated text). \n- While it’s true that the double-rewrite method ends up using entirely LLM-generated data, it's not obvious that the original data distribution has any privileged status if the goal is to understand alignment—it might even be the case that LLM-rewrites are closer to the generations of the downstream fine-tuned model. A benefit of the rewrite method (as compared to e.g. adjustment) is the ability to spotcheck rewrites to make sure they look typical of the domains we care about.\n- For our third experiment (Appendix C1), we will edit this section to give quantitative details supporting our claim that the double-rewrite method is more stable. In particular, the slopes of lines interpolating the data points differ: the single-rewrite method has a slope of -0.0413 while the double-rewrite method has a slope of -0.0114. Thus, while both are fairly stable in absolute terms, the double-rewrite is relatively more stable.", "claims_and_evidence": "See \"Theoretical Claims\" for my main feedback on the claims.\n\nRegarding the empirical results, they do align with the claimed benefit of the method. That said, I note that even the single-rewrite method produces much better results than the naïve baseline. It makes me wonder whether the issue you're trying to solve is particularly important in practice. In particular:\n- In your third experiment (Appendix C.1), both single- and double-rewrite methods yield constant treatment effects. I don't quite understand your claim that the double-rewrite results are more stable.\n- On the \"Real World Reward Models\" Experiments in Section 5.2, there appears to be **no comparison to the single-rewrite method at all**. Why is this not included? It makes the reader wonder whether it actually gives very similar results to your method here.", "methods_and_evaluation_criteria": "My main critiques do not belong in this section so I don't have much to say here. The method is well-motivated (provided the assumptions hold; see below), well-explained, and simple to implement, which is always an advantage!", "theoretical_claims": "Overall, I think your main idea is quite an elegant one, and I can see how it should work in theory, **provided all the required assumptions hold**. That said, I'm a little sceptical that they actually do hold in practice.\n\n- **Assumption 1:** I'm fairly happy that the changes to off-target attributes $\\xi$ won't typically depend on the target attribute $W$, but what if they depend on the *current values* of $\\xi$? As an illustrative example, suppose that $\\xi$ denotes the length (number of words) of the response, and that the LM always tends to make a response 20% shorter than the current length every time it rewrites. In that case, the rewrite will be shorter than the original, and the rewrite$^2$ will be *even shorter*. Your method would not cancel out any spurious length effect here, but rather would invert it. I note that my example uses a numerical off-target attribute (number of words) rather than a binary one as you consider throughout your paper, but I don't think that should be critical.\n\n- **Assumption 2:** I see no particular reason to expect this additivity assumption to hold. It seems just as plausible that the interaction between attributes is *multiplicative*, e.g. an RM gives high reward if the sentiment is positive *and* there are no spelling mistakes, and low reward otherwise.\n\nI'm open to any more persuasive arguments that these assumptions should hold in practice, and would be interested to hear if you have any.\n\nA second concern is that your method ends up evaluating the treatment effects of attributes using entirely synthetic data (i.e. the two LM rewrites of each original data point). Since some distribution shift will exist between the original and rewritten data, does this mean that the results don't properly quantify the effect of each attribute in the original data distribution? I'm not quite sure how worried to be about this, but I thought it was worth mentioning (perhaps including in the paper itself).", "experimental_designs_or_analyses": "Overall, the main experimental design using datasets with a known ground truth is appropriate, and tests what you would want to test for a method like this. It also makes sense to end with a comparison on real world models where no ground truth exists, but again, why no single-rewrite method in Figure 5?", "relation_to_broader_scientific_literature": "This paper extends a very small (but important!) literature on evaluating and interpreting language RMs, which are critical components in the alignment pipeline. The specific problem of confounding factors biasing interpretations is a novel one in this area.", "essential_references_not_discussed": "I took a look at your first citation in the Related Work section (Jiang et al. 2024) and noticed some significant overlaps in the core method, i.e. interpreting RM predictions by getting an LM to rewrite text to modify certain high-level attributes. Obviously you are citing this work already, which is good, but I do think you should acknowledge these similarities earlier in the paper, e.g. in the introduction and method sections. Doing so would help to clearly delineate the novelty of your method, namely the use of double-rewrites to cancel out off-target changes. This comment is not intended to diminish your own contribution; in fact, you could frame the possibility of off-target changes as an important limitation of Jiang et al.'s work, which your proposal (partially) rectifies.", "other_strengths_and_weaknesses": "Overall, I find the paper to be well-written, with a good discussion of the key issues and decisions made as the method is introduced.", "other_comments_or_suggestions": "- I don't think the term \"imperfect counterfactuals\" (in the title and elsewhere) quite captures the essence of the problem of off-target attribute changes. I don't have a great suggestion for an alternative, but maybe a word like \"imprecise\" or \"confounded\" or \"poorly isolated\" could be better.\n- Figure 1 isn't currently doing much to aid understanding of the problem, and is quite confusingly laid out. For example, it's not particularly clear what the red question mark is meant to represent. You should consider another attempt at this figure, or even removing it entirely.", "questions_for_authors": "1. Can you give any reassurance to my concerns in the \"Theoretical Claims\" section?\n2. Can you explain why the single-rewrite method is not shown in Figure 5, or better, actually add those results to the Figure?\n3. Can you elaborate on the claim that \"the double-rewrite estimator is more stable than the single-rewrite estimator\" in Appendix C.1? Do you just mean that the values are closer to zero?\n4. At a high-level, I understand your method as compensating for imperfect instruction following on the part of the rewriting LM (i.e. you ask it to only change one attribute, but it changes others too). As these models get better over time, would you expect your method to become less necessary?\n\n**I would like to emphasise that I'm very open to raising my score if you're able to address my concerns. I think RM interpretability is a very important problem and it's great to see novel ideas in this area.**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers have raises important concerns regarding the assumptions made in the paper and the comparison to Jiang et al. 2024.
Reviewer qahC summarises the discussion on the paper very well. Without doubt, the reliance on the two assumptions limits the proposed method, as they are unlikely to hold in practice. This is the achilles heel of the paper. On the other hand, the reviewers agree that the problem being tackled in the paper is sufficiently novel, interesting and important. Also the the approach itself is clearly explained and simple to implement, with sound theoretical justification. Overall, the positive aspects outweigh the negative one.
Decision: Accept (poster)