Preference Learning for AI Alignment: a Causal Perspective
Authors: Kasia Kobalczyk
Mihaela van der Schaar
Venue: ICML 2025 poster (2025)
Keywords: Preference learning
alignment
reward modelling
causality
robustness
confounding
heterogeneity
Primary area: deep_learning->large_language_models
TL;DR: We propose to adopt a causal framework for preference learning to define and address challenges like causal misidentification, preference heterogeneity, and crucially, confounding due to user-specific objectives.

Abstract: Reward modelling from preference data is a crucial step in aligning large language models (LLMs) with human values, requiring robust generalisation to novel prompt-response pairs. In this work, we propose to frame this problem in a causal paradigm, providing the rich toolbox of causality to identify the persistent challenges, such as causal misidentification, preference heterogeneity, and confounding due to user-specific factors. Inheriting from the literature of casual inference, we identify key assumptions necessary for reliable generalisation and contrast them with common data collection practices. We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness. Finally, we outline desiderata for future research and practices, advocating targeted interventions to address inherent limitations of observational data.

Preference Learning for Al Alignment: a Causal Perspective
Katarzyna Kobalczyk'
Mihaela van der Schaar'
Abstract
2022). Sole reliance on statistical associations observed in the training data is prone to learning rewards that pick up on spurious correlations rather than the true factors influencing user preferences (Singhal et al., 2024; Chen et al., 2024), Latent features of texts often exhibit strong correlations, making it difficult to generalise to examples where such correlations no longer hold. Moreover, pairwise preference datasets are often collected opportunistically, with LLM users both evaluating model responses and generating the prompts eliciting them. As result, recorded preferences are shaped by the interplay of the LLM's sampling distribution, latent features of the generated responses, and user-specific contexts that vary across the population. These factors raise fundamental concerns about the robustness and reliability of current approaches.
Reward modelling from preference data is a crucial step in aligning large language models (LLMs) with human values, requiring robust gen- eralisation to novel prompt-response pairs. In this work, we propose to frame this problem in causal paradigm, providing the rich toolbox of causality to identify the persistent challenges, such as causal misidentification, preference het- erogeneity, and confounding due to user-specific factors. Inheriting from the literature of causal inference, we identify key assumptions neces- sary for reliable generalisation and contrast them with common data collection practices. We illus- trate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness. Finally, we out- line desiderata for future research and practices, advocating targeted interventions to address inher- ent limitations of observational data.
In this work, we argue that building robust reward models requires addressing what if type of questions: What would happen if we intervened on specific response characteristics, such as conciseness or creativity? How might preferences change if a different objective were pursued? Current meth- ods are poorly equipped to answer such questions, motivat- ing the need to reframe preference learning through a causal perspective. A causal framework enables the disentangling of the effects of different causes on outcomes, facilitating ro- bust predictions under interventions and distribution shifts. Adopting this lens not only offers new insights but also raises pivotal questions: What assumptions are required to generalise across diverse prompts and user groups? How do these assumptions influence data collection practices? How can we address violations of causal assumptions, such as confounding due to user-specific objectives?
1. Introduction
The remarkable success of LLMs lies partially in their abil- ity to align with human values, producing responses that are helpful, harmless, and honest. A central method for achieving this alignment reinforcement learning from hu- man feedback (RLHF), with the LLM's behaviour shaped by reward models derived from datasets of human prefer- ences (Ouyang et al., 2022; Bai et al., 2022). The reward modelling, aka preference learning stage seeks to address a seemingly straightforward question: Given two responses to the same prompt, which one aligns better with human objectives? However, the challenges of this stage are of- ten underestimated, assuming that simple regression-bess based models fitted to observational datasets can generalise effec- tively to unseen texts. Meanwhile, evidence suggests this optimism may be misplaced (Tien et al., 2023; Skalse et al.,
Contributions. We introduce a causal framework of prefer- ence learning for Al alignment, defining the key challenges and identifying critical assumptions for generalising reward models to unseen texts and contexts. Through examples and real-world experiments, we highlight failure modes of na've models when causal assumptions are violated, includ- ing confounding due to user-specific objectives-an issue we are first to identify and address explicitly. We demonstrate how causal representation learning approaches can improve model robustness and propose desiderata for future prefer- ence data collection, advocating for targeted interventions to mitigate the inherent limitations of observational data.
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, United Kingdom. Correspondence to: Katarzyna Kobalezyk <knk25 @ @ww.mmm.m.......................
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Preference Learning for AT Alignment: Causal Perspective
2. Background
not. However, only one of these outcomes is observed (the factual outcome Y), while the other (the counterfactual out- come) remains unobserved. The goal of causal inference is to estimate the potential outcomes under both treatments, or their difference ((1) - ((),, known as the causal effect.
Notation. Throughout this paper, we refer to a random variable with a capital letter (e.g.. X) and the value it takes as a lowercase letter (e.g.. X = z). We let 2 denote the space of natural language. We use coloured boxes to high- light: insights (blue), key assumptions (yellow), theoretical implications (red), and empirical case studies (green).
3. The Causal Framework
Setup. We consider the standard setup of preference learn- ing for LLM alignment in which we have access to a dataset D consisting of i.i.d. realisations (x. y. U', ()) of random variables (X,Y, Y,Y,Y",, L), where X X C Σ* is the prompt, Y.Y' E y C Σr, two candidate responses, and T. E {0, }} is binary preference label with I, = indicating that (X, Y) is preferred over (X, Y"), denoted by (X,Y) > (X, Y"), and L ==0 O the opposite. Here, X' x y denotes the space of all plausible prompt-response of pairs.
Treatment
Rewards
R
Outcome
X
L
Y
R'
Reward modelling for RLHF. It standard to assume that the pairwise preference labels L,, are dependent on unobserv- able rewards R and R' assigned internally by the individual to each of (X, Y) and (X, Y"), respectively. Rewards are determined by a function r: Σx + R, so that It = r(X, Y) and R' = r(X, Y'). The likelihood of the option (X, Y) being preferred over (X, Y") is described by the Bradley- Terry-Luce (BTL) model (Bradley & Terry, 1952):
Figure I: X and the two responses Y, Y' users assigns them unob- servable rewards R. R' determining the preference label L.
The causal model of preferences. Given prompt
We can think of the observed tuples (X, Y. Y') as treatments assigned to human labellers tasked with selecting a response that they prefer and the observed labels as outcomes. Treatments are assigned according to some (often unknown) propensities: π (z. 8, y;) := P(X 2, Y y. Y' = y/). where in most cases have that Y and Y" are conditionally independent given X. The distributions P(Y x) and P = z) are not necessarily the same. For instance, given X, Y can be sampled from a base L.LM policy and Y" from a policy controlled by the researcher in a pre-defined way. We assume that conceptually, each individual provid- ing their preferences a priori associated with a set of po- tential outcomes L(X r: Y = y. Y = v') = L(x; y, y¹), for any two responses y, y V, and prompt x E X. Po- tential outcomes capture the hypothetical preferences for all pairs of texts that could have been observed. If we had a way of knowing L(x; y, 3'), for any x E X and y. y' E V, we could answer counterfactual questions regarding user preferences for hypothetical LLM's responses. However, in reality, we only observe the preference choice associated with the treatments actually received: (X)).
(1) where a stands for the sigmoid function. The reward func- tion r is approximated by parametric model re E' + R whose parameters 9 are chosen by minimising the negative log- log-likelihood of the observed examples (I, v' E) E D:
P((X,Y) (X2Y") = (r((, ((XYY)))
[(0)=
logo(ra(z, ro (2) (D)
Σ
where yd indicates the winning response and yf the loosing one. The fitted reward function is subsequently used in the RL stage to provide feedback on generations of the L.M..
Causality and Potential Outcomes. The potential out- comes framework (Rosenbaum & Rubin, 1983; Splawa- Neyman al., 1990) provides a formal approach to causal inference by conceptualising causation in terms of interven- tions. At its core, the framework models how an outcome of interest would differ under different interventions, enabling reasoning about causal effects. It considers a set of units (e.g., individuals) to which a treatment or intervention is applied. For each unit, the treatment T can take on different values where most commonly we have either T' for receiving the treatment and T 0 for being in the control group; see e.g.. (Lopez & Gutman, 2017) for extensions to multiple treatments. In the case of binary treatments, each unit has two potential outcomes denoted as Y(T- = 1). or in short Y (1he outcome if the unit receives the treat- ment, and Y(T 0) tthe outcome if the unit does
We can adopt the potential outcomes framework to the BTL model by introducing the notion of potential rewards, R(z, y) R(X = 2, Y = y) and R' (x,z')) = R'(X- = x, Y' = y²), representing the hypothetical rewards assigned by an individual to any prompt-responss pair for x E X, y. E y in our corpus so that:
E B[L(ryy. = P(L(x: 1)
= aa(( v) - Rt' (x,1))).
(3)
Thus, the BTL model directly relates the difference of po-
2

---

Preference Learning for AT Alignment: Causal Perspective
tential rewards with the expected potential outcome."
We note that the absolute values of the potential re- wards are, in general, not identifiable (any shift of the reward function by an arbitrary function of the prompt results in the same likelihood). However, we s'I}}
The fundamental challenge. The fact that for a given unit we can observe the outcome L only for the observed treatment condition" (x, Y, Y²) is known as the funda- mental challenge causal inference. Potential outcome prediction is related to the generalisation challenge conventional machine learning terminology- -ur goal is to predict the effect of hypothetical interventions on LLM's responses, by estimating (1; (.)) for any E X' and y, y/ E )
have: R(x, y) R(r,y)) = o o o- [E = z, Y -y=y. = s/1}. so that the difference in potential rewards is a function of the observable distribu- tion P(L]X, Y,Y"). and therefore is identifiable.
Corollary 1. The difference in potential rewards R(z,y) - R' (z, y²) is identifiable
"or at most a finite set of treatments {X,, Yi, 1O)
The importance of assumptions 1-3 in enabling causal in- ferences from observational data makes us questions their feasibility in the context of typical preference data collec- tion. In particular, we identify several potential challenges:
3.1. What makes potential outcomes identifiabl?
Causal inference provides a framework to answer counter- factual, 'what if" questions even when only observational data is available. The key part lies in ensuring that the causal quantity of interest can be estimated from observational data alone. The following commonly made assumptions, adapted to our preference learning setup, are sufficient to guarantee identifiability and non paraammmttiicc estimability:
R
R
X
X
(a) Self-written prompts
(b) Randomised prompts
Assumption (Consistency). For an individual with prompt-response assignment (X, Y'), we observe the associated potential outcome, i.e. L(X;Y,Y')
Figure 2: Confounding due to user-specific objectives. a) The user-specific contextual variable C' can act as a con- founder. the prompts X are written by the users them- selves, C' affects the assigned rewards R. H'' and partially determines the treatment (X, Y,Y"). b) Even if C' is not confounding, it may influence the user specific rewards, in- troducing individual-level variation in treatment effects.
Assumption 2 (Unconfoundedness). There unob- served confounders, so that ((2:0,1)) IL (X,Y,Y), for all æ E X. y E V.
? Confounding bias. The assumption of unconfounded- ness about who scores the texts and not their contents, We stress that this assumption can be easily violated. This may be the case if the prompts X are not randomly assigned, but written by the individuals who then score the generated responses. As result, users scoring the LLM's responses of one kind may systematically differ from users scoring responses of another kind. We can formalise this by introducing a user-specific contextual variable that influences both X and R (see Figure 2).
Assumption (Unconditional Positivity). Treatment as- signment non-deterministic, i.e. P(X- x, Y y, Y' v¹) for all X' and y, e..
Proposition 1. Under assumptions (1), (2) and (3), all x E X. v.yl E y,
E [L(zv.))))
=,, ='),
so that observed statistical associations have causal interpretation.
Example 1 (Violation of Unconfoundedness). Suppose the population consists of medical experts (C = expert) and non-trained users (C = -expert). Experts are more likely to ask questions about niche topics, leading to complex LLM responses filled with medical jargon, and are also more inclined to prefer such detailed responses due to their professional focus. In this case, the as- sumption of unconfoundedness would not hold, as the unobservable user-specific variable C, indicating the expertise of a user influences both the treatment assign- ment and their preference choices.
Proof. Appendix B.
'The notion of potential rewards R(x, u) and (2x y') should not be confused with the reward function. R and R' represent two random variables which, in principle, can have distinct distribu- tions so that P(R(X = x, Y v)) & P(R' (X = x, Y' y))- this can be the case, for instance, if the response seen on the left is systematically valued higher than the response seen on the right. Most commonly, it is however assumed that R = r(X, Y) and R' r(X, Y" for deterministic reward function E' + R in which case (R((X x, = v)) = P(R (X = x, Y" v)).
3

---

Preference Learning for AT Alignment: Causal Perspective
In the face of confounding due to user-specific covariates C, is necessary adjust for these variables to avoid the bias. Instead of focusing on the average [x, y, v.y"),, we shall instead consider: (x, y, v/)' = c). Assumptions 2 and 3 must be then modified accordingly as: 2) [(=;y, y, y') II (X) = с and 3) 0 < P(X = x, Y = y, Y ='(C== < 1 for all x E X. y. y y and e E C, where C the space of all levels of covariates. Throughout the rest of shorthand for
in the observational space generalisation of reward mod- els is possible by assuming that the observed texts can be compressed into a lower-dimensional latent representations capturing all latent features of texts that influence the re- wards assigned.
3.2. Robust Generalisation via Latent Treatments
We present model of latent treatments allowing us to con- sider assigned texts not on the literal level, but in terms of their underlying features. This formulation is key to facilitat- ing generalisation to unseen texts and enables the learning of human- internn-tterrettble reward models, supporting the design of targeted interventions (see Appendix C). While appealing, the introduction of the latent treatment model presents its own set of challenges. For brevity and clarity, we focus on single prompt-respone pair (X, Y). assuming the same applies to the alternative (X, XY).
this work we will use the term positivity the unconditional case and the term conditional positivity, aka the overlap, will be used when user-specific covariates are considered.
To the best of our knowledge, confounding due to user- specific covariates has not been addressed prior works on preference learning for Al alignment. We find this issue of significant importance and we will study it in greater depth in sections 3 and 4.
Heterogenous treatment effects. Even under prompt ran- domisation, the characteristics of a user or other forms of un- observable context (represented by the variable in Figure 2) may influence the rewards assigned to each response, lead- ing heterogeneity in preferences among the users. In this case, implicitly marginalizing over C' inflates the variance of the predictions and may mask the true causal relationships present among different subpopulations characterised by distinct C's. Siththaranjan et al. (2024) refer to this variable as the "hidden context" and show that averaging over C' is equivalent to adopting the so called Borda count rule, which may lead to counter-intuitive results. From the point of view of causal inference, C introduces individual-level variation, leading to heterogenous treatment effects where any given treatment (here, any given response) might affect different users in different ways. If not adequately addressed, this in turn may lead to underrepresentation issues (Wu et al., 2023), resulting in allocation of suboptimal treatments for underrepresented populations. In our context, an LLM fine- tuned to the average preferences may generate suboptimal responses for certain subgroups of users. This underscores the importance of not only focusing on the population aver- age effects, (2, y. v.) but also subgroup-level effects: E[L(x, y. s = c].
Latent Factors Z
Y
R
Figure 3: The latent treatment model. The effecc of observed texts on R can be compressed into set of latent variables Z partitioned into two kinds: = artifacts of X and z the latent treatments determined jointly by X and Y.
We assume that features of texts that affect the rewards can be effectively summarised into a set of latent features Z = (Z,,.... Zn E Z split into two parts: z -artefaccts of the prompt influenced by X only, and -T-the latent treatments influenced by (X, Y) jointly. Z- may describe features like the topic of the conversation or the type of task involved. On the other hand, Z contains the features of the response Y that cannot be determined without putting them in the context of X. This involves, for instance, factual cor- rectness or instruction mnstruction-follooinggg In our definition, ZT also subsumes the artifacts of the response that can be extracted from Y without having access to X, like its style or length.
The assumption of overlap requires that every user has a non-zero probability of being assigned any combination of texts given their covariates. In conventional, non- non- -orramamtttic treatment effect estimation, this ensures that we have enough data to estimate the causal effects across all levels of treat- ments and covariates. In our case, however, treatments belong to the extremely high-dimensions space of natu- ral language. We only observe a small subset of potential prompt-response pairs, each often being scored by at most one user, while in general, our goal is generalise to texts not part of the training corpus. With no sufficient overlap
Formally, we assume existence of feature extracting func- tion g "" + 2, (X, + Z |ZX 2T,, that can be decomposed into two maps: gx Σ + 2 and gf : Σ' + 2T, s.t. 7x = n (X) and z" = 9 (X, T = (X, The assumption of sufficiency of the latent factors in determining the rewards assigned can be then defined as:
A ssumption 4 (Latent Sufficiency). Assume there exists functions gx : + zx and g" - + zz such that

---

Preference Learning for AT Alignment: Causal Perspective
for any I E X. v.y' E 3/ we have
② The significance of sufficiency and latent positivity is that, if these assumptions hold, it is possible to esti- mate the potential outcome for a previously unobserved examples (1r, V, 3y') by only considering the average out- comes of texts observed within our corpus that have the same latent structure as (x,, ys y')
R(X
=(X=,, = -, Y, = =y) =R = ==x,z²
=
= = v, Y= =y') -))=((?'' = 2tT;;
where xx gxx (x), x = (x, (2,1), x²² = = (1,
3 3.2.1. IMPLICATIONS FOR PRACTITIONERS
? Why x matters? If the effect of xx on R is ad- ditive that for any y we have R(I, 3) fi(=") + f²(=²), then we could omit zx from the model, as it
? Limited latent positivity. The assumption of latent positivity requires that all combinations of latent factors need be observable. However, it may be that certain factors are perfectly correlated with each other, in which case disentangling their effects is not possible. When the positivity is limited, i.e. when strong correlations ex- ists, it hinders the efficiency of estimators (Hahn, 1998; Hirano et al., 2003; Crump et al., 2006), requiring large amounts of data for robust results.
would have no influence on the outcome L defined by R(x, v') = Si - (x
Vx, v. Our
claim, however, is that the effect of z* is rarely of an additive nature and thus, it cannot be disregarded.
Example 2 INon-additins effects of )x Consider xx representing the task type (e.g., summarisation or cre- ative writing), and z E I. the conciseness of a re- sponse. While in general, responses should not be ex- cessively short or excessively long, the optimal level of conciseness varies between the two task types, introduc- ing a non-additive interaction effect between z and z" which could be, for instance, described as:
Example 3 (Correlated latent factors).
a) Topic and Tone: Suppose Zx stands for the topic of the prompt and one of Z indicates the presence of a formal tone. Professional. work-related topics may naturally lead to LLM responses with a formal tone so that P(Z = informal, Z = professional) and P(Z = formal, z = causal) being near 0. Suppose 7f repre-
[ Bo(=" - 70 if x = summarisation 31 - ) if x = creative writing,
R(x,w)
b) Completeness and Conciseness: sents whether response is complete and Z indicates whether it is suceinet. Complete responses often require elaboration, making succinct yet complete responses rare, i.e., P(Z = 1, 2 1) close to 0.
where we would expect that 70 > 71. reflecting that the optimal conciseness level for summarisation is greater (i.e., more concise) than for creative writing.
The information-compresiin nature of makes it possi- ble to estimate the causal effects [[ y, s/)] for prompt- response pairs not previously observed in our training cor- pus. In particular, instead of requiring that the positivity assumption holds in the observable space, it is sufficient to consider the weaker assumption of latent positivity:
Interpretable causal effects. If the latent factors Z carry human- iamman-nteerrrttblle meaning, we can draw further insights regarding the causal effects of each individual factor on user preferences. Appendix C.1 defines the appropriate mea- surement tools. We also make connections on how such inntrrtlittereretabble models can enable targeted modifi- cations LLM''s responses and thus enable collection of interventional data (Appendix C.2).
Assumption 5 (Unconditional Latent Positivity). For == ,²,,"" E g
all xxx E x and
0 < P(Z = 21 = .T _TTx2xx = x 1.
Discovery of Z. Thus far, we have assumed that the map- ping from raw observations to latent causal factors is given. In practice, it needs to be learned from the available data. Since is not plausible to obtain an exhaustive list of all causal factors and label each example in the corpus with respect these factors, we must instead shift towards causal discovery which is significantly more challenging (Schölkopf et al., 2021). Discovering 2 in a an unsupervised or semi-supervise fashion (i.c., without explicit labels for the latent factors or only with partial labels) creates the risk of causal misidentification (Locatello et al., 2019; Makar et al., 2022: Brehmer et al., 2022; Ahuja et al., 2023), where
Proposition 2. Under assumptions 1, 2, 4, and 5
E [[L(2: v''1 11 [1/2* xxx22 -TT 2T, nT = =
- for xxx = (x), 2T = g (z. v) and /! = -Z V.
Proof. Appendix B.
5

---

Preference Learning for AT Alignment: Causal Perspective
Table I: Test time accuracy [%] of the BTL models with varying values of latent positivity in the observational dataset.
correlation is unrealistic in practice. The focus of this ex- periment is on the statistical challenges posed by limited latent overlap. The observed performance degradation reflects statistical issue arising from near-violations of the positivity assumption, rather than fundamental identifiability failure.
p
0.0
0.3
0.6
0.9
ID
69.3 0.2
68.3 = 0.2
67.8
= 0.2
67.6 0.2
OOD
64.5 + 0.2
62.0 + 0.2
59.7 = 0.2
57.8 + 0.1
the learned rewards mistakenly rely on spurious features (e.g., length (Singhal et al., 2024; Chen et al.. 2024) or formatting (Zhang et al., 2024)) instead of the true causal factors. This in turn leads to robustmess issues. The learned latent representations cannot capture features that are spuri- ously correlated with true causal factors. At the same time, none of the causal factors can be omitted-not not measuring all causal factors influencing the rewards will necessarily lead to omitted-varian bias (Fong& & Grimmer, 2023).
? Confounding & Latent Overlap. The challenges of reward modelling are exacerbated even further when con- founding effects are present. in which case we require that 0 < P(Z* = xx 2T 2T, 2 = x IC e) < 1, for all x E E =T, E ET and E C. If C' represents objective according to which the LLM's responses are evaluated and this objective C also affects the type of prompts that the user writes, then the latent overlap is likely to be particularly limited.
Experiment (Limited latent positivity). We illustrate the significance of the latent positivity by examining its influence on standard BTL models. Using the UltraFeed- back dataset (Cui et al., 2024), consider the truthful- ness and instruction following factors of each prompt- response pair, denoted Z₁ and Zz, respectively, and scored from 0 to 5. We construct five training datasets by varying the correlation coefficient AAr between Z1 and Zz. We let the reward function be r(z. y) 7221 + $222, with the true values of 21 and 22 not available for train- ing. The datasets consists of tuples (a, y. V. () with e determined by the function We assess the robustness of the learned reward models to shifts in the correlation of Z, and Zg. testing on previously unseen examples either from the same distribution as the training exam- ples (ID: o pest = p²"), or not, with fret being negative (OOD: cest < 0). Refer to Appendix D.2 for details.
Example 4. Imagine two groups of labellers, each with a distinct objective C' E {0, 1}. For C = 0, a labeller is focused on assessing the helpfulness of the model, while for C' = 1, they are focused on assessing the harmless- ness of the model. These differing objectives lead the two groups to generate prompts with distinct intents: the helpfuel (C = 0) produce assistance-related prompts, while the mennlessness-focssee (C 1) gen- erate prompts designed to elicit harmful behaviour. Con- sequently, the prompt distributions P(X CC' = 0) and P = 1) are distinct, potentially leading to over- lap violations in P(ZIC). Despite this, we wish to an- swer interventional questions regarding the preference choices of the helpfulness-focusee
users (C = 0), if presented examples (ar, y, v.3 with x P(XIC 1) and vice versa (see Figure 4). With no latent overlap, fundamental assumptions of causal inference deem this is an infeasible task. In section 4 we will study this case from an empirical perspective.
As shown in Table 1, performance on unseen ID exam- ples remains high across all p' values, demonstrating that latent sufficiency enables generalisation to unob- served prompt-response pairs. However, with OOD ex- amples breaking the training- time correlation between Zi and Z2, performance significantly declines due to reduced latent positivity. The ID-OOD accuracy gap widens as p² increases. Even at a moderate p²r = 0.6, accuracy drops to 59.7%, compared to 67.8% on unseen ID examples (note, the correlation between truthfulness and instruction-following across the entire UltraFeed- back dataset is precisely 0.63). Appendix D.I provides further analysis and visualisation explaining this be- haviour.
Even when the ground-trum latent overlap holds, learning ro- bust representations 2 from finite datasets poses significant challenges. Correlations between the observed examples Y. Y.YYY') and the objectives C' in the training data can lead to 2 entangling features of the text with user-specific objec- tives. As a result, 2 may fail to generalise to examples with X's and C's no longer correlated.
4. Case Study: The Challenge of Confounding
Modern reward learning systems frequently rely on prefer- ence data collected opportunistiaally- user types query, two candidate responses are generated, and the user is asked to select the response they prefer. In such setups, user- specific objectives can act as confounders, influencing both the types of prompts the user formulates and the subsequent
We note that this experimental setup does not strictly violate the positivity assumption, making identifiabil- ity possible in the infinite data limit. However, perfect
6

---

Preference Learning for AT Alignment: Causal Perspective
examples (2, V, 3/) with type(z) = 1?". We note that, the two objectives considered are not always aligned-a highly helpful response may not necessarily be harmless- making this a challenging machine learning problem.
Z
R
Z
R
X
(a) Observed data
(b) Intervention on X
Dataset: W create six training datasets varying the value of p. The datasets comprise of tuples (æ, yi v' c, e), with an equal number of examples for each objective с E {0, 13. Type labels are not part of the training data they wouldn't be available in a real-world data col- lection setup. Models: We train three multi-objectivv reward models. The Base model corresponds to the most straightforward architecture, where the objective label e is concatenated with the reward model's inputs (Fig- ure 5()). In contrast, the Multihead model is inspired by the ground-truth causal graph wherein the latent fac- tors Z are conditionally independent of C given (X, Y). The model learns a latent representation of the prompt- response pairs, 2. passed to two independent prediction heads corresponding to the two objectives (Figure 5(b)). Finally, the proposed Adversarial model (Figure 5(c)) incorporates an additional adversarial objective (Ganin et al., 2016) to regularise representation learning. This is inspired by recent advancements in representation learn- ing for treatment effect estimation (Bica et al., 2020; Ozery-Flato et al., 2020; Du et al., 2021) (see section 4.1 for details). Evaluation: Trained models are evaluated on unseen prompt-response pairs derived from either of the data subsets and labelled according to objectives both consistent and inconsistent with their prompt type. Refer to Appendix E. for more details.
Figure 4: A reward model relying on the set of true causal factors& Z enables prediction of outcomes under targeted interventions. We wish to predict the rewards when interven- ing on the prompt distribution X, breaking the correlations between prompt type and user-specific objectives C.
preference choices. This study highlights the challenges as- sociated with robust reward learning in the presence of such confounding effects. To illustrate these issues, we revisit Example 4 and simulate the described scenario utilising an augmented version of the well-known HH-RLHF dataset, as provided by Siththaranjan et al. (2024).
The original HH-RLHF dataset (Bai et al., 2022) con- tains prompt-response pairs categorised into two sub- sets: helpful and harmless. Siththaranjan et al. (2024) noted significant distributional differences between the prompts in these two subsets. Meanwhile, all prompt- response pairs in the helpful subset are evaluated based on the objective of helpfulness, while those in the harm- less subset are assessed according to the objective of harmlessness. To examine the challenges arising when users operate under potentially conflicting objectives, Siththaranjan et al. (2024) created an augmented version of this dataset, introducing synthetic, counterfactual la- bels that texts in the helpful subset are also scored according the harmlessness objective, and vice versa.
4.1. The Adversarial Multi-objective Reward Model.
Due to the training time correlation between type(X) and C the learned latent representations may mistakenly treat the information about typo (X) easily extractable from the observations (X, Y)s having causal effect on R. The goal of the proposed adversarial training method is make 2 not predictive of type(X) X), while retaining all features that causally influence R. Let go represent the network learning a latent representation 2 from (X, Y), s.t. 2 = go (z,, v) for all x, v E X x V. In the Multihead reward model, the rewards are obtained by passing the latent to the respective reward head fwo 2 - R or fwe 2 - IR so that:
As in Example 4, we let C' E f0, 1} encode the objec- tive of the labeller. We also denote by type(r)) E {0, 1}, whether a given example (z, y, 3/) was originally part of the helpful or harmless subset. To control the degree of confounding due to C, we define a parameter p that measures the alignment between user's objective and the type of prompts they write: p = P(lype(X X) C)) At p = 1.0, the prompt-response distribution recovers the original HH-RLHF dataset, with prompt type and user objective being perfectly correlated. In contrast, at p = 0.5, the prompt type is fully randomised, eliminat- ing the confounding effect.
Swo (10 ())
if = 0
T0 (2,
(4)
The Goal. Our aim is to investigate how the confound- ing effects related to user-specific objective influence re- ward model performance and to explore mitigation strate- gies. We seek to answer the interventional questions, such as "What would the preferences of the helpfulness- focused users (C = 0) be, have they been presented
fer (90(x,)))
if
The Adversurial model introduces an additional network hep Z + {0, 1} whose goal is to predict the objective label C' from Z as a proxy for type(X). Adversarial train- ing enourages ge to discard the spurious information about type(X) while retaining causal features relevant to the out-
7

---

Preference Learning for AT Alignment: Causal Perspective
f
0
LLM
Fu
F
y
MLP
X
LLM
MLP
LLM
MLP
2
C=1
f
y
Y
C=1
h
(a) Base
(b) Multihead
(c) Adversarial
Figure 5: Comparison of model architectures.
come. The training objective for this model is:
accuracy on inconsistent test samples improves across all models as decreases in the observational training data. Extra gains from causally models. We observe that the Multihead architecture significantly outperforms the Base model, supporting our hypothesis that separation of latent representation learning from Abjective-conditiones reward prediction enhances ro- bustness to training-tine timiingg-tmmeeee correlations between prompt types and objectives. Introducing the adversarial objec- tive further strengthens this effect, significantly boosting accuracy, especially in strong confounding regimes.
min max CR wo, w1) ALadv (0, 0,w0,
(5)
where (0, wo, w1) is the standard BTL loss for the re- ward function Ta w the second term Ladv (0, σ) is the binary cross-entroe cross-entropy loss between the true c's and their log- probabilities predicted by h& go. and À hyperparameter balancing the two objectives. Appendix contains further details regarding implementation and training.
Model:
65
Accuracy (%)
Multihead
60
Adversarial
5. Conclusions & Limitations
Consistent:
56
False
Below, we summarise key conclusions from this work, con- textualise them within existing literature, and outline future research directions. Appendix A contains an extended dis- cussion of the related work.
No confounding
Extreme confounding
True
50
0.6
0.8
1.0
Consistency training time (p)
Figure 6: Test time accuracy confounding. The label "consistent" indicates whether type(X) = C. The causally- inspired multihead architecture with additional adversarial balancing significantly reduces overfitting and improves generalisation to the inconsistent OOD examples.
The importance of data collection. This study underscores the critical role of data collection mechanisms in reward learning for preference modelling. Controlled, randomised experiments, where prompt-response pairs are allocated ran- domly across a representative population serve as the gold standard. However, in practice, preference data often collected opportunistically, relying user-written prompts and -LM-errerreed responses. This approach introduces the risk of confounding, where users' latent objectives in- fluence both the queries they pose and the feedback they provide, as demonstrated in our case study.
4.2. Case Study Results
We analyse the results of the experiments summarised in Figure 6 and presented in detail in Appendix E.2.
No overlap = failure to generalise. When the objec- tive is fully determined by the data subset (p = 1.0), all three models exhibit substantial overfitting to the train- ing distribution. While the performance on the unseen, consistent samples is high-with the multihead and adver- sarial architectures showing modest improvements all models fail to generalise to inconsistent examples, with their accuracies falling just below 55%. This outcome is expected, as the latent overlap assumption is likely violated at = 1.0. Higher p values lead to training sets with fewer inconsistent examples, making models more susceptible overfitting and causal misidentifi- cation. Increasing the overlap helps. As expected,
The challenge of unobserved confounding. Our experi- ments assumed explicit access user-specific objectives, allowing simulate and control for confounding effects. While this setup provided clear experimental framework, it does not fully reflect real-world scenarios, where user objectives are not directly observable, posing the challenge of unobserved confounding. To address this, preference data collection methods could be enhanced infer user-specific objectives through, e.g. a) explicit feedback: users could provide rationales for their preferences, offering richer in- sights into their underlying objectives; b) auxiliary data: preferences could be inferred from contextual information, such as user demographics or historical interactions. Only a limited number existing works (Li et al., 2024; Wu et al., 2024; Liu et al., 2024; Kobalezyk et al., 2024) have
Code for reproducing the experiments is made available at: htwp://asiiikoobllllyyuauaaaaaaa
8

---

Preference Learning for AT Alignment: Causal Perspective
considered incorporating user-specific information for per- sonalised alignment, yet none have explicitly examined the confounding issues identified in this work.
Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 377-407. PMLR, July 2023. URL https: /ttp///ecccc......////////....se mir.
html.
Mitigating low overlap. Our findings underscore the impor- tance of latent overlap. Strong correlations between latent response features can cause catastrophic overfitting, hinder- ing generalisation under distribution shifts, However, con- trolling overlap practice not straightforward. Existing work robust reward modelling mainly addresses biases from eesponse-specifer artefacts, such as length (Singhal et al., 2024; Chen et al., 2024) or formatting (Zhang et al., 2024), but as demonstrated, causal misidentification extends beyond these cases. While approaches derived from causal representation learning can help, architectural modifications have inherent limitations. Instead, we advocate for shift in preference data collection practices from passive data gathering to targeted interventions wherein latent factors are systematically controlled to reduce model uncertainty about the set of true causal features. Appendix C.2 outlines potential research directions in this pursuit.
Annervaz, K. M., Chowdhury, S. B. R., and Dukkipati, A. Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing. CoRR, abs/1802.0599 2018. URL http:://rat /ttpiv..x../////////./ htta/1802.05590 aivi:: 1802.05930.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Keriion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-
Z., Herman- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training a Helpful and Harmless Assistant with Rein- forcement Learning from Human Feedback, 2022. URL https: 2204.05862.
wtt.oo/gg.bb///////e 05862. eeprint:
Limitations. We note that the assumptions discussed this paper are sufficiint-bbt not strictly necessary-for identifi- ability. In practice, identifiability may still be achievable under weaker conditions, particularly when data from multi- ple environments is available (Ahuja et al., 2023; Richens & Everitt, 2024; Von Kügelgen, 2023). This is especially pertinent to recent work on reward learning across diverse datasets and personalised reward modelling areas that have garnered increasing attention in alignment research (Wang al., 2024; Ramé al., 2023; Bose al., 2025). A com- prehensive treatment of robust reward learning in the multi- dataset setting is left for future work.
Bica, I,, Alaa, A. M., Jordon, J., and Schaar, M. d. Es- timating counterfactual treatment outcomes over time through adversarially balanced representations. In In- ternational Conference on Learning Representations, 2020. URL https: 1d-BJq866NEVE.
/tttttss:///peeeeeeeeee hwwp///peeneeeeeeeeeeeeeee net/forum? forum?
Bose, A., Xiong, Z., Chi, Y., Du, S. S., Xiao, L., and Fazel, M. LoRe: Personalizing LLMs via Low-Rank Reward Modeling, 2025. URL ht //arrivaavvvvaa//aaeeaaeerreaava orr./org/aaaoo / .eprint: 2504.14439.
2504 14439.
Impact Statement
Bradley, R. A. and Terry, M. E. Rank Analysis of Incom- plete Block Designs: The Method of Paired Com- parisons. Biometrika, 39(3/4):324-344 1952. ISSN 00063444, 14643510. URL http:/ /twww//www//..m/////////// jstorr ht/sssaaleee2333/
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Publisher: [Oxford Univer- sity Press, Biometrika Trust].
Acknowledgments
Brehmer, J., Haan, P. d., Lippe, P., and Cohen, T. Weakly supervised causal representation learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural In- formation Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL https://///// //tp///ppeeeee.ee////e/ee net/ forum? i id=dz7 9MhOXWvv.
This work was supported by Azure sponsorship cred- its granted by Microsoft's AI for Good Research Lab. Katarzyna's Kobalczyk research is supported by funding from Eedi.
References
Ahuja, K., Mahajan, D., Wang, Y., and Bengio, Y. Inter- ventional Causal Representation Learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.). Proceedings of the 40th International
Butcher, B. Aligning Large Language Models with Coun- terfactual DPO, 2024. URL https: /tt///axxtt //w//aiivvoo//// .rg/
abs/2401 09566.
eprint: 2401.09566,

---

Preference Learning for AT Alignment: Causal Perspective
Cao, B., Lin, H., Han, X., Liu, F,, and Sun, L. Can Prompt Probe Pretrained I Language Models? Understanding the Invisible Risks from a Causal View. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 5796- 5808, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10.18653w1/2002.aclHorrr 398. URL https: hale::/aaaaaattt http://aaaaaoollooor///////0//// acl scl--lnge 1on...nng..//./ 398/.
10.100774100188-00--0775333 URL https: //oi.
Eisenstein, J., Nagpal, C., Agarwal, A., Beirami, A., D' D'Aouur, A. N., Dvijotham, K. D., Fisch, A., Heller, K. A., Pfohl, R., Ramachandran, D., Shaw, P., and Be- rant, J. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. In First Conference on Language Modeling, 2024, http:: / /openrevieww net/ forum? d= 5
Fong, C. and Grimmer, J. Discovery of Treatments from Text Corpora. In Erk, K. and Smith, N. A. (eds.), Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1600-1609, Berlin, Germany, August 2016. Associ- ation for Computational Linguistics. doi: 10. 18653vv1/ P16-1151. URL https: http//aaaaaanttoa.............aa org/ P1 P16-1151.
Chen, L., Zhu, C., Chen, J., Soselia, D., Zhou, T., Gold- stein, T., Huang, H., Shoeybi. M., and Catanzaro, B. ODIN: Disentangled Reward Mitigates Hacking in RLHF. In Proceedings of the 41st International Conference on Machine Learning, pp. 7935-7952. PMLR, July 2024. URthttps: /ttptpp:://rroeeeee http///oocednnnnge/ee ngs www.mllpppp. press// v235 / chene 4bn. html. ISSN: 2640-3498.
Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward Model Ensembles Help Mitigate Overoptimization. In The Twelfih International Conference on Learning Rep- resentations, 2024. URL https:/ https:://ppeeeeiiiiiee http///ppeeeee..eeeeeeeeee net/ forum? i jjddjtttMykpX
Fong, C. and Grimmer, J. Causal Inference with Latent Treatments. American Journal of Po- litical Science, 67(2):774-38,, 2023. ISSN
1540-5907. doi: 10.1111/ajps. 12649.
URL
com/
Crump, R., Hotz, V. J., Imbens, G., and Mitnik, O. Moving the Goalposts: Addressing Limited Overlap in the Esti- mation of Average Treatment Effects by Changing the Es- timand. Technical Report t0330, National Bureau of Eco- nomic Research, Cambridge, MA, October 2006. URL ht attpp:/ //t:://wwww http:/wwwrr.t/tt/////////////wtt org pttpp//pppss//3....//// pdf.
doi dbt:/abe/10..111//jps..222 http://ollleelleeaww.wwwooooo/lp
eeprint:
Ganin, Y. Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. Anvain-Advrrsariall Training of Neural Networks, May 2016. URL http:/ / waw.ia..vvo.oo...aaaaaa www./rggg/////ooooe/..o/e..e///o abs/15ls1 07818. arXiv: 150..07818 [stat].
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultra- Feedback: Boosting Language Models with Scaled AI Feedback, 2024. URL https: /ttp/aaxvvvvo///////ee org. /bs/ _eprint: 2310.01377.
Gui, L. and Veitch, V. Causal Estimation for Text Data with (Apparent) Overlap Violations. In The Eleventh International Conference on Learning Representations, 2023. URL /ttpttts:://ppeeeeeeeeee /oppnrevieweeeeeeeeeeeeeee net forum? id=Ha2Mn@M9Ph:
2310. 01377.
de Haan, P., Jayaraman, D., and Levine, S. Causal Confusion in Imitation Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: //apers.. http://appr_filllesspaeeeeeeeeee html.
Hahn, J. On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects. Econometrica, 66(2):315 1998. ISSN 00129682, 14680262, doi: 10.2307/2998560. URL http:/ //www. istor. jstor.org/stee Econometric Society].
Publisher: [Wiley,
Dekoninck, J., Fischer, M., Keuner- L.,, and Vechev, M. Controlled Text Generation via Language Model Arithmetic. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openrevi ew net forum? http::/ooorummmms1..1fffff f
Hendryeks, D. and Gimpel, K. Gaussian Error Linear Units (GELUs), 2023. URL https: / aaaaiiivvv........a.aaa org/ abs/ 1606. 08415. .ppiit:: 1606.08415.
Hirano, K., Imbens, G. W., and Ridder, G. Efficient Es- timation of Average Treatment Effects Using the Es- timated Propensity Score. Econometrica, 71(4):1161- 1189, 2003. ISSN 00129682, 14680262. URL http: Publisher:
Du, X., Sun, I., Duivesteijn, W., Nikolaev, A., and Pech- enizkiy, M. Adversarial balancing- representa- tion learning for causal effect inference with observa- tional data. Data Mining and Knowledge Discovery, 37(3):1713-1738, July 2021. ISSN 1573-756X. doi:
/ / /www. jstor. wwww.jstor [Wiley, Econometric Society).
10

---

Preference Learning for AT Alignment: Causal Perspective
Hu, Z. and Li, L. E. A Causal Lens for Controllable Text Generation. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural In- formation Processing Systems, 2021. URL https: //openreview net. forum? d= 95ROR.....
1214/17-S75612, Publisher: Institute of Mathemati- cal Statistics.
Luce, R. D. Individual choice behavior. Individual choice behavior. John Wiley, Oxford, England, 1959. Pages: xii, 153.
Kobalczyk, K., Fanconi, C., Sun, H., and Schaar, M. d. Few-shot Stecrable Alignment: Adapting Rewards and LLM Policies with Neural Processes, 2024. URL https:/ / //aarxivvv..o org/ httg///b//20 2412 1 _eprint: 2312.13998.
Maddison, C. J., Tarlow, D., and Minka, T. Aast Sampling. In Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL https://papers: nips. 309f 422lbbebb44112bbbb442424 -attaaatbbttt..... html.
Li, X., Zhou, R., Lipton, Z. C., and Leqi, L. Personalized Language Modeling from Personalized Human Feedback, December 2024. URL http:/ //s///aaaae arxiv. org/ abs/ 2402. 05133. 332.v:240..05133 [es].
Makar, M., Packer. B., Moldovan, D., Blalock, D. W. Halpem, Y., and D' Amour, A. Causally motivated short- cut removal using auxiliary labels. In Camps- G., Ruiz, F. R., and Valera, I. (eds.), International Con- ference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Leurning Research, pp. 739-766. PMIR, 2022. URL https:/ pwwppeooccidioeoooooooo ngs mmlr. press sesss/v151.m 1w1.5/mmaaa222.htt...
Liang, X., Wang, H., Wang. Y.. Song. S., Yang. J., Niu, S., Hu, J., Liu, D., Yao, S., Xiong, F., and Li, 7. Con- trollable Text Generation for Large Language Models: A Survey, 2024. URL https:tt//........ www.iv.org/aaa.. / 2408. 12599.
-eprint: 12599.
Lin, V., Ben-Miccael,, E., and Morency, L.-P. Optimizing Language Models for Human Preferences is a Causal Inference Problem. In The 40th Conference on Uncer- tainty in Artificial Intelligence, 2024. URL https: //openreview net/ feet/forume forum? wid::dddddimmmmmii 1 &QEOUK6.
Melo, L. C., Tigas, P., Abate, A., and Gal, Y. Deep Bayesian Active Learning for Preference Modeling in Large Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::///// http////peeeeeeeeeeeee/eeeeeeeee net/ forum? id=TADTT9ught
Liu, J., Zhu, Y., Wang. S., Wei, X., Min, E., Lu, Y., Wang, S., Yin, D., and Dou, Z. LLMs + Persona-Plug P alized LLMs, September 2024. URL http: //axivv. arxiv. org/ abss/a//// 11901. arXiv:2409..1999 1901 [cs].
Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Ac- tive Preference Learning for Large Language Models. In Salakhutdinov, R., Kolter, Z., Heller, K.. Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Ma- chine Learning Research, pp. 36577-36590, PMLR, July 2024. htt.ttttss://ppooeeeeeeeeee http://pocceeeenneeeeeeeee mlr press/ html.
Liu, T., Xiong, W., Ren, J., Chen, L., Wa, J., Joshi, R., Gao, Y., Shen, J.. Qin, Z., Yu, T., Sohn, D., Makarova, A., Liu, J. Z., Liu, Y., Piot, B., Ittycheriah, A., Ku- mar, A., and Salch, M. RRM: Robust Reward Model Training Mitigates Reward Hacking. In The Thirteenth International Conference on Learning Representations, 2025. URL https::////// http:////pppeeeeeeeeeeee wwwppeereeee..eeeeeoor..e.ee..ee net//orunf id=88AS5MMnmC.
v235
Oberst, M. and Sontag, D. Counterfactual Off-Policy Eval- uation with Gumbel-Max Structural Causal Models. In Proceedings of the 36th International Conference on Machine Learning, pp. 4881 4890. PMLR, May 2019. URL hwtph:tt///// htto:eeee////cceeeeeeeeooooe ngs ml r. press/v97/ oberst19a. ISSN: 2640-3498.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging Common As- sumptions in the Unsupervised Learning of Disentangled Representations. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma- chine Learning Research, pp. 4114-4124, PMLR, June 2019. URI. https:::::://// /ttp::////oceeeeeee ngs . ml r, press/ v97/1ocatello1 html.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wain- wright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F. Miller, I.., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training lan- guage models to follow instructions with human feed- back, March 2022. URL http: /arxiviivvv....i/ www.iiiwg///......... org/ abs/ 2203. 021 15. a2X [cs].
Lopez, M. J. and Gutman, R. Estimation of Causal Ef- feets with Multiple Treatments: A Review and New Ideas. Statistical Science, 32(3):432 - 454, 2017. doi: 10.1214/17-STS612. URL https: htt.//oo..oooog/////

---

Preference Learning for AI Alignment: Causal Perspective
Ozery-Flato, M., Thodoroff, P., Ninio, M., Rosen-Zvi, M., and Hl-Hay, T. Adversarial Balancing for Causal Infer- ence, September 2020. URL http:// /ttpaa.xivvvoo.o/. org/ abs/1810. 07406. urXiv:1810.000
634, May 2021. ISSN 1558-2256. doi: 10.1 10.1109/PROC. 2021.3058954. URL https: /ieeexpllrr feee. org/ hegg/documet//99 (eatmment/33330 /9639924. Conference Name: Pro- ceedings of the IEEE.
[cs].
Park, R., Rafailov, R., Ermon, S., and Finn, C. Dis- entangling Length from Quality in Direct Preference Optimization. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Findings of the Association for Com- putational Linguistics: ACL 2024, pp. 5998-5017, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/1/2222. indings-acll299 URL https: bttt:p::/aacaaathoe /ac/aaaaaannnyyy....... org/2024. findings-acl.2977
Singhal, P., Goyal, T., Xu, J., and Durrett, G. A Long Way To Go: Investigating Length Conrelations in RLHE, 2024. URL /tl.::::/pperevviii http:/neee/eeeeee net/ forum? id=sNtDKdcT1f.
Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. In The Twelfth International Conference on Learning Representations, / http///ppppeeeeeee//e/ee// net forum?
2024. URL ht i 1d=OtWTXXYPPM..
Pryzant, R., Card, D., Jurafsky, D., Veitch, V., and Sridhar, D. Causal Effects of Linguistic Properties. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 4095 4095-4109, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2211 naacl- 323. URL https: http://caaaathhoooooo/o org/2021, maa//021.aacall
Skalse, J. M. V., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and Characterizing Reward Gam- ing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. http:: owwnreview.eeee .net/ forum?id forum?id=yb3Hox03112
Splawa-Neyman, J., Dabrowska, D. M., and Speed, T. P. On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9. Statistical Science, 5(4):465 1666,-472,.10 1990. ISSN 08834237, 21688745. URL http: /www. jstor. org /t25222 Publisher: Institute of Math- ematical Statisties.
naac - maain m.t.ii..22... 3233.
Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., and Cord, M. Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine- tuned on diverse rewards, October 2023. URL http: // arxiv. org/ //0.. 04488. arXiv:2306.04488 [cs].
Tien, J., He, J. Z.-Y., Erickson, Z.,, Dragan, A., and Brown, D. S. Causal Confusion and Reward Misidentification in preference-Based Reward Learning. In The Eleventh International Conference on Learning Representations, 2023. URL https:: htttlp:://ppezeee apwnrevieneeeeeeeeeeeeeeee net/ forum? á d=ROXxvF_X324
Reber, D., Richardson, S.. Nief, T.. Garbacea, C., and Veitch, V. RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals, 2025. URL https:/ /ttes///axxii arxiv. org/ http//bs/24111 2410. 11348. eprint: 2130.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems, volume 33, pp. 12388- 12401. Curran Asso- ciates, Inc., 2020. URL neurips. httrrippsccc/paeee//20////// http://ppper/222///aaaa/aa 2tt.:///feeee222fffeaee22020 2e6fa -tw.tf/-ta/b0----eraaottttt Abstract.
Richens, J. and Everitt, T. Robust agents learn causal world models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openrevies net / forum? d
Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):11555, April 1983. ISSN 0006-3444. doi: 10.10993/bomee/70...44 URL https::///...........tt 1.41. _eprint: httpss:/accaeemiccoo wtt.://deeeccccooooooooo/m////te pdF770/1441/62955477011111
html.
Von Küigelgen, Identifiable Causal Representation Learn- ing: Unsupervised. Multi- View, and Aallii-Enviromment. PhD Thesis, Apollo University of Cambridge Reposi- tory, 2023. URL https: /wwww/wwo.ooooo.......... repository.....ooo cam. ac. / handlanndlee e/ 1810/365627.
Schölkopf, B., Locatello, F., Bauer, S., Ke, R., Kalchbren- ner, N., Goyal, A., and Bengio, Y. Toward Causal Repre- sentation Learning. Proceedings of the IEFE, 109(5):612-
Wang. F., Mo, W., Wang. Y., Zhou, W., and Chen, M. A Causal View of Entity Bias in (Large) Lan- guage Models. In Bouamor, H., Pino, J., and Bali, K.
12

---

Preference Learning for AT Alignment: Causal Perspective
(eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15173-15184, Singa-
pore, December 2023. Association for Computational
Linguistics. doi: 10. 10.18653w11/023.nndinng.. 1013. httpttts:: acc ant http:::://o//aaaoaoooooo..o.. o20/
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024. pp- 10582-10599 Association for Computational Linguistics, 2024. URL https: / www.thhowoo....oooo....oo mundings-emmn.n 620.
Wu, A., Kuang, K., Xiong, R., Li, B., and Wu, Sta- ble Estimation of Heterogeneous Treatment Effects. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th In- ternational Conference on Machine Learning, volume 202 of Proceedings of Machine learning Research, pp- 37496-37510. PMLR, July 2023. URL https: // proceedings mww. www..prees...... prwss 2w2/////w32222 html.
Wu, S., Fung, M., Qian, C., Kim, J., Hakkani- Tur, D., and Ji, H. Aligning LLMs with Individual Preferences via Interaction, December 2024. URL http:////// org/abs/
abs/ 2410. 03642. arXiv:2410.03642 [cs].
Xia, Y., Yu, T. He, Z., Zhao, H., McAuley, J., and Li, S. Aligning as Debiasing: Causality-Aware Alignment via Reinforcement Learning with Interventional Feed- back. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guisties: Human Language Technologies (Volume 1: Long Papers), pp. 4684-4695, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 12..16653/11222224aalllllloo 262. URL https:// naacl long. 262.
aclanthology, hal.anntooooy..rrg222222222 www.//24//222//
Zhang, X., Xiong, W., Chen, I.., Zhou, T., Huang, H., and Zhang, T. From Lists to Emojis: How Format Bias Af- fects Model Alignment, September 2024. URL http:// arxiv. org. abs 0abs/2409. 2409. 11704. arXiv:2409.1170 [es] version: 1.
13

---

Preference Learning for AT Alignment: Causal Perspective
A. Related Work
Causal Inference for Text Data and LLM Alignment. Causality has gained increasing attention in NLP, with several studies proposing methods for treatment effect estimation from text data, each with a different focus. Examples include the discovery of latent text attributes (Fong & Grimmer, 2016), the impact of unmeasured latent treatments (Fong & Grimmer, 2023), non- con-paammttric treatment effect estimation (Pryzant et al., 2021), and the robustness of such estimators (Gui & Veitch, 2023). These works predominantly address data analysis tasks where causality is used as interpretability tool. In large language models (LLMs), Vig et al. (2020) use causal mediation analysis to study gender biases, while Cao et al. (2022) analyse prompt-based probing from a causal perspective, Other works, such as Wang et al. (2023), introduce in-context causal interventions to alleviate entity biases in prompts, and Hu & Li (2021) adopt causality for controllable text generation. However, none of these directly apply to the pairwise preference learning setup explored in our work or the subsequent LLM fine-tuning stage via RLHF, Few notable exceptions addressing our call for adapting causal perspective for preference learning and alignment involve the work of Xia et al. (2024) who attempt to leverage pre-trained reward model as an instrumental variable for causal intervention on LLMs. Lin et al. (2024) draw on importance weighting and double robustness principles to present methods for more robust preference optimization in DPO. Reber et al. (2025) introduce a causal framework for understanding and evaluating spurious correlations of reward models. Butcher (2024) use counterfactual pairs to address spurious correlations during the aligmment process. Finally, Liu et al. (2025) propose a causal framework for learning preferences independent of response artifacts (e.g., length), assuming that prompt-independent features are spurious, and introduce a data augmentation technique to eliminate them.
Reward Hacking and Causal Confusion. Many prior works in reinforcement learning have studied reward hacking (Skalse et al., 2022), closely linked to causal misidentiiication (de Haan et al., 2019; Tien et al., 2023). This occurs when learned policies or rewards achieve high accuracy within their training distribution but fail to generalise to novel scenarios because the models do not correctly identify the underlying causal structure. In preference learning for AI alignment, this issue has primarily been explored in the context of response-speciffe biases, such as length formatting (Zhang et al., 2024) or response length itself (Singhal et al., 2024; Chen et al., 2024; Park et al., 2024), where authors demonstrate that learned reward models or downstream policies improve significantly by simply increasing response length, rather than considering other relevant features. Some evidence suggests that reward model ensembles can improve the robustness of LLM policies (Annervaz et al., 2018; Coste et al., 2024; Eisenstein et al., 2024), but they do not fully eliminate the reward hacking problem, leaving it an open challenge.
B. Identifiability Proofs
For convenience, we restate the assumptions enabling identifiability of E|L(z'y. y. s/)]. Assumption 1 (Consistency). For an individual with prompt-response assignment (X,Y,Y') we observe the associated potential outcome, i.e. L == LX;
Assumption 2 (Unconfoundedness). There are unobserved confounders, so that L(,,,,y)) IL (X,Y,Y"), for all E X,
v E V.
Assumption 3 (Unconditional Positivity). Treatment assignment is non-deterministie,, i.e. 0 P(X x, Y y, y y') < 1 for all E X and y. E y.
Proposition 1. Under assumptions (1), (2) and for all E X. y.v E v,
= = = =, ='],
E
so that observed statistical associations have a causal interpretation.
Proof.
E [[L
= =E(,,==::/==x,,,->>>=> =y)
(by positivity and unconfoundedness) (by consistency)
=E = =
14

---

Preference Learning for AT Alignment: Causal Perspective
As outlined in the main body of the paper, the assumption of overlap in the observational space can be replace with the assumptions of latent sufficiency and latent overlap:
Assumption (Latent Sufficiency). Assume there exists functions gx : " + z and g" ~ - zz such that for any
X. y, y y we have
R(X = 2, Y = -y= = =xx, xx,==²= = n
R'(X R'(x=x,Y x.Y xxx=x\-=-)> - = 2xx 77 2TTT;
= where 2x -x), =f 2 = = (x, (x,
Assumption 5 (Unconditional Latent Positivity). For all xxx xx and & _T= E Z
0 0< = _t Bj = T Z x <
Proposition 2. Under assumptions 2, 4, and
B[ = E L|2* xx ,2 22 20
for xx = (x)), 2T = = (x, v) and 28 = (x. v/)
of (x, x² =="(a, Since T, = s(R R,,U), where U is an independent exogenous noise variable² and due to the sufficiency condition, we have that:
Proof. Let E X, V. E y and let xx = gxx (xx), xT =
1) Consistency in the observational space implics consistency in the latent space, i.c. for an individual with prompt- response assignment (X, Y. Y') whose latent factors are (zx 7T, 7r²) = (a
(X, ()), we observe
the associated potential outcome, i.e. L = L(2x z², 2
2) Unconfoundedness in the observational implies unconfoundedness in the latent space, i.e. L(2x = xx; nT =T,27 ZT ) L( +==x;=T,,> / is independent of (7x nT, n
3) E [(=:,.v))] = = F [ =<<>,,")))
Thus, it follows that:
E [[ E L(s 2² 2T7]
(by sufficiency)
= 1[xx 2T, /T 12 = =xxx T. = T 7: 17
(by latent overlap & unconfoundedness)
E ||1x = xx zT = = TT = T
(by latent consistency)
C. Extended Discussion
C.1. Interpretability of Latent Factors
Aside from estimating the potential outcomes L(z, y, 3/) or rewards R(x, v) for any x E X y, y E V, we may wish to consider the average effects of their underlying latent factors. the individual components Z₁ of the set of latent factors Z enjoy hhmanhuitttnn citereiiterprettable meaning, we would like to predict how an intervention of one of these components (e.g. increasing creativity of an answer, its length) will affect user preferences while holding other features constant. If we assume that each of ' can be represented
binary value indicating the presence absence of particular feature, then
Thh re-expression of a function of R R' under the model is possible thanks to the Gumbel-softmax trick (Luce, 1959;
Maddison et al., 2014; Oherst & Sontag, 2019).
15

---

Preference Learning for AT Alignment: Causal Perspective
following the convention adopted by Fong & Grimmer (2016; 2023) such inferences are made possible by considering version of the Average Marginal Component Effect (AMCE) adopted to our pairwise preference learning setup:
AMCER := JZEZ-N
I6 [L(Z = 1, 2t =0 =, = ============= =2]m(2)dz,
(6)
where Z denotes the latent space Z except the k-th one and m(-) --) is some analyst-defined density this space, which can be taken, c.g. uniform distribution the distribution induced by the observable (X, Y, Y"). The sufficiency and latent overlap conditions imply that this measure is identifiable from observational data.
C.2. A Roadmap for Robust and Interpretable Alignment
Despite the appealing interpretability properties of the AMCE, similarly to direct potential outcome estimation, it assumes that the mapping 9 Σ" + 2 between the observable texts and their latent features is given and that it captures all factors that causally influence the reward, while disregarding any spuriously correlated features. In practice, the map needs to be learned from data and ideally. with minimal supervision-i.e. not requiring humans to label large numbers of prompt-response pairs according to a prohibitively long list of factors that can plausibly be causally linked to the outcomes. To enable learning of more robust and interpretable feature-extracting and reward predicting functions we outline some key directions for future research and data collection practices.
Collection of rationales, not only preference choices. Current preference learning frameworks typically collect binary comparisons between responses, but these do not reveal why particular choice was made. This lack of transparency obscures the true causal mechanisms driving user preferences and makes it difficult to disentangle spurious correlations from genuine reward-relevant features. Instead of solely collecting pairwise preference choices, future data collections practices could incorporate rationale elicitation, with users providing short explanations for their decisions. Such rationales could be used as weak supervision signals, guiding the learning of latent causal representations without requiring an exhaustive, predefined list of relevant features. This additional form of supervision could also for interpretable post-hoe causal analyses via causal estimands like the AMCE.
Active querying strategies and interventions. Much theoretical work has been done demonstrating how identification of causal representations requires auxiliary labels (Locatello et al., 2019; Makar et al., 2022; Brehmer et al., 2022; Ahuja et al., 2023). To gather such labels, rather than querying users at random points during the conversation and generating responses with unknown latent features, active querying strategies (Melo al., 2024; Muldrew al., 2024) can improve the efficiency and robustness of preference learning by selecting data points that maximize information gain. This applies to both feature identification (learning to predict ž's) and reward prediction (learning to predict rewards from ě's). Queries for rationales should prioritize instances where the model is least confident the feature-extracting part, reducing ambiguity of the learned representations. Reducing the ambiguity of the reward prediction part should benefit from interventional data collection so that given a prompt, two responses are generated that differ in isolation by a specific latent factors, allowing for direct causal attributions. Such interventions should also take into the account user-specific contextual variables to ensure that the latent treatments are well-balanced across different demographic subgroups.
Interpretable control over LLM-generated content. To enable targeted and interpretable interventions as described above, it is necessary that LLM -egeeeeee content can be explicitly controlled (Hu & Li, 2021; Liang et al., 2024; Dekoninck et al., 2024) to specify desired properties of text. Without such form of control, intervention- based preference learning becomes infeasible, as models would lack the ability to systematically vary latent factors. Language models should be designed to generate responses that explicitly vary along key latent dimensions, such as response verbosity or style, rather than letting these factors emerge implicitly.
Preference learning and optimisation remain an exciting field whose success necessarily relies on integrating multiple approaches, with causality playing central role. By moving beyond passive preference collection to rationale-aware learning, active data querying, and attribute-condition control, we can train models that are more robust, interpretable, and generalisable to unseen settings. This shift is crucial for aligning AI systems with human values while avoiding the negative effects of causal misidentification or confounding.
16

---

Preference Learning for AT Alignment: Causal Perspective
C.3. The connection to conventional treatment effect estimation
Our formalism follows the potential outcomes framework (Rosenbaum & Rubin, 1983; Splawa-Neyman al., 1990), but applied to preference learning rather than traditional treatment effect estimation.
In classical causal inference with binary treatments (e.g., drug trials), we define treatment assignment 7 {0, 1) and the observed outcomes Y, as well as the potential outcomes under treatment (Y (1) and potential outcomes under no treatment (O)).. We then aim to estimate treatment effects defined as: (1) - Y())). The fundamental challenge of causality lies in the fact that for a given individual we only observe one outcome, i.c. if their treatment assignment = t, then we observe (t), but not the counterfactual outcome.
Our framework extends this to preference learning by treating each prompt-response pair (1r, y, v') as distinct "treatment", Here, the tuple (X, Y, Y') is the random variable representing the treatment assignment, and Lxx; y²) represents the potential preference outcome when (X,Y.Y) traditional causal inference, we only observe the outcomes for the assigned texts and not all possible texts-a given user is only shown one (or at most finite subset of) possible prompts and responses. The observed label, defined as I,, satisfies the consistency condition so that (X, Y = (I, y, y'), then = L(x; y: y'). The key difference from conventional treatment effect estimation that while drug studies typically focus on binary treatment effects, we operate in an extremely nigh-dimeesional space of natural language where each (x, Ms y') combination effectively unique treatment. Rather than computing pairwise contrasts between all possible treatments,
(z, y, 1/) i.e. when the user observes (Ir, 3/) However, just as in
focus the expected potential preference E[L(x;, y, y ] for any given tuple (ar, y. 1/)). This is analogous to estimating (1) for each treatment / in multi-arm trial, which in turn enables making relative comparisons (t, (t2)) between different treatment choices t1, t2. In our framework, for instance, we could compare E (xx, yo: y/)l against [L, yo: 32)] to study how the expected preference change if the and the first response = Y/u
second observed response Y' is set to // instead y/1 while keeping the prompt fixed. Given the vast space of possible relative comparisons we direct our attention just the expected potential outcomes, (x, y. ). rather than their pairwise differences.
17

---

Preference Learning for AT Alignment: Causal Perspective
D. The UltraFeedback Case Study
D.I. Case Analysis
Suppose that the latent factor Z determining the rewards are two dimensional taking values in I2 (c.g. Z may correspond to the extent of truthfulness and bosllottionfollooingg of a candidate response). Suppose that the distribution of X's Y's and Y is such that Y TL y and Y, Y' are identically distributed given X. Further, assume that ((X,YYY) induces
distribution P(Z, Z') such that Z, Z std N E). As a result, we have:
8 8 - z ~ N [ 01
2 02 0.).
where WLOG 01 = 02 =
Now, suppose that the reward function is linear in Z1 and Z2, so that for all x E X, y E V we have
r(x, 3) = r(z) = 0z1 + (1 - 2) for a E [0,1].
(8)
Then, the preference label is completely determined by the vector 5. With 81 Z Z and 52 Zz Zzz we have that for any samples for which ad) + (1 a²)2 > 0 the first option is preferred, and vice versa. This can be visualised in the (8i, -plane, as in Figure? 7.
p 01.9
2
Decision boundary
= 0.75
0.5
0.25
0
-2
Figure 7: The impact of p on determining â and the classification accuracy.
The decision boundary determining the preference label L is dependent on the ground-truth value of a. It is defined by straight line with a slope of -h and the intercept at the origin. In a noise-free setting, examples (x, U.V', E) for which o falls below the decision boundary are labelled with f = 0 (i.e, the first option (z, y) wins) and all samples above the decision boundary have the label i = (i.e., the second option (x,yy) wins). An optimal fitted reward function f(z) = âz1 + (1 - 2² is s.t. à = a. Samples falling into the first and third quadrants of the (ō1, d2)-nlane are classified correctly no matter if the fitted decision boundary (determined by â) aligns with the ground- -tuth boundary (determined by a). Samples falling into the second and fourth quadrants (red-shaded region) are prone to misclassification, if the fitted and ground-truth decision boundaries diverge. If the training data exhibits high correlation (see the plot with p 0.9) few samples fall into the red-shaded region (the probability of this event is precisely 1 arcin(p)) making á sensitive to outliers and exhibiting higher variance than for non-correlated samples.
2
D.2, Experimental Details
Code for reproducing the experiments is made available at:
To emulate the setting described above, we rely on the UltraFeebback dataset (Cui et al., 2024) containing prompt-response pairs scored according to the objectives of honesty, helpfulness, truthfulness and instruction-follovi each represented as scalar values between 0 and 5. For simplicity, we focus on just the last two features: truthfulness and instruction-follove following, as the ground-truth causal factors determining the rewards.
Datasets. We create a large dataset of candidate pairs (x, y, ,,, 24, {L),, where 2 and '' are 2-dimensional vectors corre- sponding to the truthfulness and cnstructtio-followvii scores of the of the candidate options (x, y) and (ar, y"), respectively.
18

---

Preference Learning for AT Alignment: Causal Perspective
The label E is defined based on the value of rfr r(a,y'), with
r(a, r(=) = a21 + a1=a)22.
(9)
where we set a 0.25. For examples with r(z, y) r(z, v/) > we set f = 0. for examples with r(z, r(x, y/) < 0 we set é = 1 and for those with r(x, v) - r(x,,'())=) = 0 we sample l at random. Based on this large data set containing all prompts and responses in the UltraFeedback dataset we create 4 training subsets, controlling the value of the correlation p" between 21 - z and 22 - zzz where 21, 21 represents the truthfulness score and #2, 2/2 the cnstruction-oolowing score. Truining: With stratified sampling based on the value of 2 - =', we create 4 training datasets with 15.000 samples, one for each value of pT E {0.0, 0.3, 0.6, 0.9}. The resulting training sets contain tuples (ær, y. y. the latent factors Validation splits used to determine the optimal stopping point during training of the reward models are samples according to the same distribution as the training datasets and contain 2.000 samples. Testing: For each value of prr we also create a testing dataset with matching value of the correlation coefficient (ID), and a testing dataset with only samples falling into the second and fourth quadrants in the (61, -pl-ne, resulting in a correlation coefficient of -0.8 (OOD). The testing datasets are always disjoint from the training examples and contain 15.000 samples.
D () ground- truth values of not available during training and need to be approximated in an unsupervised fashion. Validation:
Reward model training. We fit standard BTL model by passing each prompt-response pair (zz y) through an LLM to obtain its embedding e. We do not perform L.M fine-tuning and simply fit the reward models on the pre-computed embeddings. In this experiment we use embeddings of the Llama-3-8B* model. The LLM embeddings are processed with 3-layer MLP with hidden dimension 512 and an output size of 64. The last linear layer maps from the 64-dimensional latent embedding to scalar value representing the reward. All models are trained by minimising the empirical estimate of the negative-loglikelihodd loss according to the BTL. model:
CR
Σ
logo ("o(x,"")rre(r,""))
(10)
(x..y""
where y" vf stand for the winning and loosing responses, respectively. All models are trained using the Adam optimiser with learning rate of le-4 for 10 epochs. Model weights 9 with the highest validation accuracy are saved for evaluation. For each value of p we train models with 3 random seeds.
E. The HH-RLHF Case Study
E.1. Experimental Details
Code for reproducing the experiments is made available at: http://shhiooooookkeooooooraaaas
Dataset. We rely on the extended version of the HH-RLHE dataset (Bai et al., 2022) provided by Siththaranjan et al. (2024). The entire dataset can be represented
tuples (t, v.y', c, e), where E {0, denotes the objective with which the choice f E {0, 1} is made and t E {0, 1} denotes the type of x, i.e. whether (z, v.z²) was originally part of the helpful (1 = 0) or harmless split (t = 1). In the original data (Bai et al., 2022) we only observe examples with t = c. Siththaranjan et al. (2024) augment this dataset with counterfactual labels for such that f * c which we refer to as inconsistent samples. We create six independent training datasets, controlling the ratio of consistent to inconsistent samples, Le. the parameter p = P(type(X) = C) E {0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. The resulting training datasets consist of 30.000 samples (z,, y, v C, f), with the label t not being part of the training sets. We also create validation splits with the same values of p's of 6.000 sample. The remaining 46518 samples is left for testing.
Models. We train three different versions of multi- multi-objective BTL models. Each model has the same pre-pot pre-processing backbone. Prompt-response pair (xc, y) are passed through an LLM to obtain its embedding e. We do not perform LLM fine-tuning and simply fit the reward models on the pre-computed embeddings. We use embeddings of the Llama-3-8B' model.
The embeddings e are concatenated with the objective label and passed through 3-layer MLP ra : Rdd - R with a hidden dimension of 512 and outputting the predicted scalar reward f. The model is trained by finding parameters 0 that maximise the log-likelihood under the BTL model as in (10).
Base:
Multihead: The embeddings are passed through 3-layer MLP go Rd + 1ss2 with a hidden dimension of 512 mapping them to a latent representation 2 E 2512 Depending on the value of c, the vectors 2 are then passed to one of
* huggingface..c/mmeteeteee Hama//Mea ---
19

---

Preference Learning for AT Alignment: Causal Perspective
the prediction heads wo 12 + R f1 R R. Here, let Jo and f1 be layer MLPs with hidden dimension of 512. Thus the reward function can be defined as:
(wo(go(E)))) v)) fwv (ge( v))
if 0 =
T0 y..
(11)
The model is trained by finding parameters (0, wy, w1) that maximise the log oog-likelhood under the BTL model analogously (10); replacing rp with "0.
Adversarial:
The adversarial model adds an additional network h& R512 + R on top of the multihead architecture, mapping from unnormalised log probabilities C Here, we let hy be 1-layer MLP with hidden dimension of 512. The model is trained under the adversarial objective:
min max (O, wo; wi) - ALnav(0, 0,
(12)
where (O, wo, w1) is the negative log-likelihood under the BTL model of the reward function re,wo,wi in the Multihead model and Lady is the binary-cross entropy loss computed across all prompt-response pairs-i.e., both for (xr, y) and (x,,y") within each sample (Ir, v. n l):
Ladv (od (6,0) =-
Σ
he h+(3)c+ + -1a(3))(( - c) + (3') + (1 - ha(()))()1 - c),
(13)
(e)
where - go (z, y) and = go (0, y/). The trade off between the two losses controlled by the hyperparameter which in our experiments we set to 1.0. The min-max optimisation problem is implemented with the gradient reversal technique (Ganin al., 2016).
All MLPs are implemented with GELU activation functions (Hendrycks & Gimpel, 2023).
Training. All models are trained using the Adam optimiser with a learning rate of 1e-4 for 10 epochs. Model weights with the highest validation accuracy are saved for evaluation. For each value we train models with 5 random seeds.
E.2. Results
Table shows the test-time accuracies of all models trained on datasets with varying values of p. Table shows the accuracies on the training sets at the training step corresponding to the best model performance on the validation set. As discussed in the main body of the paper, the Base model exhibits strong overfitting. The Multihead architecture mitigates this an extent, with the additional Adversarial objective bringing further improvements on the inconsistent samples.
Lype(X) = c
False
True
model
Base
Multihead
Adversarial
Base
Multihead
Adversarial
0.5
60.0
0.2
62.9 0.2
0.2
66.0 0.1 0.1
66.3 0.1
0.6
59.6 0.1
0.1
6.1
0.1
0.1
6.13
0.7
0.1
6.1
0.1
6.2
0.1
0.1
0.8
0.17
0.1
6..
6.27
6.29 6.26
0.9
0.1
5.1
58.9
6.22
6.24 6.2
1.0
54.9
53.8
53.5
64.1 0.2
67.9 0.2
68.4 + 0.2
Table 2: Test accuracy [%] across all model architectures trained on datasets with varying values of p.
20

---

Preference Learning for AT Alignment: Causal Perspective
type(X) C
False
True
model
Base
Multihead
Adversarial
Base
Multihead
Adversarial
p
0.5
73.5 + 0.2
72.2 0.2
0.2
74.1 0.2
74.8 0.2
74.2 + 0.2
0.6
0.2
0.2
7.2
73.1 0.1
0.1
75.8 + 0.1
0.7
0.2
0.2
68.0
0.1
0.1
72.7 0.1
0.8
0.3
6.3
6.3
0.1
0.1
74.7
0.9
73.7
0.4
0.4
6.42
8.1
0.1
74.6 0.1
1.0
7.1
0.1
0.1
Table 3: Training accuracy [%] across all model architectures trained on datasets with varying values of p.
21


Reviews:
[{"review_id": "JR3cGLWaSY", "reviewer": "Reviewer_1738", "summary": "This paper introduces a causal framework for preference learning in AI alignment, aiming to improve the robustness of reward models. Reward modeling from preference data is a crucial step in aligning large language models (LLMs) with human values. The authors propose integrating causality into reward modeling, arguing that this approach enhances model robustness.\n\nA key challenge in preference learning is that pairwise preference datasets are often collected opportunistically. LLM users both evaluate model responses and generate prompts, introducing individual-specific confounders that bias the learning process. The paper illustrates this issue with examples and experimental results, demonstrating the vulnerability of naïve reward models to OOD (out of distribution) samples.\n\nTo address these challenges, the authors propose controlled, randomized experiments, where prompt-response pairs are allocated randomly across a representative population. Additionally, they suggest that preference data collection methods can be improved to infer user-specific objectives through: Explicit feedback: Users provide rationales for their preferences, offering richer insights into their underlying objectives and Contextual information modeling: Incorporating external context to refine preference interpretation. This causal perspective provides a more systematic and reliable approach to reward modeling, ultimately strengthening AI alignment with human values.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time taken to evaluate our work. We your recognition of our work as a meaningful addition to the broader literature. Below we address your comments:\n\n**Presentation.** In the camera-ready version, we will use the additional space to clarify key intuitions, especially for readers less familiar with causal representation learning, and include citations to introductory causality resources. Thank you for this suggestion.\n\n**Q1: Treatment effects.** Thank you for your question regarding our causal framework. Due to space constraints, we have provided an intuitive explanation [here](https://imgur.com/a/YqzO5KG).\n\n**Q2: Evidence.** Thank you for prompting us to substantiate the claim about data collection practices. We are happy to provide supporting evidence:\n\n1) the widely-used Anthropic HH-RLHF dataset provides concrete evidence of this. As described in [2], the dataset construction process explicitly involved users both writing prompts and providing preference judgments on LLM’s responses.\n\n2) OpenAI's ChatGPT interface empirically demonstrates this collection approach. Users are frequently presented with pairs of responses to their own queries and asked to indicate preferences. OpenAI's privacy policy confirms these interactions are used for model improvement. \n\n**Q1: Bandits.**  Thank you for the suggestion to consider contextual dueling bandits. However, we clarify that the confounder in Example 1 is the unobserved user-specific variable $C$, which influences both the prompt $X$(during dataset generation) and the preference labels **$L$**—not $X$ itself. Secondly, contextual dueling bandits focus on optimising a selection policy for choosing responses **$Y,Y'$** given the observable context $X$ (the prompt), whereas in our setup, these responses are passively sampled from an LLM, whose policy we do not control. Our work focuses on learning a reward model from static, observational data. While applying a contextual dueling bandit framework to jointly learn preferences and control the sampling policy is an interesting direction (e.g. [3]), our work focuses on biases in preference learning from observational data rather than on designing a strategy for sampling the candidate responses. Furthermore, even in a dueling bandits, preferences are typically modelled with the BTL model, which as discussed, does not account for confounding. If an unobserved $C$ influences both the prompts $X$ and preference labels $L$, the learned preference model would remain biased.\n\n**Q2: Example 2.** We would like to clarify that Ex. 2 is not intended to prescribe a fixed functional form for the reward model for learning. It is an illustration showing why conditioning on prompt-specific features ($Z^X$) is essential. The provided equation exemplifies a possible structure of the ground-truth reward function to highlight the plausibility of the effects of $Z^X$ being non-additive.\n\nIn practice, to learn a reward model we would not assume knowledge of the parametric form or the set of relevant latent features. Instead, as discussed in the paragraph *Discovery of Z*, the latent factors are learned from data. This flexibility, however, induces challenges associated with causal discovery under minimal supervision [4, 1], including overfitting to spurious correlations, as illustrated by our case studies. \n\n**Q3: The UltraFeedback experiment.** This experiment highlights the impact of the latent overlap assumption—when strictly violated, causal identification is impossible. We show that weaker overlap makes the recovery of the ground-truth model significantly more challenging. This experiment highlights the inherent limitations of learning from observational data only, rather than promoting a specific approach.\n\nUltraFeedback was chosen for its unique structure, allowing control over correlations between latent features. Unlike other datasets like Stanford Human Preferences or HH-RLHF, which label instances based on a single objective, UltraFeedback scores each prompt-response pair across four latent factors.\n\n**Q4: DPO.** Thank you for this suggestion. While DPO bypasses the explicit reward modelling of RLHF, it defines an *implicit reward function* by shaping the policy’s logits to match preference probabilities according to the same BTL model as used in RLHF. Consequently, we expect DPO to inherit the same generalisation failures in OOD settings. While an empirical DPO comparison could offer additional insights, we prioritised analysing reward modelling, as RHLF remains the predominant approach in alignment and DPO is based on the same underlying assumptions. \n\n---\n\nThank you for your valuable feedback, which helped us improve the paper.  We hope our answer resolves your concerns, and we’d be grateful if you could reconsider your initial rating to help us make a meaningful addition to the literature.\n\n[1] arxiv.org/abs/2102.11107\n\n[2] arxiv.org/abs/2204.05862\n\n[3] arxiv.org/abs/2402.00396\n\n[4] arxiv.org/abs/1811.12359", "claims_and_evidence": "Please refer to the Questions For Authors Section.", "methods_and_evaluation_criteria": "Please refer to the Questions For Authors Section.", "theoretical_claims": "I did not find any apparent errors in the author's mathematical derivations. However, the theoretical content in this paper is relatively light. I reviewed the derivations of Proposition 1 and Proposition 2, and they appear to be correct. That said, I recommend that the authors provide better citations to relevant literature and more discussion on key intuitions, especially for readers unfamiliar with causal representation learning.\n\nFor instance, in Proposition 1, it would be helpful to clarify why the expectation in Line 149 suggests that observed statistical associations have a causal interpretation. Providing additional context or references could strengthen the argument and make the results more accessible to a broader audience.", "experimental_designs_or_analyses": "Please refer to the Questions For Authors Section.", "relation_to_broader_scientific_literature": "It opens new research directions, sparks discussions on integrating causal frameworks into reward modeling to enhance robustness, and provides valuable guidance on observational data collection processes. I believe this is a meaningful addition to the broader literature.", "essential_references_not_discussed": "Please refer to the Questions For Authors Section.", "other_strengths_and_weaknesses": "Strengths:\nPlease refers to the Relation To Broader Scientific Literature section.\n\nWeaknesses:\nPlease refer to the Questions For Authors Section.", "other_comments_or_suggestions": "Please refer to the Questions For Authors Section.", "questions_for_authors": "1. I did not fully understand the counterfactual outcome associated with not receiving the (X, Y, Y') treatment. Could you provide an intuitive example? For instance, in the case of drug treatment effects, the counterfactual outcome represents the difference in health status of the same individual under treatment versus without treatment. Typically, we compare the observed outcome under treatment with the hypothetical (counterfactual) outcome the individual would have had if they had not received the drug. However, in this paper, it seems that the author defines the expected outcome L(X,Y,Y′) as the counterfactual outcome. This approach is not intuitive to me—could you clarify the reasoning behind this definition?\n2. Lack of Supporting Evidence\nIn Line 14, the authors state:\n\"Moreover, pairwise preference datasets are often collected opportunistically, with LLM users both evaluating model responses and generating the prompts eliciting them.\"\n\nCould you provide references to studies that actually collect data in this manner? This is a key claim, as the authors argue that current practices in preference data collection are problematic. However, without supporting evidence, it is difficult to assess the validity of this concern. I would appreciate citations or empirical examples to substantiate this argument.\n\nQuestions on examples and case studies\n\n1. Example 1 shows the existence of confounding X. But why not model it as contextual dueling bandits? Such as the modeling of Nearly optimal algorithms for contextual dueling bandits from adversarial feedback?\n2. Example 2, the latent effect model, is fine. However, I am uncertain about how this approach will balance model misspecification and robust generalization. This modeling framework seems to assume precise knowledge of the reward model, which raises the question: does this approach truly enhance robustness? Could the authors clarify how the proposed modeling method mitigates misspecification while still ensuring robust generalization? Since in real world, model misspecification will be very common and it is possible that overfitting performs even better than identifying \"wrong\" or \"incomplete\" casual relationship.\n3. Table 1 highlights the issue, but I find the result unsurprising—OOD (out-of-distribution) performance degradation is expected. However, is there any evidence that modeling reward causally can actually alleviate this problem? Additionally, does similar behavior occur in other datasets, or is this effect specific to the dataset used? Could you clarify why UltraFeedback was chosen for evaluation? A broader justification or comparison with other datasets would strengthen the argument.\n4. Case Study Results. I believe it is crucial to include experimental results for Direct Preference Optimization (DPO) as well. My main concern is that reward modeling is not necessarily required for effective preference learning. If DPO is used instead of reward modeling, would the same concerns about robustness and generalization still apply? Including a comparison with DPO would provide a stronger evaluation and help clarify whether reward modeling is essential in this context.\n\nI am willing to reevaluate my rating if all my questions are clearly addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z75X3BZYWI", "reviewer": "Reviewer_sdkh", "summary": "This paper uses a causal framework to articulate several assumptions commonly made when reward modeling from preference data. Namely, users are modeled as having implicit rewards which they assign to each response: hence, a preference label is formalized as being a function of these two potential rewards. The paper then investigates what assumptions are sufficient for the Bradley-Terry-Luce model to identify user preferences.\nThe first three assumptions are typical in causal inference (consistency, unconfoundedness, and positivity/overlap). Positivity is weakened to latent-overlap under an additional assumption of latent sufficiency. The paper suggests that these assumptions are not satisfied in practice by appealing to high-level, hypothetical examples (e.g. LLM responses aren’t expected to be simultaneously informal and professional).\nThere are two sets of experiments. Firstly, the practical relevance of the latent-overlap assumption is demonstrated by showing that linear reward models fail to generalize OOD on the UltraFeedback dataset. Secondly, the second suite of experiments demonstrates that confoundedness is by default an issue on the HH-RLHF dataset which can be addressed by careful architecture design when training the reward model (especially via a causality-inspired adversarial multi-objective reward model).\n\n## update after rebuttal\nThe authors addressed my concerns to my satisfaction. I particularly appreciated the final rebuttal comment which elaborated on how a reward model could overfit conditionally to the confounder in \"multi-objective\" datasets. Hence, I am convinced that this really is saying something interesting about realistic training setups where many datasets are used during post-training: the key insight is that the way in which these datasets are mixed is itself providing information on which the reward model can overfit!\n\nI am raising my score to 4, with the expectation that the authors will include these details in particular in the main body or appendix.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful evaluation of our work. We appreciate the time and effort you put into assessing our paper. Below, we address your questions and comments point by point.\n\n**Presentation of assumptions** We acknowledge your concern regarding the potential misinterpretation of our assumptions as necessary rather than sufficient. **ACTION:** We will make appropriate adjustments in the phrasing in the revised version of our paper. In particular, we suggest that the first paragraph of section 3.1 is instead formulated as:\n\n> Causal inference provides a framework to answer counterfactual, 'what if' questions even when only observational data is available. A key part of causal analysis involves ensuring that the causal quantity of interest can be estimated from observational data alone. The following commonly made assumptions, adapted to our preference learning setup, are sufficient to guarantee identifiability and non-parametric estimability:\n> \n\n**The multi-environment setup** We greatly appreciate your suggestion to consider multi-environment setups and acknowledge their importance in broadening identifiability guarantees. **ACTION:** To further emphasise the fact that the assumptions discussed in our paper are sufficient rather than necessary, we will elaborate that alternative setups, such as having access to data from multiple environments, under certain conditions may also enable identifiability. We will reference the related literature you highlighted and contextualise it within multi-dataset reward model training. While learning reward models from multiple datasets is an emerging research area, its causal implications warrant further investigation. We view extending our framework to multi-environment settings as a promising avenue for future work.\n\nThat said, our case study in Section 4 underscores a fundamental challenge in merging datasets collected under divergent objectives. Specifically, the HH-RLHF dataset comprises two independent subsets: the helpfulness subset and the harmlessness subset, which can be interpreted as distinct environments. Our experiments highlight potential pitfalls in training a multi-objective reward model when datasets exhibit distributional differences.\n\nWe acknowledge the relevance of investigating our results in larger-scale multi-environment settings, such as those considered in ArmoRM. However, the datasets involved in ArmoRM lack counterfactual preference labels—i.e., they provide preferences only under their respective training objectives but do not indicate how preferences would transfer across datasets. The lack of overlap in objectives between the datasets of ArmoRM makes it difficult to assess generalisation beyond the dataset-specific correlations present at training time. In our experiments, the assessment of the counterfactual scenarios is possible thanks to the provision of additional, counterfactual labels to the HH-RLHF by [1]. The presented results exemplify potential risks associated with training multi-objective, steerable reward models from a composition of datasets collected under different, potentially conflicting objectives. We hope this motivates the development of new datasets tailored for robust reward model training.\n\n**Related work & contributions** We appreciate your pointers to related work. **ACTION:** we will incorporate the suggested references into our discussion of related literature. Additionally, we would like to point out that our contribution extends beyond articulating sufficiency assumptions for preference identification. A second key contribution of this work is the highlighting of the potential confounding effects due to user-specific covariates.\nWhile there indeed exist prior works that aim to infer user-specific covariates (e.g., [2, 3]), they do not examine the confounding issue.  The implicit assumption of prior works is that the covariates $C$ only influence the preference label $L$ but not the prompt distribution $X$. Our work considers cases where $C$ influences both $X$ and $L$, and thus it acts as a confounder. This is the case when users write the prompts $X$ and subsequently score the LLM’s generated responses to these prompts, which may in practice be the case (e.g. the HH-RLHF dataset).\n\n**Minor corrections** Thank you for catching the typo on line 20, we will correct it!\n\n---\n\nWe are grateful for the opportunity to refine our work based on your thoughtful feedback. Your suggestions have helped us improve the presentation clarity, depth and broader impact of our research. We hope our explanations and proposed revisions address your concerns. Thank you for your time and consideration.\n\n[1] arxiv.org/abs/2312.08358\n\n[2] arxiv.org/abs/2402.05133\n\n[3] arxiv.org/abs/2409.11901", "claims_and_evidence": "I worry that due to the presentation, many readers will misunderstand the significance of the assumptions in this paper as “necessary” when all that is shown is that they are “sufficient” for identification. Propositions 1 and 2 merely show that making these assumptions are sufficient, not necessary. The experiments show that under a particular training setup issues arise, but do not attempt to assess the scope of these issues “in the wild”: the experiments only demonstrate that misidentification can occur when these assumptions are violated on two datasets, the UltraFeedback dataset and HH-RLHF.\nIn particular, this paper should at least consider alternative assumptions which could provide identifiability guarantees while being more tenable for real-world preference learning setups. Specifically, consider the alternative assumption of “having access to observational data from multiple environments”, for which there is a rich literature demonstrating that there is rich signal (for both the causal identification task of Section 3.1, and causal representation learning task in “discovery of Z” in line 255 (right)). For instance, “Robust Agents Learn Causal World Models” https://openreview.net/pdf?id=pOoKI3ouv1, “Interventional Causal Representation Learning” https://arxiv.org/abs/2209.11924, Chapter 5 of “Identifiable Causal Representation Learning” https://arxiv.org/pdf/2406.13371, to name a few. This multi-environment assumption is more likely to be satisfied (thus providing identifiability) for leading reward models which train on multiple datasets, unlike the experiments here which only use singular datasets. This undermines the warnings for practitioners in e.g. Section 3.2.1 and Section 4 of this paper.", "methods_and_evaluation_criteria": "As discussed above, the experiments conducted here suffice for demonstrating that issues can arise, but are insufficient for claiming that these issues are *actually* observed for leading reward models and aligned LLMs. In practice, reward models are not trained on only a single dataset (as is the implicit assumption in the experiments). \nUnder the multi-environment assumption (that is, training a reward model across many different datasets simultaneously, as is done for leading reward models such as RLHFlow/ArmoRM-Llama3-8B-v0.1 https://arxiv.org/abs/2406.12845 which was trained on HelpSteer, UltraFeedback, BeaverTails-30k, CodeUltraFeedback, Prometheus, and various Argilla datasets), it seems quite possible that replicating the experiment in Section 4 will not yield the same negative results, even when inducing strong correlations in each dataset individually.\n\nIn summary: how much are the experimental results of Section 4 simply due to the restriction to single datasets?", "theoretical_claims": "I checked the proofs of Propositions 1 and 2, which look fine: they use standard proof techniques from causal inference.", "experimental_designs_or_analyses": "Please note how the experimental results in Section 4 do not localize confounding alone, as they also implicitly make a single-environment assumption. Do these results still hold even when models are trained on multiple datasets, as specified above?", "relation_to_broader_scientific_literature": "The causal assumptions and latent discussion are well-established approaches in the literature on causal representation learning. \nI am not aware of any work which applies it to reward models in precisely the way done in this paper. There are somewhat close papers, namely 1. “RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals” https://arxiv.org/abs/2410.11348v2, which introduces a causal framework for understanding and evaluating the spurious correlations of reward models regardless of whether they identify the original user preferences and leverages a similar latent variable framing and 2. “Aligning Large Language Models with Counterfactual DPO” https://arxiv.org/abs/2401.09566 which uses counterfactual pairs to address spurious correlations during the alignment process.\nGiven this context, the main contribution of this paper is in clearly articulating several sufficiency assumptions for identification of preferences. The assumptions themselves are unsurprising to anyone versed in causal inference, but may be new to reward model developers. (Which is precisely why it is important that this paper address that the assumptions here are sufficient, not necessary, and also highlight competing assumptions like multi-environment settings).", "essential_references_not_discussed": "Line 173 (left): “To the best of our knowledge, confounding due to user-specific covariates has not been addressed in prior works.”\n\nThere is lots of work which view language models as inferring latent user attributes. e.g. “Language Models as Agent Models”: https://arxiv.org/pdf/2212.01681. Since in practice leading reward models are almost always fine-tuned LLMs, any work that discusses user-specific covariates in the context of LLMs is relevant for reward modeling.", "other_strengths_and_weaknesses": "The paper is clearly organized, and can serve as a good introduction for practitioners not already familiar with causal representation learning. \nThe adversarial multi-objective reward model was neat.\n\nWeaknesses have already been sufficiently noted elsewhere in this review.", "other_comments_or_suggestions": "Typo: line 20 (left) should be “causal” not “casual”.", "questions_for_authors": "Do the results of Section 4 still hold under a multi-environment assumption, that is, even when models are trained simultaneously on multiple datasets, e.g. HelpSteer, UltraFeedback, BeaverTails-30k, CodeUltraFeedback, Prometheus, and various Argilla datasets as used for leading real-world reward models? Leading reward models are not trained on single datasets anymore, so it is not clear that the base model’s worse performance is due to confounding (as claimed) or of only using a single environment.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nvr5if6AUT", "reviewer": "Reviewer_tb1d", "summary": "- This paper introduces a causal framework for preference learning in AI alignment, specifically focusing on reward models trained on LLM prompts and response pairs. The authors frame prompt-response-response tuples as treatment variables, with latent rewards for each prompt-response combination serving as mediators for the observed binary preference label (the outcome variable).\n- Reward models aim to learn to predict these latent rewards as a function of text, essentially estimating a treatment effect. However, these models may fail if they don't account for the underlying causal structure. The authors particularly focus on cases where contextual variables also affects the reward (and in some cases, the prompt too). For example, if a user's domain knowledge determines both the prompt and their assessment of responses, this creates confounding that precludes causal interpretation without additional assumptions. The paper draws on standard causal inference techniques and insights from causal representation learning to identify these necessary assumptions.\n\n## update after rebuttal: the authors have addressed all of my concerns and suggestions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful review. We appreciate the overall positive evaluation of our work. Below we address your comments and concerns:\n\n**The Multihead architecture** We appreciate your suggestion, **ACTION:** we clarify how the Multihead architecture incorporates the causal structure in the camera-ready version.\n\n**Additional References** Thank you for your suggestion regarding the additional references. **ACTION:** We suggest including [1, 2, 3, 4] as additional references strengthening our claims on page 5 and Appendix C.\n\n**Overlap/Positivity.**\n\n- **Terminology:** Preference learning conventionally assumes homogeneity of user preferences, so we stated identifiability assumptions in the *unconditional* case; this also reduces the notational clutter. However, we acknowledge that *overlap* is usually used conditionally. **ACTION:** We propose to rename Assumption 3 as *Unconditional Positivity* and Assumption 5 as *Unconditional Latent Positivity*. We will also clarify that we use *positivity* as shorthand for the unconditional case, while *Conditional Positivity (a.k.a. Overlap)* applies when user-specific covariates are considered (e.g., Example 4, Section 4).\n- **Sufficiency**:  Indeed, positivity is not necessary for identification but is required for non-parametric estimation as shown in the proofs of Proposition 1 and Proposition 2.  Thank you for pointing this out. **ACTION**: We will adapt the presentation of these results to make this clear.\n- **Statistical vs. causal confusion**: We agree with the reviewer that the UltraFeedback experiment primarily addresses statistical generalisation. The setup does not strictly violate the positivity condition, making the problem solvable in the limit of infinite data. Yet, a perfect correlation is not expected to occur in the real world so we focus on the practical challenge of *limited* latent positivity. It has been shown that near-violations of positivity inflate estimator variance and necessitate impractical amounts of data for robust inference [5, 6, 7]. Thus, while the observed performance degradation is a statistical issue, it stems from the weak satisfiability of the causal positivity assumption. **ACTION:** We appreciate the feedback and agree that the presentation of this experiment could be sharpened. We will explicitly frame this experiment as a statistical challenge arising from near-violations of the causal assumption and not a causal non-identifiability problem.\n\n**Other comments** Thank you for catching all the typos and your suggestions regarding the introduction. **ACTION:** we will implement them!\n\n**Heterogeneity of user preferences** We would like to highlight that while the case study of section 4 is presented as a study of confounding, the issues discussed are inherently a consequence of the heterogenous preferences of the two user groups: the helpfulness and the harmlessness preferring, where the two objectives are often *conflicting*.\nWe find the investigation of more fine-grained variability in user preferences, going beyond the simplified setup of just two, an exciting direction for future work.\n\n**Distinction from prior work with user-specific covariates** Siththaranjan et al. highlight the possibility of heterogeneity in human preferences, pointing out the existence of potentially unobservable, hidden contexts that can influence user preferences. They demonstrate the negative consequences of implicitly aggregating over these hidden contexts when performing preference learning under a homogenous BTL model. Using our notation, they show that applying $\\mathbb{E}[L(x; y, y')]$  as a global estimate of the expected preference for all users (instead of $\\mathbb{E}[L(x; y, y') \\mid C = c]$ separately for each set of user-specific covariates $c$) can lead to misalignment, especially with respect to the minority groups.\n\nHowever, their work does not address the issue of confounding, which arises when user-specific covariates $C$ not only influence the preference $L$ but also the distribution of the prompts $X$, making $C$ a confounder.  More generally, the limited number of related works considering user-specific covariates in preference learning assumes that user-specific preference datasets are collected under a randomised distribution of treatments. Our work challenges this assumption and explicitly considers the scenario where this distribution is influenced by $C$ (which is the case for instance when the prompts $X$ are written by the users themselves), leading to confounding issues examined in the case study of Section 4.\n\n---\n\nWe thank the reviewer for their positive evaluation of our work and are happy to answer any further questions!\n\n[1] arxiv.org/abs/1811.12359\n\n[2] arxiv.org/abs/2105.06422\n\n[3] arxiv.org/abs/2209.11924\n\n[4] arxiv.org/abs/2203.16437\n\n[5] doi.org/10.2307/2998560\n\n[6] doi.org/10.1111/1468-0262.00442\n\n[7] doi.org/10.3386/t0330", "claims_and_evidence": "- The paper's primary theoretical contribution is a causal framing of preference learning. The theoretical results about causal identification are standard results from the causal inference literature, adapted to the preference learning context and accounting for the latent structure of textual features. They claim to be the first to address confounding due to user-specific covariates, though they note previous work that has identified the existence of user-specific covariates and their effect on rewards [but not necessarily through confounding].", "methods_and_evaluation_criteria": "- The HH-RLHF study effectively addresses the motivating challenge of confounding in a semi-synthetic environment. The dataset is appropriate.\n- See my comments under “Other Strengths And Weaknesses,” where I suggest that the UltraFeedBack experiment may be unnecessary or out of place.", "theoretical_claims": "- Overall the theoretical claims are well-justified. But see my comments in “Other Strengths and Weaknesses.”", "experimental_designs_or_analyses": "- The HH-RLHF study provides a very nice comparative analysis of reward models under confounded data.\n- There could be more explanation of how the Multihead architecture incorporates the causal structure of the data. (The review suggests that the $\\hat{z}$ being learned as a function of just $(x, y)$ means it doesn't use information from $c$, but it would be helpful to clarify that $c$ doesn't use information from $\\hat{z}$ either, mirroring the conditional independence found in the underlying causal graph.)\n- See my comments under “Other Strengths And Weaknesses,” where I suggest that the UltraFeedBack experiment may be unnecessary or out of place.\n- The experiments don't address treatment effect heterogeneity, which is one of the paper's theoretical motivations.", "relation_to_broader_scientific_literature": "- The related works section nicely situates this paper within the broader causal inference and reward modeling literatures.", "essential_references_not_discussed": "- In the discussion of causal representation learning on page 5 and in Appendix C, the authors should reference the fundamental challenges in learning causal representations from stationary data. Much theoretical work has been done demonstrating how identification of causal representations requires auxiliary labels (for instance, using paired examples where only a small number of ground-truth causal variables have been intervened upon). This is especially relevant to the paragraph “Active querying strategies and interventions” in Appendix C.\n- See Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations by Locatello et al., https://arxiv.org/abs/1811.12359 and any more recent relevant work citing this paper.", "other_strengths_and_weaknesses": "- Strengths\n    * The paper is very readable with clear visual cues and figures.\n    * Extensive use of examples enhances understanding of the concepts.\n- Weaknesses\n    - Overlap/Positivity\n        * Inconsistent terminology: The use of the term \"overlap\" in Assumption 3 and Assumption 5 to cover the case without conditioning on covariates is confusing, as this term is typically used to refer to overlap across covariate subpopulations (which the paper itself discusses on page 6, line 315). Even before this section, the paper describes the overlap assumption in the more standard \"conditioning on covariates\" way on page 4, line 202, noting that \"The assumption of positivity requires that every user has a non-zero probability of being assigned any combination of texts given their covariates.\"\n        * Unnecessary for identification: The overlap assumption as defined in Assumption 3 and Assumption 5 is not actually necessary for the proofs of Proposition 1 or Proposition 2. What the paper appears to be getting at is that overlap is not needed for causal identification in the case without confounding, but it is needed if one wants to estimate those quantities in a non-parametric form. If we assume a parametric model (e.g., a linear model), then we don't actually need to observe all possible combinations of X, Y, Y', contrary to what the paper claims.\n        * Statistical vs. causal confusion: The UltraFeedBack Case Study (Appendix D, referenced in Table 1) is fundamentally about misgeneralization due to failure of the IID assumption. The experiments show how models fail when correlations present during training disappear at test time, but this doesn't necessarily involve causal misconceptions - it's about the IID assumption failing. A truly causal problem would exist if, even with IID data, infinite data would not allow us to estimate a treatment effect of interest. Including this study dilutes the paper's causal focus and may confuse readers about which problems are genuinely causal versus purely statistical.", "other_comments_or_suggestions": "- Page 2, line 86: “maximising” should be corrected to “minimizing.”\n- Page 4, line 194: “underepresentation” should be corrected to “underrepresentation.”\n- Page 5, line 260: ensure clarity regarding the distinction between causal discovery and causal representation learning.\n- Though some motivation for the causal framing is presented in the introduction, some additional comments here may be helpful, perhaps by emphasizing the downstream role of reward models in alignment.", "questions_for_authors": "* Could you clarify why the UltraFeedback case study is framed as a causal problem rather than a statistical generalization issue? What makes the correlation shifts in this experiment fundamentally causal rather than just IID violations?\n* Your theoretical motivation discusses both confounding and heterogeneous treatment effects, but your experiments focus primarily on confounding. Do you have plans to extend your empirical analysis to demonstrate the benefits of your causal approach for handling heterogeneous treatment effects?\n* Your paper mentions that \"confounding due to user-specific covariates has not been addressed in prior works.\" Could you elaborate on the distinction between your contribution and prior work that has identified the existence of user-specific covariates (like Siththaranjan et al.'s work on \"hidden context\")?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cchIWSFm2K", "reviewer": "Reviewer_Leyp", "summary": "This paper proposes a causal framework for preference learning in the context of aligning LLMs with human values. The authors argue that relying solely on observational data can lead to reward functions that pick up spurious correlations rather than true causal drivers of user preferences. To address this, the authors develop a model that incorporates causal inference principles such as potential outcomes, confounding, and latent treatments. Through theoretical analysis, they emphasize the critical assumptions needed for causal identification, most notably unconfoundedness and latent overlap, and discuss how these assumptions often fail in real-world data collection settings. Empirically, they demonstrate that strong correlations among latent features or user-specific objectives can cause overfitting and hinder the robustness of reward models under distribution shifts. The paper then proposes an “adversarial multi-objective reward model” to mitigate confounding effects, showing improved generalization, particularly in highly confounded scenarios.\n\n## update after rebuttal\nThanks the authors for their detailed responses. My questions and concerns are fully addressed, and I will keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for engaging with the core ideas of our work. We appreciate your time and feedback, and we’d like to address your concerns in hopes of clarifying our contributions and positioning.\n\n**The semi-synthetic design** We appreciate the reviewer’s attention to our experimental design and understand their concern about the use of semi-synthetic tasks. However, we’d like to clarify why semi-synthetic evaluations are a necessary and principled choice for studying causality in preference learning. Evaluation of causal inference methods demands knowledge of the true data-generating process (e.g., presence of confounders, the true set of causal variables), which is *unknowable* in purely observational real-world datasets. Semi-synthetic data allows us to:\n\n- Control the degree of overlap.\n- Inject controlled confounding (to simulate real-world biases).\n- Precisely measure how well the competing methods recover true relationships.\n\nReal-world datasets only show the observed human choices, not the counterfactual \"what-if\" responses needed to assess how a user would react if the same prompt were answered differently or if they acted according to a different objective. By augmenting these real-world public datasets with synthetic variations (e.g., simulating distribution shifts or conflicting objectives), we create a controlled testbed to isolate causal challenges—something impossible with static, observational data. Our approach aligns with established causal inference literature, where semi-synthetic data is the standard for evaluating methods [1, 2]. \n\n**Useful vs. harmful correlations** We agree with your observation that correlations like “academic-style queries → rigorous answers\" are often valid and desirable—indeed, in many cases, these patterns reflect genuine user needs.  However, our focus is on the challenges such correlations pose when training reward models from observational data where confounding is present. This issue is particularly critical in the context of steerable alignment, where we aim to gain control over which objective an LLM aligns to at inference time, rather than letting the model infer it purely from its input. Our case study in Section 4 examines the problem of learning a multi-objective, steerable reward model, where the two objectives considered are learned from different data distributions. We demonstrate that the naive approach fails when the train reward model is applied to data distributions different than at training time, which can plausibly occur in the real world. We also show how insights from causal representation learning can help build more robust reward models.\n**ACTION:** We acknowledge that not all correlations are harmful, and we will clarify this nuance in the revised version of our manuscript.\n\n**Practicality vs. coverage** You also raise a valid concern about the practicality of ensuring broad coverage of latent factors. We agree that exhaustive interventions are often infeasible, and we do not advocate for unrealistic data collection burdens. However, our goal is not to advocate for rigid control but rather to bring attention to challenges in learning across heterogeneous user populations, merging datasets collected with different objectives, or collecting feedback directly from users who wrote the prompts—practices that are present in preference learning today. While training reward models in an unsupervised, uncontrolled manner is flexible and scalable, we argue that the community should consider seeking a better balance between practicality and robustness. We highlight cost-effective strategies such as active querying (Appendix C) and causal regularisation techniques (as exemplified with the adversarial model) that can improve robustness without excessive overhead.\n\n**ACTION:** We will explicitly discuss the trade-offs between practicality and robustness in the final version, clarifying our position.\n\n---\n\nWe hope this response alleviates your concerns about our paper’s positioning. Our work does not dismiss observational data but instead provides tools to diagnose its limitations—a step we find crucial to achieving scalable and robust alignment. We believe the paper’s insights are valuable for the community, especially as alignment research moves towards greater personalisation.\n\nThank you again for your time and feedback. We are happy to answer any further questions you may have and incorporate further revisions to ensure clarity in the final version.\n\n[1] arxiv.org/abs/1606.03976\n\n[2] arxiv.org/abs/1705.08821", "claims_and_evidence": "Yes. The paper provides detailed explanation on the problem formulation, theoretical statements, and numerical experiments.", "methods_and_evaluation_criteria": "The paper’s methods and evaluation align well with its theoretical goals with well-known dataset like HH-RLHF and UltraFeedback, but they mainly use synthetically augmented datasets instead of real-world data.", "theoretical_claims": "The propositions in the main paper seems fine to me.", "experimental_designs_or_analyses": "The authors conducted case study over two public datasets. The analyses look fine to me, but the synthetic task design remain my key concern.", "relation_to_broader_scientific_literature": "This paper provides an interesting causal perspective over AI alignment, and discusses the key challenges critical assumptions for generalizing reward models to unseen texts and contexts.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**  \nA key strength is its clear explanation of the preference learning problem and seamless integration of a causal framework. It effectively connects user preferences, latent features, and rewards, making it easy to see how spurious correlations arise and hurt generalization. The solid theoretical backing on latent overlap and confounding, supported by a case study, reinforces these ideas. Overall, it offers a well-structured causal perspective.\n\n**Weaknesses**  \nSome arguments in the paper seem either insufficiently supported or somewhat overstated. For example, the concern over \"unobserved confounding\" suggests that latent user attributes (like professional background) might create misleading correlations, such as academic-style queries leading to a preference for rigorous answers. However, in many real-world cases, these correlations are exactly what a model should capture. Medical experts, for instance, genuinely need detailed and precise explanations. If most academic queries come from users who actually require rigorous answers, then learning the pattern “academic → rigorous” isn’t necessarily a problem.\n\nRegarding the “low overlap” issue, the paper argues that rare combinations of latent features can hurt generalization under distribution shifts and suggests extensive interventions to ensure broad coverage. However, in practice, dedicating significant resources to extremely rare cases may not always be worthwhile—focusing too much on edge cases could come at the cost of improving more common scenarios. The paper doesn’t address this trade-off, making its push for systematically controlling all latent factors feel somewhat idealistic in real-world data collection.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper applies causal thinking to the learning of reward models. Following an extensive discussion, all reviewers agree it constitutes an interesting and possibly impactful perspective on understanding reward models. The authors should incorporate the reviewer comments into the final version (per the discussion).
Decision: Accept (poster)