Causally Robust Reward Learning from Reason-Augmented Preference Feedback
Authors: Minjune Hwang
Yigit Korkmaz
Daniel Seita
Erdem Biyik
Venue: ICLR 2026 Poster (2026)
Keywords: Preference-based learning
causal confusion
learning from human feedback
Primary area: causal reasoning
TL;DR: We develop a framework that utilizes natural language rationales to mitigate causal confusion in preference learning.

Abstract: Preference‑based reward learning is widely used for shaping agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co‑occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in an embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that is unrelated to the stated reason. Because the same rationales (e.g., "_avoids collisions_", "_completes the task faster_") can appear across multiple tasks, ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge to novel tasks without extra data or language‑model fine‑tuning. Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features. ReCouPLe outperforms baselines by up to 1.5x in reward accuracy under distribution shifts, and 2x in downstream policy performance in novel tasks. We have released our code at [https://github.com/mj-hwang/ReCouPLe](https://github.com/mj-hwang/ReCouPLe).

Published as a conference paper at ICLR 2026
CAUSALLY ROBUST REWARD LEARNING FROM PREFERENCE FEEDBACK
Minjune Hwang
Yigit Korkmaz'
Daniel Seita If
Erdem Bıyak' It
Thomas Lord Department of Computer Science, University of Southern California
* "Corresponding author
'quual advising (minjuneh, ykorkmaz, seita, biyik) @usc.
ABSTRACT
Preference-basee reward learning is widely used for shaping agent behavior match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co-occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in an embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that unrelated to the stated reason. Because the same rationales (c.g.. "avoids collisions", "completes the task faster") appear across multiple tasks, ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge novel tasks without extra data or language-model fine-tuning. Our learned reward model can ground preferences generalizing beyond spurious features. ReCouPLe outperforms baselines by up 1.5x in reward accuracy under distribution shifts, and 2x in downstream policy performance in novel tasks. We have released our code at ht :ttpps::/gge //g1 htt//ttttttt hubb com/ mj hwangg wwwwang//Rooo..e.... ReCouPLe.
the articulated reason, aligning better with user intent and
INTRODUCTION
Designing reward functions that faithfully capture human intent is one of the central obstacles to deploying learning agents in the real world. Preference-based reinforcement learning (PbRL) removes the need for hand-crafted rewards by asking a human to compare two trajectories and indicate their preference (Christiano et al., 2017; Sadigh et al., 2017; Bryık et al., 2019; Hejna & Sadigh, 2023; Lee et al., 2021; Ouyang al., 2022). Unfortunately. this binary feedback conveys at most single bit of information and leaves the reward model free to explain the preference with any correlating feature in its observation space. Under the presence of non-causal distractor features that are spuriously correlated with preference labels, reward models often learn to rely such features (Tien et al., 2023). These features, however, are irrelevant to the task success. When they disappear or change at test time, the agent can suffer from reward misidentification and fail to generalize. Since each comparison supplies little information, it leaves many causal explanations indistinguishable. Without extra guidance, the learner cannot tell whether users prefer a trajectory for its smoothness, its speed, or some spurious cue in the background.
For example, suppose we want to train robotic arm to pick up box large enough to store toys (Figure 1). During data collection, every preference query shows a large red box and small blue box, and the annotator always prefers the former. Because size and color are perfectly correlated in these comparisons, a reward model can reach zero training error by attending to the color cue instead of true size. At test time, when it encounters large blue box next to small red box, the learned reward could mistakenly favor the small red box.
A natural way to overcome this ambiguity is to supply richer feedback. Following advancements in natural language processing, recent works in robot learning employed language for task planning (Ahn et al., 2022; Sharma et al., 2022; Singh et al., 2023; Bucker al., 2022), policy learning (Cui

---

Published as a conference paper at ICLR 2026
Trajectory A
Trajectory B
Data Collection
I want the robot to pick up box
large enough to put toys.
Preference Learning
prefer trajectory over B.
?
prefer trajectory because it picks a larger hox.
Wait, the blue bux larger. Does the user prefor because of the color? the size?
Abal should pick larger box.
Without reasoning
With reasoning
Figure 1: Preference learning can be susceptible to causal confusion, especially with the presence of non-causal distractor features that merely co-occur with preferred trajectories. In the example above, the reward model struggles identify the exact feature of a trajectory that determined the user's preference. By providing reason, the agent can identify the causal feature.
et al., 2023; Dai et al., 2025; Shi et al., 2024), and reward shaping (Goyal et al., 2019; Yang et al., 2024; Holk al., 2024). We claim that short natural language rationales carry exactly the causal signal the model is missing, "I prefer this trajectory because it picks up the large box" tells the learner which feature matters for the user's preference.
We present ReCouPLe (Reason-basee based Confusion Mitigation in Preference Learning). lightweight framework which consists of data modality that couples preferences with their reasons/rationales and a learning algorithm that treats each rationale as directional guide in shared trajectory- language representation space. We design simple loss that encourages the preference be based on the direction of the reason specified by the user, rather than on incidental correlations in a pair of trajectories. The language encoder shared across all tasks makes sure we retain the same semantics between them. Subsequently, this decoupling of reason components enables to leverage shared rationales appearing across tasks, learning reward function that generalizes zero-shot from one task to another.
In summary, our contributions are:
1. Based on the observation that pairwise preferences provide limited information in the presence of non-causal distractor features, leaving reward models prone causal confusion, design new feedback type by supplying complementary causal cues that help disambiguate the true preference signal.
2. We introduce ReCouPLe, framework that injects causal structure into preference learning by aligning trajectory representations with rationale embeddings.
3. We demonstrate that augmenting comparisons with rationales produces reward models that significantly reduce causal confusion relative to state-of-the-art baselines. Moreover, these models transfer across tasks without additional preference queries by leveraging shared underlying reasons.
2 RELATED WORK
Learning from Human Feedback. Policy learning from human feedback has taken many forms, including demonstrations (Schaal, 1996; Ho & Ermon, 2016), corrections (Bajesy et al., 2017; 2018), interventions (Korkmaz & Bıyık, 2025; Kelly et al., 2019), language (Shi et al., 2024; Sharma et al., 2022), and preferences (Bıyık et al., 2019; Christiano et al., 2017; Hejna Sadigh, 2023). Among these, preferences and language occupy opposite ends of the spectrum: binary comparisons are intuitive and easy to provide but limited in expressivity, whereas natural language is rich and flexible but often underconstrained due to the vagueness and ambiguity of natural language, necessitating an additional modality (Casper et al., 2023; Zeng et al., 2023; Shi et al., 2024; Cui et al., 2023). this work, complement pairwise comparisons with natural language rationales, which provides richer causal information while preserving the ease of use.
Preference-based Learning. Preference-basee methods have become popular for reinforcement learning, especially where explicit rewards are unavailable (Christiano et al., 2017; Sadigh et al., 2017; Hejna& & Sadigh, 2023; Bryık et al., 2019). While effective, their reliance binary comparisons poses
2

---

Published as a conference paper at ICLR 2026
two main challenges: (i) each query conveys at most one bit of information, and (ii) the feedback is ambiguous when multiple task-relevant features exist (Casper et al., 2023). This makes reward models prone to relying on spurious, non-causal cues (Tien et al., 2023). When policies are optimized against such flawed proxies, this vulnerability frequently triggers "Causal Goodhart" effects, where optimizing learned proxy reward degrades actual policy performance because the proxy relies on features that correlate with, but do not cause, the desired behavior (Manheim & Garrabrant, 2018; Gao et al., 2023). Prior efforts to enrich preference signals include active query selection strategies such as information gain (Bryık et al., 2019), volume removal (Sadigh et al., 2017) and maximum regret (Wilde et al., 2020), and augmentations with feature-level queries (Basu et al., 2018). More recently, Peng et al. (2024) proposed feature-wise preference learning, asking users why example preferred. However, these methods assume access to structured, task-specific features and have been demonstrated in limited settings like linear bandits. In contrast, we employ free-form natural language rationales alongside binary preferences, aligning the reward model with causal explanations while reducing reliance on spurious correlations.
Robot Learning with Language Feedback. There have been several recent works that utilize natural language for improving learned robot policies via different strategies such as task planning (Ahn et al., 2022; Singh et al., 2023), policy learning (Cui et al., 2023; Dai et al., 2025; Shi et al., 2024), and reward shaping (Goyal al., 2019). Shi al. (2024) employ language-condittoned behavior cloning (LCBC) for corrective language commands and improving policies. Cui et al. (2023) introduce an approach to use human language feedback correct robot manipulation real-time via shared autonomy. Dai et al. (2025) propose a data generation pipeline that automatically augments expert demonstrations with failure recovery trajectories and fine-grained language annotations for training recovery policies. In the domain of preference learning, Yang et al. (2024) learn a shared latent space for trajectories and comparative language like "move farther from the stove", showing that language can make reward learning faster and more intuitive. Holk et al. (2024) instead use predefined set of relevant features and human prompts to extract trajectory segments corresponding to each feature, which are then used for reward modeling. These existing approaches treat language as an additional input to the reward model, without exploiting its compositional structure or underlying rationale. In contrast, we explicitly use rationales to reveal the causal factors behind preferences, thereby mitigating causal confusion and enabling better transfer across tasks.
PRELIMINARIES
We consider a collection of tasks modeled finite-horizon Markov decision processes (MDPs) M = A,, p. 7, T), where S is the state space, AA is the action space, P(s st, 4:) is the transition kernel, r: x A + R is the reward function, 7 E [0. 1) the discount factor, and T is the maximum time horizon. The reward function is unknown and must be inferred from the user's pairwise preference feedback augmented with rationales.
3.1 REWARD LEARNING FROM PREFERENCE DATA
We assume access to preference data in the form of binary comparisons. Given a pair of trajectory segments (TA, of horizon < T, a user (either human, or a proxy) provides preference label y::
-= 1 if TB:
otherwise.
where each segment is a sequence of observations and actions (sk, Ck...., 8k+ H: Og ak+H), +I). Following the Bradley-Terry model (Bradley & Terry, 1952), the probability that the trajectory TA is preferred over the trajectory Tв is given by:
exp(r(TA))
PA(TA> ((TA>TB) exp(r(TA)) + e
where r(T) is the cumulative discounted reward of trajectory 7. In order to infer the reward function, prior works (Christiano al., 2017; Sadigh et al., 2017; Lee et al., 2021) in preference- RL train a reward function fw. : S x A + E parameterized by on a dataset D of (TA, TB, y) triplets by minimizing the binary cross-entropy (BCE) loss with the Bradley- -erry model:
LECE.BT BT TE (TA.TT))) loy Pr., (TA 7B) (1 y) log(1
(TA +TB))
(2)
3

---

Published as a conference paper at ICLR 2026
3.2 LANGUAGE INTERFACES
In addition to the standard MDP formulation, each task is associated with a task description Ensk. a short instruction such as "pick up the cup" "push the cube." These descriptions provide high-level semantic grounding for the task but are not sufficient on their own to fully specify the reward function. To capture the finer distinctions that matter the users, we rely on preference queries. Additionally, each preference label has an optional reason feason that explains why one trajectory is preferred over the other (e.g., "because it avoids collisions"). frozen language encoder LM maps these strings to fixed embeddings of dimensionality d: 0 LM(f IR and vá =
E R
4 METHOD
Preference-basee reinforcement learning typically fits single-task reward by maximizing the likeli- hood of observed comparisons (Eq. 1). We extend this framework to the multi-task setting, where each task is identified by its language description. Specifically, model the reward the inner product between the trajectory representation and the task embedding:
r(T, bunk) = ( LM(4uss) = ((T)
(3)
where the trajectory encoder Φ T Red is the only trainable component, as the task embedding 0 = (42ak is frozen. We use this reward formulation across all methods for consistency. Although linear in structure, the nonlinearity both the trainable trajectory encoder and the frozen language model allows this simple form to capture complex, task-specific reward structures, similar to CLIP- based reward models (Sontakke et al., 2023). In our experiments, we use the pretrained T5 (Raffel et al., 2020) language model encoder LM.
4.1 RECOUPLE REASON-BASER CONFUSION MITIGATION IN PREFERENCE LEARNING
Our key idea is that short natural language ra- tionales may reveal the causal features underly- ing preference and thereby mitigate spurious correlations. For example, the statement "I pre- fer this path because il avoids collisions" explic- itly identifies the relevant factor influencing the user's choice. Given rationale Lrassnn we obtain its embedding via the frozen language encoder, LM (- ReCouPLe then treats projection axis and decomposes the trajectory em- (i) a reason- aligned part parallel to vú, and (ii) an orthogonal part corresponding to features unrelated to the ra- tionale. Figure 2 visualizes this process.
Trajectories
Joint Representation Space
FA
(((A)
pEas)
(L
(TTT)
FB
<pTT
Language
bedding (TT) into two components:
reason
(1(TT)
D1(Ta)
y
L.M
avih puck lifte*
é task
Goal: decompose trajectory embedding lo ground preference on reasons
"pick pack, hypuss maiii and place the puck to goal"
LM
θ
Mathematically, this orthogonal projection onto Vp induces two disjoint subspaces:
Rewards
((T)=
(I(T)
TT = 0,
Reason-Aligned,
($ (+)) (p)(T
Reason-Orthogonal: (t))) (T))))
rcason-aligned
reason-omtorg casson-othogonal
which is achieved by
Figure 2: ReCouPLe decomposes the task re- ward by orthogonally projecting the trajectory representation to the reason language embedding and decomposing the representation into reason- aligned and reason- reason-orthogonal components. This allows the reward model to isolate the causal fea- ture specified in the rationale to explain the user's preference.
(T(T) | ) @L(T) =6(r) (T)
6 (T)
Subsequently, the reward term (Eq. 3) decomposes as
r(T, (rask)
(T, Lrnck) explained by rationale
(T, Cmsk) residual task signal
4

---

Published as a conference paper at ICLR 2026
with the following components:
TT(T, fusk) = 11 is the reason-aligned, causal component explicitly justified by the user's 's rationale. ri (T, fiask) = 0² 0 is the orthogonal component that captures any lask-relevant information the rationale overlooks (e.g., shaping rewards or domain priors).
Our insight is to ground the pairwise preference on the stated reason and prevent the model from relying on incidental correlations. This achieved by forcing reward differences to depend solely on "I while holding neutral.
Given the decomposed reward, we train our trajectory representation by three loss terms:
1. Reason loss: BCE loss using Bradley-Terry model with ri only, enforcing that preferences explained through the stated causal feature.
Lresson - - ply log Pry (TA > TB) + (1 - y) log(1 - Pry (TA > Tn))
Orthogonal consistency loss: Consistency loss prevents non cannaalal features explaining pairwise preferences but still encourages them to capture any task-relevant information.
We propose two different versions of this term, resulting in variants of ReCouPLe:
(a) ReCouPLe-EC:
Equality constraint (TA, (rskk) RS Tu (TB, lmsk) for every comparison, ensuring $o carries no preference signal:
Leg (T_(TA, lunk) r (TB, (TB: busk))
(b) ReCouPLe-IC: Inequality constraint encouraging the difference between to be greater than that of r. for given pair of trajectories. A regularizer term computed by taking the BCE loss on total reward (r') + r (Eq. 2) is also added to prevent potential mode collapses:
diff, (A> B) r(TA, Pansk) r(TB: lank)
exp(diff,, (A B))
S(A> B) exp exp(diff,, (A> (ifidi,,,, B)) + exp (A > B))
Liney B( ply log S(A B) (1 y) log(1 S(A B))] + LACE-BT (6)
ReCouPLe-EC imposes a strict condition, requiring the reason-orthogonal components to be identical for compared trajectories. In contrast, ReCouPLe-IC less restrictive, incentivizing differences in the ceason-aligned component to dominate differences in total task rewards. As such, ReCouPLe-FCC is more effective when a small, recurring set of reasons largely governs preferences and variability in the reason-orthogonal components is minimal, whereas ReCouPLee e- IC preferable when many plausible reasons may explain comparisons and when variation in reason-orthooonal reward is non-negligible.
3. Reward-ratio regularizer: Lratio for keeping the magnitude of rii below fraction of the total reward magnitude + TL, preventing the trivial collapse of the reward into the causal subspace:
fratio = RcLU |r Hriiittee a). where a small constant c is included in the denominator to prevent division by zero.
Ir|l
The final objective is the following:
CReCouPLe = Crcason + Aratio + Dayfory (ReCouPLe-FCC) Xinoq Lineq (ReCouPLe-IC).
EXPERIMENTS
We evaluate ReCouPLe on two complementary suites that probe distinct facets of the method. The first suite focuses causal robustness in single visuomotor task whose visual cues deliberately confounded; the second investigates cross-task generalization in a multi-task manipulation benchmark. Together they address research questions:
5

---

Published as a conference paper at ICLR 2026
RQ1 (Robustness against causal confusion): Can ReCouPLe maintain preference accuracy when the covariate distribution shifts in a way that exposes spurious correlations?
RQ2 (Task transfer): Does the reason-aligned subspace learned on small set of tasks transfer to semantically related, novel task without additional preference queries?
We address RQ1 with custom ManiSkill (Tao al., 2025) suite that deliberately entangles distractor features (object color) with the ground-truth causal factor (object size) and then evaluates under a color-swapped distribution. We assess RQ2 with set of Meta-Won World (Yu et al., 2020) tasks that are widely used to test few-shot/zero-sho transfer.
For both experiments, we compute per-step embedding by first encoding each state-action pair (st, (z.) with modality-aprropriaae encoder: a convolutional encoder for image observations (Yarats et al., 2021) and fully connected network for state-based control tasks. Concretely, we use a neural network encoder e S x A + IR' to encode every atate-action pair of trajectory into the corresponding per-step embedding. We then obtain a trajectory embedding ((T) by aggregating these stepwise embeddings over time. This additive design mirrors prior works in preference learning that formulates trajectory features as sum of per-step features (Yang et al., 2024).
Baselines. We compare ReCouPLe against two baselines that share the same multi-task reward formulation but differ in the loss terms used to train the trajectory encoder e.
Multi-Task Bradley-Terrry (BT-Multi): This baseline learns the trajectory encoder, é by minimiz- ing the binary cross-entropy loss CACEART (Eq, 2) across all tasks, using the shared reward definition above and ignoring the rationale reason. It therefore serves as the baseline without reason inputs. Reason-Feature Preference (RFP): This baseline is an extension of Pragmatic Feature Preferences (PFP) (Peng et al., 2024). PFP assumes that each state can be represented by explicit, hand- designed feature vector. Humans (i) specify which of those features are relevant to the task and (ii) give pairwise labels that compare each relevant feature across items. In our setting, such features do not exist. Instead, we treat the frozen rationale embedding &b = (f as a single implicit feature direction in the representation space. Besides the shared reward r(T, fmsk) from Eq. 3, we also define a reason score, Q(T, reason) ) = Φ(T) e,, and minimize the standard BCE loss from Eq. 2 and an additional auxiliary BCE loss term for the specified reason's 's score.
Metries. After training each reward model, we report reward accuracy: the proportion of held-out preference pairs where the preferred trajectory labeled with a higher reward. We evaluate ManiSkill on in-distribution (ID) and color-swapped (OOD; out-of-distributio splits, and Meta-World on three training tasks (validation split) plus a held-out novel task. We then validate the reward model's usefulness in downstream policy learning. To assess this, we use the learned model to assign reward values to trajectories in offline play data and train an offline RL. policy for each task to report its task success rate. These offline datasets have trajectories with different levels of optimality, so simply imitating the behavior in the dataset would not produce successful policies. Rather, the learned reward model has to discern the difference between preferred and undesirable behaviors and assign appropriate rewards to trajectories with varying optimality.
5.1 MANISKILL TASK SUITE FOR RQ1
Task design. To test whether our proposed method can mitigate causal confusion in preference- learning, we design a set of object manipulation tasks in ManiSkill. Each scene has two cubes of different sizes and colors (red vs. blue) on tabletop, and the agent must manipulate the larger cube. We have 4 total tasks: Mis-Pick-Larger, During training, the larger cube always a fixed color for each task, creating a perfect correlation between color and the correct behavioral choice; The larger cube is always red in MS- MS-Pick-Larger and MS-Pull-Laare Larger whereas the larger cube always blue in MS-Place-Larree and Push--uhheerr At test time we swap the colors so that the distribution shift induces the classic "shortcut" failure: model that latches onto color will choose the wrong object. Implementation details are described in Appendix A.1.
JS--Push-Large
AIS-Place-Large
and MS-Pull Pull-Larger.
Data generation. We collect synthetic preference queries for each task by pairing trajectories that manipulate the larger and smaller cubes, respectively. These trajectories generated by training oracle SAC (Haarnoj et al., 2018) policies that manipulate either larger cubes or smaller cubes. Details in the environments and data collection process are introduced in Appendix A.2. Then, the

---

Published as a conference paper at ICLR 2026
Table 1: Reward accuracy comparison for ManiSkill (RQ1), averaged over 3 seeds. Left block: 2-task setting; right block: 4-task setting. Methods with highest accuracies in each OOD task are bold-faced.
2-task
4-task
In Distribution
Color Swapped
In Distribution
Color Swapped
Model
Pick
Place
Pick
Place
Pick
Push
Place
Pull
Pick
Push
Place
Pull
Single Task
BT
0.980
1,000
0.540
0,830
0.980
1,000
1,000
1,000
0.540
0.987
0.830
0.867
Multi- Tasti BT-Mulli
0.953
1,000
0.600
0.820
0.987
1.000
1,000
1,000
0.707
1,000
0.840
0.907
Multi-Task Reasons
RFP
0.940
1,000
0.620
0.800
0.993
0.980
1,000
0.700
0.700
0.980
0.807
0.913
ReConPLe-FCC
0.993
1,000
0.820
0.940
1,000
1,000
1,000
1,000
0.773
1,000
0.880
0.860
ReCouPLe-IC
1,000
0.633
0.807
0.993
0.987
0.807
0.867
preference label selects the trajectory handling the larger cube. The accompanying rationale fressnn. is "(because) the cube is largez" The task label lus is simply "/manipulating verb) the larger cube." " Even though the task text already mentions "larger cube," ReCouPLe uses the rationale shared causal axis, projecting trajectories to isolate the size feature in the reason-aligned component while keeping the manipulation verb in a reason-orthogonal. task-specific residual. By contrast, methods without this decomposition often entangle causal (size) and distractor cues (e.g.. color), leading to overfitting and failures under color-swap distribution shifts.
Preference prediction results. Table shows that performance saturates for all methods on the ID validation split, but clear gaps appear on the color-swapped OOD split that flips the color-size correlation. Single-task BT and BT-Multi drop markedly, and a naive reason auxiliary loss (RFP) offers only modest recovery. In contrast, ReCouPLe-ECC attains the best OOD accuracy across tasks, with the only exception of MS- MSSPull-Larger. This indicates that ReCouPLe can extract the common causal feature (cube color) from tasks with different color-size correlations (e.g., red larger cube in MS-Pick-Large and blue larger cube in MS- MS-PPace-LLnie Place-Larger) by projecting trajectory embeddings onto the common reason embedding and isolating such causal features from distractor fcatures (cube size). ReCouPLe-IC is comparatively less effective here, since single causal feature dominates all pairs and the reason-orthogonal components vary little in this suite, and the stricter equality constraint in ReCouPLe-EC yields clearer advantage.
Policy Learning Results. Using the learned rewards to train policies on the OOD environments for 2-task and 4-task settings (Figure 3) mirrors the reward accuracy trends: offline RL datasets with BeCCoulLe-trainee rewards yield policies with higher success than BT-Multi and RFP. A noticeable advantage is observed in MS-Pick PirkeLarger and MS-Pull-Larger Both ReCouPLe-EC and ReCouPLe- IC consistently outperform naive behavior cloning on the play dataset, suggesting that improvements in reward preference prediction translate into downstream policy learning in OOD environments.
2-Task Suite
Task Suite
Pick (OOD)
Pick (OOD)
Push (OOD)
1.00
0.4
04
Secoess Rate
0.25
0.0
0.2
0.2
0.25
0.0
0.00
000000
100000
100000
20000
Step
Step
Step
Place (OOD)
Place (OOD)
Pull (OOD)
0.55
0.50
8.15
0.15
0.10
0.05
0.05
0.00
0.00
40.00
100000
Step
Step
Step
BT-Muli
REP
(AlI)
BC (Pieferned)
ReCouPla-IC
ReConPlaeFCC
Figure 3: ManiSkill policy learning results, averaged over 3 seeds (mean + std).
7

---

Published as a conference paper at ICLR 2026
5.2 WOOAL TASK SUITE FOR RQ2
Task design. We select three training tasks from Meta-World. Push, Push Wall, and Pick-Place Wall. We reserve Pick-Place, a variant of Pick-Place-Wee task without a wall that parallels the structural difference between Push and Push- Wall. Each task's ground-truth reward is linearly decomposed into interpretable components (grasp, lift, collision avoidance waypoints, etc.) provided by the benchmark.
Data generation. We first collect trajectories by rolling out policies with different levels of optimality and Gaussian noise, similar to the data collection procedure in Hejna & Sadigh (2023). Then, for each query, we randomly sample two trajectory segments ТA and тB and generate the preference label based on their total reward E; r(s;, 04), where Meta-Worldh pre-defined environment reward can be linearly decomposed into feature components { r(s, a) = Σj (8, a). Without loss of generality, suppose тA is preferred over TB. Now, we synthetically generate the reason label by computing component-wise wise advantages A5 Wy (TA( (TB) and convert them to softmax distribution from which we sample the reason behind the preference:
exp(A,)
Pr(choose reason j)
Σx (A))
Each sampled reason is free-form sentence such as "keeps a firm grasp while steering toward the goal." Please refer to Appendix B.3 for details. We generate 2000 preference-rationll pairs for each training task (6000 total).
Preference prediction results. As demonstrated in Table 2, there is no meaningful difference in validation re- ward accuracy of training tasks. How- ever, in the novel Pick-Place task, both ReCouPLe-IC and ReCouPLe- EC outperform our baselines. This indicates that the reason-aa aliigedd sub- space learned on the training tasks captures transferable cues for held-out task. ReCouPLe can better transfer preference to novel tasks by decomposing and recomposing semantically grounded features in training tasks (e.g., Pick-Place- Wall PPush-Wall - Push) ~ Pick-Place).
Table 2: Reward accuracy comparison for Meta- WorldWorld
(RQ2), averaged over 3 seeds.
Model
Training Tasks
Novel Task
Push
Push- Wall
Pick-Pllae- Wall
Pick-Place
Multi-Task B1-Multi
0.873
0.893
0.577
0.547
Multi-Task w/ Reasons
REP
0.870
0,900
0.647
0.553
ReCouPLe-EC ReCouPLe.IC
0.863
0.843
0.650
0.663
0.893
0.823
0.657
0.627
Policy learning results. We label the rewards of the novel Pick-Place task's play data using learned mod- els assess how well our reward model transfers to the downstream policy learning of a novel task. As indicated the Figure 4, a similar pattern per- sists from reward learning results. ReCouPLe can better transfer to a novel task without additional pref- erence queries, compared to both of our baselines. Among our variants for orthogonal consistency loss, ReCouPLe- IC slightly outperforms ReCouPLe-EC in terms of overall preference accuracies. Unlike our ManiSkill experiment, in which preference queries consist of a pair of trajectories with a minimal differ- ence in features other than the stated reasons, datasets in Meta- World contain noisy trajectories with differ- ent levels of optimality. Also, each query has a dif- ferent reason behind its preference. Thus, it is less re- components
Pick Place (Novel Task)
0.25
Success Rate
0.15
0.10
0.05
0.00
50000
100000
150000
200000
250000
Step
BT-Multi
ReCouPLe-IC
RFP
ReCouPLee-CC
alistic to assume that peason-orthogome should remain identical across compared trajectories. In this setting, the strict equality constraint enforced by ReCouPLe-EC may overly penalize legitimate dif- ferences unrelated to the stated reason, harming its performance.
Figure Meta- Meta-World policy earning on held-out task, averaged over 3 seeds (mean + std). Both ReCouPLe variants outperform the baselines, showing task transfer capability.
8

---

Published as a conference paper at ICLR 2026
5.3 ABLATIONS AND ADDITIONAL ANALYSIS
Ablations. Table 3 verifies the efficacy of each component our method: ReCouPLe-ne no- consistency removes the consistency constraint and CocouuPL--oo-ccsistteeee----e-ee
Table 3: Mean reward prediction accuracy over ManiSkill manipulation tasks, averaged over seeds. ID and OOD refer to in-distribution and out-of-distriuution tasks, respectively.
further ab- lates the reward-ratio regularizer in the loss fune- tion. Both ablations retain high ID accuracy, but suffer large drops on OOD cases. The consistency loss ensures that preferences are not explained by the non-causal, reason-orthogonal component, and the ratio regularizer prevents the trivial collapse of this component. Together, they account for ReCouPLe's robustness gains under distribution shift.
2-task
ID
00D
ID
00D
ReCouPLe RcCoulPLeeno-comissaayy
0.995
0.872
1,000
0.878
0.745
0.987
0.727
0.950
0.730
Effectiveness of ReCouPLe in image- based control tasks. While our main experiments focus on state- mate- control, previous works demonstrate that manipulation tasks are especially vulnerable to causal confusion from distribution shifts in non-causal visual features (Park et al., 2021). To address this, we evaluate ReCouPLe on image-based tasks across both ManiSkill and a challenging Meta-World visual suite. Table 4 shows the ManiSkill results: with raw visual inputs, all baselines yield significantly lower reward prediction accuracy on the out-o--distribution (OOD) validation set, as their reward models overly attend to spurious features (cube color). RFP raises OOD accuracy only slightly, confirming that naively adding an auxiliary BCE loss for reasons provides an informative but insufficient signal. In contrast, ReCouPLe significantly improves OOD accuracy and outperforms all baselines. Under the 4-task setting, both ReCouPLe variants reach near- serreperfect accuracies (2 0.96).
Table 4: Reward accuracy comparison for in image- over 3 seeds. Methods with highest accuracies in each OOD task are bold-faced.
image-based based ManiSkill (RQ1) enviromment, averaged
2-task
4-task
Distribution
Color Swapped
Distribation
Color Swapped
Model
Pick
Push
Pick
Push
Pick
Push
Place
Pull
Pick
Push
Place
Pull
Single Task
BT
0.833
1,000
0.833
1.000
0,980
1.000
0.167
0.460
0.053
Multi-Task
BT-Multi
0.870
0.999
0.167
0.673
0.867
1.000
0.987
1.000
0.533
0.867
0.833
0.587
Multi-Task Task with Reasons
RFP
0.847
0.990
0.290
0.813
0.867
1.000
0.993
1.000
0.807
0.967
0.947
0.833
ReCouPLe-EC Oreato = 0.4)
0.980
1.000
0.707
0.987
0,980
1.000
0.993
1.000
0.960
1.000
1.000
0.980
ReConPLe-IC (Auto 0.4)
0.940
1.000
0.433
0.927
0.960
1.000
1.000
1.000
0.960
1.000
0.993
0.987
To test robustness in more challenging settings, we designed a Meta- World experiment (Push and Push- Wall) introducing background color non-causal distractor. For Push, preferred trajectories were collected in environments with a blue background, while non-preferred trajectories with compa- rably more suboptimal behavior had gray background. For Push- Wall, this color scheme was flipped (see Figure 6 in Appendix C for a visual overview). We evaluated the reward models based on their accuracy in correctly predicting preferences between trajectory pairs in an OOD environment where these background colors were swapped. We maintained diverse rationales across tasks (4 reasons for Push, 5 for Push- Wall) sharing high-level semanties, sampled from softmax distribution of component- wise advantages identical to the generation process in Section 5.2. As shown in Table 5, the reward prediction accuracy of standard baselines collapse when visual confounders are swapped (e.g.. BT baseline drops to 0.233 on Push OOD). However, ReCouPLe-IC successfully disentangles the shared causal reasoning from the spurious background color and demonstrates robustness under the distribution shift, maintaining accuracies of 0.793 on Push and 0.900 Push-Wall.
Table 5: Reward accuracy comparison for custom visual causal confusion tasks in Meta- World. ReCouPLe successfully avoids overfitting to spurious background colors under diverse reasoning.
Model
Push (Visual OOD)
Push-Wall (Visual OOD)
BT (Baseline) 0.233
0.113
BT-Multi
0.367
0.227
ReCouPLe-EC
0.713
0.760
ReCouPLe-IC
0.793
0,900

---

Published as a conference paper at ICLR 2026
Robustness to Linguistic Diversity. A common concern with canguage-conditione reward models is their tendency to overfit to specific, simplistic instruction templates. To verify that ReCouPLe leams robust semantic representations rather than overfitting to specific phrasing, evaluated its robustness to phrasing diversity in reason labels. We replaced the single canonical reason ("cube is larger") with 16 distinct paraphrases per task, incorporating synonyms ("cube is bigger"). specific descriptors ("red cube is larger"), passive voice ("larger cube is picked"), and negations ("smaller cube is not picked"). Notably, training with diversified rationales actually improved OOD generalization: ReCouPLe-EC's reward accuracies on the Pick (OOD) task increased from 0.820 with original to 0.867 with diversified rationales. This result suggests that our method is not brittle to specific sentence structures. demonstrates the trajectory encoder successfully extracts common causal features and aligns them with lexically different but semantically similar reasons, rather than memorizing exact text strings. We detail the reason labels and full results in Appendix D.
Scalability to Sparse Explanations. Requiring a natural language rationale for every preference annotation can be prohibitively expensive in large-scale offline datasets. To evaluate ReCouPLe's label efficiency, we trained the reward model on datasets where rationales were provided for only subset (25% and 50%) of the preference pairs. For the remaining pairs, the model relied solely on the standard binary cross-entropy loss without rationale alignment. Remarkably, ReCouPLe proved highly label-efficient. Even when 75% of reasons were missing (the 25% subset), ReCouPLe- EC retained a robust out-of-distribttion reward accuracy of 0.783 on the Pick OOD task, vastly outperforming the standard BT baseline (0.540) which lacked reason labels entirely. This indicates that the causal signal derived from a small fraction of rationales successfully propagates through the shared trajectory encoder regularize the entire dataset. Full results for this sparse setting are detailed in Appendix E.
Impact of Preference Sample Size. Beyond the sparsity of rationales, we also analyzed how the absolute volume of preference queries impacts downstream policy robustness. We conducted an extensive scaling experiment on the 2-task ManiSkill setup, varying the dataset size from 200 to 2,000 queries per task. Note that the original experiment in Section 5.1 had 1000 preference queries per task. We observed that baseline methods, such as BT-Multi and RFP, show only marginal improvements as data increases and quickly hit performance ceiling. In contrast, ReCouPLe- EC successfully leverages the additional preference queries. By anchoring the reward function to the provided rationale, the model uses the extra samples to sharpen the distinction between the causal feature and the confounders. Consequently, the performance steadily improves with dataset size, with its reward accuracy on the Pick (OOD) task improving 0.913 at 2,000 queries. The full scaling analysis and tables are provided in Appendix F.
6 CONCLUSION
Summary. We introduce ReCouPLee a lightweight preference learning method that is robust against causal confusion and effective for task transfer by leveraging natural language reasons behind preferences. ReCouPLe turns free-form language rationales into causal projection axes for preference- based reward learning, and uses them to separate out the part of the trajectory that explains the preference, ensuring the model focuses on the feature that actually matters. Furthermore, ReCouPLe leverages shared causal structure across multiple tasks to transfer reward signals without additional preference data or language model fine-tuning. Across two complementary evaluations, ReCouPLe consistently mitigates causal confusion and exhibits strong zero-shot transfer to novel tasks.
Limitations and Future Work. While ReCouPLe shows strong gains, some limitations remain. Our method assumes that rationales are easily available and reliable, yet in practice they may be noisy or costly to obtain. A promising future direction is to explore active learning strategies that selectively query rationales when they are expected to provide the most causal signal. Another limitation is that our evaluation is restricted to simulated domains; validating ReCouPLe in real-worke robotic settings will be essential to assess its robustness under natural human input. Finally, our current framework focuses on reward modeling, leaving additional RL step for policy learning. Extending it to directly learn policies from rationales and preferences (An et al., 2023; Rafailov et al., 2023; Hejna al., 2024), bypassing the reward-learning loop, could further improve efficiency and practicality.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reprodueibility, we provide the source code in the supplementary material and publicly on GitHub https: //github. com/mmmmmmmmo..m mmjmmj- ha-hwwagg//// Lwwag/ReCouPLac We include README file containing commands for all experiments. We describe the evaluation environment and training details in the Appendix.
LLM USAGE
We only used large language models (LLMs) to assist with grammar correction and rewording. No mooell-generated content was used for scientific claims, experiments, or core contributions. All ideas and analyses are original and solely developed by the authors.
ACKNOWLEDGEMENTS
We thank the USC Center for Advanced Research Computing (CARC) for providing us with compute resources. YK and EB acknowledge funding by the Airbus Institute for Engineering Research (AIER) for this project.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. In Conference Robot Learning (CoRL), 2022.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Andrea Bajesy, Dylan Losey, Marcia O'malley, and Anca D Dragan, Learning robot objectives from physical human interaction. In Conference Robot Learning (CoRL), 2017.
Andrea Bajesy, Dylan P Losey, Marcia O'Malley, and Anca D Dragan. Learning from physical human corrections, one feature at a time. In ACM/EEEE International Conference on Stuman-Robo Interaction (HRI), 2018
Chandrayee Basu, Mukesh Singhal, and Anca D Dragan. Learning from richer human guidance: Aug- menting comparison- based learning with feature queries. In ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2018.
Erdem Bıyık, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-frientt ureenfriendly approach to active reward learning. In Conference on Robot Learning (CoRL), 2019.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
Arthur Fender C. Bucker, Luis F. C. Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma. Sai H. Vemprala, and Rogerio Bonatti. Latte: Language trajectory transformer. In International Conference Roboties and Automation (ICRA), 2022.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jeremy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphail Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen. Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadeelld-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research RT,, 2023.
11

---

Published as a conference paper at ICLR 2026
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2023.
Yinpei Dai, Jayjun Lee, Nima Fazeli, and Joyce Chai. Racer: Rich language guided failure recovery policies for imitation learning. International Conference on Robotics and Automation (ICRA), 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning (ICMLR), 2023.
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney, Using natural language for reward shaping in reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (ICML), 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Simon Holk, Daniel Marta, and Iolanda Leite. Predilect: Preferences delineated with zero-shot language- based reasoning in reinforcement learning. In ACM/IEEE International Conference on Robot Interaction (HRI), 2024.
Michael Kelly, Chelsea Sidrane, Katherine Driggs-Camphell, and Mykel J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In International Conference on Robotics and Automation (ICRA), 2019.
Yigit Korkmaz and Erdem BıyIk. Mile: Model-based intervention learning. International Conference Roboties and Automation (ICRA), 2025.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedhack-efficiett interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning (ICML), 2021.
R Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959.
David Manheim and Scott Garrabrant. Categorizing variants of goodhart's law. arXiv preprint 1rXivv1803.04585, 2018
Daniel McFadden. Conditional logit analysis of qualitative choice behavior. 1972.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Jongjin Park, Younggyo Seo, Chang Liu, Li Zhao, Tao Qin, Jinwoo Shin, and Tie- Yan Liu. Object- aware regularization for addressing causal confusion in imitation learning. In Advances Neural Information Processing Systems (NeurIPS), 2021.
12

---

Published as a conference paper at ICLR 2026
Andi Peng, Yuying Sun, Tianmin Shu, and David Abel. Pragmatic feature preferences: Learning reward-relevant preferences from human input. In International Conference on Machine Learning (ICML), 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems (RSS), 2017.
Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems (NeurIPS), 1996.
Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint wxXv:2204.00586 2022.
Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. In Science and Systems (RSS), 2024.
Robotics:
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In International Conference on Robotics and Automation (ICRA), 2023.
Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem BIyik, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36:55681155555 555693, 2023.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao. Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. In Robotics: Science and Systems (RSS), 2025.
Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-bate based reward learning. In International Conference Learning Representations (ICLR), 2023.
Nils Wilde, Dana Kulić, and Stephen L. Smith. Active preference learning using maximum regret. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
Zhaojing Yang, Miru Jun, Jeremy Tien, Stuart Russell, Anca Dragan, and Erdem B1yık. Trajectory improvement and reward learning from comparative language feedback. In Conference on Robot Learning (CoRL), 2024.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint 1rXiv:2107.09644 2021.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S. Yu. Large language models for robotics: survey. arXiv preprint urXiv:2311.07226, 2023.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
Table of Contents
A ManiSkill Experimental Details
14
Details and Implementation of Tasks
14
A.2 Data Collection Details
15
Wera-World Experimental Details
15
B.1 Data Collection
15
B.2 Meta-World Natural Language Task Descriptions
16
B.3 Meta-World Natural Language Rationales
16
B.4 Justification of Rationale Extraction and Softmax Sampling
17
C Details on Meta-Wordd Custom Visual Causal Confusion Tasks
17
Robustness to Linguistic Diversity
17
Scalability to Sparse Explanations
18
F Impact of Preference Sample Size
18
Correlation between Reward Accuracy and Policy Performance
19
A MANISKILL EXPERIMENTAL DETAILS
A.1 DETAILS AND IMPLEMENTATION OF TASKS
As introduced in Section 5.1, design 4 custom manipulation environments (MS-Pick- Larger, MS- MS-Push- Push-Larger, MSsPlacc-Larger, and MS-Pull-Lani Pull--Larrrrr in which each scene has two cubes of different sizes and colors by modifying set of tabletop manipulation tasks provided in ManiSkill (Tao et al., 2025). All the original tasks require the robot to manipulate a single cube or sphere, but we instead put two cubes with different sizes and colors in the scene entangle causal feature (object size) with a non-causal distractor feature (object color), as shown in Table 6. The top row of Figure displays exemplar terminal states for each task, where the robot successfully manipulates the larger cube.
Table 6: Task descriptions and settings for ManiSkill tasks.
Task
Task Description
Larger Cube Color
Smaller Cube Color
MS Pick PirkeLarger
"pick up larger cube to target sphere" "place larger cube target bin" "push larger cube toward target line" "pull larger cube toward green line"
Red
Blue
MS- MIS-Place-Larger
Blue
Red
Si-Push--Lagge
Blue
Red
MS-Pull-Larger
Red
Blue
For all tasks, we randomize the initial pose of two cubes and the target object. Additionally, for each task, we design two versions ground-truth reward function that incentivizes the robot manipulate either the larger cube or the smaller cube. This is later used for training expert RI policies with which we collect trajectories preference dataset.
Observation modalities. For state-based experiments in Section 5.1, we design compact state that includes, for each of the two cubes, its color, size, and pose. To avoid trivial dependence on input ordering, randomize the order of the two cubes in the concatenated state at every episode. This forces methods to identify the correct causal feature (size) rather than relying on position/order color; it thus mirrors the causal-confusion risk present in visuomotor policy learning. We then concatenate this with the robot proprioception and the target object pose to create the state. For -mage-based experiments in Section 5.3, use 128x129 RGB frame input modality.
14

---

Published as a conference paper at ICLR 2026
MS-Pick-Larger
MS-Place-Larger
MS-Push-Larger
MS-Pull-Larger
Figure 5: Terminal states for custom ManiSkill tasks. Each column represents specific task, with the top row showing preferred trajectories (manipulating the larger cube) and the bottom row showing non con-prefrreed trajectories (manipulating the smaller cube). Tasks and their respective color confounder are defined in Table 6.
A.2 DATA COLLECTION DETAILS
For the state- based experiments (Section 5.1), we train SAC (Haarnoja et al., 2018) policies that either manipulate the larger cube the smaller cube, for each manipulation task. The former policy is used to collect preferred trajectories, and the latter is used to collect trajectories that are not preferred. Then, for each task, we generate 1000 synthetic preference pairs by pairing trajectory segments that manipulate the larger and smaller cubes, respectively. These segments are sub-sampled from the oracle's collected trajectories, with the length of For the image-based experiments (Section 5.3), we design motion planning solutions that manipulate either the larger or the smaller cube for each task due to higher sample complexity in training visuomotor RL policies. Similar to the state- generate 500 synthetic preference pairs for each task, where the sub-sampled
based experiments, segment length 64. The terminal states the full trajectories from which these segments are sub-sampled visualized in Figure 5.
B META-WORLD EXPERIMENTAL DETAILS
B.1 DATA COLLECTION
In this section, we provide details on how offline datasets for World tasks are constructed. Similar to Hejna & Sadigh (2023), we first collect trajectories with varying levels of optimality and behavior for each task. These ground-truth policies are scripted, inverse-kinematics. kinematics-base expert policies provided by the Meta-World benchmark (Yu et al., 2020). The collection scheme is as follows:
400 trajectories collected with the ground-trutt policy for the target task, with varying levels of random noise. We use Gaussian noise with standard deviations [0.1, 0.3, 0.5].
100 random trajectories with uniform random actions.
100 suboptimal trajectories collected with the ground-trute truth policy, but for a different random initialization in each target environment.
100 suboptimal trajectories collected with the ground-trutt policy for a different task, other than each target environment. For tasks without wall in the center (i.e., Pick-Place and Push),
These data are generated using the scripted, inverse- -enersseinnmaiicsss based expert policies provided by the Meta-World benchmark. We over-sample trajectories with the noisy ground-truth policy, in order to
15

---

Published as a conference paper at ICLR 2026
guarantee diversity in sampled reasons. If we only collect substantially suboptimal nearly random trajectories, we empirically observed that it significantly limits the diversity of reasons; reasons that related to near-goal states are barely sampled as most trajectories achieve limited task progression.
After we collect trajectories, we construct our preference dataset with the following procedure. We first uniformly sample segments with sub-tru trajetrajeetryy length 32 from the collected trajectories. For each task, we sample 4000 segments and create 2000 preference pairs. Assignment of preference labels and corresponding reason texts is described in Section 5.2. This process results in 6000 total preference pairs, as we have three training tasks (Pick-Place- Place-Wall, Push, bash.
For held-out validation dataset for reward accuracy evaluation, simply repeat the same process as training dataset. We however collect 100 preference pairs per task, which results in 400 total preference pairs training tasks and novel task). We also collect play data (i.e.. offline RL data without reward label) in the same process. For each task, we collect 2000 preference pairs, which we later label reward values with our learned reward models and train offline RL policies with.
B.2 META-WORLL NATURAL LANGUAGE TASK DESCRIPTIONS
Table 7 has compiled list of task descriptions lask. As described in Section 3.2, convert these task descriptions to create task embeddings 0.
Table 7: Natural language task descriptions for Meta- World tasks.
Task
Task Description
Push
"make contact and push puck to goal"
Push Wall
"make contact, bypass wall via waypoint, and push puck goal" "pick up puck, lift it, and place on goal"
Pick-Plac Place
Dick-Place- Wall
"pick puck, use waypoint to bypass wall, and place it on goal"
B.3 META-WORKS NATURAL LANGUAGE RATIONALES
Table has a compiled list reasons feeason As described in the data generation section of Section 5.2, sample the reason behind the preference from softmax distribution of component wise advantages. Here, each reason corresponds to an appropriate feature component in the total reward function, where the environment reward be linearly decomposed into interpretable feature components. This decomposition is based on the reward formulation of Meta-world environments Yu et al. (2020).
Table 8: Reason codes used per task.
Code
Reason Text
Push
"pushes more decisively after making contact" "pushes puck closer goal"
Push
Push-War
"pushes puck toward waypoint"
Push-Wir Wall
"guides puck past wall"
Push-Wall
"pushes puck toward goal after clearing wall" "maintains firm grip on puck"
Push & Push-Waa
Push & Push-Waa
"makes contact with puck sooner"
Pick-Place
"keeps firm grip while moving puck toward goal"
Pick-Place
"carries puck toward goal while lifted"
Pick-Place- Wall
"keeps firm grip while moving puck toward waypoint"
Fick-Place-War
"carries puck toward waypoint while lifted"
Pick-Places Place-Wall
"clears wall with puck lifted"
Pick-Place Wall
"carries puck toward goal after clearing wall" "lifts puck cleanly" "finishes goal spot"
Pick-Place & Pick-Place- Wall
All tasks
16

---

Published as a conference paper at ICLR 2026
B.4 JUSTIFICATION OF RATIONALE EXTRACTION AND SOFTMAX SAMPLING
As described in Section 5.2, we sample the reason behind the preference from a softmax distribution of component-wise advantages, rather than from deterministic argmax. This design choice is grounded in Discrete Choice Theory, specifically the Random Utility Model (McFadden, 1972) and the Luce Choice Axiom (Luce, 1959). In numan-in-the-loop setting, users are probabilistic; while they likely cite the dominant factor, they occasionally cite secondary relevant factors. Sampling from the softmax simulates a realistic user providing reasons proportional to their saliency. Using an argmax would assume "perfect" annotator, artificially reducing the diversity of the training signal.
C DETAILS ON WORLD CUSTOM VISUAL CAUSAL CONFUSION TASKS
We provide additional implementation details for the custom Meta- World environments introduced in Section 5.3. Similar to the data collection scheme in Appendix B, the scripted expert policies (Yu et al., 2020) with different levels of Gaussian noise. Specifically, preferred trajectories were collected with Gaussian noise of 0.1, while non-preferred trajectories were collected with Gaussian noise = 1.0, resulting in significantly more suboptimal behavior. By design, the background floor color serves non-causal distractor that perfectly correlated with these preference labels during training (e.g., blue for preferred and gray for non-preferred in Push, as shown in Figure 6). This setup allows to evaluate whether reward models rely on these spurious visual cues or instead capture the underlying task completion.
Push Task
Push- Wall Task
(a) Pref. (c 0.1)
(b) Non-pref. (e 1.0)
(c) Pref. 0.1)
(d) Non-pref. (c 1.0)
Figure 6: Visualizations of terminal states from the custom visual causal confusion environments. During training. background color is perfectly correlated with trajectory preference. Robustness is evaluated on an OOD distribution where this mapping is inverted, requiring the reward model to generalize beyond spurious visual features.
D ROBUSTNESS LINGUISTIC DIVERSITY
To verify that our approach generalizes across different natural language expressions rather than overfitting to specific rationale phrases, we assess ReCouPLe's robustness to phrasing and semantic diversity. In the original ManiSkill experiments, single canonical reason was used (Section 5.
We generated 16 distinct paraphrases per task to replace the single canonical reason. These variations included:
Synonyms: "cube bigger", "object is larger"
Specific Descriptors: "red cube larger", "blue object bigger"
Passive Voice: "larger cube is picked"
Negation/Contrast: "smaller cube is not picked", "tinier object is not placed"
As shown in Table 9, training with diversified rationales actually improves OOD performance. This demonstrates that the trajectory encoder successfully extracts common causal features and aligns them with semantic representations shared across varied phrasing, rather than relying on simple pattern matching.
17

---

Published as a conference paper at ICLR 2026
Table 9: Robustness Rationale Diversity (ManiSkill OOD Reward Accuracies)
Model
Pick (OOD)
Place (OOD)
BT (Baseline)
0.540
0.830
ReCouPLe-EC (Original)
0,820
0,940
ReCouPLe-EC (Diversified)
0.867
0.967
ReCouPLe-IC (Original)
0.633
0,807
ReCouPLe-IC (Diversified)
0.620
0.820
E SCALABILITY TO SPARSE EXPLANATIONS
Annotating every preference pair with rationale be expensive. We investigated method's label efficiency by training on datasets where rationales were provided for only a subset (25% and 50%) of the preference pairs. For pairs without reasons, simply minimized the standard binary cross-entropy loss following the Bradley-Terry model. As shown in Table 10, ReCouPLe is highly label-efficient: when 75% of reasons are missing, the causal signal from small set of rationales propagates to the entire dataset.
Table 10: Scalability to Sparse Explanations (ManiSkill OOD Reward Accuracies)
Model
Pick (OOD)
Place (OOD)
BT (Baseline)
0.540
0.830
ReCouPLe--CC (25% Reasons)
0.783
0.940
ReCouPLe-EC (50% Reasons)
0.767
0.947
ReCouPLe-IC (25% Reasons)
0.673
0.807
ReCouPLe-IC (50% Reasons)
0.660
0.847
IMPACT OF PREFERENCE SAMPLE SIZE
To address how preference sample size impacts our method, we conducted an extensive scaling experiment on the 2-task ManiSkill setup, varying the dataset size from 200 to 2,000 queries per task. While baseline methods like BT-Multi and RFP show marginal improvements data increases, they quickly hit performance ceiling because adding more samples merely reinforces their reliance on spurious features. In contrast, ReCouPLe- EC successfully leverages additional data to sharpen the distinction between causal features and confounders.
Table 11:
Scaling Analysis on ManiSkill (2-Task Setup). The 1000-query column corresponds to the original dataset size used in our main experiments in Section 5.1
Preference Queries
200
500
1000
2000
Task: Pick (OOD)
BT-Multi
0.573
0.627
0.600
0.693
RFP
0.540
0.613
0.620
0.760
ReCouPLe-EC
0.733
0.820
0.820
0.913
ReCouPLe-IC
0.547
0.580
0.633
0.793
Task: Place (OOD)
BT-Multi
0.667
0.760
0.820
0.767
RFP
0.733
0.720
0.800
0.760
ReCouPLe-EC
0.747
0.847
0.940
0.920
ReCouPLe-IC
0.773
0.793
0.807
0.807
18

---

Published as a conference paper at ICLR 2026
G CORRELATION BETWEEN REWARD ACCURACY AND POLICY PERFORMANCE
In standard reward learning, high reward model accuracy does not always translate better policies due to "Causal Goodhart" effects (Manheim & Garrabrant, 2018; Gao et al., 2023), where a proxy reward model relies on features that correlate with the gold label but do not cause it. ReCouPLe is explicitly designed to address this by using the rationale to project the reward onto the causal axis, forcing the model to ignore spurious features. Consequently. OOD reward accuracy becomes reliable binary detector of causal disentanglement, Our empirical results confirm this: we observed Pearson correlation of 0.924 and a Mean Spearman's 's Rank correlation of 0.933 between OOD reward accuracy and policy success in Meta- World tasks in Section 5.2.
19


Reviews:
[{"review_id": "zpqh5JV1p3", "reviewer": "Reviewer_mHRs", "summary": "This proposes a method to reduce causal confusion in preference-based reinforcement learning, where reward models often overfit to spurious correlations. ReCouPLe augments standard binary preference data with short natural-language rationales, treating each rationale as a causal projection axis that aligns trajectory embeddings with the stated reasons. This decomposition isolates causal features that truly drive preferences while ignoring irrelevant correlations. Experiments on ManiSkill and Meta-World show that ReCouPLe significantly improves out-of-distribution robustness and enables zero-shot reward transfer to new tasks without additional annotations.", "questions": "- It seems that the RFP baseline is just simple extension of Multi-BT operations twice, once for each of two distinct text instructions. Given this, what could explain RFP’s superior performance compared to Multi-BT, despite their apparent similarity?\n- Could the authors clarify the specific domains or conditions under which ReCouPLe-EC and ReCouPLe-IC are most advantageous? At present, the explanations appear somewhat post-hoc, matched to results rather than grounded in theory.\n- Can the method scale to setups with more than three distractors? For instance, beyond object size and color, could it handle additional factors such as object shape while maintaining consistent reasoning and performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Employs language-based causal alignment by combining simple preference data with natural-language rationales, guiding the model to focus on causally relevant features that reflect the user’s true intent.\n- Demonstrates zero-shot reward transfer to unseen environments without requiring any additional preference collection or reward model training.", "weaknesses": "- The task instructions used are overly simple, and it is unclear whether incorporating language reasoning truly provides an advantage in this setup. It would strengthen the work to include experiments using more diverse language rationales or to analyze whether varying the linguistic expressions for the same rationale improves performance.\n- The rationale extraction process appears heuristic and heavily dependent on ground-truth rewards (especially in MetaWorld). This reliance limits the method’s applicability to benchmarks where reward engineering is difficult or unavailable.\n- Because each preference annotation requires a corresponding rationale, the method is not easily scalable to online RL settings. It would be valuable to discuss how ReCouPLe could be modified or extended for online applications.\n- Prior work [1] has shown that image-based control tends to exacerbate causal confusion—particularly in MetaWorld—but the paper does not provide results for visual reward learning and policy training on MetaWorld to validate this.\n\n**References**\\\n[1] Subtask-Aware Visual Reward Learning from Segmented Demonstrations, ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYtYxNlosS", "reviewer": "Reviewer_fFvS", "summary": "The paper presents a method to incorporate preference rationales into the learning of the trajectory representation used by the reward model to assign rewards per state-action pair. The method relies on the presence of a language-based task description that is embedded and then compared to each state-action pair embedding from the given trajectory. The trajectory learning approach combines the current representation of each state-action pair in a trajectory with an embedding of the provided preference rationale, and relies on a representation of the trajectory that excludes those features most relevant to the preference rationale. The training objective then combines three objectives that aim to: (1) force the state-action pair embedding to focus on rationale-relevant features, (2) ignore features not relevant to the rationale, and (3) prevent a collapse of the trajectory representation. The experiments are run on two different domains, ManiSkill and Metaworld, each of which have different levels of difficulty and rationale complexity. The results suggest the proposed method, ReCouPLE, out performs baselines on reward modelling and learned-policy quality.", "questions": "- How much data is used in the ManiSkill experiments?\n- What is the strength of correlation between the reward model's performance and policy performance? This important to answer as there is evidence on the LLM side of reward learning and PbRL that reward model preference-labelling performance is not always strongly predictive of how well the reward can be used to train a policy. This makes reward model selection challenging.\n- Why is the color swap reward model accuracy higher in Table 1 for 4-task + pull + RFP?\n- How do the baseline methods compare ReCouPLe in terms of number of learnable parameters? This would speak to the generally upper bound on method expressivity, which impacts general overall performance.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper applies a method that has been used in goal-conditioned RL to the preference learning setting to boost the limited amount of information available in a preference label.\n- Addressing the limited amount of information available in a preference signal is a key problem to solve improve general field ability to learn effective and robust reward models.\n- The paper is easy to read and follow.", "weaknesses": "- Some experiments are missing to truly understand where ReCouPLE performs well:\n     - impact of number of preference samples on reward and learned-policy quality\n     - impact of noisy preference labels\n     - impact of noisy rationales\n     - combining data with and without rationales as, in practice, collecting large datasets with rationales will be expensive and impractical\n- The ManiSkill experiments lack diversity in rationales, so it is not clear how well the results will generalize.\n- While one of ReCouPLe-EC or -IC typically are a best performer, frequently one or more baseline is better than at least one of the two versions, and this varies by task for ManiSkill. This suggests the method is fragile and task-specific. There is no discussion of what might lead to this performance sensitivity. Given the sensitivity of ReCouPLEe performance to the task in ManiSkill, there should be more test tasks for MetaWorld. \n- There should be MetaWorld test tasks less closely related to the training tasks to understand how well the method works in more challenging and realistic multi-task settings.\n- It is not possible to assess how impact or meaningful the ReCouPLe performance gains are. When presenting the test results for policy training, results should be provided when training on the true reward. For example, if the training on the true reward provides a success rate of 0.8, a success rate difference of 0.07 is less meaningful.\n- For the ablations and analysis results in Section 5.3 (ablations and image-based control tasks), results should be provided for policy learning, not only reward model accuracy, especially as the reward model accuracy score differences are large. While there may be a difference in reward model accuracy, it is important to understand how much of an impact that difference has on learned-policy performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXjZdWm8tV", "reviewer": "Reviewer_NmL7", "summary": "This paper considers the problem of reward learning in PbRL, where the data can be confounded. The authors propose ReCouPLe, which use natural language task description/explanations to resolve such spurious correlations. Specifically, the rewards are decomposed into two components (i.e., a component parallel to the reason and an orthogonal component), and only the parallel component is used to capture preferences via the BT model. The orthogonal component (non-causal feature) is then penalized to be consistent between trajectories. Experiments show that the proposed approach outperforms standard baselines that do not utilize the auxiliary information.", "questions": "1. Line 393: This sampling distribution may lead to completely incorrect reasons given trajectory pairs. Can the authors elaborate on this design choice?\n2. Line 409: What does it mean to decompose and recompose semantically? Can the authors provide additional evidence for this behavior?\n3. How sensitive is the method regarding task descriptions / reason codes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and easy to follow. The environments and datasets are carefully designed to capture confounding problems in PbRL and the experiments clearly demonstrate the effectiveness of the proposed approach. Although it is not immediately clear how frequently causal confusions occur in real-world settings, I believe the method has potential applications beyond the specific cases studied here. In particular, it could be useful for more general problems where explanations are available during reward modeling (e.g., Table 4 shows that in-distribution tasks can also benefit from the proposed method), making it of interest to the broader community.", "weaknesses": "My main concern is that it is well-known that language models can be sensitive to phrasing, and this is not really explored in the paper (e.g., using different LMs, paraphrasing).\nIt is also not clear what kind of LMs should be used to best extract the task/reason embeddings.\nFor Fig 3, I think it would be nice to have in-distribution performances also reported; otherwise, it is difficult to tell if the performances stagnate because of problems from RL training or reward modeling.\nAside from the ablations presented in Sec 5.3, I would suggest the authors also ablate on the LM side to demonstrate the robustness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)