Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
Authors: Jeremy Tien
Jerry Zhi-Yang He
Zackory Erickson
Anca Dragan
Daniel S. Brown
Venue: ICLR 2023 poster (2023)
Keywords: reward learning
robustness
preference-based learning

Abstract: Learning policies via preference-based reward learning is an increasingly popular method for customizing agent behavior, but has been shown anecdotally to be prone to spurious correlations and reward hacking behaviors. While much prior work focuses on causal confusion in reinforcement learning and behavioral cloning, we focus on a systematic study of causal confusion and reward misidentification when learning from preferences. In particular, we perform a series of sensitivity and ablation analyses on several benchmark domains where rewards learned from preferences achieve minimal test error but fail to generalize to out-of-distribution states---resulting in poor policy performance when optimized. We find that the presence of non-causal distractor features, noise in the stated preferences, and partial state observability can all exacerbate reward misidentification. We also identify a set of methods with which to interpret misidentified learned rewards. In general, we observe that optimizing misidentified rewards drives the policy off the reward's training distribution, resulting in high predicted (learned) rewards but low true rewards. These findings illuminate the susceptibility of preference learning to reward misidentification and causal confusion---failure to consider even one of many factors can result in unexpected, undesirable behavior. 

Published as a conference paper at ICLR 2023
CAUSAL CONFUSION AND REWARD MISIDENTIFICA- TION IN PREFERENCE-BASEI REWARD LEARNING
Jeremy Tien
Jerry Zhi-Yans He University of California, Berkeley
University of California, Berkeley rtienbberkelly edu
Zackory Erickson Camegie Mellon University
Anca D. Dragan University of California, Berkeley
Daniel S. Brown University of Utah
ABSTRACT
Learning policies via preference-based reward learning is an increasingly popu- lar method for customizing agent behavior, but has been shown anecdotally to be prone to spurious correlations and reward hacking behaviors. While much prior work focuses on causal confusion in reinforcement learning and behavioral cloning, we focus on a systematic study of causal confusion and reward misiden- tification when learning from preferences. In particular, we perform a series of sensitivity and ablation analyses on several benchmark domains where rewards learned from preferences achieve minimal test error but fail to generalize out- of-distribution states-resulting in poor policy performance when optimized. We find that the presence of non-causal distractor features, noise in the stated prefer- ences, and partial state observability can all exacerbate reward misidentification. We also identify set methods with which to interpret misidentified learned rewards. In general, we observe that optimizing misidentified rewards drives the policy off the reward's training distribution, resulting in high predicted (learned) rewards but low true rewards. These findings illuminate the susceptibility of pref- erence learning to reward misidentification and causal confusion-failee to con- sider even one of many factors can result in unexpected, undesirable behavior.
INTRODUCTION
Preference- reward learning (Wirth et al. 2017: Christiano et al., 2017 Stiennon et al. 2020 Shin et al. 2023) is popular technique for adapting AI systems to individual preferences and learning specifications for tasks without requiring demonstrations or an explicit reward function. However, anecdotal evidence suggests that these methods prone to learning rewards that pick up on spurious correlations in the data and miss the true underlying causal structure, especially when learning from limited numbers of preferences (Christiano al. 2017]] Ibarz al. 2018 Javed] et al. 202I). While the effects of reward misspecification have recently been studied in the context of reinforcement learning agents that optimize proxy reward function (Pan al. 2022). and the effects of causal confusion have been emphasized in behavioral cloning approaches that directly mimic an expert (De Haan al. 2019 Zhang al. 2020 (waamy et al. 2022). we provide the first systematic study of reward misidentification and causal confusion when learning reward functions.
Consider the assistive feeding task in Fig. 26 Note that all successful robot executions will move the spoon toward the mouth in the area in front of the patient's 's face-few, if any, executions demonstrate behavior behind the patient's head (the exact likelihood depending on how trajectories for preference queries generated), In practice, we find that the Icarned reward will often pick up the signed difference between the spoon and the mouth rather than the absolute value of the distance. The two correlate the training data, but using the former instead of the latter leads to the robot thinking that moving behind the patient's s head is even better than feeding the person!
Our contribution is study of causal confusion and reward misidentification it occurs in preference-bast reward learning. First, we demonstrate the failure of preference- based reward learning to produce causal rewards that lead desirable behavior three benchmark tasks, even

---

Published as a conference paper at ICLR 2023
when given large amounts of preference data in these settings, the learned reward has high test ac- curacy but leads to poor policies when optimized. We then study the effect of several factors related to reward model specification and training: the presence of non-causal distractor features, reward model capacity, noise in the stated preferences, and partial state observability. For each of these, perform an analysis of what errors the learned reward has, how it compares to the ground truth reward, and the amount of distribution shift it induces during policy optimization. Another of our contributions is to point to the importance of data coverage and interactive techniques that iteratively or actively query for feedback to help learners disambiguate causal features from correlates. Over- all, our findings caution that there are many aspects that can make preference learning challenging, from the way we define state-whrr to include and what not to include in the input to the learned todel-to to how we mitigate the effects of noisy or biased preference data.
1.1 RELATED WORK
Reward Hacking and gaming behaviors are known to commonly occur in reinforcement learn- ing (Ng et al., 1999; Krakovna et al. 2020) and reward learning (Christiano et al.] 2017 Ibarz et al., 2018: He & Dragan 2021). However, these behaviors
often only mentioned anecdo- tally Krakovna et al.] 2020). Recently, Pan et al. (2022) proposed to systematically analyze reward misspecification in RL by creating set of domains where the agent optimizes a hand-engineered proxy reward function and then studying when this leads to incorrect behavior. By contrast, we study cases where the reward function must be learned.
Preference Learning, is a popular method for training AI systems when a reward function is un- available (Wirth al. 2017: Christiano et al.] 2017: Biyik & Sadigh 2018: Brown al., 2020a Ouyang et al.) 2022 Shin et al.) [003]] Liu et al. 2023). Preferences are often easier to provide than demonstrations or raw numerical values (Wirth et al., 2017) since they do not require expert proficiency or fine-grained feedback. However, optimizing reward function that has been trained using preferences can sometimes lead to unintended behaviors. Anecdotal evidence of this has been documented for Atari games (Christiano et al.) 2017] Ibarz et al.] [018]] [Bronn et al.) [2020a), simple robot navigation tasks (Javed et al. 2021). and language model fine-tuning (Stiennon et al. 2020); however, there has been no systematic study of causal confusion when learning reward functions.
Causal Confusion in Imitation Learning has previously been studied in the context of behavioral cloning (Pomerleau 1988, Torabi et al. 2018). Prior work shows that behavioral cloning approaches suffer causal confusion due to "causal misidentification. " where giving imitation learning policies more information leads to worse performance (De Haan et al., 2019) due temporally correlated noise in expert actions (Swamy et al.] 2022). Similarly, we find strong evidence of causal misiden- tification when expert noise is present. Zhang et al. (2020) use causal diagrams to investigate causal confusion for simplified imitation learning tasks with discrete actions and small numbers of states when the features available the demonstrator are different from those of the imitator. By contrast, we study the effects of changes to the observation space when performing preference learning over continuous states and actions and when using non-linees reward function approximation.
Goal Misgenerallization happens when an RL agent has known goal and some environment fea- tures are correlated and predictive of the reward on the training distribution but not out of distri- bution (Langosco et al.) 2022 Shah et al., (022). This setting is similar to ours in that there is misidentification of the state features that are causal with respect to the reward. However, goal mis- generalization assumes that the ground-truth reward signal is always present during training. We show that when learning a reward function, the learned reward can be misidentified, leading to poor RL performance. By contrast, Langosco al. (2022) and (Saah et al. (2022) show that even if the learned reward is perfect, RL can still fail due to spurious correlations during training.
1.2 BACKGROUND: REWARD LEARNING FROM PREFERENCES
We model the environment as finite horizon MDP PPteeran 2014), with state space S, action space A, horizon T, and reward function r S x A + R. The reward function is unobserved and must be learned from preferences over trajectories. Using the Bradley-Terry model (Bradley & Terry 1952), the probability trajectory тB is preferred over another trajectory TA is given by exp(r(TB)) P(TA x TB) = exp(r(ra)) -exp(r(TB)
(1)
2

---

Published as a conference paper at ICLR 2023
Jj (1r -bl.....
(1): :0l--.. ...
(2t 2/1
( =+))
(10 20. 2bl.- .... 11 x/
(1r Till
1
Σ
Ex(tt)
Σ r(zz ui]
En(lo will
f([z, 21)
f([x, 21)
(a)
(b)
Figure I: Causal structure of preference- reward learning. The reward function i is learned from preference labels over trajectory pairs (Ti, Tj). Unobserved variables are denoted by dashed lines. Unobservable user noise v) affects the preference node. In (a) the true reward is not affected by nuisance variables 2. In (b) the true reward based on unobserved state features u.
where r(r) (a.ajer r(s, a) and r = (sg; ao........................ S8, ,r,
To learn reward function from preferences, we assume access to set of pairwise preference labels P over trajectories F...... TN, where (i, j) E P implies that т; < Tj. We then optimize a reward function rg :S x A parameterized by that maximizes the following likelihood (see Alg. T
exp(ro(T,))
((0)=
II (i.))P
+
REWARD MISIDENTIFICATION
Fig. displays the causal structure of preference- based reward learning where pairwise preferences (in the form of binary label) are given based features of that are causal and influence r(z), as well as other features z that are nuisance variables and have bearing r(z). Note that 2 may very well exhibit correlations with 24, potentially due to their sharing the same causal parent or there being biases during data collection. Given preference labels
observed reward function (Fig. lap. There are
trajectory pairs, the goal of prefenence-based reward learning is to learn reward function f(x; 2) that best matches the stated preferences. In Fig. 1b there are state features which are causal with respect the true reward, but are unobserved by the learning agent. both cases, unobserved human bias and noise, denoted by 7), also affects the preferences labels.
The learned reward f may be able achieve low and out test set by making use of 1) nuisance variables that correlate with causal variables or 2) faulty/incomplet correlations between causal variables and r that happen hold true for the train- ing data distribution. However, performing reinforcement learning on misidentified learned reward values f[xx, z)) leads distribution shift, resulting in behaviors with low performance under the true reward function We define this behavior of learning reward that achieves low test error but results in poor performance (under the true reward function) when the learned reward function is op- timized via RL as reward misidentification. We note that the causal graphs provide in Fig. [ a meant to shed light on a typical way misidentification can occur. In reality, the sources of misiden- tification can vary widely variable may be causal in certain context and not in another, causal variables may not combined properly, etc.
near-perfect performance
held-
3 EXPERIMENTAL SETUP
To facilitate reproducibility and encourage future research causal reward learning, we open-source our code and training datasets: https: /tpp:s//sites.. google com/vi com/view/ mewssal-rewrdd-coom
This combination of domains and training data forms the first set of benchmarks for studying reward misidentification and causal reward confusion.
3

---

Published as a conference paper at ICLR 2023
Environments for Preference Learning. We identify a set of benchmarks that exhibit re- ward misidentification. In Reacher (Brockman et al., 2016) (Fig. (2)). the goal is to move an end effector desired goal location. In Feed- ing (Erickson et al., (020) (Fig. 2b). the goal is feed the human using a spoon carrying pieces of food. Finally, in Itch Scratching (Erickson al., 2020) (Fig. (2)). the goal to scratch an itch location on the human's 8 arm.
Test Data
Learned Policy
-15.7
(a) Reacher
Test Data
Learned
True Rewards and Preference Generation. Each domain has a predefined "true" reward function r (see Appx. (A.3). This enables us to create synthetic demonstrations and prefer- ence labels via noise injection: adding differ- ent amounts of noise to expert policy trained
-136.4
143.2
566.3
on r (details in Appx. A.2)). As shown by Brown al. (2020b), adding this type of dis- turbance will result in monotonically decreas- ing performance in expectation and produce diverse dataset for preference learning. Gleave 20200) propose a similar approach that switches between an expert policy and a ran- dom policy to produce good coverage dis- tribution over states. In Appx. A. we com- pare generating trajectories using noise injec- tion versus using different RL checkpoints, proposed by Brown et al. (2019), and find that noise injection leads to similar or better perfor- mance. Note that while we use the ground-trun reward function for obtaining preference labels, we assume no access to this reward function during policy learning; we first learn model of the true reward function from preference labels P, and then RL on the learned reward function to produce policy. We then evaluate the learned policy on the true reward function
Test Feeding
Folley
-1.8
25.3
23.3
(c) Itch Scratching
Figure 2: Poor behaviors resulting from learned rewards (rightmost column), despite high accu- racy on the test data (left two columns). The pre- dicted trajectory rewards produced by the learned reward function are displayed under each image; each image corresponds a trajectory.
Evaluating Learned Reward Functions. To establish that learned reward misidentifies the causal structure, we first check for low test error and establish that the learned reward performs well on unseen in-distribution test data (thereby ruling out model selection and training failures like not having enough data, capacity, regularization). We then show that the learned reward fails in two ways: 1) it leads
policy (called PREF) that has poor performance with respect to the true reward, and 2) it prefers its poorty-performing optimized policy PREF over the optimal policy with respect to the true reward (GT), indicating that PREF's poor performance not due to RL failures. Finally, we analyze the learned reward qualitatively via gradient saliency maps and quantitatively via the EPIC pseudometric (Gleave et al.) 2020) and KI. divergence to elucidate the effects of the reward error by quantifying the distribution shift induced by policy optimization of the learned reward. Details on hyperparameters and evaluation methods are in Appx. A. and A.6
EVIDENCE OF CAUSAL CONFUSION
Before varying different factors that affect the performance the learned reward, we start with a generous setting where provide large amounts of data and add features to the default observation such that all necessary information needed to infer the ground truth reward, TRUE, is available. We produce the preference training data as detailed in the previous section. Table T details the results.
We find that the learned reward achieves high preference test accuracies that are comparable to the training accuracy. This indicates that the learned model does not overfit, and there is sufficient
4

---

Published as a conference paper at ICLR 2023
Table 1: Empirical evidence of causal confusion. We compare policies optimized with a reward learned from preferences (PREF) against policies optimized with the true reward (GT). State fea- tures on which preferences are based are oully-observable. Reward functions were trained with 52326 unique pairwise preferences. Both PREF and GT are optimized with IM RL iterations and averaged over 3 seeds. Despite high pairwise preference classification test accuracy, the policy per- formance achieved by PREF under the true reward is very low compared with GT. However, the reward learned from preferences consistently prefers PREF over GT. This suggests that preference- based reward learning fails to learn a good reward for each of these tasks.
PREF. LEARNING ACC.
RL POLICY PERFORMANCE
LEARNED
TRUE (PREF/GT)
SUCCESS (PREF/GT)
DOMAIN
TRAIN
VAL
TEST
(PREF/GT)
REACHER
0.954
0.956
0.966
44.988 / 3.395
-42.716 -5.560 0.100 / 0.827
FEEDING
0.987
0.976
0.976
277.152 124.016
-27.432
128.933
0.603 / 0.990
ITCHING
0.954
0.933
0.928
16.588 / 10.282
-47.190
248.397
0.013 / 0.970
model capacity and data. In later sections, we observe that even models with over 99% test accuracy sometimes fail to produce good polices. We also find that the learned reward prefers PREF to GT. This shows that the poor performance is not due to issue in RL training. Unfortunately, the actual performance of PREF is disastrous: it has poor TRUE reward (compared to GT) and poor success rates. Overall, the learned reward incentivizes poor behavior, despite high test accuracy.
For Reacher, we observe that PREF chooses to simply spin very fast rather than reaching for the target. We note, however, that the learned reward correctly classifies the leftmost image in Fig. 2a (where the agent just folds its arm) as being worse than the middle image (where the agent success- fully reaches the target), and does so with 96.6% accuracy on such diverse pairs of comparisons. One would think that this would apply to the behavior in the rightmost image, where the agent folds its arm as well (and subsequently spins), but the learned reward turns out to strongly prefer the right- most case. For Feeding, the learned reward encourages minimizing the signed difference between the spoon and the mouth rather than the absolute value of the difference. This is because the ma- jority of trajectories approach the mouth from in front. Fig. 88 shows that there far more states that have low reward at negative 'diff.y' values than states that have low reward at positive 'diff y values. As a result, the learned reward (Fig. 9b) correctly identifies spilling food (left) as being worse than feeding (middle), but goes further and incentivizes bringing the spoon towards and even behind the head in an attempt to minimize the signed difference. For Itch Scratching, the learned reward correctly identifies flailing (left) as being worse than actual scratching (middle). However, as seen in Fig. Pc] the Itch Scratching agent (spuriously) learns a higher weight on two components of the action (corresponding to two robot arm joints), which results in the agent turning the last section of the arm in a circle while trying to keep the end effector close to the itch type of flailing! It also falls into the same trap as the Feeding agent in minimizing the signed rather than the absolute difference (Fig, |c shows there is a bias toward negative values of 'diff_x" rather than even distribution of states across both positive and negative values). Fig. 2 summarizes this behavior. Appx. A.3 provides plots of the aforementioned features' spurious correlations with the true reward and gradient saliency maps of the learned rewards.
FACTORS THAT MAY LEAD TO CAUSAL REWARD CONFUSION
We examine various factors of the preference-based reward learning problem setup and their effects on reward misidentification. The motivation for exploring each is as follows: Our experiments with Distractor Features draw directly from the findings in Sec. 4. where we find that certain non- causal features may be spuriously correlated with the reward. Experiments on Model Capacity are inspired by Pan et al. (2022)'s 's findings the effects of increased agent capabilities on reward hacking. Exploring Noise in Stated Preferences is inspired by the fact that preference data collected from humans is often rife with various biases and noise. Studying Partial Observability of Causal Features is motivated by the fact that it is not always possible to fully-observe all causal features in the real world. Complex Causal Features is inspired by the observation that the causal reward is
5

---

Published as a conference paper at ICLR 2023
Reward
Reward
DATASET SIZE
DISTRACTORS
40
-10
WITH
WITHOUT
-60
M
53.731
6.762
0.0
2.5
5.0
7.5
46.032
7.366
M
Distractor features
(a) RL Performance
(b) Removing distractors
(c) KL Divergence
Figure 3: Distractor features. Fig. 3a compares performance with distractors present in the learned
reward with performance without distractors present in the learned reward across two dataset sizes.
Fig. 3b sensitivity analysis the number of distractor features done the M dataset size from Fig. 3al In Fig. 3el we see that distractor features result much larger distribution shift (as measured by the KL divergence between state-action pair distributions at reward learning and RL).
(a) Gradients With distractors
(b) Feature Ranges
With distractors
Figure 4: Saliency maps: distractor features. Fig. 4a is gradient saliency map the (misidenti- fied) learned reward. Fig. 4b is heatmap displaying the ranges of each feature over the course of trajectory produced by the learned reward. The large range of the 'vel_1' feature and its correspond- ing positive gradient are the reason why the trajectory does well under the learned reward.
often a complex function of state variables and, in complex tasks, features may be causal in some contexts and non-causal in others. The above factors (which may often be overlooked) contribute significantly to reward misidentification and the eventual success or failure of the reward model.
5.1 DISTRACTOR FEATURES
One reason for reward misidentification is the presence of nuisance features in the input that are spuriously correlated with preference labels. To study this effect, we incrementally remove such 'distractor' features and test the impact this has on the learned reward. For example, for Reacher, we know that joint angles and angular velocities purely on distance between end effector and target and the norm of the action (a control penalty) (see Appx. A.4 for complete list of causal and distractor features). As all distractors drastically improves performance, and, indeed, the more such features are left in the input, the worse the performance gets Reacher, almost linearly. Granted, removing distractors does also slightly improve the validation and test performance in this case-indicating that there is some signal in the training data to help discern the spuriousness of certain features. Nonetheless, over the several experiments in this paper, we find that the validation error is not strongly correlated performance-aee some cases, increasing the validation accuracy results worse performance.
not causal to the ground truth reward, which is based
in Fig. removing
Fig. 2a depicts the policy of the learned reward with distractors- the Reacher robot learns to simply fold its arm and spin. Looking at the gradient saliency maps (Fig. 4 illuminates why this is the case. Firstly, in Fig. 4a we observe that the learned reward is misaligned with respect to 'diff_y', y the feature corresponding to the difference between end effector position and target position along the y- axis; specifically, we note that the learned reward actually rewards an increase in 'diff_y', rather than a decrease. Next, we notice that the reward doesn't 't penalize action evenly. penalizes 'torque_1' (the one responsible for the spinning), but rewards 'torque_2 across nearly all timesteps. It should instead be largely a negative penalty, as seen in Fig. [Tal Lastly and perhaps most importantly, we observe in Fig. 4b that 'vel 1 and 'vel 2' have very large feature ranges, corresponding to the large

---

Published as a conference paper at ICLR 2023
variations in angular velocity achieved by the Reacher robot's spinning behavior. Looking back to the gradient for each of these velocity features in Fig. 4a we observe that the reward has slight positive gradient with respect to each which stems from slight correlation between angular velocity and reward. Fig. [a shows this correlation; there is slight bias in the training data toward states with high reward and high velocity. By spinning fast, the Reacher robot is thus able to achieve much higher (learned) reward than performing the proper reaching action. The KL divergence (Table (3cb between the distribution of observation-action pairs seen during reward learning and those seen during policy optimization provides further insights in incentivizing the Reacher robot to spin fast, it leads the RL optimization toward states that were not seen during reward learning (specifically, states where the robot is spinning very fast).
Appx. A.7 displays results for the other tasks. Similar to the Reacher task, we find that removing distractor features generally helps performance. Note that the reason removing non-causal distractor features does not appear beneficial for Feeding is that the main spurious correlation (discussed in Section 4 involves one of the causal features-namer the difference between spoon position and target position. Thus, removing purely non-causal features fails to address this issue in Feeding.
5.2 MODEL CAPACITY
we study the effect of model capacity on the learned reward. We find that, despite careful tuning of hyperparameters with each model and dataset size, increasing the capacity of the reward model does not necessarily result in increase in subsequent policy performance.
In Appx. A.8
5.3 NOISE IN STATED PREFERENCES
Because user noise is often inevitable, we explore the effect of various types of user noise on reward misidentification. Following Lee et al. (2027), modify Eq. 1 to include a rationality constant and myopic discount factor E 1]
H 7t 1 at)) exp(8 ビ
P(TA <T)
(3)
2 az)) + exp( Ei= - aa²))
Using Eq. 3] we explore four types of user noise (varied independently of each other): STOCHAS- TIC, where the user is rational with = 1; MYOPIC, where earlier rewards are discounted with a 0.99; SKIP, where the user skips pair of trajectories if both have rewards below a certain threshold; and MISTAKES, where the preference label is randomly flipped with probability 0.1.
ORACLE has B
Results are displayed in Fig. 5
We find that noise in the stated preferences exacerbates reward misidentification test accuracy stays high while policy performance plunges (Fig. 5ah. Notably, two instances of user noise, Stochastic and Skip, have test accuracies greater than or equal to the Oracle (despite far worse performance). Importantly, the poor performance of the learned policies is not well predicted by the validation accuracies, with models achieving 0.995 resulting in worse alignment than models trained on less data with lower validation accuracy of 0.985. Additional results are located in Appx. A.9
We use EPIC (Gleave et al. 2020) as one way to measure the difference betweeen reward functions. Table 9 shows that when the coverage distribution for EPIC is chosen the distribution of state- action pairs seen during reward learning, the misidentified rewards due to user noise are closer to the ground truth rewards, in EPIC distance, than rewards learned without user noise. However, the op- posite is the case when the coverage distribution is chosen to be the distribution of state-action pairs seen during RL. This implies that the misidentified reward model "mimics" the ground truth reward on the reward learning distribution but fails to generalize when taken out of distribution by RL. We further observe in Tableeatha the KL divergence between reward learning and reinforcement learn- ing state distributions is greater when the learned reward contains noise (Stochastic). This indicates that the rewards trained with noisy data are misidentified: they have low test error but incentivize RL to deviate from the optimal behavior encountered during reward learning.
5.4 PARTIAL OBSERVABILITY OF CAUSAL FEATURES
Partial observability over aspects of the state on which the user's preferences are based is nearly inevitable in the real world. Interestingly, RL with the ground truth reward is able learn proper
7

---

Published as a conference paper at ICLR 2023
Crida
Stecharte
Myople
Task success
0.5
0.0
0.978
0.960
0.861
0.978 0.897 - test acc.
(a) Feeding, User Noise
Figure 5: Noise in stated preferences results in significantly degraded performance.
PERFORMANCE
OBSERVABILITY
REWARD
SUCCESS
PREF-FULL
126.120
0.973
PREF-PART
-120.825
0,040
GT-PART
128.933
0.990
Figure 6: Partial observability. Minimizing spoon-mouth distance without observability of food leads spilling food (left). PREF and GT refer policies optimized with the learned and true reward, respectively. FULL and PART refer to the amount of observability over causal features.
feeding behavior in spite of this lack of critical state information. However, as displayed in Fig. 6 reward learning is only successful when the model is able to observe all the causal reward features.
we analyze gradient saliency plots and find that partial observability causes the reward model to incorrectly over-weigh causal features that available and pick up spurious correlations with non-causal variables. With Feeding, we observe that the reward model learns stronger weights bearing on the true reward. Simultaneously. the reward also learns a greater weight on the second component of the feature corresponding to the vector distance between the end effector and the mouth. This results in the behavior depicted in Fig- robot manipulator is able to successfully maneuver the spoon close to the patient's mouth (by observing the distance feature), but does so without ensuring that the food particles themselves stay on the spoon and end up in the patient's s mouth.
In Appx. A.10
of the joint angles and two components of the action, all of which have
5.5 COMPLEX CAUSAL FEATURES
To explore how results may differ we increase the complexity of the task, evaluate reward misidentification on the complex task of Itch Scratching. We find that even after increasing the amount of training data for reward learning (Fig. 7ap. increasing the reward model capacity (Fig. 7b). removing distractor features that are not causally related to the ground truth reward, having per- feetly stated preferences, and ensuring full observability over all the features that are involved in the ground truth reward, performing preference-batt successfully scratches the itch location on the patient. However, as shown in Table [ standard model- free RL algorithm trained using the ground truth reward is able to solve such task.
reward learning still fails to produce a policy that
The task's scratching motion requires not only making contact with the target using the and-effector, but also that the target contact position be greater than a 8 away from the previous target contact position and that the exerted force be no more than a F Fraz. Although the preferences are based on how well this motion is performed and despite the reward model having access to all the necessary aforementioned information (including information about the state at the previous timestep; see Appx. A.4), we find that the reward model is not able to learn the scratching motion. As seen in Fig. 7e once we explicitly include high-level indicator feature, "scratched" (whether the robot has successfully performed the scratching motion), performance drastically increases. We suspect that the reward model's tendency to pick up on spurious correlations that occur consistently over the course of the trajectory involving just few variables prevents it from learning the true causal
8

---

Published as a conference paper at ICLR 2023
200
Reward
500
Reward
Reward
100
-50
-100
-100
1000
M
L.
128x64
256x256x256
M
Increasing dataset size
Increasing model capacity
With 'scratched" indicator
Figure 7: Complex causal features. Learning the "scratched" feature from low-level causal features difficult, despite increasing dataset size and model capacity. Performance significantly improves when given access to a high-level 'scratched' feature.
relation that involves many variables, each of which are causal only in a particular context. In Appx.[A.11] we analyze this further and find that the learned reward without explicit "scratched" feature leads to a greater amount of distribution shift. Future work should address the problem of learning this kind of complex, multi-feature causal relationship.
6 CONCLUSION
Our work provides an analysis reward misidentification and causal reward confusion. We identify three tasks where preference learning often results in misaligned learned reward models. Interest- ingly, these models have good validation and test accuracy-Sometimes sometimes even 99.5%-and seem to distinguish the basics of task success versus failure. However, optimizing these rewards via RL pushes the policy outside of the training distribution, where the model falsely believes the reward higher-we demonstrate this via saliency maps, EPIC distance, and KL divergence and by ex- amining the resulting behaviors. This out-of-distribution effect results in policies that achieve high learned rewards but have poor true rewards. We find that it is easy for reward models to pick up on non-causal features, as some issues go away when we eliminate these non-causal features from the input. Furthermore, noisy preference data aggravates poor generalization. And when not all causal features are observable, the learned reward model will struggle even though there exists high-pent gegh-performin policy that only uses observable state information.
Based on our results, we have identified several directions for future work. First, our results show that reward misidentification induces distribution shift such that the learned reward appears de- ceptively close to the true reward on the training distribution, despite leading to misaligned be- havior when optimized via RL. As such, future work should investigate methods for penalizing excess exploration beyond the training data distribution. In Appx. A.13 we examine the effect of adding a direct penalty on the KL divergence into cost function during RL. We find that RL agent is again able to hack the learned reward performance becomes even worse because of the additional degree of freedom in the reward function afforded by the discriminator penalty. How- ever, we hypothesize there are other ways successfully incorporate such penalty. Next, results demonstrate that spurious ("nuisance")) features can significantly increase the chance re- ward misidentification. Thus, in cases where we can query for human knowledge on exactly which features are spurious or causal, we can use this feedback to learn which features are non-causal or remove non-causal features entirely. Further, our results indicate that high-level features such the "scratch" feature are difficult for neural networks to learn, even when all the necessary low-level information available. Future work should examine incorporating methods for learning high-level features (Bobu et al. 2021). Using alternatives to EPIC (Gleave al.) 2020) such DARD (Wulfe) al. 2022) to compare learned rewards may also prove fruitful in detecting and alleviating reward misidentification. Finally, recognize that active and iterative methods for data acquisition is also a widely popular solution-as such, we provide a preliminary exploration in Appx. A.14
Our work cautions that reward learning is brittle- choices for acquiring data, deciding on the amount of data, or defining the input space can lead to models that seem very close to the true reward, but lead to spectacular failures when optimized. While active and iterative methods for data acquisition may alleviate some of these issues, learning reward models when some of the causal features unobservable remains open challenge.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like thank the InterACT lab for their insightful feedback and fruitful discus- sions over the course of this work. This work was supported in part by an ONR YIP award.
REFERENCES
Erdem Biyik and Dorsa Sadigh. Batch active refference-basee learning of reward functions. In Conference on robol learning, pp. 519-528. PMLR, 2018.
Andreea Bobu, Marius Wiggert, Claire Tomlin, and Anca D Dragan. Feature expansive reward learn- ing: Rethinking human input. In Proceedings of the 2021 ACM/IEEE International Conference Human-Robot Interaction, pp. 216-224, 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, pp. 783-792. PMLR, 2019.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning. PMLR, 2020a.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. senter-than-demonstall imitation learning via automatically-rankeed demonstrations. In Conference robot learning, pp. 330-359, PMLR, 2020b.
Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NIPS, 2017.
Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Ad- vances Neural Information Processing Systems, 32, 2019.
Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C Karen Liu, and Charles C Kemp. Assis- tive gym: A physics simulation framework for assistive robotics. In 2020 IEEE International Conference on Roboties and Automation (ICRA), pp. 10169-10176, IEFF, 2020.
Adam Gleave, Michael D Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differ- ences in reward functions. In International Conference on Learning Representations, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishck Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05903 2018.
Jerry Zhi- Yang He and Anca D Dragan. arXiv:2111.09884, 2021.
Assisted robust reward design.
arXiv preprint
Ferenc Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: 2017.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. arXiv preprint arXiv: I.06521, 2018.
Zaynah Javed, Daniel S Brown, Satvik Sharma, Jemry Zhu, Ashwin Balakrishna, Marek Petrik, Anca Dragan, and Ken Goldberg. Policy gradient bayesian robust optimization for imitation learning. In International Conference on Machine Learning, pp.
PMLR, 2021.
10

---

Published as a conference paper at ICLR 2023
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Ku- mar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 2020.
Cassidy Laidlaw and Anca Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. In International Conference on Learning Representations, 2021.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgenerallizatin in deep reinforcement learning. In Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp- 11004-12015 PMLR, 17-23 Jul 2022.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking weference-based reinforcement learning. arXiv preprint arXiv:2 11,2212100002 2021.
Yi Liu, Gaurav Datta, Ellen Novoseller, and Daniel Brown. Efficient preference bbaed reinforce- ment learning using learned dynamics models. In International Conference on Robolics and Automation (ICRA). IEFE, 2023.
Eric Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint arXiv:20/2.05862 2020.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application to reward shaping. In Ieml, volume 99, pp. 278-287, 1999.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint wr.ivv2201.03544 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong. Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralizaaion: Why correct specifications aren't enough for correct goals. arXiv preprint arXiv:2210.01790 2022.
Daniel Shin, Anca Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference- based reward learning. Transactions on Machine Learning Research, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Zhiwei Steven Wu. Causal imitation learning under temporally correlated noise. arXiv preprint 22Xiv:2202.01312, 2022.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv: 2018.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
Blake Wulfe, Logan Michael Ellis, Jean Mercat, Rowan Thomas McAllister, and Adrien Gaidon. Dynamics-aware comparison of learned reward functions. In International Conference on Leam- ing Representations, 2022. htt.:/ppp//peeeeeeeeeeeeeeerrrrp

---

Published as a conference paper at ICLR 2023
Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unobserved confounders. Advances in neural information processing systems, 33:12263-1222 2020.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1
PREFERENCE LEARNING TRAINING DETAILS
To learn reward function from preferences, assumme access to set of pairwise preference labels d over trajectories Th,-... TN, where (i.j) E P implies that т; < Tj. We then optimize a reward function rg 8 A IR parameterized by 0 that maximizes the likelihood:
exp(ra(Tz))
((0) = II exp(re(TT) exp(ro(T,))
(4)
(JJ)
This likelihood function is differentiable, allowing leverage non-linear function approximation to learn the reward function from trajectory preferences. In practice, we use the Adam optimizer in PyTorch learn the reward function, Ta, and then use PPO Schulman al. (2017) SAC Haarnoja et al. 2018) for policy optimization given re-
We perform preference learning on three dataset sizes (given in terms of unique pairwise compar- isons): SMALL (780), MEDIUM (7 (7140), and LARGE (52326). Our test set is composed of 1770 unique pairwise preferences drawn from disjoint set of 60 trajectories. See Appx. A.2 for dataset generation details. Hyperparameters- g rate and weight decay-are tuned coarsely using the MEDIUM dataset size due to runtime limits and cost of computation. The tuned hyperparame- ters (best performance on held-out validation set) for each environment are as follows: Reacher: weightdecay=000000 1=0.01, Feeding: weightdecay=000000) 1r=0.001, Itch Scratching: weight- decay=0.001, 1r=0.001.
Using the dataset of preferences (SMALL, MEDIUM, or LARGE) obtained offline, we train a neural network reward function approximator with two hidden layers (128 units and 64 units, respectively) and Leaky ReLU activations, after which we perform 1 1,000,000 timesteps of reinforcement learning with PPO Schulman al. (2017) (for Feeding and Itch Scratching) and SAC HHarnoja et al.) (2018) (for Reacher) using the learned reward function in place of the ground-truth reward function. We optimize the reward function approximator using Adam with weight decay and early-stopping on the validation loss (with patience of 10 epochs). The full preference-based reward learning procedure we use is detailed in Alg. The hyperparameters for the PPO and SAC agents are as follows (if not specified, then hyperparameters are set to RLLib's bbs default values):
PPO:
training batch size 19200
number of SGD iterations = 50
SGD minibatch size 128
lambda 0.95
dimensions of fenet hidden layers = [100, 100]
SAC:
timesteps_per. per_rteratttttonnnn = 400
learning_starts 1000
Q_mddel fenet hindenhsdeess [100, 100]
policy model fcnet hiddens = [100, 100]
train batch. size 4096
actor Jettonnearnngg_rtt = 3e-3
critic lriticcle 3e-3
entropy Jearningnnnnnnnne rate = 3e-3
13

---

Published as a conference paper at ICLR 2023
Algorithm Preference-Based Reward Learning
Input: Training dataset of pairwise preferences derain
initialize network (random)
for epoch (0, 100) do
for trajs, trajj: label E derain do
f 4- f(traj,)
f3 + f(trajj)
loss Cross Entropy label) loss. bossbbackward()
end for
loss + calculate cross-entropy loss on validation set
LosSpal doesn't 't decrease for 10 epochs then break end if end for
f + Run RL (PPO or SAC) using as reward for IM iterations
Output: F 5
A.2 SYNTHETIC PREFERENCE GENERATION
To enhance scalability and reproducibility, we automatically generate large amount of synthetic trajectory preferences. This was done using an expert RL policy trained using the ground-truth reward function provided with each of environment. We then generate large number of diverse trajectories by adding e-greedy noise during policy rollouts, where e is the probability that the policy takes an action uniformly at random from its action space. Thus, trained RL policy and = corresponds to a uniformly random policy. As noted by [Bronn et al.) (2020b), adding this type of disturbance will result in monotonically decreasing performance in expectation.
corresponds to the fully
To generate pairwise preferences over trajectories, we select all pairs trajectories from a set of 40, 120, and 324 total trajectories (for the SMALL, MEDIUM, and LARGE dataset sizes, respectively) generated with e-greedy rollouts for e E {0, 0.2, 0.4, 0.6, 0.8, 1}. We use held-out sets of trajec- tories for validation and testing. We then use the ground-trun truth reward functions provided by each environment to provide ground-truth preference labels.
A.3 EVIDENCE OF CAUSAL CONFUSION
Table 2 displays results when dataset size is varied (S, M, L). Table 3 displays additional results for the Half Cheetah environment (Brockman et al.) 2016), where notice that, unlike the other environments presented, results vary depending on dataset size. We observe the same pattern in the S data size of high reward model performance on held-out set, low subsequent policy (PREF) performance when the reward model is optimized, and the reward model's preference for the learned suboptimal policy rather than the expert policy (GT). However, the M and L dataset sizes, serve that the learned reward is more or less aligned with the true reward-in recognizes that the GT policy better (L) or about the same (M). Thus, the poor performance of the policy in HalfChee- tah (I.) is more due to a failure in RI, and one could reasonably expect that running RL for more carefully-tuner hyperparameters and iterations on this reward model would result in good perfor- mance. This agrees with the HalfCheetah results in Brown al. (2019). Nonetheless, that this is due to the simplicity the true HalfCheetah reward (with which preferences are gener- ated), which essentially only depends one feature: the x-velocity (which the reward model has explicit access to as one the features the observation).
ob-
maintain
Fig. provides plots of select features' spurious correlations with the true reward. Fig, shows gradient saliency maps of the learned rewards that elucidate how the learned reward may improperly weight feature.
14

---

Published as a conference paper at ICLR 2023
C000EC--455 113000
15
2.2891-- 2.0 032927
125-
0.1
Reward
Reward
103-
25-
-0.5
'vell'
diff
(a) Reacher, 'vel_ 1 (feature 6)
(b) Feeding, 'diff y (feature 8)
Fe DOB651
Reward
-2-
-CA
0.8
"diff
(c) Itch Scratching, 'diff x (feature 7)
Figure 8: Correlations of Spuriously Weighted Features with Reward.
(a) Reacher
(b) Feeding
(c) Scratching
Figure 9: Gradient Saliency Maps for Fig. 2 Fig. 9a shows the slight positive gradient for the angular velocity feature 'veL_', which results in spimning behavior. Fig. 9b shows the nearly constant positive gradient in the 'diff feature (feature 8) that incentivizes the robot to move into, past, and behind the human's head. A properly learned reward would exhibit goes behind the human's head.
gradient that 'flips" negative the moment the robot Fig. 9c shows significant positive gradients for features 7, 9, 35, and 36, which correspond to ddff", "dif f", andthelastuuodeeeeeeee http:///wwwweeeeeeee/eeedooonnon freedomintheT DOFrobotam, DO''
15

---

Published as a conference paper at ICLR 2023
Table 2: Empirical evidence of causal confusion (different dataset sizes). We compare policies optimized with reward learned from preferences (PREF) against policies optimized with the true reward (GT). State features on which preferences are based are fully-observable all three tasks. S, M. and L. correspond to training dataset sizes of 780. 7140, and 52326 unique pairwise preferences, respectively. Both PREF and GT are optimized with 1M RL iterations and averaged over seeds. Despite high pairwise classification accuracy, the policy performance achieved by PREF under the true reward is very low compared with GT, irrespective of data size. However, the reward learned from preferences consistently prefers PREF over GT. This, combined with the low success rates of PREF compared to GT, suggests that preference-basee bbased reward learning fails to learn a good reward, the amount of data increases, for all of these tasks. (Lunar Lander does not have predefined success metric, so we leave this column blank.)
PREF LEARNING ACC.
RL POLICY PERFORMANCE
LEARNED
TRUE
SUCCESS (PREF/GT)
DOMAIN
TRAIN
VAL
TEST
(PREF/GT)
(PREF/GT)
REACHER (S)
0.955
0.913
0.939
-1.097 -6.002
-13.331 -5.560
0.040 0.827
REACHER (M)
0.957
0.949
0.962
-12.002 / -14.936
-11.890 / -5.560
0.053 / 0.827
REACHER (1.)
0.954
0.956
0.966
44.988 3.395
-42.716 622.116-5.5660 -5.560
0.100 / 0.827
FEEDING (S)
0.976
0.902
0.891
90.671 / 13.206
1153.012 128.933
0.057 / 0.990
FEEDING (M)
0.979
0.968
0.960
106.415 / 68.835
-45.427 / 128.933
0.437 / 0.990
FEEDING (L)
0.987
0.976
0.976
277.152
124.016
-27.432
128.933
0.603 0.990
ITCHING (S)
0.974
0.908
0.869
18.757 / 10.337
-56.591 / 248.397
0.000 / 0.970
ITCHING (M)
0.967
0.924
0.918
17.871 12.685
-68.024 248.397
0.003 0.970
ITCHING (L)
0.954
0.933
0.928
16.588 / 10.282
-47.190 / 248.397
0.013 / 0.970
LUNAR (S)
0.967
0.920
0.921
12.730 1.548
-5496.430 178.617
LUNAR (M)
0.947
0.945
0.937
73.342 -4.511
-6272.095 / 178.617
LUNAR (L)
0.940
0.948
0.943
-1.375 / -6.307
-747.081 / 178.617
Table 3: Empirical evidence of causal confusion (addt'l.), We observe that the the learned reward (LEARNED) more or less aligned with the true reward (TRUE) in the M and L dataset sizes and only really misidentified in S.
PREF ACC.
RL POLICY PERFORMANCE
LEARNED
TRUE
SUCCESS (PREF/GT)
DOMAIN
TRAIN
VAL
(PREF/GT)
(PREF/CT)
HALFCHEETAH (S)
1.000
0.971
1608.499 1101.144
1626.549 3346.6618
0.397 0.8660
HALFCHEETAH (M)
0.999
0.993
684.118
677.197
2968.347 / /3446.618
0.647 0.860
HALFCHEETAH (1.)
0.999
0.997
190.574 815.975
1236.655 334466188
0.330 0.860
A.4 INPUT FEATURES FOR ENVIRONMENTS
Features that are distractors are labeled with a D. Features that are causal are labeled with a C. The dimensions of each feature included in parentheses. Lunar Lander:
C x-coordinates of the lander (1)
C g-coordinates of the lander (1)
C Linear velocity in z (1)
C Linear velocity in (1)
C Angle of the lander (1)
D Angular velocity of the lander (1)
C Whether each leg is in contact with ground (2)
C Action - one of (do nothing, fire left orientation engine, fire main engine, fire right orientation engine) (1)
16

---

Published as a conference paper at ICLR 2023
Reacher:
D Cos of angle of first and second arm (2)
D Sin of angle of first and second arm (2)
D Coordinates of target (2)
D Angular velocity of first and second arm (2)
C Position cuattion_innertip position_targ target (3)
C Action - Torque applied at first and second hinge (2)
Feeding:
D rpeon (3)
D Spoon orient real (4)
C spoon ror. real target-pos.rees real (3)
D Ronoteoo (7)
D) Head pos real (3)
D Head_.riennt (4)
C Spoon_force_on_hunge (1)
C Action (7)
C io frods_.i_moutte mouth (1)
C Foodo_on_floor (1)
C Foods hht.hhthuuuh human (1)
C Sum food mouth velocities (1)
C Prev_spoon real (3)
C force_force. on human (1)
Itch Scratching:
C Tool pos real (3)
D Tool_orientrraal (4)
C Tool_pos_real rarget_pos_real (3)
D Target-pps.reet (3)
D jongtajosnt (7)
D Shoulder_pos_real (3)
D Elbow_pos_ret real (3)
D Wrist.pos-real (3)
C force (1)
C Action (7)
C Prev tool pos real (3)
C Robot_force_on_hune (1)
C Prev tool force (1)
17

---

Published as a conference paper at ICLR 2023
A.5 GROUND TRUTH REWARD FUNCTIONS FOR ENVIRONMENTS
We outline the ground truth reward functions for each environment below. We also refer the reader the publicty available code repositories for each environment, which describe the rewards in more detail.
Lunar Lander (https://github) http:hhww...coo/oaaa/a/aaaa///
S(x) - 100 (position + position?))
100
(velocity + velocity))
+ - -100/angle
+ 10(conlache.:)) + 10(collac
R(x) (xx Spree (x)
R(x) = R(x) (.3(action)
Reacher http:///ggthab..cmmmpeem////////
R(x) =
||ation|
Idistance postarget - p08spoon 1/2
Faction
|actionl2
Fowesahhhoooooooo..o. Foods_om_floor)
Food hht_hhhmaann Sum Summ_food.
preferences.soee score 9 Fooods
reall preu_spoon.pos.ree
R(x)) W Wisstance Idiistanee
W Waction Taction
W food + preferences
Itch httpt//ttwp........oo///aaaaaata
ddistance
postarget po8spoon ||2
Taction |action|l2
Tscratch /T poo__rlll T rarget.pos.seal, Prev fool ros_r
Tool Tore_, freeJool_fore
preferences.scort = g(Spoon Seppoonposs_raoll. real, Prev ppee_sspon.pposeee
Robot-ffrcee.o www.ooooo.ooooor.... T fool-_orce, T pos_raal)
R(x:) Wdistance Iddistance + W action Taction + W scratch * Toood + pre ereecssencssscooee
A.6 EVALUATING LEARNED REWARD FUNCTIONS
Saliency maps are one of the few methods that allow one to interpret learned reward functions in isolated, relatively lightweight manner. Following Michaud et al. (2020), we saliencies, or Ø(s.0) -thh gradient with respect to each element the input. We extend upon this by examining gradient saliencies per timestep along with feature spread maps-maps of each input feature's variation (standard deviation, variance, range) over the course of a trajectory.
gradient
aR
We produce saliency maps of the learned reward as follows: forward propagate single rollout from the learned reward's policy through the reward network. Then, with the reward model's 's weights fixed, backpropagate the output from the forward pass through the network and into the input (the policy rollout) to obtain the gradient with respect to each feature in the abservation-actiin pair each timestep.
EPIC, or Equivalent-Police Invariant Comparison, is a pseudometric proposed by Gleave et al. (020) that quantifies the difference between reward functions given coverage distribution and proposes to predictive of policy performance without the need for policy optimization. We apply the EPIC pseudometric to compare the distance between various learned rewards and the
18

---

Published as a conference paper at ICLR 2023
Table 4: KL Divergence: Distractor Features.
ENV
WITH
WITHOUT
FEEDING
4.732
6.985
ITCH SCRATCHING
18.749
8.570
ground truth reward on the coverage distributions seen during reward learning and those seen during policy training. Using other metrics to compare learned rewards (such as DARD, by Wulfe et al. 2022 may also prove fruitful in future work.
Hellback-Leibber divergence: We use discriminator trained to minimize the cross-cntropy loss states from two different distributions following the approach proposed by Huszár (2017) and Laidlaw & Dragan (202T). Specifically, we approximate the KL. divergence between the distribution of state-action pairs seen during reward learning and those seen during RL on the learned reward. We use this to measure the amount of distribution shift from the reward learning distribution induced by optimizing (potentially misidentified) learned rewards during policy training.
Specifically, for each learned reward, we sample 50 trajectories from the reward's training data and from the resulting policy (taking care to label each trajectory's origin distribution). From these 50x2 trajectories, we create the training and validation splits and then flatten each group of trajectories into dataset of observation-acee 12x 128) to distinguish between observation-acteo action pairs seen during reward learning and those during RL by minimizing the binary cross-entropy loss. We tune hyperparameters (learning rate, weight decay) on the validation loss and accuracy.
pairs. We train a discriminator model (hidden dimensions of
With this trained discriminator model, we calculate (KI(p))4) by taking the discriminator's nega- tive mean return/logit of all reward learning observation-action pairs, where p is the reward learning distribution and q is the policy optimization distribution. Similarly, we calculate Dx (gllp) by tak- ing the mean return of all RL observation-action pairs. Since the KL divergence is not symmetric, we report D L (p) (g) + Dx (gllp) or the symmetric KL divergence. For a proof on why a dis- criminator can be used approximate the KL divergence, refer the reader to Appendix B.2 of Laidlaw & Dragan (2021)
A.7 DISTRACTOR FEATURES
Fig. 10 shows the additional results for removing distractors. Fig. shows the saliency map for Reacher learned reward without distractors. Table 4 shows additional KI. divergences for the Feeding and Itch Scratching environments. We note here that the reason removing non-causal dis- tractor features appears to not benefit for Feeding is that the main spurious correlation (discussed in the difference between spoon position and
Section 4 involves one of the causal target position. Thus, removing purely non-causal features fails to address this issue Feeding.
A.8 MODEL CAPACITY
We study the effect of model capacity on the learned reward. We find that, despite careful tuning of hyperparameters with each model and dataset size, increasing the capacity of the reward model does not necessarily result in an increase in subsequent policy performance. In fact, as seen in Fig. [2] only the Feeding task trained with the large (L) dataset size (52326 preferences) benefits from steadily increasing reward model capacity. Indeed, although increasing the capacity for the Reacher task appears to initially increase performance for the large dataset case (Fig. [2ab, per- formance drops back down to below the performance of the smallest model size when the model size further increased. Further, increasing model size seems to decrease the performance on the small datasets. This is not surprising, as benefiting from larger capacity tends to require increasing the amount data. As such, the validation-see accuracies here tend to agree with learned reward performance.
To examine these results further, we compare the learned rewards directly with the ground truth reward (without performing policy optimization) using the EPIC pseudometric in Table 5 Inter-
19

---

Published as a conference paper at ICLR 2023
1.00
With distractors
Without distractors
1.00
With distractors
Without distractors
0.75-
0.75
Success
Success
0.50-
0.50
0.25
0.25
0.00
0.00
M
M
(a) Reacher, Success
(b) Feeding, Success
distractors
distractors
200
100
Reward
Reward
100
0
-100
100
M
M
(c) Feeding, Reward
(d) Itch Scratching, Reward
Figure 10: Distractor Features. In general, removing distractor features that have no influence on preferences improves performance.
Table 5: EPIC Distances, Model Capacity. The reward model with larger capacity (hidden layer dimensions of 256x256x256) appears closer the ground truth than the reward model with smaller capacity (hidden layer dimensions of 128x64) on the distribution of states seen during reward leam- ing, but is much further from the ground truth on the distribution of states seen during reinforcement learning.
REWL/RR RI.
256x256x256
128x64
GROUND TRUTH
0.210 0.707
0.233 / 0.598
20

---

Published as a conference paper at ICLR 2023
Gradint per timestep
Fadure Rarqees
12
28-
a
(a) Gradients W/out distractors
(b) Feature Ranges
W/out distractors
Figure II: Saliency Maps: Without Distractor Features.
0
-0
Reward
Reward
-40
-60
-200
128x64
(256)3 (256)4
128x64
(256)
(256)
(a) Reacher
(b) Feeding
Model Capacity. Increasing model capacity increases performance when the large (L) dataset (324) 52326 preferences), certain extent. However, increasing capacity decreases performance on the small (S) dataset ((2) 780 preferences).
Figure 12:
estingly, the EPIC distances between the Iearned rewards and the ground truth reward vary widely depending on the choice of coverage distribution. With the distribution of states seen during re- ward learning (generated by randomly switching between expert and random policy), the larger (256x256x256) model appears closer to the ground truth reward than the smaller (128x64) one. However, these results are flipped when instead evaluate the pseudometric of states seen during policy optimization. This suggests that increasing the capacity of the reward model allows it
the distribution
closely 'mimic" the ground truth reward on the reward learning distribution (hence the lower EPIC distance) but worsens its ability to generalize to the reinforcement learning distribution. We see similar effect with the divergence metric in Table
A.9 NOISE IN STATED PREFERENCES
Fig. 1 shows additional results. Increasing the amount of preference data when noise is present appears to have a negative effect (as seen in Fig. 13b). despite the proportion of mislabeled data
Table KL Divergence Approximations, Model Capacity, Increasing model capacity may result in much larger distribution shift (as measured by the KL Divergence).
128x64
256x256x256
128x64
256x256x256
256x256x256x256
REACHER
64.165
57.724
46.031
18.187 4.732
88.927 9.909
FEEDING
25.352
32.836
16.900
21

---

Published as a conference paper at ICLR 2023
Oracle
Stochastic
Oracle
Stochastic
1.0
Task success
Reward
-2
0.5
-4
0.0
M
M
(a) Reacher, Data Size
(b) Reacher, Data Size
Stochastic
1.0
Oracle
Stochastic
100
Task success
Reward
50
0.5-
0-
0.0
With
Without
With
Without
Feeding, Distractors
(d) Feeding, Distractors
Figure 13: Noise in Stated Preferences.
Table 7: Percentage (%) of Mislabeled Data Provided by a Stochastic User. Numbers here correspond Fig. 13b
MEDIUM
LARGE
STOCHASTIC
2.726
2.870
remaining constant across dataset sizes (Table 7 Further, there compounding effect with dis- tractor features, where both noise and distractors result in a large loss in performance (Fig. l3dh.
To examine these results further, we compare the learned rewards directly with the ground truth reward (without performing policy optimization) with the EPIC pseudometric in Table Interest- ingly, the EPIC distances between the learned rewards and the ground truth reward vary depending on the choice of coverage distribution. With the distribution of observation-acto action pairs seen during reward learning (generated by randomly switching between an expert and random policy), the re- ward trained with labeling errors appears to be closer (in EPIC distance) to the ground truth than the reward trained without labeling errors. However, these results are flipped when we instead evaluate the distribution of observation-actio
the pseudometric This suggests that the misidentified reward model more closely 'mimics' the ground truth reward on the reward learning distribution (hence the lower EPIC distance) but fails to generalize to the reinforcement learning distribution.
pairs
during policy optimization.
Table 8 shows the KI. divergences for each model configuration in Fig.
22

---

Published as a conference paper at ICLR 2023
Table 8: KL Divergence Approximations, Noise in Stated Preferences.
ORACLE
STOCHASTIC
REACHER, MEDIUM DATASET
6.762
8.856
REACHER, LARGE DATASET
7.365
6.739
FEEDING, WITH DISTRACTORS FEEDING, WITHOUT DISTRACTORS
4.732
13.188
6.985
8.046
Table 9: EPIC Distances, Noise in Stated Preferences. Misidentified rewards (Stochastic) are closer to the ground truth reward the reward learning distribution (RewL) but further from it the RL distribution (RL) when compared to the properly learned rewards (Oracle). EPIC distances computed the rewards learned with distractors in the Feeding environment in Fig. 13d and averaged over three seeds.
REWL/ RL
GROUND TRUTH
STOCHASTIC
ORACLE
GROUND TRUTH
0.000 / 0.000
0.136 / 0.164
0.165 0.130
STOCHASTIC
0.136 / 0.164
0.000 / 0.000
0.220 0.209
ORACLE
0.165 / 0.130
0.220 0.209
0.000 0.000
A.10 PARTIAL OBSERVABILITY OF CAUSAL FEATURES
Fig. 14 displays gradient maps of reward that partially observes the causal features and a reward that fully observes the causal features. Table T0 displays the EPIC distances. Unlike with previous factors, EPIC distance the reward learning distribution is predictive of eventual policy perfor- mance. The KL divergences (Table again reveal that the misidentified reward (this time, due to lack of information in the observation) leads greater distribution shift during RL training.
We see that without having access to all the causal reward features, an incorrect reward model is learned due over-weighting the causal features that are available and also due spurious corre- lations with non-causal variables. With Feeding, we observe both cases: the reward model learns stronger weights on one of the joint angles and two components of the action ('act'), all of which have no bearing on the true reward. Simultaneously, the reward also learns greater weight on the second component of the feature corresponding to the vector distance between the end effector and the mouth ('diff"), which is causal in the ground truth reward. (Note that similar weighting also occurs in the learned reward that has access to all features of the true reward.) This results in the behavior depicted in Fig. 6-the robot manipulator is able to successfully maneuver the spoon close to the patient's 's mouth (by observing the distance feature), but does so without ensuring that the food particles themselves stay the spoon and end up in the patient's mouth (since the reward is unable to observe the number of food particles).
A.11 COMPLEX CAUSAL FEATURES
In the Itch Scratching task rewards the robot needs to perform a 'scratching" motion, which entails only making contact with the target using the end-effector (being within certain radius of the target coordinates), but also requires that the target contact position be greater than certain ó away from the previous target contact position and that the exerted force be no more than a specified
Table 10: EPIC Distances, Partial Observability.
REWL/ / RL
GROUND TRUTH
PARTIAL
FULL
GROUND TRUTH
0.000 0.000
0.684 0.699
0.165 0.309
PARTIAL
0.684 / 0.699
0.000 / 0.000
0.685 / 0.697
FULL
0.165 0.309
0.685 0.697
0.000 / 0.000
23

---

Published as a conference paper at ICLR 2023
timecap
1009
004
000
175
die pirt
pelt pegles
J0ly
oriees
crieee
beald
Isturt
(a) Gradients PART
Gradient per timestes
122
coven alle
der
cot Wast
colend
tooll minicl vetuction
de Borrin
pres goe
perre Amort
(b) Gradients FULL
Pouer Lualel
0000
(c) Feature Standard Deviations PART
(d) Feature Standard Deviations -FUL
Figure 14: Saliency Maps, Partial Observability.
24

---

Published as a conference paper at ICLR 2023
Table 11: KL divergence: partial observability. The reward with partial observability on causal features induces greater distribution shift than the reward with full observability.
FULL
PARTIAL
FEEDING
4.732
45.731
Table 12: KL Divergence: Complex Causal Features. Learning a reward without the 'scratched' indicator feature leads greater amount of distribution shift (KL divergence) during policy opti- mization.
WITH 'SCRATCHED' WITHOUT 'SCRATCHED"
8.570
15.553
Fmaa- Although the preferences are based on how well the robot performs this scratching motion and despite the reward model having access to all the necessary low-level information to infer this scratching motion (including information about the state at the previous timestep; Appx. A.4 for details), we find that the reward model not able to learn the scratching motion. As seen in Fig. [7c] once explicitly include high-level indicator feature, 'scratched', denoting whether the robot has successfully performed the aforementioned scratching motion, performance drastically increases. We suspect that the reward model's tendency pick up spurious correlations that occur consistently over the course of the trajectory involving just few variables prevents it from learning the true causal relation that involves many variables, each of which are causal only particular context, This is supported by Fig. 9c which shows gradients staying relatively constant for each feature rather than varying with time and context. As result, the learned reward without an explicit 'scratched' feature leads to a greater amount of distribution shift. This is shown in Table T2 which shows the KL divergences with and without the -nnd-spedified complex 'scratch" feature.
A.12 DATA GENERATION METHODS
Fig. 15 displays comparison of our method of generating diverse dataset of trajectory preferences and the method of using a checkpointed RL policy (proposed by (Brown et al. 2019)).
A.13 PENALIZING THE KL DIVERGENCE
Inspired by the results showing that misidentified rewards result in greater distribution shifts during policy optimization, we attempt to address this by adding a KL divergence penalty term to the reward. Specifically, pretrained discriminator (trained to discriminate between state-action pairs from the reward learning distribution and the RL distribution) to estimate the KL divergence, as detailed earlier in Section and Appx. AA6 This KL divergence is scaled by hyperparameter À and subtracted from the reward learned from preferences.
addition to using the reward learned from preferences, we incorporate
We show results for 0, 0.01, 0.1, 10 Fig. 16 Using discriminator that is pretrained offline (the discriminator isn't updated during the RL. process) to penalize the KL divergence does not appear to help the performance or alleviate reward misidentification in any way. It appears that the RL agent is again able to hack the reward-this time, the performance and reward misidentification are even worse because of the additional degree of freedom in the reward function afforded by the discriminator penalty.
A. ITERATIVE PREFERENCE- BASED REWARD LEARNING FROM ONLINE DATA
Following Christiano et al. 2017). we verify the occurrence of reward misidentification when learn- ing from data that is actively queried online (during the learning process). As before, initial set of offline trajectory and preference data (acquired via Appx. AA22p to train a reward func- tion, which then optimize policy using RL. We then sample TO trajectorris from the most
the
25

---

Published as a conference paper at ICLR 2023
T-REX (synthetic checkpoints)
RL+noise
0.0
-2.5
Mean cumulative reward
-7.5
-10.0-
-12.5
-15.0
-17.5-
190 preferences
500 preferences
2000 preferences
Figure 15: TREX vs. RL.Noise. Our method of generating diverse trajectories for preference
learning performs on par with, if not better than, the method using rollouts taken from check- pointed RL policy, as proposed by (Brown et al. 2019). Displayed are cumulative trajectory rewards from the Reacher environment.
A-D
A-001
A-0.1
A-1.0
A-10.0
A=0
A=b00
A=10
A=100
0.10
25
Success
Reward
50
0.05
-5
0.00
(a) Reacher, Reward
(b) Reacher, Success
Figure 16: Penalizing the KL Divergence.
26

---

Published as a conference paper at ICLR 2023
1.00
0.4
Test accuracy
Test loss
0.3
0.96-
0.2
0.94
0.1
0.92
0.90
10
20
30
40
50
10
20
30
40
50
Iteration
Iteration
(a) Reacher, Test Loss
(b) Reacher, Test Accuracy
0.20
-45
0.15
True Reward
-50-
0.10
Task
555-
0.05
0.00
20
30
40
50
10
40
50
Iteration
Iteration
(c) Reacher, True Reward
(d) Reacher, Success
Learned Reward
40-
100-
120
Iteration
(e) Reacher, Learned Reward
Figure 17: Training the reward and policy together iteratively.
recently optimized policy and generate preferences for all possible pairs (1) between sampled and of- fline trajectories and (2) within the set of sampled trajectories, which we concatenate to our existing training dataset. Using the (now augmented) dataset, we fine-tune the reward function and policy for 10 epochs and 100 iterations, respectively. We repeat this process of taking rollouts from the most recent policy and fine-tuning the existing learned reward and policy for 50 iterations. Results are displayed in Fig- 17 We observe that reward misidentification appears to still be presentttest error very low, while resulting policy performance (measured by true reward and success) remains poor.
27


Reviews:
[{"review_id": "xL2Ot_hsHsQ", "reviewer": "Reviewer_Ek1w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focus on causal confusion in preference-based reward learning. The study was conducted by performing a series of sensitivity and ablation studies on three robot control benchmarks, where the preference-based reward learning methods fail to generalize to out-of-distribution states.  The study indicated some sources that exacerbate causal confusion. Moreover, a set of methods were identified to interpret causally confused rewards. ", "summary_of_the_review": "Overall, this work provides a novel study for causal confusion in preference-based reward learning. To make it stronger, I suggest the authors to provide formal definition of causal confusion in reward learning, and showcase on how to leverage the findings in this study to avoid the causal confusion (if it is true) in reward learning.  ", "strength_and_weaknesses": "## Strength\n\n+ This paper provides the first concrete and systematic study of causal confusion for preference-based reward learning on three robot control benchmarks. \n\n+ Several techniques including saliency maps, EPIC reward distance and Kullback-Leibler divergence were used for evaluating the learned reward functions. This study tried to provide evidence of causal confusion in preference-based reward learning, and identify several factors that lead to causaul confusion.\n\n\n## Weakness\n\n+ There is no formal definition of causal confusion in preference-based reward learning. How to distinguish between the cuasal confusion and other factors that result in similar performance in the benchmarks, for example, overfitting or distribution shift? \n\n+ EPIC distance might no be a proper method for evaluating the learned reward in distribution shift cases. A recent work [1] has identify that EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics. This is problematic for learned reward functions because it entails evaluating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards. \n\n+ It is not clear how to use the findings in this study to avoid causal confusion in reward learning. It would be more interesting if this work can provides some suggestions or algorithms to do so. \n\n[1] Wulfe, Blake, et al. \"Dynamics-Aware Comparison of Learned Reward Functions.\" In ICLR 2022.\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "la1LcdUrvi", "reviewer": "Reviewer_VP1L", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an analysis of reinforcement learning reward functions learned from preference labels over trajectory pairs.  In particular, the authors show that such learned reward functions are prone to pick up on spurious correlations in the features, to noise in the preference labels, and to distribution shifts when optimizing the reward model.  Furthermore, the authors demonstrate that is not simply an over-fitting problem as reward models can reliably predict the preferred trajectory within a pair of trajectories in validation and test data, but rather that the policy is unable to exploit the reward as the policy shifts from the training distribution.", "summary_of_the_review": "Preference-based reward learning is becoming increasing popular, especially for domains in which it is challenging to hand-define a reward function.  Understanding the impact and limitations of learned rewards is important, and the work in this paper provides insights towards that.\n\nI have some concern about the flow of the training — it’s not that what is done here is wrong, but rather there are better ways to train the policies and rewards together.  However, with the authors open sourcing their code, others can extend the work to consider how they specifically go about training.\n\n", "strength_and_weaknesses": "+The paper drives home the message that being able to predict preferences does not necessarily translate to quality task completion. \n+The experiments are thorough and consider various issues that could be the cause of the failure with learned reward functions.\n+The test environments offer a range of difficulty, and the findings are consistent across tasks.\n\n-The paper reads as though the policies and rewards are trained independently.  First train a policy using the ground-truth environment reward, sample behaviors and add noise to form the dataset for reward learning, optimize the learned reward function, then finally train a new policy on this reward.  However, as noted in Cristiano et al. (Section 2.2), it is important the the policy and the reward are learned together.\n-The number of pairs in the datasets is somewhat large.  For Table 1, the large dataset is >50k pairs.  For a real problem this is too large to imagine being labelled by people.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dVNfj6YDWz", "reviewer": "Reviewer_hap7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies reward hacking that happens when a policy is trained using a reward model that is learned from human preference data. In reward hacking, the predicted reward of the learned policy is high but the actual reward is low. The paper shows multiple lines of evidence that the observed reward hacking stems from spurious/non-causal correlations: Firstly, reward hacking disappears when non-causal features are removed from the input space. Secondly, the predicted reward is sensitive/salient to non-causal features that we know do not cause ground truth reward. Thirdly, after learning a policy, the state-action distribution differs from the training distribution, more than it does when non-causal features are not present. Thus, non-causal features exacerbate distribution shift, breaking spurious correlations even more. Then, the paper shows (a little bit of) evidence that causal confusion is exacerbated by noisy preferences, greater model scale, and unobservability of features that cause reward. ", "summary_of_the_review": "The paper addresses an important, interesting, and understudied problem by showing a variety of empirical evidence for spurious correlations and factors that exacerbate it in RL from preferences. To fully recommend acceptance it would need stronger evidence for its conclusions, or focus on conclusions that can be supported well. \n\nEDIT: my concerns are sufficiently addressed now that I score the paper as marginally above acceptance. ", "strength_and_weaknesses": "RL from preferences has increasingly high-impact applications and great promise to be scaled further. This paper addresses reward hacking which is a common and severe difficulty with RL from feedback. Therefore it is important to understand the causes of this reward hacking. This paper identifies and analyzes one cause: spurious features that correlate with reward during training, are therefore rewarded by the reward model, which leads to troublesome behavior when training a policy with this reward. The paper also identifies exacerbating factors. Showing empirical evidence for these is a useful contribution since the problem is important and understudied. \n\nHowever, the evidence for key findings is limited, which makes it hard to know how much the findings will generalize to other settings. Some claims also lack an explanation. I expand on this below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UtWvXaHQeY", "reviewer": "Reviewer_DXr6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work analyzes the occurrence of spurious correlations in learning reward functions from preferences in the commonly used Bradley-Terry model. The authors show that picking up such correlations can have a drastic effect on the performance of an RL agent that is trained using the learned reward. In their empirical analysis, the authors examine the effect of distractor features, noise, partial observability and model capacity. The paper quantifies how these factors affect the trained RL agent on the basis of several different metrics.", "summary_of_the_review": "Overall, I'm torn on this paper. On one hand, the authors analyze a relevant problem in a very interesting and relevant problem domain. The writing is clear and easy to understand and comes with detailed empirical analysis that follows a coherent structure.  \nOn the other hand, this feels like reading only the first half of the paper: After the problem has been clearly identified, a (partial) solution is missing. Of course that's not necessary for each paper, but without a solution method the limited novelty and in points not 100% clear presentation weigh more heavily.  \nFor me, the lacking novelty currently outweighs the intriguing setting and therefore I'm voting for weak reject. I'm looking forward to an improved version of the paper (even just addressing some of the weaknesses) because there's obvious potential here.\n", "strength_and_weaknesses": "**Strengths**\n\n- Assistive robotics is a very interesting and relevant problem domain.\n- The analyzed failure cases are of high practical relevancy.\n- The empirical analysis is easy to follow.\n- Videos on the companion website qualitatively illustrate the failure modes of trained RL agents very well.\n- I appreciated the authors making a concerted effort to convey their findings graphically.\n\n**Weaknesses**  \nMy main concerns regarding the paper are:\n\n- The paper has limited novelty. While the setting is new and interesting, the methods used by the authors to measure causal confusion are not novel, except for a method to approximate the KL divergence between two distributions of state-action pairs.\n- No proposed solution method. The paper reads like a very nice and thorough exposition of an important problem but then it \"abruptly\" ends. Once the authors introduced their method to measure the KL divergence between state-action distributions, I was immediately thinking of whether this could be used to penalize learned rewards and thereby encouraging agents to stay within the \"known\" region of the state-action space. Even negative results on \"straightforward\" (devil's in the details of course) remedies like this would be nice.\n\nMore minor weaknesses:\n\n- Noise evaluations could be more detailed. I agree with the authors that user noise is inevitable, but in my opinion the setting they evaluate ($\\epsilon$-greedy action noise) is rather uninteresting. \"*B-Pref: Benchmarking Preference-Based Reinforcement Learning*\" (Lee et al.) introduces several types of noise based on known cognitive biases. It would be interesting to see how such structured noise affects performance and I also think evaluating it would mesh very well with the practical relevancy of the author's work.\n- It would have been nice to have an explanation of the noise procedure in the main paper as I think it's relevant to understanding.\n- The authors evaluate against the ground-truth reward functions very often. It's frustrating that these are neither stated in the main paper nor in the appendix. \n- Hyperparameters for the SAC and PPO agents where missing. \n- Evaluation methods are not used consistently for all settings. There are no saliency maps and EPIC distances for itch scratching. They would help paint a more complete picture even though I presume they are omitted because the agent fails to learn in most settings anyway.\n- In A.4: \"*With the trained model, we calculate $D_{\\text{KL}} (p \\parallel q)$ by taking the negative mean return/logit of all\nreward learning observation-action pairs*\". I assume it's the mean return according to the reward model? Would still be nice to have it explicitly stated here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PeKH-VMBaA", "reviewer": "Reviewer_DzdV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies causal confusion in the context of reward learning from preferences. It is shown that rewards learning from preferences achieve poor performance in terms of learning a policy in three robot learning benchmarks. Several factors for causal confusion are identified, including distractor features and noise in stated preferences.", "summary_of_the_review": "This work considers an interesting idea. However, the narrowness of this scope of this study as well as the lack of full support for certain claims limits the contribution of this work. It seems that it is not yet ready for publication and a larger scope would improve considerably its contribution.", "strength_and_weaknesses": "The paper is well-written and the ideas are exposed clearly.\n\nHowever, the scope of this study is very limited. The experimental setup only includes continuous robot learning benchmarks, with three similar tasks, and is not extended to additional, potentially very different, environments. Besides, a specific model is used for the preferences. It might also be interested to compare the results here with alternative preference-based reinforcement learning approach, including additional value-based approaches and relation-based techniques (see for example the review: \"A Survey of Preference-Based Reinforcement Learning Methods\", Christian Wirth et al.). Therefore, it is not clear if the conclusions reached in this study may be extended beyond this narrow scope.\n\nAnother limitation of this work is the use of only synthetic preferences, computed from the real reward using a noise model. Would the same conclusions hold with real human raters preferences? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster