On the Sensitivity of Reward Inference to Misspecified Human Models
Authors: Joey Hong
Kush Bhatia
Anca Dragan
Venue: ICLR 2023 notable top 5% (2023)
Keywords: reward learning
inverse reinforcement learning
misspecification

Abstract: Inferring reward functions from human behavior is at the center of value alignment – aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.

Published as a conference paper at ICLR 2023
ON THE SENSITIVITY OF REWARD INFERENCE TO MISSPECIFIED HUMAN MODELS
Joey Hong
Kush Bhatia Stanford
Anca Dragan UC Berkeley
UC Berkeley
bhey_hong@berrkeeey..oo edu
ABSTRACT
Inferring reward functions from human behavior is at the center of value alignment aligning Al objectives with what we, humans, actually want. But doing so relies models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order the reward inference to be accurate? On the one hand, small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.
INTRODUCTION
The expanding interest in the area of reward learning stems from the concern that it is difficult (or even impossible) to specify what we actually want AI agents to optimize, when it comes to increasingly complex, real-world tasks (Ziebart et al., 2009; Muelling et al., 2017). At the core of reward learning is the idea that human behavior serves as evidence about the underlying desired objective. Research on inferring rewards typically uses noisy-rationality as model for human behavior: the human will take higher value actions with higher probability. It has enjoyed great success in variety of reward inference applications (Ziebart et al., 2008; Vasquez et al., 2014; Wulfmeier et al., 2015), but researchers have also started to come up against its limitations (Reddy et al., 2018). This is not surprising, given decades of research in behavioral economics that has identified a deluge of systematic biases people have when making decisions on how to act, like myopia/hyperbolic discounting (Grüne- Yanoff, 2015), optimism bias (Sharot al., 2007), prospect theory (Kahneman & Tversky, 2013), and many more (Thompson, 1999; Do et al., 2008). Hence, the noisy-rationality model has become complication many reward learning tasks AI researchers are interested in. For instance, in shared autonomy (Javdani et al., 2015), human operating robotic arm may behave suboptimally due to being unfamiliar with the control interface or the robot's dynamics, leading to the robot inferring the wrong goal (Reddy et al., 2014; Chan et al., 2021).
Recent work in reward learning attempts to go beyond noisy rationality and consider more accurate models of human behavior, by for instance looking at biases as variations on the Bellman update (Chan et al., 2021), )), modeling the human's 's false beliefs (Reddy et al., 2018), or learning their suboptimal perception process (Reddy et al., 2020). And while we might be getting closer, we will realistically never have a perfect model of human behavior. This raises an obvious question: Does the human model need to be perfect in order for reward inference to be successful? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, especially as it applies to value alignment: will never have perfect models, and will therefore never have guarantees that the agent does not do something catastrophically bad with
Work conducted student UC Berkeley

---

Published as a conference paper at ICLR 2023
respect what people actually value. On the other hand, if we can show that models improve, we have a guarantee that reward accuracy also improves, then there is hope: though modeling human behavior is difficult, we know that improving such models will make Al agents more aligned with us.
The main goal of this work is to study whether we can bound the error in inferred reward parameters by some function of the distance between the assumed and true human model, specifically the KL divergence between the two models. We study this question both theoretically and empirically. Our first result is a negative answer: we show that given finite dataset of demonstrations, it is possible to hypothesize a true human model, under which the dataset is most likely to be generated, that is "close" to the assumed model, but results in arbitrarily large error in the reward we would infer via maximum likelihood estimation (MLE). However, we argue that though this negative scenario can arise, it unlikely to occur in practice. This is because the result relies on an adversarial construction of the human model, and though the dataset is most likely, it is not necessarily representative of datasets sampled by the model. Given this, our main result is thus a reason for hope: we identify mild assumptions on the true human behavior, under which we can actually bound the error in inferred reward parameters linearly by the error of the human model. Thus, if these assumptions hold, refining the human model will monotonically improve the accuracy of the learned reward. We also show how this bound simplifies for particular biases like false internal dynamics or myopia.
Empirically, also show similar, optimistic message about reward learning, using both diagnostic gridworld domains (Fu et al., 2019b), as well as the Lunar Lander game, which involves continuous control over continuous state space. First, we verify that under various simulated biases, when the conditions on the human model are likely to be satisfied, small divergences in human models do not lead to large reward errors. Second, using real human demonstration data, we derive a natural human bias and demonstrate that the same finding holds even with real humans. Overall, our results suggest an optimistic perspective on the framework of reward learning, and that efforts in improving human models will further enhance the quality of the inferred rewards.
2 RELATED WORK
Inverse reinforcement learning (IRL) aims to use expert demonstrations, often from a human, to infer a reward function (Ng & Russel, 2000; Ziebart et al., 2008). Maxinum-entropp (MaxEnt) IRL is popular IRL framework that models the demonstrator as noisily optimal, maximizing reward while also randomising actions as much as possible (Ziebart et al., 2008; 2010). This is equivalent to modeling humans as Boltzmann rational. MaxEnt IRI. is preferred in practice over Bayesian IRL. (Ramachandran & Amir, 2007), which learns a posterior over reward functions rather than a point estimate, due to better scaling in high-dimensional environments (Wulfmeier et al., 2015). More recently, Guided Cost Learning (Finn et al., 2016) and Adversarial IRL (Fu et al., 2018) learn reward functions more robust to environment changes, but build off similar modeling assumptions as MaxEnt IRL. Gleave & Toyer (2022) connected MaxEnt IRL to maximum likelihood estimation (MLE), which is the framework that consider in this work. One of the challenges with IRL is that rewards are not always uniquely identified from expert demonstrations (Cao et al., 2021; Kim et al., 2021). Since identifiability is orthogonal to the main message of our work-sensitivity to misspecified human models-we assume that the dataset avoids this ambiguity.
Recent IRL algorithms attempt to account for possible irrationalities in the expert (Evans al., 2016; Reddy et al., 2018; Shah et al., 2019). Reddy et al. (2018; 2020) consider when experts behave according to an internal physical and belief dynamies, and show that explicitly learning these dynamics improves accuracy of the learned reward. Singh et al. (2018) account for human risk sensitivity when learning the reward. Shah et al. (2019) propose learning general biases using demonstrations across similar tasks, but conclude that doing so without prior knowledge is difficult. Finally, Chan et al. (2021) show that knowing the type of irrationality the expert exhibits can improve reward inference over even optimal expert. In this work, do not assume the bias can be uncovered, but rather analyze how sensitive reward inference is to such biases.
More generally, reward learning is a specific instantiation of an inverse problem, which is well-studied in existing literature. In the framework of Bayesian inverse problems, prior work has analyzed how misspecified likelihood models affect the accuracy of the inferred quantity when performing Bayesian inference. Owhadi et al. (2015) showed that two similar models can lead to completely opposite inference of the desired quantity. Meanwhile, Sprungk (2020) showed inference is stable under
2

---

Published as a conference paper at ICLR 2023
different measure of distance between models. In this work, we also derive both instability and stability results, but consider the different problem of reward learning using MLE.
PROBLEM SETUP
Reward parameters. We consider Markov decision processes (MDP), which are defined by tuple (,,,,) Here, S.A represent state and action spaces, P(s'|s. a) and r(s, a) represent the dynamics and reward function, and (0, 1) represents the discount factor. In this work, we interested in the setting where the reward function r is unknown and needs to be inferred by a learner. We assume rewards are bounded [r(s, a)l S Rmax. We assume that the reward can be parameterized by reward function parameters 0 E θ. We denote by r(::0) the reward function with & as parameters.
True vs. assumed human policy. Instead of having access to the reward, we observe the behavior of an "expert" demonstrator. Let πT" θ x S + ((A) be the and D {({t 0e) } be a dataset of demonstrations provided the learner, sampled from π We use (s, a) ~ w" to denote observations generated by policy n, where w" denotes the discounted stationary distribution. We shorthand w" as w". Finally, let स. θ x S - A(A) be the model that the learner assumes generated the dataset. In practice, स is often the Boltzmann rational policy (Ziebart et al., 2008), while x² is an irrational policy based on human biases
demonstrator policy,
Reward inference using the assumed policy. Many popular algorithms inverse reinforcement learning (IRL) (Ziebart et al., 2008; 2010) infer the reward function parameters via maximum- likelihood estimation (MLE). This is because unlike Bayesian IRL methods that learn a posterior over rewards, such MLE methods are shown to scale to high- digeeniienii environments (Wulfmeier et al., 2015). Using a dataset DD the learner would estimate parameters
1 Σ
i arg min
log ((x | 8;; 0) == arg min L),;
Let 8 be the true reward function parameters. Though 0 cannot always be uniquely determined (Cao et al., 2021; Kim et al., 2021), for simplicity of analysis, assume that 6* is identifiable: Assumption 1. There exists unique 0 satisfying 0 = arg ming L(0; n²' D).
Though Assumption I is rather strong, we make it only because we view identifiability as orthogonal to the subject of our work sensitivity to misspecified models.
Goal: effect of error in the model on the error in the inferred reward. The goal of our paper to answer whether we can bound the distance between the inferred reward and the true reward, do 0), as a function of the distance between the assumed human model and the true human policy, dz (T" ) for some useful notions of distance. If so, then we know that more accurate policies will monotonically improve the fidelity of the learned rewards. We discuss our choice of distances below.
Reward inference error. The inferred reward is typically used to optimize a policy in a test environment, and ultimately evaluated using the policy's performance in the new environment (Ng Russel, 2000; Ziebart et al., 2008). However, the reason we infer reward (instead of simply cloning the human policy) is because we do not necessarily know the test environment having the reward means we can optimize it in environments with different dynamics or initial state distributions. And unfortunately, adversarial environments exist whose dynamics amplify small disagreements between the learned and true reward functions. Hence, without prior knowledge about the test environment, we cannot derive any bound on a performance- based distance metric. We thus focus on analysis directly on a distance between the rewards themselves, via their parameters, do(0", 0) = 1 0 l|2 We choose squared distance as a natural and general distance metrie, but admit that rewards may not be smooth in their parameters. As future work, we can improve our work using more robust measures of reward similarity (Gleave et al., 2021).
Human model error. Since policies are probability distributions, we can measure error in the human model as the KL-divergence between the model and demonstrator policies. We consider two different instatiations of policy divergence. The first is a worst-case policy divergence that takes the supremum over all reward parameters and states:
TuE ( 7)- sup sup (TL.(TT(: 8; ()( 8; 0))
(2)
0 SES
3

---

Published as a conference paper at ICLR 2023
We use the forward direction of KL-divergence because it contains expectation over the demon- strator policy. which aligns with the sampling distribution of the dataset. Also note that this direction implies that the human model must cover actions of the true human to have small divergence. Finally, note that a small model error entails having small error across all states. This is thus a strong metric for a lower bound. It is not, however, strong metric for upper bound: when used in upper bound, it would allow the model to be wrong on states and rewards without paying any more penalty than model that is only wrong on one state and reward. We thus also consider an average error that makes for stronger upper bound - it considers the model only on the true reward parameters 0* and takes an expectation over states. We term this the weighted policy divergence:
(M = = [PKL(T s;; 0*) 7 s; s)("))]
(3)
The weighted policy divergence only looks at the states visited under the true human behavioral policy πt as compared to the worst case divergence, which compares against all states and rewards.
4 THE BAD NEWS: INSTABILITY RESULTS
We begin our theoretical analysis with negative result. We prove that even under the worst-case policy divergence from eq. (2), small difference between the model and the true human policy d= #) < can lead to a large inference error d0"0 0)). Since a trivial way for this to happen is to get "unlucky" with the data set D drawn from 'τ, and contain actions that are unlikely even n² = T,, strengthen the result by excluding tail events and imposing the strong requirement that D contains the most likely actions under the demonstrator policy and the true reward:
Definition 1. A policy π "likely generates" 'D if for every (st,a,) most likely one under the true reward, i.e., a; = arg max i (a: 14; 0*).
az) E DD the observed action is the
This means that actions that appear in DD are always at the modes of the policy .-. Our theorem shows that despite the two policies being close on each state and reward pair, under the stronger notion of worst-case policy divergence, the inference procedure can lead to large errors.
Theorem 2. For any MDP M with continuous actions, policy error O, assumed model म, and dataset D, there exists a demonstrator policy π that likely generates 'D such that the worst-case policy divergence satisfies d (n"", π) < E, but the reward inference error satisfies
0 A- 11
sup - |0--0
2 0
The theorem shows that for any continuous-action MDP, and any observed dataset D, even a small perturbation in the assumed human model can lead large inference error. We are able to prove such a strong result by perturbing the policy x² on only the observed maate-action pairs in the dataset. We defer the proof of the theorem to Appendix C, and provide an illustrative example in Appendix A. Though the theorem presents pessimistic worse-case scenario, there are aspects of the proof of that make it impractical and potentially unlikely to occur in practice. First, the construction of T² is adversarial, by biasing the demonstrator at exactly the actions in the dataset. Second, though Definition means the dataset is most likely to be generated by the demonstrator, it may contain actions that are not representative of what they would take. Since actions continuous, this means that actions that are similar to the observed one (via some distance metric) are in fact very unlikely.
THE GOOD NEWS: A STABILITY RESULT
Theorem 2 paints a pessimistic picture on the feasibility of reward inference from human demon- strations, but as discussed towards the end of Section 4, the mechanisms used to derive Theorem may be impractical and unlikely to occur. In this section, we show that under reasonable assumptions on the true policy we can indeed obtain positive stability result wherein we can upper bound the reward inference error by a linear function of the weighted policy error. We identify the following assumption that enables such result:
Assumption 2. parameters E O. Formally, there exists constant e. > such that for any s E S, a E A, '²" satisfies
The true and model policies π², . are strongly log-concave with respect to reward
log x" (a) s;)') I log x (a s; 0) + V logo nT" (a s: 8) T (0- 00) _ 110 011 and analogously for न..
c
4

---

Published as a conference paper at ICLR 2023
The adversarial construction of demonstrator policy x² in deriving Theorem 2 violate the above log-concavity assumption as they involve drastic perturbations of the probabilities of just the observed actions, creating policies that are not smooth in their parameters.
Intuition. We know that cog.cvicyv is violated by unnatural, adversarial constructions, but, we aim answer: does log-concavity always hold outside of such contrived examples? Intuitively, notice that log-concavity holds only if, as the reward parameter increases, an action that has become less preferred cannot become more preferred in the future. This appears to be natural property of many policies. For example, if someone already prefers chocolate over vanilla, increasing the reward of chocolate will never cause the person to prefer vanilla over chocolate. However, it turns out there are simple problems where this is violated. In Appendix B, we present simple navigation example violated. Though counter examples exist, we show our experiments that many natural biases still result in human models that satisfy Assumption 2.
where Assumption
Under Assumption 2, we can show that the reward inference error can be bounded linearly by weighted policy divergence. We state the formal result below, and defer its proof to Appendix C. Theorem 3. Under Assumption with parameter c > O. for any policies π², ग with corresponding MLE reward parameters 0, 8", the reward inference error de A00
(0"000000 00) is bounded as
ED [ 12] Eyww [DeL (- 0*) [[ s;; s;0*))]
Theorem 3 differs from Theorem 2 in two important ways: (1) the reward inference error is in expectation over sampled datasets, and (2) the policy divergence is the weighted policy divergence. Both these properties are desirable, as we are agnostic to tail events due to randomness in dataset sampling, and as discussed Section 3, we use the smaller of the two notions of divergence in the upper bound.
5.1 INSTANTIATING THE UPPER BOUND FOR SPECIFIC BIASES
Theorem shows that the reward inference error can be bounded by the weighted policy divergence between the assumed and true policies. To understand the result in more detail, we now consider different systematic biases that could appear in human behavior, and show how they affect the weighted policy divergence and thus the upper bound.
Without loss of generality, we parameterize both the true and assumed policies as acting noisily optimal with respect to their own "Q-functions", i.e., T"(a s; s; x exp((" (s, a; 0)) and i(a s; f) x cxp(Q(s, a; f)). Importantly, note that even though this parameterization is used in MaxEnt IRL with the soft Q-values (Ziebart et al., 2008; 2010), neither Q* nor Q need necessarily be optimal in this analysis, we will use Q. the human model, as the soft Q-value function, and show what happens when the true model coming from Q* suffers from certain biases. Following prior work (Reddy et al., 2018; Chan et al., 2021). examine biases that can be modelled as deviations from the Bellman update. For a tabular MDP M with |S|, |A|| < co, the soft Bellman update satisfies:
: + E I (s; V(s: 0) == log Σ eep((Q(s
(4)
AEEAA
Formally, we study examples under which the human demonstrator's Q-values Q* (s, a; 0) satisfy (4) but under a biased MDP M We consider two specific sources of bias in the MDP: (1) the transition model P and (2) the discounting factor 7. By parameterizing the biases in this way, we now have an intuitive notion of the degree of bias, and can study how the magnitude of the bias affects the policy divergence in (3). For brevity, we simply state the results as corollaries and defer proofs to Appendix C.
Internal dynamics. We first consider irrationalities that result from human demonstrators having an internal dynamics model pa that is misspecified. For example, studies in cognitive science have shown that humans tend to underestimate the effects of inertia in projectile motion (Caramazza et al., 1981). Similar studies have also shown that humans overestimate their control over randomness in the environment (Thompson, 1999), dubbed illusion of control. The latter irrationality can be formalized in our parameterization by assuming that (t((1 8, x (P(: 8, a))", where oc, the human
5

---

Published as a conference paper at ICLR 2023
will believe the dynamics of the MDP are increasingly deterministic. In Corollary 4, we show that the policy distance can be bounded linearly by the bias transition dynamics:
Corollary 4. Let Ap su || 8.a) P( 8, a)ll Also, let x² 7 be the policies that
result from from value iteration using (4) with dynamics models P P., respectively. Then, their weighted policy divergence bounded
2| Rmax 80 8;01)
Band (Tx |s: (s;0))
(1 (1-7)2 Ap.
Myopia Bias. The other irrationality we study is when humans overvalue near-term rewards, dubbed myopia (Grüne- 2015). Such bias can be captured in our parameterization through biased discount factor y", where 7 - 0, the human will act more greedily and prioritize immediate reward. In Corollary 5, we bound the distance between policies by the absolute difference in their internal discount factor.
Corollary 5. Let α², I be the policies that result from value iteration using (4) with discount factors y", 3, respectively. Then, their weighted policy divergence bounded as
2) R Rmmx
Egwa' [Da((T' (-1 s;8") ( (-11 s; - - (1 7--7-1
The above result shows that the degree of bias linearly upper-bounds the weighted policy divergence and hence, from Theorem 3, the expected reward inference error.
EMPIRICAL ANALYSIS
Our theoretical results predict that the worst-case, reward inference error can be arbitrarily bad relative to the human model error, but average, under some assumptions, the reward error should be small. Our empirical results aim to complement our theory by answering the following question under natural biases in human models in various environments: do find stable relationship between policy divergence and reward error?
We tackle this in three ways: (I) simulating the specific biases analyzed in Section 5.1, (2) simulating a non-Bellman-update structured kind of bias (a demonstrator that is still learning about the environment), and (3) collecting real human data. We consider both tabular navigation tasks on gridworld, as well as more challenging continuous control tasks on the Lunar Lander game (Brockman et al., 2016b).
Experiment design. Each experiment has bias we study and an environment (gridworld or LunarLander). When considering simulated biases, we manipulate x² by manipulating the magnitude of the bias starting at x' т the Boltzmann optimal policy. This helps us simulate different hypothetical humans, and see what degree of deviation from optimality ends up negatively impacting reward inference. When modeling bias with real human data, we instead fix x the real human policy, and manipulate ग by interpolating between the Boltzmann optimal policy and the real human policy this emulates practical process where human models get increasingly more accurate.
6.1 TABULAR EXPERIMENTS WITH STRUCTURED BIASES
First, we consider tabular navigation in gridworld domains (Fu al., 2019a), where the task the reach the goal state and earn reward of 0 > 1, which is not known to the agent, while avoiding getting trapped lava states. To further complicate the task, the agent can also get stuck at "waypoint" states that yield a reward of 1. Depending on the environment, it can be better for the agent stop at the waypoint state, circumvent taking the longer, more treacherous path to the goal state. The agent is able to move in either of the four directions, still. To introduce stochasticity in the transition dynamics, there is 30% chance that the agent travels in different direction
Start
Lava
Waypolnt (r
(r
choose to stay
Figure 3: Gridworld environments.

---

Published as a conference paper at ICLR 2023
0.07
1.0
1.0
0.06
Deusee wnherrcce emoo
0.4
0.4
0.22
0.11
0.00
0.05
0.10
0.15
0.70
0.25
0.
0.00
0.00
0.10
0.35
0.20
0.25
10..110.00 0.51 0:07 0:03 0.00 6.05 0.05 0.0 0.08 -0
Transitio eroor
Transiion arror
Weigheed palicy divergance
Figure 1: Effect of transition error (measured as the degree of underestimation of unintended transitions) (a) weighted policy divergence and (b) reward inference emorr three Gridworld environments (A,B,C). In (c). show a seatter plot of the policy and reward errors for different biased transition model. Note that small policy divergence results in small reward inference error.
LON0 entrrnnce
Rouar etrence
28-
0.15
805
1.34
Docoun 61101
DhusAun 21151
Weggrool potly Ar1
Figure 2: Effect of discount error in the three Gridworld environments. Like Figure 1, we see strong correlation between policy divergence and reward inference error.
than commanded. We consider three different gridworlds (which we simply call environments B, and C) where we vary in the location of the waypoint state (shown in Figure 3).
In each environment, we want the learn the underlying reward parameter 0 from demonstrations; however, the model i is noisily optimal, whereas the demonstrator policy π is irrational by suffering from false internal dynamics, or myopia. We model these irrationalities by either modifying the transition matrix discount factor, respectively, in the soft Bellman update in Equation (4). Note that the stationary distribution w" for a policy т can be exactly computed; hence, instead of sampling w" to compute exact quantities (see Appendix D.I for technical details).
data from n²"
Internal dynamics. The first irrationality we consider is illusion control, where the demonstrator policy significantly underestimates the stochasticity in the environment. Such biased policies x* obtained via value iteration on a biased transition matrix P where the human wrongly believes the probability p of unintended transitions is smaller than the true value. As O, the demonstrator becomes more confident that they can reach the goal state, and will prefer reaching the goal over the waypoint state, even when the latter is much closer and safely reachable (see Appendix D. for visualizations of the biased policies). In Figure 1, we show the effect of the transition bias (error p) on both the weighted policy divergence, and the reward inference error. The sub-linear trend in Figure la agrees with Corollary 4. Figure lb and show sub-linear dependence of the reward inference error on the policy divergence, as predicted by Theorem 3. For environment A, the reward error goes up most quickly with the dynamics error, but so does the weighted policy divergence, making this divergence a better indicator of reward error than simply the dynamies error.
Myopia. The next irrationality we look at is myopia, where the demonstrator policy assumes a biased discounting factor 7" that underestimates the true one. As 7" - O, the biased agent will much more strongly prefer the closer waypoint state over the goal state. In Figure 2, see analogous results to the internal dynamics bias. Namely, Figure 2a agrees with Corollary 5, and Figure 2b and e shows a sub- lub-linear correlation between policy and reward error, predicted by Theorem 3.
6.2 CONTINUOUS CONTROL EXPERIMENTS
Next, we consider more challenging domain of navigation with continuous states and actions. The exact navigation environment is modification the Lunar Lander game with continuous actions (Brockman al., 2016a), where the agent receives a reward for landing safely on the landing pad. The agent is able to take a continuous action in [-1, 1]2 that encodes the directions it wants to move (left, right, up) via its sign, as well as how much power wants to use in each direction via its cardinality.
7

---

Published as a conference paper at ICLR 2023
Wrogtod gotey dhrrggos
2.5
0.20
Reaard sreeecceeen eree
inteerce
0.35
2.5
3.5
0.30
2.0
0.5
0.05
0.0
0.0
0.5 Tronsition crror
0.4
0.8
1.0
0.00
1.0
00:4
0.6
1.0
0.25
8.5
0.0
Tharsttion
Weigheed disegence
Figure Effeect of transition error (measured error in p) in the continuous Lunar Lander environments. The
results are consistent with earlier Gridworld results.
Weggtod paltp diergence
0.20
2.0
Roword Roword Inforence
6.55
Braane intwereee eene
1.5
3.5
1.0
0.0
0.
0.9 6.8 DJ7 6.6
0.4
53
0.7
0.00
10
0.5
044
0.3
0.2
-0.5
0.25
8.15
020
Training terations 155 moxium)
Training korations 1% moxium))
Weighted policy divergnnce
Figure 5: Effect of the simulated human learning bias the continuous Lunar Lander environments.
In constrast to the classic version of the game where the landing pad is always in the middle, vary its location. The unknown reward parameter E (0, 1) is the location of the landing pad (as horizontal displacement normal- ized by the total width of the environment). We consider three different environments that differ in the location of the landing pad (see Figure 6). In each environment, the human model से is the near-optimal one obtained by soft actor-critic (Haarnoja et al., 2018). We provide details on the training procedure in Appendix D.2. In these ex- periments, we simulate the internal dynamics bias as well as new one based on the notion of demonstrator that is still themselves learning.
Figure 6: Lunar Lander environments.
Internal dynamics. We first study of the effect of demonstrator policies with biased dynamics models. We bias dynamics model by varying parameter p that describes how much one unit of power will increase acceleration in the corresponding direction. This is a plausibly natural human bias as people tend to underestimate the effects of inertia in projectile motion (Caramazza et al., 1981) For each false setting of p. we learn biased policy that is near-optimal for that p. As p increase, the biased policy tends to underestimate the amount of power required to move the lander enough to the right reach the landing pad (see Appendix D.2 for visualizations). In Figure 4, show the effect of the transition bias (error in p) on both the weighted policy divergence and the reward inference error. We that even in challenging continuous control domain, Theorem still holds.
Demonstrators that are learning. We next simulate a bias that might arise from humans that are learning how do the task (as would be the case, for instance, in Lunar Lander task). We do so by varying the amount of training iterations in learning the policy. The degree of such bias is captured in a parameter p E [0, 1], that denotes the number of training iterations, normalized by the amount used to learn the near-optimal believed policy. In Figure 5, we show the effect of on both the weighted policy divergence and the reward inference error. Reassuringly, we again notice sub-linear correlation in line with Theorem 3.
6.3 ANALYSIS OF REAL HUMAN POLICIES
The previous experiments have considered natural but sim- ulated biases to construct demonstrator policies modeling biased humans. However, it remains to be seen whether Theorem remains predictive even when biases are from real humans. We consider the same Lunar Lander game in Section 6.2 but discretize the action space so that people intuitively interact in it. Using this environment,
Figure 8: Visualization of trajectories under the human policy.
8

---

Published as a conference paper at ICLR 2023
2.5
0.20
2.0
Reward srleenceeee erer
inteerne
0.35
2.5
3.5
0.30
2.0
0.5
0.05
0.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.6
1.0
0.25
0.10
8.15
0.50
Probobilty of human trajectory
Probebility
Weighoed paley disegencee
Figure 7: Effect of modeling human hias (measured by prohability of acting according human policy) on (a) weighted policy divergence and (h) reward inference error on discrete Lunar Lander environments. In (c), we show a scatter plot of the policy and reward errors for different probabilities. We see that more accurate human models correspond to lower reward inference error.
create a demonstrator policy grounded in real human demonstrations. We do this by collecting trajectories from 10 human demonstrators, then learning a policy that imitates human behavior by running behavior cloning (BC) on the aggregated trajectories. We visualize trajectories from this policy in Figure 8. We observe that in general, humans tend to be unable to properly account for the effect of gravity, causing them to crash the lander before it has moved enough horizontally, particularly in environments B and C where the landing pad is horizontally displaced from the middle.
Then, we emulate a process through which the learner's model of the human, म. would evolve to align more and more with the true human policy. Specifically, we vary मे to interpolate between near- optimal and the true human policy, while keeping x fixed to the latter. To do so, we vary a parameter that controls the probability of sampling from the human policy (vs. the near-optimal one). In Figure 7, we show the effect of a on the weighted policy divergence and reward inference error, and conclude that larger result in smaller policy divergence well reward error. Importantly, this suggest that as the model gets closer and closer to the true human policy, the reward estimate gets increasingly better, with no sign that a small model error leads to terrible reward. In addition, we also match the simulation experiments by keeping से fixed as the optimal policy, and interpolate between the optimal policy and the real human policy for ' (see Appendix D.3). This gives us hope that even in real-world problems, better human models गे can translate to better reward inference.
DISCUSSION
Summary. In this paper, we conduct theoretical and empirical study of how sensitive reward learning from human demonstrations is to misspecification of the human model. First, we provide an ominous result that arbitrarily small divergences in the assumed human model can result in large reward inference error. While this is in theory possible, it requires rather adversarial construction that makes it unlikely to occur in practice. In light of this, we identify assumptions under which the reward error can actually be upper-bounded linearly by the model error. Experiments with multiple biases in different environments, as well as an analysis of the true human policy (which potentially suffers from unknown, yet to be characterized suboptimalities), reassuringly show remarkably consistent results: over and over again, we see that as the human model and the true human behavior are more and more aligned, the reward error decreases. Overall, our results convey the optimistic message that reward learning improves as we obtain better human models, and motivate further research into improved models.
Limitations and future work. Our upper-bound relies on Assumption 2 of log-concavity of the human policy. However, we hypothesize that weaker assumptions exist from which we can derive similar bounds as Theorem 3. In addition, via Assumption 1, we ignore ambiguity in reward identification. may important in the future consider reward error and identifiability jointly, potentially through equivalence classes of reward functions. Finally, as alluded to in Section 3, more robust measures of reward similarity exist that avoid needing to implicitly assume smoothness of the reward function in their parameters (Gleave et al., 2021). Interesting directions of further investigation include: (I) how to better model human biases, or orthogonally, (2) how to modify existing reward inference algorithms to be more robust to misspecification. A negative side-effect of our work could occur when we mistakenly rely on the upper- upper-bound in Theorem 3 when its conditions are not met, i.e. Assumption 2 does not hold, resulting in catastrophically bad inference without knowing it. More broadly, reward learning in general has the issue that it does not specify whose reward to learn, and how to combine different people's values.
9

---

Published as a conference paper at ICLR 2023
ACKNOW
We thank the members of the InterACT lab, and BATR as a whole, at UC Berkeley for their helpful discussions, as well as providing human demonstrations for experiments. We also thank anony- mous reviewers for feedback on an carly version of this paper. This research is funded in part by the Office of Naval Research Young Investigator Program, and UCSF Weill Institute for Neurosciences. Additionally, KB was supported in part by grant from Long-Tern Future Fund (LTFF).
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0154: 2016a.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016b.
Haoyang Cao, Samuel N. Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. CoRR, abb/2116.034990 2021.
Alfonso Caramazza, Michael McCloskey, and Bert Green. Naive beliefs in "sophisticated" subjects: Misconceptions about trajectories of objects. Cognition, 9, 1981.
Lawrence Chan, Andrew Critch, and Anca Dragan. Human irrationality: both bad and good for reward inference. preprint arXiv:21 11022111.0000 069956 2021.
Amy M Do, Alexander V Rupert, and George Wolford. Evaluations of pleasurable experiences: The peak-end rule. Psychonomic Bulletin & Review, 2008.
Owain Evans, Andreas Stuhlmueller, and Noah D. Goodman. Learning the preferences of ignorant, inconsistent agents. In AAAI Conference on Artificial intelligence, 2016.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML), 2016.
Fu, Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2018.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-laningng algorithms. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019a
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep Q-learning algorithms. arXiv preprint arXiv: 1902. 10250, 2019b.
Adam Gleave and Sam Toyer. A primer on maximum causal entropy inverse reinforcement learning. preprint arXiv:2203. 1409, 2022.
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. In International Conference on Leurning Representations (ICLR), 2021.
Till Grüne-Yannn Models of temporal discounting 1937-2000: An interdisciplinary exchange between economics and psychology. Science in context, 2015.
T. Haamoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In arXiv, 2018. URL httpttppss//xxx..oro....../ pdf/1801. 01290. pdf.
Javdani, S. Srinivasa, and J. A. Bagnell. Shared autonomy via hindsight optimization. In Robotics: Science and Systems (RSS), 2015.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Handbook of the fundamentals of financial decision making: Part 2013.
10

---

Published as a conference paper at ICLR 2023
Kumo Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning. 2021.
Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John E Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B Schwartz, Jennifer L Collinger, and J Andrew Bag- nell. Autonomy infused teleoperation with application to brain computer interface controlled manipulation. Autonomous Robots, 2017.
Andrew Y. Ng and Stuart Russel. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
Houman Owhadi, Clint Scovel, and Tim Sullivan. On the brittleness of bayesian inference. SIAM Review, 2015.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, 2007.
Siddharth Reddy, Anea Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. In Robotics: Science and Systems (RSS), 2014.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Where do you think you re going?: Inferring beliefs about dynamics from behavior. In Advances in Neural Information Processing Systems, 2018.
Siddharth Reddy, Sergey Levine, and Anca D. Dragan, Assisted perception: Optimizing observations to communicate state. CoRR, abs/2008.0288 2020.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca D. Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. International Conference on Machine Learning, 2019.
Tali Sharot, Alison M Riccardi, Candace M Raio, and Elizabeth A Phelps. Neural mechanisms mediating optimism bias. Nature, 2007.
Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar, and Marco Pavone. Risk-sensitive inverse reinforcement learning via semi- and con-parametric methods. The International Journal of Robotics Research, 2018.
Bjöm Sprungk. On the local lipschitz stability of bayesian inverse problems. arxiv preprint arxiv: /e0///99000000 2020.
Suzanne C. Thompson. Illusions of control: How we overestimate our personal influence. Current Directions in Psychological Science, 1999.
Dizan Vasquez, Billy Okal, and Kai O. Arras. Inverse reinforcement learning algorithms and features for robot navigation in crowds: An experimental comparison. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014.
M. Wulfmeier, P. Ondruska, and I. Posner. Maximum entropy deep inverse reinforcement learning. In Neural Information Processing Systems Conference, Deep Reinforcement Learning Workshop. 2015.
B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In International Conference Artificial Intelligence (AAAI), 2008
B. D. Ziebart, A. Bagnell, and A. K. Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning (ICML), 2010.
Brian D Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J Andrew Bagnell, Martial Hebert, Anind Dey, and Siddhartha Srinivasa. Planning prediction for pedestrians. In JEEE/RSJ International Conference Intelligent Robots and Systems, 2009.
11

---

Published as a conference paper at ICLR 2023
A EXAMPLE WHERE LOWER-BOUND OCCURS
Let us consider stochastic bandit with continuous actions A E 10, 1.. Since bandits consist of single, stationary state, we drop dependence on state in all quantities. The reward for choosing action E A is r(a; 8) a1 a) 1-6 for some parameter 8 E (0, 1). When @ is close to 0, the reward is higher for actions close to O, and vice-versa when / close to
For simplicity, let us only consider a dataset of a single action 0i Let us consider a Boltzmann rational policy as the assumed model, namely F(a; 0) exp(r(a; 0)) and have the demonstrator policy be an adversarial perturbation of से that overestimates the reward of a1:
x (a; 0) exp(r(a; 0)) {1 & (a1 2..a 2)} 10 1fa (a1 2. + 9)) if < 0.001 exp(r(a;))
otherwise,
for some E (0, 1). The interpretation of this is that the human is believed to be noisily optimal; however, the human actually overestimates the value of an infinitesimal region centered at action ai only if 0 is close to O. Note that ( 7)) < cō for some constant
we can choose 8 such that the two policies are "close" each other. When 01 1, we will infer 0 ze however, 0r x leading to reward inference error equal to the range of reward parameters.
(
B EXAMPLE WHERE LOG- LOG-CONCAVITY IS VIOLATED
The environment is a 3 3 gridworld with deterministic transitions and discount y 1. Let s be the center cell, and be going up. In Figure 9, show that natural policy that chooses "up" according to x (a 8; ()) xx exp(max (max 10 - 0)) violates log-concavity. The reason that "up" is optimal for f E 00, 4] U (6, 10] but not in between.
-36
-10
-12
Figure 9: Simple navigation environment where a near optimal policy violates Assumption 2.
C PROOFS
C.I PROOF OF THEOREM 2
Without loss of generality, let स satisfy #(a 0) = exp($(s, a; 0))/ 0). Note that this parameterization can be used to express any probability distribution. Also, for any 8 > 0, let define
BD( = Ü ar
5 ar + 8
2
as union of -balls around the actions that appear in dataset Then, for any 0" E ④ let satisfy
( (exp( (s, a; (a & Bs(D)) + C 1fae Bs(D)) if - 8", [(:00) exp($(s,a;0)
n² (a 5;
otherwise,
where is the supremum C sup, exp( (s, a; 0)). By construction, is clear that MLE on D using n² would yield reward parameter 0*, Since such x² can be constructed for any 0*, we can choose 0* that satisfies ||0 0* l1 supa.e or ||0 2/12 What remains is showing that there exists 5 such that (n="
(n" 7) <
12

---

Published as a conference paper at ICLR 2023
Bounding the worst-case policy divergence. satisfied at 0" Fix any state 8 S. We have,
Note that the worst-case divergence is necessarily
/ T-" (a (a 0*) log
a' s:0")
DRL( ( (1 s;0*))
da
7(a
8; s;0")
- (ABB:(D) (a s; 80) log T (a 8; 8; da /
log (a s; s;0") i[a s;8-) da.
a' (;;0)
i(a s;
Bs(D)
We consider each term individually. Starting with the first term, we have
/ JAPBADD) aa s; 0;
Ta s;8") #(a 8; 0*) da & log Z(s (s; 0*)
((s;0) [s; - (ss
where we use that the policies only differ in their normalizers in the first inequality, and that log(t) - log(s)) I min(t,s} It - s| in the second. Now, using that
(x;00)) JAYB. exp(#(s, a, 0² da + C CIB)(D))
we have
SA x" (a 8; 0²) log
(a 8;0") T(a s; 8; da S
C
[Bs(D))
())
Now, let us consider the second term. we have
(a s; 0*)) T(a 8; 8;
da S / Bs(D)
(a s; llog C - o(s, a: 0*) da + log 7(x;0)
Ba(D) log
(a 8; 8")) (::0")oop
Z' 8+)
C
2og C CBB] + 7(s; 0+) [Bx(D),
where we reuse the bound for the first term, and use that (ø(s, a; 0* I log C. Combining the two bounds yields
2C
DT | (- s; 0*)) < 2 log C' [Bx(D)
|B (D))
Z(s; 0+)
Using that (D) I nō by construction, we can solve for (EE/n) such that ("" < E. as desired. This completes the proof.
C.2 PROOF OF THEOREM 3
Recall that L(0; n, D) is the negative log-likelihood of demonstrations D under policy π and reward parameters Note that we can write
L(O; D) 1
n Σ [ml
n =at
863 0)
log ((t)) 8x;
log π (az 181 + log
7(at
81; 0)
L(0 D) log
1 72 n
(at
81;
188 0) ((a
By the law of large numbers, have that under expectation over dataset D,
log n" (a, sri sr: = Eewar [DKI, s;0*) 7(- s; ()))
E E n
F(a 0);0)
Using Assumption 2 on क. for any θ we also have
L(0;m", D) 2 L(O"; m. D) + VoL(O'; " ") T0 -0) + CR
0 0-
2
13

---

Published as a conference paper at ICLR 2023
By definition of l" and Assumption 1, we know that VoL(6 n², D) 0. Substituting 0 = and rearranging yields
10 10-07 0113 2
((0 L'0'==
Analogously, using Assumption 2 on मं we have that
2
1 - [((0 D) - L(0:
Combining the two bounds yields,
||0 - 11 -
;T L(0"; ; ,D) +
10; D) L(θ; म. D)
" (a, a1
sc: 0*) #(a
1
x (a, sc; 0)
log
log
Sy: 0*)
n t=1 i(ay sc; 0)
t=1
Taking expectation over dataset D yields the desired result
IE 10-
01 - Band [DKL (: s;,0))))))))))))))))))) || #- s; ss)0)) (T (. (1:50) || # s; 0)
< Band (P (:18;0))))00000))0)))))))))))() (- 8; 0;))
which is the desired result.
C.3 PROOF OF COROLLARY 4
Recall that म. n² are parameterized by Q-values Q Q" that satisfy the soft Bellman update in (4).
Fix state s. We have
E. exp(((s,a)))))
DKL (T s; ) {;)")) (
[(5,a: log E. exp((((ss,0;00*)
For any action a, we have
Q" Q"(ss 0*) - a)) =7 Σ Pn (s,"_=)))(((''0)) - E ((8(ss,))((s;0)
(s' (8,)) + 1p(51s,,
Σ 1, - Σ
- P(d 8,a))(s;00)
(- s.a) P(: s, s. + Σ (
(s))
Rmax Ap r M.'Epp (s s. max ((''''''0))
R Ap.
(1-7)2
Now, let consider the normalization term. We have
Σ.. expe a'; (');
log explo(s, 0'))
Σ Ex exp(( (s, al; 8)))
log
E. exppp"(spsss d'; )))
E. exp(@(s,d;;0)))
((s, 0r) (x,a';)0*)
A |||mmax Ap. (1 (1-7)²
Combining the two bounds and taking an expectation over s yields the desired result.
'In the statement of A Assumptio 2 in the main paper, we only assume log-concavity for x". This will be corrected in future revision to include both x"
14

---

Published as a conference paper at ICLR 2023
C.4 PROOF OF COROLLARY 5
The proof follows the format of the proof for Corollary 4. Fix state 8. We have
(x |-| ss; (1 s;(*)) Bo (1s00) - log En exp(@(s,,';80))
En exp(Q"(s,
For any action a, we have
(" - (s. =1 Σ P(s (s0)))())))) E P(s' (,,))\))))))))
7" Σ ((=1),)) + + Σ (('11,,)V((3000
Y E (((11,,))((000)) -
(7" 7) Σ P(s' (s.))((6:00 +
(,,a((V [ (()))))
Rmaxx 14* 71 t Σ P(s' (ax -(()
1 1
Rimax h" 71.
Now, let us consider the normalization term. We have
En exp a'; (*))
a':;
Ex exp(@(s,at Σ (e log exp(Q" (s, a'; 0r)
log
E. exp(p' (s, a:; ))
-s,
Rmax 7) 17 7"
Combining the two bounds yields the desired result.
15

---

Published as a conference paper at ICLR 2023
Polccy pao.
Palicy vistaation for
Figure 10: Visualization of gridworld policies (as state-visitation distributions) with (a) different transition
biases (probability of unintended transitions), and (b) different discount factors.
D EXPERIMENT DETAILS
D.1 TABULAR EXPERIMENTS
Environment and training details Recall that the gridworld environments we consider are de- scribed by 8 4 grids, with start and goal state, and walls, lava, and exactly one waypoint state placed in between. We consider a sparse reward where the agent earns a reward of 3 upon reaching the goal state. Alternatively, if the agent reaches lava waypoint state, then its reward or 1, respectively, for the rest of the trajectory. The agent is able to move in either of the four direction (or choose stay still), and there is p 30% chance that the agent travels in different direction than commanded. We choose 7* = 0.98 high enough that the goal state is preferred over the closer waypoint state under the optimal policy.
A reward-conditioned policy (model or demonstrator) under each environment is given by T(a s; 0) exp(Q(s, a; 0)). where ((s, a; a; were derived by value iteration using the an MDP model (can be the true underlying MDP or a biased one) of the environment. During reward inference, we discretize the reward parameter space tabular, instead of sampling demonstrations D from π,, we can instead compute w the discounted stationary distribution. Specifically, let p be the distribution of the starting state (which in our case, an indicator vector at the start state of each environment), then w satisfies:
[1. 4] with resolution 64. Because the environment
= (1- r)p(s)
ΣΣ w" | 8))(sss',a) e'ESa'EA
We can use this to solve for the true state visitations w" for any demonstrator policy x², which can be used compute the weighted policy divergence in (3) without explicitly sampling dataset D of demonstrations.
Visualization of biased policies In Figure 10, we visualize the demonstrator policies x² under the systematic biases considered. We see that in Figure 10(a). when the demonstrator underestimates the probability of unintended transitions, heavily prefers the goal state, which has higher reward but is much more dangerous to reach, over the waypoint state Conversely, in Figure 10(b), when the demonstrator underestimates the discount factor, they strongly prefer the waypoint state that yields lower reward but is much closer.
D.2 CONTINUOUS CONTROL EXPERIMENTS
Environment and training details Recall that the domain we consider is the Lunar Lander game, where agent needs navigate lander onto the landing pad. The reward function yields large
16

---

Published as a conference paper at ICLR 2023
Figure 11: Visualization of Lunar Lander trajectories for policies with (a) biased internal dynamics that underestimate left-right acceleration and (b) correct internal dynamics.
2.5
0.70
2.0
Resand inteeneeeen erme
0.35
0.30
1.0
Wesghod
0.5
0.09
0.0
0.2
0,4
0.6
0.8
1.0
0.00
0.0
0.2
0.4
0.8
1.0
-0.5
0.05
0.13
0.20
Probsbilty of human trajectory
Probabilltty human trajectoryy
Weigheed policy divergnnce
Figure 12: Effect of human bias (measured probability of acting according human policy) on (a) weighted policy divergence and (b) reward inference error on discrete Lunar Lander enviromments. In (c), we show scatter plot of the policy and reward errors for fixed probabilities.
reward for landing the pad, and penalty for crashing going out of bounds. The magnitude of the reward depends on the speed and tilt of the lander upon reaching the landing pad. The physics of the game deterministic. The reward parameter [0, try infer is the location of the landing pad (expressed as normalized horizontal displacement).
In this domain, we train a reward-conditioned policy <(a s; 0) by folding the reward parameter into the state representation, which an 8-dimensional vector capturing the lander's current location, velocity, and tilt. The policy parameterized as a 3-layer fully-connected neural network with hidden dimension of 128, and outputs a squashed Gaussian distribution over actions. Because the state and action space are continuous, we use soft-actor-critic (SAC) (Haarnoja al., 2018) with fixed entropy regularization
1. We train the policy for 600 episodes of length at most 1, 000, with batch size of 264, until the policy was able to land on the landing pad with a high success rate. During reward inference, we discretize the reward parameter space θ [0, 1] with resolution 32. We sample datasets D consisting of 10, 000 observations, and report the mean and standard error of policy and reward error across 10 independent samples of datasets.
Visualization of biased policies In Figure 11, we visualize the demonstrator policies x* under different degrees of internal dynamics bias. Recall that parameter p describes how much one unit of power will increase acceleration in the left-right directions. When p is underestimated, the policy will not move right enough to reach the landing pad; in contrast, when p properly estimated, the policy will reach the landing pad with high success rate.
D.3 ADDITIONAL EXPERIMENT WITH HUMAAN POLICIES
In line with the experiments with simulated biases, we run an additional experiment similar to the one in Section 6.3, where we instead keep ã fixed as the optimal policy, and interpolate between the optimal policy and the real human policy for demonstrator policy ~*. We show the effect of the interpolation proportion policy divergence and reward inference error in Figure 12. Again, notice the consistent message that policy error bounds reward error.
17


Reviews:
[{"review_id": "jA5BaqKTZn", "reviewer": "Reviewer_vrff", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inferring human reward functions requires 'human models' which specify how a human acts given their reward function. This paper studies how errors in the human model translate into errors in the inferred reward. First it shows a pessimistic result: in an adversarial setting, a small error in the human model can lead to a large error in the inferred reward. However, under some assumptions the paper shows that the reward error can be bounded linearly in the human model error (for particular choices of error metrics). The paper show that this result applies to several known human biases. The result matches up with experiments in two environments. The authors thus make an optimistic conclusion: reward inference with misspecified human models is not very difficult. ", "summary_of_the_review": "The paper makes a significant contribution to filling an important knowledge gap: can we accurately infer reward functions with a slightly misspecified human model? The theoretical results are clear and useful. While the overall positive conclusion seems to need caveats, these do not affect the strength of the contributions. If the caveats can be clarified, I would consider raising my score since I believe the paper will be among the more useful ones accepted to ICLR. \n\nEDIT: The authors partially addressed the suggestions I made and argued for the generality of the results (for example by noting that the KL upper bounds the TV distance). This is one of the somewhat rare papers in modern ML where a clean theoretical result provides a clear and non-obvious practical benefit. In addition, the benefit pertains to a problem that is becoming increasingly important. Assuming the authors make the minor changes I suggested, I think the paper should be considered for an award. ", "strength_and_weaknesses": "The paper studies an important problem: To infer reward functions, a human model is necessary, but it is infeasible to infer it accurately. This is because inferring it would at least require the human's reward function, which is initially unknown. This problem becomes more important as AI systems attempt more complex tasks where human behavior is nowhere near optimal and cannot be modeled as optimal.  If we know the relationship between reward errors and human model errors, we can better understand if we can improve reward learning by learning better human models. This understanding also helps clarify if reward learning is a viable strategy at all for complex tasks where humans are increasingly suboptimal. Both the optimistic and pessimistic result yield useful insight on these questions. The results are clear and succinct and to my knowledge novel, providing a clearly significant improvement in our knowledge of the important question at hand. \n\nWeaknesses\n\nIn the first and last section, the authors make an optimistic conclusion overall, saying that their results imply that reward inference with misspecified human models is not very difficult. The positive conclusion needs some strong caveats stated in the first and last section to avoid false impressions. Though I think these caveats should be stated, I don't think they detract, and in fact add to, the value of the paper. By order of importance:\n\nFirstly, the introduction speaks of mild assumptions for the positive conclusion that a slightly misspecified human model will lead to small errors in the inferred reward. However there is an assumption baked into the choice of error metric for human models: the KL(human_policy | model_policy). To assume that the human model error is small, this KL divergence assumes that the human policy would *never take an action that the human model places ~0 probability on*. Otherwise, the KL will be large, allowing large reward errors, and challenging the paper's positive conclusion.\n\nAside from noting this caveat, the authors could reproduce their results with a sufficiently different metric such as the reverse KL. Furthermore, the authors could note that having a small forward KL requires that the human model 'covers' the support of true human, as otherwise the KL would be large. This is an interesting result, as it means that, all we need is to learn human models with good coverage. (And it is not a problem if the human model sometimes takes actions that humans would not take). \n\nSecond, the authors construct a case where the inference error is large, but then argue that their construction is unlikely to occur in practice and that this supports their optimistic conclusion. However, they have not ruled out that other constructions exist. Some of these may be more likely in practice. \n\nThird, the paper could justify its choice of reward distance metric, the squared distance between reward parameters. In particular, a small error in inferring the reward parameters might still lead to a large error in the resulting return and policy. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eZKbPs4ufqJ", "reviewer": "Reviewer_tbzC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work analyzes the effect of model accuracy on reward inference accuracy when fitted to human behavior. Specifically, it shows it is possible that a small model error leads to a very large error in inferring the reward. However, this scenario is unlikely. The authors backed their claims with simulations and real human data.", "summary_of_the_review": "It is a good solid paper to me both in terms of theoretical analysis and experimental results, but human data experiment should be more clear.  ", "strength_and_weaknesses": "Strength: \nThe paper tackles an important problem. It contains both theoretical and empirical results. Moreover, the experiments contain both simulation and real data.\n\nWeakness:\n My main concern: I did not find IRB approval information on the human experiment. If there is, it should be mentioned, if not the authors should explain why it is not necessary in this case (and should be validated with the conference chairs). Also, the details of the experiment and instructions to the demonstrators should accompany the paper. \n\nWorst case v.s. average (end of page 3, beginning of page 4): If I understood correctly the worst-case error is based on all states but the average is based on available states in the data. This looks inconsistent, what is the rationale behind it? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aLpjdxDEaOp", "reviewer": "Reviewer_Se9W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the question of whether misspecification of human behavior models for IRL can lead to detrimental effects on reward inference accuracy, and provide theoretical guarantees showing that while adversarial situations can be constructed such that small human behavior models lead to \"catastrophic\" inference errors, under a log-concavity assumption the effect is more stable. The authors then provide empirical results showing a correlation between human model accuracy and reward inference accuracy.  ", "summary_of_the_review": "\nMy primary concern and reason for rejection is the rather simplified assumptions the authors make in order to \"get the theory to work\".  It is also unclear what the ultimate takeaway here is - if the goal of studying the sensitivity of reward inference is to make some kind of claim on whether reward learning is a valid direction, then only studying restricted forms of human model misspecification does not really lead to any meaningful insights. I think a more clear, isolated section stating the author's assumptions is necessary for acceptance. ", "strength_and_weaknesses": "\nStrengths: the paper is very well-written, and I appreciate the attempt to explore different types of human irrationality (e.g. myopia), and the experiments include real human data (albeit w/ a very simplified form of misspecification via simple re-weighting)\n\n- I am a bit confused why measuring reward misspecification by distance in parameter space works except for very restrictive, simple linear reward models. If the reward function is a NN, then it is well known that two networks that behavior similarly can have widely different parameters. Any necessary assumptions here should be clearly stated. \n- I am likewise confused why KL diverge between demonstration policy and the human model is the right way to measure distance. Given the motivations discussed in the paper of modeling humans via noisy-rationality, systemic biases, etc., it seems like humans may deviate from an idealized model in ways much stronger and more state-specific than the KL measure assumes. \nWhile I like the idea of fitting certain types of irrationalities (myopia and illusion of control) into this framework, it feels a bit cherry-picked to only study irrationalities that satisfy the specific notion of distance the authors chose. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6hXmpQLKSO", "reviewer": "Reviewer_PHhN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper theoretically and empirically targets the question how/if small errors in learned (human) reward models can lead to large inference errors, thereby making them unusable. The goal then becomes to try to bound the inference error by a function to guarantee usability. The authors argue that it is theorerically provable that the inference error (using maximum likelihood estimation) can be arbitrarily large for sufficiently close human reward functions (learned on finite datasets) for worst-case policy divergence, but that the case is unlikely for \"non-adversarial\" sampled datasets compared under a weighted policy divergence. The paper identifies central assumptions for the human reward functions, which lead to a possible bound of the inference error. The authors also provide an empirical evaluation for diagnostic grid world domains and LunarLandor for generated as well as real human biases. All experiments support the claims of a bounded inference loss.", "summary_of_the_review": "The paper clearly describes and situates the targeted problem for learned reward models and inference and provides thorough theoretical and empirical analyses. The problem at hand is important and the contribution has sufficient impact. I am still wondering if/how the results can be applied to other forms of reward learning.\n\n## Post author response\nI want to thank the authors for their answers to my questions. Also based the other reviews for the paper, my positive evaluation of the paper remains. ", "strength_and_weaknesses": "Pros:\n* High clarity wrt motivation, related work, assumptions and conducted theoretical and experimental work\n* Important topic - it is a baseline for numerous human-centered RL approaches focussing on AI alignment\n* Careful evaluation\n\nCons:\n* Minor open questions to reward comparison / generalization to other reward learning settings. How, for example, does recent work on  equivalent-policy invariant comparison fit compared to the chosen reward similarity? Would it be possible derive similar results here or are such approaches prone to get caught in problems for adversarial samples? Is the derived bound dependent on chosen IRL methods / methods that assume a dataset of expert states and actions or would it transfer to reward functions learned based on trajectory comparisons or other kinds of datasets?\n* The related work could also other works on reward learning, e.g. from human preferences", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%