Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
Authors: Joar Max Viktor Skalse
Alessandro Abate
Venue: ICLR 2024 poster (2024)
Keywords: inverse reinforcement learning
reward learing
misspecification
Primary area: learning theory
TL;DR: We analyse how sensitive the inverse reinforcement learning problem is to misspecification of the behavioural model.

Abstract: Inverse reinforcement learning (IRL) aims to infer an agent's *preferences* (represented as a reward function $R$) from their *behaviour* (represented as a policy $\pi$). To do this, we need a *behavioural model* of how $\pi$ relates to $R$. In the current literature, the most common behavioural models are *optimality*, *Boltzmann-rationality*, and *causal entropy maximisation*. However, the true relationship between a human's preferences and their behaviour is much more complex than any of these behavioural models. This means that the behavioural models are *misspecified*, which raises the concern that they may lead to systematic errors if applied to real data. In this paper, we analyse how sensitive the IRL problem is to misspecification of the behavioural model. Specifically, we provide necessary and sufficient conditions that completely characterise how the observed data may differ from the assumed behavioural model without incurring an error above a given threshold. In addition to this, we also characterise the conditions under which a behavioural model is robust to small perturbations of the observed policy, and we analyse how robust many behavioural models are to misspecification of their parameter values (such as e.g. the discount rate). Our analysis suggests that the IRL problem is highly sensitive to misspecification, in the sense that very mild misspecification can lead to very large errors in the inferred reward function.

Published as a conference paper at ICLR 2024
QUANTIFYING THE SENSITIVITY OF INVERSE REIN- FORCEMENT LEARNING TO MISSPECIHICATION
Joar Skalse & Alessandro Abate
Department of Computer Science Oxford University {joar. mooar.skall skalse, aabate @cs. ac..c.a......................... uk
ABSTRACT
Inverse reinforcement learning (IRL) aims to infer an agent's preferences (repre- sented as a reward function Rt) from their behaviour (represented as policy т). To do this, we need behavioural model of how π relates to R. In the current litera- ture, the most common behavioural models are optimality, and causal entropy maximisation. However, the true relationship between a hu- man's 's preferences and their behaviour is much more complex than any of these behavioural models. This means that the behavioural models are misspecified, which raises the concern that they may lead to systematic errors applied to real data. In this paper, we analyse how sensitive the IRL problem is to misspecifica- tion of the behavioural model. Specifically, we provide necessary and sufficient conditions that completely characterise how the observed data may differ from the assumed behavioural model without incurring an error above a given threshold. In addition to this, we also characterise the conditions under which behavioural model is robust to small perturbations of the observed policy, and we analyse how robust many behavioural models are to misspecificcation of their parameter values (such as c.g. the discount rate). Our analysis suggests that the IRL problem is highly sensitive to misspecification, in the sense that very mild misspecification can lead to very large errors in the inferred reward function.
ienttmaann-aationnalln
INTRODUCTION
Inverse reinforcement learning (IRL) is a subfield of machine learning that aims to develop tech- niques for inferring agent's preferences based on their actions in sequential decision-making problem (Ng & Russell 2000). There are many motivations for IRL. One motivation to use it as tool for initation learning, where the objective is to replicate the behaviour of an expert in some task (e.g. Hussein et al.) 20.7). In this context, it is not essential that the inferred preferences reflect the actual intentions of the expert, as long as they improve the imitation learning process. Another motivation for IRL is to use it as a tool for preference elicitation, where the objective is to under- stand an agent's goals or desires (c.g. Maddield-Menell importance that the inferred preferences reflect the actual preferences of the observed agent. In this paper, we are primarily concerned with this second motivation.
al. 2016). In this context, it is of central
An IRL algorithm must make assumptions about how the preferences of the observed agent relate to its behaviour. Specifically, in IRL, preferences are typically modelled as reward function R. and behaviour is typically modelled as a policy æ. An IRL algorithm must therefore have behavioural model that describes how computed from R: by inverting this model, can then be deduced based on In the current literature, the most common behavioural models optimality, Boltzmann rationality, or causal entropy maximisation. These behavioural models essentially assume that the observed agent behaves in way that is (noisily) optimal according to its preferences.
One of the central difficulties in IRL is that the true relationship between a person's preferences and their actions in general is incredibly complex. This means that it typically is very difficult to specify a behavioural model that is perfectly accurate. For example, optimality, Boltzmann rationaally. and causal entropy maximisation are all very simple models that clearly do not capture all the nuances

---

Published as a conference paper at ICLR 2024
of human behaviour This means that these behavioural models are misspecified, which raises the concern that they might systematically lead to flawed inferences if applied to real data.
In this paper, study how robust IRL is to misspecification of the behavioural model. If an IRL algorithm that assumes a particular behavioural model is shown data from an agent whose behaviour violates the assumptions behind this model, then will the inferred reward function still be close the true reward function? We will analyse this question mathematically, and provide several quantitative answers. In particular, we provide necessary and sufficient conditions that completely characterise what types of misspecification a wide class of behavioural models will tolerate. In addition to this, we also study two specific types of misspecification in more detail (namely, perturbation of the observed policy, and misspecificatioo of the parameters in the behavioural model) and provide additional results that describe how they affect the quality of the inferred reward. Our analysis is highly general it applies to the three behavioural models that are most common in the current IRI. literature, but is also directly applicable to a much wider class of models.
The motivation behind this paper is to contribute towards theoretically principled understanding of when IRL methods are applicable to the problem of preference elicitation. Human behaviour is very complex, and while we can create behavioural models that are more accurate, it will never be realistically possible to create a behavioural model that is totally free from misspecification. It is therefore crucial to have understanding of how robust the IRL problem is to misspecification, and whether a small amount of misspecification leads to proportionally small error in the inferred reward function. Our work aims to further our understanding of these question.
1.1 RELATED WORK
There are two previous papers that analyse how robust the IRL problem is to misspecified be- havioural models; Hong et al. (2022) and [Skalse & Abate (203)) Our work is more complete than these earlier works in several important respects. To start with, our problem setup is both more realistic, and more general. In particular, in order to quantify how robust IRL to misspecification. we first need a way to formalise what it means for two reward functions to be "close". Skalse & Abate (2023) formalise this in terms of equivalence relations, under which two reward functions are either equivalent or not. As such, their analysis is somewhat blunt, and is unable to distinguish between small errors and large errors in the inferred reward. HHog et al. (2022) instead use the (2-distance between the reward functions. However, this choice is also problematic, because two reward functions can be very dissimilar even though they have a small didditancc,, and vice versa (cf. Section 2.2) By contrast, our analysis is carried out in terms of specially selected metrics on the space of all reward functions, which are backed by strong theoretical guarantecs. Moreover, [Hong let al. (2022) assume that there is unique reward function that maximises fit to the training data, but this is violated in most real- world cases (Ng & Russell (2000) Dvijotham & Todorov] 2010) Cao
al. |202 Kim et al., [202]] Skalse et al.) 2022 Schlaginhaufen & Kamgarpour (023). In addition to this, many of their results also assume "strong log-concavity", which is rather opaque condi- tion that is left mostly unexamined. Indeed, Hong et al. (2022) explicitly do not answer strong cog-ciicy should be expected to hold under typical circumstances. Both (Skalse & Abate (2023) and Hong et al. (2022) recognise these issues as limitations that should lifted future work. The analysis we carry out in this paper is not subject to any of these limitations. Moreover, in addition to being based on a more sound problem formulation, our paper also contains several novel results that are not analogous to any results derived by Skalse & Abate (2023) or Hong et al. (2022).
There are also earlier papers that study some specific types of misspecification in IRL. In particular, Freedman et al. (2020) study the effects of misspecified choice sets in IRL, and show that such misspecificatioon in some cases can be catastrophic, and Viano et al.] (202T) study the effects of misspecified environment dynamics, and propose an algorithm for reducing this effect. By contrast, we present a broader analysis that covers all forms of misspecification, within a single framework. Also relevant is the work by Armstrong & Mindermann (2018), who show that it is impossible to simultancously learn a reward function and behavioural model from single data set, given an inductive bias towards joint simplicity.
Indeed, there are detectable differences between data collected from human subjects and data synthesised using these standard behavioural models, see Orsini 2021
2

---

Published as a conference paper at ICLR 2024
1.2 PRELIMINARIES
A Markov Decision Processes (MIP) is tuple T,, R, is a set of actions, T : SxA - (S) is a transition function, Me E ASS) is an initial state distribution, R: AxAxS + R is reward function, and 7 E (0, is a discount rate. Here A(X) denotes the set of all probability distributions over X. In this paper, we assume that S and A are finite, and that all states are reachable under 7 and P.o policy
where S is set of states, A
function ι :S + A(A). A trajectory& Ç (su, ag. $1, a1 ...) is a possible path in an MDP. The return function G gives the cumulative discounted reward of a trajectory, G(E) = Σr TR((s,, 0e. 8t+1), and the evaluation function J gives the expected trajectory return given a policy, ((x
Re- [G(E]]. A policy maximising J is an optimal policy. The value function V* S - R of a policy encodes the expected future discounted reward from each state when following that policy, and the Q-function is Q" (s. a) = E [R(s, a, S") + 7V= (S').. Q" and V* denote the optimal Q- and ffnctionn
An IRL algorithm needs& a behavioural model that describes how the observed policy relates to the underlying reward function R. In the current IRL. literature, the most common models are optimal- ity, where it is assumed that ı is optimal under R (e.g. [ & R 2000), Boltzmann- rattinaall,, where it is assumed that (()(s)) machandran & Amir, 2007), and maximal causal entropy (MCE), where it is assumed that πT max- imises the causal entropy objective, which is given by B B > (R(S,, A,, Se41) (H(()))))
a) x e²Q"(s.a), where |3 is a temperature parameter (e.g. Ra-
where a is a weight and H is the Shannon entropy function (e.g. Ziebart 2010).
Two reward functions R1, Rg are said to differ by potential shaping (with 7) if there is a function t : S - R such that R2(8, a, 8¹) Rí (8, a. 8) + (8') ((s) (Ng et al. 1999). by S" redistribution (with 7) if (gr [R, (s, a, S')] = (ss R2 a, S"] (Skalse et al. 2022). and by positive linear scaling if there is a positive constant e such that R2 = c -1.
Given a set X, pseudometric on X any function d : X x X + R that satisfies the conditions that d(ær = 0, d(x, y) 2 0, d(x, y) = d(y, x)) and d(z, 2) S d(x, y) + d(y,2), for all x, y, E X. A metric additionally satisfies that if d(xx, 3)
then x y.
If a reward function R satisfies that J(TT)) = (T2) for all and T2, then we say that R is trivial. All constant reward functions are trivial, but there are always non-constant trivial rewards as well
2 THEORETICAL FRAMEWORK
In this section, we will introduce the theoretical definitions and machinery that we will later use to analyse how robust the IRL. problem is to different forms of misspecification.
2.1 DEFINING MISSPECIFICATION ROBUSTNESS
Before we can derive formal results about what types of misspecification different IRL algorithms are robust to, we first need to construct an abstract model of the IRL problem. First of all, assume that we have a fixed set of states S and fixed set of actions A, that R is the set of all reward functions R. AxX + R, and that II is the set of all policies ι : S - ((A). We say that behavioural model a function f R - II, i.e. a function that takes a reward function and returns policy. For example, the function that, given It, returns the noltzmann-rattonns policy of R under transition function т. discount 7, and temperature B, is an example of a behavioural model. Using this, we can now model the IRL learning problem follows: first, we assume that there is true underlying reward function R", and that the training data is generated by a hehavioural model g. so that the learning algorithm observes the policy given by g R(R Moreover, we assume that an IRL. algorithm L has a model f how the observed policy relates to Rt, where ff is also a behavioural model, such that L converges to a reward function Rd, which satisfies (Ra) π = g(It). If f + g. then f is misspecified, and otherwise f is correctly specified.
For convenience, if ((R, = f(R) whenever R2 and R2 differ by potential shaping, then we say that f is invariant to potential shaping, and similarly for S'-redistribution and positive linear scaling.
"These are given by applying potential shaping and S' s'rrddistribution to constant reward function.
3

---

Published as a conference paper at ICLR 2024
In many contexts, we have prior information about R", beyond the information provided by g(R)), For example, we may know that R. is defined in terms of some sparse state features, or we may know that it only depends on the states, and on. To model this, we give our definitions relative to arbitrary set of reward functions R C R. We then assume that the true reward function R* is contained in R. and that the learning algorithm L will learn a reward function R2 that is also contained in R. Unless otherwise stated, our results apply for any choice of R (including R = R).
Intuitively speaking, we want to say that a behavioural model f is robust to misspecification with g if a learning algorithm that is based on f is guaranteed to learn a reward function that is "close" the true reward function if it is trained on data generated from g. To make this statement formal, we need a definition of what it means for two reward functions to be "close". ". In this paper, we assume that this is defined in terms of some pseudometric d on R. In Section 2.2 we discuss how to choose this pseudometric. Unless otherwise stated, our results apply for any choice of pscudometric. Note also that while dº of course may be proper metric, we allow it to be pseudometric since we may want to consider distinct reward functions to be equivalent. Using this, we can now give our formal definition of misspecification robustness:
Definition 1. Given set of reward functions R C R, pseudometric dl on R, and two behavioural
models f,9 R + II, we say that f is e-robust to misspecification with y if each of the following conditions are satisfied:
1. If ((R) (R2)) then d' (Ri, R2) S e, for all R1, R2 R.
2. If f(R) =(2a then dI (R) R2) S e, for all R1, R₂ E R.
3. Im(g)= C Im(f) on R.
4. There exists R1, Ra E R such that ((R)) + g(R2).
Before moving on, let us explain each of these conditions intuitively. The first condition is saying that any learning algorithm L based on / is guaranteed to learn reward function that has distance of at most e to the true reward function when trained on data generated from gi this is the core property of misspecification robustness. Similarly, the second condition is saying that any learning algorithm L based on f is guaranteed to learn a reward function that has a distance of at most e to the true reward function when there is no misspecificatiion; this condition is included to rule out certain pathological edge cases. The third condition is effectively saying that L can never observe a policy that is impossible according to its model. Depending on how L behaves, it may in some cases be possible to drop this condition, but we include it to make our analysis as general as possible. The fourth condition is simply saying that f and g are distinct on R (otherwise is not misspecified!). More extensive discussion of Definition including more subtle issues, is given in Appendix A
We are particularly interested in the three behavioural models that are most common in the current IRL literature, and so we will use special notation for these models. Given a transition function т and discount parameter y, let br.y.8 R > II be the function that returns the Boltzmann-rational policy of R with temperature 33 and let Cr.7.a R > II be the function that returns the MCE policy of R with weight a. Similarly, let Or,, R IT be the function that returns the optimal policy of R that takes all optimal actions with equal probability.
2.2 REWARD FUNCTION METRICS
We wish to obtain results that describe how different the learnt reward function R₁, may be compared to the underlying true reward function R*, given different forms of misspecifiication To do this, we need a way to quantify the difference between R₁ and R*, in the form of a pseudometric dR on R. However, finding an appropriate choice of d is not straightforward. For example, suppose we sim- ply let (RR,, R²) = RR, R* 12 In that case, we would have that dL (Rr, Rt) can be arbitrarily large, even if R₁ and R have the same ordering of policies, and similarly, that dLR (RR, II*) can be arbitrarily small, even if R2 and R" have the opposite ordering of policies [] This means that the l distance between RA and R* does not quantify their qualitative difference useful way.
Too see this, let R be any nontrivial reward function. Then for any positive we have that R and cR have the same ordering of policies, but by making e large, we can make| R cRIla arbitrarily large. Similarly, for
4

---

Published as a conference paper at ICLR 2024
Intuitively, we want pseudometric dd with the property that (R,, R*) is small if (and only if) would be safe to optimise RG, instead of R*. As we have just argued, the e--orrm does not satisfy this condition. Instead,
will use a STARC-metric, introduced by Skkals et al. (2023):
Definition 2. Let T be a transition function and y be a discount factor. Given a reward function
R. let Vr,, (R) be the set of all reward functions that differ from R by potential shaping (with ?) and S'-redistributtin (with r). Let 67,77
STARC
R - R. be the function where eSTARC
(R) returns
the element of (It)(I) that has the smallest e- ! Moreover, let SSTACC R R. be the (R) (TARC (R)ll2
7.7 ll2 if |, (R)l22 0, and 6,77
function where STARC (R) = 777
&TARC (R)ARC(R)/l 7.7
[[
eSTACC (R)
7,y
otherwise. Finally, let (STARC : x R - [0,
be the function given by (Ri, R2) 0.5 1155-117777 118777 (TSSTTAR 87 87,7
F,
STARC 7,7
STARC (R2)112-
There is extensive justification for measuring the distance between reward functions in terms of dSTARC_ However, because the full justification is quite long, and because this justification is essential to understand most of our results, we have decided to provide it in Appendix (B instead of the main text. Appendix also contains less formal, more intuitive explanation of Definition 2 For now, it is sufficient to know that dSTARC RR.. R₂) = 0 if and only if Rı and R2 induce the same ordering of policies under 7 and y. Moreover, while some of our results are given in terms of dSTARC, we also have many results that apply for any choice of pseudometric on R. This will be 7,7 clearly stated each theorem.
Another prominent pseudometric on R is EPIC, proposed by Gleave et al. (202)) Appendix puts forward reasons to not use EPIC for the kind of analysis that we undertake this paper.
2.3 BACKGROUND RESULTS
In this section, give few important results from earlier works that we will rely on throughout this paper, and which will also helpful for contextualising our results. First, is useful to know under what conditions two reward functions have the same ordering of policies:
Proposition AS, T, uu: 71 and (S,A, T, Ho, R2,7) have the same ordering of policies if and only if Rı and R2 differ by potential shaping (with 7/, S'-redistribution (with T), and positive linear scaling.
see Skalse & Abate 2023) (their Theorem 2.6). Also recall that dSTARC (Ri, R2)= 0 if and only if Ri and R2 induce the same ordering of policies. Next, it is also useful know under what conditions ((Rz S(R)) when f is the Boltzmann-rational model or the maximal causal entropy model:
For a proof of Proposition [
Proposition 2. For any % and 3, we have that br. and o, are invariant to potential shaping (with 7) and S' -redistribution (with T), and no other transformations.
For proof of Proposition 2 see Skalse et al. 2022) (their Theorem 3.3). Note that Propositions and 2 together imply that any learning algorithm that based on either the Boltzmann-rations model or the MCE-model is guaranteed to learn reward function R2 that has the same ordering of policies as the true reward function R* when there is no misspecification.
MISSPECIFICATIOO ROBUSTNESS
In this section, we provide main results about the misspecifiictiion robustness of various be- havioural models, including the behavioural models that are most common in the IRL literature. Section 3.1 is quite dense, but 3.2 and 3.3 both provide more intuitive takeaways.
3.1 NECESSARY AND SUFFICIENT CONDITIONS
Given a behavioural model f. it is desirable to have necessary and sufficient conditions that com- pletely characterise when f is robust to misspecification with g. Surprisingly, first result shows
any positive e, we have that eR and -eR have the opposite ordering of policies. However, by making e small, we can make| | eR - (-eR)]l2 arbitrarily small.
"V. (R) is affine subspace of so there a unique element of Vr,((R) that minimises the &
5

---

Published as a conference paper at ICLR 2024
de R2) = 0, then can derive such necessary and sufficient conditions in a relatively straightforward way:
that if f(R)) = f(R2)
Theorem 1. Let R be a set of reward functions, let f: R II be a behuvioural model, and let dR be a pseudometric on R. Assume that {(Ri) = [(Ra) = d (R1, R2) = -0 for all R1, R2 E R. Then f is e-robust to misspecification with g (as defined by d if and only if g f o for some t: R + R such that de (R, ((R)) S for all R E RR and such that I ++1 g.
Recall that d. dSTARC (Bi, (2) 0 if and only if R₁ and R2 induce the same ordering of policies.
Thus, our reward metric is d dSTARC, then Theorem I applies to any behavioural model f for which
f(t) = ((2) implies that Ri and R2 have the same policy ordering. Moreover, also recall that both the --ottzmann-ratiion
model and the MCE model satisfy this condition (Propositions and 2). Thus, if we can find the set T. of all transformations : R R such that d t(R) I c, then we can get the set of all behavioural models to which f is e-robust to misspecification by simply composing with each element of Tc. We next derive Te:
Proposition A transformation & R - R satisfies that dSTARCC (R, ((R)) S for all E R if T.y and only if can be expressed as 1 .... 1n-1 o ln ln+1 o... lm for some n and m where
||R f. (R) 12 S [TTAARC (R)l2 sin(2 arcsin(c/2))
for all R, and for all i * and all R, we have that R and (R) differ by potential shaping (with y), s'rredistribution (with r). or positive linear scaling.
The statement of Proposition is quite terse, so let us briefly unpack it. First of all, STARC is invariant to any transformation that preserves the policy ordering of the reward function, and these transformations are exactly those that can be expressed as a combination of potential shaping. S'- redistribution, and positive linear scaling. As such, we can apply an arbitrary number of such transformations. Moreover, we can also transform R in any way that does not change the stan- dardised reward function SSTARC (R) by more than e; this equivalent to the stated condition on 7,y
4.7
b2- Note that sin(2 arcsin(e/2)) s for small e, so the right-hand side is approximately equal to STARC (R))l22 However, also note that ||CTAARC (Rt)l2 < ||/2||2. Intuitively speaking, a reward transformation satisfies the conditions given in Proposition B if never changes the policy order of the reward function by a large amount.
e (C.
Using this, we can now state necessary and sufficient conditions that completely characterise all types of misspecification that the aaltzmann--attions model and the MCE model will tolerate:
Corollary 1. Let R be a set of reward functions, T be a transition function, 7 a discount faccourfaactor, |3 temperature parameter, and a a weight parameter. Let T. be the set of all functions t R + R that satisfy Proposition 3 and additionally satisfy that t(R) E for all R E R. Then by R - II is e-robust to misspecification with g (as defined by dSTARC) ) if and only if g = br t for some t E T. such that b + g. and Cr, R - II is e-robust to misspecification with g (as defined (STARC) if and only if = Cr.rra o t for some t E T. such that C7,7 # g- by dST
In principle, Corollary [ completely describes the misspecification robustness of the Boltzmann- rational model and of the MCE model (for any R). However, the statement of Corollary | is rather opaque, and difficult to interpret qualitatively. For this reason, we will the subsequent sections examine a few important special types of misspecification, and analyse them individually.
We should also briefly comment on the fact that Corollary dooes not cover 0, i.c the optimality model. The reason for this is that, unless |S] = and [A] = 2, there are reward functions R₁, Rg such that or.7(1;) = 0r,7 (Rz), but 7.77 functions can have the same optimal policies, but have different policy orderings. This means that Theorem I does not apply to or,; when dR = d Moreover.
TSTRRC (R2, R2) > O. This ought to be intuitive: two reward
Proposition Unless (S] that for all < E, there is no behavioural model g such that 0 is c-robust to misspecification with g (as defined by d dSTARC).
and |A| 2, then for any and any y there exists an E > 0 such
This is simply consequence of the fact that the second condition of Definition I will be violated for any e that is sufficiently small. An analogous result will hold for any behavioural model f and any pseudometric dR for which f(R)) f(R)) + d (Ri, R2) 0.
6

---

Published as a conference paper at ICLR 2024
3.2 PERTURBATION ROBUSTNESS
It is interesting to know whether or not behavioural model f is robust to misspecification with any behavioural model y that "close" to f. But what does mean for I and g to be "close"? One option is to say that f and ! are close if they always produce similar policies. In this section, we will explore under what conditions f is robust to such misspecification, and provide necessary and sufficient conditions. Our results are given relative to pscudometric dll on 11. For example, d'l 1 T2) may be the e di-ttannce between T1 and T2, or it may be the KL divergence between their trajectory distributions, or it may be the di-ditancce between their occupancy measures, etc. As usual, our results apply for any choice of d" unless otherwise stated. We can now define a notion of a perturbation and a notion of perturbation robustness:
Definition 3. Let R be a set of reward functions, let f.g R + 11 be two behavioural models, and let dl be pseudometric on II. Then g is a d-perturbation of f if g + f and for all I E R we have that d' (f()), (I)) 6.
Definition 4. Let R be a set of reward functions, let ʃ R + IT be behavioural model, let dR be pseudometric on R. and let dll be pseudometric on I1. Then f is e-robust to d-perturbation if is e-robust to misspecification with g (as defined by d for any behavioural model g : R - II that is a 8- 6-perturbation of f (as defined by d with Im(g) (g) = Im()).
A ò- d-perturbation of f simply is any function that is similar to f on all inputs, and f is e-robust to S-perturbation if small perturbation of the observed policy leads reward function. It would be desirable for a behavioural model to be robust in this sense. To start with, this captures any form of misspecification that always leads small change in the final policy. Moreover, in practice, we can often not observe the exact policy of the demonstrator, and must instead approximate from a number of samples. In this case, we should also expect to infer a policy that is a perturbation of the true policy. Before moving on, we need one more definition:
small error in the inferred
Definition Let R be a set of reward functions, let f R -> IT be a behavioural model, let RR be a pseudometric on R, and let d be a pseudometric on II. Then f is c/ /esseparating d (R1, R2) > E = d' (f()), f(R2)) > 8 for all R1, R2 E R.
Intuitively speaking, f is e e/8-separating if reward functions that are far apart, are sent to policies that are far apart Using this, we can now state our main result for this section:
Theorem 2. Let R be a set of reward functions, let f R - II be a behavioural model, let di be a pseudometric on R2 and let dll be a pseudometric on II. Then I is e-robust to 5-perturbation (as defined by dR and dil if and only if f es0ō-eppaaainn (as defined by dR and dil) )
We have thus obtained necessary and sufficient conditions that describe when a behavioural model is robust to perturbations - namely, it has to be the case that this behavioural model sends reward functions that far apart, to policies that are far apart. This ought to be quite intuitive; if two policies are close, then perturbations may lead us to conflate them. To be sure that the learnt reward function is close to the true reward function, we therefore need to be the case that policies that are close always correspond to reward functions that are close (or, conversely, that reward functions which are far apart correspond to policies which are far apart).
Our next question is, of course, whether or not the standard behavioural models are c/ c/6-separating. Surprisingly, we will show that this is not the case, when the distance between reward functions is measured using dSTARC, and the policy metric dll is similar to Euclidean distance. Moreover, we only need very mild assumptions about the behavioural model to obtain this result:
Theorem 3. Let dR be d and let d be a pseudometric on II which satisfies the condition that for all 5 there exists a 5' such that [[]] < 8' then dl (11, π2) < 5. Let c be any positive constant, and let R be set of reward functionn such that if |||2 e then R E R. Let R II
be a behavioural model that is continuous. Then f is not e/ e/8-separating for any e < 1 or 5 > 0.
'Note that this definition is not saying that reward functions which are close must be sent to policies which are close. In other words, being e/6-separating is not continuity condition. It is also not local property of f. but rather, global property. It is, however, a continuity condition on the inverse of f.

---

Published as a conference paper at ICLR 2024
This theorem is telling us several things at once. To make things easy, we can begin by noting that may let R R, and that we may assume that d" simply is the e-normm i.e. (m1, (22) || - T2ll2. Theorem 3 is then telling us that no continuous behavioural model is c/ c/6-separating for any or 8 (and therefore, by Theorem[] [ also not c-robust to 5-perturbation for any c or 8). Note that the Boltzmann-rational model and the maximal causal entropy model (i.e. br,y,8 and Cr Cra.a)
both are continuous, and hence subject to Theorem 3 The condition given on dll in Theorem 3 is simply generalisation, that covers other policy-metrics than the ( Similarly, the condition on R is also a generalisation certain restricted reward spaces. We give more in-depth, intuitive interpretation of Theorem 3
and an explanation of why it is true, in Appendix D
3.3 MISSPECIFIED PARAMETERS
A behavioural model typically defined relative to some parameters. For example, the Boltzmann- rational model is defined relative to a temperature parameter / and discount parameter Y, as well as the transition dynamics т. Moreover, determining the exact values of these parameters ex post facto can often be quite difficult. For example, there is a sizeable literature that attempts to estimate the rate at which humans discount future reward, and there is a fairly large range in the estimates that this literature produces (e.g. Percocoo & Nijkamp 2009). It is therefore interesting to know to what extent a behavioural model is robust to misspecification of its parameters. rational for discount parameter V/, but IRL algorithm interprets it as being Boltzmann-rattons for discount parameter 7/2, where "/1 2 7/2, then are we still guaranteed to learn a reward function that is close to the true reward function? These the questions that we will study in this section.
I is Boltzmann-
We will first consider the case when the discount parameter, 7, is misspecified. Say that a transition function I is trivial if for all states and all actions d1, 02, we have that T(S, = =(s, We now have the following rather surprising result:
42, then fr, is not Theorem 4. If fr R + 11 is invariant to potential shaping with 7, and 7n # 72. e-robust to misspecification with f72 under (STARC for any non-trivial T, any Y3, and any < 0.5.
41,7
Note that Theorem 4 permits that 73 = or T3- = 72- Of course, any interesting environment will have a non-trivial transition function, so this requirement very mild. Moreover, a df dSTARC -distance desaac-distance of 0.5 is very large; this corresponds to the case where the reward functions are nearly orthogonal. This means that Theorem iss saying that a behavioural model f is invariant to potential shaping, then it is not robust to any misspecification of the discount parameter. Note that this holds even 71 and 7/2 are arbitrarily close! Moreover, optimal policies, Boltzmann-ratione MCE policies are all invariant to potential shaping, and hence Or,y, br,y,8, and Cr,n, are subject to Theorem In general, we should expect any behavioural model that uses exponential discounting to be invariant to potential shaping, and Theorem 4 will apply very widely.
policies, and
We will next consider the case when the transition function, т, is misspecified. Here, we similarly find that very wide class of behavioural models are non-robust to any amount of misspecification:
Theorem 5. If fr R -> II is invariant to S' rreddiittribution with т, and Tı # T2, then fr, is not c-robust to misspecification with fr, under d STARC for any T3, any ", and any < 0.5.
T3:7
Note that 5 permits that T3 = 71 or = 72. Thus, Theorem i saying that a behavioural model is invariant to S' S'-redistribution, then it is not robust any degree of misspecification of (even if Ti and are arbitrarily close). Moreover, optimal policies, Boltzmann- and maximal causal entropy policies, are all invariant to S' d'rredistriuttion, and hence 0r,y, br,y.9, and Cryy. are subject to Theorem 5 Indeed, since S" o'-redistribution does not change the expected value of any policy, we should expect almost all sensible behavioural models to be invariant to S' "'-redistribution. As such, Theorem will also apply very widely.
policies,
Theorem 4 and show that a very wide range of behavioural models in principle are highly sensi- tive to arbitrarily small misspecification of two of their core parameters. To make this result more accessible and easier to understand, we have included two examples in Appendix E that explain the intuition behind these two theorems.
Note that while Theorem 3 uses a "special" pscudometric on R, in the form of de STARC we do not need to special (pseudo)metric IT, because for policies, L² does capture the relevant notion of similarity.
8

---

Published as a conference paper at ICLR 2024
Before moving on, also want to note that the Boltzmann- model is robust to arbitrary misspecification of the temperature parameter, B, and that the maximal causal entropy model is robust arbitrary misspecification of the weight parameter, a. This was shown by [Skalse & Abate 22023) in their Theorems 3.2 and 3.4. Specifically, we have that for any T and 7, any 6 2 0, and any B1, f2, 01, 02 we have that 0,,,,8 is c-robust to misspecification with d7, and that Cr is c-robust to misspecification with Cr.y.0a. defined by dSTARC. For a detailed description of how to connect the results of Skalse & Abate (023) to ours, see Appendix G
4 DISCUSSION
We have quantified how robust IRL is misspecification of the behavioural model. We first pro- vided necessary and sufficient conditions that fully describe what types of misspecification many behavioural models will tolerate. In principle, these conditions give a complete answer to how tol- erant given behavioural model is to any given type of misspecification. However, these conditions rather opaque, and difficult interpret. Therefore, we have also separately provided necessary and sufficient conditions that characterise when a behavioural model is robust to perturbation, and we have analysed how robust many behavioural models are to misspecification of the discount factor Y or the environment dynamics r. Our analysis suggests that the IRL. problem is highly sensitive to particular, a very wide class of behavioural models are unable to guarantee robust inference under arbitrarily small perturbations of the observed policy, or under arbitrarily small misspecification of y or T.
many plausible forms of misspecification.
Our results present serious challenge to IRL in the context of preference elicitation. The relation- ship between human preferences and human behaviour is very complex, and while it is certainly possible to create increasingly accurate models of human behaviour, it will never be realistically possible to create a model that is completely free from all forms of misspecification. Therefore, if IRL. is unable to guarantee accurate inferences under even mild misspecification of the behavioural model, as our results suggest, then we should expect it to be very difficult (and perhaps even pro- hibitively difficult) situations. This in turn means that IRL should be used cautiously, and that the learned reward func- tions should be carefully examined and evaluated (as done by e.g. Michaud et al. 2020 Jenner & Gleave 2022). It also means that we need IRL algorithms that are specifically designed to be more robust under misspecification, such as e.g. that proposed by Viano al. (202 It may also be fruitful to combine IRI. with other data sources, as done by e.g. Ibarz et al. 2018). or consider policy optimisation algorithms that assume that the reward function may be misspecified, as done by c.g. Krakovna al. (2018 2020); Turner et al. (2020); Griffin et al. (2022).
guarantee that IRL reliably will produce accurate inferences real-world
We also need more extensive investigations into the issue of how robust IRL is to misspecification, and there are several ways that our analysis can be extended. First of all, it may in some cases be possible to mitigate some of our negative results if R is restricted. For example, Theorem 4 and rely on the fact that we for any reward R1 can find reward R₂ such that R1 and R₂ differ by potential shaping or S' s'-reiistribution for a given choice of 7 and r, but such that R3 and R2 have large STARC-distance for other choices of y and We may thus be able to circumvent these results by restricting R in way that removes all such reward pairs. However, this of course not straightforward, not least because we need to ensure that the true reward in fact is contained in R. Moreover, some of our results rely on the fact that we use STARC-metrics to quantify the difference between reward functions. While there are compelling theoretical justifications for doing so (cf. Appendix B and C). there may be other relevant options. STARC-metrics are quite strong, and we may be able to derive weaker guarantees using other forms of reward metrics. Furthermore, it may also be fruitful to modify Definition [ for example by making it more probabilistic, generalising it in other ways. This topic is discussed in Appendix A Finally, our analysis can also be extended to other types of behavioural models and other types of misspecification. For example, policies that use (c.g.) hyperbolic discounting subject to a result that is analogous to Theorem A Such investigations also present interesting direction for future work.

---

Published as a conference paper at ICLR 2024
REFERENCES
Stuart Armstrong and Sören Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, volume 31, pp. 5603-5614, Montréal, Canada, 2018. Curran Associates, Inc., Red Hook, NY, USA.
Haoyang Cao, Samuel N. Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learn- ing. arXiv preprint, arXiv:2106.0555 03.99398888 [cs.LG], 2021.
Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solvable MDPs. In Proceedings the 27th International Conference Machine Learning, pp. 335-342, Haifa, Israel, June 2010. Omnipress, Madison, Wisconsin, USA.
Rachel Freedman, Rohin Shah, and Anca Dragan. Choice set misspecification in reward inference. In IUCAI-PRICAI-20 Workshop on Artificial Intelligence Safery, 2020. doi: 10.48S50/ARXIV. 2101.07691. URL https:://axxiiiis
http:///...///1.1101. 07691
Adam Gleave, Michael D Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differ- ences in reward functions. In International Conference on Learning Representations, 2021. 2021. URL
https://openrevier net www.eeeeoo....t... Pid LwEQnn6CYev
Charlic Griffin, Joar Max Viktor Skalse, Lewis Hammond, and Alessandro Abate. All's well that ends well: Avoiding side effects with distance-impact penalties. In NeurlPS ML Safety Workshop, 2022.
Dylan Hadfeeld-Menel,, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative in- verse reinforcement learning. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL httptttsss/prrcced.ng....tt http://gwg...oopppocccc/c/er//// pctppp/pe////11///1eeee.. 53355084cc3447776477666666838883
pdf
Joey Hong, Kush Bhatia, and Anca Dragan. On the sensitivity of reward inference to misspecified human models, 2022.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv. 50(2), 4 2017. ISSN 0360-0300. doi: 10.1145/3054912. / org/10. 111.111.1155335591110
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving. Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Proceedings of the 32nd In- ternational Conference on Neural Information Processing Systems, volume 31, pp. 8022-8034, Montréal, Canada, 2018. Curran Associates, Inc., Red Hook, NY, USA.
Erik Jenner and Adam Gleave. Preprocessing reward functions for interpretability, 2022.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in- verse reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5496-5505, Virtual, July 2021. PMLR.
Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability, 2018. URL httpps://arxii..org////.///
1186600185
Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding side effects by considering future tasks, 2020. URL httttt:///axxiii/ org/ abss/210... 07877
Eric J. Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions, 2020.
Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, volume 1, pp. 663-670, Stan- ford, California, USA, 2000. Morgan Kaufmann Publishers Inc.
10

---

Published as a conference paper at ICLR 2024
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping, In Proceedings of the Sixteenth International Con- ference Machine Learning, pp. 278 287, Bled, Slovenia, 1999. Morgan Kaufmann Publishers Inc.
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? arXiv preprint, wxiiv2106.00667 [cs.LG], 2021. To appear in Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021.
Marco Percoco and Peter Nijkamp. Estimating individual rates of discount: meta-analysis, Applied Economics Letters, 16: 1235 1239, 2009.
CorpusTD: 154901945
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, pp. 2586-2591, Hyderabad, India, 2007. Morgan Kaufmann Publishers Inc.
Andreas Schlaginhaufen and Maryam Kamgarpour. Identifiability and generalizability in con- strained inverse reinforcement learning, 2023.
Joar Skalse and Alessandro Abate. Misspecificatiin in inverse reinforcement learning, 2023.
Joar Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. In- variance in policy optimisation and partial identifiability in reward learning. arXiv preprint 2750022033.0475. 2022.
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro Abate. Stare: A general framework for quantifying differences between reward functions, 2023.
Alex Turner, Neale Ratzlaff, and Prasad Tadepalli. Avoiding side effects in complex envi- ronments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systemx, volume 33, pp. 21406 21415. Curran As- sociates, Inc., 2020. URL httpihtttps::/proceedinnss neurips cc/ wttpcc/cpppeeeeeee pappp/202/e/11111111// E5G60028T00238000311100-----2-22
pdf
Luca Viano, Yu-Tue Yu-Ting Huang, Parameswaran Kamalaruban, Adrian Weller, and Volkan Cevher. Ro- bust inverse reinforcement learning under transition dynamics mismatch. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Process- ing Systems, 2021. URL
wwt. http://wwwwee//tooo/tttttttaeeee tasdmmaddo
Blake Wulfe, Logan Michael Ellis, Jean Mercat, Rowan Thomas McAllister, and Adrien Gaidon. Dynamics-aware comparison of learned reward functions. In International Conference on Learn ing Representations, 2022. /tt.htttss///openeesttt
www.tee/oooumm forum ott.ee/eeeeeeooooooooooooooorro
Brian D Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Camneiie Mellon University, 2010.

---

Published as a conference paper at ICLR 2024
A MOTIVATING OUR DEFINITION OF MISSPECIFICATION ROBUSTNESS
In this section, we provide further discussion and motivation for our formalisation of misspecifica- tion robustness, given in Definition beyond the discussion we give in Section 2.1
A.1 ADDITIONAL COMMENTS ON THE CONDITIONS FOR MISSPECIFICATION ROBUSTNESS
In this section, we make few additional comments some of the four conditions in Definition T In particular, while the first condition ought to be reasonably clear, we have further comments on each of the remaining three conditions.
Condition 2 says that for all R1, R2 R. ((Ra) f(2)) then (R1,, R2) S e. In other words, any learning algorithm L based on f is guaranteed to learn reward function that has a distance of most e to the true reward function when trained on data generated by f. i.e. when there is no misspecification. It may not be immediately obvious why this assumption is included, since we assume that the data is generated by g. where + g. To see this, suppose R = {R,, R2, R3, Ra} where dR (R1, R2) c, d Ra) < c, and d (R2, Rs) > and let f.g R - II be two behavioural models where f(Ri) = W1, [(Bb) (Ra) = T2. f(Ra) = т3. and g (R) =
g(2)) nı. RRs)) - g(Ra) wz. This is illustrated in the diagram below:
T2
T3
f
9
R1
R2
R3
R4
6
E
>
E
In this case, we have that =f (Ra)) but (R2, Rs) >> e. As such, f violates condition 2 in Definition [ learning algorithm L based on is not guaranteed to learn a reward function that has distance at most e to the true reward function when there is no misspecification, because f cannot distinguish between Rg and Rg, which have large distance. However, if f (R) g(R),, it does in this case follow that (R, It') < In other words, if the training data is coming from g, then a learning algorithm L based on f is guaranteed to learn reward function that has distance at most e to the true reward function. As such, we could define misspecification robustness in such way that f would be considered be robust to misspecification with g in this case. However, this seems unsatisfactory, because g essentially has to be carefully designed specifically to avoid certain blind spots in f. In other words, while condition in Definition is met, only met spuriously. To rule out these kinds of edge cases, we have therefore included the condition that for all R1, Ra E R. f(1)) (R2), then it must be that dR (R1, R2) c.
=
Condition 3 says that there for all R1 R exists RS R such that f(R) o(R2). Stated differently, the image of g on R. is subset of the image of ff on R. The reason for why this assumption is necessary is to ensure that the learning algorithm can never observer data that is impossible according to its assumed model. For example, suppose f maps each reward function to a deterministic policy; in that case, the learning algorithm L will assume that the observed policy must be deterministic. What happens if such an algorithm is given data from a nondeterministic policy? This is undefined, absent further details about L, because L cannot possibly find a reward function that fits the training data under its assumed model. Since we do not want to make any strong assumptions about L. it therefore seems reasonable to say that if always produces a deterministic policy, and g sometimes produces nondeterministic policies, then ff is not robust to misspecification with g. More generally, it has to be the case that any policy that could be produced by g, can be
12

---

Published as a conference paper at ICLR 2024
explained under f. This is encompassed by the condition that there for all R₁ R exists R₂ R such that fR2z = g R1). Of course, in many cases we may have that Im(f) = II, ie. that f can produce any policy, and in that case this condition is vacuous.
Condition 4 says that there exists R1, R2 R such that f(R)) + f(22); in other words, that f + g on R. This condition not strictly necessary - from a mathematical standpoint, very little would change if we were to simply remove it from Definition[] [ Indeed. the only effect that this condition has on the results in this paper that Theorem | and Corollary [ add the condition that f * g. and that f F g is part of the definition of 6-perturbations (Definition } Rather, the reason for including the assumption that f + g is purely to make Definition [ more intuitive. If f = 9. then f is not misspecified, and it would seem odd to say that "f is e-robust to misspecification with itself". As such, there is no deeper significance to this condition besides making our terminology more clear.
A.2 ON THE ASSUMPTION THAT BEHAVIOURAL MODELS ARE FUNCTIONS
Here, we will comment on the fact that behavioural models are assumed to be functions; i.e., we assume that behavioural model associate each reward function R with unique policy This is true for the Boltzmann-
model and the maximal causal entropy model, but it may not be natural assumption in all cases. For example, there may in general be more than one optimal policy. Thus, an optimal agent could associate some reward functions R with multiple policies w. This particular example is not too problematic, because the set of all optimal policies still form convex set. As such, it is natural to assume that an optimal agent would take all optimal actions with equal probability, which is what we have done in the definition of 0,,7 However, imagine alternative criteria which would associate some rewards with multiple policies, and where there may not be any canonical way to select a single policy among them. Such criteria may then not straightforwardly translate into a functional behavioural model.
could
There are several ways to handle such cases within our framework. First of all, we may simply assume that the observed agent still has some fixed method for breaking ties between policies that it considers to be equivalent (as we do for 0r In that case, we still ultimately end up with a function from R to II, in which case our framework can be applied without modification. We expect this approach to be satisfactory in most cases.
It is worth noting that this approach does not necessarily require to actually know how the ob- served agent breaks ties between equivalent policies. To see this, let G R + P(II) be a function that associates each reward function with set of policies. We can then say that a behavioural model g R - II implements G if g R)) E G(R) for all R E R. Using this definition, we could then say that f R + II is robust to misspecification with G R - (III) if f is robust misspecification with each g that implements G. where ʃ being robust to misspecification with is defined as in Definition [] In other words, we assume that the observed agent has fixed method for breaking ties between policies in G, but without making any assumptions about what this method is. Using that definition, our framework can then be applied without modification.
An alternative approach could be to generalise the definition of behavioural models to allow them to return a set of policies, i.c. f R P(II). Most of our results can be extended to cover this case in a mostly straightforward manner. However, this approach is somewhat unsatisfactory, because we would then assume that the learning algorithm L gets to observe all policies in the set f(R)). ). However, in reality, it seems more realistic to assume that L only gets to observe a single element of [(R)), unless perhaps L gets data from multiple similar agents acting the same environment.
A.3 ON RESTRICTED SPACES OF REWARD FUNCTIONS
Our definitions are given relative to a set of reward functions R., which in general may be any subset of R.. It may not be immediately obvious why this is necessary, and so we will say a few words about that issue here.
NNot also that this is equivalent to assuming that an optimal agent would take all optimal actions with positive probability, but that the exact probability that associates with each action does not convey any further information about R.
13

---

Published as a conference paper at ICLR 2024
First of all, we should note that we always allow R R. This means that the introduction of R. makes our analysis strictly more general, in the sense that always can assume that 'R. is unrestricted. In other words, nothing is lost by giving our definitions and theorem statements relative to set of reward functions R. instead of the set of all reward functions.
Moreover, there are many cases where it is interesting to restrict R. To start with, functions with type signature S x A x S + R. but is quite common to use reward functions with a different type signature, such as for example 8 A + R or S + E. We can ensure that our analysis covers these settings as well, by noting that we can allow R to be equal to [R& R | Vs, a, $1, 82 R(s, 0, $1) = R(s, a, $2)}, or {R R Vs, 01, 02 $2 R(s, $1) = R(s, a2 $2) }, and so on. As such, by using (potentially restricted) set of rewards R. can make sure that our results do not depend on these design choices.
reward
Additionally, there many cases where may have prior information about the underlying true reward function R" over and above the information provided by the observed policy. For example, may know that the reward function cannot depend on certain features of the environment, or we may know that it only depends on the state of the environment at the end of an episode, and so on. This information may come from expert knowledge, or from auxiliary data sources, etc. In these cases, makes sense restrict R the set of all reward functions that are viable in light of this prior knowledge. Moreover, restricted reward sets also allow us to handle the case where this information given in the form of Bayesian prior, see Appendix[A.A
Another reason for restricting R is that Definition T is existential, in the sense that single coun- terexample in principle is enough to prevent f from being e-robust to misspecification with g, even if f(R)) g( R2) implies d² RR, R2) S for "most" R1 and R2, etc. As such, even if f is not e-robust to misspecification with g. could in theory still be the case that a learning algorithm L based on f is guaranteed to learn a reward function Ra that is close to the true reward function It* for most choices of R We can rule out this possibility by restricting R in a way that excludes gerrymandered counter-examples.
As such, by giving our definitions relative to a set of reward functions R, we make our analysis more versatile and more general.
A.4 ON MAKING THE ANALYSIS MORE PROBABILISTIC
The formalisation of misspecification robustness in Definition 1 is essentially a worst-case analysis, in the sense that it requires each condition to hold for all reward functions. For example, a single pair of rewards R1, Rg with f(R) = g(R2) and dR (R1, R2) > is enough to make it so that not e-robust to misspecification with g. even if (R1) "most" reward functions. This makes sense if we do not want to make any assumptions about the true reward function R', or about the inductive bias of the learning algorithm. However, in certain cases, we may know that R is sampled from particular distribution D over R. In those cases, it may be more relevant to know whether dR²( RA) S with high probability.
g(R2) implies di (R1, R2) S e for
To make this more formal, we may assume that we have two behavioural models f.g: - II and a distribution D over R. that R' is sampled from D, and that the learning algorithm L observes the policy π g(R*). We then assume that L returns the reward function R2 such that fRR) that L seleets among all such reward functions using some (potentially nondeterministic) inductive bias. We then want to know if dR RR* RA)) S with probability at least 1 5, for some 8 and c.
and
Our framework can, to an extent, be used to study this setting as well. In particular, suppose we pick a set R of "likely" reward functions such that TP PR-DD(P (R E R) 2 - 5, and such that the learning algorithm L will return reward function Rs E R. if there exists a reward function RA E R such that ((R)) y( R²). Then if f is e-robust to misspecification with g on R, we have that L will learn a reward function RG such that dR (R', Ra)) S with probability at least 1 8.
So, for example, suppose R is the set of all reward functions that have "low complexity". for some complexity measure and complexity threshold. The above argument then informally tells that if the true reward function is likely to have low complexity, and if C will attempt to fit a complexxt reward function to its training data, then the learnt reward function will be close to the true reward
14

---

Published as a conference paper at ICLR 2024
function with high probability,, long as is e-robust to misspecification with g the set of all low-complexity reward functions.
Thus, while Definition I gives worst-case formalisation of misspecification robustness, it is relatively straightforward to carry out a more probabilistic analysis within the same framework.
B EXPLAINING AND MOTIVATING STARC-METRICS
In this section, we will explain Definition 2 and provide the theoretical justification for measuring the difference between reward functions using d:. dsTARC
Let us first walk through the definition dSTARC, and explain each the steps. Intuitively speak- ing, we want to consider R1 and Rg to be equivalent if (and only if) they induce the same ordering of policies. Moreover, also recall that R1 and R2 have the same ordering of policies and only if they differ by potential shaping, S'-redistributton, and positive linear scaling (see Proposition D.. For this reason, 4,7 single representative in their respective equivalence class, before measuring their difference.
dSTARC first standardises each reward function in a way that maps all equivalent rewards to
To do this, we first use STARC to map all rewards that differ by potential shaping and S"- redistribution to a single representative. Note that for all R. the set of all rewards that differ from R by potential shaping and S'-redistribution forms an affine subspace. This means that there is a well-defined "smallest" element of each such equivalence class, which is the reward function that STARC returns. It is also worth noting that STARC is an orthogonal linear transformation, that F,7 maps R to an |S (IA] -)-dimensional linear subspace of R.
Cr,y
After this, we normalise the resulting reward functions, by dividing them by their no-mr This collapses positive linear scaling, which now means that SSTRCC S7,7 (R1) ) ST.y SSTRRC (R2) if and only if
R1 and R2 have the same ordering of policies. We then measure the distance between the resulting
reward functions, and multiply this distance by 0.5 to ensure that the resulting value is between and 1. For more details, see Skalse et al. 2023).
To get an intuitive sense of how SSTARC behaves, first note that STARC is pseudometric on R. T.Y
Moreover, as we have already alluded to, di dSTARC (Ri, R2) 0 if and only if R& and R2 induce the same ordering of policies under 7 and y. In addition to this, we also have that STTARC (R1, R₂) = if and only if R2 and Rg induce the opposite ordering of policies under and 7. Furthermore, if Ro is trivial and Rt is non-trivial, then we have that dSTARC (R, Ro) = O.5. More generally, if R& and 7.7
&STARC gives each pair Ra are approximately orthogonal, then dSTARC (R1, R2) = 0.5. As such, d 7.7 of reward functions R1, R2 distance between 0 and where a distance close to 0 means that R and R2 have approximately the same policy order, distance close to 1 means that they have ap- proximately the opposite policy order, and a distance close 0.5 means that they are approximately orthogonal. Almost all reward functions have distance close to 0.5.
In addition to this, /STARC induces an upper bound on worst-case regret. Specifically: r,y
Definition 6. pseudometric d on R is sound there exists a positive constant U, such that for any reward functions R, and R2, if two policies and T2 satisfy that J2(TT) ) 2 J2(mi), then
JI (I(TT) J1 (12) < U (max (T()) min (1(T)) d(R1, R2).
Proposition
STARC sound. 7,"
5, see Skalse al. (2023). Before moving on, let us briefly unpack Definition 6 [ - (T22 is the regret, as measured by Ri, of using policy n2 instead of Z Division by max; J1 (T) min, Jı (T)) normalises this quantity to lie between 0 and 1 (though the term is put on the right-hand side of the inequality, instead of being used as a denominator, in order to avoid division by zero when Rj is trivial. The condition that J2(m2) Z (2() says that R2 prefers 22 over K₁. Taken together, this means that pseudometric d on R is sound if d(R,, R2) gives an upper bound on the (normalised) maximal regret that could be incurred under R2 if an arbitrary policy Ti is optimised to another policy πa according Rg. Note that we, special case, may assume that म, is optimal under R, and that Ta is optimal under Rg. Since T,7 sound, it induces such bound.
For a proof of Proposition
STARC is
15

---

Published as a conference paper at ICLR 2024
In addition this, dSTARC also induces lower bound worst-case regret. It may not be immedi- ately obvious why this property is desirable. To see why this is the case, note that if pseudometric d on R does not induce lower bound on worst-case regret, then there are reward functions that have a low regret, but large distance under d. This would in turn mean that d is not tight, and that it should be possible to find better way to measure the distance between reward functions. If pseudometric induces a lower bound on regret, then these kinds of cases are ruled out. When a pseudometric has this property, we say that it is complete:
Definition 7. A pseudometric d on R is complete if there exists positive constant I,, such that for any reward functions Rt and Rz, there exists two policies _1 and I2 such that (2()) 2 J2(mI) and
Ji (I( J1 (x2) > L (max J1 (T) min J/ (T) d(R,, R2),
and moreover, if R1 and R2 have the same policy order then d' R1 R2) 0.
Proposition 6. dSTARC is complete.
T,7
Note that if R1 and R2 have the same policy order and max, (1( min, J1(x) > 0. then d(R1, R2) = 0; the last condition ensures that this also holds when max,, J1 (TT) min, J1 (T) = 0. Intuitively, if d is sound, then a small d is sufficient for low regret, and if d is complete, then a small d is necessary for low regret. Soundness implies the absence of false positives, and completeness the absence of false negatives. As per Proposition we have that dSTARC is complete, and hence tight. For a proof, see (Skalse al. (2023).
7,7
Moreover, if a pseudometric is both sound and complete, then this implies that it, in a certain sense, unique. Specifically:
Proposition 7. If two pseudometrics d1, d2 on R are both sound and complete, then dj and d2 are bilipschitz equivalent.
For proof, see Skalse al. (2023). Note that this means that any pseudometric on R that is both sound and complete must be bilipschitz equivalent to @STARC, As such, d dSTARC is a canonical pseu- Tn
dometric on R. in the sense that small aSTARC -distance is both necessary and sufficient for low worst-case regret, and that any other pseudometric on R with this property also must be equivalent to 7.77 dSTARC, Therefore, we think it is justified to regard dSTARC as the "right" way to quantify the difference between reward functions.
d. dSTARRC-dstance
Recent literature has proposed other pseudometrics for quantifying the difference between reward functions, namely EPIC (Gleave et al. 202T) and DARD (Wulfe et al., 2022). However, these do not enjoy the same strong theoretical guarantees as df dSTARC, In particular, they are neither sound nor complete in the sense of @STARC, For more details, see Skalse al. 2023).
7,7
C WHY NOT USE EPIC?
Many of results are invariant to the choice of pseudometric on R. but when we do have to pick a particular metric, we use d Another prominent pseudometric on R is EPIC, which was first proposed by Gleave et al. (022I) and has since become the most widely used pseudometric on R (as judged by the number of citations at the time of writing). So why are we not using EPIC in this paper? There is a simple reason for this, namely that EPIC is sensitive to S'-redistribution. Specifically, for any reward function R and any 5 E (0, 1] there exists two reward functions R1, R2 such that R. Ri, and R2 differ by S'-redistribution, but such that the EPIC-distance between Ri and R2 is 1 - 8. In other words, starting from arbitrary reward function It and using only S'-redistribution, we can find reward functions whose EPIC-distance is arbitrarily close to I.
This is problematic, because essentially any behavioural model of interest is invariant to S'- redistribution (including 0, by,y,3, and C,,,a). This means that any such model will violate con- dition in Definition [ for all e < 1 when d is the EPIC pseudometric. Moreover, this also means that if f is e-robust to misspecification to S' s'-redistribution, then it must be the case that e 2 0.5 (c.f. Lemma I}. Since an EPIC-distance of is too
with g (as defined by the EPIC distance), and g is invariant
0.5 is very large, such results are essentially vacuous. In other words, the EPIC-pseudomeed loose, and cannot be used to derive any non-trivial results within the setting that we are concerned with in this paper.
16

---

Published as a conference paper at ICLR 2024
In addition to this, dSTARC also yields stronger theoretical guarantees than EPIC; Appendix B and Skalse et al. 2023).
D WHY ARE CONTINUOUS MODELS NoT ROBUST TO PERTURBATIONS?
In this section, we give more in-depth interpretation and explanation of Theorem S
Intuitively speaking, the fundamental reason that Theorem holds because there a mismatch between e di-istance and STARC-distance In particular, if continuous, then it must send reward functions that are close under the norm to policies that are close under the e -orm. However, there reward functions that are close under the (e-norm but have large STARC distance. Hence, f is continuous then it will send some reward functions that are far apart under dSTARC (but close under ( to policies which are close (under e²), which in turn means that is not c/ e/ō-separating.
7,7
To see this, R be an arbitrary non-trivial reward function, and let be any positive constant. We then have that R and -e R have the opposite policy ordering, which means that d dSTARC R, - R)- 1. However, by making small enough, we can ensure that the fedistaacc between c. R and - is arbitrarily small, and hence that d (c. R, - R) < Thus, we have two reward functions that have large STARC-distance that are sent to policies that are close.
This example is not too concerning by itself, because only demonstrates that we may run into trouble for reward functions that very close 0, and we may expect such reward functions be unlikely (both the sense that the observed agent unlikely to have such a reward function, and in the sense that the inductive bias of the learning algorithm unlikely to generate such hypothesis). would therefore be natural to restrict R in some way, for example by imposing a minimum size on the f--oomm of all considered reward functions, or by supposing that they are normalised. However, Theorem 3 tells us that this will not work either: as long as there some positive constant e such that R||2 e then E R. then we can always find reward functions R, Rg such that their (2 (i-distance is small but (TARCC (RR, R2) is large. Theorem thus applies very widely. 7,7
E WHY Is IRL SENSITIVE TO MISSPECIFIED PARAMETERS?
In this section, we give more in-depth explanation of Theorems and
To start with, the reason that Theorem 4 is true is that we for any reward function R1 can find a reward function Rz such that R, and R2 differ by potential shaping with ?1, but such that R and Rg have different policy ordering under "22 (when "1 * 72). To see this, consider a simple environment with three states Su, $1, $2, where Sp is the initial state, and where the agent can choose to either go directly from Su $2, choose to first visit state $1:
82
20
7/1 x
start
80
Let RA be any reward function over this environment, and let R be the reward function that we get if we take Rí and increase the reward of going from so 81 by n s,, and decrease the reward of
17

---

Published as a conference paper at ICLR 2024
going from $1 to $2 by z. Now, the policy order under discounting with V is completely unchanged. At $1, the value of every action is changed by the same amount, and so there is no reason to change action. Similarly, at sp, the value of going to 8 is changed by Y1 x "i x O. and there is likewise no reason to change action. This transformation corresponds to potential shaping where ((s1) and (sp)) (((2) = 0. Therefore, if f R IT is invariant to potential shaping with 71, then (R1) = f(R2).
However, if we discount with 72, then Ri and Rg have a different policy order. In particular, the value of going from to 81 is changed by 71 I 722 I (71 72) x o 0. Thus, if the optimal action under R1 at sq is to go to $1, then by making x sufficiently large or sufficiently small (depending on whether Y/1 > Y2, or vice versa), then we can create reward function Rg for which the optimal action instead to go to 82, and vice versa.
Thus, in this environment, for every reward function R1 and every 71, 72 such that 71 * 72, we can find a reward function R2 such that R1 and R2 differ by potential shaping with 7/1 but such that they have a different ordering of policies when we discount with 7/2 This in turn means that we cannot be robust to misspecification of y; if the observed policy is computed using 72, then there are reward functions that would lead to the same observed policy (and which hence cannot be distinguished by the learning process) but which nonetheless are a large distance from each other as evaluated by ?i- This issue is present long 71 # 72. and so the degree of misspecification does not matter.
This is the basic mechanism behind Theorem although this theorem additionally shows that the dynamic which describe above shows up for any non-trivial transition function. Intuitively speak- ing, we can use potential shaping to move reward around in the MDP (so that the agent receives a larger immediate reward at the cost of a lower reward later, or vice versa). However, because of the discounting, later rewards must be made larger than immediate rewards. If the discount values do not match, then this "compensation" will also not match, leading to distortion of the policy ordering. Indeed, we can make it so that this distortion dominates the rest of the reward function. For the full details, see the proof of Theorem 4
As for Theorem 4 this theorem is similarly true because we for any reward function Ry can find a reward function R2 such that R1 and R2 differ by S' s'rreiisttibution with Tı, but such that R₁ and R2 have different policy ordering under 72 (when
# T2). To see this, suppose we have an MDP with (at least) three states 8p, 81, 82, and that taking action a in state sg under transition function TI takes you to state 81 with probability p, and 82 with probability 1 p. Similarly, taking action a in state so under transition function 72 takes you to state $1 with probability and $2 with probability 1 q, where p # q.
81
$2
p. q
1 1 p. 1 -
start
80
Let R2 be any reward function, and X any real number. Now note that we, regardless of the values of Rí and X, can find values of R2 a, 81) and R2(s0, a, 82) such that
p- R2(50, a, 81) + (1 p) RR a, 82) p. Rí (50, a, 81) + (1-p) p) Rí (30, a, 82)
9) R2(s0, a, 52) X. Note that this means that Eg(rr(()),a) (s a, S')] = Es (s. RR (s0, a, S')], and that Egtwr)(*) [22 a, S')] = X. Note also that X was selected arbitrarily. In other words, the fact that Ri and R2 differ by S'-redistribution under TI. leaves the expectation pletely unconstrained for all transitions where TI + 72. If TI + 72 for all states, then the policy order of R₂ under T2 can be literally any possible policy ordering. This in turn means that we cannot be robust to misspecification of 7; if the observed policy is computed using T2, then there are reward functions that would lead to the same observed policy (and which hence cannot be distinguished by the learning process) but which nonetheless have arbitrarily large distance under TI.
and such that q- R2(50, a, s1) + (1 g)
Ri under 72 com-
18

---

Published as a conference paper at ICLR 2024
It is also important to note that Theorem 4 does not require that T1 # To for all states; indeed, it enough for them to differ at just a single transition 8, a. Using the same strategy as above, we can find two reward functions R and R2 such that R, and R2 differ by S' sr-redistributton under Ti, but such that under 72, the value a given policy under Ri depends primarily on visiting 8, a as many times possible, but the value of under R2 depends primarily visiting 8, as few times as possible. For the full details, see the proof of Theorem 4
We can also give somewhat less artificial example, make this point more intuitive. Consider a simple N x N gridworld environment. We assume that the agent has four actions, down, left, and right. We assume that 71 is deterministic, so that if the agent takes action up, then moves one step up, etc. Moreover, we assume that is slippery, so that if the agent takes action up, then it moves up, up-left, and up-right with equal probability, and that it takes action r right, then it moves right, up-right, and down-right with equal probability, etc. For simplicity, we will also assume that the environment has "PacMan-like" border, so that the agent moves up from the top of the environment, then it ends up at the bottom, etc
Now suppose that R₁ and R2 reward each transition depending on how the agent moves, according to the following schemas:
-1
-3
3
1
-1
1
-1
R1
R₂
These two reward functions are identical under 72, and give the agent reward for going right, -1 for going left, and () for going up or down. However, under they are opposites; Rı rewards the agent for going right, and Rg rewards the agent for going left. Thus, we observe policy computed under 72, then we will not be able to distinguish between Ri and Rg, even though they have large distance under Tı. Similar issues will occur given any discrepancy between T and T2-
"In other words, the environment is shaped like torus.
19

---

Published as a conference paper at ICLR 2024
F PROOFS
Here, we will provide the proofs of all our theorems and other theoretical results. The proofs split up in three sections, mirroring the three subsections in Section
F.1 NECESSARY AND SUFFICIENT CONDITIONS
Theorem 1. Let R. be set of reward functions, let : II be behavioural model, and let dR be pseudometric
R. Suppose that f ((R1) f(R2) = dR (B1, RR2) 0 for all R1, R2 E R. Then f is e-robust to misspecification with g (as defined by d if and only if g = f for some t: R + such that di (R, t(R)) I for all R E R, and such that f + g.
Proof. For the first direction, let L R + R. be transformation such that dh (R, ((R) S e for all Re E R. and let g = f fot t. To show that f is e-robust to misspecification with g. we need to show that:
1. For all R1, R2 ((R)) = (R2)
then d² (R1, R2) S
2. For all R1, Rg E R, if ((1) = f(22) then da RR,, R2) S <.
3. For all R1 E R there exists an Rg E R such that f(22) = ((11).
4. There exists R1, R2 E R such that f(R1) + 9(R2).
For the first condition, suppose f(R,) = g Rg,, which implies that [R)) = fo ((2)). By assumption, have that if (R) f(R'), then d (R. R') This implies that d ( (R1, t(2)) 0. Moreover, we have that d (R, ((R) < E for all R; this implies that d (R2, ((R)) S e. By the triangle inequality, we then have that d (Ri, (R) < 2++e-
Since Rí and R2 were chosen arbitrarily, this means that condition holds. For condition 2. note that we by assumption have that (R, f(R2). then dR (R1, R2) = O. Since < this implies that condition 2 holds. For condition 3, let Ri be any reward function, and let R₂ = t(R,). Now f(2) = g(R,) Since R was chosen arbitrarily, this means that condition assumption. We have thus shown that if = f for some & R > R such that (R, t(R) < e for all E R and such that f * g. then f is e-obust to misspecification with (as defined by dRR.
satisfied. Condition 4 is satisfied by direct
For the other direction, let f be c-robust to misspecification with g (as defined by dR²). For each y E Im(g), let Ry E R be some reward function such that ((R,) = yy since Im(g) S. &Im()), such an Ry E R always exists. Now let t be the function that maps each E R to Rg(n). Since by construction g( ()) [( (Rg(R)),, and since is e-robust to misspecification with g R, we have that d² (R, Rg) S Since by construction t(R) Rgt R) this means that dº (R, (R)) S Thus : R + R. satisfies the condition that dR (R, (R)) S c. Moreover, since f is c-robust to misspecification with g, we have that f A g. Finally, note that g = f tt. This completes the proof of the other direction, which means that we are done.
Proposition 3. A transformation t R + R satisfies that STARC di,7 (RRRRCCR,REEE (R)((R) for all R R if and only if can be expressed as t1 o ... tt-1 o tn tn+1 o... tm for some n and m where
|| (R) < || (TARR sin(2 aresin(e/2))
for all R, and for all i # and all R, we have that R and (2(R) differ by potential shaping (with 7), S' -reedistribution (with T) or positive linear scaling.
Proof. For the first direction, suppose STARC (R, t(R) S c for all R E R, and let R be an Try arbitrarily selected reward function. We will show that it is possible to navigate from to t(R) using the described transformations.
20

---

Published as a conference paper at ICLR 2024
Recall that 7,7 (STARO(R1, R2) is computed by first applying STARC to both R₁ and R2, then normal- T,y
ising the resulting vectors, and finally measuring their e disttance. This means that SSTARC (R) and SSTARC (t(R)) can be placed in the following diagram, where S c: ST,y
5.7
STARC (R)
T:7
STARC (t(R)) T,7
Now, elementary trigonometry tells us that 0 = 2 arcsii(é'/2)) Moreover, suppose we extend .STARC (R) make the triangle right triangle, follows: 57,7
0
Here elementary trigonometry again tells us that x/(1 + y) = sin arcsin(é//2)) or that x (1 + y) sin aresin(e' (2)).. This means that can go from R to t(R) as follows:
STARC, Since R and STARC (R) differ by potential shaping and S'-redistribution, 1. Apply c this transformation can be expressed as combination of potential shaping and S'- redistribution. Call the resulting vector R'.
c.,
2. Normalise R', that its magnitude is 1. This transformation is an instance of positive linear scaling. Call the resulting vector R"
3. Scale R until it forms right triangle with STARC This transformation is an s instance of positive linear scaling. Call the resulting vector R"
21

---

Published as a conference paper at ICLR 2024
4. Move from R" (t(R)) This will move R" by (1 + y) sin(2 aresin(e' /2)), $1,7
where (1 + y) || Moreover, since R" in the image of Tn eTARC; we have that R" = SSTARC CTTARC(R"))) ), and so || = || [R"))l2- This means that R" is
7:7
moved by || CT.7
STARC (R") sin(2 (rosin(c (2))))(((//))). Since c' S < S n/2, this means that RR STARC (t(R)))ll S T (RARC(R"")) sin arcsin(e/2)).
5. Move from 8
STARC (I(R)) to 7,7
STARC (I(R)) is simply normalised STARC (I)).. Since 87,7
version of STARC (t(It),, this is an instance of positive linear scaling.
7,7
6. Move from STARC (t))) to {RR). Since (R) and TTY (TARC(((R)) 7.77
(t(R)) differ by potential shap- ing and s'-redistribution, this transformation can be expressed as combination of potential shaping and 5" -'drriiitribttinn.
Thus, for an arbitrary reward function R we can find a series of transformations that fit the given description. This completes the first direction.
For the other direction, suppose can be expressed as t .... t--1 o tn o tx+1 .... by where
R - (R)lle S STARC (R)ll sin(2 arcsin(e/2))
for all R, and for all o and all R. have that R and (2(R)) differ by potential shaping (with 7), s'-redistribution (with 7), or positive linear scaling.
7/7 Recall that dSTARC is invariant potential shaping (with 7). Srrrediitriiiation (with r), and positive
linear scaling: this means that (STARC (R, (Rt)) = for + n.
7.7
For fa recall that STARC is a linear orthogonal projection; this means that ||e G,7
STARC (R))
STARC [R2)112 < [|/1 R2ll2. As such, if R - Ln (/t)ll2 < [|CSTARC(RR))) (R))22 sin arcsin(e/2)) T.77
(t., (R)) I ||7,71 [|cSSARC(())] (alllsssin222 arcsin(c/2)) well. Consider the then | (R) C,7
-STARCC (t,,
following diagram:
STARC (R) 8..7
[GSTARR (R) la aresin(e/2)) (2)]
Now STARC STARC (R,,()) is located within circle in the diagram above. The vector within this circle that C,yy
maximises the distance .TARC (R)) after normalisation lies the tangent of the circle: c
22

---

Published as a conference paper at ICLR 2024
SSTARC
[1 (R)] ain(2arcsin((/22
Elementary trigonometry now tells us that
||
STARO (R))l sin(2 arcsin(c/2))
si
[ CSTTAC (R)ll2
which gives that arcsin(e/2)) From this, we have that ô = e, and so STARC (R, t(R)) S c. This completes the other direction, and hence the proof.
L
Corollary 1. Let R be a set of reward functions, T be a transition function, 7 discount factor, a lemperature parameter, and a a weight parameter. Moreover, let T. be the set of all functions t R + R. that satisfy Proposition and udditionally satisfy that U(R) E R for all E R. dSTARC) if and only if Then br,7,9 : R + is c-robust to misspecification with (as defined by T.7
g br.,.9 o for some t Te, and Cryaa R + II is e-robust to misspecification with g (as defined by (STARC) if and only if = Cr,ra o t for some t Te.
Proof. Immediate from Theorem D and Proposition
Proposition 4. Unless |S| 1 and |A| 2, then for any and any "i there exists E 0 such that for all < E, there is no behavioural model g such that 07,7 is e-robust to misspecification with g (as defined by de dSTARC).
Proof. We will first show that unless [S] and IAl 2, there exists reward functions R1, R and F. such that or,, (R,) or,, (R2), but d STARC (R1, R22) F. In particular, note that |S| [A| > then there exists uncountably many reward functions that do have the same ordering of policies. Moreover, also note that Im(or,)) is finite. By the pigeonhole principle, this means that there must exist a policy क E Im(or,r) and reward functions R2 Rg such that or,, (R,) (R,)(R) R,, and such that R3 and R2 do not have the same ordering of policies. Moreover, recall that .STARC/ (R), R2) = 0 if and only if R₁ and R2 have the same ordering of policies. Thus, unless 7,
SS) = and |A| 2, there exists reward functions R1, R2 such that (R,) = or,, but
SSTARC (R1 R₂) = 0. Thus Or violates condition 2 of Definition for all < E. 7.
n
F.2 PERTURBATION ROBUSTNESS
Theorem 2. Let R be a set of reward functions. let f R - II a behavioural model, let dR. be a pseudometric on R. and let all be pseudometric on TI. Then I is e-robust 5-perturbation (as defined by dR and dll) if and only if is f
(as defined by dR and d
Proof. For the first direction, suppose f is e/ e/6-separating, and let be a 8-perturbation f with Im(g) C Im(f). We will show that f and g satisfy the conditions of Definition 1 For the first
23

---

Published as a conference paper at ICLR 2024
condition, let R1, Rg be two arbitrary reward functions in R such that f(R)) = g(R2). Since 9 is a o-perturbation of f. we have that d (g( R2), f f(2)) S 8. Since f(Ri) = g R2), straightforward substitution thus gives us that d' (f((R)) f(R2)) S 8. Since f is e/6-spaarating, this means that d (Ri, R2) I 6. Since R1 and R2 were chosen arbitrarily, this means that if f(R)) R( 6. Thus, the first condition of Definition holds. For the second condition, note that if f(R)) = ((2)), then (f f(R)) = 0 S Since f is e/ 6-separating, this means that dº (R1, R2) S E, which means that the second condition satisfied as well. The third condition is satisfied, since we assume that [mgg) S Im(f), and the fourth condition is satisfied by the definition of 6-perturbations. This means that f and g satisfy all the conditions of Definition []] and thus f is e-robust to misspecification with g. Since y was chosen arbitrarily, this means that / is e-robust to misspecification with any 8-p 5-perturbation g such that Im(g)) S Im(s). Thus ʃ is e- e-robust to 5-perturbation.
then (R1, R2)
For the second direction, suppose f is not e/ //-sepaaraiing This means that there exist R1, Ro E R and (f
such that dR (R1, R2) model where g R1) = (R2), g (R2) = f(R)). and g( = (() for all R E [R,, R2]. Now is 6-perturbation of f. However, f is not e-robust misspecification with g, since g(R)) = (R2),
(R2)) I 8. Now let g R R. be the behavioural
but dd (R1, R2) > Thus, if f is not e/ //ô-separating then f is not e-robust to 6-perturbation, which turn means that f is c-robust to 6-perturbation, then f must be e/8--/s-eeeeee sepaarrtn.
Theorem 3. Let dR be dSTARC, and let d be a pseudometric on II which satisfies the condition T,,7 that for all 61 there exists 52 such that if [| Tzll2 < 62 then d (T1,, (2)) < 61. Let be any positive constant, and let R be a set of reward functions such that if || = c then R E R. Let
f: R + II be continuous. Then f is not e/--separating for any < 1 or 8 > O.
Proof: Let R be non-trivial reward function that is orthogonal to all trivial reward functions. Since the set of all trivial reward functions form a linear subspace, such a reward function R exists. Note that must not necessarily be contained in R.
We now have that for any positive constant it is the case that eR and - have the opposite T.7
ordering of policies, and thus dSTARC (ERR, -eft) = 1. Next, let Rg be some potential-shaping
reward function such that ||eR+ + R& ||2 = c. Since potential shaping does not change the ordering of policies, we have that elt R& and el + R& must have the opposite ordering of policies as well, and so &STARC (eR Rg. +cR+ R&²) = 1. Moreover, since Rg is trivial we have that both eR and -RR T:7
are orthogonal to Rg. and so || - eR + R = c as well. Since ||eR + l2 = ||R + Ra l|2 = c, we have that both el + Rs and -eR + R& are contained in R.
Let oi be any positive constant. By assumption, there exists a d2 such that if [|]] T2 < 82 then d (IT,,, T2) < 81- Moreover, since f is continuous, there exists an 61 such that if [R R2 l2 < 61, then [[ - f(R))ll < 62. Next, note that by making sufficiently small, we can ensure that the -ddistance between eR + R4, and elt + R& is arbitrarily small (and, in particular, less than e1).
Thus, for any positive å there exist reward functions elt Rs and -eR+ Rs that are both contained in R, such that a (f(eR R4). f(-efffRRff eR R&)) 8. and such that d 1. Thus f is not e/0-separating for any 8 > 0 and any e < 1.
@STARC (ER.+ RĄ. -RR+ Rx+)
F.3 MISSPECIFIED PARAMETERS
Lemma 1. Let R be set of reward functions, let f.g R + II be two behavioural models, and let d be a pseudometric on R. Suppose I is e-robust to misspecification with g (as defined by dR ).
Then if (RR) = g( R2), we have that d (R1, R2) S 2c.
Proof. Let f.g: R + R be two behavioural models, and suppose f is e-robust to misspecification
with g. Let R1, R2 E R be any two reward functions such that g (R) = g( RRz). From condition 3 in
Definition I we have that there must be a reward function R3 such that f (Rs) g ( 9 (R2). From condition I in Definition] [ we have that d (Ra, R₁) S e and dR RR.. R2) S e. The triangle inequality then implies that d (Ri, R2) S 2
24

---

Published as a conference paper at ICLR 2024
Lemma 2. If fy R - Il is invariant potential shaping with %, then for all and all 11, 72 such that "Y o 722 and is non-trivial, then there exists reward function R such that frx (R) (R
f + a for all R E R and all a E R.
Proof. Analogous to the proof of Lemma A.18 in Skalse & Abate 2023
Lemma 3. If fr : R - II is invariant to S' -'-rriistribution with T, then for all ry and all Ti, such that + T2, there exists reward function R that is non-trivial under T2 and %, such that Jr. (R) = Jn (R + all) ) for all R E R and all a E R.
Proof. Since fn is invariant S' sr-redistribution with TI, we have that fr (R1) Sr. (R2) for any two reward functions R1, R2 such that
Eg (s.a) [Ri Sa)] (s,aa)(sa)) (s2 S')
Note that R2 and R2 satisfy this condition if and only if
(s.a) [ - R1)(s, a, =')]=0.
That is to say, if R' is reward function such that (s [R'(s, a, S")] O, then fn (R) fr, (R + R.) for all R. Next, note that the set of all such reward functions R' form a linear subspace of R. with {S] [S|S)][[[S] - 1) dimensions. We will show that this subspace contains reward functions that are non-trivial under y and T2.
Since TI # T2, we have that there exists a state 8 and action a such that TI a) # T2(s, a)- Let Rt be reward function that is O everywhere, except that Eg (s(s, S")) O, and Essrra(s,)) (s, tem of linear equations. In particular, the values of Ri as,a,,s) for each transition s, a, s form a |s)-dimensional vector space. The set of all values for these variables that satisfy (s,))) [f''(s, a, 8')] 0 form an (IS) 1)-dimensional linear subspace, and the set of all values that satisfy Es (s"'s,, S')] = form an ([8) - 1)-dimensional affine subspace. These two sets must intersect, unless they are parallel. However, since Σ, P( (s. a) = s') 1, they cannot be parallel. Thus, such reward function R must exist.
1. Note that there is always solution to this sys-
P() =') 8')
It is clear that Rt is non-trivial under y and 72. To spell it out; since all states are reachable under T2 and Ho. there exists a policy æ that visits state with positive probability. Let т and T2 be two policies that are identical to everywhere, except that Ti takes action with probability 1 in state 8, and T2 takes action a with probability 0 in state 8. Then (mx) > (xz).. Moreover, since Estan (s.a) [R (s, a, S')] = 0 for all and a, we have that R and R+ differ by S' s'-redistribution (with TI) for all reward functions R E R and all scalars a E R. Thus fr, (R) fr, (R + aR
Thus, if fr R + 11 invariant to S'-redistribution with t, then for all y and all Tı, 72 such that TI * 72, there exists a reward function Rt that is non-trivial under 72 and y, such that fr, (R) fr. (R+ + aRt) for all R E R and all a E R.
Theorem 4. If fr R I1 is invariant potential shaping with n, and 7/1 # *2 then fm. is not e-robust to misspecification with frz under SSTARC for any non-trivial T, any Ya. and any < 0.5.
6,77
Proof. If m # 72, then either 71 * 73 or 72 2-
If 'n * 73. then Lemma 2 implies that there exists reward function Rt that is non-trivial under T and 7a. such that fyy (R) foy (R + RRt) for all R E R and all a E TR This means that
Ja (Rt)) = fm (-Rt) and STAACC (Rt - = 1. Thus Sy, violates condition 2 of Definition for all 1.
7,78
If 7/ # 7a. then Lemma 2 implies that there exists a reward function R that is non-trivial under
7 and 73, such that fyp (R) = fyz (R + art) for all R E R and all R. This means that 1. Then Lemma implies that there can be no that is r-bbustt to misspecification with fo2 (as defined by SSTAC for any 6 < 0.5.
fr (Rl) (-R)) and dSTARC (Rt--R))
7,7a
Theorem 5. fr= : R + II is invariant to S' rredistribbuton with T, and T * T2, then fr. is not 6-robust to misspecification with J, under d STARC for any T3, any 7, and any < 0.5.
Tsin
25

---

Published as a conference paper at ICLR 2024
Proof. If # T2. then either + 73 72 + T3-
If Ti * T3, then Lemma implies that there exists a reward function R that is non-trivial under T3 and % such that fr. (R) for (R + a for all R R and all E R. This means that
fr. (RI) = fr. (-R') RR)) and ASTATIC (R), -R' = Thus fr, violates condition of Definition for
73.7
all <.
If T2 * Ts. then Lemma implies that there exists a reward function R that is non-trivial under for R aRt) for all R R and all E R. This means that Jr, fra( (-) and @STAR (R - RR) = 1. Then Lemma implies that there can be no f that is e-robust to misspecification with fra (as defined by dSTARC) for any 0.5.
Ta and % such that fr
ra.7
13r7
G CONNECTING OUR ANALYSIS TO EARLIER PROPOSALS
In this section, will explain how to connect the results of Skalse & Abate (2023) to our results rigorous way. Skalse & Abate 2023) assume that we have a partition on R, which of course corresponds to an equivalence relation Ep, and say that two reward functions R1. Rg should be considered to be "close" if R1 =p R2. Like us, they consider functions f.g R + II that take a reward function and return policy. They then say that f is "P-robust to misspecification with g" if each of the following conditions hold:
1. f(Ra) R22) - R1 Er R2.
2. f(Ri) 2(R)) R1 = R2.
3. For all R1 there exists an Rg such that f(R)) 9
4. f +
Note that this definition is analogous to Definition [ except that an equivalence relation P plays the role that pseudometric d does in framework. Next, note that for any pseudometric d can define an equivalence relation p such that Rg =p R2 if and only if dR RR, R2) = 0. In that case, would have that f is P-robust to misspecification with g (in the terminology of [Skalse & Abate 2023) if and only if f is O-robust to misspecification with (as evaluated by d in our terminology (i.e. Definition D. Moreover, if f is O-robust to misspecification with g. then it of course follows that f is c-robust to misspecification with for all 2 O. In this way. their results can be expressed within our more general framework.
Next, also recall that dSTARC (R1, R₂) = 0 if and only if Ri and R2 induce the same ordering of dT. policies (under and 7). Skalse & Abate (203) use "ORD denote this equivalence relation.
Thus, if is ORD-M to misspecification with g (as defined by SSkals & Abate, 2023) then I is r-bbust to misspecification with (as evaluated by d
aSTARC) for all c 2 0.
26


Reviews:
[{"review_id": "o5olaQj2Ij", "reviewer": "Reviewer_5VPU", "summary": "The authors investigate the question of robustness of an IRL algorithm against reward misspecification, i.e. if the expert follows a behavioural model (which computes $\\pi$ from $r$) and an IRL algorithm assumes another behavioural model, when will the inferred reward be close to the true reward? To understand this, the authors use the STARC metric [1] to define closeness between two given rewards. The authors then identify necessary & sufficient conditions for robustness to hold when misspecification exists, and further discuss parameter misspecification, continuity of the behavioural model, etc. Overall, the paper is well written and provides useful insights fundamental to IRL, but is still less clear how these insights translate to practical rules for IRL practitioners.", "questions": "1. What if we have two IRL algorithms that differ in implementation but assume the same underlying behavioural model? Will this affect the results?\n2. Does the IRL reward parameterization affect the results?\n3. Are there more ways in which reward functions can be different but yield the same optimal policy? (except for potential shaping, S' redistribution, positive linear scaling)\n4. Shouldn't a reward change of the type $R'=R+b$ where $b$ is a constant also lead to the same optimal policy? Is this discussed somewhere (may have missed it)?\n5. Can these results be used to quantify unidentifiability better? For example, unidentifiabilty in RL/IRL states that there may exist multiple rewards that yield the same policy, so given a policy, an IRL algorithm can output multiple valid reward functions. But what is the size of a class of rewards that are equivalent in some way? What is this size affected by?\n\n**Typos**\n- Page 2, Paragraph 2, Line 6: \"misspecifiaction\" => \"misspecification\"\n- Section 3 title: \"rubustness\" => \"robustness\"\n\n**References**\n1. Starc: A general framework for quantifying differences between reward functions, Skalse et al. (2023)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Lots of good explanations, especially in the appendix", "weaknesses": "Understanding the results in this paper takes multiple readings, and it is difficult to follow in a few places. This is due to choices that seem arbitrary, but are well explained later. For example, the choice of $sin(2arcsin(\\epsilon/2))$ in Proposition 3 seems arbitrary, but is justified in the Appendix. Overall, the writing is good, but can be improved by providing more intuition about the theorems/propositions in the main text, possibly through more examples, rather than relegating all explanations to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "am6RqI0VjD", "reviewer": "Reviewer_Bsty", "summary": "The authors analyze the sensitivity of inverse reinforcement learning, i.e., learning the reward model to misspecification of the behavior model adopted by the agent, i.e., the objective function that the agent is trying to optimize. The authors analyse robustness of IRL to misspecification in the behavior model by first quantifying the impact of misspecification of behavior model in terms of a pseudometric defined on the space of reward functions. Their first theorem states necessary and sufficient conditions for a behavior model to be $\\epsilon$ robust to a defined family of behavior models. The authors then state the necessary and sufficient conditions under which a behavioral model is robust to perturbations as their second theorem. Their final theorems state conditions under which behavioral models with perturbations to their parameters are not robust in terms of learning a reward function that preserves preferences.", "questions": "1. Can't the difference in observed versus predicted behavior of a human assuming a particular behavior model be explained by incorrect or incomplete optimization rather than incorrectness of the behavior model?\n2. Doesn't the IRL method also need access to the environment to collect samples?\n3. For a given dataset, can't two different behavioral models yield the same reward function? If so, isn't accuracy of reward prediction an insufficient criterion for estimating the validity of the behavioral model? In this paper, do the authors wish to learn a true reward function, albeit woith an incorrect behavior model, or do they wish to establish correctness of the behavior model as well?\n4. Are the brackets correctly places in the definition of S' redistribution in the third paragraph of Sec 1.2? If so, I do not understand this definition. Can the authors please clarify?\n5. In Point 3 of Defintion 1, should $R_1$ and $R_2$ be necessarily different? If so, why? What happens in the case when $f = g$?\n6. Is the orderting of policies mentioned in Proposition 1 a partial ordering?\n7. I did not understand the 2-norm of the first term in the equation written in Proposition 3. Can the authors please explain this?\n8. What happens to IRL robustness when the true behavior model is Cumulative Prospect Theory and the assumed one is Boltzmann-rationality?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors present interesting propositions that will help characterise the robustness of IRL algorithms towards learning preferences.", "weaknesses": "1. I am not sure about the extent of novelty in this contribution. The problem of robustness of IRL to behavior model misspecification has been studied in the past as mentioned in the literature survey in the paper. The new pseudometric used by the authors has also been introduced in an earlier paper cited by the authors. If this is correct, and the analysis is the main novel contribution, then I would expect a more detailed comparison of the analysis with the other works in literature.\n2. The main contribution of this paper is its theoretical results. The authors hhave stated the theorems, propositions and corollaries in the main paper without any proofs or even proof sketches. Furthermore, there are no numerical experiments conducted to demonstrate the impact of the theoretical findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sscBS5Sc94", "reviewer": "Reviewer_jn4g", "summary": "Inverse reinforcement learning aims to infer a reward function $R$ from a policy $\\pi$. To achieve this, IRL needs to make an assumption on the underlying behavior model, which maps reward functions to policies and characterizes the relation between $R$ and $\\pi$. However, the behavior model could be misspecified and differs from the true one. This paper is to contribute towards a theoretical understanding on the robustness of IRL to misspecification. First, this paper characterizes the sufficient and necessary condition to ensure that IRL is robust to a general misspecification. Then this paper considers a specific type of misspecification: the behavior model is a perturbation of the true one, and provides the corresponding sufficient and necessary condition. Finally, this paper considers another type of misspecification on the parameters of behavior models and provides a negative result that IRL is often not robust to such misspecification.", "questions": "Typos:\n1. In Proposition 3, $\\\\| R, t_n (R) \\\\|_2$ should be $\\\\| R - t_n (R) \\\\|_2$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper indeed makes notable progress toward theoretical understandings of misspecification in IRL. This paper conducts a systematic analysis and provides complete answers to this problem. The main results are insightful and clean and can help deepen the understanding of IRL.\n2. The paper is well-written and easy to follow. Although this paper is a theoretical paper, it provides many intuitive examples and explanations that can help readers easily understand the main results.", "weaknesses": "1. In this paper, the authors consider a general IRL procedure that outputs an arbitrary reward function $R$ from the class $\\tilde{\\mathcal{R}} = \\\\{R_1: f (R_1) = \\pi = g(R_2) \\\\}$. However, in practice, we often run a specific IRL algorithm such as MaxEnt IRL [Ziebart, 2010], which performs reward selection and outputs the chosen reward. Therefore, it could be more meaningful to analyze the robustness of a specific IRL algorithm to misspecification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3H8qOeiAS", "reviewer": "Reviewer_CDtC", "summary": "This paper presents a theoretical framework aimed at investigating the discrepancy or 'misspecification' between the assumed and actual behavioural models during the Inverse Reinforcement Learning (IRL) process. Moreover, it explores how this misspecification impacts the reward obtained through the IRL algorithm.\n\nThis research primarily concentrates on the most prevalent behavioural models, namely optimality, Boltzmann-rationality, and causal entropy maximisation.\n\nIn the initial segment of the theoretical framework, it introduces a formal definition of 'misspecification robustness' and the 'STARC-metric'. This metric is used to measure the differences between a reward function that remains invariant to potential shaping, redistribution, and possibly rescaling. Leveraging these definitions, the paper outlines the necessary and sufficient conditions for identifying the degree of robustness against misspecification between two behavioural models.\n\nThe second segment of the theoretical framework presumes a distance between the returns of behavioural models (policies). It examines the capability of the IRL algorithm to learn precise rewards under two conditions: 1) when the behavioural models are misspecified, and 2) when there is a gap in their returns. The findings suggest that no continuous behavioural model can maintain robustness against these two levels of misspecification.\n\nIn the final part of the theoretical framework, the paper delves into the scenarios where the parameters of behavioural models are misspecified. The research primarily focuses on the discount parameter, demonstrating that behavioural models are, in principle, highly sensitive to even minor misspecifications of two of their core parameters.", "questions": "1. In proposition 3, what's the meaning of $\\|R,t_n(R)\\|_2$? Should it be |R-t_n(R)\\|_2? \n\n2. According to the results of this paper, all the commonly applied behavior models are not $\\epsilon/\\delta$-separating, are there any behavior model is $\\epsilon/\\delta$, maybe under the mild relaxation on the distance under the reward and policy space?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The issue of misspecification between the assumed and actual behavioural models in IRL reresents a significant and underexplored challenge, to the best of the reviewer's knowledge.\n\n2. The paper is logically structured and easily navigable, with definitions clearly articulated and supplemented by explanations. These help the reader to intuitively comprehend the main theses and their implications in terms of characterizing the robustness of misspecification in the IRL problem.\n\n3. The theoretical framework is presented with clarity, and the main claims appear generally sound.", "weaknesses": "1. The inclusion of concrete examples of the transformation would be beneficial. Currently, it is challenging to image the types of transformations that would satisfy Proposition 3, and more importantly, how to interpret these transformations requires clarification.\n\n2. As per the reviewer's understanding, an MCE (Maximum Causal Entropy) policy corresponds to a Boltzmann-rational policy. Specifically, the optimal policy representation in soft-Q learning complies with Boltzmann-rationality. This point may warrant further verification.\n\n3. It would be beneficial to present some empirical results, which could illuminate the consequences of failing to achieve robustness under the misspecification of behavioural models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)