Dynamics-Aware Comparison of Learned Reward Functions
Authors: Blake Wulfe
Logan Michael Ellis
Jean Mercat
Rowan Thomas McAllister
Adrien Gaidon
Venue: ICLR 2022 Spotlight (2022)
Keywords: Reward Learning
Inverse Reinforcement Learning
Reinforcement Learning
Comparing Reward Functions

Abstract: The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world. However, $\textit{comparing}$ reward functions, for example as a means of evaluating reward learning methods, presents a challenge. Reward functions are typically compared by considering the behavior of optimized policies, but this approach conflates deficiencies in the reward function with those of the policy search algorithm used to optimize it. To address this challenge, Gleave et al. (2020) propose the Equivalent-Policy Invariant Comparison (EPIC) distance. EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics. This is problematic for learned reward functions because it entails evaluating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards. To address this problem, we propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric. DARD uses an approximate transition model of the environment to transform reward functions into a form that allows for comparisons that are invariant to reward shaping while only evaluating reward functions on transitions close to their training distribution. Experiments in simulated physical domains demonstrate that DARD enables reliable reward comparisons without policy optimization and is significantly more predictive than baseline methods of downstream policy performance when dealing with learned reward functions.

Published as a conference paper at ICLR 2022
KINAMICS-AWARE COMPARISON OF LEARNED REWARD FUNCTIONS
Blake Wulfe, Logan Ellis, Jean Mercat, Rowan McAllister, Adrien Gaidon Toyota Research Institute (TRI) (first. last @tri globll
Ashwin Balakrishna
University of California, Berkeley ashwin www.kkiihhateereeeelleeo edu
ABSTRACT
The ability to learn reward functions plays an important role enabling the de- ployment of intelligent agents in the real world. However, comparing reward fune- tions, for example as a means of evaluating reward learning methods, presents a challenge. Reward functions are typically compared by considering the behavior of optimized policies, but this approach conflates deficiencies in the reward func- tion with those the policy search algorithm used optimize it. To address this challenge, Gleave et al. 2020) propose the Equivalent-Policy Invariant Compar- ison (EPIC) distance. EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics. This problematic for learned reward functions because it entails eval- uating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards. To ad- dress this problem, propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric. DARD uses an approximate transition model of the environment transform reward functions into form that allows for compar- isons that are invariant to reward shaping while only evaluating reward functions transitions close to their training distribution. Experiments simulated physi- cal domains demonstrate that DARD enables reliable reward comparisons without policy optimization and is significantly more predictive than baseline methods of downstream policy performance when dealing with learned reward functions. [[
INTRODUCTION
Reward functions provide general way specify objectives for reinforcement learning. Tasks such as game playing have simple, natural reward functions that enable learning of effective poli- cies (Tesauro 1995] Mnih et al. 2013) Silver al. 2017); however, many tasks, such as those involving human interaction, multiple competing objectives, or high-dimensional state spaces, do not have easy to define reward functions (Knox al. 2021 Holland al. 2013 Cabi al. 2019). In these cases, reward functions can be learned from demonstrations, explicit preferences, or other forms of reward supervision (Ziebart al. (2008) Christiano et al. 2017; ; Cabi al. 2019). How- ever, evaluating the quality of learned reward functions, which may be too complicated to manually inspect, remains an open challenge.
Prior work on comparing reward functions typically involves a computationally expensive and indi- rect process of (i) learning a reward function, (ii) learning a policy on that reward function, and (iii) evaluating that policy
known, ground-truth reward function. Gleave et al. (2020) refer to this as "rollout method" for evaluating methods for learning reward functions. Furthermore, the rollout method can be potentially dangerous in safety critical environments because (i) it cannot distinguish deficiencics between the reward function and the policy optimization procedure, and (ii) rewards are
Videos available httphhttp///sttteee wttooole..com/ie///ddrr.aa

---

Published as a conference paper at ICLR 2022
evaluated on the distribution of transitions induced by an optimized policy, which may not capture all the transitions required to accurately quantify reward differences.
To address these challenges, Gleave al. (2020) introduce Equivalent-Poole Invariant Comparison (EPIC). EPIC quantifies differences in reward functions directly, without learning policies. EPIC uses offline dataset of transitions, and consists of two stages. First, transitions from this dataset are used to convert reward functions to a canonical form that is invariant to reward transformations that do not affect the optimal policy. Second, the correlation between reward values is computed on transition samples, yielding a pseudometric capturing reward function similarity. However, when computing this canonical form, EPIC evaluates rewards on transitions between all state-state com- binations, regardless of whether those state-state transitions are possible or not. While this makes possible for EPIC to quantify reward distances that are robust to arbitrary changes in transition dynamics, in practice evaluating learned reward functions on physically impossible transitions leads to unreliable reward values, as these are outside the distribution of the transitions on which the re- ward functions are learned. We show empirically in Section S that these out-of -istriiuitiitttonn reward evaluations can render EPIC an ineffective reward comparison metric.
To address this issue, we propose new method for quantifying differences in reward functions, Dwnamics-Aware Reward Distance (DARD), that transforms reward functions into a form allow- ing for soward-shaping invaiiant comparisons while ensuring that learned reward functions are only evaluated on transitions that are approximately physically realizable. We achieve this by assuming access to transition model for the environment of interest, and evaluate reward functions sitions sampled from this transition model. As a result, DARD only evaluates learned reward func- tions close their training distribution, and is well motivated in a variety of physical environments in which transition models are unlikely to change significantly between learning and deployment. Experiments suggest that DARD better reflects reward function similarity than does EPIC in com- paring learned reward functions in two simulated, physics-based environments: the first involving multi-agent navigation, and the second involving manipulating a robotic arm.
2 RELATED WORK
2.1 LEARNING REWARD FUNCTIONS
A variety of methods exist for learning reward functions, which can be categorized based on the form of supervision used. Rewards can be learned from supervision provided by explicit labels. For example, human labeled preferences between demonstrations can be used to learn reward func- tions (Christiano al. 2017). Rewards can also be labeled on per-timestep basis, allowing for learning via supervised regression (Cabi et al. 2019).
Alternatively, inverse reinforcement learning (IRL) methods typically assume access to expert demonstrations, and attempt to find a reward function under which the expert is uniquely opti- mal (Ziebart et al., 2008). IRL is underconstrained in that many reward functions can make the expert policy appear optimal. This problem is often addressed by favoring reward functions that induce high-entropy optimal policies (Ziebart et al., 2008). Recent work in IRL focuses on learning reward functions with expressive function approximators such as neural networks, particularly in an adversarial formulation (Fu al.] (2077). There has also been recent interest in offline IRL meth- ods (Lee et al. 2019). In this work, we study how these various reward learning methods can be evaluated efficiently and accurately in practice.
2.2 EVALUATING METHODS OF LEARNING REWARD FUNCTIONS
Reward functions are typically evaluated by first learning an optimal policy under the reward func- tion. The value of the induced policy is found by aggregating the returns from online trajectory rollouts under the known, ground-truth reward function. This requirement of online rollouts from experimental policies is often prohibitive in safety-critical domains such autonomous driving or decision making in healthcare (Thapa et al. 2005).
Off- Off-policy evaluation (OPE) methods attempt address these issues by estimating the value of policy in a purely offline setting by using previously collected data. Traditional OPE methods ad- dress policy mismatch by using form of importance sampling (Precup et al. 2000). This approach
2

---

Published as a conference paper at ICLR 2022
can be effective, but is unbiased only if the underlying expert policy is known, and can be subject to large variance. Direct OPE methods (Dudik et al. (201T)) apply regression- based techniques in either model- (Paduraru 2013) or model-free setting (Le et al., 2019). Contemporary works combine these two methods using Doubly Robust Estimation, which originated in a contextual ban- dits setting (Dudik al. 201T), and was later extended reinforcement learning by Piang & Li (2016). More recent works seek to forego models and importance sampling entirely, such as Irpan al. (2019), which recasts OPE positive-unlabeled classification problem with the Q-function as the decision function.
All OPE methods indirectly evaluate the reward by estimating the value of the induced optimal pol- icy. This introduces additional complexity into the evaluation procedure and can obfuscate sources of error as resulting either from aspects of policy learning or due to the underlying reward function. EPIC (Gleave et al., (020) seeks to circumvent these shortcomings by comparing reward functions directly. Our work builds on EPIC as covered in Section 4 but uses a transition model of the envi- ronment to significantly improve the accuracy of comparisons involving learned reward functions.
3 PRELIMINARIES
3.1 MARKOV DECISION PROCESSES
Markov decision process (MDP) is a tuple (S, A. T, R, 7 do) where S and A are the state and action spaces. The transition model 7 A
[0. 1] maps state and action probability distribution over next states, the reward R: S x A x S + IR measures the quality of a transition. Finally, 7 E [0, 1] and dg are respectively the discount factor and initial state distribution. A trajectory consists of sequence of state-action pairs: T = {(so, ao), ($1 e1),...} and the return g of trajectory
defined as the sum of discounted rewards along that trajectory: g(T)
Et=o 7 R(st, lt, $t-1). The goal in an MDP is to find policy æ : S x A - [0, 1] that maximizes the expected return, E (T)( where So dg. ar- ~ I (a:]st), and 8t+1 ~ ((s++1|8t at).
We denote distribution over a space S as Ds, and the set of distributions over a space S as A(S).
3.2 REWARD FUNCTION EQUIVALENCE
Our goal is to define a metric for comparing reward functions without learning policies. Neverthe- less, the measure of similarity we are interested in is the extent to which the optimal policies induced by two reward functions are the same. Because different reward functions can induce the same set of optimal policies, we in fact seek a pseudometric as formalized in the following definition.
Definition 3.1 (Pseudometric (Definition 3.5 from Gleave al. (2020))- Let X be set and d X x X + [0, oo] a function. d is a premetric if d(x,, x) = for all x E X. is a pseudometric if, furthermore, it is symmetric, d(x,, y) d(y, for all ar y X; and satisfies the triangle inequality, + d(y, for all x. y, z E X. d is a metric if. furthermore, for all x, E X, r =y- y.
d(x, =) < d(xx y) d(æ, y)
Which rewards induce the same set of optimal policies? Ng et al. (1999) showed that without additional prior knowledge about the MDP only reward functions that are related through a dif- ference in state potentials are equivalent. This "reward shaping" refers to an additive transfor- mation F applied to an initial reward function R to compute a shaped reward R': R'(s, a, s') R(s, a, s') + F(s, o, s'). Reward shaping is typically applied in reward design setting to produce denser reward functions that make policy optimization easier in practice (Ng et al. 2003 Schulman
et al. 2015). Ng et al. (1999) showed that if F is of the form F(s, a, 8') 7P(8') ((s) for
arbitrary state potential function (s). the set of optimal policies under R and R' are the same. This provides a definition for reward equivalence:
Definition 3.2 (Reward Equivalence, Definition 3.3 from Gleave et al. 2020 Bounded rewards RA and RB, are equivalent if and only if
YE > 0, and & S + R bounded s.I.
RB(s, a. s') ARA(s, s') 7255s) 2ss) Vs, S, A.
3

---

Published as a conference paper at ICLR 2022
This form of equivalence is only possible in reward functions that depend on (s, s'), which moti- vates our focus on these types of rewards in this paper. This choice further motivated by the fact that there are cases where conditioning example in highly stochastic MDPs, or in cases where rewards are a simple function of (s, a, s.), but not of (s, a). We next describe existing pseudometric that in theory is capable of identifying these reward equivalence classes.
the next state s' can make reward learning simpler, for
3.3
SINIVALENT-POLIICO INVARIANT COMPARISON (EPIC)
EPIC pseudometric for comparing reward functions and involves two stages. The first stage converts reward functions into a canonical form that removes the effect of reward shaping, and the second stage computes the correlation between the canonicalized rewards.
The following definition describes the canonicalization stage.
Definition 3.3 (Canonically Shaped Reward (Definition 4.1 from Gleave al. (2020)) ). Let R S x AAxx x S - I& be reward function. Given distributions 'Ds E (S) and D E ((A) over states and actions, let and S' be independent random variables distributed as Ds and A distributed as DA Define the canonically shaped R to be:
(
R(s, a, 8') + E[yR(s',,
(,(,, -
(I)
Fig. [ visualizes this process. Informally, this operation computes the mean of the potentials of and 8' with respect to 5" using only evaluations of the reward R, and subtracts/adds those averages from/to the original, shaped reward, thereby canceling out their effects. This cancellation behavior is formalized in the following proposition.
Proposition (The Canonically Shaped Reward is Invariant to Shaping (Proposition 4.2 from Gleave et al. ()020)). Let R : S A x S + T. be a reward function, and $ : S + R a po- tential funcctin.. Let 7 [0, be a discount rate, and Ds (S)) and D E ((A) be distributions over states and actions. Let R' denote R shaped by & R' R''s, a,) ) = R(s, s') > (als')->)) (()). Then the canonically shaped R' and R equal: CDs,DA
CDs.DA
This invariance means that canonicalized rewards can be compared without reward shaping impact- ing the comparison. In practice, canonicalization is performed over an offline dataset of transitions. The rewards for (s, a, s') transitions are computed, and then canonicalized using ( The distri- butions Ds and D are the marginal distributions of states and actions in the dataset on which evaluation proceeds. As we will discuss in the next section, this results in evaluating learned reward functions outside their training distribution, leading to inaccurate reward values.
EPIC's s second stage computes the Pearson distance between two canonicalized reward functions.
Definition 3.4 (Pearson Distance (Definition 4.4 from Gleave et al. (2020)). The Pearson distance
between random variables X and is defined by the expression (X. VI p(X,Y)/ where p(X,Y) is the Pearson correlation between X and Y.
In combining these two steps, EPIC is capable of quantifying the similarity of two reward functions while being invariant to shaping (due to canonicalization) and invariant to changes in shift and scale (due to the use of the Pearson distance).
Definition 3.5 (EPIC pseudometric (Definition 4.6 from Gleave et al. (2000))- Let S, A, S' be ran- dom variables jointly following some coverage transition distribution. Let Ds, and D respectively be the state and action distributions. The EPIC pseudometric between rewards R. and RB is:
DEPIC RA, RB) = D2 (B.
Note that during the canonicalization stage, S, A, S' are independent while in the stage computing the Pearson correlation between canonical rewards, the joint distribution of S, A, S' is used.
4 DARD: DYNAMICS-AWARE REWARD DISTANCE
Here we present DARD, a new distance measure between learned reward functions that uses in- formation about system dynamics to prevent errors due to out-of-distribution queries of learned
4

---

Published as a conference paper at ICLR 2022
R(s,s")
A(a,6')
E
F(((S,5))
E. (R(S'S5"))
RIS. .7
Cr(Mi(s,s')
RIS,S)
2-2
0
S
2
S 2
Figure 1: These plots depict the canonicalization procedure of EPIC (left) and the transformation of DARD (right) applied to a single transition from to s' 2 in simple MDP. The contours depict a reward function for a ID, continuous-state environment with a continuous action space of displacements in [-1, 1]. The axes show the state S and next state S' (omitting the action because is deterministic). The reward function is zero everywhere, but shaped by the potential function ((s). The plotted points correspond to different terms in Eq. 1 or Eq. (3) indicated by the legends. EPIC canonicalizes reward functions by computing reward values for transitions that may not be possible under the transition model of the environment. This is reflected in the visualization by samples comprising the different expectations being dispersed globally over S and/or S". Our approach in contrast uses a transition model to transform the reward function using approximately in-distribution transitions as illustrated visually by its local sampling in computing the expectations. Both approaches arrive at the unshaped value of zero depicted by the red triangle.
rewards. As described in 3.3 the EPIC distance measure first transforms reward functions as shown in equation (I]. This transformation procedure involves computing expectations E E[R(s', A, S')], E[R(s, A, S')J. and E[R(S, A, S")] with independent random variables S and S distributed as Ds. which in practice is the marginal distribution of states in the dataset. Thus, the resulting transitions may not actually be possible under the transition model of the environment of interest. Comput- ing transformed rewards with these out-of-distribution transitions allows EPIC to be robust to (arbi- trary) changes in the transition model. Moreover, without additional information about the transition model, two reward functions that behave identically on in in-distribution transitions, but differently on dii---i-ittitttionn transitions, could justifiably be regarded as different.
However, if these reward functions are learned from data, evaluating them on out-of-distribbtion transitions can lead to unreliable reward values, introducing errors into the reward distance computed by EPIC. For example, the reward value of transition in which home robot teleports across a room likely to be arbitrarily incorrect, since no similar transition would be observed when the robot interacts with its environment. Because EPIC necessarily evaluates learned reward functions on -iitt--distribution data when computing reward distances, these distances can have significant inaccuracies
show empirically in Section 5
To address this challenge, we assume access to a transition model of the environment (which may also be learned in practice), and propose Dynamics- Aware Reward Distance (DARD), a new reward distance measure, which builds on EPIC. DARD computes an alternative reward transformation that evaluates reward functions on transitions that are much closer those observed in the environment. This helps to encourage evaluation of reward functions closer to the distribution of transitions on which they were trained while still providing distance metric between reward functions that is invariant to potential shaping.
Definition 4.1 (Dynamics-Awwa) Aware Transformation). Let R: AxS - R. be a reward function. Given distributions Ds E A(S) and DA ((A) over states and actions, let S be random variable distributed as Ds, and A be a random variable distributed as D Furthermore, given probabilistic transition model defining conditional distribution over next states T(S A), let
5

---

Published as a conference paper at ICLR 2022
S' and S" be random variables distributed [(S'|s, and ((5 |s A) respectively. Define the dynamics-ave aware transformation of R as follows:
Cr(R)(s,,, a. R(s, s') 1 A, A,) H1,, A" 7 A,S"))
(3)
By sampling next states from the transition model conditioning on the current state (either s or s'), this transformation able to evaluate reward models (s,a,')) transitions that are closer to their training distribution. In the case of the term E. R(S', A, S")). the transitions evaluated are not perfectly in-distribution with respect to T. This is because S" is distributed conditionally on the original s' as opposed to the random variable S". These transitions physically realizable than those sampled EPIC. In Section we show that learned reward functions relatively insensitive to these slight errors in practice. We select D to densely cover the space of possible actions. We do this because during policy optimization, policies may select a wide range of possible actions, resulting in evaluating learned reward functions this same distribution of actions. See Figure [ for a visualization of our dynamics-awand transformation. Informally, this operation normalizes the reward values of transition (s, a. based on the reward values of nearby transitions sampled from the transition model. This normalization allows for comparison between reward functions that invariant to potential shaping.
nevertheless much closer to
Proposition 2 (The dynamics-aware transformed reward is invariant shaping). Let R 8 A x - TR. be reward function, and क S - IR a state potential function. Let E [0, 1] be discount factor, and Ds A(S) and DA ((A) be distributions over states and actions, and T S A - (S)) be a conditional distribution over next states. Let R' denote R shaped by R(s,a.s') + (s). Then the ceniition-relative transformations of R and R are equal: Cr CR') (R).
R' (s, a, s')
Proof: See Section A.4
Definition 4.2 (DARD Distance Measure). Let S, A, be random variables jointly following some coverage transition distribution, and let T be the transition model of the corresponding environment. We define the DARD pseudometric between rewards RA and Ra as follows:
DARDD (RA. RB) = =D,(CrrBANVSAAA)))CCCBB
(4)
We prove that the above distance measure is pseudometric in Section A.4
The key insight behind DARD is that we can use knowledge of transition dynamics define a dis- tance measure between reward functions that only evaluates those functions at realizable transitions. Though, unlike EPIC, DARD assumes knowledge of transition dynamics and that these dynamics will remain similar during deployment, DARD uses these additional assumptions to produce signif- icantly more accurate and predictive distances between learned reward functions in practice.
EXPERIMENTS
In experiments we evaluate the following hypotheses: H1: EPIC is a poor predictor of policy return for learned reward functions due its out-of-distribute evaluation of these reward functions. H2: DARD is predictive of policy return even for learned reward models, while maintaining EPIC's in- variance reward shaping. H3: DARD is predictive of policy performance distributions. H4: DARD is predictive of policy return when using learned, high-fidelity transition models. We test these hypotheses by computing each metric across a set of reward models, and then evaluate a policy trained on each reward model against a ground truth reward function through on-policy evaluation. We perform these steps across two environments as described next.
diverse coverage
5.1 EXPERIMENTAL SETUP
Environments We evaluate DARD in two environments. We first study the Bouncing Balls en- viromment, which requires an agent to navigate from random initial position to a random goal position while avoiding set of balls randomly bouncing around the scene. This environment cap- tures elements of real-world, physical environments with multiple interacting agents, as is the case, for example, in autonomous driving. In these types of environments, EPIC samples transitions in

---

Published as a conference paper at ICLR 2022
manner such that agents teleport around the scene, causing issues for learned reward models. The second environment is adapted from the Reacher environment from OpenAl Gym (Brockman et al. 2016). This environment captures aspects of robotic manipulation tasks where arbitrary changes in joint configuration between and s' can cause issues for learned reward models. Videos illustrating both environments be found at htt.pt:: http:///itess.m goo. googl com/v ew wwwp//drrpppprrrrr
Coverage Distributions In order to compute reward distances for DARD, we must sample transi- tions from the environments. In the experiments we consider two coverage distributions. The first the distribution induced by a policy that takes actions uniformly at random across magnitude bounds (denoted by uni.). The second is the distribution over transitions of an expert policy n², which we learn in both environments using Proximal Policy Optimization (PPO) Schulman al. (017).
Action Sampling and Transition Models
Both environments have a 2D, continuous action space. DARI) samples actions from these spaces (i.e., defines 'DA) by defining uniformly-spaced,, 21) grid of action values within magnitude bounds.
We consider two versions of DARD in the experiments. The first uses (approximate) ground-truth transition models, and the second uses learned transition models, which we refer to as DARD-L in the results. For the ground-truth transition model in the Bouncing Balls environment we use a constant velocity model that deterministically samples next states conditional on each sampled action. For the Reacher environment, we use the underlying MuJoCo simulator (Todorov et al., 2012) as the transition model, again deterministically sampling next states.
We additionally learn neural network transition models for each environment. In the Bouncing Balls environment, we learn transition model that independently predicts position deltas for each ball based on its physical state and the action (in the case of the controlled agent). the Reacher environment, we learn transition model that takes as input the state and action, and predicts a state delta used to produce the next state, See Appendix 1.1.4 for model architecture and training details.
Reward Models For each environment, we define learn the following reward models:
The reward function against which we perform on-policy evaluation. In both environments, this is goal-based reward given when the agent position is within some range of the goal location. The Reacher environment also contains additive terms penalizing distance from the goal and large action magnitudes.
GROUND TRUTH:
The ground truth reward function, shaped by a potential function. In both environments, a potential shaping based on the distance to the goal location. In the Bouncing Balls envi- ronment, we take the difference in the square root of the distance to goal (due to the large distance values). In the Reacher environment we use the difference of distances from the end effector to goal directly.
SHAPED:
FEASIBILITY: A namually-definee reward model that returns the shaped reward function when evaluated with a dynamically feassbbe transition, and returns unit Gaussian noise otherwise. This serves as a sanity check to illustrate that EPIC performs poorly on models that exhibit inaccurate reward values on OOD transitions.
REGRESS: A learned model trained via regression the shaped reward model. We regress the shaped reward function because otherwise the metrics do not need to account for shaping (see Ap- pendix I for an example). Although simple, this method used in practice in cases where people manually label reward functions (Cabi et al. 2019). and was used as a baseline in Christiano et al. (017) ("target" method, section 3.3).
REGRESS OUT-OF-DISTRIBUTION A model trained similarly to REGRESS, except that it is trained diiitoff-dittrbutioon" " in the sense that during training it samples (a, s') pairs randomly from the dataset such that its training distribution matches the evaluation distribution of EPIC.
PREFERENCES: reward model learned using preference comparisons, similar to the approach from Christiano et al. (2017). We synthesize preference labels based on ground- truth return.
All learned models are fully-connected neural networks. In the Bouncing Balls environment, the input to the network is specific set of features designed to be susceptible to OOD issues (a squared

---

Published as a conference paper at ICLR 2022
distance is computed between and s' as of the features). For the Reacher environment, concatenate 8, a, and as in the curned-reware experiments in Gleave et al. (2020).
Baselines We compare with two baseline methods. First, compare with EPIC. Second, compare with a simple calculation the Pearson distance p) We include this baseline order assess the extent to which reward shaping is issue that needs to be accounted for in comparing rewards (see Appendix A.2. for additional motivation).
5.2 RESULTS AND DISCUSSION
Table 1: Results for the Bouncing Balls (Top) and Reacher (Bottom) environments. Center: For each reward model, we compute each distance metric 1000 from GROUND TRUTH (GT). The coverage distribution is indicated by the policy (Tii corresponding to the distribution resulting from uniformly random actions, '" to that from expert actions). Right: The average episode return and its standard error of the policy trained on the corresponding reward model. Values are averaged over executions of all steps (data collection, reward learning, reward evaluation) across different random seeds (see Appendix/A....ffoo standard errors of distances). Distance values that are inversely corre- lated with episode return are desirable (i.e., increasing distance within column should correspond with decreasing episode return). Additionally, for hand-designed models that are equivalent to GT (SHAPING, FEASIBILITY), and for well-fit learned models (REGRESS), lower is better. Learned models are fit to SHAPED, and may produce higher episode return than those fit to GT as a result.
Reward Function
1000
DARRD
1000
D DRRD-L
1000 Tuni
D DEII n²
1000 D x*
Episode Return
Tuni
x²
Tuni
Ruii
GT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
9.61 +: 0.41
SHAPING
0.00
0.00
0.00
0.00
0.00
0.00
642
602
12.0 -: 0,20
FEASIBILITY
0.00
0.00
0,00
0.00
642
601
642
602
12.2 + 0.11
REGRESS OOD
12.3
9.63
12.3
9.55
14.4
6,20
642
602
11.9 0.13
REGRESS
28.4
12.7
29.0
12.6
690
613
642
602
12.0 + 0.13
PREF
110
60.0
108
58.9
697
650
668
639
10.8 + 0.13
1000 DAARD
1000 DDARD-I. m²
1000 DEPIC
1000 D,
Reward Function
Episode Return
Tuni
x²
Tumi
Tuni
n²
Tani
GT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
82.4 + 1.13
SHAPING
0.00
0.00
0.00
0.00
0.00
0..0
369
88.9
78.4 + +.22
HEASIBILITY
0.00
0.00
0.00
0.00
529
221
368
88.3
83.4 + 0.66
REGRESS OOD
30.9
130
29.6
100
27.3
60.8
370
105
80.4 0.86
REGRESS
49.7
184
46.6
143
182
175
371
112
82.3 + 1.30
PREF
63.5
63.4
210
165
136
32.8 4.99
Table summarizes the experimental results, which provide support for H1 across both environ- ments. We first note as a sanity check that EPIC assigns large distances between GROUND TRUTH and FEASIBILITY This demonstrates EPIC's tendency assign large distances to reward functions that are equivalent when assuming a fixed transition model, but different under an arbitrary transi- tion model. This behavior manifests with learned models in practice due EPIC's OOD evaluation of those models. For example, in both environments EPIC assigns large distances to the REGRESS model when, based on policy performance, we see that it is quite similar the ground truth (disre- garding shaping). This results from EPIC's evaluation of these reward models with OOD transitions, which is illustrated by the fact that the metric more much predictive of on-policy performance on the REGRESS OOD model (which does not suffer from OOD evaluation issues). Perhaps most importantly, EPIC fails be predictive episode return when comparing learned models. For ex- ample, in the Reacher environment, the metric assigns lower distance to PREFERENCES (PREF) than REGRESS despite the lower episode returns of PREF,
The results from both environments also support H2, In the Bouncing Balls environment, DARD gives low distances for the well-fit models (FEASIBILITY. REGRESS, REGRESS OOD) as a result of only evaluating them in their training distribution. In the Reacher environment, DARD exhibits gap between REGRESS and REGRESS OOD, which indicates that the algorithm's ' evaluation the reward model slightly out-of-distribution transitions does hurt performance. Nevertheless, on the
8

---

Published as a conference paper at ICLR 2022
reward models that can be learned in practice (REGRESS, PREF) DARD produces distances that are more predictive of episode return compared to baseline methods.
The results also demonstrate that DARD is predictive of episode return across coverage distributions (H3). DARD is sensitive to the choice of coverage distribution because the transitions the metric samples are tightly grouped around this distribution. This is reflected in the different distance scales between Jai and x. Nevertheless, the relative value of DARD within a sampling distribution is still predictive of episode return. For example, the metric maintains a consistent ordering of the models across Tuni and π in both environments.
Finally, the results largely support the hypothesis (H4) that replacing ground truth transition models in DARD with high-fidelity learned ones still results in distance values predictive of policy return. This conclusion supported by DARD-L's predictiveness of policy return across both environments and coverage distributions. In most cases, the distance values of DARD-L closely match those of DARD. Three reasons for this close correspondence are: (i) high-fidelity transition models can be fit in the relatively low-dimensional environments considered, (ii) DARD uses transition models to make single-step predictions, which avoids the compounding errors that may occur when sampling rollouts, and (iii) the environments and coverage distributions considered generally produce reward functions insensitive to small errors in the predicted states. The lower distance values of DARD-L in the Reacher environment when using the expert coverage distribution By analyzing the errors made by the learned transition model, we concluded that the lower distances of DARD-LL from the reward functions being quite sensitive to small errors in predicted states in this case (see Appendix A...2 for detailed analysis). Although the distance values are still pre- dictive of policy return, this result is an instance of the false positive equivalence that can result from errors in the transition model (as discussed in Appendix A.3.I (case 2.B)), and illustrates the importance of applying DARD with well-fit learned transitions models.
exception this trend.
Limitations In our experiments, we discovered that DARD is poorly suited for reward comparison in environments where nearby transitions exclusively have similar reward values. This is the case,
for example, in robotic domains with small timesteps and in which a smooth reward function is
used (such as negative distance to the goal). We illustrate this behavior and explain why it occurs in Appendix T with experiments the Point Maze environment used by Gleave et al. (2020). In short, because DARD performs reward comparisons between transitions relative to transition model. if all the transitions considers (across all possible actions) have nearly-identical reward values, it can be sensitive to noise in the reward model. For this reason, DARD should be used in environments for which, at least for some parts of the state-action space, nearby transitions yield well-differentiated reward values.
Overall, these experiments demonstrate that in situations where we have confidence the transition model will not change significantly, as is the case in many physical settings, DARD allows for comparing learned reward functions with other reward functions without policy optimization.
6 CONCLUSION
The ability to reliably compare reward functions is a critical tool for ensuring that intelligent sys- tems have objectives that are aligned with human intentions. The typical approach to comparing reward functions, which involves learning policies on them, conflates errors in the reward with er- rors in policy optimization. We present Dynamics-Aware Reward Distance (DARD), a pseudomet- ric that allows for comparing rewards directly while being invariant to reward shaping. DARD uses knowledge of the transition dynamics to avoid evaluating learned reward functions far outside their training distribution. Experiments suggest that for environments where we can assume the transition dynamics will not change significantly between evaluation and deployment time that DARD reliably predicts policy performance associated with reward function, even learned ones, without learning policies and without requiring online environment interaction.
There are number of exciting areas for future work. For example, DARD could be used to define benchmarks for evaluating reward learning methods (e.g., in applications such as autonomous driv- ing or robotics; Appendix A.3.3). Second, DARD could be applied to high-dimensional (e.g., visual) domains where learned transition models are necessary and for which OOD evaluation issues are likely particularly challenging to overcome.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have attempted to make the experimental results easily reproducible by providing the following: first, source code necessary reproduce results is available in the supplementary material. Second, the software for this research is designed to allow for reproducing the results in the main body of the text through a single program execution. Third, results are computed over five random seeds and standard error values are reported in Appendix A. to provide reasonable bounds within which reproduced results should fall. Fourth, we have attempted to provide extensive details regarding the experimental setup in Appendix A.1
ETHICS STATEMENT
The ability to reliably and efficiently evaluate reward functions without executing policies in the environment has the potential to ensure that AI systems will exhibit behavior that is aligned with the objective of the system designer before risking deploying these systems for making safety critical decisions. This has the potential to facilitate the application of reinforcement learning algorithms in settings in which this might otherwise be too risky, such as healthcare, finance, and roboties.
The primary risk associated with these methods is that system designers may rely on them inappro- priately (e.g.. in situations where they do not apply, or in lieu of additional validation procedures), which could result in unsafe system deployment. We have attempted to address this risk by stating limitations of the method (see Section 5.2). and by providing example applications we believe to be appropriate (see Appendix A.3.3)
REFERENCES
Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and Sculley. The ml test score: rubric for ml production readiness and technical debt reduction. In 2017 IEEE International Conference on Big Data (Big Data), pp. 1123-1132. IEFE, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: 1606.0 2016.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov. Ksenia Konyushkova, Scott Reed, Rae Jeong. Konrad Zolna. Yusuf Aytar, David Budden. Mel Vecerik, al. Scaling data- driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv: /909. /2200, 2019.
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv: 1706110667.777 03741, 2017.
Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv 2011.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568. 2018
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11246: 2017.
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. arXiv preprint arXiv:20066. 13900, 2020.
Jessica E Holland, Mykel J Kochenderfer, and Wesley A Olson. Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance. Air Traffic Control Quarterly, 21(3):275-29, 2013.
Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. arXiv preprint 190iv: 2019.
10

---

Published as a conference paper at ICLR 2022
Nan Jiang and Lihong Li. Doubly robust off- off-policy value evaluation for reinforcement learning. arXiv preprint arXiv:1511.03722 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2:::412.6980 2014.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis)design for autonomous driving. arXiv preprini arXiv:2 13906, 2021.
Hoang M. Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. arXiv 2019.
preprint
Donghun Lee, Srivatsan Srinivasan, and Finale Deshi-Velez. Truly batch apprenticeship learning with deep successor features. arXiv preprint arXiv: 1903. /0077, 2019.
Eric Liang. Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp. 5053-3062. PMLR, 2018
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 1rXivv:311 13... 5602, 2013.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, volume 99, pp. 278-287, 1999.
Andrew Y Ng, H Jin Kim, Michael I Jordan, Shankar Sastry, and Shiv Ballianda. Autonomous he- licopter flight via reinforcement learning. In Neural Information Processing Systems, volume 16. Citeseer, 2003.
Cosmin Paduraru. Off-policy evaluation in Markov decision processes. PhD thesis, McGill Univer- sity, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perf-peerormance deep learning library. Neural Information Processing Systems, 32:8026-8037, 2019.
Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In International Conference Machine Learning, pp. 759-766, San Francisco, CA. USA, 2000. Morgan Kaufmann Publishers Inc.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024:: 02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. nature, 550(7676):344355,, 2017.
Gerald Tesauro. Temporal difference learning and TD-Gammon. Communications of the ACM, 38 (3):58-68, 1995.
Devinder Thapa, In-Sung Jung, and Gi-Nam Wang. Agent Based Decision Support System Using Reinforcement Learning Under Emergency Circumstances, volume 3610. 2005.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, volume 8, pp. 1433-1438. Chicago, II,, USA, 2008
11

---

Published as a conference paper at ICLR 2022
A
SUPPLEMENTARY MATERIAL
A.1
EXPERIMENTAL DETAILS
The following sub-sections provide details on the different components involved in the experiments.
A.1.1 ENVIRONMENTS
Bouncing Balls Environment This is custom environment with traits of multi-agent navigation environments such autonomous driving. The agent attempts navigate to random goal location while avoiding a set of balls (other agents) bouncing around the scene. The state consists of the position and velocity of the balls (the agent included), well the goal location. The action space is a 2D continuous space of accelerations between [-5, 5] to apply to the agent state. The transition function applies the agent accelerations to its state, while the other agents apply random accelera- tions sampled from zero-mean, low variance Gaussian distribution. The reward function gives positive value once the agent within some distance of the goal, which results in the agent and goal position being randomly moved. This environment executes for fixed horizon of 400 timesteps to approximate a continuing, infinite horizon MDP. We make the environment approximately contin- uing in order to ensure that S and S" are distributed identically as is required by EPIC. Sampling and 5' from their marginal distributions in the dataset results in transitions where the agents in have completely different positions and velocities in s', an issue that would be encountered in real-world multi-agent settings.
Reacher Environment This environment is adapted from the Reacher environment from OpenAI gym (Brockman et al. 2016). We adapt the environment in two ways. First, we add a goal-based reward component. Second, we increase the timesteps between states 5 from original 2. We make both these changes so that the rewards nearby transitions are sufficiently differentiated (see I for detailed motivation).
A.1.2 POLICY LEARNING
We used the RLlib Liang et al. 2018) implementation of Proximal Policy Optimization (PPO) for learning control policies (Schulman al. 2017). Parameters are provided in table
Parameter
Value
Discount 7
0.95
GAE A
0.95
# Timesteps Per Rollout
64
Train Epochs Per Rollout
# Minibatches Per Epoch
Entropy Coefficient
0.01
PPO Clip Range
0.2
Learning Rate
x 10
Hidden Layer Sizes.
[256,256]
# Workers
# Environments Per Worker
40
Total Timesteps
8B
10M
Table PPO parameters. Superscripts indicate Bouncing Balls (BB) or Reacher (R) environments.
A.1.3 DATA COLLECTION
We collect sets datasets for use in reward learning and evaluation. Each set contains three sub- datasets for (i) reward model training, (ii) reward model validation, and (iii) reward evaluation. We collect these dataset sets using two policies. The first policy (Tui)) yields a uniform distribution on the action space. The second policy is expert (x trained described in A.12 Table 3 gives the sizes of these datasets, which were set such that the validation and train losses were similar during reward learning.
12

---

Published as a conference paper at ICLR 2022
Environment
Reward Learning Train
Reward Learning Validation
Reward Evaluation
Bouncing Balls Reacher
2,000,000
500,000
500,000
4,000,000
1,000,000
1,000,000
Table 3: The sizes of the datasets used for reward learning and evaluation for each environment.
1.1.4 TRANSITION MODEL LEARNING
This section provides details regarding the architecture and training of the learned transition models.
For the Bouncing Balls transition model, we split the state (originally containing the concatenated information for each ball in the scene and the goal position) into states for each ball and predict the motion of each independently. We simplify the task in this manner in order to make learning the transition model easier, but a model that predicts the next state jointly could be learned instead. The absolute velocity of each ball is then passed to a fully-connected network, which outputs position delta to be applied to the original state. In the case the controlled ball, the action is also provided to the network. Given this formulation, the network is learning to translate velocity into change in position over some time period, optionally applying an acceleration value as well.
For the Reacher environment, pass the concatenated state and action fully colly-connected net- work, and again predict changes in the state. We normalize the scale of each state dimension by dividing by its standard deviation across the dataset.
In both cases, we use a mean squared error loss computed between (1) the model output (i.c., state delta) applied to the original state and (ii) the ground truth next state. We train the transition model on data sampled from Tani- which is common in learning transition models (Ebert et al. 2018). Model and training hyperparameters are given in Table
Parameter
Value
Batch Size
2048
Hidden Layer Sizes
[256, 256]
Activation Function
ReLU
Normalization Layer
Batch Norm Rt None BB
Learning Rate
(2 10-4)R, (5 10-4)88
Gradient Clip
1.0
Optimizer
Adam (Kingma & Ba (0014))
Table 4: Parameters for learning transition models. For information datasets used for training see section[A.1.3] Superscripts indicate parameters specific to Bouncing Balls environment (BB) or Reacher environment (R).
A.1.5 REWARD LEARNING
Three methods of learning reward functions were considered in this research: (i) direct regression of per-timestep rewards (REGRESS), (ii) regression shuffled dataset matching the evaluation dis- tribution of EPIC (REGRESS OOD), and (iii) preference-batee
learning of rewards. These methods each fit a fully-connected neural network, and their model and training hyperparameters are pro- vided in table 5 We used early stopping based on validation loss for all models to determine the number of training epochs.
For the Bouncing Balls environment we input into the model set of manually extracted features including distance the agent to the goal the current and next state, the difference distances between states, an indicator of whether the goal has been reached, squared differences in position of other agents, and others. Features involving squared distances are what make these model suscep- concatenate the state, action, and next state,
tible to OOD issues. For the Reacher environment and pass that directly into the model. In either case, these features are normalized by subtracting their mean and dividing by their standard deviation.
13

---

Published as a conference paper at ICLR 2022
Parameter
Value
Batch Size
2048
Hidden Layer Sizes
[256, 256]
Activation Function
tanh
Learning Rate
10-4
Gradient Clip
1.0
Discount y
0.95
Optimizer
Adam (Kingma & Ba (2014)
Reward Regularization
0.01 0.001*
Trajectory Length
25 15R
Random Pairs OUD
10 5
Table 5: Parameters for reward learning algorithms. For information datasets used for training
section 1.1.3. Superscripts indicate parameters specific to preference learning (p), OOD regression (OOD), Bouncing Balls environment (BB), Reacher environment (R).
A.1.6
REWARD EVALUATION
EPIC We compute the sample- based approximation of EPIC described in section 1.1.1 Gleave et al. (020). Table gives the parameters we use in computing EPIC. We use the same values of Ny and NM as Gleave al. (2020), though use fewer seeds. In our case each seed represents full execution of the pipeline (data collection, reward model training, reward evaluation), whereas in Gleave al. (2020) it represents different sub-samplings single dataset. We take this approach in order to assess sensitivity to randomness in the data collection and reward learning processes.
Parameter
Value
State Distribution Ds
Marginalized from 2 Marginalized from D
State Distribution D
# Seeds
Samples Ny
32768
Mean Samples Nas
32768
Discount y
0.95
Table EPIC parameters.
DARD Similarly EPIC, compute sample-based approximation of DARD in the exper- iments. We sample a batch By Ny (s, transitions. For computing the expectations E[R(s, A, S')l and R[R(s', A, S")] sample NA actions u using a sampling policy (described in Appendix A.1.3). We then sample Ny samples of S" or S" x' and x", conditional on or s' and u. For the last expectation, E[R(S', A, S")], we sample actions and next states similarly for S', but repeat the process for S" overall computing the mean over O(NANN)) transitions. We com- pute the aynamics-aware transformation (definition 3) for each transition by averaging over these sampled actions and next states:
()())(s,a,5)) = als,a,s') BB S,, R(s, AR(S A,
(5)
s R(s,a,ss)
R(s u4, u,,z
NANT ==
NA Nr R(s,u;, U.,
(6)
NANT 1= ==
NA Nr NA No ΣΣΣΣ
R(aly unt- x())
N2 == j=0 K= 1=0
14

---

Published as a conference paper at ICLR 2022
Overall this yields O(Ny-NANN) algorithm. In experiments the transition model is assumed to be deterministic, making Nr = 1. The actions are sampled from the N-dimensional action space by taking the cross product of linearly-sampled values for each dimension. For example, in the Bouncing Balls environment, we sample actions linearly between [-5, 5 for each the two dimensions, overall yielding NA 64. The complexity of computing E. RR("", A, could be reduced by instead sampling A actions and Ny transitions for each of x' and x²..
For the Bouncing Balls environment, we approximate the transition function with constant velocity model for all agents, implemented in Pytorch and executed on the GPU (Paszke et al. 2019). For the Reacher environment, we use the ground truth simulator, which we execute in parallel across cores. See table 7 for summary of the DARD parameters used in the experiments.
Parameter
Value
# Seeds Samples Ny
5
100,000
Action Samples NA
64 16R
Transition Samples Nr
1
Discount ?
0.95
Table 7: DARD parameters. Superscripts indicate parameters specific to the Bouncing Balls (BB) or Reacher (R) environments.
Policy Evaluation We compute average episode returns of policies trained both manually- defined and learned reward models. Training parameters are the same as those used for learning the expert policy (see Table 2 We evaluate policies for million steps in the respective environment, and report the average, undiscounted episode return. Similarly to the other reward evaluation steps, we train one policy for each random seed (of which there are 5).
A.1.7 STANDARD ERROR OF RESULTS
Tables and provide the standard errors the estimates of the mean distances and episode returns, computed over 5 random seeds.
1000
DAARD T
10000 x DDARD
1000 DEPIC n
1000 D x²
Episode Return
Reward Function
Tuni
Tusi
5²
Tuni
Tasi
GT
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.41
SHAPING
0.00
0.00
0.00
0.00
0.00
0.00
1.56
0.31
0.20
FEASIBILITY
0.00
0.00
0.00
0.00
3.05
0.70
1.56
0.31
0.11
REGRESS OOD
1.10
0.90
1.32
0.92
0.84
0.40
1.53
0.49
0.13
REGRESS
1.70
0.92
1.38
2.07
1.56
0.37
0.13
PREF
7.20
3.20
7.52
2.10
7.74
1.14
0.13
Table 8: Standard error values for Bouncing Balls environment.
1000 DDARD
1000 x DDARD-L
1000
DEPIC n
1000 D, n
Episode Return
Reward Function
T²
T²
GT
0.00
0.00
0.00
0.00
0.00
1.13
SHAPING
0.00
0.00
0.00
0.00
0.00
0.00
0.38
1.85
3.22
FEASIBILITY
0.00
0.00
0.00
0.00
0.68
6.65
0.36
1.61
0.68
REGRESS OOD
1.37
15.9
1.42
6.62
1.71
4.74
0.47
2.87
0.86
REGRESS
2.07
11.5
1.52
2.78
10.8
5.13
0.49
2.26
1.30
PREF
2.02
14.7
2.38
11.5
3.91
6.15
0.45
5.02
4,99
Table 9: Standard error values for Reacher environment.
15

---

Published as a conference paper at ICLR 2022
1000 x DDARD π²²
1000
DEPIC n
1000 x Dp
Episode Return
Reward Function
Tuni
Tuni
Tuni
n
GT
0.00 195
0.00
0.00
0.00 34.2
0.00
0.00
-4.73 + 0.48
REGRESS
480
14.8
6.59
32.1
-8.02 + 0.95
1000 x DDARD π²
1000 DEPiC n²
1000 x Dp
Episode Return
Reward Function
Tuni
Tuni
n²*
GT REGRESS
0.00
0.00
0.00
0.00
0.00
0.00
-2.1 = 0.06
16.3
47.5
6.34
25.4
6.19
20.2
-2.59 + 0.10
Table 10: Top: Results for the original Point Maze environment. Bottom: Results for the Point Maze environment when using a larger timestep. See the caption of farr how to interpret these results. DARD distances are much closer to those of EPIC in the large-timestep case, indicating that the differences in rewards assigned to nearby transitions has a significant impact on DARD performance. Parameters used for reward learning match those of Gleave al. (2020). We were unable to exactly reproduce the episode returns of the policies trained on learned reward models. Note that in the larger-timestep case the episode returns will differ due to the change in timestep.
A.2 ADDITIONAL. EXPERIMENTS
A.2.1 POINT MAZE EXPERIMENTS
In this section we present results for the Point Maze environment considered in Gleave et al. 2020)- These results illustrate a disadvantage of DARD. As discussed section [3] because DARD performs reward comparisons between transitions relative a transition model, if all the transitions it consid- ers (across all possible actions) have near-identical reward values, it can be sensitive to noise in the reward model. The Point Maze environment fits this description, and DARD performs poorly on it as a result (see table IO). We provide evidence for the hypothesis that the lack of eeward-differeeniaaed transitions causes this poor performance by also providing results for an adapted Point Maze en- vironment, which is identical to the original except that we increase the time difference between timesteps by a factor of 5 (such that nearby transitions yield different reward values). In this adapted case, DARD and EPIC distances are significantly closer than in the original environment.
Two additional traits of the environment are relevant. First, the ground truth reward only a function of the current state and action, but not the next state. As result, learned reward models tend to ignore the next state, which minimizes the effect of the OOD reward evaluations performed by EPIC. Second, in these experiments (unlike those in section (5) we train reward models to fit the ground truth reward model (as opposed to a shaped version of it). We do this to be consistent with the experiments provided in Gleave et al. (2020); however, this results in the learned reward models exhibiting little shaping (as illustrated by the low distance values of Dp)). This a property of the reward learning algorithm and specific environment in which it is applied, but learned reward models may exhibit significant shaping in general (Fu et al. 2017).
A.2.2 METRIC SAMPLE SIZE SENSITIVITY
How sensitive are reward comparison metrics to the size of the dataset on which they are computed? The answer depends on the specific environment, reward functions, and comparison metric under consideration. As such, in this section we evaluate the sensitivity to sample size variations of DARD and EPIC in the Bouncing Balls and Reacher environments across variety of manually defined and leamned reward functions.
Evaluation Procedure Because we do not have "ground-truth" values that each metric should as- sume for reward functions in general, we focus instead on the variability of each reward comparison metrie for a given sample size. We quantify this variability by computing, for each metric, a 95% confidence interval (CI) of the mean reward distance across different sample sizes. In this context, smaller CI width indicates that a metric is less sensitive to the random variations in the sampling of the data. We compute the CI by sampling K
100 datasets of size N for varying N from much
16

---

Published as a conference paper at ICLR 2022
larger population of samples (we use the reward evaluation dataset described in Appendix A.1.3) For each dataset, we compute each metric comparing each reward function to the ground-trume truth re- ward function. We compute the CI based on the K resulting distances for each sample size N. For the evaluation datasets, we use x for the Bouncing Balls environment, and Tuii for the Reacher environment. We use x² for Bouncing Balls because the reward is sufficiently sparse that for small sample sizes, N, the metric values are frequently undefined (due to only having samples with zero reward). Using instead of Tuni increases the fraction of transitions within the goal threshold addressing the problem.
Reward Functions We compare a set of manually defined and learned reward functions. For the manually defined reward functions, we desire a set of rewards that smoothly vary in similarity to the ground-truth reward. To accomplish this, we use noisy version of the ground-truth reward defined as Rmois((ss (8, a, 5') = R(s a.,") + e for e ~ (0) By increasing a we produce a sequence of reward functions that are increasingly different from the ground-truth reward. For the learned rewards, we use the REGRESS and REGRESS OOD reward models introduced in Section Recall that REGRESS is trained transitions sampled from dataset of demonstrations collected from the environment, whereas REGRESS OOD is trained on the distribution of transitions resulting from sampling actions and next states independently from the starting state. The resulting training distribution matches that used by EPIC in comparing rewards.
Results Fig. 2 shows the results of the experiment, with each row corresponding to different different environment (Bouncing Balls on the
reward function, and each column corresponding left and Reacher on the right). We see that the 95% CI width for DARD and EPIC are quite similar for the noisy ground -oound-truth reward functions (top three rows of the figure) across all sample sizes. This indicates that, for these types of reward functions, the two metrics are similarly sensitive to the size of the dataset. Note that as the amount of noise increases so does the width of the CI, but the similarity of results across metrics holds.
For the learned reward models we do observe different behavior between the metrics. the case of the REGRESS model (4th row from the top), see that DARD yields significantly smaller CI widths than does EPIC. The reason for this is that EPIC evaluates the reward function at transitions outside its training distribution, which can yield arbitrary values. The additional randomness (with respect to the random sampling of the data subset) incurred in these reward calculations results in more variation in the reward distance, which produces larger Cls. In contrast, DARD avoids these OOD evaluations of the reward, instead staying close to the reward function training distribution where it is more consistent (across data subsets), producing smaller Cls.
For the REGRESS OOD reward model (bottom row), we observe the inverse: EPIC produces smaller CI widths than does DARD. This is because the reward model training distribution matches the one used by EPIC. As result, the "imagined" transitions of EPIC effectively increase the size of the evaluation dataset producing less variation for given sample size. DARD also uses "imagined" transitions, but only from the distribution of transitions encountered in practice in the environment.
We evaluated the sensitivity of DARD and EPIC to variations in sample size across different environments and reward models. We found that for noisy ground-truth reward functions, the two metrics performed similarly, and that for learned reward functions, the relative sensitivity of the metrics depends on the training distribution of the reward function under consideration.
Conclusion
A.2.3 RANDOMIZED REWARD FUNCTIONS
The results in Section S focus on a special set of reward functions that are relatively similar to the ground-truth reward function. this section we instead attempt to address the question, how does DARD perform in the full range of reward similarity? We investigate this question by (i) randomly generating a large number of reward functions in the Reacher environment, (ii) optimizing a policy for each reward function, (iii) evaluating that policy against the ground-truth reward function, and (iv) computing the DARD distance between the randomly generated and ground-trum -tuth rewards.
Evaluation Procedure The rewards are generated by randomly sampling weights for the three reward components: di stance from goal, action magni goa -reached
17

---

Published as a conference paper at ICLR 2022
Ground Truth Noise
Ground Truth Noise 0001)
95% Confidence Interval Width 95% Confidence Interval Width 95% Confidence interval Width 95 Confidence Interval Width 95% Confidence Interval Width
DARD
EPIC
0.0115
0.003
0.0000
0.000
0.0001
Ground Truth Noise (a = 0.1)
Ground Truth Noise (a 0.1)
0.014
0.012
0.100
0,022
0.008
0.001
0.002
Ground Truth Noise (a 1)
Ground Truth Noise (a 1)
0.0150
0.000
0.008
0.001
0.000
Regression
Regression
0.008
0.0175
0.0150
0.010
0.001
0.0075
0.0025
0,100
0.1000
Regression Out-of- Distribution
Regression Out-of- Distribution
0.0088
0.0007
0.005
0.0000
0.006
0.001
0.063
0.002
10
10
10
10a
Sample Size
Sample Size
Figure 2: Comparison of the sensitivity of DARD and EPIC variations in the size of the dataset. Left: Results for the Bouncing Balls environment. Right: Results for the Reacher environment. See Appendix A.2.2 for detailed description of the figure and analysis of the results.
18

---

Published as a conference paper at ICLR 2022
indicator. Weights for the first two components are sampled uniformly between [0, 1]. The reason we disallow negative weights for these components is that they make policy optimization challenging ill-posed: in the distance from goal case it creates challenging exploration problem, and in the act on magn tude case it encourages the policy to output increasingly large actions. Weights for the goa soal-reacien ched ind cator component are sampled uniformly from [-1, 1J. The reward function is then computed as linear combination of the reward compo- nents weighted by these sampled values. We sample 128 reward functions total. Policy optimiza- tion, reward evaluation, and policy evaluation are performed as described in Appendix A.1
Results Fig. 3 visualizes the results from the experiment, computing DARD the random (Tass) and expert (Λ coverage distributions. We see that in both cases DARD is predictive of policy return indicated by the association between small DARD distances and large policy return. The DARD distance and policy return are both heavily dependent on the sign of the goal-reached indicctt weight, which we visualize by coloring points based on the sign of this weight. When the sign is negative, the policy learns to avoid getting too close to the goal, therehy dramatically re- ducing its expected return under the ground-trute reward. The Tuni plot shows policy return decaying more quickly as function of DARD distance than does the plot of x². The reason for this is that in the Tuni case, the action magnitude contributes much more significantly to the DARD distance than in the x² case (because nT generally consists of small actions near the goal state).
There are some outliers in the plot, which seem to result from extreme combinations of weight val- ues that can cause policy optimization issues. For example, cases of low DARD distance and low policy return in general tend to have small distance from goal weights and large, positive goa scal-reached lems for policy optimization (based on prior experiments, the policy does eventually achieve high expected return, but only with more training timesteps). In the Tuni case, instances of large DARD distance and high policy return tend to result from extremely small action magnitude weights and large, positive goal-reachhe takes large magnitude actions to quickly reach the goal state. This incurs large penalty over the first few timesteps of the episode, after which the agent remains mostly stationary, accruing the goal-reached reward. This trend does not hold for xt" which demonstrates DARD's sensitivity to the coverage distribution.
cator weights. In these cases, the reward quite sparse, causing prob-
indicator weights. In these cases, the optimized policy
Conclusion In this section, we evaluated DARD on large number of random reward functions in the Reacher environment, many of which differ significantly from the ground-truth reward. Results demonstrate that DARD maintains its predictiveness of policy return in this setting across both coverage distributions.
A.3 ADDITIONAL DISCUSSION
A.3.1 THE IMPACT OF TRANSITION MODEL ERRORS ON DARD
In this section, we discuss how errors made by the transition model can impact the performance of DARD. We organize our discussion of this topic around different cases of reward equivalence and different types of errors that a transition model might make. The high-level summary of the following discussion is that errors in the transition model only impact the performance of DARD on reward functions that are not equivalent everywhere (case 2 below), and in this case DARD may produce both false negative and false positive equivalences as a result of transition model errors.
Case 1: Equivalent Reward Functions Consider two reward functions that are equivalent across the full space of transitions accounting for reward shaping (this includes the case of identical re- wards). The proof of Proposition 2 does not rely on any proprieties of the transition model. As result, we can conclude that even arbitrary errors in the transition model will not result in two werryhhereeqquivleen reward functions being considered different. This logic indicates that errors in the transition model may only impact the performance of DARD when the reward functions are not whweeeeeuueiieeeene as discussed next.
Case 2: Different Reward Functions
19

---

Published as a conference paper at ICLR 2022
DARD Distance vs Policy Return (Tun))
DARD Distance vs Policy Return (ar
Goal Weight
Goal Weight
80
Goal Weight
80
Goal Weight
60-
60-
Policy Return
Policy Return
40-
40
20
20-
0-
-00
-20-
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.0
0.2
0.4
0.6
0.8
1.0
DARD Distance
DARD Distance
Figure 3: Comparison of DARD distance with policy return for randomized reward functions in the Reacher environment. Each point in these scatter plots corresponds to a different randomly generated reward function. We train policy on each these reward functions, and then evaluate that policy against the ground-truth reward function. The resulting mean policy return is plotted on the y-axis. On the x-axis we plot the DARD distance computed with respect to the Tuni (Left) and r (Right) coverage distributions. See Appendix A.2.3 for analysis of the results.
SUBCASE A: REWARDS ARE EQUIVALENT ON ALL Realizable TRANSITIONS Given the ground truth transition model, DARD would correctly identify the two rewards as equivalent (with respect to realizable transitions). However, if the transition model erroneously samples unrealizable transi- tions on which the rewards differ, the transformation term of Eq. (3) could assume arbitrary values, thereby resulting in the two rewards being considered different. This might occur, for example, when dealing with learned reward functions for which sampled next states are OOD with respect to their training distribution.
SUBCASE B: REWARDS DIFFER ON SOME REALIZABLE TRANSITIONS Given the ground truth transition model and sampling of the full action space, DARD would correctly identify the two rewards as different (assuming states producing transitions on which the rewards differ exist in the dataset). However, if the transition model incorrect such a way that it never samples those transitions which the reward models differ, DARD may produce false positive equivalences.
For example, suppose transition model learned for an autonomous driving environment. Because collisions are rare in this environment, the transition model might never sample transitions involving collisions. As a result, two reward models that differ only with respect to transitions involving collisions may be considered equivalent by DARD. In the preceding text, say that DARD may produce false positive equivalences in this case because even with an erroneous transition model, the dataset which the rewards compared itself contains transitions which the rewards differ DARD will still recognize them as different (due to the R(8, a, s') term of Eq. 33.. Continuing the autonomous driving example: if the dataset itself contains collisions, then the rewards will be
identified as different even if the transition models do not sample collisions. This case illustrates the importance of using a comprehensive coverage distribution comparing reward functions.
A.3.2 ANALYSIS OF DARD-L IN REACHER WITH x COVERAGE DISTRIBUTION
This section provides detailed analysis of the results for DARD-L in the Reacher environment when using the coverage distribution of the expert policy.
Unlike the uni coverage distribution, the states sampled from the expert distribution in the Reacher environment generally have small distances between the end-effector and goal location. As a result, the reward is dominated by whether not the end-effector is within the threshold distance which
20

---

Published as a conference paper at ICLR 2022
Predicted Distance Discrepancies Between Transition Models
Goal Reached Threshold
Learned Model Distances
Ground Truth Distances
400
Frequency
300
200
100
0.00
0.02
0.04
0.06
0.10
0.12
0.14
Predicted Distance Between End-Efffector and Goal Position (m)
Figure 4: This plot shows histograms of the ground truth and predicted next state displacements between the end-effector and goal location in the Reacher environment. These displacements computed on 200 transitions for which the learned transition model yields transformed reward values that most differ from those of the ground truth transition model. The displacements predicted by the learned transition model are consistently larger than those of the ground truth model, and are often beyond the threshold which the "goal reached" reward assigned.
the "goal reached" reward is given. This causes the reward functions to be sensitive to small errors in the predicted state because those errors can move the end-effector or out of range of the goal.
On average, the learned transition model predicts larger next state end-effector displacements than the ground truth transition model. This can be seen in Fig. 4 where we plot the histogram of ground truth and predicted displacements the most between the ground truth and learned transition models. These erroneous, larger predicted distances result from using mean squared error loss, which causes displacements near zero to contribute a minuscule amount to the loss.
set of transitions for which the transformed reward differs
As result, the next states predicted by the learned transition model are average easier assign rewards to than those of the ground-truth transition model: they are more clearly beyond the limit of the "goal reached" threshold, and (due being farther away) reward shaping (which is easier to predict) constitutes larger fraction of the reward. Overall this results in the learned transition model yielding lower DARD distances than those of the ground truth transition model. This is an instance of Case 2.B in Appendix [.3.]] states at which the reward models differ more are sampled relatively less frequently due to errors in the learned transition model.
A.3.3 APPLICATIONS OF REWARD COMPARISON METRICS
This section discusses two potential applications of reward comparison metrics.
Benchmarking Reward Learning Algorithms Suppose have large amount of (potentially suboptimal) demonstration data from which we would like to learn a reward function using one of a variety of potential IRL algorithms. Additionally, suppose that we have (through an expen-
21

---

Published as a conference paper at ICLR 2022
sive process) manually labeled rewards in relatively small set of demonstrations. Furthermore, assume that evaluating a policy online involves an expensive or dangerous process, and that learn- ing policy for this environment is itself challenging (e.g.. sometimes fails or produces policies of highly-variable quality). Note that this setting might occur in practice in physical robotic domains such autonomous driving robotic manipulation.
In summary, we would like to know which of the set of possible IRL algorithms produces the best reward function using the large of demonstrations, but we would prefer to avoid policy learning and evaluation. One way to accomplish this goal would be to learn a "benchmark" reward model on the small set of manually labeled rewards (e.g., through supervised learning as in Cabi et al. (2019), and then to compare that with the rewards produced by the various IRL algorithms. If reward function learned via IRL. similar to the benchmark reward function, this indicates that is similar to the reward function manually defined by experts, and should therefore be preferred. In this way, we can evaluate the different IRI. methods we are considering, and choose the best for our problem. Furthermore, this approach could be applied in creating open benchmark enabling the comparison of reward learning methods.
Validating Learned Reward Functions Suppose that we have deployed a system that uses learned reward function. With deployed learning systems, it is often necessary to continuously collect data and retrain models in order to account for changing data distributions arising from variety of sources (Breck et al. 2017). As such, we would like to frequently retrain the reward function newly-collected data; however, we would also like to ensure that the reward function has not changed "too much" from previous versions that we know to be of high quality (through expensive deployment). Similar the previous application, suppose that online policy evaluation is expensive or dangerous, and that policy optimization is challenging.
Given this problem setting, could use reward comparisons follows. First, learn new reward function on all the data collected so far. Second, compare that reward function with the old reward function on the validation dataset of the old reward. If the comparison yields a small distance be- tween rewards, this indicates that the reward functions are likely quite similar, thereby providing support for continuing the process of deploying the new reward function. Alternatively, if the com- parison yields large distance between rewards, this indicates that the new reward is potentially quite different from the old one, and that we should carefully inspect the cases where the rewards differ before proceeding with deployment. In this way. we have defined (part of) a validation procedure for continuously learning and deploying reward functions.
A.4 PROOFS
A.4.1 COMPARING LEARNED REWARD FUNCTIONS
Proposition (The dynamics-aware transformed reward is invariant to shaping). Let R S x A x S + IR be reward function, and & + R potential function. Let "y [0, 1] be discount rate, and Ds E A(S) and DA E A(A) be distributions over states and actions, and ASS) be conditional distribution over next states. Let R' denote R. shaped by ::RR(ss (s = R(s, a, s') + 70((s) ((s). Then the dynamics aware transformations of R! and R are equal: Cr Cr (R).
T S S
22

---

Published as a conference paper at ICLR 2022
Proof. a, E A S. Then by substituting in the definition of R' and using linearity of expectation:
(T(R)(s,
A,
(7)
( ((05,,,))+++()))+))))))))))))+))
(8)
+ +,
-
-E }0("))
-(6,,..))++E/(,344555) ( - A,S"")
(9)
((=) - [ss)
- -)0(5))
R(s, B R(s, A,5") - -S' A,S"))
(10)
(T,
(11)
Lemma A.1 (The Pearson distance Dp is pseudometric). (Lemma 4.5 from Gleave et al. 2020
Namely, for set of random variables, D, satisfies the following properties:
Identity: D,(X, X) 0 X E X
Symmetry: Dx(X, ,Y, VX,Y X, and
Triangle Inequality: D (x Z) +, VXY
Proof: See Gleave et al. (2020)
Proposition 4 (The DARD distance metric pseudometric). Let A, S' be random variables jointly following some coverage transition distribution, and let T be the transition model of the corresponding environment. We define the DARD pseudometric between rewards RA and Rg as follows:
(RARD(RA, RB) A.
(12)
Proof. This proof closely follows the proof of Theorem 4.7 from Gleave al. (2020). Namely, the result follows from direct application of Lemma A.1
Let RA RB, and Re be reward functions mapping from transitions A x S to real numbers Identity: It immediately follows that
DDARD RA. RA) = (rr( A, (R(,),, = =0 since D, is pseudometric,
Dx(X, X) O4X e
Symmetry: This again immediately follows from the fact that D, is pseudometric. We have that:
DDARD RR,, RB) = (
(13)
= =D, A,S"), C,
(14)
-DDARD (RB, RA)
(15)
since D, X), because D, is a pseudometric.
Triangle Inequality: This again immediately follows from the fact that D, is pseudometric, We have that:
DDARD RA, (c) (..
(16)
DD (T (...A,
(17)
+ D, (c,(R))((SAAA (RRC)
(18)
DDARD (AA, RB) + DDARD(Rg, Rc) since D,(X, Dx(X)) (Y, Z) because D, is pseudometric.
(19)
23


Reviews:
[{"review_id": "avJhcOgh5Hn", "reviewer": "Reviewer_vDTh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of comparing reward function without training policies is very appealing for Reinforcement Learning. The authors take a previous method (EPIC) and improve it by making it consider only feasible state transitions. As a results the distance between reward function become more aligned with the trained policies scores.\n\nThe proposed idea is techincally sound and well presented. It is an incremental change based on a previous method, but the authors explain in which possible scenarios it would provide benefits. The evaluation is scritly oriented to validate this claim. The authors compare few reward functions crafted for these scenarios. In my opinion, it is hard to draw (or support) the conclusion from 5 reward functions. It would be very beneficial if the authors could provide a scenario where the DARD compares hundreds (or thousands) of arbitrarily chosen reward functions, and based on its results the authors could train a policy that outperforms the other. This is actually related to a high level comment that I have. Reading the paper, it was not easy for me to understand how the proposed method can be used in a large scale reward designing process. I think that the theme of \"reward designing based on DARD for a very complex (or unknown) problem\" could make the paper more impactful. \n\nThe method assumes access to the dynamics model, and the authors state that this can be approximated (or learned) as well. I believe that an example based on learned dynamics model would make the paper much better.\n", "summary_of_the_paper": "The paper proposes a new reward pseudometric called Dynamics Aware Reward Distance (DARD) which uses approximate transition model of the environment to compare reward functions while being indifferent to reward shaping. Previous work Equivalent Policy Invariant Comparison (EPIC) addresses the same problem, but it uses all possible state transitions feasible or not. The authors show that this can cause problems for reward functions that are technically equivalent in the feasible state transition state, but unequal on the infeasible transitions (which should not matter). The authors evaluate the method in \"bouncing balls\" (a discrete navigation task) and \"reacher\"  (robotic manipulation) environments. The results show that the proposed distance is more accurate compared to EPIC (distance is zero for \"FEASIBILITY\", 2 reward functions that are technically different but practically the same) and the distances are more aligned with the learned policies scores.", "summary_of_the_review": "The paper proposes an incremental method that improves over past work for to ignore the infeasible state transitions. The idea is incremental, but has a good potential. The evaluation could be improved to show the impact of the given method. I think the paper can also use a higher level use-case of reward comparison where the proposed method is used to design a reward function to learn a problem that is otherwise unsuccessful.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RTD-9udJhwu", "reviewer": "Reviewer_4Nse", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. DARD does not relies on the input datasets, and therefore avoids unrelied reward values due to inconsistent samples.\n2. The numerical results show that DARD is more powerful compared to EPIC. In particular, the experiments demonstrate that samples \"out of distribution\" would harm the performance of EPIC significantly.\n\n\nWeaknesses:\n1.  Compared to EPIC, DARD needs access to the transition model $T$ when computing the reward function, which may fail in many practical cases. Even assuming the transition model $T$ is accessable, DARD requires additional samples, which may worse the sample efficiency. \n2.  The theorectical intuition of DARD is not clear. In intuition, the samples \"out of distribution\" leads to inaccuracy. Then the questions are that: how to define a sample \"out of distribution\" and how it harms the performance of EPIC? There are many measurements to describe the inconsistency between the offline distribution and online distribution and I think it is possible to derive some theorectical guarantee using these arguments.\n", "summary_of_the_paper": "The paper proposes a new reward pseudometric (DARD), which is invariant to reward shaping and computationally efficient. Assuming access to the transition model of the environment, DARD could avoid the error due to samples out of the training distribution. The numerical results in simulated physical domains show that DARD outperforms previous methods. ", "summary_of_the_review": "Overall, the paper is well written. My current score is 5 and I look forward to further discussion with the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QNppun44IUP", "reviewer": "Reviewer_LfGd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Pros\n  * The paper clearly identifies an issue with the prior work (EPIC) and proposes an appropriate solution.\n  * The paper is very well-written. \n  * The experiments are well-designed. \n* Cons\n  * The empirical results could be more comprehensive.\n  * The proposed method is a bit incremental to EPIC. \n\n---\n\n* The paper motivates the problem well by clearly identifying the issue with the prior work (EPIC) and proposes a reasonable solution that takes into account the actual dynamics of the environment. \n* The experiments (various manually designed / learned reward functions) are carefully designed to verify the hypotheses of the paper. I also appreciate the authors for acknowledging the limitation of the proposed method with smooth reward functions.\n* The presentation of the paper is excellent. The problem is introduced nicely with appropriate preliminaries, and the main idea is illustrated with a good motivating example (Figure 1).\n* Although the empirical results are good, it could be more comprehensive if the paper showed how the amount of the data affects the results, because the proposed method seems a bit more sensitive to the the amount of transitions in the dataset, because this is how the proposed method taking into account the dynamics. For example, it could be interesting to see how the results change as the amount of data available is varied. \n* Although the idea of considering dynamics for reward comparison in EPIC is novel, the overall method seems a little incremental because the method heavily builds upon EPIC in that it uses a similar canonicalization step followed by the same Pearson distance measure.", "summary_of_the_paper": "This paper considers a reward comparison method in RL, where the goal is to evaluate how close an alternative reward function is to the ground truth reward function in terms of how well the set of optimal policies for each reward function is aligned. The proposed method (DARD) builds upon EPIC [Gleave et al.] and addresses an issue of EPIC by taking into account the actual dynamics of the environment when canonicalizing/transforming reward function as opposed to considering all possible (s, a, r') triples in EPIC. This allows DARD to be more robust to out-of-distribution transitions compared to EPIC while still being invariant to reward shaping like EPIC. The empirical results on Bouncing Balls and Reacher environments show that DARD can represent the reward distance more reliably EPIC does.", "summary_of_the_review": "Although the overall method is still quite similar to EPIC, this paper proposes a reasonable solution to overcome a clear limitation of EPIC. In addition, the experiments are quite well-designed, and the paper is very well-written. \n\n\n---\nUpdate after the rebuttal: Thank you for adding experiments with learned transition models. The results look good. Since my major concern is addressed, I increased my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HjosULUNrgE", "reviewer": "Reviewer_6Mwn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes progress on the important problem of comparing reward functions by addressing a significant shortcoming of the EPIC distance. The paper is clearly written and well-motivated. The theoretical results seem sound based on a quick skim of the proofs. The method is evaluated empirically on a wide range of reward models, and claims H1-H3 seem well-supported by the results. \n\nThe advantage of DARD over EPIC comes at the cost of assuming knowledge of the transition dynamics, which is a limitation for scaling the method to more complex settings where an environment simulator may not be available. It would be great to see how well DARD works for learned transition models as suggested in the conclusion, though I understand this is out of scope for this paper.\n\nHere are some suggestions for improving the paper:\n* The fact that DARD does not perform well on environments like PointMaze where nearby transitions always have similar reward values is an important limitation, and I would suggest adding the PointMaze experiment to the main text of the paper. \n\n* Define a feasibility reward model for the Reacher environment as well, to demonstrate that this shortcoming of EPIC is not specific to one environment.\n\n* To further support hypothesis H3 that DARD is effective across diverse coverage distributions, it may be useful to also test it on a coverage distribution induced by a non-random suboptimal policy. \n\n* It would be good to see the two methods compared on more than two environments, e.g. other MuJoCo environments. \n\n* Clarify the difference between $D_\\rho$ and $D_p$, which seem to both be defined as the Pearson distance\n\n", "summary_of_the_paper": "This paper proposes a novel method for comparing reward functions without policy optimization, the Dynamics-Aware Reward Distance. DARD improves on the state-of-the-art EPIC distance by using an approximate transition model to evaluate reward functions on transitions close to the training distribution, while EPIC evaluates on arbitrary transitions (which can be infeasible). The authors prove that DARD is also invariant to reward shaping, and experimentally demonstrate on the Bouncing Balls and Reacher environments that DARD is a better predictor of policy return than EPIC and works for coverage distributions induced by either random or expert actions. ", "summary_of_the_review": "This paper makes progress on the important problem of comparing reward functions by introducing a new reward distance that addresses a significant shortcoming of the EPIC distance. The results are significant and novel, and I would recommend accepting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new pseudometric, DARD, for comparing reward functions that avoid policy optimization. DARD builds on a recent work by Gleave et al. 2020 where the pseudometric EPIC was proposed. In contrast to EPIC, DARD operates on an approximate transition model and evaluates reward functions only on transitions close to their training distribution. Empirical experiments in different domains demonstrate the effectiveness of the proposed pseudometric. The reviewers acknowledged the importance of the studied problem setting and generally appreciated the results. I want to thank the authors for their detailed responses that helped in answering some of the reviewers' questions and increased their overall assessment of the paper. At the end of the discussion phase, there was a clear consensus that the paper should be accepted. The reviewers have provided detailed feedback in their reviews, and we strongly encourage the authors to incorporate this feedback when preparing a revised version of the paper.
Decision: Accept (Spotlight)