Quantifying Differences in Reward Functions

For many tasks , the reward function is inaccessible to introspection or too complex to be specified procedurally , and must instead be learned from user data . Prior work has evaluated learned reward functions by evaluating policies optimized for the learned reward . However , this method can not distinguish between the learned reward function failing to reflect user preferences and the policy optimization process failing to optimize the learned reward . Moreover , this method can only tell us about behavior in the evaluation environment , but the reward may incentivize very different behavior in even a slightly different deployment environment . To address these problems , we introduce the Equivalent-Policy Invariant Comparison ( EPIC ) distance to quantify the difference between two reward functions directly , without a policy optimization step . We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy . Furthermore , we find EPIC can be efficiently approximated and is more robust than baselines to the choice of coverage distribution . Finally , we show that EPIC distance bounds the regret of optimal policies even under different transition dynamics , and we confirm empirically that it predicts policy training success . Our source code is available at https : //github.com/HumanCompatibleAI/evaluating-rewards . 1 INTRODUCTION . Reinforcement learning ( RL ) has reached or surpassed human performance in many domains with clearly defined reward functions , such as games [ 20 ; 15 ; 23 ] and narrowly scoped robotic manipulation tasks [ 16 ] . Unfortunately , the reward functions for most real-world tasks are difficult or impossible to specify procedurally . Even a task as simple as peg insertion from pixels has a non-trivial reward function that must usually be learned [ 22 , IV.A ] . Tasks involving human interaction can have far more complex reward functions that users may not even be able to introspect on . These challenges have inspired work on learning a reward function , whether from demonstrations [ 13 ; 17 ; 26 ; 8 ; 3 ] , preferences [ 1 ; 25 ; 6 ; 18 ; 27 ] or both [ 10 ; 4 ] . Prior work has usually evaluated the learned reward function R̂ using the “ rollout method ” : training a policy πR̂ to optimize R̂ and then examining rollouts from πR̂ . Unfortunately , using RL to compute πR̂ is often computationally expensive . Furthermore , the method produces false negatives when the reward R̂ matches user preferences but the RL algorithm fails to optimize with respect to R̂ . The rollout method also produces false positives . Of the many reward functions that induce the desired rollout in a given environment , only a small subset align with the user ’ s preferences . For example , suppose the agent can reach states { A , B , C } . If the user prefers A > B > C , but the agent instead learns A > C > B , the agent will still go to the correct state A . However , if the initial state distribution or transition dynamics change , misaligned rewards may induce undesirable policies . For example , if A is no longer reachable at deployment , the previously reliable agent would misbehave by going to the least-favoured state C. We propose instead to evaluate learned rewards via their distance from other reward functions , and summarize our desiderata for reward function distances in Table 1 . For benchmarks , it is usually possible to directly compare a learned reward R̂ to the true reward function R. Alternatively , benchmark creators can train a “ proxy ” reward function from a large human data set . This proxy can then be used as a stand-in for the true reward R when evaluating algorithms trained on a different or smaller data set . ∗Work partially conducted while at DeepMind . Distance Pseudometric Invariant Efficient Robust Predictive . Comparison with a ground-truth reward function is rarely possible outside of benchmarks . However , even in this challenging case , comparisons can at least be used to cluster reward models trained using different techniques or data . Larger clusters are more likely to be correct , since multiple methods arrived at a similar result . Moreover , our regret bound ( Theorem 4.9 ) suggests we could use interpretability methods [ 12 ] on one model and get some guarantees for models in the same cluster . We introduce the Equivalent-Policy Invariant Comparison ( EPIC ) distance that meets all the criteria in Table 1 . We believe EPIC is the first method to quantitatively evaluate reward functions without training a policy . EPIC ( section 4 ) canonicalizes the reward functions ’ potential-based shaping [ 14 ] , then takes the correlation between the canonical rewards over a coverage distribution D of transitions . We also introduce baselines NPEC and ERC ( section 5 ) which partially satisfy the criteria . EPIC works best when D has support on all realistic transitions . We achieve this in our experiments by using uninformative priors , such as rollouts of a policy taking random actions . Moreover , we find EPIC is robust to the exact choice of distribution D , producing similar results across a range of distributions , whereas ERC and especially NPEC are highly sensitive to the choice of D ( section 6.2 ) . Moreover , low EPIC distance between a learned reward R̂ and the true reward R predicts low regret . That is , the policies πR̂ and πR optimized for R̂ and R obtain similar returns under R. Theorem 4.9 bounds the regret even in unseen environments ; by contrast , the rollout method can only determine regret in the evaluation environment . We also confirm this result empirically ( section 6.3 ) . 2 RELATED WORK . There exist a variety of methods to learn reward functions . Inverse reinforcement learning ( IRL ) [ 13 ] is a common approach that works by inferring a reward function from demonstrations . The IRL problem is inherently underconstrained : many different reward functions lead to the same demonstrations . Bayesian IRL [ 17 ] handles this ambiguity by inferring a posterior over reward functions . By contrast , Maximum Entropy IRL [ 26 ] selects the highest entropy reward function consistent with the demonstrations ; this method has scaled to high-dimensional environments [ 7 ; 8 ] . An alternative approach is to learn from preference comparisons between two trajectories [ 1 ; 25 ; 6 ; 18 ] . T-REX [ 4 ] is a hybrid approach , learning from a ranked set of demonstrations . More directly , Cabi et al . [ 5 ] learn from “ sketches ” of cumulative reward over an episode . To the best of our knowledge , there is no prior work that focuses on evaluating reward functions directly . The most closely related work is Ng et al . [ 14 ] , identifying reward transformations guaranteed to not change the optimal policy . However , a variety of ad-hoc methods have been developed to evaluate reward functions . The rollout method – evaluating rollouts of a policy trained on the learned reward – is evident in the earliest work on IRL [ 13 ] . Fu et al . [ 8 ] refined the rollout method by testing on a transfer environment , inspiring our experiment in section 6.3 . Recent work has compared reward functions by scatterplotting returns [ 10 ; 4 ] , inspiring our ERC baseline ( section 5.1 ) . 3 BACKGROUND . This section introduces material needed for the distances defined in subsequent sections . We start by introducing the Markov Decision Process ( MDP ) formalism , then describe when reward functions induce the same optimal policies in an MDP , and finally define the notion of a distance metric . Definition 3.1 . A Markov Decision Process ( MDP ) M = ( S , A , γ , d0 , T , R ) consists of a set of states S and a set of actions A ; a discount factor γ ∈ [ 0 , 1 ] ; an initial state distribution d0 ( s ) ; a transition distribution T ( s′ | s , a ) specifying the probability of transitioning to s′ from s after taking action a ; and a reward function R ( s , a , s′ ) specifying the reward upon taking action a in state s and transitioning to state s′ . A trajectory τ = ( s0 , a0 , s1 , a1 , · · · ) consists of a sequence of states si ∈ S and actions ai ∈ A . The return on a trajectory is defined as the sum of discounted rewards , g ( τ ; R ) = ∑|τ | t=0 γ tR ( st , at , st+1 ) , where the length of the trajectory |τ | may be infinite . In the following , we assume a discounted ( γ < 1 ) infinite-horizon MDP . The results can be generalized to undiscounted ( γ = 1 ) MDPs subject to regularity conditions needed for convergence . A stochastic policy π ( a | s ) assigns probabilities to taking action a ∈ A in state s ∈ S . The objective of an MDP is to find a policy π that maximizes the expected return G ( π ) = Eτ ( π ) [ g ( τ ; R ) ] , where τ ( π ) is a trajectory generated by sampling the initial state s0 from d0 , each action at from the policy π ( at | st ) and successor states st+1 from the transition distribution T ( st+1 | st , at ) . An MDP M has a set of optimal policies π∗ ( M ) that maximize the expected return , π∗ ( M ) = arg maxπ G ( π ) . In this paper , we consider the case where we only have access to an MDP\R , M− = ( S , A , γ , d0 , T ) . The unknown reward function R must be learned from human data . Typically , only the state space S , action space A and discount factor γ are known exactly , with the initial state distribution d0 and transition dynamics T only observable from interacting with the environment M− . Below , we describe an equivalence class whose members are guaranteed to have the same optimal policy set in any MDP\R M− with fixed S , A and γ ( allowing the unknown T and d0 to take arbitrary values ) . Definition 3.2 . Let γ ∈ [ 0 , 1 ] be the discount factor , and Φ : S → R a real-valued function . Then R ( s , a , s′ ) = γΦ ( s′ ) − Φ ( s ) is a potential shaping reward , with potential Φ [ 14 ] . Definition 3.3 ( Reward Equivalence ) . We define two bounded reward functions RA and RB to be equivalent , RA ≡ RB , for a fixed ( S , A , γ ) if and only if there exists a constant λ > 0 and a bounded potential function Φ : S → R such that for all s , s′ ∈ S and a ∈ A : RB ( s , a , s ′ ) = λRA ( s , a , s ′ ) + γΦ ( s′ ) − Φ ( s ) . ( 1 ) Proposition 3.4 . The binary relation≡ is an equivalence relation . LetRA , RB , RC : S×A×S → R be bounded reward functions . Then ≡ is reflexive , RA ≡ RA ; symmetric , RA ≡ RB implies RB ≡ RA ; and transitive , ( RA ≡ RB ) ∧ ( RB ≡ RC ) implies RA ≡ RC . Proof . See section A.3.1 in supplementary material . The expected return of potential shaping γΦ ( s′ ) − Φ ( s ) on a trajectory segment ( s0 , · · · , sT ) is γTΦ ( sT ) − Φ ( s0 ) . The first term γTΦ ( sT ) → 0 as T → ∞ , while the second term Φ ( s0 ) only depends on the initial state , and so potential shaping does not change the set of optimal policies . Moreover , any additive transformation that is not potential shaping will , for some reward R and transition distribution T , produce a set of optimal policies that is disjoint from the original [ 14 ] . The set of optimal policies is invariant to constant shifts c ∈ R in the reward , however this can already be obtained by shifting Φ by cγ−1 . * Scaling a reward function by a positive factor λ > 0 scales the expected return of all trajectories by λ , also leaving the set of optimal policies unchanged . If RA ≡ RB for some fixed ( S , A , γ ) , then for any MDP\R M− = ( S , A , γ , d0 , T ) we have π∗ ( ( M− , RA ) ) = π ∗ ( ( M− , RB ) ) , where ( M− , R ) denotes the MDP specified by M− with reward function R. In other words , RA and RB induce the same optimal policies for all initial state distributions d0 and transition dynamics T . Definition 3.5 . Let X be a set and d : X ×X → [ 0 , ∞ ) a function . d is a premetric if d ( x , x ) = 0 for all x ∈ X . d is a pseudometric if , furthermore , it is symmetric , d ( x , y ) = d ( y , x ) for all x , y ∈ X ; and satisfies the triangle inequality , d ( x , z ) ≤ d ( x , y ) + d ( y , z ) for all x , y , z ∈ X . d is a metric if , furthermore , for all x , y ∈ X , d ( x , y ) = 0 =⇒ x = y . We wish for d ( RA , RB ) = 0 whenever the rewards are equivalent , RA ≡ RB , even if they are not identical , RA 6= RB . This is forbidden in a metric but permitted in a pseudometric , while retaining * Note constant shifts in the reward of an undiscounted MDP would cause the value function to diverge . Fortunately , the shaping γΦ ( s′ ) − Φ ( s ) is unchanged by constant shifts to Φ when γ = 1. other guarantees such as symmetry and triangle inequality that a metric provides . Accordingly , a pseudometric is usually the best choice for a distance d over reward functions .

Summary: The paper introduces a pseudometric on reward functions, EPIC (Equivalent-Policy Invariant Comparison), based on the potential-based reward shaping (Ng. 2020). It formally analyzes the EPIC distance in detail and demonstrates its usefulness in comparing learned reward functions without the necessity of optimizing reward-specific policies. The empirical results show that the EPIC is more predictive of the policy returns than some baseline variants and robust to visitation distributions, even in unseen test environments.