Invariance in Policy Optimisation and Partial Identifiability in Reward Learning

Designing reward functions for complex , real-world tasks is challenging . Reward learning lets one instead infer reward functions from data . However , multiple reward functions often fit the data equally well , even in the infinite-data limit . Prior work often considers reward functions to be uniquely recoverable , by imposing additional assumptions on data sources . By contrast , we formally characterise the partial identifiability of popular data sources , including demonstrations and trajectory preferences , under multiple standard sets of assumptions . We analyse the impact of this partial identifiability on downstream tasks such as policy optimisation , including under shifts in environment dynamics . We unify our results in a framework for comparing data sources and downstream tasks by their invariances , with implications for the design and selection of data sources for reward learning . 1 INTRODUCTION . A wide range of problems can be represented as sequential decision-making tasks , where the goal is to maximise some numerical reward ( Sutton & Barto , 2018 ) . However , designing an appropriate reward function remains a challenge in complex real-world tasks ( Amodei et al. , 2016 ; Leike et al. , 2018 ; Dulac-Arnold et al. , 2019 ) . Reward learning algorithms infer task reward functions from data sources such as expert demonstrations ( Ng & Russell , 2000 ) , preferences over trajectories ( Christiano et al. , 2017 ) , and many others ( Jeon et al. , 2020 ) . This approach has extended the applicability of sequential decision-making techniques to more complex tasks ( e.g . Abbeel et al. , 2010 ; Christiano et al. , 2017 ; Singh et al. , 2019 ; Stiennon et al. , 2020 ) . Multiple reward functions are often consistent with the data source , even in the infinite-data limit . For most data sources , this fundamental ambiguity has been acknowledged , but its extent has not been characterised . In section 3 , we formally characterise the ambiguity of several popular data sources including expert demonstrations and trajectory preferences . These infinite-data limits bound the information recoverable from finite data sets using any algorithm , so they are useful for evaluating algorithms relative to their limits , and data sources relative to each other . Uniquely identifying a reward function is unnecessary when all plausible reward functions lead to the same downstream outcome in a given application , such as policy optimisation . Characterising this ambiguity tolerance for various applications allow us to evaluate the ambiguity of a data source relative to a given application . Learnt reward functions are often used for policy optimisation , for example via reinforcement learning ( RL ) 1 . In section 3 we formally characterise the ambiguity tolerance of policy optimisation under arbitrary dynamics . Ambiguity and ambiguity tolerance are formally related . Both concern invariances – of data sources or downstream outcomes – to reward function transformations . Thus , our main contribution is to catalogue the invariances of various mathematical objects derived from the reward function . In section 4 , we explore a partial order on these invariances , and its implications for the selection and evaluation of data sources , addressing an open problem in reward learning ( Leike et al. , 2018 , §3.1 ) . 1We focus on RL applications . Further applications arise in other fields , where reward functions are used in models to understand and predict the behaviour of humans , animals , and other systems ( see , e.g. , Schoemaker , 1982 ; Dennett , 1989 ; Rust , 1994 ; Howes et al. , 2014 ; Peterson et al. , 2021 ; Collins & Shenhav , 2021 ) . 1.1 RELATED WORK . Inverse reinforcement learning ( IRL ; Russell , 1998 ) is the prototypical example of reward learning . IRL infers a reward function from the behavioural data of a task expert by inverting a model of the expert ’ s planning algorithm ( Armstrong & Mindermann , 2017 ; Shah et al. , 2019 ) . Existing work partially characterises the inherent ambiguity of behaviour for certain planning algorithms ( Ng & Russell , 2000 ; Cao et al. , 2021 ) and classes of tasks ( Dvijotham & Todorov , 2010 ; Kim et al. , 2021 ) . We extend these results to more planning algorithms and arbitrary time-unbounded , stochastic tasks , using a more expressive space of reward functions that reveals novel ambiguity . Reward learning models have been proposed for many other data sources ( Jeon et al. , 2020 ) . A popular and effective data source is preferences over behavioural trajectories ( Akrour et al. , 2012 ; Christiano et al. , 2017 ) . Unlike for IRL , the ambiguity arising from these data sources has not been formally characterised . We contribute a formal characterisation of the ambiguity for central models of evaluative feedback including trajectory preferences . Several studies have explored learning from expert behaviour and preferences ( Ibarz et al. , 2018 ; Palan et al. , 2019 ; Bıyık et al. , 2020 ; Koppol et al. , 2020 ) , or other multi-modal data sources ( Tung et al. , 2018 ; Jeon et al. , 2020 ) . One motivation is that different data sources may provide complementary reward information ( Koppol et al. , 2020 ) , eliminating some ambiguity . Similarly , Amin et al . ( 2017 ) and Cao et al . ( 2021 ) observe reduced ambiguity by combining behavioural data across multiple tasks . Our partial order provides a general framework for understanding these results . Computing an optimal behavioural policy is a primary application of learnt reward functions ( Abbeel & Ng , 2004 ; Wirth et al. , 2017 ) . Ng et al . ( 1999 ) proved that potential shaping transformations always preserve the set of optimal policies , and so are always tolerable for this application . We extend this result , characterising the full set of transformations that preserve optimal policies in each task , including for additional policy optimisation techniques such as maximum entropy RL . Ambiguity corresponds to the partial identifiability ( Lewbel , 2019 ) of the reward function modelled as a latent parameter . The prevailing response to partial identifiability in reward learning has been to impose additional constraints or assumptions until the data identifies the reward function uniquely ( or , at least , sufficiently for policy optimisation ) . Following Manski ( 1995 ; 2003 ) and Tamer ( 2010 ) , we instead describe ambiguity given various constraints and assumptions . This gives practitioners results appropriate for their real data ( and the ambiguity tolerance of their actual application ) . IRL is related to dynamic discrete choice ( Rust , 1994 ; Aguirregabiria & Mira , 2010 ) , a problem where identifiability has been extensively studied ( e.g. , Aguirregabiria , 2005 ; Srisuma , 2015 ; Arcidiacono & Miller , 2020 ) . We study a simpler setting with known tasks . IRL also relates to preference elicitation ( Rothkopf & Dimitrakakis , 2011 ) and inverse optimal control ( Ab Azar et al. , 2020 ) . Preferences over sequential trajectories are not typically considered as a data source in other fields . 1.2 PRELIMINARIES . We consider an idealised setting with finite , observable , infinite-horizon sequential decision-making environments , formalised as Markov Decision Processes ( MDPs ; Sutton & Barto , 2018 , §3 ) . An MDP is a tuple pS , A , τ , µ0 , R , γq where S and A are finite sets of environment states and agent actions ; τ : SˆAÑ ∆pSq encodes the transition distributions governing the environment dynamics ; µ0 P ∆pSq is an initial state distribution ; R : SˆAˆS Ñ R is a deterministic reward function2 ; and γ P p0 , 1q is a reward discount rate . We distinguish states in the support of µ0 as initial states , and states s with τps|s , aq “ 1 and Rps , a , sq “ 0 for all a as terminal states . We represent the transition from state s to state s1 using action a as the tuple x “ ps , a , s1q . We classify ps , a , s1q as possible in an MDP if s1 is in the support of τps , aq , otherwise it is impossible . A trajectory is an infinite sequence of concatenate transitions ξ “ ps0 , a0 , s1 , a1 , s2 , . . .q , and a trajectory fragment of length n is a finite sequence of n concatenate transitions ζ “ ps0 , a0 , s1 , . . . , an´1 , snq . A trajectory or fragment is possible if all of its transitions are possible , and is impossible otherwise . A trajectory or fragment is initial if its first state is initial . A state or transition is reachable if it is part of some possible and initial trajectory . Given an MDP , we define the return function G as the cumulative discounted reward of entire trajectories and trajectory fragments : Gpζq “ řn´1 t “ 0 γ tRpst , at , st ` 1q for a trajectory fragment ζ of length n , and similarly for trajectories . We primarily consider this return function with various restricted domains , such as only possible or initial trajectories or trajectory fragments . A policy π : S Ñ ∆pAq encodes an agent ’ s behaviour as a state-conditional action distribution . Together with an MDP ’ s transition distribution τ , a policy π induces a distribution of trajectories starting from each state . We denote such a trajectory starting from s with the random variable Ξs , and its remaining components with random variables A0 , S1 , A1 , S2 , and so on . Given an MDP and a policy π , and the value function encodes the expected return from states , Vπpsq “ EΞs „ π , τ “ GpΞsq ‰ ; and the Q-function of π encodes the expected return given an initial action , Qπps , aq “ EΞs „ π , τ “ GpΞsq ˇ ˇ A0 “ a ‰ . Qπ and Vπ satisfy a Bellman equation : Qπps , aq “ ES1 „ τps , aq “ Rps , a , S1q ` γVπpS1q ‰ , Vπpsq “ EA „ πpsq “ Qπps , Aq ‰ , ( 1 ) for all s P S and a P A . Their difference , Aπps , aq “ Qπps , aq ´ Vπpsq , is the advantage function . We further define a policy evaluation function , J , encoding the expected return from following a particular policy in an MDP , J pπq “ ES0 „ µ0 “ VπpS0q ‰ . J induces an order over policies . A policy maximising J is an optimal policy , denoted π‹ . Similarly , Q‹ , V‹ , and A‹ denote the Q- , value , and advantage functions of an optimal policy . Since J may be multimodal , we often discuss the set of optimal policies . However , Q‹ , V‹ , and A‹ are unique . Moreover , we consider several policies resulting from alternative planning algorithms . Given a base policy π0 , and an inverse temperature parameter β ą 0 , we define the Boltzmann policy with respect to π0 , denoted ππ0β , using the softmax function : ππ0β pa|sq “ exp ` βAπ0ps , aq ˘ ř a1PA exp ` βAπ0ps , a1q ˘ . ( 2 ) The Boltzmann-rational policy , π‹β , is the Boltzmann policy with respect to optimal policies , as used for IRL by Ramachandran & Amir ( 2007 ) . The popular Maximum Entropy policy , πβ , is defined using a modified policy evaluation function with an entropy regularisation term ( Haarnoja et al. , 2017 ) , and solves the recurrence πβ “ π πβ β ( i.e. , πβ is a Boltzmann policy with respect to itself ) . 2Notably , we consider deterministic reward functions that may depend on a transition ’ s successor state . Alternative spaces of reward functions are often considered ( such as functions from S or SˆA , or distributions ) . The chosen space has straightforward consequences for invariances , which we discuss in appendix C .

Summary: This is a theoretical work on understanding the intrinsic limits of various data sources that are used for reward learning in RL. In particular, by considering the infinite data limit of the data source, they study the level of reward ambiguity that can be obtained for a given downstream task. For example, for the expert behavior data source, they characterize the reward transformations that are determined by the optimal Q-function. Similar attempts have been made previously for specific data sources and specific planning algorithms (Ng & Russell, 2000). However, this work makes substantial contributions by conducting this study in a unified and rigorous way for variety data sources and downstream tasks.