Identifiability in inverse reinforcement learning

1 Introduction . Inverse reinforcement learning aims to use observations of agents ’ actions to determine their reward function . The problem has roots in the very early stages of optimal control theory ; Kalman [ 1964 ] raised the question of whether , by observation of optimal policies , one can recover coefficients of a quadratic cost function ( see also Boyd et al . [ 1994 ] ) . This question naturally generalizes to the generic framework of Markov decision process and stochastic control . In the 1970s , these questions were taken up within economics , as a way of determining utility functions from observations . For instance , Keeney and Raiffa [ 1976 ] set out to determine a proper ordering of all possible states which are deterministic functions of actions . In this setup , the problem is static and the outcome of an action is immediate . Later in Sargent [ 1978 ] , a dynamic version of a utility assessment problem was studied , under the context of finding the proper wage through observing dynamic labor demand . As exemplified by Lucas ’ critique1 , in many applications it is not enough to find some pattern of rewards corresponding to observed policies ; instead we may need to identify the specific rewards agents face , as it is only with this information that we can make valid predictions for their actions in a changed environment . In other words , we do not simply wish to learn a reward which allows us to imitate agents in the current environment , but which allows us to predict their actions in other settings . 1The critique is best summarized by the quotation : “ Given that the structure of an econometric model consists of optimal decision rules of economic agents , and that optimal decision rules vary systematically with changes in the structure of series relevant to the decision maker , it follows that any change in [ regulatory ] policy will systematically alter the structure of econometric models. ” ( Lucas [ 1976 ] ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we give a precise characterization of the range of rewards which yield a particular policy for an entropy regularized Markov decision problem . This separates the main task of estimation ( of the optimal policy from observed actions ) from the inverse problem ( of inferring rewards from a given policy ) . We find that even with perfect knowledge of the optimal policy , the corresponding rewards are not fully identifiable ; nevertheless , the space of consistent rewards is parameterized by the value function of the control problem . In other words , the reward can be fully determined given the optimal policy and the value function , but the optimal policy gives us no direct information about the value function . We further show that , given knowledge of the optimal policy under two different discount rates , or sufficiently different transition laws , we can uniquely identify the rewards ( up to a constant shift ) . We also give conditions under which action-independent rewards , or time-homogenous rewards over finite horizons , can be identified . This demonstrates the fundamental challenge of inverse reinforcement learning , which is to disentangle immediate rewards from future rewards ( as captured through preferences over future states ) . 2 Background on reinforcement learning . The motivation behind inverse reinforcement learning is to use observed agent behavior to identify the rewards motivating agents . Given these rewards , one can forecast future behavior , possibly under a different environment . In a typical reinforcement learning2 ( RL ) problem , an agent learns an optimal policy to maximize her total reward by interacting with the environment . In order to analyse the inverse reinforcement learning problem , we begin with an overview of the ‘ primal ’ problem , that is , how to determine optimal policies given rewards . We particularly highlight a entropy regularized version of the Markov decision process ( MDP ) , which provides a better-posed setting for inverse reinforcement learning . For mathematical simplicity , we focus on discrete-time problems with finitely many states and actions ; our results can largely be transferred to continuous settings , with fundamentally the same proofs , however some technical care is needed . 2.1 Discrete Markov decision processes with entropy regularization . The environment . We consider a simple Markov decision process ( MDP ) on an infinite horizon . The MDP M = ( S , A , T , f , γ ) is described by : a finite state space S ; a finite set of actions A ; a ( Markov ) transition kernel T : S × A → P ( S ) , that is , a function T such that T ( s , a ) gives probabilities3 of each value of St+1 , given the state St = s and action At = a at time t ; and a reward function f : S ×A → R with discount factor γ ∈ [ 0 , 1 ) . An agent aims to choose a sequence of actions { A0 , A1 , ... } from A in order to to maximize the expected value of total reward ∑∞ t=0 γ tf ( St , At ) . It will prove convenient for us to allow randomized policies π , that is , functions π : S → P ( A ) , where π ( ·|s ) is the distribution of actions the agent takes when in state s. For a given randomized policy π , we define Tπ ∈ P ( S ) , the distribution of St+1 given state St , by Tπ ( St+1 = s′|St = s ) = ∑ a∈A T ( s′|s , a ) π ( a|s ) . Given an initial distribution ρ ∈ P ( S ) for S0 and a policy π : S → P ( A ) , we obtain a ( unique ) probability measure Pπρ such that , and any a ∈ A , s , s′ ∈ S , Pπρ ( S0 = s ) = ρ ( s ) and Pπρ ( At = a|St = s ) = π ( a|s ) , Pπρ ( St+1 = s′|St = s , At = a ) = T ( s′|s , a ) , for all t. We write Eπρ for the corresponding expectation and Eπs when the initial state is given by s ∈ S . The classic objective in a MDP is to maximize the expected value Eπs [ ∑∞ t=0 γ tf ( St , At ) ] . With this 2Reinforcement learning and optimal control are closely related problems , where reinforcement learning typically focuses on the challenge of numerically learning a good control policy , while optimal control focuses on the description of the optimizer . In the context of the inverse problem we consider they are effectively equivalent and we will use the terms interchangeably . 3Here , and elsewhere , we write P ( X ) for the set of all probability distributions on a set X . objective , one can show ( for example , see Bertsekas and Shreve [ 2004 ] , Puterman [ 2014 ] ) that there is an optimal deterministic control ( i.e . a policy π , taking values zero and one , which maximizes the expected value ) . This implies that , typically , an optimal agent will only make use of a single action for each state , and the choice of this action will not vary smoothly with changes in the reward , discount rate , or transition kernel . Entropy regularised MDP . Given the lack of smoothness in the classical MDP , and to encourage exploration , a well-known variation on the classic MDP introduces a regularization term based on the Shannon entropy . Given a policy π and regularization coefficient λ ≥ 0 , the entropy regularized value of a policy π , when starting in state s , is defined by V πλ ( s ) : = Eπs [ ∞∑ t=0 γt ( f ( st , at ) −λ log ( π ( at|st ) ) ) ] = Eπs [ ∞∑ t=0 γt ( f ( st , at ) +λH ( π ( ·|st ) ) ) ) ] . ( 1 ) Here H ( π ) = − ∑ a∈A π ( a ) log ( π ( a ) ) is the entropy of π . We call this setting the regularised MDPMλ = ( S , A , T , f , γ , λ ) . The optimal value is given by V ∗λ ( s ) : = maxπ V πλ ( s ) , where the maximum is taken over all ( randomized feedback4 ) policies π : S → P ( A ) . We define the state-action value of π at ( s , a ) ∈ S ×A by Qπλ ( s , a ) = f ( s , a ) + γ ∑ s′∈S T ( s′|s , a ) V πλ ( s′ ) . ( 2 ) The dynamic programming principle ( e.g . Haarnoja et al . [ 2017 , Theorem 2 ] ) gives V ∗λ ( s ) = max m∈P ( A ) [ ∑ a∈A ( f ( s , a ) − λ log ( m ( a ) ) + γEs1∼T ( ·|s , a ) [ V ∗λ ( s1 ) ] ) m ( a ) ] = λ max m∈P ( A ) [ 1 λ ∑ a∈A ( f ( s , a ) + γEs1∼T ( ·|s , a ) [ V ∗λ ( s1 ) ] ) m ( a ) +H ( m ) ] . ( 3 ) Observing that on the right hand side we are maximizing over a linear function in m plus an entropy term , and applying [ Dupuis and Ellis , 2011 , Proposition 1.4.2 ] , we have that for any s ∈ S V ∗λ ( s ) = V π∗λ λ ( s ) = λ log ∑ a∈A e 1 λ ( f ( s , a ) +γEs1∼T ( ·|s , a ) [ V π∗λ λ ( s1 ) ] ) = λ log ∑ a∈A exp ( 1 λ Q π∗λ λ ( s , a ) ) , ( 4 ) and the maximum in ( 3 ) is achieved by the randomized policy m ( a ) = π∗λ ( a|s ) , where π∗λ ( a|s ) = exp ( 1 λQ π∗λ λ ( s , a ) ) ∑ a′∈A exp ( 1 λQ π∗λ λ ( s , a ′ ) ) for a ∈ A . ( 5 ) From ( 4 ) we see that exp ( V ∗λ ( s ) /λ ) = ∑ a∈A exp ( Q π∗λ λ ( s , a ) /λ ) and so we can write the optimal policy as π∗λ ( a|s ) = exp ( ( Q π∗λ λ ( s , a ) − V ∗ λ ( s ) ) / λ ) = exp ( ( f ( s , a ) + Es1∼T ( ·|s , a ) [ γV ∗ λ ( s1 ) − V ∗λ ( s ) ] ) / λ ) , ( 6 ) From this analysis , we make the following observations regarding the regularized MDP : • The optimal policy will select all actions in A with some positive probabilities . • If λ is increased , this has the effect of ‘ flattening out ’ the choice of actions , as seen in the softmax function in ( 5 ) . Conversely , sending λ→ 0 will result in a true maximizer being chosen , and the regularized problem degenerates to the classical MDP . • Adding a constant to the reward does not change the policy . Remark 1 . In many modern approaches , one replaces dependence on the state with dependence on a space of ‘ features ’ . This has benefits when fitting a model , but does not significantly change the problem considered . 4Given the Markov structure there is no loss of generality when restricting to policies of feedback form . Further , by replacing A with the set of maps S → A if necessary , all feedback controls a ( s ) can be written as deterministic controls a ( · ) in a larger space , so when convenient we can consider controls which do not depend on the state without loss of generality .

Summary: The paper considers the problem of reward identifiability in a maximum entropy inverse optimal control setting when access to the expert policy is available. It is shown that the reward function can only be recovered up to a state-dependent shaping term (which relates to the temporal credit assignment). If access to the optimal policies for two variants (different discount factor or different transition model) of a given MDP is available, it is shown that the reward function can be recovered (up to a constant). The results are validated on numerical experiments on a finite MDP, with known transition model policies.