Identifiability in inverse reinforcement learning

1 Introduction . Inverse reinforcement learning aims to use observations of agents ’ actions to determine their reward function . The problem has roots in the very early stages of optimal control theory ; Kalman [ 1964 ] raised the question of whether , by observation of optimal policies , one can recover coefficients of a quadratic cost function ( see also Boyd et al . [ 1994 ] ) . This question naturally generalizes to the generic framework of Markov decision process and stochastic control . In the 1970s , these questions were taken up within economics , as a way of determining utility functions from observations . For instance , Keeney and Raiffa [ 1976 ] set out to determine a proper ordering of all possible states which are deterministic functions of actions . In this setup , the problem is static and the outcome of an action is immediate . Later in Sargent [ 1978 ] , a dynamic version of a utility assessment problem was studied , under the context of finding the proper wage through observing dynamic labor demand . As exemplified by Lucas ’ critique1 , in many applications it is not enough to find some pattern of rewards corresponding to observed policies ; instead we may need to identify the specific rewards agents face , as it is only with this information that we can make valid predictions for their actions in a changed environment . In other words , we do not simply wish to learn a reward which allows us to imitate agents in the current environment , but which allows us to predict their actions in other settings . 1The critique is best summarized by the quotation : “ Given that the structure of an econometric model consists of optimal decision rules of economic agents , and that optimal decision rules vary systematically with changes in the structure of series relevant to the decision maker , it follows that any change in [ regulatory ] policy will systematically alter the structure of econometric models. ” ( Lucas [ 1976 ] ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we give a precise characterization of the range of rewards which yield a particular policy for an entropy regularized Markov decision problem . This separates the main task of estimation ( of the optimal policy from observed actions ) from the inverse problem ( of inferring rewards from a given policy ) . We find that even with perfect knowledge of the optimal policy , the corresponding rewards are not fully identifiable ; nevertheless , the space of consistent rewards is parameterized by the value function of the control problem . In other words , the reward can be fully determined given the optimal policy and the value function , but the optimal policy gives us no direct information about the value function . We further show that , given knowledge of the optimal policy under two different discount rates , or sufficiently different transition laws , we can uniquely identify the rewards ( up to a constant shift ) . We also give conditions under which action-independent rewards , or time-homogenous rewards over finite horizons , can be identified . This demonstrates the fundamental challenge of inverse reinforcement learning , which is to disentangle immediate rewards from future rewards ( as captured through preferences over future states ) . 2 Background on reinforcement learning . The motivation behind inverse reinforcement learning is to use observed agent behavior to identify the rewards motivating agents . Given these rewards , one can forecast future behavior , possibly under a different environment . In a typical reinforcement learning2 ( RL ) problem , an agent learns an optimal policy to maximize her total reward by interacting with the environment . In order to analyse the inverse reinforcement learning problem , we begin with an overview of the ‘ primal ’ problem , that is , how to determine optimal policies given rewards . We particularly highlight a entropy regularized version of the Markov decision process ( MDP ) , which provides a better-posed setting for inverse reinforcement learning . For mathematical simplicity , we focus on discrete-time problems with finitely many states and actions ; our results can largely be transferred to continuous settings , with fundamentally the same proofs , however some technical care is needed . 2.1 Discrete Markov decision processes with entropy regularization . The environment . We consider a simple Markov decision process ( MDP ) on an infinite horizon . The MDP M = ( S , A , T , f , γ ) is described by : a finite state space S ; a finite set of actions A ; a ( Markov ) transition kernel T : S × A → P ( S ) , that is , a function T such that T ( s , a ) gives probabilities3 of each value of St+1 , given the state St = s and action At = a at time t ; and a reward function f : S ×A → R with discount factor γ ∈ [ 0 , 1 ) . An agent aims to choose a sequence of actions { A0 , A1 , ... } from A in order to to maximize the expected value of total reward ∑∞ t=0 γ tf ( St , At ) . It will prove convenient for us to allow randomized policies π , that is , functions π : S → P ( A ) , where π ( ·|s ) is the distribution of actions the agent takes when in state s. For a given randomized policy π , we define Tπ ∈ P ( S ) , the distribution of St+1 given state St , by Tπ ( St+1 = s′|St = s ) = ∑ a∈A T ( s′|s , a ) π ( a|s ) . Given an initial distribution ρ ∈ P ( S ) for S0 and a policy π : S → P ( A ) , we obtain a ( unique ) probability measure Pπρ such that , and any a ∈ A , s , s′ ∈ S , Pπρ ( S0 = s ) = ρ ( s ) and Pπρ ( At = a|St = s ) = π ( a|s ) , Pπρ ( St+1 = s′|St = s , At = a ) = T ( s′|s , a ) , for all t. We write Eπρ for the corresponding expectation and Eπs when the initial state is given by s ∈ S . The classic objective in a MDP is to maximize the expected value Eπs [ ∑∞ t=0 γ tf ( St , At ) ] . With this 2Reinforcement learning and optimal control are closely related problems , where reinforcement learning typically focuses on the challenge of numerically learning a good control policy , while optimal control focuses on the description of the optimizer . In the context of the inverse problem we consider they are effectively equivalent and we will use the terms interchangeably . 3Here , and elsewhere , we write P ( X ) for the set of all probability distributions on a set X . objective , one can show ( for example , see Bertsekas and Shreve [ 2004 ] , Puterman [ 2014 ] ) that there is an optimal deterministic control ( i.e . a policy π , taking values zero and one , which maximizes the expected value ) . This implies that , typically , an optimal agent will only make use of a single action for each state , and the choice of this action will not vary smoothly with changes in the reward , discount rate , or transition kernel . Entropy regularised MDP . Given the lack of smoothness in the classical MDP , and to encourage exploration , a well-known variation on the classic MDP introduces a regularization term based on the Shannon entropy . Given a policy π and regularization coefficient λ ≥ 0 , the entropy regularized value of a policy π , when starting in state s , is defined by V πλ ( s ) : = Eπs [ ∞∑ t=0 γt ( f ( st , at ) −λ log ( π ( at|st ) ) ) ] = Eπs [ ∞∑ t=0 γt ( f ( st , at ) +λH ( π ( ·|st ) ) ) ) ] . ( 1 ) Here H ( π ) = − ∑ a∈A π ( a ) log ( π ( a ) ) is the entropy of π . We call this setting the regularised MDPMλ = ( S , A , T , f , γ , λ ) . The optimal value is given by V ∗λ ( s ) : = maxπ V πλ ( s ) , where the maximum is taken over all ( randomized feedback4 ) policies π : S → P ( A ) . We define the state-action value of π at ( s , a ) ∈ S ×A by Qπλ ( s , a ) = f ( s , a ) + γ ∑ s′∈S T ( s′|s , a ) V πλ ( s′ ) . ( 2 ) The dynamic programming principle ( e.g . Haarnoja et al . [ 2017 , Theorem 2 ] ) gives V ∗λ ( s ) = max m∈P ( A ) [ ∑ a∈A ( f ( s , a ) − λ log ( m ( a ) ) + γEs1∼T ( ·|s , a ) [ V ∗λ ( s1 ) ] ) m ( a ) ] = λ max m∈P ( A ) [ 1 λ ∑ a∈A ( f ( s , a ) + γEs1∼T ( ·|s , a ) [ V ∗λ ( s1 ) ] ) m ( a ) +H ( m ) ] . ( 3 ) Observing that on the right hand side we are maximizing over a linear function in m plus an entropy term , and applying [ Dupuis and Ellis , 2011 , Proposition 1.4.2 ] , we have that for any s ∈ S V ∗λ ( s ) = V π∗λ λ ( s ) = λ log ∑ a∈A e 1 λ ( f ( s , a ) +γEs1∼T ( ·|s , a ) [ V π∗λ λ ( s1 ) ] ) = λ log ∑ a∈A exp ( 1 λ Q π∗λ λ ( s , a ) ) , ( 4 ) and the maximum in ( 3 ) is achieved by the randomized policy m ( a ) = π∗λ ( a|s ) , where π∗λ ( a|s ) = exp ( 1 λQ π∗λ λ ( s , a ) ) ∑ a′∈A exp ( 1 λQ π∗λ λ ( s , a ′ ) ) for a ∈ A . ( 5 ) From ( 4 ) we see that exp ( V ∗λ ( s ) /λ ) = ∑ a∈A exp ( Q π∗λ λ ( s , a ) /λ ) and so we can write the optimal policy as π∗λ ( a|s ) = exp ( ( Q π∗λ λ ( s , a ) − V ∗ λ ( s ) ) / λ ) = exp ( ( f ( s , a ) + Es1∼T ( ·|s , a ) [ γV ∗ λ ( s1 ) − V ∗λ ( s ) ] ) / λ ) , ( 6 ) From this analysis , we make the following observations regarding the regularized MDP : • The optimal policy will select all actions in A with some positive probabilities . • If λ is increased , this has the effect of ‘ flattening out ’ the choice of actions , as seen in the softmax function in ( 5 ) . Conversely , sending λ→ 0 will result in a true maximizer being chosen , and the regularized problem degenerates to the classical MDP . • Adding a constant to the reward does not change the policy . Remark 1 . In many modern approaches , one replaces dependence on the state with dependence on a space of ‘ features ’ . This has benefits when fitting a model , but does not significantly change the problem considered . 4Given the Markov structure there is no loss of generality when restricting to policies of feedback form . Further , by replacing A with the set of maps S → A if necessary , all feedback controls a ( s ) can be written as deterministic controls a ( · ) in a larger space , so when convenient we can consider controls which do not depend on the state without loss of generality .

Summary: This work presents theory and experiments on the problem of reward function non-identifiability in IRL. The key result is Theorem 1, which entails that the reward function (in the entropy-regularized MDP setting) cannot be uniquely identified given an unknown value function. Following this line of reasoning, the authors argue that it is possible to identify the reward function (up to a constant) provided we can disentangle immediate rewards from preferences over future states (e.g., with data from two agents with different discount factors in the LQG setting). This result is also demonstrated empirically using a small example.