Inverse Reinforcement Learning in a Continuous State Space with Formal Guarantees

1 Introduction . Reinforcement learning in the context of a Markov Decision Process ( MDP ) aims to produce a policy function such that the discounted accumulated reward of an agent following the given policy is optimal . One difficulty in applying this framework to real world problems is that for many tasks of interest , there is no obvious reward function to maximize . For example , a car merging onto a highway must account for many different criteria , e.g. , avoiding crashing and maintaining adequate speed , and there is not an obvious formula of these variables producing a reward which reflects expert behavior [ 3 ] . The Inverse Reinforcement Learning ( IRL ) problem consists of having as input an MDP with no reward function along with a policy , and the goal is to find a reward function so that the MDP plus the learned reward function is optimally solved by the given policy . Describing expert behavior in terms of a reward function has been shown to provide a succinct , robust , and interpretable representation of expert behavior for purposes of reinforcement learning [ 3 ] . Furthermore , this representation of agent preference can give insight on the preferences or goals of an agent . Despite much work over the years , there still exist major limitations in our understanding of IRL . One major gap is the lack of algorithms for the setting of continuous state space . This setting is of primary importance since many promising applications , such as autonomous vehicles and robotics , occur in a continuous setting . Existing algorithms for IRL in the continuous setting generally do not directly solve the IRL problem , but instead take a policy matching approach which leads to a less robust description of agent preference [ 3 ] . Finally , there is a lack of formal guarantees for IRL methods which make principled understanding of the problem difficult . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . For a comprehensive list of results on the IRL problem , we refer the reader to [ 3 ] as we next describe works that are closest to ours . Throughout the years , various approaches have been proposed to solve the IRL problem in the continuous state setting . Aghasadeghi and Bretl [ 1 ] used a finite set of parametric basis functions to model the policy and reward function for a policy matching approach . Boularias et al . [ 6 ] matched the observed policy while maximizing Relative Entropy and applied this method to continuous tasks through discretization . Levine and Koltun [ 12 ] maximized the likelihood of the observed policy while representing the reward function as a Gaussian Process . Finn et al . [ 8 ] modeled the reward function through a neural network to allow for more expressive rewards . Ramponi et al . [ 18 ] and Pirotta and Restelli [ 17 ] search for an optimal reward based on minimizing the gradient of the continuous policy . Metelli et al . [ 14 ] selects a reward function over a set of basis functions such that the policy gradient is zero . These previous methods rely on heuristic arguments and empirical results to support each approach , but do not provide any theoretical guarantees or the insight that comes with them . Finally , Komanduru and Honorio [ 11 ] and Metelli et al . [ 15 ] provided theoretical analysis similar to our work , but only for the finite state space setting . Contributions . In contrast to the work above , this paper provides a formally guaranteed solution to the IRL problem over a continuous state space when transition dynamics of the MDP are unknown . We accomplish this by representing the estimated transition dynamics as an infinite sum of basis functions from R2 → R. Then , under natural conditions on the representations , we prove that a linear program can recover a correct reward function . We develop a model of IRL in the continuous setting using the idea of infinite matrices , which in contrast to the more typical non-parametric functional analysis approach , allow us to derive “ explicit , computable , and meaningful error bounds ” [ 20 ] . We exhibit our main theorem by proving that if the transition functions of the MDP have appropriately bounded partial derivatives , then by representing the transition functions over a trigonometric basis , our algorithm returns a correct reward function with high probability . However , we emphasize that our results apply under a range of appropriate technical conditions and orthonormal bases . To the best of our knowledge , we present the first algorithm for IRL in the continuous setting with formal guarantees on correctness , sample complexity , and computational complexity . The total sample and computational complexity of our algorithm depends on the series representations of the transition dynamics over a given basis . However , if a given set of transition functions can be represented with sufficiently small error over a given basis by a k × k matrix of coefficients , then our algorithm ( see Algorithm 2 ) returns a solution to the IRL problem with probability at least 1− δ in O ( k2 log kδ ) samples by solving a linear program with O ( k ) variables and constraints . We show how the magnitude of k needed , i.e. , the size of the representation , depends on characteristics of a given IRL problem . This sample complexity matches that of the discrete case algorithm of [ 11 ] . For more detailed complexity results , see Section 4 . We experimentally validate our results by testing our IRL algorithm on a problem with random polynomial transition functions . Finally , we show how our algorithm can be efficiently extended to the d-dimensional setting . 2 Preliminaries . In this section , we first define the Markov Decision Process ( MDP ) formalizing the IRL problem . We briefly introduce the Bellman Optimality Equation , which gives necessary and sufficient conditions for a solution to the IRL problem . We then describe the problem of estimating the transition dynamics of the MDP . Finally , we show how to represent the Bellman Optimality Criteria for the given MDP in terms of infinite-matrices and infinite-vectors . For now , we restrict our attention to a one dimensional continuous state space . Definition 1 A Markov Decision Process ( MDP ) is a 5-tuple ( S , A , { Pa ( s′|s ) } a∈A , γ , R ) where • S = [ −1 , 1 ] is an uncountable set of states1 , • A is a finite set of actions , • { Pa ( s′|s ) } a∈A is a set of probability density functions such that Pa ( s′|s ) : S × S → R+ . Pa ( s ′|s ) represents the transition probability density function from state s to state s′ when taking action a , 1While this paper considers S = [ −1 , 1 ] , S can easily be generalize to any interval . • γ ∈ [ 0 , 1 ) is the discount factor , • R : S → R is the reward function . We consider the inverse reinforcement problem . That is , given an MDP without the reward function ( MDP \R ) along with a policy π : S → A , our goal is to determine a reward function R such that the policy π is optimal for the MDP under the learned reward function R. Bellman Optimality Criteria . First we define the value function and Q-function V π ( s0 ) = E [ R ( s0 ) + γR ( s1 ) + γ2R ( s2 ) + ... |π ] , ( 1 ) Qπ ( s , a ) = R ( s ) + γE [ V π ( s′ ) ] , s′ ∼ Pa ( s′|s ) . ( 2 ) Using these terms , we present the Bellman Optimality Equation which gives necessary and sufficient conditions for policy π to be optimal for a given MDP [ 21 ] . π ( s ) ∈ arg max a∈A Qπ ( s , a ) , ∀ s ∈ S. ( 3 ) Estimation Problem . We impose the additional task of estimating the transition dynamics of the MDP . While the policy is assumed to be exactly known , the transition probability density functions { Pa ( s′|s ) } a∈A are unknown . Instead , we assume that for an arbitrary state s , we can sample the one-step transition s′ ∼ Pa ( ·|s ) , as previously done by Komanduru and Honorio [ 11 ] . Infinite Matrix Representation . We now introduce a natural assumption on the transition probability density functions , which will make further analysis possible . Let { φn ( s ) } n∈N be a countably infinite set of orthonormal basis functions over S. That is∫ 1 −1 φ2n ( s ) ds = 1 , and ∫ 1 −1 φi ( s ) φj ( s ) ds = 0 , ∀ i 6= j . Examples of bases that fulfill the above conditions are the Legendre polynomials , the Chebyshev polynomials , and an appropriately scaled trigonometric basis . We assume that the transition probability density functions { Pa ( s′|s ) } a∈A can be represented over the given basis by , Pa ( s ′|s ) = ∞∑ i , j=1 φi ( s ′ ) Z ( a ) ij φj ( s ) , Z ( a ) ij ∈ R. ( 4 ) In Section 5 , we demonstrate sufficient conditions on the transition functions to guarantee that our algorithm is correct when using a trigonometric basis as an example . The following proposition justifies considering only reward functions which are a linear combination of { φn ( s ) } n∈N . Proposition 1 If the transition functions can be represented over { φn ( s ) } n∈N as in Equation 4 and there is a at least one reward function such that the policy is optimal , then there is at least one reward function R equal to a linear combination of { φn ( s ) } n∈N such that the Bellman Optimally criterion is fulfilled for the given policy . ( Missing proofs can be found in the Appendix A . ) This justifies considering only reward functions represented by a series such as , R ( s ) = ∞∑ k=1 αkφk ( s ) , αk ∈ R. ( 5 ) The above conditions are equivalent to requiring that the transition function is a linear operator on the function space spanned by the Schauder basis { φn ( s ) } . Proposition 1 guarantees that if there exists a reward which optimally solves the IRL problem , then there is an optimal reward in the closure of the function space spanned by { φn ( s ) } n∈N . It will be useful to identify the coefficients Z ( a ) ij and αk in Equations 4 and 5 as entries of an infinitematrix and infinite-vector respectively . This identification allows us to keep all the usual definitions of matrix-matrix multiplication , matrix-vector multiplication , matrix transposition , matrix powers , as well as addition and subtraction . We define additional simplifying notation . Let the infinite matrix Z ( a ) be defined as the matrix of coefficients associated with Pa ( s′|s ) . Let the infinite vector α correspond to coefficients of an arbitrary reward function R. We drop the superscript a and write Z where possible . Let [ kZ ] represent the k-truncation of Z . That is , [ kZ ] ij = Zij for all i , j ≤ k , and [ kZ ] ij = 0 otherwise . Define the complement of the k-truncation of Z as [ ∞k Z ] = Z − [ kZ ] . Let φ ( s ) be an infinite-vector where the n-th entry equals φn ( s ) . Note that in contrast to finite matrices , typical objects , such as matrix products , do not always exist in the infinite matrix case . We add additional conditions on the infinite matrices where necessary to ensure that all needed operations are well-defined . This typically amounts to requiring absolute summability of the reward coefficients α and bounded row absolute-summability of the matrix Z ( a ) . We define operations on infinite matrices and vectors to follow typical conventions . Let A and B be infinite matrices and ν be an infinite vector . Then let ‖A‖∞ denote the induced infinity norm defined as ‖A‖∞ = maxi∈N ∑∞ j=1 |Aij | . Let the vector ` 1-norm be defined as ‖ν‖1 = ∑∞ i=1 |νi| and the ` ∞- norm defined as ‖ν‖∞ = maxi∈N |νi| . Let [ A+B ] ij = Aij +Bij . Let the infinite matrix product be defined as [ AB ] ik = ∑∞ j=1AijBjk . Matrix-vector multiplication is defined as [ Aν ] i = ∑∞ j=1Aijνj and [ ν > A ] j = ∑∞ i=1 νiAij . Matrix transposition is defined as [ A > ] ij = Aji . Define matrix power as the repeated product of a matrix , i.e. , Ar = AA ... A , r times . The outer product of infinite vectors ν and µ is denoted as νµ > and equals an infinite matrix such that [ νµ > ] ij = νiµj . These definitions match those of finite matrices and vectors except that the index set is countably infinite . For more information on infinite matrices , we refer the reader to [ 7 ] . Finally , we can give an infinite matrix analogue to the equations of Ng and Russell [ 16 ] to characterize the optimal solutions . We restrict our definition to only strictly optimal policies as motivated by Komanduru and Honorio [ 11 ] . For simplicity , we also assign the given optimal policy π as π ( s ) ≡ a1 , ∀s ∈ S , as done in [ 16 ] and [ 11 ] . Following the same intuition as in the discrete case , we can represent the expectation of a reward function after a sequence of transitions through matrix-matrix and matrix-vector multiplication . Therefore , we can use the following definition to write the difference in Q-functions of the optimal action and any other action under the optimal policy : F ( a ) = ∞∑ r=0 γr ( Z ( a1 ) ) r ( Z ( a1 ) −Z ( a ) ) . ( 6 ) Necessary and sufficient conditions for policy optimality . The following Lemma gives necessary and sufficient conditions for policy π to be strictly optimal for a given MDP of the form described above . The condition is analogous to the condition in [ 16 ] . However , a key difference is that entries of the infinite transition matrices represent coefficients of basis functions and not transition probabilities directly . Therefore , our method of proof is necessarily different . Lemma 2 The given policy π ( s ) is strictly optimal for the MDP ( S , A , { Pa ( s′|s ) } a∈A , γ , R =∑∞ n=1 αnφn ( s ) ) if and only if , α > F ( a ) φ ( s ) > 0 , ∀s ∈ S , ∀ a ∈ A \ { a1 } . Lemma 2 will be used in Section 4 to prove correctness of our main result , Algorithm 2 , as it provides a relatively easy to verify sufficient condition on the reward function representation .

Summary: The paper studies the IRL problem in the presence of a compact state space and finite actions. Under the assumption that the transition model of the MDP can be represented by means of an (infinite) set of orthogonal basis functions, the authors prove that there always exists a reward function explaining the expert's behavior that can be constructed using the same basis functions. Then, the paper proposes an approach for estimating a reward function endowed with sample complexity guarantees. Synthetic experiments are then provided and the paper concludes with a discussion for the extension to multiple dimensions.