Stochastic Inverse Reinforcement Learning 

1 INTRODUCTION . The IRL problem addresses an inverse problem that a set of expert demonstrations determines a reward function over a Markov decision process ( MDP ) if the model dynamics are known Russell ( 1998 ) ; Ng et al . ( 2000 ) . The recovered reward function provides a succinct , robust , and transferable definition of the learning task and completely determines the optimal policy . However , the IRL problem is ill-posed that the policy may be optimal for many reward functions and expert demonstrations may be optimal for many policies . For example , all policies are optimal for a constant reward function . In a real-world scenario , experts always act sub-optimally or inconsistently , which is another challenge . To overcome these limitations , two classes of probabilistic approaches for the IRL problem are proposed , i.e. , Bayesian inverse reinforcement learning ( BIRL ) Ramachandran & Amir ( 2007 ) based on Bayesians ’ maximum a posteriori ( MAP ) estimation and maximum entropy IRL ( MaxEnt ) Ziebart et al . ( 2008 ) ; Ziebart ( 2010 ) based on frequentists ’ maximum likelihood ( MLE ) estimation . BIRL solves for the distribution of reward functions without an assumption that experts behave optimally , and encode the external a priori information in a choice of a prior distribution . However , BIRL also suffers from the practical limitation that a large number of algorithmic iterations is required for the procedure of Markov chain Monte Carlo ( MCMC ) in a sampling of posterior over reward functions . Advanced techniques , for example Kernel technique Michini & How ( 2012 ) and gradient method Choi & Kim ( 2011 ) , are proposed to improve the efficiency and tractability of this situation . MaxEnt employs the principle of maximum entropy to resolve the ambiguity in choosing demonstrations over a policy . This class of methods , inheriting the merits from previous non-probabilistic IRL approaches including Ng et al . ( 2000 ) ; Abbeel & Ng ( 2004 ) ; Ratliff et al . ( 2006 ) ; Abbeel et al . ( 2008 ) ; Syed & Schapire ( 2008 ) ; Ho et al . ( 2016 ) , imposes regular structures of reward functions in a combination of hand-selected features . Formally , the reward function is a linear or nonlinear combination of the feature basis functions which consists of a set of real-valued functions { φi ( s , a ) } i=1 hand-selected by experts . The goal of this approach is to find the best-fitting weights of feature basis functions through the MLE approach . Wulfmeier et al . ( 2015 ) and Levine et al . ( 2011 ) use deep neural networks and Gaussian processes to fit the parameters based on demonstrations respectively but still suffer from the problem that the true reward shaped by the changing environment dynamics . Influenced by the work of Finn et al . ( 2016a ; b ) , Fu et al . ( 2017 ) propose a framework called adversarial IRL ( AIRL ) to recover robust reward functions in a changing dynamics based on adversarial learning and achieves superior results . Compared with AIRL , another adversarial method called generative adversarial imitation learning ( GAIL ) Ho & Ermon ( 2016 ) seeks to directly recover the expert ’ s policy rather than reward functions . Many follow-up methods enhance and extend GAIL for multipurpose in various application scenarios Li et al . ( 2017 ) ; Hausman et al . ( 2017 ) ; Wang et al . ( 2017 ) . However , GAIL is in a lack of an explanation of expert ’ s behavior and a portable representation for the knowledge transfer which are the merits of the class of the MaxEnt approach , because the MaxEnt approach is equipped with the `` transferable '' regular structures over reward functions . In this paper , under the framework of the MaxEnt approach , we propose a generalized perspective of studying the IRL problem called stochastic inverse reinforcement learning ( SIRL ) . It is formulated as an expectation optimization problem aiming to recover a probability distribution over the reward function from expert demonstrations . The solution of SIRL is succinct and robust for the learning task in the meaning that it can generate more than one weight over feature basis functions which compose alternative solutions to the IRL problem . Benefits of the class of the MaxEnt method , the solution to our generalized problem SIRL is also transferable . Since of the intractable integration in our formulation , we employ the Monte Carlo expectation-maximization ( MCEM ) approach Wei & Tanner ( 1990 ) to give the first solution to the SIRL problem in a model-based environment . In general , the solutions to the IRL problem are not always best-fitting in the previous approaches because a highly nonlinear inverse problem with the limited information is very likely to get trapped in a secondary maximum in the recovery . Taking advantage of the Monte Carlo mechanism of a global exhaustive search , our MCEM approach avoids the secondary maximum and theoretically convergent demonstrated by pieces of literature Caffo et al . ( 2005 ) ; Chan & Ledolter ( 1995 ) . Our approach is also quickly convergent because of the preset simple geometric configuration over weight space in which we approximate it with a Gaussian Mixture Model ( GMM ) . Hence , our approach works well in a real-world scenario with a small and variability set of expert demonstrations . In particular , the contributions of this paper are threefold : 1 . We generalize the IRL problem to a well-posed expectation optimization problem SIRL . 2 . We provide the first theoretically existing solution to SIRL by the MCEM approach . 3 . We show the effectiveness of our approach by comparing the performance of the proposed method to those of the previous algorithms on the objectworld . 2 PRELIMINARY . An MDP is a tupleM : = 〈S , A , T , R , γ〉 , where S is the set of states , A is the set of actions , and the transition function ( a.k.a . model dynamics ) T : = P ( st+1 = s′ ∣∣st = s , at = a ) for s , s′ ∈ S and a ∈ A is the probability of being current state s , taking action a and yielding next state s′ . Reward functionR ( s , a ) is a real-valued function and γ ∈ [ 0 , 1 ) is the discount factor . A policy π : S → A is deterministic or stochastic , where the deterministic one is written as a = π ( s ) , and the stochastic one is as a conditional distribution π ( a|s ) . Sequential decisions are recorded in a series of episodes which consist of states s , actions a , and rewards r. The goal of reinforcement learning aims to get optimal policy π∗ for maximizing the expected total reward , i.e . π∗ : = arg max π E [ ∞∑ t=0 γt · R ( st , at ) ∣∣∣π ] . Given an MDP without a reward functionR , i.e. , MDP\R=〈S , A , T 〉 , and m expert demonstrations ζE : = { ζ1 , · · · , ζm } . Each expert demonstration ζi is a sequential of state-action pairs . The goal of the IRL problem is to estimate the unknown reward functionR ( s , a ) from expert demonstrations ζE and the reward functions compose a complete MDP . The estimated complete MDP yields an optimal policy that acts as closely as the expert demonstrations . 2.1 REGULAR STRUCTURE OF REWARD FUNCTIONS . In this section , we provide a formal definition of the regular ( linear/nonlinear ) structure of reward functions . The linear structure Ng et al . ( 2000 ) ; Ziebart et al . ( 2008 ) ; Syed & Schapire ( 2008 ) ; Ho et al . ( 2016 ) is an M linear combination of feature basis functions , written as , R ( s , a ) : = M∑ i αi · φi ( s , a ) , where φi : S ×A 7−→ Rd are a d-dimensional feature functions hand-selected by experts . The nonlinear structure Wulfmeier et al . ( 2015 ) is of the form as follows , R ( s , a ) : = N ( φ1 ( s , a ) , · · · , φM ( s , a ) ) , where N is neural networks of hand-crafted reward feature basis functions { φi ( s , a ) } Mi=1 . In the following section , we propose an optimization problem whose solution is a probability distribution over weights of basis functions { φi ( s , a ) } i=1 . 2.2 PROBLEM STATEMENT . Given an MDP\R : = ( S , A , T , γ ) with a known transition function T : = P ( st+1 = s′ ∣∣st = s , at = a ) for s , s′ ∈ S and a ∈ A and a hand-crafted reward feature basis function { φi ( s ) } Mi=1 . A stochastic regular structure on the reward function assumes weightsW of the reward feature functions φi ( s ) , which are random variables with a reward conditional probability distributionDR ( W|ζE ) conditional on expert demonstrations ζE . Parametrizing DR ( W|ζE ) with parameter Θ , our aim is to estimate the best-fitting parameter Θ∗ from the expert demonstrations ζE , such that DR ( W|ζE , Θ∗ ) more likely generates weights to compose reward functions as the ones derived from expert demonstrations , which is called stochastic inverse reinforcement learning problem . Suppose a representative trajectory class CE satisfies that each trajectory element set O ∈ CE is a subset of expert demonstrations ζE with the cardinality at least · |ζE | , where is a preset threshold and |ζE | is the number of expert demonstrations , written as , CE : = { O ∣∣∣O ⊂ ζE with |O| ≥ · |ζE | } . Integrate out unobserved weightsW , and the SIRL problem is formulated to estimate parameter Θ on an expectation optimization problem over the representative trajectory class as follows : Θ∗ : = arg max Θ EO∈CE [ ∫ W fM ( O , W|Θ ) dW ] , ( 1 ) where trajectory element set O assumes to be uniformly distributed for the sake of simplicity in this study , and fM is the conditional joint probability density function of trajectory element O and weightsW for reward feature functions conditional on parameter Θ . 2.2.1 NOTE : . • Problem 1 is well-posed and typically not intractable analytically . • Trajectory element set O is usually known from the rough estimation of the statistics in expert demonstrations in practice . • We introduce representative trajectory class to overcome the limitation of the sub-optimality of expert demonstrations ζE Ramachandran & Amir ( 2007 ) , which is also called a lack of sampling representativeness in statistics Kruskal & Mosteller ( 1979 ) , e.g. , driver ’ s demonstrations encode his own preferences in driving style but may not reflect the true rewards of an environment . With the technique of representative trajectory class , each subset of driver ’ s demonstrations ( i.e . trajectory element set O ) constitutes a subproblem in Problem 1 . 3 METHODOLOGY . In this section , we propose a novel approach to estimate the best-fitting parameter Θ∗ in Problem 1 , which is called the two-stage hierarchical method , a variant of MCEM method .

Summary: The paper proposes a novel method for inverse reinforcement learning: inferring a (distribution over) reward functions from a set of expert demonstrations. Prior work has either learned a point-estimate, notably maximum entropy IRL, or used Bayesian methods to learn a probability distribution over reward functions. Maximum entropy IRL has scaled to complex environments with unknown dynamics and non-linear rewards (with methods such as AIRL), but do not learn a probability distribution. By contrast, Bayesian IRL is more theoretically principled, but has not scaled to complex environments or non-linear rewards. This paper performs maximum likelihood estimation of a parameter for a *generative model* over probability distributions, using a Monte-Carlo expectation-maximization (MCEM) method. It therefore still outputs a probability distribution like Bayesian IRL, but is able to learn non-linear rewards unlike prior Bayesian methods.