Sample Efficient Imitation Learning for Continuous Control

1 INTRODUCTION . Recent advances in reinforcement learning ( RL ) have achieved super-human performance on several domains ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Mnih et al. , 2016 ; Lillicrap et al. , 2015 ) . On most of such domains with the success of RL , the design of reward , that explains what agent ’ s behavior is favorable , is obvious for humans . Conversely , on domains where it is unclear how to design the reward , agents trained by RL algorithms often obtain poor policies and behave worse than what we expect them to do . Imitation learning ( IL ) comes in such cases . The goal of IL is to enable the learner to imitate expert behavior given the expert demonstrations without the reward signal . We are interested in IL because we desire an algorithm that can be applied to real-world problems for which it is often hard to design the reward . In addition , since it is generally hard to model a variety of real-world environments with an algorithm , and the state-action pairs in a vast majority of realworld applications such as robotics control can be naturally represented in continuous spaces , we focus on model-free IL for continuous control . A wide variety of IL methods have been proposed in the last few decades . The simplest IL method among those is behavioral cloning ( BC ) ( Pomerleau , 1991 ) which learns an expert policy in a supervised fashion without environment interactions during training . BC can be the first IL option when enough demonstration is available . However , when only a limited number of demonstrations are available , BC often fails to imitate the expert behavior because of the problem which is referred to compounding error ( Ross & Bagnell , 2010 ) – inaccuracies compound over time and can lead the learner to encounter unseen states in the expert demonstrations . Since it is often hard to obtain a large number of demonstrations in real-world environments , BC is often not the best choice for real-world IL scenarios . Another widely used approach , which overcomes the compounding error problem , is Inverse Reinforcement Learning ( IRL ) ( Russell , 1998 ; Ng & Russell , 2000 ; Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) . Recently , Ho & Ermon ( 2016 ) have proposed generative adversarial imitation learning ( GAIL ) which is based on prior IRL works . Since GAIL has achieved state-of-the-art performance on a variety of continuous control tasks , the adversarial IL ( AIL ) framework has become a popular choice for IL ( Baram et al. , 2017 ; Hausman et al. , 2017 ; Li et al. , 2017 ) . It is known that the AIL methods are more sample efficient than BC in terms of the expert demonstration . However , as pointed out by Ho & Ermon ( 2016 ) , the existing AIL methods have sample complexity in terms of the environment interaction . That is , even if enough demonstration is given by the expert before training the learner , the AIL methods require a large number of state-action pairs obtained through the interaction between the learner and the environment1 . The sample complexity keeps existing AIL from being employed to real-world applications for two reasons . First , the more an AIL method requires the interactions , the more training time it requires . Second , even if the expert safely demonstrated , the learner may have policies that damage the environments and the learner itself during training . Hence , the more it performs the interactions , the more it raises the possibility of getting damaged . For the real-world applications , we desire algorithms that can reduce the number of interactions while keeping the imitation capability satisfied as well as the existing AIL methods do . The following three properties of the existing AIL methods which may cause the sample complexity in terms of the environment interactions : ( a ) Adopting on-policy RL methods which fundamentally have sample complexity in terms of the environment interactions . ( b ) Alternating three optimization processes — learning reward functions , value estimation with learned reward functions , and RL to update the learner policy using the estimated value . In general , as the number of parameterized functions which are related to each other increases , the training progress may be unstable or slower , and thus more interactions may be performed during training . ( c ) Adopting Gaussian policy as the learner ’ s stochastic policy , which has infinite support on a continuous action space . In common IL settings , we observe action space of the expert policy from the demonstration where the expert action can take on values within a bounded ( finite ) interval . As Chou & Scherer . ( 2017 ) suggests , the policy which can select actions outside the bound may slow down the training progress and make the problem harder to solve , and thus more interactions may be performed during training . In this paper , we propose an IL algorithm for continuous control to improve the sample complexity of the existing AIL methods . Our algorithm is made up mainly three changes to the existing AIL methods as follows : ( a ) Adopting off-policy actor-critic ( Off-PAC ) algorithm ( Degris et al. , 2012 ) to optimize the learner policy instead of on-policy RL algorithms . Off-policy learning is commonly known as the promising approach to improve the complexity . ( b ) Estimating the state-action value using off-policy samples without learning reward functions instead of using on-policy samples with the learned reward functions . Omitting the reward learning reduces functions to be optimized . It is expected to make training progress stable and faster and thus reduce the number of interactions during training . ( c ) Representing the stochastic policy function of which outputs are bounded instead of adopting Gaussian policy . Bounding action values may make the problem easier to solve and make the training faster , and thus reduce the number of interactions during training . Experimental results show that our algorithm enables the learner to imitate the expert behavior as well as GAIL does while significantly reducing the environment interactions . Ablation experimental results show that ( a ) adopting the off-policy scheme requires about 100 times fewer environment interactions to imitate the expert behavior than the one on-policy IL algorithms require , ( b ) omitting the reward learning makes the training stable and faster , and ( c ) bounding action values makes the training faster . 1Throughout this paper , we refer to “ number of interactions “ as the number of state-action pairs obtained through interaction between the learner and the environment during training the learner . 2 BACKGROUND . 2.1 PRELIMINARIES . We consider a Markov Decision Process ( MDP ) which is defined as a tuple { S , A , T , R , d0 , γ } , where S is a set of states , A is a set of possible actions agents can take , T : S×A×S → [ 0 , 1 ] is a transition probability , R : S×A → R is a reward function , d0 : S → [ 0 , 1 ] is a distribution over initial states , and γ ∈ [ 0 , 1 ) is a discount factor . The agent ’ s behavior is defined by a stochastic policy π : S×A → [ 0 , 1 ] and Π denotes a set of the stochastic policies . We denote SE ⊂ S and AE ⊂ A as sets of states and actions observed in the expert demonstration , and Sπ ⊂ S and Aπ ⊂ A as sets of those observed in rollouts following a policy π . We will use πE , πθ , β ∈ Π to refer to the expert policy , the learner policy parameterized by θ , and a behavior policy , respectively . Given a policy π , performance measure of π is defined as J ( π , R ) = E [ ∑∞ t=0 γ tR ( st , at ) |d0 , T , π ] where st ∈ S is a state that the agent receives at discrete time-step t , and at ∈ A is an action taken by the agent after receiving st . The performance measure indicates expectation of the discounted return ∑∞ t=0 γ tR ( st , at ) when the agent follows the policy π in the MDP . Using discounted state visitation distribution denoted by ρπ ( s ) = ∑∞ t=0 γ t P ( st = s|d0 , T , π ) where P is a probability that the agent receives the state s at time-step t , the performance measure can be rewritten as J ( π , R ) = Es∼ρπ , a∼π [ R ( s , a ) ] . The state-action value function for the agent following π is defined as Qπ ( st , at ) = E [ ∑∞ u=t γ u−tR ( su , au ) |T , π ] , and Qπ , ν denotes its approximator parameterized by ν . 2.2 ADVERSARIAL IMITATION LEARNING . We briefly describe objectives of RL , IRL , and AIL below . We refer the readers to Ho & Ermon ( 2016 ) for details . The goal of RL is to find an optimal policy that maximizes the performance measure . Given the reward function R , the objective of RL with parameterized stochastic policies πθ : S×A→ [ 0 , 1 ] is defined as follows : RL ( R ) = argmaxθ J ( πθ , R ) ( 1 ) The goal of IRL is to find a reward function based on an assumption that the discounted returns earned by the expert behavior are greater than or equal to those earned by any non-experts behavior . Technically , the objective of IRL is to find reward functions Rω : S × A → R parameterized by ω that satisfies J ( πE , Rω ) ≥ J ( π , Rω ) where π denotes the non-expert policy . The existing AIL methods adopt max-margin IRL ( Abbeel & Ng , 2004 ) of which objective can be defined as follows : IRL ( πE ) = argmaxω J ( πE , Rω ) − J ( π , Rω ) ( 2 ) The objective of AIL can be defined as a composition of the objectives ( 1 ) and ( 2 ) as follows : AIL ( πE ) = argminθ argmaxω J ( πE , Rω ) − J ( πθ , Rω ) ( 3 ) 2.3 OFF-POLICY ACTOR-CRITIC . The objective of Off-PAC to train the learner can be described as follows : argmaxθ Es∼ρβ , a∼πθ [ Qπθ , ν ( s , a ) ] ( 4 ) The learner policy is updated by taking the gradient of the state-action value . Degris et al . ( 2012 ) proposed the gradient as follows : Es∼ρβ , a∼πθ [ Qπθ , ν ( s , a ) ∇θ log πθ ( a|s ) ] ( 5 ) Heess et al . ( 2015 ) provided another formula of the gradient using “ re-parameterization trick “ in the case that the learner policy selects the action as a = πθ ( s , z ) with random variables z ∼ Pz generated by a distribution Pz : Es∼ρβ , z∼Pz [ ∇aQπθ , ν ( s , a ) |a=πθ ( s , z ) ∇θπθ ( s , z ) ] ( 6 )

Summary: This paper proposes a new method to imitate expert efficiently. The paper first proposes a way to compute reward function from expert demonstration and uses the log probability to represent this reward function.  Then they find a form of bellman equation that can optimize the reward stably. After the 'Q learning without IRL', an off-policy RL off-pac is applied. So this paper achieves comparable results to GAIL but uses much less data amount.