Learning from Demonstrations with Energy based Generative Adversarial Imitation Learning

1 INTRODUCTION . Motivated by applying reinforcement learning algorithms into more realistic tasks , we find that most realistic environments can not feed an explicit reward signal back to the agent immediately . It becomes a bottleneck for traditional reinforcement learning methods to be applied into more realistic scenarios . So how to infer the latent reward function from expert demonstrations is of great significance . Recently , a lot of great work have been proposed to solve this problem . They are also successfully applied in scientific inquiries , such as Stanford autonomous helicopter Abbeel et al . ( 2006 ) Abbeel et al . ( 2007 ) Ng et al . ( 2004 ) Coates et al . ( 2008 ) Abbeel et al . ( 2008a ) Abbeel et al . ( 2010 ) , as well as practical challenges such as navigation Ratliff et al . ( 2006 ) Abbeel et al . ( 2008b ) Ziebart et al . ( 2008 ) Ziebart et al . ( 2010 ) and intelligent building controls Barrett & Linder ( 2015 ) . The goal of imitation learning is to mimic the expert behavior from expert demonstrations without access to a reinforcement signal from the environment . The algorithms in this field can be divided into two board categories : behavioral cloning and inverse reinforcement learning . Behavioral cloning formulate this problem as a supervised learning problem which aims at mapping state action pairs from expert trajectories to policy . These methods suffer from the problem of compounding errors ( covariate shift ) which only learn the actions of the expert but not reason about what the expert is trying to achieve . By the contrast , inverse reinforcement learning recovers the reward function from expert demonstrations and then optimize the policy under such an unknown reward function . In this paper , we propose energy-based generative adversarial imitation learning which views the discriminator as an energy function without explicit probabilistic interpretation . The energy function computed by the discriminator can be viewed as a trainable cost function for the generator , while the discriminator is trained to assign low energy values to the regions of expert demonstrations , and higher energy values outside these regions . We use an auto-encoder to represent the discriminator , and the reconstruction error is thought to be the energy . There are many other choices to learn the energy function , but an auto-encoder is quite efficient . Our main contributions are summarized as follows . • An EB-GAIL framework with the discriminator using an auto-encoder architecture in which the energy is the reconstruction error . • Theoretical analysis shows that the policy rolls out the trajectories that are indistinguishable from the distribution of the expert demonstrations by matching the occupancy measure with the expert policy . • Experiments show that EB-GAIL outperforms several SoTA imitation learning algorithms while the training process for EB-GAIL can be more stable . 2 BACKGROUND . In this section , we ’ ll briefly introduce the basic concepts in direct reinforcement learning Sutton & Barto ( 1998 ) Sugiyama ( 2015 ) , inverse reinforcement learning Ng & Russell ( 2000 ) , imitation learning Bagnell ( 2015 ) , and energy based models LeCun et al . ( 2006 ) . 2.1 DIRECT REINFORCEMENT LEARNING . Reinforcement Learning Sutton & Barto ( 1998 ) Sugiyama ( 2015 ) , which is usually used for sequential decision making problems , can help us to learn from the interactions with the environment . The process for direct reinforcement learning is that at each time step , the agent receives a state st and chooses an action at from the action space A , following a stochastic policy π ( at|st ) . After that , the environment transit to a new state st+1 and gives a scalar reward rt back to the agent . This process continues until the agent reaches a terminal state . In this case , the training model can be thought as a Markov decision process ( MDP ) , which is a tuple ( S , A , T , γ , D , R ) . In this tuple , S is the state space ; A is the action space ; T = Psa is a probability matrix for state transitions , owing to the environment dynamics ; γ ∈ ( 0 , 1 ] is a discount factor ; D is the initial-state transition distribution ; and R : S → A is the reward function , which is assumed to be bounded in absolute value by 1 . In addition , we also define that a policy π is a mapping from states to probability distributions over actions , which is also called a stochastic policy . For a certain task , the goal of direct reinforcement learning is to maximize the total future reward . For simplification , we define the value function to be a prediction of the total future reward , which can be shown as a discounted future reward : V = ∑∞ t=0 γ tRt . Besides , we also define the action value function as Q ( s , a ) = E [ Rt|st = s , at = a ] , which is the expected return for selecting action a in state s. According to the Bellman Optimality , an optimal action value function Q∗ ( s , a ) is the maximum action value achievable by any policy for state s and action a . 2.2 INVERSE REINFORCEMENT LEARNING . The goal of inverse reinforcement learning is to infer the reward signal with respect to the expert demonstrations which are assumed to be the observations of optimal behaviors Ng & Russell ( 2000 ) . In the past decade , a lot of great work have been proposed towards enlarging the ability of reward function representation . Take some traditional methods for example , in 2010 , FIRL Levine & Popvić ( 2010 ) was proposed to learn a set of composites features based on logical conjunctions with nonlinear functions for the reward signal . Later , non-parametric methods based on Gaussian Process ( GP ) Rasmussen & Williams ( 2006 ) are proposed to enlarge the function space of latent reward to allow for non-linearity in Levine & Popvić ( 2010 ) . For undertaking the learning of an abstract structure with smaller data sets , Jin et al . Jin & Spanos ( 2015 ) combined deep belief networks and gaussian process to optimize the existing algorithm GP-IRL . To solve the substantial noise in the sensor measurements , some work have been applied here such as bayesian programming , probabilistic graphical model and so on . For example , in 2007 , Ramachandran et al . proposed a Bayesian nonparametric approach Ramachandran & Amir ( 2007 ) to construct the reward function features in IRL , which is so-called Bayesian IRL . Later , Choi et al . Choi & Kim ( 2014 ) extend this algorithm by defining a prior on the composite features , which are defined to be the logical conjunctions of the atomic features . By assuming that each trajectory can be generated by multiple locally consistent reward functions , Nguyen et al . used an expectation-maximization ( EM ) algorithm to learn the different reward functions and the stochastic transitions between them in order to jointly improve the likelihood of the expert ’ s demonstrated trajectories Nguyen et al . ( 2015 ) . As a result , the most likely partition of a trajectory into segments that are generated from different locally consistent reward functions selected by EM can be derived . Experiments show that the algorithm outperforms the SoTA EM clustering with maximum likelihood IRL . 2.3 IMITATION LEARNING . Imitation learning is a study of algorithms that can mimic the experts ’ demonstrations or a teachers ’ behavior . Unlike inverse reinforcement learning , the goal of imitation learning is to obtain a policy from teachers ’ behaviors rather than to recover the reward function for some certain tasks . The algorithms in imitation learning can be classified into two categories : behavioral cloning , and inverse reinforcement learning . One fatal weakness for behavioral cloning is that these methods can only learn the actions from the teacher rather than learn the motivation from teachers ’ behaviors . To solve this problem , inverse reinforcement learning was proposed to recover the reward function for decision making problems . By assuming that the teachers ’ behavior is optimal , these methods tend to recover the reward function in a Markov decision process . So when combined with direct reinforcement learning methods , inverse reinforcement learning can realize the process for mimicking the teachers ’ behaviors . 2.4 ENERGY BASED MODEL . The essence of the energy based model LeCun et al . ( 2006 ) is to build a function that maps each point of an input space to a single scalar , which is called “ energy ” . The learning phase is a data driven process that shapes the energy surface in such a way that the desired configurations get assigned low energies , while the incorrect ones are given high energies . Supervised learning falls into this framework : for each x in the training set , the energy of the pair ( x , y ) takes low values when y is the correct label and higher values for incorrect y ’ s . Similarly , when modeling x alone within an unsupervised learning setting , lower energy is attributed to the data manifold . The term contrastive sample is often used to refer to a data point causing an energy pull up , such as the incorrect y ’ s in supervised learning and points from low data density regions in unsupervised learning . Denoted that the energy function is ε , the connection between probability and energy can be built through Gibbs distribution : P ( y|x ) = exp ( −βε ( y , x ) ) ∫ y∈Y exp ( −βε ( y , x ) ) , ( 1 ) the denominator here is the partition function which represents the total energy in the data space and β is an arbitrary positive constant . The formulation of this connection might seem arbitrary , but other formulation can be obtained by re-defining the energy function . 3 ENERGY BASED GENERATIVE ADVERSARIAL IMITATION LEARNING . ( EB-GAIL ) 3.1 METHODOLOGY . The output of the discriminator goes through an objective functional in order to shape the energy function , assigning low energy to the regions near the expert demonstrations and higher energy to the other regions . In this work , we use an auto-encoder to represent the discriminator and the reconstruction error of the auto-encoder is assumed to be the energy . Meanwhile , we use a margin loss function to train EB-GAIL while one loss function is to train the discriminator and the other loss function is assumed to be the reward function for the reinforcement learning procedure . Given a positive margin margin , a state-action pair χE sampled from expert demonstrations , and a state-action pair χi rolled out by a trained policy , the discriminator loss function LD is formally defined by : LD = D ( χE ) + [ margin−D ( χi ) ] + ( 2 ) where [ · ] + = max ( 0 , · ) . Meanwhile , the reward function for the reinforcement learning procedure is : r ( χi ) = −D ( χi ) ( 3 ) Maximizing the total reward for the reinforcement learning is similar to minimizing the second term of LD . In practice , we observe that the loss function can effectively avoid gradient vanishing and mode collapse problems .

Summary: The authors propose a discriminator-based approach to inverse reinforcement learning (IRL). The discriminator function is trained to attain large values ("energy") on trajectories from the current policy and small values on trajectories from an expert policy. The current policy is then improved by using the negative discriminator as a reward signal. The specific discriminator suggested is an autoencoder loss. The authors continue to provide a proof that assuming their discriminator/generator attain a Nash equilibrium, the occupancy measure of the trained policy matches that of the expert policy. They follow up with demonstrating better performance of their approach compared to certain baselines when tested on a number of tasks on Physics simulators.