SIMILE: Introducing Sequential Information towards More Effective Imitation Learning

1 INTRODUCTION . In recent years , Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) has been widely used in many research areas . Different from supervised learning ( SL ) in which a set of input/output pairs are provided , the fundamental idea of RL is to interact with an environment and learn from trial-anderror . The goal of an RL algorithm is to find a policy that maximizes the total reward in a complete episode . However , as the learning task becomes more and more complex , researchers often encounter the difficulty in defining a reasonable reward function . To alleviate this issue , imitation learning ( IL ) was proposed ( Schaal , 1999 ) , which instead provides a few expert demonstrations for the agent to learn from . Based on the assumption that the expert is always the best , there are two types of IL approaches , known as behavioral cloning ( BC ) ( Abbeel & Ng , 2004 ) and inverse reinforcement learning ( IRL ) ( Ng & Russell , 2000 ) , respectively1 . Both of them train the agent so that its behavior is indistinguishable from that of the expert – BC tries to mimic the behavior of the expert , and IRL aims at recovering the underlying reward function so that its inner-loop , an RL algorithm , can learn from it . A noticeable challenge of IL lies in the limited amount of expert demonstrations , mainly due to the potential time and financial costs especially in real-world environments . On the other hand , conven- 1 Throughout this paper , we use “ imitation learning ” ( IL ) to name all the RL methods which learn from expert demonstrations . Among them , all methods which explicitly recover the reward function are named “ inverse reinforcement learning ” ( IRL ) , otherwise “ behavioral cloning ” ( BC ) . For example , GAIL ( Ho & Ermon , 2016 ) belongs to BC because it bypassed the reward function and directly generate the agent policy . tional IL approaches often interpret the behavior of an expert using individual state-action pairs . We argue that this introduces ambiguity and noise into the training process , as one single state may be insufficient to provide all necessary information for the agent to make a decision . For example , in playing a video game ( Bellemare et al. , 2013 ) , each state contains a still image , which does not tell the moving direction and speed of each object . In such cases , the expert can take completely different actions under the “ same ” situation ( two states seem identical in the defined representation , but the expert can refer to extra information and realize the difference ) , which confuses the agent and introduces additional noise to the training process . Please refer to Section 3.2 for detailed examples . This paper provides a straightforward approach named Sequential IMItation LEerning ( SIMILE ) , which allows the agent to see sequential information ( i.e. , past state-action pairs ) in order to acquire extra information – as an intuitive example , in playing a video game , the moving direction and speed of an object can be predicted by seeing a few consecutive states . Therefore , we formulate the policy into a recurrent model , and instantiate it using an LSTM ( Hochreiter & Schmidhuber , 1997 ) which effectively fuses long-term and short-term information . SIMILE is a generalized algorithm which can be instantiated in two important examples of IL , namely IRL and BC . In either case , we also fuse the information produced by each LSTM cell using AdaBoost ( Freund & Schapire , 1997 ) , a sequential classifier which adjusts the weight of each state according to its importance . We evaluate SIMILE in OpenAI Gym ( Brockman et al. , 2016 ) , a popular benchmarking environment for RL algorithms . We investigate eight robot controlling tasks , varying from basic objects to complex motion systems . Beyond both IRL and BC baselines , our approach achieves consistent overall performance gain – in some tasks , the trained agent performs comparably or even better than the expert . SIMILE also enjoys two additional benefits . First , the training process converges faster , which not only remedies the drawback in running speed , but also reduces the number of interaction with the environment which can be expensive . Second , the testing performance becomes more stable in terms of the standard deviation of total reward , suggesting that the agent becomes more reliable . Both these advantages imply that our approach makes use of training data more effectively , and owe to the exploitation of sequential information . Moreover , these advantages also suggest that our approach is more friendly to training an RL system in real-world environments . The remainder of this paper is organized as follows . Section 2 briefly reviews related work . Section 3 introduces the SIMILE approach and its application to IRL and BC pipelines . After experiments are shown in Section 4 , we conclude this work in Section 5 . 2 RELATED WORK . Reinforcement learning ( RL ) ( Kaelbling et al. , 1996 ; Sutton & Barto , 1998 ) is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward2 . It implies , and has made significant progresses in a wide range of applications , including robot navigation ( Zhu et al. , 2017 ) , urban route planning ( Kuderer et al. , 2015 ) , training multi-agent systems ( Mnih et al. , 2015 ) , playing video games ( Bellemare et al. , 2013 ) and chess games ( Silver et al. , 2016 ) , etc . Compared to supervised learning ( SL ) which provides a set of input-output pairs , RL often trains an agent by facilitating it to interact with an environment and get trained from trial-and-error . An RL system is often formulated by a Markov Decision Process ( MDP ) in which a reward function is defined at each state transition , and the goal is to maximize the total reward in a pre-defined task , i.e. , a complete episode . One of the key issues in RL is to sample effective training data from the environment , for which a few popular algorithms were proposed , such as Monte-Carlo sampling ( Sutton & Barto , 1998 ; Dosovitskiy & Koltun , 2017 ) , temporal differencing ( Sutton , 1988 ; Amiranashvili et al. , 2018 ) and deep Q-learning ( Mnih et al. , 2015 ; Wang et al. , 2015 ) . To improve training efficiency , off-policy or asynchronous methods were also designed ( Schulman et al. , 2015 ; Mnih et al. , 2016 ; Schulman et al. , 2017 ) for parallel optimization . The strategy of combining different sampling tricks achieves superior performance was studied in ( Hessel et al. , 2017 ) . One of the key factors of RL is to define a reward function , however , this is difficult in many complex scenes . Imitation learning ( IL ) was then proposed to deal with this problem . The core idea of IL is to provide a few expert demonstrations , and ask the agent to mimic the behavior of the expert . 2 https : //en.wikipedia.org/wiki/Reinforcement learning There are mainly two types of methods in IL . The first type is named behavioral cloning ( BC ) or apprenticeship learning ( Abbeel & Ng , 2004 ; Abbeel et al. , 2010 ; Hadfield-Menell et al. , 2016 ) , in which a set of state-action pairs was extracted from the expert demonstrations , and the agent learned from these data like the supervised setting . This method can be generalized to optimizing an agent policy that is indistinguishable from the expert policy ( Ho et al. , 2016 ) , and thus be formulated in the form of a generative adversarial network ( GAN ) ( Ho & Ermon , 2016 ) . The second type called inverse reinforcement learning ( IRL ) ( Ng & Russell , 2000 ) instead aimed at recovering the underlying reward function from expert demonstrations , so that a standard RL algorithm can learn from the reward function . Therefore , IRL is often equipped with an inner-loop which is an RL algorithm , and the training process alternatively updates the agent policy and the reward function , assuming that the expert always makes the optimal choice . Previous studies have included max-margin or max-entropy approaches ( Ziebart et al. , 2008 ; Wulfmeier et al. , 2015 ) as well as probabilistic approaches ( Ramachandran & Amir , 2007 ; Levine et al. , 2011 ) into IRL . The equivalence of IRL and GAN was later proved in ( Finn et al. , 2016 ) , which leads to adversarially training an IRL system towards stronger transfer ability ( Fu et al. , 2018 ) . This work aims at introducing sequential information into both IRL and BC frameworks . This recalls the family of recurrent neural networks ( RNNs ) ( Hopfield , 1982 ; Graves et al. , 2009 ) , and a popular instantiation of RNNs named long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) which has shown its ability of incorporating sequential information in many problems such as natural language processing ( Kumar et al. , 2016 ) , video processing ( Donahue et al. , 2015 ) and reinforcement learning ( Parisotto & Salakhutdinov , 2018 ) . Also , the sequential outputs from LSTM need to be combined into a discriminator , which is related to the problem of fusing several weak classifiers into a strong one . This problem was investigated in Adaptive Boosting ( AdaBoost ) ( Freund & Schapire , 1997 ) and its variants such as the multi-class AdaBoost ( Hastie et al. , 2009 ) . 3 APPROACH . 3.1 BACKGROUND . The environment of an RL system can be defined as a quintuple 〈S , A , P , γ , R〉 abstracting a finitestate Markov Decision Process ( MDP ) . S denotes the space of all possible states , and A denotes the set of actions . Given a pair of state s ∈ S and action a ∈ A , P ( · | s , a ) is the state transition function , i.e. , P ( s′ | s , a ) is the probability that another state s′ is achieved when the agent takes action a at state s. As a special case , P0 ( s0 ) . = P0 ( s0 | ∅ ) determines the probability of the first state s0 . γ ∈ [ 0 , 1 ] is named discounted factor measuring how much the action of the current state impacts a future reward . R : S × A 7→ R denotes the reward function to measure the benefit of a state-action pair . R is often bounded by Rmax in its absolute value : |R ( · , · ) | 6 Rmax . The goal of RL is to learn a policy , i.e. , a function π : S ×A 7→ [ 0 , 1 ] which defines the probability of each action a to be taken by the agent under state s. Given a policy π and a starting state s0 , the MDP can sample an action at each time t ∈ N and then go to the next random state : at ∼ π ( st , · ) , st+1 ∼ P ( · | st , at ) . ( 1 ) This produces a trajectory τ = { s0 , a0 , . . . , sT−1 , aT−1 , sT } , and we denote it by τ ∼ P ( π , P | s0 ) . Compared to directly learn policy π , it is often easier to learn from a critic , e.g. , a function that implicitly defines the optimal policy π ? . An example is to construct a value function Vπ ( · ) which estimates the reward obtained by policy π at each state s : Vπ ( s ) = E [ ∞∑ t=0 γt ×R ( st , at | s0 = s ) ] , ( 2 ) and the optimal policy π ? satisfies Vπ ? ( s ) > Vπ ( s ) at each state s and all policy π . Another critic is named Q-function , which estimates the reward obtained by policy π at each state s if action a is taken as the first step ( and all the remaining steps are determined by π ) : Qπ ( s , a ) = E [ ∞∑ t=0 γt ×R ( st , at | s0 = s , a0 = a ) ] . ( 3 ) Once aQ-function is constructed on a policy π , we can continue to obtain a better policy π′ ( namely , Vπ ( s ) 6 Vπ′ ( s ) for all states s ∈ S ) by setting π′ ( s , argmaxa { Qπ ( s , a ) } ) = 1 at each state s. In many scenarios , the reward function R is difficult to define . IL starts with constructing a set of expert demonstrations { τ1 , τ2 , . . . , τK } . Each τk is named a trajectory which records the expert ’ s behavior in a complete episode τk = ( sk,0 , ak,0 , sk,1 , ak,1 , . . . , sk , Tk ) . This is to implicitly define an expert policy πE . Then , based on the assumption that the expert always performs the best , there are two ways to train the agent . BC uses all pairs ( sk , t , ak , t ) as training data to optimize the agent policy π in a supervised manner , and IRL updates π and the reward function R alternatively so that R always prefers πE to π . Mathematically , both BC and IRL solves the following minimax problem : R ? = argmin R { max π { λ×H ( π ) + Eπ [ R ( s0 ∼ P0 ) ] } − EπE [ R ( s0 ∼ P0 ) ] } , ( 4 ) where H ( π ) is the casual entropy term of π , measuring how confident the policy π is ( the larger H ( π ) , the more confident , the better ( Bloem & Bambos , 2014 ) ) . R takes different forms in BC and IRL . In BC , it can be written as Eπ [ R ( s0 ) ] = ∑∞ t=0 − lnD ( st , at ) , where all ( st , at ) pairs are sampled from π using Equation 1 , and D ( s , a ) is a discriminative function to judge the likelihood that ( s , a ) comes from the decision of an expert . In IRL , Eπ [ R ( s0 ) ] = Vπ ( s0 ) is the value function estimating the total reward obtained by π. Optimizing Equation 4 is equivalent to GAN ( Goodfellow et al. , 2014 ) , in which the agent ( RL inner-loop ) plays the role of a generator , and the reward function ( IRL outer-loop ) plays the role of a discriminator ( Finn et al. , 2016 ) . The final goal is to make it difficult for the reward function to distinguish π from πE .

Summary: The paper proposes to integrate sequential information into imitation learning techniques.  The assumption is that mostly all the IL techniques are learning a policy which depends on state at time t, while the information contained in this state may be not sufficient to choose the right action (actually, this is the POMDP setting, the notion of POMDP not appearing in the paper....). The authors thus propose to use a recurrent neural network to encode the state by aggregating past information, instead of just using the features of the state at time t. They thus instantiate this idea on different methods and show that, on some problems, this approach can increase the quality of the final policy.