Reinforced Imitation Learning from Observations

1 INTRODUCTION . Learning by imitating is one of the most fundamental forms of learning in nature ( McFarland , 1999 ; Jones , 2009 ) . Its critical role in cognitive development is also supported by the fact that human brains have special structures , such as the mirror neurons , which are presumed to support this ability ( Heyes , 2010 ) . Due to this significance , it has also played a key role in machine learning and robotics ( Pomerleau , 1989 ; Ratliff et al. , 2007 ) , especially for the problems where reinforcement learning ( RL ) can easily be inefficient , e.g. , due to the sparsity of the reward signals . Imagine an infant ( the learner ) observing a caregiver ( the expert ) who is performing a task , e.g. , opening a door . From this example , we can derive the following observations on imitation learning ( IL ) . First , unlike the typical imitation learning in machine learning , the true action labels ( motor command ) executed by the expert is not available to the learner . Although in some cases such as autonomous cars , it may be possible to have access to the internal action labels by deploying special equipment , it is still expensive in general and in many applications not possible at all . Second , the actions that can be executed by the expert and the learner are different because of the differences in body development . This challenge can also easily be raised in real world applications . For instance , we may have a new version of a home robot that needs to learn from demonstrations of the old version which supports only a naive set of actions , e.g. , only a subset of the new version . Third , it may be reasonable and realistic to augment imitation learning with sparse reward signals . Even if having access to the labels to every action is unrealistic , in many cases the sparse rewards , e.g. , the completion of a task signaled by language or facial expression , can easily and cheaply be obtained . These challenges together with the inherent difficulties of reinforcement learning such as the sparsity of the reward signal and sensitivity to hyperparameter tuning , are required to be dealt with in order to make imitation learning applicable to real and complex challenges . In this paper , we propose a method for Reinforced Imitation Learning from Observations ( RILO ) to tackle the aforementioned challenges . Following the above observations , the proposed method aims to work efficiently in a setting where ( i ) labels for expert actions are not available , ( ii ) a reward signal is only sparsely provided , and ( iii ) the expert and learner operate in different action spaces . To achieve this , we extend generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) to improve efficiency for the cases where the expert actions are not available and different from the learner actions . The proposed approach can automatically balance learning between imitation and environment rewards . This makes our method learn as fast as imitation learning but also potentially converge to a better policy than the one demonstrated . This is done by gradually , but automatically , releasing the reliance on the imitation reward and then by learning more from the environment rewards . Through a series of experiments in simple navigation scenarios ( both fully and partially observable ) , we show that an agent is able to combine both the state observations from an expert and the sparse rewards to achieve better performance than either pure RL or IL with state-only observations . The main contribution of the paper is as follows . By leveraging environment sparse rewards , we propose a method that outperforms standard IL from observations and validate it through a series of systematic experiments . We provide an algorithm that can be applied when expert and learner do not share the same action space . The performance of the learner is not bounded by that of the expert due to suitable use of the environment sparse rewards . 2 RELATED WORK . Imitation learning ( IL ) is a common approach to learn a policy from expert demonstrations , i.e . sequences of state-action pairs . IL includes two main categories : ( i ) behavioural cloning ( Bain & Sammut , 1995 ; Pomerleau , 1989 ) and ( ii ) inverse reinforcement learning ( Abbeel & Ng , 2004 ; Ng & Russell , 2000 ) . Behavioral cloning directly learns the mapping from a state to an action by using the true action-labels from demonstrations and thus is a supervised learning method . Inverse reinforcement learning derives a reward function from demonstrations that may be then used to train a policy using the learned reward function . Recently , Ho & Ermon ( 2016 ) proposed GAIL , a method that uses demonstration data by forcing the learner to match state-transition occupancy distribution of the expert using an approach similar to GANs ( Goodfellow et al. , 2014 ) . Although GAIL is very effective and has attracted research attention ( Stadie et al. , 2017 ; Li et al. , 2017 ) , it requires expert state-action pairs which are expensive to obtain in many applications . For this reason , we focus on an IL scenario in which an agent does not use the expensive true action labels from an expert , but uses only the state observations . Following the previous literature , we call this imitation learning from observation ( ILO ) . Among few existing works that focus on this problem ( Torabi et al. , 2018a ; Kimura et al. , 2018 ; Stadie et al. , 2017 ; Aytar et al. , 2018 ; Liu et al. , 2017 ) , there are two that we find to be the closest to our method ( Merel et al. , 2017 ; Torabi et al. , 2018b ) . While both methods are built on top of GAIL , unlike GAIL , they target the case where the state-only observations are provided . Contrary to our approach , these methods work under the pure IL setting , i.e . they do not take advantage of the potential availability of sparse rewards . Some other line of works consider state-only expert observations during training but also require expert observations at test time ( Pathak et al. , 2018 ; Duan et al. , 2017 ; Borsa et al. , 2017 ) . Another related line of works is to use demonstrations to improve the exploration efficiency of RL under sparse reward settings ( Nair et al. , 2017 ; Hester et al. , 2017 ; Vecerı́k et al. , 2017 ) . Unlike us , these works however use the expensive state-action paired demonstrations , and treat them as selfgenerated . They also consider the optimal demonstrations only . While Kang et al . ( 2018 ) and Zhu et al . ( 2018 ) do not rely on these assumptions , they still use state-action demonstrations and assume both expert and learners share the same action space . Also , they used the full reward by combining the environment reward and imitation reward as a convex combination whose weights are manually set as a hyperparameter . Our method can , however , automatically adapt the balance . The last two works that we would like to mention are ( Gupta et al. , 2017 ; Gao et al. , 2018 ) . The former considers agents that may be morphologically distinct , however their approach assumes that time alignment is trivial which does not hold in our experimental setup . The latter works on imperfect demonstrations which makes the work related to our ( considering different actions spaces implies dealing with imperfect demonstrations ) , however their approach assumes the access to the demonstrations ( including expert actions ) . 3 METHOD . Our method is based on the recently proposed GAIL ( Ho & Ermon , 2016 ) . This method suggests an adversarial training approach as a way to make the distribution of state-action pairs of the learner as indistinguishable as possible from that of the expert . Although it achieves good performances , \ { s^l_i , s^l_j\ } _0^ { N_l } \ { s^e_i , s^e_j\ } _0^ { N_e } GAIL relies on two strong assumptions , which we want to overcome in our method : ( i ) the learner has access to the actions of the expert and ( ii ) the expert and the learner possess the same action space . Our setting differs from the standard GAIL approach in three ways . First , we assume that stateonly expert observations are provided as a dataset of trajectories Te = { ( si1 , ... , sini ) } Ni=1 . These trajectories consist of observations derived by executing an expert policy on a ( possibly partially observable ) Markov decision process with state space S , action space A , a transition probability p ( st+1|st , at ) , a reward function r , and discount factor γ . In our setting , an expert policy πe ( at|st ) performs actions in Ae ⊂ A . Second , we also consider the learner to have actions in Al ⊂ A , potentially different from Ae . Note that both the expert and the learner operate on the same state space S and we assume the same transition probability defined on the superset of agents ’ action spaces , i.e Ae ∪ Al ⊆ A . Since the expert and the learner can perform different actions , the two agents may have different optimal policies . In this case , pure imitation learning methods would end up with a learner having a sub-optimal policy . We propose self-exploration method to utilize the availability of sparse environment rewards to escape from the sub-optimal policy ( see Subsection 3.3 ) . 3.1 OVERVIEW . Our approach is composed of two different components ( see Figure 1 ) . A policy πθ : S → Al outputs an action given its current state . A discriminator Dφ : S × S → [ 0 , 1 ] is responsible for identifying the agent ( either expert or learner ) that has generated a given pair of states . Our goal is to optimize the following minimax objective : min θ max φ Eπθ [ log ( 1−Dφ ( si , sj ) ) ] + ETe [ log ( Dφ ( si , sj ) ) ] , ( 1 ) where ( si , sj ) is a tuple of states generated by the policy πθ or πφ . Similar to Zhu et al . ( 2018 ) , the policy is trained to maximize the discounted sum of the final reward rt , which is the sum of the environment reward renvt and the imitation reward r imt t given by the discriminator : rt = r env t + λ · rimtt , λ > 0 . ( 2 ) In the next subsection , we describe how we define the imitation reward rimtt . The reward rt from Equation 2 encourages the learned policy to visit similar paths as the expert , while obtaining high environment reward by achieving the goal . In all our experiment we always used λ = 1 . Each policy iteration consists of the following steps : ( i ) collect observations with the current learner policy , ( ii ) compute the discriminator scores for pairs of states where each pair is from either the expert or the learner , and update the discriminator , and ( iii ) compute the final rewards { rt } and update the policy . To update the policy , we used the A2C algorithm , a synchronous variant of A3C ( Mnih et al. , 2016 ) . See Figure 1 for the illustration of this procedure . For the rest of this section , we describe the two main contributions that we found to be fundamental to make an agent learn to complete a task in the RILO setting : different new imitation rewards and a method to efficiently combine state-only observations and sparse rewards .

Summary: The paper proposes to combine expert demonstration together with reinforcement learning to speed up learning of control policies. To do so, the authors modify the GAIL algorithm and create a composite reward function as a linear combination of the extrinsic reward and the imitation reward. They test their approach on several toy problems (small grid worlds).