Adversarial Exploration Strategy for Self-Supervised Imitation Learning

1 INTRODUCTION . Over the past decade , imitation learning ( IL ) has been successfully applied to a wide range of domains , including robot learning ( Englert et al. , 2013 ; Schulman et al. , 2013 ) , autonomous navigation ( Choudhury et al. , 2017 ; Ross et al. , 2013 ) , manipulation tasks ( Nair et al. , 2017 ; Prieur et al. , 2012 ) , and self-driving cars ( Codevilla et al. , 2018 ) . Traditionally , IL aims to train an imitator to learn a control policy π only from expert demonstrations . The imitator is typically presented with multiple demonstrations during the training phase , with an aim to distill them into π . To learn π effectively and efficiently , a large set of high-quality demonstrations are necessary . This is especially prevalent in current state-of-the-art IL algorithms , such as dataset aggregation ( DAgger ) ( Ross et al. , 2011 ) and generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) . Although these approaches have been the dominant algorithms in IL , a major bottleneck for them is their reliance on high-quality demonstrations , which often require extensive supervision from human experts . In addition , a serious flaw in the learned policy π is its tendency to overfit to demonstration data , preventing it from generalizing to new ones . To overcome the aforementioned challenges in IL , a number of methods have been investigated to enhance the generalizability and data efficiency , or reduce the degree of human supervision . Initial efforts in this direction were based on the idea of meta learning ( Duan et al. , 2017 ; Finn et al. , 2017 ; Yu et al. , 2018 ) , in which the imitator is trained from a meta learner that is able to quickly learn a new task with only a few set of demonstrations . However , such schemes still require training the meta-learner with tremendous amount of time and demonstration data , leaving much room for improvement . Thus , a rapidly-growing body of literature based on the concept of using forward/inverse dynamics models to learn π within an environment in a self-supervised fashion ( Agrawal et al. , 2016 ; Nair et al. , 2017 ; Pathak et al. , 2018 ) has emerged in the past few years . One key advantage of the concept is that it provides an autonomous way for preparing training data , removing the need of human intervention . In this paper , we call it self-supervised IL . Self-supervised IL allows an imitator to collect training data by itself instead of using predefined extrinsic reward functions or expert supervision during training . It only needs demonstration during inference , drastically decreasing the time and effort required from human experts . Although the core principles of self-supervised IL are straightforward and have been exploited in many fields ( Agrawal et al. , 2016 ; Nair et al. , 2017 ; Pathak et al. , 2017 ; 2018 ) , recent research efforts have been dedicated to addressing the challenges of multi-modality and multi-step planning . For example , the use of forward consistency loss and forward regularizer have been extensively investigated to enhance the task performance of the imitator ( Agrawal et al. , 2016 ; Pathak et al. , 2018 ) . This becomes especially essential when the lengths of trajectories grow and demonstration samples are sparse , as multiple paths may co-exist to lead the imitator from its initial observation to the goal observation . The issue of multi-step planning has also drawn a lot of attention from researchers , and is usually tackled by recurrent neural networks ( RNNs ) and step-by-step demonstrations ( Nair et al. , 2017 ; Pathak et al. , 2018 ) . The above self-supervised IL approaches report promising results , however , most of them are limited in applicability due to several drawbacks . First , traditional methods of data collection are usually inefficient and time-consuming . Inefficient data collection results in poor exploration , giving rise to a degradation in robustness to varying environmental conditions ( e.g. , noise in motor control ) and generalizability to difficult tasks . Second , human bias in data sampling range tailored to specific interesting configurations is often employed ( Agrawal et al. , 2016 ; Nair et al. , 2017 ) . Although a more general exploration strategy called curiosity-driven exploration was later proposed in Pathak et al . ( 2017 ) , it focuses only on exploration in states novel to the forward dynamics model , rather than those directly influential to the inverse dynamics model . Furthermore , it does not discuss the applicability to continuous control domains , and fails in high dimensional action spaces according to our experiments in Section 4 . Unlike the approaches discussed above , we do not propose to deal with multi-modality or multi-step planning . Instead , we focus our attention on improving the overall quality of the collected samples in the context of self-supervised IL . This motivates us to equip the model with the necessary knowledge to explore the environment in an efficient and effective fashion . In this paper , we propose a straightforward and efficient self-supervised IL scheme , called adversarial exploration strategy , which motivates exploration of an environment in a self-supervised manner ( i.e. , without any extrinsic reward or human demonstration ) . Inspired by Pinto et al . ( 2017 ) ; Shioya et al . ( 2018 ) ; Sukhbaatar et al . ( 2018 ) , we implement the proposed strategy by jointly training a deep reinforcement learning ( DRL ) agent and an inverse dynamics model competing with each other . The former explores the environment to collect training data for the latter , and receives rewards from the latter if the data samples are considered difficult . The latter is trained with the training data collected by the former , and only generates rewards when it fails to predict the true actions performed by the former . In such an adversarial setting , the DRL agent is rewarded only for the failure of the inverse dynamics model . Therefore , the DRL agent learns to sample hard examples to maximize the chances to fail the inverse dynamics model . On the other hand , the inverse dynamics model learns to be robust to the hard examples collected by the DRL agent by minimizing the probability of failures . As a result , as the inverse dynamics model becomes stronger , the DRL agent is also incentivized to search for harder examples to obtain rewards . Overly hard examples , however , may lead to biased exploration and cause instability of the learning process . In order to stabilize the learning curve of the inverse dynamics model , we further propose a reward structure such that the DRL agent is encouraged to explore moderately hard examples for the inverse dynamics model , but refraining from too difficult ones for the latter to learn . The self-regulating feedback structure between the DRL agent and the inverse dynamics model enables them to automatically construct a curriculum for exploration . We perform extensive experiments to validate adversarial exploration strategy on multiple OpenAI gym ( Brockman et al. , 2016 ) robotic arm and hand manipulation task environments simulated by the MuJoCo physics engine ( Todorov et al. , 2012 ) , including FetchReach , FetchPush , FetchPickAndPlace , FetchSlide , and HandReach . These environments are intentionally selected by us for evaluating the performance of inverse dynamics model , as each of them allows only a very limited set of chained actions to transition the robotic arms and hands to target observations . We examine the effectiveness of our method by comparing it against a number of self-supervised IL schemes . The experimental results show that our method is more effective and data-efficient than the other self-supervised IL schemes for both low- and high-dimensional observation spaces , as well as in environments with high-dimensional action spaces . We also demonstrate that in most of the cases the performance of the inverse dynamics model trained by our method is comparable to that directly trained with expert demonstrations . The above observations suggest that our method is superior to the other self-supervised IL schemes even in the absence of human priors . We further evaluate our method on environments with action space perturbations , and show that our method is able to achieve satisfactory success rates . To justify each of our design decisions , we provide a comprehensive set of ablative analysis and discuss their implications . The contributions of this work are summarized as follows : • We introduce an adversarial exploration strategy for self-supervised IL . It consists of a DRL agent and an inverse dynamics model developed for efficient exploration and data collection . • We employ a competitive scheme for the DRL agent and the inverse dynamics model , enabling them to automatically construct a curriculum for exploration of observation space . • We introduce a reward structure for the proposed scheme to stabilize the training process . • We demonstrate the proposed method and compare it with a number of baselines for multiple robotic arm and hand manipulation tasks in both low- and high-dimensional state spaces . • We validate that our method is generalizable to tasks with high-dimensional action spaces . The remainder of this paper is organized as follows . Section 2 introduces background material . Section 3 describes the proposed adversarial exploration strategy in detail . Section 4 reports the experimental results , and provides an in-depth ablative analysis of our method . Section 5 concludes . 2 BACKGROUND . In this section , we briefly review DRL , policy gradient methods , as well as inverse dynamics model . 2.1 DEEP REINFORCEMENT LEARNING AND POLICY GRADIENT METHODS . DRL trains an agent to interact with an environment E . At each timestep t , the agent receives an observation xt ∈ X , where X is the observation space of E . It then takes an action at from the action space A based on its current policy π , receives a reward r , and transitions to the next observation x′ . The policy π is represented by a deep neural network with parameters θ , and is expressed as π ( a|x , θ ) . The goal of the agent is to learn a policy to maximize the discounted sum of rewards Gt : Gt = T∑ τ=t γτ−tr ( xτ , aτ ) , ( 1 ) where t is the current timestep , γ ∈ ( 0 , 1 ] the discount factor , and T the horizon . Policy gradient methods ( Mnih et al. , 2016 ; Sutton et al. , 2000 ; Williams , 1992 ) are a class of RL techniques that directly optimize the parameters of a stochastic policy approximator using policy gradients . Although these methods have achieved remarkable success in a variety of domains , the high variance of gradient estimates has been a major challenge . Trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) circumvented this problem by applying a trust-region constraint to the scale of policy updates . However , TRPO is a second-order algorithm , which is relatively complicated and not compatible with architectures that embrace noise or parameter sharing ( Schulman et al. , 2017 ) . In this paper , we employ a more recent family of policy gradient methods , called proximal policy optimization ( PPO ) ( Schulman et al. , 2017 ) . PPO is an approximation to TRPO , which similarly prevents large changes to the policy between updates , but requires only first-order optimization . PPO is superior in its generalizability and sample complexity while retaining the stability and reliability of TRPO 1 .

Summary: The paper proposes a novel exploration strategy for self-supervised imitation learning. An inverse dynamics model is trained on the trajectories collected from a RL-trained policy. The policy is rewarded for generating trajectories on which the inverse dynamics model (IDM) currently works poorly, i.e. on which IDM predicts actions that are far (in terms of mean square error) from the actions performed by the policy. This adversarial training is performed in purely self-supervised way. The evaluation is performed by one-shot imitation of an expert trajectory using the IDM: the action is predicted from the current state of the environment and the next state in the expert’s trajectory. Experimental evaluation shows that the proposed method is superior to baseline exploration strategies for self-supervised imitation learning, including random and curiosity-based exploration.