Task-Relevant Adversarial Imitation Learning

We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent . We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning ( GAIL ) . Our proposed solution , Task-Relevant Adversarial Imitation Learning ( TRAIL ) , uses a constrained optimization objective to overcome task-irrelevant features . Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators , where other agents such as behaviour cloning ( BC ) , standard GAIL , improved GAIL variants including our newly proposed baselines , and Deterministic Policy Gradients from Demonstrations ( DPGfD ) fail to find solutions , even when the other agents have access to task reward . 1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have produced breath-taking conditional image synthesis results ( Goodfellow et al. , 2014 ; Brock et al. , 2019 ) , and have inspired adversarial learning approaches to imitating behavior . In Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) , a discriminator network is trained to distinguish agent and expert behaviour through its observations , and is then used as a reward function . GAIL agents can overcome the exploration challenge by taking advantage of expert demonstrations , while also achieving high asymptotic performance by learning from agent experience . Despite the huge promise of GAIL , it has not yet had the same impact as GANs ; in particular , robust GAIL from pixels for control applications remains a challenge . Here , we study a key shortcoming of GAIL : the tendency of the discriminator to mainly exploit task-irrelevant features . For example , by focusing on slight background differences , a discriminator can achieve perfect generalization , assigning zero reward to all held-out agent observations . However , this discriminator does not yield an informative reward function because it ignores behavior . Assuming there is an expert policy πE that is optimal for an unknown reward function , here we refer to a feature as task-irrelevant if it does not affect that reward . For example , if the task is to lift a red block , the positions of other blocks would be task-irrelevant ; see Figures 1 and 2 . This paper makes the following contributions : 1 . It reveals a fundamental limitation of GAIL by showing that discriminators do in practice exploit task-irrelevant information , thereby resulting in poor task performance . 2 . It introduces powerful GAIL baselines . In particular , it shows that standard regularization and data augmentation are generally useful and improve upon standard GAIL . 3 . It shows that these improvements to GAIL , as well as other improvements proposed by Reed et al . ( 2018 ) , do not completely solve the problem , allowing GAIL agents to fail catastrophically with the addition of task-irrelevant distractors . 4 . It introduces Task Relevant Adversarial Imitation Learning ( TRAIL ) , using constrained optimization to force the discriminator to focus on the relevant aspects of the task , which improves performance dramatically on manipulation tasks from pixels ( see Figure 1 ) . 2 RELATED WORK . The use of demonstrations to help agent training has been studied extensively in robotics ( Bakker & Kuniyoshi , 1996 ; Kawato et al. , 1994 ; Miyamoto et al. , 1996 ) with approaches ranging from Q-learning ( Schaal , 1997 ) to behavioral cloning ( BC ) ( Pomerleau , 1989 ) . BC : BC is effective in solving many control problems ( Pomerleau , 1989 ; Finn et al. , 2017 ; Duan et al. , 2017 ; Rahmatizadeh et al. , 2018 ) . It has also been successfully applied to initialize RL training ( Rajeswaran et al. , 2017 ) . It , however , suffers from compounding errors as initial small deviations from the expert behaviors tend to cause bigger differences ( Ross et al. , 2011 ) . This often necessitates a large number of demonstrations for satisfactory performance . Furthermore , BC typically does not lead to agents that are superior to their demonstrators . Inverse RL : Ziebart et al . ( 2008 ) ; Ng et al . ( 2000 ) ; Abbeel & Ng ( 2004 ) propose inverse reinforcement learning ( IRL ) as a way of learning reward functions from demonstrations . Reinforcement learning can then be used to optimize that learned reward . Recently , Finn et al . ( 2016b ) approached continuous robotic control problems with success by applying Maximum Entropy IRL algorithms which are very closely related to GAIL ( Finn et al. , 2016a ) and have similar drawbacks . Learning from Demonstrations : Hester et al . ( 2018 ) developed deep Q-Learning from demonstration ( DQfD ) , in which expert trajectories are added to experience replay and jointly used to train agents along with their own experiences . This was later extended by Vecerik et al . ( 2017 ) and Pohlen et al . ( 2018 ) to better handle sparse-reward problems in control and Atari games respectively . Despite their efficiency , this class of methods still requires access to rewards in order to learn . GAIL : Following the success of Generative Adversarial Networks ( Goodfellow et al. , 2014 ) in image generation , GAIL ( Ho & Ermon , 2016 ) applies adversarial learning to the problem of imitation . Although many variants are introduced in the literature ( Li et al. , 2017 ; Fu et al. , 2018 ; Merel et al. , 2017 ; Zhu et al. , 2018 ; Baram et al. , 2017 ) , making GAIL work for high-dimensional input spaces , particularly raw pixels , remains a challenge . A few papers ( Peng et al. , 2018 ; Reed et al. , 2018 ; Blondé & Kalousis , 2018 ) seek to address the problem of overfitting the discriminator . Peng et al . ( 2018 ) introduces the Variational Bottleneck to regularize the discriminator . Reed et al . ( 2018 ) proposes to not train the vision module of the discriminator ( e.g . use the vision module of the critic network instead ) and only train a tiny network on top of the vision module to discriminate . Blondé & Kalousis ( 2018 ) follow a similar approach . Unstructured regularization , however , can not stop the discriminator from fitting to features that are systematically different between the agents ’ and the demonstration ’ s behavior , like those illustrated in Figure 2 . TRAIL , on the other hand , is much less prone to overfitting to these features . Stadie et al . ( 2017 ) extend GAIL to the setting of third person imitation , in which the demonstrator and agent observations come from different views . To prevent the discriminator from discriminating based on viewpoint domain , they use gradient flipping from an auxiliary classifier to learn domain-invariant features . Our approach is not to learn domain-invariant features , but instead learn domain-agnostic discriminators that only focus on behavior . Several recent works have focused on improving the sample efficiency of GAIL ( Blondé & Kalousis , 2018 ; Sasaki et al. , 2018 ) . Common to these approaches and to this work , is the use of off-policy actor critic agents and experience replay , to improve the utilization of available experience . 3 REINFORCEMENT LEARNING AND ADVERSARIAL IMITATION . Following the notation of Sutton & Barto ( 2018 ) , a Markov Decision Process ( MDP ) is a tuple ( S , A , R , P , γ ) with states S , actions A , reward function R ( s , a ) , transition distribution P ( s′|s , a ) , and discount γ . An agent in state s ∈ S takes action a ∈ A according to its policy π and moves to state s′ ∈ S according to the transition distribution . The goal of RL algorithms is to find a policy that maximizes the expected sum of discounted rewards , represented by the action value function Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ tR ( st , at ) ] , where Eπ is an expectation over trajectories starting from s0 = s and taking action a0 = a and thereafter running the policy π . To apply RL , it is essential that we have access to the reward function which is often hard to design and evaluate ( Singh et al. , 2019 ) . In addition , sparse rewards can cause exploration difficulties that pose great challenges to RL algorithms . We therefore look to imitation learning and particularly GAIL to derive a reward function from expert demonstrations . In GAIL , a reward function is learned by training a discriminator network D ( s , a ) to distinguish between agent and expert state-action pairs . The GAIL objective is thus formulated as follows : min π max D E ( s , a ) ∼πE [ logD ( s , a ) ] + E ( s , a ) ∼π [ log ( 1−D ( s , a ) ) ] − λHH ( π ) , ( 1 ) where π is the agent policy , πE the expert policy , and H ( π ) an ( optional ) entropy regularizer . The reward function is defined simply : R ( s , a ) = − log ( 1−D ( s , a ) ) . GAIL is theoretically appealing and practically simple . The discriminator , however , can focus on any features to discriminate , whether these features are task-relevant or not . In the next subsection we describe a way to constrain the discriminator network in order to prevent it from using task-irrelevant details to distinguish agent and expert data . 4 TASK-RELEVANT ADVERSARIAL IMITATION LEARNING ( TRAIL ) . We want the discriminator to focus on task-relevant features . Our proposed solution , TRAIL , prevents the discriminator from being able to distinguish expert and agent behaviour based on selected aspects of the data . For instance , the discriminator should distinguish agent and expert frames only when meaningful behavior is present in those frames . In the absence of behavior useful to solve the task , e.g . in initial frames prior to the execution of the behavior , the discriminator should be agnostic . To this end , we propose a constraint for the discriminator , such that its accuracy must not be greater than chance on a particular set of observations I that we will call the invariant set . I will include only observations that can be distinguished as agent or expert in task-irrelevant ways . Precisely , we formulate TRAIL in terms of the following constrained optimization problem : max ψ Es∼πE [ logDψ ( s ) ] + Es∼πθ [ log ( 1−Dψ ( s ) ) ] ( 2 ) s.t . 1 2 Es∼πE [ 1Dψ ( s ) ≥ 12 | s ∈ I ] + 1 2 Es∼πθ [ 1Dψ ( s ) < 12 | s ∈ I ] ≤ 1 2 . To apply the above constraint in practice , we can optimize the reverse of the objective function on I . That is , given a batch of N examples se ∼ πE , sθ ∼ πθ from the expert and agent , and ŝe ∼ πE and ŝθ ∼ πθ both in the set I , we maximize the following augmented objective function : Lψ ( se , sθ , ŝe , ŝθ ) = ∑N i=1 logDψ ( s ( i ) e ) + log ( 1−Dψ ( s ( i ) θ ) ) ( 3 ) − λ [ ∑N i=1 logDψ ( ŝ ( i ) e ) + log ( 1−Dψ ( ŝ ( i ) θ ) ) ] 1accuracy ( ŝe , ŝθ ) ≥ 12 , where accuracy ( · , · ) is defined as the average of discriminator accuracies : accuracy ( ŝe , ŝθ ) = 1 2N N∑ i=1 [ 1 Dψ ( ŝ ( i ) e ) ≥ 12 + 1 Dψ ( ŝ ( i ) θ ) < 12 ] . ( 4 ) The scalar λ ≥ 0 is a tunable hyperparameter . We used single frames as states but sequences can also be used , and we do not require expert actions . Not requiring actions enables learning in very off-policy settings , where the action dimensions and distributions of the demonstrator ( another robot or human ) are different from those of the agent .

Summary: The paper addresses the acute problem of irrelevant discrimination in adversarial imitation learning methods. Ideally, the discriminator in such methods should extract task-dependent features and base its discrimination rule upon them only. However, in practice, the opposite usually occurs. For example, if the state distributions of the expert and agent come from different modalities, e.g., different background light, different POV camera, etc., then almost surely the discriminator will base its classification rule on such differences. In turn, the reward signal will be (constant and) irrelevant. This is a well-known problem, that indeed received little attention in previous work.