Visual Imitation with a Minimal Adversary

1 INTRODUCTION . Many real world tasks , especially in robotics , involve high-dimensional inputs such as video pixels , and may have sparse rewards or even no rewards . This is a challenging problem for training reinforcement learning agents , because a randomly-initialized agent may need to interact with the environment for a very long time before receiving any learning signal . On a physical robot , this problem is compounded by the small number of actors gathering experience ( typically 1 ) , the slow speed compared to simulation , and the high cost of an episode that goes wrong . One way to make this exploration problem easier is to create a dense reward function . Often this is done by human hand-crafting of a staged reward . For example , in a block stacking task , the rewards can be provided in increasing magnitude for a robotic arm reaching toward a block , then more reward for grasping and lifting , then more for placing one on top of the other , and finally more for removing the arm from the blocks . However , this requires human expertise to design , and also requires access to the state of objects in the environment . Further , it can lead to sub-optimal policies . In the case of learning from pixels only , these stages may also be difficult to distinguish perfectly . Instead of hand-crafting a dense reward function , another popular approach is to collect expert demonstrations , and use these to derive a dense reward function , e.g . via imitation . Contemporary imitation learning approaches have a variety of trade-offs and requirements . For example , many approaches require expert actions in addition to the observations , or access to the expert states in order to perform special initialization , tracking , or warm starting ( Hosu & Rebedea ( 2016 ) ) the agent from the later stages of expert trajectories . Generative Adversarial Imitation Learning ( GAIL , Ho & Ermon ( 2016 ) ) , one of the most popular reinforcement learning methods for imitation , does not have these particular limitations . In GAIL , a discriminator network learns to distinguish agent from expert trajectories ; the agent is then rewarded for “ fooling ” this discriminator into thinking its behavior is that of the expert . However , GAIL tends to be inefficient in terms of environment interactions , is typically trained on-policy , and to our knowledge has not been shown to solve challenging high-dimensional sparse reward tasks . This might be because intuitively , in high dimensions such as pixel space , there are simply too many minute differences between the agent and expert for a deep neural network discriminator to exploit . Furthermore , optimizing deep adversarial networks is notoriously difficult , although there is progress towards stabilizing this problem ; see e.g . Arjovsky & Bottou ( 2017 ) ; Arjovsky et al . ( 2017 ) . In this work , we show that if GAIL is provided the right kind of features , it can actually easily handle high-dimensional pixel observations using a single layer discriminator network . We can also improve its efficiency in terms of environment interactions by using a Deep Distributed Deterministic Policy Gradients ( D4PG ) agent ( Barth-Maron et al. , 2018 ) , which is a state-of-the-art off-policy method for control , that can take advantage of a replay buffer to store past experiences . We show that several types of features can be used successfully with a tiny , single-layer adversary , in cases where a deep adversary on pixels would fail completely . Specifically : self-supervised embeddings , e.g . Contrastive Predictive Coding ( van den Oord et al. , 2018 ) ; surprisingly , random projections through a deep residual network ; and value network features from the D4PG agent itself . Concurrently with ( Kostrikov et al. , 2018 ) , we additionally show how to modify GAIL for the off-policy case of D4PG agents with experience replay . In our experiments , we demonstrate that our proposed approach is able to , from pixels , solve a challenging simulated robotic block stacking task using only demonstrations and a sparse binary reward indicating whether or not the stack has completed . Previous imitation approaches to this task have all used dense , staged task rewards crafted by humans and/or true state instead of pixels . In addition to reducing the dependency on hand-crafted rewards , our approach learns to stack faster than the dense staged reward baseline agent with the same amount of actor processes . The main contributions of this paper are the following : • A 6-DoF Jaco robot arm agent that learns block stacking from demonstration videos and sparse reward . On a simulated Jaco arm it achieves 94 % success rate , compared to ∼ 29 % using a behavior cloning agent with an equivalent number of demonstrations . • An adversary-based early termination method for actor processes , that improves task performance and learning speed by creating a natural curriculum for the agent . • An agent that learns with no task reward using an auxiliary goal recognizer adversary , achieving 55 % stacking success from video imitation only . • Ablation experiments on Jaco stacking as well as a 2D planar walker benchmark ( Tassa et al. , 2018 ) , to understand the specific reasons for improvement in our agent . We find that random projections with a linear discriminator work surprisingly well in some cases , and that using value network features is even better . 2 BACKGROUND . Following the notation of Sutton et al . ( 1998 ) , a Markov Decision Process ( MDP ) is a tuple ( S , A , R , P , γ ) with states S , actions A , reward function R ( s , a ) , transition distribution P ( s′|s , a ) , and discount γ . An agent in state s ∈ S takes action a ∈ A according to its policy π and moves to state s′ ∈ S according to the transition distribution . The goal of training an agent is to find a policy that maximizes the expected sum of discounted rewards , represented by the action value function Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ tR ( st , at ) ] , where Eπ is an expectation over trajectories starting from s0 = s and taking action a0 = a and thereafter running the policy π . 2.1 DEEP DETERMINISTIC POLICY GRADIENTS ( DDPG ) . DDPG ( Lillicrap et al. , 2016 ) is an actor-critic method in which the actor , or policy π ( s|φ ) and the critic , or action-value function Q ( s , a|θ ) are represented by neural networks with parameters φ and θ , respectively . New transitions ( s , a , r , s′ ) are added to a replay buffer B by sampling from the policy according to a = π ( s|φ ) + , with ∼ N added for better exploration . The action-value function is trained to match the 1-step returns by minimizing L1 = E ( s , a , r , s′ ) ∼B [ R ( s , a ) + γQ′ ( s′ , π′ ( s′|φ′ ) |θ′ ) −Q ( s , a|θ ) ] 2 ( 1 ) Figure 1 : Actor and learner process pseudocode . Algorithm 1 Actor process Replay buffer B , environment E for nepisodes do for t = 0 to T do at ← π ( st ) rt , st+1 ← E ( st , at ) B ← ( st , at , rt , st+1 ) // Add to replay rimitation ← D ( st+1 ) if rimitation < β break end for end for Algorithm 2 Learner process Replay buffer B , expert dataset D for nupdates do // Sample from replay and expert data . ( s , a , r , s′ ) ∼ B , sexpert ∼ D // Add imitation to reward . r = αr + ( 1− α ) D ( s′ ) // Optimize Equations 1 , 2 , and 4 . Q , π , D ← Update ( s , a , r , s′ , sexpert ) end for where the transition is sampled from the replay buffer B , and π′ , Q′ are target actor and critic networks parameterized by φ′ and θ′ respectively . To improve the stability of weight updates , the target networks are updated every K learning steps . The policy network is trained via gradient descent to produce actions that maximize the action-value function using the deterministic policy gradient ( Silver et al. , 2014 ) : ∇φJ ( φ ) ≈ E ( s , a ) ∼B [ ∇φQ ( s , π ( s|φ ) ) ] ( 2 ) Building on top of the basic DDPG agent , we also leverage several subsequent improvements following Barth-Maron et al . ( 2018 ) , called D4PG . We summarize these in section 7.6 . 2.2 GENERATIVE ADVERSARIAL IMITATION LEARNING ( GAIL ) . In GAIL , a reward function is learned by training a discriminator network D ( s , a ) to distinguish between agent transitions and expert transitions . The GAIL objective is formulated as follows : max π min D Eπ [ logD ( s , a ) ] + EπE [ log ( 1−D ( s , a ) ) ] + λH ( π ) ( 3 ) where π is the agent policy , πE the expert policy , and H ( π ) an entropy regularizer . GAIL is closely related to MaxEnt inverse reinforcement learning ( Ziebart et al. , 2008 ; Finn et al. , 2016 ) . 3 MODEL . To make use of all the available training data , we use a D4PG agent that does off-policy training with experience replay with buffer B . The actor and critic updates are the same as in D4PG , but in addition we jointly train the reward function using a modified equation 3 for D : min D Es∼B [ logD ( s ) ] + EπE [ log ( 1−D ( s ) ) ] ( 4 ) As pointed out by Kostrikov et al . ( 2018 ) , the use of a replay buffer changes the original expectation over the agent policy in equation 3 , so that the discriminator must distinguish expert transitions from the transitions produced by all previous agents . This could be corrected by importance sampling , but in practice , we also find this not to be necessary . Note that we do not use actions a in the discriminator because we only assume access to videos of the expert . Our reward function interpolates imitation reward and a sparse task reward : R ( s , a , s′ ) = αRtask ( s , a , s ′ ) + ( 1− α ) D ( s′ ) ( 5 ) where s′ is the state reached after taking action a in state s. Note that by D ( . ) we mean the sigmoid of the logits , σ ( x ) = 1/ ( 1 + exp ( −x ) ) , which bounds the imitation reward between 0 and 1 . This is convenient because it allows us to choose intuitive values for early termination of episodes in the actor process , e.g . if the discriminator score is too low , and it is scaled similarly to the sparse reward that we use in our block stacking experiments . context vectors target vectors ( 2 ) Train the agent and the reward function Agent transition Expert transition ... ... Contexts Targets P ( Firsttransitionis agent ) Reward network Pixel observation CPC expert future prediction Above we include pseudocode for the actor and learner processes . In practice we use many CPU actor processes in parallel ( 128 in our experiments ) and a single learner process on GPU . The actor processes receive updated network parameters every 100 acting steps . An important detail in the actor is the use of early termination of the episode when the discriminator score is below a threshold β , which is a hyperparameter . This prevents the agent from drifting too far from the expert trajectories , and avoids wasting computation time . In practice we set β = 0.1 . We include a plot of the training dynamics induced by this early stopping mechanism in figure 7b .

Summary: The submission describes a sort of hybrid between reinforcement learning and imitation learning, where an auxiliary imitation learning objective helps to guide the RL policy given expert demonstrations.  The method consists of concurrently maximizing an RL objective--augmented with the GAIL discriminator as a reward—and minimizing the GAIL objective, which optimizes the discriminator between expert and policy-generated states.  Only expert states (not actions) are required, which allows the method to work given only videos of the expert demonstrations.  Experiments show that adding the visual imitation learning component allows RL to work with sparse rewards for complex tasks, in situations where RL without the imitation learning component fails.