One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

1 INTRODUCTION . One-shot imitation is a powerful way to show agents how to solve a task . For instance , one or a few demonstrations are typically enough to teach people how to solve a new manufacturing task . In this paper , we introduce an AI agent that when provided with a novel demonstration is able to ( i ) mimic the demonstration with high-fidelity , or ( ii ) forego high-fidelity imitation to solve the intended task more efficiently . Both types of imitation can be useful in different domains . Motor control is a notoriously difficult problem , and we are often deceived by how simple a manipulation task might appear to be . Tying shoe-laces , a behaviour many of us learn by imitation , might appear to be simple . Yet , tying shoe-laces is something most 6 year olds struggle with , long after object recognition , walking , speech , often translation , and sometimes even reading comprehension . This long process of learning that eventually results in our ability to rapidly imitate many behaviours provides inspiration for the work in this paper . We refer to high-fidelity imitation as the act of closely mimicking a demonstration trajectory , even when some actions may be accidental or irrelevant to the task . This is sometimes called over-imitation ( McGuigan et al. , 2011 ) . It is known that humans over-imitate more than other primates ( Horner & Whiten , 2005 ) and that this may be useful for rapidly acquiring new skills ( Legare & Nielsen , 2015 ) . For AI agents however , learning to closely imitate even one single demonstration from raw sensory input can be difficult . Many recent works focus on using expensive reinforcement learning ( RL ) methods to solve this problem ( Sermanet et al. , 2018 ; Liu et al. , 2017 ; Peng et al. , 2018 ; Aytar et al. , 2018 ) . In contrast , high-fidelity imitation in humans is often cheap : in one-shot we can closely mimic a demonstration . Inspired by this , we introduce a meta-learning approach ( MetaMimic — Figure 1 ) to learn high-fidelity one-shot imitation policies by off-policy RL . These policies , when deployed , require a single demonstration as input in order to mimic the new skill being demonstrated . AI agents could acquire a large and diverse set of skills by high-fidelity imitation with RL . However , representing many behaviours requires the adoption of a model with very high capacity , such as a very large deep neural network . Unfortunately , showing that RL methods can be used to train massive deep neural networks has been an open question because of the variance inherent to these methods . Indeed , traditional deep RL neural networks tend to be small , to the point that researchers have recently questioned their contribution ( Rajeswaran et al. , 2017b ) . In this paper , we show that it is possible to train massive deep networks by off-policy RL to represent many behaviours . Moreover , we show that bigger networks generalize better . These results therefore provide important evidence that RL is indeed a scalable and viable framework for the design of AI agents . Specifically this paper makes the following contributions1 : • It introduces the MetaMimic algorithm and shows that it is capable of one-shot high-fidelity imitation from video in a complex manipulation domain . • It shows that MetaMimic can harness video demonstrations and enrich them with actions and rewards so as to learn uncoditional policies capable of solving manipulation tasks more efficiently than teleoperating humans . By retaining and taking advantage of all its experiences , MetaMimic also substantially outperforms the state-of-the-art D4PG RL agent , when D4PG uses only the current task experiences . • The experiments provide ablations showing that larger networks ( to the best of our knowledge , the largest networks ever used in deep RL ) lead to improved generalization in high-fidelity imitation . The ablations also highlight the important value of instance normalization . • The experiments show that increasing the number of demonstrations during training leads to better generalization on one-shot high-fidelity imitation tasks . 2 METAMIMIC . MetaMimic is an algorithm to learn both one-shot high-fidelity imitation policies and unconditional task policies that outperform demonstrators . Component 1 takes as input a dataset of demonstrations , and produces ( i ) a set of rich experiences and ( ii ) a one-shot high-fidelity imitation policy . Component 2 takes as input a set of rich experiences and produces an unconditional task policy . Component 1 uses a dataset of demonstrations to define a set of imitation tasks and using RL it trains a conditional policy to perform well across this set . Here each demonstration is a sequence of observations , without corresponding actions . This component can use any RL algorithm , and is applicable whenever the agent can be run in the environment and the environment ’ s initial conditions can be precisely set . In practice we make use of D4PG ( Barth-Maron et al. , 2018 ) , an efficient off-policy RL algorithm , for training the agent ’ s policy from demonstration data . In Section 2.1 we give a detailed description of our approach for learning one-shot high-fidelity imitation policies . Here we also describe the neural network architectures used in our approach to imitation ; our results will show that as the number of imitation tasks increases it becomes necessary to train large-scale neural network policies to generalize well . Furthermore , the process of training the imitation policies results in a memory of experiences which includes both actions and rewards . As shown in Section 2.2 we can replay these experiences to learn unconditional policies capable of solving new tasks and outperforming human demonstrators . 1Videos presenting our results are available at https : //vimeo.com/metamimic . 2.1 LEARNING A POLICY FOR ONE-SHOT HIGH-FIDELITY IMITATION . We consider a single stochastic task and define a demonstration of this task as a sequence of observations d= { d1 , ... , dT } ∼ pd ( · ) . While there is only one task , there is ample diversity in the environment ’ s initial conditions and in the demonstrations . Each observation dt can be either an image or a combination of both images and proprioceptive information . We let πθ represent a deterministic parameterized imitation policy that produces actions at =πθ ( ot , d ) by conditioning on the current observations and a given demonstration . We also let E denote the environment renderer , which accepts an arbitrary policy and produces a sequence of observations o= { o1 , ... , oT } =E ( π ) . We can think of this last step as producing a rollout trajectory given the policy , as illustrated on the right side of Figure 2 . The goal of high-fidelity imitation is to estimate the parametersθ of a policy that maximizes the expected imitation return , e.g . similarity between the observationso and sampled demonstrationsd∼pd . That is , θ∗=argmax θ Ed∼pd [ ∑ t γtsim ( dt , E ( πθ ( ot , d ) ) t ) ] , ( 1 ) where sim is a similarity measure ( reward ) , which will be discussed in greater detail at the end of this section , and γ is the discount factor . In general it is not possible to differentiate through the environment E . We thus choose to optimize this objective with RL , while sampling trajectories by acting on the environment . Finally , we refer to this process as one-shot imitation because although we make use of multiple demonstrations at training time to learn the imitation policy , at test time we are able to follow a single novel demonstration d ? using the learned conditional policy πθ∗ ( ot , d ? ) . We adopt the recently proposed D4PG algorithm ( Barth-Maron et al. , 2018 ) as a subroutine for training the imitation policy . This is a distributed off-policy RL algorithm that interacts with the environment using independent actors ( see Figure 2 ) , each of which inserts trajectory data into a replay table . A learner process in parallel samples ( possibly prioritized ) experiences from this replay dataset and optimizes the policy in order to maximize the expected return . MetaMimic first builds on this earlier work by making a very specific choice of reward and policy . At the beginning of every episode a single demonstration d is sampled , and the initial conditions of the environment are set to those of the demonstration , i.e . o1 =d1 . The actor then interacts with the environment by producing actions at =πθ ( ot , d ) . While this policy representation is popular in the feature-based supervised one-shot imitation literature , in our case the observations and demonstrations are sequences of high-dimensional sensory inputs and hence this approach becomes computationally prohibitive . To overcome this challenge , we simplify the model to only consider local context at =πθ ( ot , dt+1 ) . In this formulation , the future demonstration state can be interpreted as a goal state and the approach may be thought of as goal-conditional imitation with a time-varying goal . At every timestep , we compute the reward rt =r ( ot+1 , d ) . While in general this reward can depend on the entire trajectory—or on small subsequences such as ( dt , dt+1 ) —in practice we will restrict it to depend only on the goal state dt+1 . Ideally the reward function can be learned during training ( Ganin et al. , 2018 ; Nair et al. , 2018 ) . However , in this work we experiment with a simple reward function based on the Euclidean distance2 over observations : rimitatet =r ( ot+1 , dt+1 ) =β1exp ( −‖oimaget+1 −d image t+1 ‖22 ) +β2exp ( −‖obodyt+1 −d body t+1 ‖22 ) ( 2 ) where oimaget are the raw pixel observations and o body t are proprioceptive measurements ( joint and end effector positions and velocities ) . Both components of the reward function have limitations : obodyt has no information about objects in the environment , so it may fail to encourage the imitator to interact with the objects ; where as oimaget contains information about the body and objects , but is insufficient to uniquely describe either . In practice , we found a combination of both to work best . Again we note that the next demonstration statedt+1 can be interpreted as a goal state . In this perspective the goals are set by the demonstration , and the agent is rewarded by the degree to which it reaches those goals . Because the imitation goals are explicitly given to the policy , the imitation policy is able to imitate many diverse demonstrations , and even generalize to unseen demonstrations as described in Section 2.2 . High-fidelity imitation is a fine-grained perception task and hence the choice of policy architecture is critical . In particular , the policy must be able to closely to mimic not only one but many possible ways of accomplishing the same stochastic task under different environment configurations . This representational demand motivates the introduction of high-capacity deep neural networks . We found the architecture , shown in Figure 3 , with residual connections , 20 convolution layers with 512 channels for a total of 22 million parameters , and instance normalization to drastically improve performance , as shown in Figure 6 of the Experiments section . Following the recommendations of ( Rajeswaran et al. , 2017b ) , we compare the performance of this model with a smaller ( 15 convolution layers with 32 channels and 1.6 million ) network proposed recently by Espeholt et al . ( 2018 ) and find size to matter . We note however that the IMPALA network of Espeholt et al . ( 2018 ) is large in comparison to previous networks used in important RL and control milestones , including AlphaGo ( Silver et al. , 2016 ) , Atari DQN ( Mnih et al. , 2013 ) , dexterous in-hand manipulation ( OpenAI et al. , 2018 ) , QT-Opt for vision-based robotic manipulation ( Kalashnikov et al. , 2018 ) , and DOTA among others .

Summary: This paper presents an RL method for learning from video demonstration without access to expert actions. The agent first learn to imitate the expert demonstration (observed image sequence and proprioceptive information) by producing a sequence of actions that will lead to the similar observations (require a renderer that takes actions and outputs images). The imitation loss is a similarity metric. Next, the agent explores the environment with both the imitation policy and task policy being learned; an off-policy RL algorithm D4PG is used for policy learning. Experiments are conducted on a simulated robot block stacking task.