Visual Imitation with Reinforcement Learning using Recurrent Siamese Networks

1 INTRODUCTION . Imitation learning and Reinforcement Learning ( RL ) often intersect when the goal is to imitate with incomplete information , for example , when imitating from motion capture data ( mocap ) or video . In this case , the agent needs to search for actions that will result in observations similar to the expert . However , formulating a metric that will provide a reasonable distance between the agent and the expert is difficult . Robots and people plan using types of internal and abstract pose representations that can have reasonable distances ; however , typically when animals observe others performing tasks , only visual information is available . Using distances in pose-space is ill-suited for imitation as changing some features can result in drastically different visual appearance . In order to understand how to perform tasks from visual observation a mapping/transformation is used which allows for the minimization of distance in appearance . Even with a method to transform observations to a similar pose space , each person has different capabilities . Because of this , people are motivated to learn transformations in space and time where they can reproduce the behaviour to the best of their own ability . How can we learn a representation similar to this latent space ? An essential detail of imitating demonstrations is their sequential and causal nature . There is both an ordering and speed in which a demonstration is performed . Most methods require the agent to learn to imitate the temporal and spatial structure at the same time creating a potentially narrow solution space . When the agent becomes desynchronized with the demonstration , the agent will receive a low reward . Consider the case when a robot has learned to stand when its goal is to walk . Standing is spatially close to the demonstration and actions that help the robot stand , as opposed to falling , should be encouraged . How can such latent goals be encouraged ? If we consider a phase-based reward function r = R ( s , a , φ ) where φ indexes the time in the demonstration and s and a is the agent state and action . As the demonstration timing φ , often controlled by the environment , and agent diverge , the agent receives less reward , even if it is visiting states that exist elsewhere in the demonstration . The issue of determining if an agent is displaying outof-phase behaviour can understood as trying to find the φ that would result in the highest reward φ′ = maxφR ( s , a , φ ) and the distance φ′ − φ is an indicator of how far away in time or out-ofphase the agent is . This phase-independent form can be seen as a form of reward shaping . However , this naive description ignores the ordered property of demonstrations . What is needed is a metric that gives reward for behaviour that is in the proper order , independent of phase . This ordering motivates the creation of a recurrent distance metric that is designed to understand the context between two motions . For example , does this motion look like a walk , not , does this motion look precisely like that walk . Our proposed Visual Imitation with Reinforcement Learning ( VIRL ) method uses Recurrent Siamese Networks ( RSNs ) and has similarities to both Inverse Reinforcement Learning ( IRL ) ( Abbeel & Ng , 2004 ) and Generative Advisarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) . The process of learning a cost function that understands the space of policies to find an optimal policy given a demonstration is fundamentally IRL . While using positive examples from the expert and negative examples from the policy is similar to the method GAIL uses to train a discriminator to recognize in distribution examples . In this work , we build upon these techniques by constructing a method that can learn policies using noisy visual data without action information . Considering the problem ’ s data sparsity , we include data from other tasks to learn a more robust distance function in the space of visual sequence . We also construct a cost function that takes into account the demonstration ordering as well as pose using a recurrent Siamese network . Our contribution consists of proposing and exploring these forms of recurrent Siamese networks as a way to address a critical problem in defining reward structure for imitation learning from the video for deep RL agents and accomplishing this on simulated humanoid robots for the challenging single shot learning setting . 2 RELATED WORK . Learning From Demonstration Searching for good distance functions is an active research area ( Abbeel & Ng , 2004 ; Argall et al. , 2009 ) . Given some vector of features , the goal is to find an optimal transformation of these features , such in this transformed space , there exists a strong contextual meaning . Previous work has explored the area of state-based distance functions , but most rely on pose based metrics ( Ho & Ermon , 2016 ; Merel et al. , 2017 ) that come from an expert . While there is other work using distance functions , including for example Sermanet et al . ( 2017 ) ; Finn et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Dwibedi et al . ( 2018 ) , few use image based inputs and none consider the importance of learning a distance function in time as well as space . In this work , we train recurrent Siamese networks ( Chopra et al. , 2005 ) to learn distances between videos . Partially Observable Imitation Without Actions For Learning from Demonstration ( LfD ) problems the goal is to replicate the behaviour of expert πE behaviour . Unlike the typical setting for humans learning to imitate , LfD often assumes the availability of expert action and observation data . Instead , in this work , we focus on the case where only noisy actionless observations of the expert are available . Recent work uses Behavioural Cloning ( BC ) to learn an inverse dynamics model to estimate the actions used via maximum-likelihood estimation ( Torabi et al. , 2018 ) . Still , BC often needs many expert examples and tends to suffer from state distribution mismatch issues between the expert policy and student ( Ross et al. , 2011 ) . Work in ( Merel et al. , 2017 ) proposes a system based on GAIL that can learn a policy from a partial observation of the demonstration . In this work , the discriminator ’ s state input is a customized version of the expert ’ s state and does not take into account the demonstration ’ s sequential nature . The work in ( Wang et al. , 2017 ) provides a more robust GAIL framework along with a new model to encode motions for few-shot imitation . This model uses an Recurrent Neural Network ( RNN ) to encode a demonstration but uses expert state and action observations . In our work , the agent is limited to only a partial visual observation as a demonstration . Additional works learn implicit models of distance ( Yu et al. , 2018 ; Pathak et al. , 2018 ; Finn et al. , 2017 ; Sermanet et al. , 2017 ) , none of these explicitly learn a sequential model considering the demonstration timing . An additional version of GAIL , infoGAIL ( Li et al. , 2017 ) , included pixel based inputs . Goals can be specified using the latent space from a Variational Auto Encoder ( VAE ) ( Nair et al. , 2018 ) . Our work extends this VAE loss using sequence data to train a more temporally consistent latent representation . Recent work ( Peng et al. , 2018b ) has a 2D control example of learning from video data . We show results on more complex 3D tasks and additionally model distance in time . In contrast , here we train a recurrent siamese model that can be used to en- able curriculum learning and allow for computing distances even when the agent and demonstration are out of sync . 3 PRELIMINARIES . In this section , we outline the general RL framework and specific formulations for RL that we rely upon when developing our method in Section 4 . Reinforcement Learning Using the RL framework formulated with a Markov Dynamic Process ( MDP ) : at every time step t , the world ( including the agent ) exists in a state st ∈ S , wherein the agent is able to perform actions at ∈ A , sampled from a policy π ( at|st ) which results in a new state st+1 ∈ S and reward rt according to the transition probability function T ( rt , st+1|st , at ) . The policy is optimize to maximize the future discounted reward J ( π ) = Er0 , ... , rT [ T∑ t=0 γtrt ] , ( 1 ) where T is the max time horizon , and γ is the discount factor , indicating the planning horizon length . Inverse reinforcement learning refers to the problem of extracting a reward function from observed optimal behavior Ng et al . ( 2000 ) . In contrast , in our approach we learn a distance that works across a collection of behaviours . Further , we do not assume the example data to be optimal . See Appendix 7.2 for further discussion of the connections of our work to inverse reinforcement learning . GAIL VIRL is similar to the GAIL framework ( Ho & Ermon , 2016 ) which uses a Generative Advasarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , where the discriminator is trained with positive examples from the expert trajectories and negative examples from the policy . The generator is a combination of the environment , policy and current state visitation probability induced by the policy pπ ( s ) . min θπ max θφ EπE [ log ( D ( s , a|θφ ) ) ] + Eπθπ [ log ( 1−D ( s , a|θφ ) ) ] ( 2 ) In this framework the discriminator provides rewards for the RL policy to optimize , as the probability of a state generated by the policy being in the distribution rt = D ( st , at|θφ ) . While this framework has been shown to work in practice , this dual optimization is often unstable . In the next section we will outline our method for learning a more stable distance based reward over sequences of images . 4 CONCEPTUAL DISTANCE-BASED REINFORCEMENT LEARNING . Our approach is aimed at facilitating imitation learning within an underlying RL formulation over partially observed observations o . Unlike the situation in GAIL , we do not rely on having accces to state , s and action , a information – our idea is to minimize a function that determintes the distance between two sequences observations , o , one from the desired example behavior oe , and another from the current agent behavior oa . We can then define the reward used within an underlying RL framework in terms of a distance function D , such that rt̂ ( o e , oa ) = −D ( oe , oa , t̂ ) = t̂∑ t=0 −d ( oet , oat ) , ( 3 ) where in our setting here D ( oe , oa , t̂ ) models a distance between video clips from time t = 0 to t̂ . A simple formulation of the approach above can be overly restrictive on sequence timing . While these distances can serve as RL rewards , they often provide insufficient signal for the policy to learn a good imitative behaviour , especially when the agent only has partial observations of the expert . We can see an example of this in Figure 1a were starting at t5 the agent ( in red ) begins to exhibit behaviour that is similar to the expert ( in blue ) yet the spatial distance indicates that this state is further away from the desired behaviour than at t4 . To encourge the agent to match any part of the expert behaviour we propose decomposing the distance into two distances , by adding a type of temporal distance shown in green . To compute a time independant distance we can find the state in the expert sequence that is closest to the agent ’ s current state argmin t̂∈T d ( oet̂ , o a t ) and use it in the following distance measure dT ( oe , oa , t̂ , t ) = . . .+ d ( oe t̂−1 , o a t−1 ) + d ( o e t̂ , oat ) + d ( o e t̂+1 , oat+1 ) + . . . ( 4 ) Using only a single state time-alined may lead to the agent fixating on mataching a single state in the expert demonstration . To avoid this the neighbouring states given sequence timing readjustment are used in the distance computation . This framework allows the agent to be rewarded for exhibiting behaviour that matches any part of the experts demonstration . The better is learns to match parts of the expert demonstration the more reward it is given . The previous spatial distance will then help the agent learn to sync up its timing with the deomonstration . Next we describe how we learn both of these distances . Distance Metric Learning Many methods can be used to learn a distance function in state-space . Here we use a Siamese network f ( oe , oa ) with a triplet loss over time and task data ( Chopra et al. , 2005 ) . The triplet loss is used to minimize the distance between two examples that are positive , very similar or from the same class , and maximize the distance between pairs of examples that are known to be unrelated . For more details see supplementary document . Sequence Imitation The distance metric is formulated in a recurrent style where the distance is computed from the current state and conditioned on all previous states d ( ot|ot−1 , . . . , o0 ) . The loss function is a combination of distance Eq . 9 and VAE-based representation learning objectives from Eq . 7 and Eq . 8 , detailed in the supplementary material . This combination of sequencebased losses assists in compressing the representation while ensuring intermediate representations are informative . The loss function used to train the distance model on a positive pair of sequences is : LV IRL ( oi , op , · ) =λ0LSN ( oi , op , · ) + λ1 [ 1 T T∑ t=0 LSN ( oi , t , op , t , · ) ] + λ2 [ 1 T T∑ t=0 LV AE ( oi , t ) + LV AE ( op , t ) ] + λ3 [ LAE ( oi ) + LAE ( op ) ] . Where λ = { 0.7 , 0.1 , 0.1 , 0.1 } . With a negative pair , the second sequence used in the VAE and AE losses would be the negative sequence . The Siamese loss function remains the same as in Eq . 9 but the overall learning process evolves to use an RNN-based deep networks . A diagram of the full model is shown in Figure 2 . This model uses a time distributed Long Short-Term Memory ( LSTM ) . A single convolutional network conva is first used to transform images of the demonstration oa to an encoding vector eat . After the sequence of images is distributed through conva there is an encoded sequence < ea0 , . . . , e a t > , this sequence is fed into the RNN lstma until a final encoding is produced hat . This same process is performed for a copy of the RNN lstma producing hbt for the agent ob . The loss is computed in a similar fashion to ( Mueller & Thyagarajan , 2016 ) using the sequence outputs of images from the agent and another from the demonstration . The reward at each timestep is computed as rt = ||hat −hbt ||+ ||eat − ebt || = ||lstma ( conva ( sat ) ) − lstma ( conva ( sbt ) ) ||+ ||conva ( sat ) − conva ( sbt ) || . At the beginning of each episode , the RNN ’ s internal state is reset . The policy and value function have 2 hidden layers with 512 and 256 units , respectively . The use of additional VAE-based image and Auto Encoder ( AE ) -based sequence decoding losses improve the latent space conditioning and representation . Algorithm 1 Learning Algorithm Initialize model parameters θπ and θd Create experience memory D ← { } while not done do for i ∈ { 0 , . . . N } do τi ← { } { st , oet , oat } ← env.reset ( ) for t ∈ { 0 , . . . , T } do at ← π ( ·|st , θπ ) { st+1 , oet+1 , oat+1 } ← env.step ( at ) rt ← −d ( oet+1 , oat+1|θd ) τi , t ← { st , oet , oat , at , rt } { st , oet , oat } ← { st+1 , oet+1 , oat+1 } end for end for D ← D ⋃ { τ0 , . . . , τN } Update d ( · ) parameters θd using D Update policy θπ using { τ0 , . . . , τN } end while Unsupervised Data labelling To construct positive and negative pairs for training we make use of time information in a similar fashion to ( Sermanet et al. , 2017 ) , where observations at similar times in the same sequence are often correlated and observations at different times will likely have little similarity . We compute pairs by altering one sequence and comparing this modified version to its original . Positive pairs are created by adding noise to the sequence or altering a few frames of the sequences . Negative pairs are created by shuffling one sequence or reversing it . More details are available in the supplementary material . Imitation data for 24 other tasks are also used to help condition the distance metric learning process . These include motion clips for running , backflips , frontflips , dancing , punching , kicking and jumping along with the desired motion . For details on how positive and negative pairs are created from this data , see the supplementary document . Importantly the RL environment generates two different state representations for the agent . The first state st+1 is the internal robot pose . The second state ot+1 is the agent ’ s rendered view , shown in Figure 2 . The rendered view is used with the distance metric to compute the similarity between the agent and the demonstration . We attempted using the visual features as the state input for the policy as well ; this resulted in poor policy quality . Details of the algorithm used to train the distance metric and policy are outlined in the supplementary document Algorithm 1 .

Summary: This paper presents visual imitation with reinforcement learning (VIRL), an algorithm for learning to imitate expert trajectories based solely on visual observations, and without access to the expert’s actions.  The algorithm is similar in form to GAIL and its extensions, learning a reward function which captures the similarity between an observed behavior and the expert's demonstrations, while simultaneously using reinforcement learning to find a policy maximizing this reward, such that the learned policy will replicate the demonstrated behavior as well as possible.  A key feature of this method is that the learned reward function is defined by a learned distance metric, which evaluates the similarity between the agent's current trajectory, and the nearest demonstrated expert trajectory.