Visual Imitation with Reinforcement Learning using Recurrent Siamese Networks

1 INTRODUCTION . Imitation learning and Reinforcement Learning ( RL ) often intersect when the goal is to imitate with incomplete information , for example , when imitating from motion capture data ( mocap ) or video . In this case , the agent needs to search for actions that will result in observations similar to the expert . However , formulating a metric that will provide a reasonable distance between the agent and the expert is difficult . Robots and people plan using types of internal and abstract pose representations that can have reasonable distances ; however , typically when animals observe others performing tasks , only visual information is available . Using distances in pose-space is ill-suited for imitation as changing some features can result in drastically different visual appearance . In order to understand how to perform tasks from visual observation a mapping/transformation is used which allows for the minimization of distance in appearance . Even with a method to transform observations to a similar pose space , each person has different capabilities . Because of this , people are motivated to learn transformations in space and time where they can reproduce the behaviour to the best of their own ability . How can we learn a representation similar to this latent space ? An essential detail of imitating demonstrations is their sequential and causal nature . There is both an ordering and speed in which a demonstration is performed . Most methods require the agent to learn to imitate the temporal and spatial structure at the same time creating a potentially narrow solution space . When the agent becomes desynchronized with the demonstration , the agent will receive a low reward . Consider the case when a robot has learned to stand when its goal is to walk . Standing is spatially close to the demonstration and actions that help the robot stand , as opposed to falling , should be encouraged . How can such latent goals be encouraged ? If we consider a phase-based reward function r = R ( s , a , φ ) where φ indexes the time in the demonstration and s and a is the agent state and action . As the demonstration timing φ , often controlled by the environment , and agent diverge , the agent receives less reward , even if it is visiting states that exist elsewhere in the demonstration . The issue of determining if an agent is displaying outof-phase behaviour can understood as trying to find the φ that would result in the highest reward φ′ = maxφR ( s , a , φ ) and the distance φ′ − φ is an indicator of how far away in time or out-ofphase the agent is . This phase-independent form can be seen as a form of reward shaping . However , this naive description ignores the ordered property of demonstrations . What is needed is a metric that gives reward for behaviour that is in the proper order , independent of phase . This ordering motivates the creation of a recurrent distance metric that is designed to understand the context between two motions . For example , does this motion look like a walk , not , does this motion look precisely like that walk . Our proposed Visual Imitation with Reinforcement Learning ( VIRL ) method uses Recurrent Siamese Networks ( RSNs ) and has similarities to both Inverse Reinforcement Learning ( IRL ) ( Abbeel & Ng , 2004 ) and Generative Advisarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) . The process of learning a cost function that understands the space of policies to find an optimal policy given a demonstration is fundamentally IRL . While using positive examples from the expert and negative examples from the policy is similar to the method GAIL uses to train a discriminator to recognize in distribution examples . In this work , we build upon these techniques by constructing a method that can learn policies using noisy visual data without action information . Considering the problem ’ s data sparsity , we include data from other tasks to learn a more robust distance function in the space of visual sequence . We also construct a cost function that takes into account the demonstration ordering as well as pose using a recurrent Siamese network . Our contribution consists of proposing and exploring these forms of recurrent Siamese networks as a way to address a critical problem in defining reward structure for imitation learning from the video for deep RL agents and accomplishing this on simulated humanoid robots for the challenging single shot learning setting . 2 RELATED WORK . Learning From Demonstration Searching for good distance functions is an active research area ( Abbeel & Ng , 2004 ; Argall et al. , 2009 ) . Given some vector of features , the goal is to find an optimal transformation of these features , such in this transformed space , there exists a strong contextual meaning . Previous work has explored the area of state-based distance functions , but most rely on pose based metrics ( Ho & Ermon , 2016 ; Merel et al. , 2017 ) that come from an expert . While there is other work using distance functions , including for example Sermanet et al . ( 2017 ) ; Finn et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Dwibedi et al . ( 2018 ) , few use image based inputs and none consider the importance of learning a distance function in time as well as space . In this work , we train recurrent Siamese networks ( Chopra et al. , 2005 ) to learn distances between videos . Partially Observable Imitation Without Actions For Learning from Demonstration ( LfD ) problems the goal is to replicate the behaviour of expert πE behaviour . Unlike the typical setting for humans learning to imitate , LfD often assumes the availability of expert action and observation data . Instead , in this work , we focus on the case where only noisy actionless observations of the expert are available . Recent work uses Behavioural Cloning ( BC ) to learn an inverse dynamics model to estimate the actions used via maximum-likelihood estimation ( Torabi et al. , 2018 ) . Still , BC often needs many expert examples and tends to suffer from state distribution mismatch issues between the expert policy and student ( Ross et al. , 2011 ) . Work in ( Merel et al. , 2017 ) proposes a system based on GAIL that can learn a policy from a partial observation of the demonstration . In this work , the discriminator ’ s state input is a customized version of the expert ’ s state and does not take into account the demonstration ’ s sequential nature . The work in ( Wang et al. , 2017 ) provides a more robust GAIL framework along with a new model to encode motions for few-shot imitation . This model uses an Recurrent Neural Network ( RNN ) to encode a demonstration but uses expert state and action observations . In our work , the agent is limited to only a partial visual observation as a demonstration . Additional works learn implicit models of distance ( Yu et al. , 2018 ; Pathak et al. , 2018 ; Finn et al. , 2017 ; Sermanet et al. , 2017 ) , none of these explicitly learn a sequential model considering the demonstration timing . An additional version of GAIL , infoGAIL ( Li et al. , 2017 ) , included pixel based inputs . Goals can be specified using the latent space from a Variational Auto Encoder ( VAE ) ( Nair et al. , 2018 ) . Our work extends this VAE loss using sequence data to train a more temporally consistent latent representation . Recent work ( Peng et al. , 2018b ) has a 2D control example of learning from video data . We show results on more complex 3D tasks and additionally model distance in time . In contrast , here we train a recurrent siamese model that can be used to en- able curriculum learning and allow for computing distances even when the agent and demonstration are out of sync . 3 PRELIMINARIES . In this section , we outline the general RL framework and specific formulations for RL that we rely upon when developing our method in Section 4 . Reinforcement Learning Using the RL framework formulated with a Markov Dynamic Process ( MDP ) : at every time step t , the world ( including the agent ) exists in a state st ∈ S , wherein the agent is able to perform actions at ∈ A , sampled from a policy π ( at|st ) which results in a new state st+1 ∈ S and reward rt according to the transition probability function T ( rt , st+1|st , at ) . The policy is optimize to maximize the future discounted reward J ( π ) = Er0 , ... , rT [ T∑ t=0 γtrt ] , ( 1 ) where T is the max time horizon , and γ is the discount factor , indicating the planning horizon length . Inverse reinforcement learning refers to the problem of extracting a reward function from observed optimal behavior Ng et al . ( 2000 ) . In contrast , in our approach we learn a distance that works across a collection of behaviours . Further , we do not assume the example data to be optimal . See Appendix 7.2 for further discussion of the connections of our work to inverse reinforcement learning . GAIL VIRL is similar to the GAIL framework ( Ho & Ermon , 2016 ) which uses a Generative Advasarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , where the discriminator is trained with positive examples from the expert trajectories and negative examples from the policy . The generator is a combination of the environment , policy and current state visitation probability induced by the policy pπ ( s ) . min θπ max θφ EπE [ log ( D ( s , a|θφ ) ) ] + Eπθπ [ log ( 1−D ( s , a|θφ ) ) ] ( 2 ) In this framework the discriminator provides rewards for the RL policy to optimize , as the probability of a state generated by the policy being in the distribution rt = D ( st , at|θφ ) . While this framework has been shown to work in practice , this dual optimization is often unstable . In the next section we will outline our method for learning a more stable distance based reward over sequences of images . 4 CONCEPTUAL DISTANCE-BASED REINFORCEMENT LEARNING . Our approach is aimed at facilitating imitation learning within an underlying RL formulation over partially observed observations o . Unlike the situation in GAIL , we do not rely on having accces to state , s and action , a information – our idea is to minimize a function that determintes the distance between two sequences observations , o , one from the desired example behavior oe , and another from the current agent behavior oa . We can then define the reward used within an underlying RL framework in terms of a distance function D , such that rt̂ ( o e , oa ) = −D ( oe , oa , t̂ ) = t̂∑ t=0 −d ( oet , oat ) , ( 3 ) where in our setting here D ( oe , oa , t̂ ) models a distance between video clips from time t = 0 to t̂ . A simple formulation of the approach above can be overly restrictive on sequence timing . While these distances can serve as RL rewards , they often provide insufficient signal for the policy to learn a good imitative behaviour , especially when the agent only has partial observations of the expert . We can see an example of this in Figure 1a were starting at t5 the agent ( in red ) begins to exhibit behaviour that is similar to the expert ( in blue ) yet the spatial distance indicates that this state is further away from the desired behaviour than at t4 . To encourge the agent to match any part of the expert behaviour we propose decomposing the distance into two distances , by adding a type of temporal distance shown in green . To compute a time independant distance we can find the state in the expert sequence that is closest to the agent ’ s current state argmin t̂∈T d ( oet̂ , o a t ) and use it in the following distance measure dT ( oe , oa , t̂ , t ) = . . .+ d ( oe t̂−1 , o a t−1 ) + d ( o e t̂ , oat ) + d ( o e t̂+1 , oat+1 ) + . . . ( 4 ) Using only a single state time-alined may lead to the agent fixating on mataching a single state in the expert demonstration . To avoid this the neighbouring states given sequence timing readjustment are used in the distance computation . This framework allows the agent to be rewarded for exhibiting behaviour that matches any part of the experts demonstration . The better is learns to match parts of the expert demonstration the more reward it is given . The previous spatial distance will then help the agent learn to sync up its timing with the deomonstration . Next we describe how we learn both of these distances . Distance Metric Learning Many methods can be used to learn a distance function in state-space . Here we use a Siamese network f ( oe , oa ) with a triplet loss over time and task data ( Chopra et al. , 2005 ) . The triplet loss is used to minimize the distance between two examples that are positive , very similar or from the same class , and maximize the distance between pairs of examples that are known to be unrelated . For more details see supplementary document . Sequence Imitation The distance metric is formulated in a recurrent style where the distance is computed from the current state and conditioned on all previous states d ( ot|ot−1 , . . . , o0 ) . The loss function is a combination of distance Eq . 9 and VAE-based representation learning objectives from Eq . 7 and Eq . 8 , detailed in the supplementary material . This combination of sequencebased losses assists in compressing the representation while ensuring intermediate representations are informative . The loss function used to train the distance model on a positive pair of sequences is : LV IRL ( oi , op , · ) =λ0LSN ( oi , op , · ) + λ1 [ 1 T T∑ t=0 LSN ( oi , t , op , t , · ) ] + λ2 [ 1 T T∑ t=0 LV AE ( oi , t ) + LV AE ( op , t ) ] + λ3 [ LAE ( oi ) + LAE ( op ) ] . Where λ = { 0.7 , 0.1 , 0.1 , 0.1 } . With a negative pair , the second sequence used in the VAE and AE losses would be the negative sequence . The Siamese loss function remains the same as in Eq . 9 but the overall learning process evolves to use an RNN-based deep networks . A diagram of the full model is shown in Figure 2 . This model uses a time distributed Long Short-Term Memory ( LSTM ) . A single convolutional network conva is first used to transform images of the demonstration oa to an encoding vector eat . After the sequence of images is distributed through conva there is an encoded sequence < ea0 , . . . , e a t > , this sequence is fed into the RNN lstma until a final encoding is produced hat . This same process is performed for a copy of the RNN lstma producing hbt for the agent ob . The loss is computed in a similar fashion to ( Mueller & Thyagarajan , 2016 ) using the sequence outputs of images from the agent and another from the demonstration . The reward at each timestep is computed as rt = ||hat −hbt ||+ ||eat − ebt || = ||lstma ( conva ( sat ) ) − lstma ( conva ( sbt ) ) ||+ ||conva ( sat ) − conva ( sbt ) || . At the beginning of each episode , the RNN ’ s internal state is reset . The policy and value function have 2 hidden layers with 512 and 256 units , respectively . The use of additional VAE-based image and Auto Encoder ( AE ) -based sequence decoding losses improve the latent space conditioning and representation . Algorithm 1 Learning Algorithm Initialize model parameters θπ and θd Create experience memory D ← { } while not done do for i ∈ { 0 , . . . N } do τi ← { } { st , oet , oat } ← env.reset ( ) for t ∈ { 0 , . . . , T } do at ← π ( ·|st , θπ ) { st+1 , oet+1 , oat+1 } ← env.step ( at ) rt ← −d ( oet+1 , oat+1|θd ) τi , t ← { st , oet , oat , at , rt } { st , oet , oat } ← { st+1 , oet+1 , oat+1 } end for end for D ← D ⋃ { τ0 , . . . , τN } Update d ( · ) parameters θd using D Update policy θπ using { τ0 , . . . , τN } end while Unsupervised Data labelling To construct positive and negative pairs for training we make use of time information in a similar fashion to ( Sermanet et al. , 2017 ) , where observations at similar times in the same sequence are often correlated and observations at different times will likely have little similarity . We compute pairs by altering one sequence and comparing this modified version to its original . Positive pairs are created by adding noise to the sequence or altering a few frames of the sequences . Negative pairs are created by shuffling one sequence or reversing it . More details are available in the supplementary material . Imitation data for 24 other tasks are also used to help condition the distance metric learning process . These include motion clips for running , backflips , frontflips , dancing , punching , kicking and jumping along with the desired motion . For details on how positive and negative pairs are created from this data , see the supplementary document . Importantly the RL environment generates two different state representations for the agent . The first state st+1 is the internal robot pose . The second state ot+1 is the agent ’ s rendered view , shown in Figure 2 . The rendered view is used with the distance metric to compute the similarity between the agent and the demonstration . We attempted using the visual features as the state input for the policy as well ; this resulted in poor policy quality . Details of the algorithm used to train the distance metric and policy are outlined in the supplementary document Algorithm 1 .

Summary: This paper studies the problem of visual imitation learning: given a video of an expert demonstration, take actions to reproduce that same behavior. The proposed method learns a distance metric on videos and uses that distance metric as a reward function for RL. Experiments show that this method does recover reasonable behaviors across a range of simulated robotic tasks.  Compared with prior methods, the main contribution of this work is that the distance metric is parametrized and trained as a siamese network.