Visual Imitation with Reinforcement Learning using Recurrent Siamese Networks
Authors: Glen Berseth
Florian Golemo
Christopher Pal
Venue:  (2021)
Keywords: Reinforcement Learning
Imitation learning

Abstract: It would be desirable for a reinforcement learning (RL) based agent to learn behaviour by merely watching a demonstration. However, defining rewards that facilitate this goal within the RL paradigm remains a challenge. Here we address this problem with Siamese networks, trained to compute distances between observed behaviours and an agent's behaviours. We use an RNN-based comparator model to learn such distances in space and time between motion clips while training an RL policy to minimize this distance. Through experimentation, we have also found that the inclusion of multi-task data and an additional image encoding loss helps enforce temporal consistency and improve policy learning. These two components appear to balance reward for matching a specific instance of a behaviour versus that behaviour in general. Furthermore, we focus here on a particularly challenging form of this problem where only a single demonstration is provided for a given task -- the one-shot learning setting. We demonstrate our approach on humanoid, dog and raptor agents in 2D and a 3D quadruped and humanoid. In these environments, we show that our method outperforms the state-of-the-art, GAIfO (i.e. GAIL without access to actions) and TCNs.

Under review as a conference paper at ICLR 2021
VISUAL IMITATION WITH REINFORCEMENT LEARN- ING USING RECURRENT SIAMESE NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
It would be desirable for a reinforcement learning (RL)-based agent learn be- haviour by merely watching a demonstration. However, defining rewards that facilitate this goal within the RL paradigm remains challenge. Here we ad- dress this problem with Siamese networks, trained to compute distances between observed behaviours and agent's behaviours. We work (RNN)-based comparator model to learn such distances in space and time be- tween motion clips while training an RL policy to minimize this distance. Through experimentation, we have also found that the inclusion of multi-task data and an additional image encoding loss helps enforce temporal consistency and improve policy learning. These two components appear to balance reward for matching a specific instance of a behaviour versus that behaviour in general. Furthermore, we focus here on particularly challenging form of this problem where only a single expert demonstration is provided for a given task. We demonstrate our approach on simulated humanoid, dog and raptor agents in 2D and 3D quadruped and humanoid. In these environments, we show that our method outperforms the state-of oftte-of-the-art Generative Adversarial Imitation from Observation (GAIfO) (i.e. Generative Adversarial Imitation Learning (GAIL) without access to actions) and Network (TCN).
recurrent neural net-
INTRODUCTION
In nature, many intelligent beings (agents) can imitate their peers (experts) by watching them. In order to learn from observation alone, the agent must compare its own behavior to the expert's, 's mimicking their movements (Blakemore & Decety, (001). While this process seems to come as second nature to humans and many animals, formulating framework and metrics that can measure how different expert's 's demonstration from an agent's reenactment in this setting is challenging. While robots have access to their state information, humans and animals simply observe others performing tasks relying only upon visual perceptions of demonstrations, creating mental representation the target motion. In this work ask: Can agents learn these representations in order to learn imitative policies from a single demonstration?
One of the core problems of imitation learning is how to align a demonstration in space and time with the agent's 's own state. To address this, the imitation framework has to learn a distance function between agent and expert. The distance function in our work makes use of positive and negative examples, including types of adversarial examples, similar to GAIL (Ho & Ermon 2016) and GAIFO TTorai al. 2018b). These works require expert policies generate large amount of demonstration data and GAIL need action information as well. These works train discriminator to recognize in in-distribution examples. In this work, we extend these techniques by learning distances between motions, using noisy visual data without action information, and using the distance function as reward signal to train RL policies. is given. As show in the paper, this new formulation can be extended to assist in training the distance function using multi-task data, which improves the model's accuracy and enables its reuse on different tasks. Additionally, while previous methods have focused computing distances between single states, we construct cost function that takes into account the demonstration ordering as well the state using a recurrent Siamese network to learn smoother distances between motions.
Figure Tb an outline of our method for visual imitation

---

Under review as a conference paper at ICLR 2021
ConvNet encider
Encoded state
state Loss terms
Agont
Agent
LSTM
LSTM
Boward
LSTM
LSTM
LSTM
Expert
Encodder
Decoder
(a) Losses for training the encoders/decoders
(b) Reward generation for the agent
Figure 1: Overview of our method: We aim to learn distance function Ia) and then use that distance
function as reward function for RL (1b). At the current timestep, observations (o) of the reference motion and the agent are encoded (e) and fed into LSTMs (leading to hidden states h). Fig. Ta shows how the reward model is trained using both Siamese and AE losses. There are: VAE reconstruction losses on static images (CvAEE) and one for the agent (which we do not show in pink to simplify the figure). There is Siamese loss between encoded images (CsNI) and Siamese loss that is computed between encoded states over time (CsNs). Fig. shows how the reward is calculated at every timestep. Reward for the agent at every timestep consists of the distance between encoded images and encoded LSTM hidden states.
sgguence-to-sequenee AE losses (Craes), one for the reference
Our contribution, Visual Imitation with RL (VIRL), consists of proposing and exploring these forms of recurrent Siamese networks as a way to address a critical problem in defining the reward structure for imitation learning from video for deep RL agents. We accomplish this using simulated humanoid robots inhabiting a physics simulation environment and for the challenging setting of imitation learning from single expert demonstration. Our approach enables us train agents that can imitate many types of behaviours that include walking, running and jumping. We perform experiments for multiple simulated robots in both 2D and 3D, including recent Sim2Real quadruped robots and humanoid with 38 degrees of freedom (DoF), which is a particularly challenging problem domain.
2
PRELIMINARIES
Here we provide very brief review of some fundamental methods that are related to the new approach we present here. Reinforcement Learning (RL) frequently formulated within the framework of Markov decision process (MDP) where at every time step t, the world (including the agent) exists in state s; E S. where the agent is able perform actions a, and where states and actions are discrete. The action to take is determined according to policy T(als))) which results in new state 8t+1 E and reward rt = R(s,, at, 8(+1) according the transition probability function T(r,, $14115e, a.). The policy is optimized to maximize the future discounted reward
E Brg....r Σn 7're where T is the max time horizon, and y is the discount factor, indicating the planning horizon length. The formulation above generalizes continuous states and actions, which is the situation for the agents we consider in our work.
Imitation Learning is typically cast as the process of training new policy to reproduce the behaviour of some expert policy. Behavioral cloning is fundamental method for imitation learning. Given an expert policy TE possibly represented as a collection of trajectories T = ((50, on), (ST, aT)) a be learned to match this trajectory using supervised learning and maximizing the
new policy
expectation Ex ITT log (T(a) St, 0m) While this simple method can work well, often suffers
from distribution mismatch issues leading to compounding errors as the learned policy deviates from the expert's behaviour (Ross et al. 201 Inverse reinforcement learning avoids this issue by extracting a reward function from observed optimal behaviour (Ng et al., 2000). In our approach, we learn distance function that allows agent to compare observed behavior to its own current behavior to define its reward rt at a given time step. Our comparison is performed with respect to reference activity but the comparison network behaviours. Further, we do not assume the example data to be optimal. See Appendix 772 for further discussion of the connections of our work to inverse reinforcement learning.
be trained across collection of different
2

---

Under review as a conference paper at ICLR 2021
VAEs are popular approach for learning lower-dimensional representations of distribution (Kingma & Welling 2014). A VAE consists of two parts, an encoder qs, with parameters & and decoder Py with parameters db. The encoder maps inputs to latent encoding and in turn the decoder transforms z back to the input space pe (x|().. The model parameters for both &i and to are trained jointly to maximize
LVAE(S, v)) = D (u. (2ol2)))))ll( + (ula (x|[))]
where DKL. is the Kullback-Leibhe divergence, p(z) is prior distribution over the latent space. The encoder la- or inference model takes the form of diagonal covariance multivariate Gaussian distribution q = o where the mean, (x( is typically given by a deep neural network.
Sequence to sequence models can be used to learn the conditional probability of one sequence given another p(yo..... .... y/ ( ZT), where 20...., 2T and y yu..... 3/r sequences. Here we will use extensions of encoder-decoder structured, autoencoding recurrent neural networks which learn latent representation h that compresses the information in 2T. Our model for decoding the sequence can then be written as
ply) = p(yo h) II pyo............................. P.................. 11-1).h)
(2)
t=1
This method has been used for learning compressed representations for transfer learning (Zhu et al. 2016) and 3D shape retrieval (Zhuang et al.) (015). In our case this type of autoencoding can help regularize model, which has primary goal of computing distances between sequences using Siamese structured autoencoding RNN.
3 VISUAL IMITATION WITH REINFORCEMENT LEARNING
High-level Overview Our method is similar other Imitation Learning frameworks like GAIfO in that we train a system to give the agent a reward depending on how closely is imitating the expert. We interleave training between refining the reward generator with rollouts and using the reward generator to train the policy and gather more rollouts. The reward generator consists of several components and losses that are described in the following section but coarsely, observations of both the expert and agent are encoded with VAEs and LSTMs, to be later decoded in inverse order. Contrastive loss ("Siamese Network triplet loss") is used maximize similarity between the encoding of similar frames/sequences and dissimilarity between incorrect frames and shuffled sequences Hadsell et al. 2006). Once this system has been initialized, every timestep a reward for the agent's 's policy is calculated as difference between the current encoded observation and also the difference of the sequence so far between expert and agent. the following section, we first discuss how the encoder/decoder networks are trained, then how they generate reward for the agent, and finally which data augmentation techniques we used to make the system more robust.
The Sequence Encoder/Decoder Networks Figure Ia shows an outline of the system. A single convolutional network Conv" is used to transform observations (images) at time t of the expert demonstration of to encoding vector eç. After the sequence of observations Conv" there is an encoded sequence (e....., ef), this sequence is fed into the RNN LSTMM until final encoding is produced hç- This same process is performed for copy of the RNN I.STM producing hę for the agent o". The final encoding of the expert is fed into a separate RNN LSTM which generates series of decoded latent representations (ef.... ef) which are then decoded back to images with a deconvolutional network Deconvé. The same applied to the agent with RNN LSTM². latent representations e?)), and deconvolutional network Deconv", respectively.
passed through
Loss Terms The encoding of single observation of either agent or expert at given timestep is trained using the VAE loss AV from Eq[[] full sequence of observations of either agent or expert is encoded and then decoded back, and the L.STMs are trained with the loss LRAES from E
We found these frame- and sequence sgquence-autoenconn
to improve latent space conditioning. A
wwhich is different from existing methods like GAIFO in that enforce similarity over sequences, not just individual transitions. This allows us temporally align the demonstration with the agent.
3

---

Under review as a conference paper at ICLR 2021
by-ffmmee Siamese loss between ef of the expert and ef of the agent enforces individual frames to be encoded similarly. This Siamese Network image loss LSNI is defined below in Eq Lastly and primarily, Siamese loss between a full encoded sequence of the expert h; and sequence of the agent hy forces not just individual frames but the representation of whole sequences to match up if they are alike. This Siamese Network sequence loss LsNs is also defined in Eq same formula, just expects a sequences instead of frames as input. The Siamese Network loss (both for images and sequences) is defined as:
3 since it uses the
LSN (o,, Op. y: 6) y* [If( o) f( ((1-)))())))))))))))0 (max(p-(11f(o;o)) (m f(on: .)
(3)
where y E [0, 1] is the indicator for positive/negative samples. When y 1, the sample is positive and the distance between current observation o, to positive sample op should be minimal. When y = 0, the sample is negative and the distance between o; and negative example 0n should be maximal. This loss is computed over batches of data that are half positive examples and half negative, The margin p is used as an attractor anchor to pull the negative example output away from o; and push values towards [0, 1] range. f(-) computes the output from the underlying network (i.e. Conv or LSTM). The data used to train the Siamese network is a combination of observation trajectories O .... .r) generated from simulating the agent in the environment and the expert demonstration. For our recurrent model the observations Op. On, O, are sequences. This combination of image-hased based and requence-based losses assists in compressing the representation while ensuring intermediate representations remain informative. The combined loss to train the model on a positive pair of sequences (y= 1) is:
CVIRI(O;: Po yr bs w, = >1 Liss Op. Up A2
E CSNI Op., vb)
Contrastive sequence loss
Contrastive frame loss
A3 >s|CRABs(O:: o, v w, p) + L CRAES (Op Φ, 1/, w. p)]
Recurrent autoeneader loss (full sequence)
X1
(VAeE(OO.:i000) LV (0 (Oprr (Op.i t=
Variational autoencoder loss (individual frames)
(4)
Where the relative weights of the different terms A1 {0.7, 0.1, 0.1, the image encoder convnet is é, the image decoder a), the recurrent encoder w. and the recurrent decoder p.
Reward Calculation The model trained using the method described above is used to calculate the distance between two sequences of observations seen thus far up to time t as d(O", O"; Φ w) = |[w (06::: 6) - wof 6)]l and the reward as r r(0 o O0 0": o, w). This means every timestep, the reward is computed as r, = ||g he + lee elll This can be expanded to rr I|M (CONV* (o).)) I.STM ( ||conv (o;) Conv"(o?) and is shown in Figure lb During RL training, we compute a distance given the sequence observed so far in the episode. This method allows us to train distance function in the observations space where all we need to provide is labels that denote if two observations or sequences are similar or not.
Training the Model Details of the algorithm used to train the distance metric and policy are outlined in Algorithm| I We consider variation on the typical RL environment that produces 3 different outputs. two for the agent and for the demonstration and no reward. The first is the internal robot pose, which we shall refer to as the state 8,. The second and third representation is the agent's s rendered view, or observation of and the demonstration of, shown in Figure 1b The rendered views used with the distance metric to compute the similarity between the agent and the demonstration. We Iearn the policy of our agents using RL and the Region Policy Optimization (TRPO) algorithm (Schulman al. 2015) with a reward signal that is learned as discussed below.
Unsupervised Data labelling To construct positive and negative pairs for training, we make use of time information in similar fashion to (Sermanet et al.) (2017) and adversarial information similar to GAIL. Timing information is used where observations at similar times in the same sequence are often correlated, and observations at different times will likely have little similarity. We compute
4

---

Under review as a conference paper at ICLR 2021
these sequence pairs by altering one sequence and comparing this modified version to its original. Positive pairs are created by adding Gaussian noise with o 0.05 to the images in the sequence or swapping or duplicating random frames of the sequences. Negative pairs are created by shuffling, cropping or reversing one sequence. Additionally, we include adversarial pairs where positive pairs come from the same distribution, for example, two motions for the agent two from the expert. Negative pairs then include one from the expert and one from the agent. More details are available in the supplementary document.
Data Augmentation We apply several data augmentation methods produce additional data for training the distance metric. Using methods analogous to the cropping and warping methods popular in computer vision (He et al. 2015) we randomly crop sequences and ran- domly warp the demonstration timing, The crop- ping is performed by both initializing the agent to random poses from the demonstration motion and terminating episodes when the agent's 's head,
Algorithm 1 Learning Algorithm
1: Initialize parameters 0z 0a. D D+b
2:
while not done do for E (O.....N N do
3:
4:
{s,, 0f 0f +- env.resetO, 7 0
5:
for 1 E (0......7 7) do
6:
at t (-1ss,, 0x)
7:
{s of 0f 2441 f env.step(a:)
8:
Ti t fsr. of of at)
9:
{sr, of. of)
ssti of 1,11 of+1}
hands or torso contact the ground. As the agent improves, the average length of each episode
10:
end for ro:: +
11:
(0:1111 o 1a)
increases, and so to will the average length of the cropped window. The motion warping is done by replaying the demonstration motion at different speeds. Two additional methods influ- ence the data distribution. The first method is Reference State Initialization (RSI) (Peng et al.) 2018a), where the initial state of the agent and expert randomly selected from the expert demon- stration. With this property, the environment can also be thought of form of memory replay. The environment allows the agent to go back to random points in the demonstration as if replaying remembered demonstration. The second is Early Episode Sequence Priority (EESP) where the probability a sequence x is cropped ending at p(i) ien(x) carlier in the episode.
12:
end for
13:
D - --UU(T) DU(T",...,,*.,
14:
Update d(-)) parameters 0g using D
15:
Update 0₂ with {{r", ,r,,,,))..... rr....., { rN)
16:
end while
.increasing the likelihood of starting
Σ
4 RELATED WORK
For the purposes of this work, we group existing imitation learning methods based on the type and quantity of data needed to learn. In the first tier, there is GAIL (Ho & Ermon, 2016) and related methods, which require access to expert policies, states and actions and require large quantities of expert data. In the second tier, the need for expert actions is relaxed in methods like GAIfO [Torabi] et al. 2018b). In the third tier, the need for ground truth states relaxed in favor of images which casier to obtain in methods like T-REX and D-REE et al. 2019 2020). These methods still require many examples of data from a policy trained on the agent in the same simulation with the same dynamics. Lastly, in the fourth tier, the need for multiple demonstrations and matching dynamics is relaxed in methods like TCN (Sermanet et al. 2018) and ours.
Methods that require access to expert states and actions. Generative Adversarial Imitation Learn- ing or GAIL (Ho & Ermon, 2016). uses the well known Generative Adversarial Network (GAN) framework applied to learning an RL policy (Goodfellow et al. 2014). In GAIL, the GAN's discrimi- nator is trained with positive examples from expert trajectories and negative examples from the current policy. However, using a discriminator is only one possible way of judging the distance between expert and agent and searching for good distance functions between states is an active research area (Abbeel & Ng 2004] AArgal et al. 2009. Finn al. 2016 Brown et al.) 2019). Given some vector of features, the goal of distance- based imitation learning is to find an optimal transformation of these features, such that in this transformed space, there exists a more meaningful distance between expert demonstrations and agent trajectories. Previous work has explored the area of state-based distance functions, but most rely on the availability of an expert policy to sample data (Ho & Ermon 2016; Merel et al.) 2017). In the section hereafter we demonstrate how VIRL learns more stable distance- based reward over sequences of images (as opposed to states) and without access to actions or expert policies.
5

---

Under review as a conference paper at ICLR 2021
Methods that don't require access to actions. For learning from demonstrations (LfD) problems, the goal is to replicate the behaviour of an expert Tg. GAlfO (Torabi et al., 2018b) has been proposed as extension of GAIL that does not require actions. This and other recent works this area require access to an expert policy to sample more states (Sun et al.) 2019 Yang et al. 2019). By comparison, our method only needs a single fixed demonstration. Other recent work uses behavioural cloning (BC) to learn an inverse dynamics model to estimate the actions used via -ikimuum-ikklihoodd estimation (Torabi et al. 2018a). Still, BC often needs many expert examples and tends to suffer from state distribution mismatch issues between the expert policy and student (Ross et al. 2011a).
Additional works learn implicit models of distance YYu et al., 2018 Finn et al. 2017; Sermanet et al.) 2017; Merel et al., 2017]] Edwards et al. 2019 Sharma et al. 2019) they require large amounts demonstration data and none of these explicitly learn a sequential model considering the demonstration timing. The work in (Wang et al.) 2017 Li et al. 2017, Peng et al., (018b) includes a more robust GAIL framework along with a new model to encode motions for few-shot imitation but they need access to an expert policy to sample data from. In this work, we train recurrent Siamese networks (Chopra et al., 2005) to learn more meaningful distances between videos. Other work uses state-only demonstration ranking to out-perform the demonstration data but requires many demonstrations and ranking information (Brown et al.] 2019: 2020). We show results on more complex 3D tasks and additionally model distance in time, i.e. due to the embedding of the full sequence, our model can compute meaningful distances between agent and demonstration even they are out of sync.
Methods that work on images instead of states Some works like Sermanet et al. (2017) Finn et al. (2017); Liu et al. (2017); Dwibedi et al. (2018), use image-based inputs instead of states but require many demonstrations. Further, these models only address spacial alignment (i.e. matching joint positions/orrintaations)) but not temporal alignment (i.e. getting the sequence of motion correct rather than just the individual frames) between expert demonstration and agent motion like our recurrent sequence model does. Other works that perform imitation from only image-based information like (Pathak et al. 2018) do so between goal states.
Methods that require few visual observations and allow for a different source environment Time-Contrastive Networks (TCNs) (Sermanet et al. (2018) were proposed as way to use metric learning loss to embed simultaneous viewpoints of the same object. They use TCN embeddings as features in the system state which are provided to reinforcement learning algorithm, specifically, PILQR (Chebotar et al. (017) which combines model- based learning, linear time varying dynamics and model-free corrections. In contrast, our Siamese network-bar approach is used learn the reward for an arbitrary subsequent RI. algorithm. Our method does not rely on multiple views and we use an RNN-based autoencoding approach regularize the distance computations used for rewards generated by our models.
In summary, all existing methods either (a) require ground truth states and actions, (b) require access to states as opposed to images, (c) require large amounts of training data or require the expert to be trained in the same environment under the same dynamics, or (d) require the expert motion to be leamned from spacial alignment alone, as opposed to spatiotemporal aligmment. Our method requires none of these and therefore aims to provide a more generic solution.
RESULTS AND ANALYSIS
We use a collection of different simulation environments to validate VIRL's ability to train imitative agents. In these simulated robotics environment, the agent is learning to imitate a given reference demonstration. Each of these simulation environment provides a hard-coded reward function based on the robot's pose that is used to evaluate the policy quality independently. The demonstration M the agent is learning to imitate is produced from a clip of mocap data. The mocap data is used to animate, kinematically, a second robot in the simulation. Frames from the simulation are captured and used as video input to train the distance metric. The images captured from the simulation are converted to grey-scale with 48 x 48 pixels. The policy instead received the state data, often as link distances and velocities relative to the robot's centre of mass (COM). These simulation environments are new and have all been updated to take motion capture data and produce view video data that can be used for training RI. agents generating data for computer vision tasks. The environment
6

---

Under review as a conference paper at ICLR 2021
includes challenging and dynamic tasks for humanoid, dog and raptor robots. Some example tasks are imitating running, jumping, trotting, and walking, shown in Figure 2 and Figure
2D Video Imitation Results Our first experiments evaluate the method's ability to learn a complex eyclic motion for a simulated robots given a single motion demonstration, similar to (Peng & van de Panne 2017), but instead using video. For each of these simulated robots VIRL is able to learn a robust gate even though is only given noisy partial observations of a demonstration. Results for these environments can found in Figure (humanoid2d) and in Figure ?? (dog2d and raptor2d).
Figure Frames from the humanoid2d, dog2d and raptor2d environments in our experiments.
3D Robot Video Imitation We train imitation policies from videos over number of environments including two quadrupedal robots simulators used for Sim2Real research, the Laikago (Peng et al.) 2020) and Pupper (Kau et al.). In these two quadruped simulators the environment is altered to produce additional video from a recorded demonstration of the robot performing a task. Additionally, we use environments with simulated humanoid robot, the agent is learning to imitate given reference motion of a walk, run, jump or zombie motion. A single motion demonstration is provided by the simulation environment from all other tasks for the walking task this would be: walking-dynamic-spae front-flips, back-flips, dancing, jumping, punching and kicking) that only used to train the distance metric. We also include data from a modified version of the task that has a randomly generated speed modifier w [0.5, 2.0] dalking-dynamic-speer additional data is used to provide richer understanding of distances in space and time to the distance metric. The method is capable of learning policies that produce similar behaviour to the expert across diverse set of tasks. We show example trajectories from the learned policies in Figure B and in the supplemental Video, takes 5- days to train each policy in these results on a core machine with an Nvidia GTX1080 GPU.
cyclic motion. During learning, we can include additional data
running, jogging,
which warps the demonstration timing. This
Algorithm Analysis and Comparison In Fig- ure 4a we show evaluation of the learning ca- pabilities and improvements of VIRL compared with two other methods that learn distance function in state space, GAIFO (Torabi et al. 2018b) and VAF trained encode agent and reference observations and compute distances between those encodings, similar to Nair al.. (2018) and TCNs. We find that the VAE alone does not appear to model distances between states in a way that helps with RL, possibly due to the decoding complexity. Similarly, the GAIFO baseline produces very jerky motion or stands still, both of which are contained in the imita- tion distribution. Our full VIRL method considers the temporal structure of the data, learns faster and produces higher value policies.
Figure 3: Rasterized frames of the agent's motion after training on humanoid3d walking and running. Addition- ally, zombie walk and jumping policy can be found on the project website: https://sites.oogl google.coomvieww/vil Also see Appendix Fig.
In Figure 4b we compare the importance of adding the spatial VAE ||e: - e/122 and temporal LSTM ||اg h I components of VIRL. Using the recurrent representation alone allows learning to progress quickly but can lead to difficulties informing the policy of how to best match the desired example. On the other hand, using only the encoding between single frames as is done with TCNs slows learning due to limited reward when the agent quickly becomes out-of-sync with the demonstration behaviour. We achieved the best results by combining the representations from these two models. This is shown for completely different agent type (a 2d walking dog) and across many humanoid tasks in 4 The use of multi-task data not necessary but provides an improvement and was only used for the Walking task. Note that we experimented with using visual features as the state input for the policy as well; however, this resulted in poor policy quality.
Ablation Analysis We conduct ablation studies for learning policies for 3D humanoid control in Fig- ure 5a and (5b] We compare the effects of data augmentation methods, network models and the use of additional data from other tasks (24 additional tasks like back- flips, see appendix 7.4). We compared using different length sequences for training, shorter (where the probability of the length decays
7

---

Under review as a conference paper at ICLR 2021
TCN
URR CAIOO
humanoid2d walk
(b) humanoid2d walk
dog2d
(d) raptor2d
ViR
Walking
ZombieWalk
Running
(h) Jumping
Figure 4: (a) Comparisons between VIRL, simple VAE and GAIfO for the humanoid walking task. (b) Comparing our model with both an image VAE and and LSTM autoencoder (VIRL) with a model only having the LSTM autoencoder, versus TCN. (c) Comparisons of VIRL with TCN. In these plots, the large solid lines are the average performance of a collection of policy training simulations.
VIRL-MID
(a)
Walking Ablations
(h) Distance Metrics
(c) KombieWalk, LSTM (d) Running, MultiTask
Figure 5: (a) Ablation analysis of VIRL on the Walking Task showing the mean reward over of the number of simulated actions. The legend is the same as (b) where we examine the impact on our loss under the diffcrent distance metries resulting from the ablation analysis. We find that including multi-task data (only available for the humanoid3D) and both the VAE and recurrent AE losses provide the performant models. (c) Ablating recurrent autoencoder from VIRL dramatically impairs the ability to how to walk like a Zombie. (d) The use of multi-task training data helps learn better policies for running (away from Zombies if desired).
linearly), uniform random and max length available. For these more complex and challenging three dimensional humanoid (humanoid3d) control problems, the data augmentation methods, including EESP, increases average policy quality marginally. The use of multitask data Figure 5d and additional recurrent sequence autoencoder (RSAE) greatly improves the methods ability learn as observed in Figure [c] As one can observe, our method performs better this setting. Further analysis is available in the Appendix including additional comparison with TCNs in Figure 11 and using the 2D Raptor agent Figure 4d
Sim2Real for Quadreped Robots We use VIRL to train policies for two simulated quadrapeds in Fig- ure 66. that have been used for Sim2Real transfer. With these trained policies
possible to transfer the VIRL policies trained from single demonstration, a real robot. The resultsing behaviours are availble at: httpss///sites.. gwwgle.coooooooooooooo www.gl..oo//viive...... wwt.//owo//wwwo///////// We find that the Laikago environemnt is particularaly chal- lenging to learn; however, we are able to learn good policies on the pupper in a day.
Figure 6: Pupper and Laikago Envs.
DISCUSSION AND CONCLUSION
this work, we have created a new method for learning imitative policies from single demonstration. The method uses Siamese recurrent network to learn distance function in both space and time. This distance function is trained on video data where the true state of the agent is noisily and partially observed. We this to learn reward function for training RL policy. Using data from other motion styles and regularization terms, VIRL produces policies that demonstrate similar behaviour to the demonstration.
8

---

Under review as a conference paper at ICLR 2021
We believe VIRL will benefit from a more extensive collection of multi-task data and increased variation of each task. Additionally, if the distance metric confidence is available, this information could be used to reduce variance and overconfidence during policy optimization. We also believe that it is likely that learning a reward function while training adds additional variance to the policy gradient. This variance may indicate that the bias of off- methods could be preferred over the added variance of on-policy methods used here. Another approach may be to use partially observable RL that can learn a better value function model given changing RNN-based reward function. Training the distance metric could benefit from additional regularization, such as constraining the <l-divergence between updates to reduce variance. Learning& sequence- based policy as well, given that the rewards are now not dependent on single state observation is another area for future research that could improve performance.
We have compared our method to GAIfO, but we found GAIFO has limited temporal consistency. GAIfO led to learning jerky and overactive policies. The use of a recurrent discriminator for GAIfÒ may mitigate some of these issues and is left for future work. It is challenging to produce results better than the carefully manually crafted reward functions used by the RL simulation environments that include motion phase information in the observations (Peng et al. 201 Sa 2017). However, have shown that our method can compute distances in space and time and has faster initial learning. A combination of starting with our method and following with manually crafted reward function, if true state information is available, could potentially lead to faster learning of high-quality policies. Still, as environments become increasingly more realistic and grow in complexity, we will need more robust methods to describe the desired behaviour we want from the agent. One might expect that the distance metric should be trained early and fast so that quickly understands the difference between good and bad demonstration. However, we have found that in this setting, learning too quickly can confuse the agent, as rewards can change, which can cause the agent to diverge off toward an unrecoverable policy space. In this setting, slower is better, as the distance metric may not yet be accurate. However, may be locally or relatively reasonable, which is enough to learn a good policy. As learning continues, these two optimizations can converge together.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-firss International Conference on Machine Learning, ICML '04, pp. 1-, New York, NY, USA, 2004. ACM. ISBN --88133838-5 doi: 10.1145/1015330.001522 10154330 URL
http:///o...c..org//ooo 00100//00001010113030003.00 5330.1 533011011310
Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Roboties and Autonomous Systems, 57(5):469 483, 2009. ISSN 0921-8890. doi: http://to...rg/10..116....ooo htt.://www...............oo/....
10.024. URL
Sarah-Jayne Blakemore and Jean Decety. From the perception of action to the understanding of intention. Nature reviews neuroscience, 288):561-567, 2001.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub-
optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstaal imitation learning via automatically-rankedd demonstrations. In Conference on Robot Learning, pp. 330-359, 2020.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining based and model-free updates for trajectory- reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- -aaning-Volluee 70, pp. 703-711, 2017.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pp. 539-546. IEEE, 2005.
D. Dwibedi, J. Tompson, C. Lynch, and P. Sermanet. Learning Actionable Representations from Visual Observations. ArXiv e-prints, August 2018.

---

Under review as a conference paper at ICLR 2021
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International Conference on Machine Learning, pp. 1755-1763, 2019.
Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, and Sergey Levine. Generalizing skills with semi- supervised reinforcement learning. CoRR, abs/ 2016. URL http:://a//xxi.or/////
abs/1612.00429
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. CoRR, (tss/1709.0490. 2017. URL /nttp://arxiv..rrg/ /abs/
1709.04905
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In 7. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27. pp. 2672-2680. Curran Associates, Inc., 2014. URL htte::/ppaeersnpsscccppeeeee 5423 -ew.-a2iiggeeeiieeeeee adversarial nets.pdf
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(CVPR (06), volume 2, pp. 1735-1742. IEEE, 2006.
K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9): 1904-1916, Sept 2015. ISSN 0162-8828. doi: 10.1109/TPAM..215 23898844
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2016. URL http: /pattr:/ppapers..t wwt.ss.iis.cccceeee pttpppppppppppppppppppppppp / gene ive avvaavvvraaaraall im taaittiionn inattaaiinneeee con-learninggpa
Curran Associates, Inc..
Nathan Kau, Aaron Schultz, Tarun Punnoose, Laura Lee, and Zac Manchester. Woofer and pupper: Low-cost open-source quadrupeds for research and education.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International Conference onLearning Representations (ICLR), 2014.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Guyon, V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3812-3822. http:////ppappe/ee1111n-eoooee Curran Associates, Inc., 2017. URL http:::/ppppeeeapppp.c/////////s
interpretable imitation
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2015. URL http::://aaxiirrrr////////303300
Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. CoRR, abs/1707 0077003777444. 2017.
URL http::://axxvvvorr///o abs 707. 07.03374
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-22) 2602, 2008.
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. CoRR, abs/1707.02201, 2017. URL /R//Eppp///aavvooorr//////////
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CoRR, hass/180..04744 2018. URL http://
/tw..///111000007703 Drxiv.org/abs/1000
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 2, 2000.
10

---

Under review as a conference paper at ICLR 2021
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. sero-shot visual imitation. CoRR, 2018.
Xue Bin Peng and Michiel van de Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? In Proceedings of the ACM SIGGRAPH / Eurographics Symposium Computer Animation, SCA 17, pp. 12:1-12:13, New York, NY, USA, 2017. ACM. ISBN 978- 108-1-4003-0011-11 -4, doi: 10. 10.11445/309956433099566, URL httt:tt::22 http::///d......oooo///o/o 5099564.3099588
Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph. 36(4):
41:1-41:13, July 2017. ISSN 0730-0301. doi: 10.1 10.114577072959.3776600
URL http:/7
Xue Bin Peng, Picter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-hased character skills. ACM Trans. Graph., 37
(4::43:1-114:14,.uubb 8a. ISSN 0730-0301. doi: 10.1145/3197517.2203111 URL
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv: 2018b.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang. Tsang-Wer Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 07 2020. doi: 10.15607/RSS. WWI.RRSS..00.XXT 2020.XV1.064,
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudk (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Laud- erdale, FL, USA, 11-13 Apr 201 la. PMLR. URL /ttpptpp::/pppceeinnn..mmmcceep rosslla.html
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 627-635, 201 1b.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. ArXiv e-prints, July 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastive networks: Self- supervised learning from multi-view observation. CoRR, abs/1704.0688s. 2017. URL http:
//aaxii.orgg/aaa abs/ abw. 1704.06888
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-persee per-soeroon visual imitation learning via decoupled hierarchical controller. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'lché-Buc tion Processing Systems 32. pp. 2597-2607. Curran Associates, Inc., 2019. URL iittalliinnaioonnn
Fox, and R. Garnett (eds.), Advances in Neural Informa-

---

Under review as a conference paper at ICLR 2021
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imita-
tion learning from observation alone. In JCML. pp. 9036-6045 2019. URL http://
ppoceeddinggmm..press//// stt.:/.tmmmb.mmmm
Faraz Torabi, Garreu Warnell, and Peter Stone. Behavioral Cloning from Observation. (July), 2018a.
URL htttp://arxiv.ortt http.o/////bb/0055011110
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint 1807.0/00700//5555 2018b.
Hado Van Hasselt. Reinforcement learning in continuous state and action spaces. Reinforcement Learning, pp. 207-251. Springer, 2012.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems 30, pp. 5320-5329. Curran Associates, Inc., 2017. URL htthttp::/ pttp:/ppppprs...cco... 2aper/7116-robus wtt.::aatt..ttt----eeeeeeeeeeeee
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'lché- Buc. E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems 32, pp. 239-249, Curran Associates, Inc., 2019. URL http://ppp//appprsssaosp/////e/p 1tt///8811--1mmmmm htta:////atpatttttttioooeeaaaaar Leariniigg-- from 3r2e:7/2......pp.mmm.aiiii------
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. CoRR, abs/1 802.01557, 2018. /tt//////p/////aaarrrrr//a//////
Zhuotun Zhu, Xinggang Wang, Song Bai, Cong Yao, and Xiang Bai. Deep learning representation using autoencoder for 3d shape retrieval. Neurocomputing, 204:41-50, 2016.
Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. Supervised representa- tion learning: Transfer learning with deep autoencoders. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI088, pp. 1433-1438, AAAI Press, 2008, ISBN 978-1-57735-368-3, URL
http:://l.a...orccctaaaa http:p///.aeeooo/aaooooooooaaaa.
12

---

Under review as a conference paper at ICLR 2021
7
APPENDIX
This section includes additional details related to VIRL.
7.1 IMITATION LEARNING
Imitation learning the process of training new policy to reproduce the behaviour of some expert policy. BC is a fundamental method for imitation learning. Given expert policy possibly represented as collection of trajectories < (so, (ST, (er,ar) > a new policy can be learned to match these trajectory using supervised learning.
max I
log ग (a: 0z)]
While this simple method can work well, it often suffers from distribution mismatch issues leading to compounding errors the learned policy deviates from the expert's behaviour during test time.
7.2 INVERSE REINFORCEMENT LEARNING
Similar to BC, Inverse Reinforcement Learning (inverse reinforcement learning (IRL) also learns to replicate some desired, potentially expert. behaviour. However, IRL uses the RL enviromment to learn reward function that learns to tell the difference between the agent's 's behaviour and the example data. Here we describe maximal entropy IRI. (Ziebart et al.) (008). Given an expert trajectory policy can be trained produce similar trajectories by discovering a distance metric between the expert trajectory and trajectories produced by the policy m
(so, do).... (3T,aT) ar) >
max max a)] H())) Exg [x a)]
eEC'
where c is some learned cost function and H (T) is a causal entropy term. g is the expert policy that is represented by collection of trajectories. IRL is searching for a cost function that is low for the expert TH and high for other policies. Then, a policy can be optimized by maximizing the reward function F2 -c(St, 0.).
7.3 PHASE-BASED IMITATION
If we consider a phase-based reward function r= R(s, a, 6) where é indexes the time in the demonstration and s and is the agent state and action. As the demonstration timing é, often controlled by the environment, and agent diverge, the agent receives less reward, even if it is visiting states that exist elsewhere in the demonstration. The issue determining if agent is displaying out-of-phase behaviour can be understood trying to find the á that would result in the highest reward ø' max R(s, a, Φ) and the distance ø' - <) is an indicator of how far away in time or form can be seen However, this naive description ignores the ordered property of demonstrations. What is needed metric that gives reward for behaviour that is in the proper order, independent of phase. This ordering motivates the creation of a recurrent distance metric that designed to understand the context between motions. For example, does this motion look like a walk, not, does this motion look precisely like that walk.
out-of-phase the agent is. This phase-independer
form of reward shaping.
To encourage the agent match any part of the expert behaviour, VIRL be understood as decomposing the distance into two distances and learning Dynamic timme warping, by adding type of temporal distance shown in green. To compute time-independent distance we can find the state in the expert sequence that is closest to the agent's 's current state arg min ier d(o of) and use it in the following distance measure
df do 0 f,t) =... + ,f + d(os, of) + d(o of41)
Using only a single state time-alimeee alignmennnent may lead to the agent fixating on matching a particular state in the expert demonstration. To avoid this alignment issue, the neighbouring states added to enforce sequential structure in the distance computation. This framework allows the agent to be rewarded for exhibiting behaviour that matches any part of the expert's demonstration. The better it
13

---

Under review as a conference paper at ICLR 2021
Figure 7: Rasterized frames of the imitation motions on humanoid3d walking (row 1), running (row 2), zombie (row 3) and jumpingroow 4). https://sites..oogg/ hitp:/sses...ooooeccooo//t//i///
Active Data Collection
Train RT.
.(6,)
1s, a, rJ)
Siamese
Env
xs)))
Sampled transitions (s, 4)
3. Update V(s)
Update a(s,)
Sampled
0. crestePosittveee))
<da, o²)
sequencce
3t createNogativetto)
To, ((55)
Update with (o, of, o")
Figure 8: The flow of control for the learning system.
learns match parts of the expert demonstration, the more reward it is given. The previous spatial distance will then help the agent learn to sync up the timing with the demonstration. Next, we describe how both distances can be learned together.
7.4 DATA
We are using the mocap data from the CMU Graphics Lab Motion Capture Database from 2002 http://mocaa.cc.cmuu. edu/). To be thorough we provide the process length. This data has been preprocessed to map the mocap markers to a human skeleton. Each recording contains the positions and orientations of the different joints of a human skeleton and can therefore directly be used to animate humanoid mesh. This is standard approach that has been widely used in prior literature like [Gleicher 1998, Rosales 2000, Lee 2002]. To be precise: at each mocap frame, the joints of humanoid mesh model are to the positions and orientations of their respective values in the recording. a full humanoid mesh is not available, it is possible to add capsule mesh primitives between each recorded joint. This 3D mesh model is then rendered camera that follows the center of mass of the mesh at fixed distance.
image through 3rd person
For the humanoid experiments, imitation data for 24 other tasks was used help condition the distance metric learning process. These include motion clips for running, backflips, frontflips, dancing, punching, kicking and jumping along with the desired motion. The improvement due to these additional unsupervised training data generation mechanisms are shown in Figure 5a
7.5 TRAINING DETAILS
The learning simulations are trained using graphics processing unit (GPU)s. The simulation is not only simulating the interaction physics of the world but also rendering the simulation scene to capture video observations. On average, takes 3 days to execute single training simulation. The process of rendering and copying the images from the GPU is one of the most expensive operations with VIRL..
14

---

Under review as a conference paper at ICLR 2021
eucode doode
Lem
Figure 9: We use Siamese autoencoding network structure that can provide a reward signal every step reinforcement learning algorithm. For the Humanoid experiments here. the convolutional portion of network includes 2 convolution layers of filters with size stride 2 x 2. The features are then flattened and followed by two dense layers of 256 and 64 units. The majority of the network uses ReLU activations except for the last layer that uses sigmoid activation. Dropout is used between convolutional layers. The RNN-based model uses LSTM layer with 128 hidden units, followed by dense layer of 64 units. The decoder model has the same structure in reverse with deconvolution in place of convolutional layers.
and and stride 2 16 filters size
We collect 2048 samples between training rounds. The batch size for TRPO is 2048. The term is 0.5.
The simulation environment includes several different tasks that are represented by collection of motion capture clips to imitate. These tasks come from the tasks created in DeepMimic (Peng et al.) We include all humanoid tasks this dataset.
2018a).
In Algorithm|] I we include outline of the algorithm used for the method and diagram in Figure g The simulation environment produces three types observations, 81+11 the agent's 's proprioceptive pose, 6141 1 the image observation of the agent and m the image-based observation of the expert demonstration. The images are grayscale 64 x 64.
7.6 DISTANCE FUNCTION TRAINING
In Figure T0a the learning curve for the Siamese RNN is shown during pretraining phase. We see the overfitting portion the occurs during RL training. This overfitting can lead to poor reward prediction during the early phase of training. In Figure [Ob] we show the training curve for the recurrent Siamese network after starting training during RL. After an initial distribution adaptation, the model learns smoothly, considering that the training data used continually changing the RL agent explores.
$.700
3.00
Erfor
3.85
100
116
115
123
Iteration x160.0
Iteration x160.0
(a) Siamese loss during pretraining
(b) Siamese loss after pretraining
Figure 10: Training losses for the Siamese distance metric.
can be challenging train a sequenced based distance function. One particular challenge is training the distance function to be accurate across the space of possible states. We found good strategy was to focus on earlier parts of the episode. When the model not accurate on states earlier the episode, it may never learn how get into good states later, even if the distance function understands those better. Therefore, when constructing batches to train the RNN on, we give a higher probability of starting carlier in episodes. We also give higher probability of shorter sequences as function of the average episode length. As the agent gets better average episodes length increase, so to will the randomly selected sequence windows.
15

---

Under review as a conference paper at ICLR 2021
7.7 DISTANCE FUNCTION USE
We find it helpful to normalize the distance metric outputs using = exp(r22 ww)) wa) where wa - 5.0 scales the filtering width. This normalization is common method convert distance-based rewards to be positive, which makes it easier to handle episodes that terminate early (Peng et al. 2018 Peng & van de Panne 2017). Early in training. the distance metric often produces large, noisy values. The RL method regularly tracks reward scaling statistics; the initial high variance data reduces the significance of better distance metric values produced later on by scaling them to small numbers. The improvement of using this normalized reward is shown in Figure Tla In Figure 11b we compare to few baseline methods. The manual version uses a carefully engineered reward function from (Peng
al. 2017).
Omalsed Aetrossea hoeeetteen exc
Celee do Soatt, Thaliig lodeses 21ffc
(a) Reward smoothing
(b) Comparisons with other reward methods humanoid2d
Figure 11: Ablation analysis of VIRL. We find that training RL policies is sensitive to the size and distribution of rewards. We compare VIRL to number of other simple basclines.
7.8 SEQUENCE ENCODING
Using the learned sequence encoder, we processed collection of different motions create a t- distributed Stochastic Neighbor Embedding (t-SNE) embedding of the encodings (Maaten & Hinton 2008). In Figure 12b we plot motions both generated from the learned policy π and the expert trajectories gg. Overlaps in specific areas of the space for similar classes across learned and expert TE data indicate a well-formed distance metric that does not separate expert and agent examples. There is also separation between motion classes in the data, and the cyclic nature of the walking cycle is visible. We also show how the choice in policy variance affects the RL jump task learning process in Figure T2a
08
(a) Variance Comparison, Jump (b) t-SNE embedding manoid3d)
(hu-
Figure 12: Variance and (-SNE Visualization
7.9 POSITIVE AND NEGATIVE EXAMPLES
We use two methods to generate positive and negative examples. The first method is similar to TCN, where we can assume that sequences that overlap more in time are more similar. For each episode, two sequences generated, one for the agent and one for the imitation motion. Here list the methods used to alter sequences for positive pairs.
1. Adding Gaussian noise to each state in the sequence (mean 0 and variance 0.02)
2. Out of sync versions where the first state is removed from the first and the last ones from the second sequence
3. Duplicating the first state in either sequence
4. Duplicating the last state in either sequence
We alter sequences for negative pairs by
16

---

Under review as a conference paper at ICLR 2021
1. Reversing the ordering of the second sequence in the pair.
2. Randomly picking state out of the second sequence and replicating it to be as long as the first sequence.
3. Randomly shuffling one sequence.
4. Randomly shuffling both sequences.
The second method we use to create positive and negative examples is by including data for additional classes of motion. These classes denote different task types. For the humanoid3d environment, we generate data for walking-dynamic-spe
running, back-flipping and front-flipping. Pairs from the same tasks are labelled as positive, and pairs from different classes are negative.
7.10 RL ALGORITHM ANALYSIS
It is not clear which RL algorithm may work best for this type of imitation problem. A num- ber of RL algorithms were evaluated on the humanoid2d environment Figure 13a Surprisingly, TRPO (Schulman al. (2035) did not work well in this framework, considering it has controlled policy gradient step, we thought it would reduce the overall variance. We found that Deep Deter- ministic Policy Gradient (DDPG) (Lillicrap et 2015) worked rather well. This result could be related to having a changing reward function, in that if the changing rewards are considered off-policy data, can be easier to learn. This can be seen Figure 13b where DDPG is best estimating the future discounted rewards in the environment. We also tried Continuous Actor-Critic Learning- Automaton (CACLA) [Van Hasselt 2012) and Proximal Policy Optimization (PPO) (Schulman et al. 2017); we found that PPO did not work particularly well on this task; this could also be related to added variance.
AdooN miknnn
araiiddiaooattio aegpppaeesion 21000
(a) Average Reward
Bellman Error
(c) sequence-basel comparison
DDPG
Figure 13: RL algorithm comparison on humanoid2d environment.
17


Reviews:
[{"review_id": "mhBOpqQtBHJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\nThe authors propose an extension of recent imitation from observation techniques called VIRL that explicitly incorporates Siamese network and LSTM encodings in order to attempt to better overcome some of the challenges of imitation learning from visual observations. They evaluate their algorithm in several simulation domains and find reasonable results.\n\n\nSTRENGTHS:\n\t(S1) The paper considers an important and timely problem in trying to improve upon the state-of-the-art in imitation learning from visual observations.\n\t(S2) The paper proposes an interesting new technique that, in some ways, seems to perform very well in experiments.\n\n\nWEAKNESSES:\n\t(W1) The motivation for the technique in the sense of being for single-shot learning is not clear. Learning a distance metric between behaviors would seem to be applicable for learning from _many_ demonstrations -- in what way is the proposed approach specialized for single-shot learning?\n\t(W2) The presentation of the experimental results is, at times, incomplete and confusing. For example:\n\t\t(a) The ablation study shows lines for random/max sequence lengths, but (as best I can tell), the methods that generated these curves are never discussed.\n\t\t(b) The ablation study presented in Figures 5a and 5b generates more questions that it answers. For example, how is it that \"(EESP) + pretrain\" gets the lowest loss in 5b but does very poorly in 5a?\n\t\t(c) The experimental setup is unclear to the point where its doubtful that a reader could reproduce the results. In particular, at the beginning of Section 5, it's not clear exactly how the demonstrations are generated. The authors state that the simulation environments are \"new and have all been updated to take motion capture data and produce view video data that can be used for training RL agents,\" but do not describe in any detail how the mocap data was processed to actually generate the demonstration video data.\n\n\nRECOMMENDATION STATEMENT:\nWhile the paper considers and important problem and may indeed describe a very promising new approach, the current experiments section is in need of a major rework before the story is clear.\n\n\nQUESTIONS FOR AUTHORS:\n\t(Q1) How, exactly, were the demonstrations from mocap data produced?\n\n\nMINOR COMMENTS:\n\t(MC1) The authors need to state in the abstract that their experiments are done on _simulated_ agents.\n\t(MC2) The authors state in the introduction that \"the fundamental problem of imitation learning is how to align a demonstration in space and time with the agent's own space.\" Respectfully, don't think this is true. It is perhaps \"a\" fundamental problem, but certainly not \"the\" fundamental problem. Even with aligned state spaces, the general problem imitation learning poses several other challenges such as how to learn robust policies, how to incorporate multiple demonstrations if they are available, etc. The authors should revise the wording of this statement.\n\t(MC3) The caption of Figure 4 does not address (d)-(g).\n\t(MC4) The legend in Figure 4a says \"GAILfO\" but the text says the comparison algorithm is \"GAIfO.\"\n\t(MC5) The text in Figures 4 and 5 is so small that it is unreadable. The figures need to be reformatted to address this.\n\t(MC6) The authors should reference and discuss how their work relates to / differs from a very similar recent paper [https://arxiv.org/pdf/1909.13392.pdf]", "extra_scores": {}, "extra_text": {}}, {"review_id": "mHBfhuse353", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: This paper studies the problem of visual imitation learning: given a video of an expert demonstration, take actions to reproduce that same behavior. The proposed method learns a distance metric on videos and uses that distance metric as a reward function for RL. Experiments show that this method does recover reasonable behaviors across a range of simulated robotic tasks.  Compared with prior methods, the main contribution of this work is that the distance metric is parametrized and trained as a siamese network.\n\n**Novelty and Significance**: While the exact method seems novel, it is very similar to a number of prior methods, most notably GAIL. At a high-level the main difference is that this paper uses a siamese network to parametrize the discriminator, and employs a few types of data augmentation. If the experiments had shown that this architectural choice made a significant improvement in performance, and was useful inside a range of imitation learning frameworks (e.g., GAIL, AIRL, and Value Duce), then I think it'd represent a significant contribution. As is, the paper has not convincingly shown that this architectural choice is critical for significantly improved performance.\n\n\n**Experiments:**\n* I found it challenging to assess the experimental results without any quantitative comparisons with baselines (besides TCN). I'd highly recommend comparing against recent imitation learning methods. For example, some reasonable baselines would be\n  * GAIL or InfoGAIL\n  * Zero-shot visual imitation\n  * Imitating latent policies from observation\n  * Generative adversarial imitation from observation (Though this method is discussed, it's not compared against in the figures.)\n  * Learning robust rewards with adversarial inverse reinforcement learning\n  * Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning\n  * Imitation learning via off-policy distribution matching\n* The videos of the humanoid imitation behavior on the accompanying website actually look quite poor. The videos for other domains seemed to have been rendered incorrectly.\n* I found it somewhat irksome that the method is entitled \"visual imitation\" but the learned policy doesn't take visual inputs (\"Note that we experimented with using visual features as the state input for the policy as well; however, this resulted in poor policy quality.\")\n\n\n\n**Clarity:**\n* The introduction makes imitation learning sound like a new problem. I'd recommend clarifying the relationship with prior work earlier in the introduction.\n* I found Fig 1 hard to follow, largely because it's unclear what the method is supposed to produce. One idea is to explicitly say something like: \"We aim to learn a distance function (left) and then use that distance function as a reward function for RL (right). The distance function is learned by ...\"\n* I found Eq 4 hard to parse. It might be helpful to label each term (using \\underbrace{}) with its semantic meaning.\n* Reward Calculation: One thing that's unclear here is which trajectories are used for computing the reward function. I'm guessing that the agent's trajectory is compared to an expert trajectory, but I don't think this is ever stated explicitly. If so, it's unclear how to compute the reward function when multiple expert trajectories are given.\n\n\nOverall, I give this paper a score of 4/10, primarily because of the lack of discussion and comparison with prior work. I would consider increasing my score if the paper compared against recent visual imitation learning methods (see above). The proposed method also seems quite similar to GAIL; if possible, it'd be great to formally explore the connections between these two methods. Finally, I'd recommend focusing the related work section to only discuss the most related works (see list above), but to discuss the exact differences between these prior methods and the proposed method in more depth.\n\n**Questions for discussion**:\n* Precisely, what are the differences between this method and GAIL? (I want to make sure I didn't miss an important difference)\n* For the experiments in \"3D Robot Video Imitation\", is the expert demonstration provided as an RGB video, a mocap trace, or something else?\n* Where is the comparison with GAIfO shown? (\"We have compared our method to GAIfO\")\n* \"that takes into account demonstration ordering\" -- In a Markovian environment with a Markovian expert, shouldn't comparisons based on state-action pairs be sufficient? Is the assumption that the expert's motion is not Markovian?\n* \"Mutual information loss (”Siamese Network triplet loss”)\" -- Eq 3 doesn't look like a mutual information objective. Can you explain precisely how maximizing Eq 3 leads to maximizing mutual information?\n\n\n**Minor comments:**\n* Abstract: No need to define \"GAIfO\" as an acronym in the abstract. Instead, just use \"GAIL without access to actions.\"\n* \"In nature ... their movements\" -- Please add citations.\n* \"formulating ... is challenging\" -- How is this different from the large body of prior work on imitation/apprenticeship learning?\n* \"The fundamental problem ...\" -- This is only true for *trajectory*-based imitation learning, not *transition*-based imitation learning (e.g., BC, GAIL, AIRL).\n* \"re-use\" -> \"reuse\"\n* \"Additionally, While\" -> \"Additionally, while\"\n* \"Including, ...\" This sentence is missing a subject.\n* \"Markov Decision Processes\" -> \"Markov decision processes\"\n* When defining a trajectory, use \\langle and \\rangle.\n* \"it often suffers from distribution mismatch\" -- Cite DAGGER, or something similar.\n* \"behaviour Ng\" -- Use \\citep{} for references that are not used as nouns.\n* Eq 2: Can you define the distribution over $h$? (I assume it will be p(h | x)).\n* Eq 3: Perhaps cite prior methods that use this contrastive margin loss (e.g., FaceNet)\n* Define VIRL = \"Visual Imitation with RL\" in S3. It took me awhile to figure out what VIRL was when it was first mentioned in S4.\n* \"is an active research area\" -- I agree, but using citations from '04 and '09 doesn't make this area sound particularly \"recent.\"\n* \"the goal is to\" -- The goal of what? Of all \"good distance functions\"?\n* \"state-based metrics ... image based inputs\" -- Aren't image-based metrics a special case of state-based metrics? For example, the [Ho & Ermon] citation for state-based metrics can be applied to images.\n* \"Additional works..., none of these ...\" -> \"While additional works ..., none of these ...\"\n\n----------------------------\n**Update after author response**: Thanks to the authors for answering my questions and for incorporating feedback into the paper. Through discussion, I think we got to the crux of the method: distance functions seem to work better than classifiers for imitation learning. I think this is a really neat observation, and potentially quite important; the experiments definitely support this hypothesis. That said, I don't think the paper goes far enough in exploring this hypothesis, either experimentally or theoretically. There are a number of confounding variables, such as data, loss function, and architecture, which each will need to be accounted for. I therefore stand by my previous vote (4) to reject the paper. With more thorough experiments and ablations, I think this will make a fantastic submission to a future conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y5WFAROzcx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The approach described in the paper uses a relatively complicated architecture, with multiple losses and a very processed training set, but it achieves strong results when compared to two of the most similar published methods - TCNs (Time-Contrastive Networks) and GAIfO (Generative adversarial imitation from observations). A set of ablations is presented which is not exhaustive, but which is adequate to understand the comparative importance of the different parts of the approach. Overall, I think that there are some rough edges to the paper which could be improved, but the contribution of the paper is enough to warrant publication.\n\nOne problem is that the method is poorly explained in some places. Some of the terms need to be explained earlier in the paper - Siamese Network or Siamese Loss is never properly explained and TCN is used in the abstract without reference or definition. At the beginning of section 3, it is stated that the 'Siamese network triplet loss' is used in the proposed method, but Eq. 3 shows a contrastive loss with margin, not a triplet loss. Adding to the problems with clarity, there are a number of obvious typos which are distracting to the reader and make it clear that this is not a fully polished submission:\n\ntypos - oultine -> outline; Advisarial -> Adversarial; primariliy -> primarily; subsquent -> subsequent; independanlty -> independently; kinematiccally -> kinematically \nfragments: 'Including recent Sim2Real quadreped robots and a huanoid with 38 DoF, which is a\nparticularly challenging problem domain.'; 'Where states and actions are discrete.'\n\nThe paper claims that the method can train an agent to do imitations from noisy visual data from single demonstrations, but this is not clearly shown by the experiments. The domains where the agent performs well (judging by the videos available on the linked website) are simple and cyclic and have little or no variation. It is unclear whether the agent would be able to 1-shot imitate a novel demonstration which was not within the training set already. On the more challenging domains, the videos show that the agent imitates the expert for a very short timespan before diverging. However, visual imitation is very difficult and the proposed method does perform better than the other baselines.\n\nThe paper would have more impact if the authors had shown whether the trained approach could be used for better performance or learning speed on new tasks where expert data was not available, perhaps by training a new policy on top of the LSTM, or distilling to a new agent and then finetuning.\n\nOverall, the paper is exciting because it shows the value of the complex architecture for solving a challenging problem. The experiments are convincing and show that this is a promising approach which may be useful in a real-world domain.\n\nPros: \n- The method is well-designed and is a natural extension on existing work (TCNs, Siamese nets, GAIfO). There is enough novelty for the work to be published.\n- The results show that the approach works well on a number of different continuous control environments\n- The comparisons to other published methods, the baselines and ablations, are well chosen.\n- The supplementary contains additional details on training, and additional analysis, which is valuable\n\nCons:\n- The text is poorly written in places - Acronyms and terms need to be explained, and spelling and grammar need to be proofed. However, this is an easy fix that should not prevent the paper from being published.\n- The authors have not adequately explained whether the model is actually capable of 1-shot imitation. It is not clear from the domains whether the agent has simply memorized the full data distribution.\n- The model is limited if there is no way to reuse the model without demonstrations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CojTLvd8ElK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper summary\n\nThis paper proposes using a recurrent siamese network to learn distance functions between observed behaviours and agent behaviours, and the use of this distance for reinforcement learning. Experiments on a set of simulated walking tasks show that this approach works reasonably well.\n\nUnfortunately, I am recommending rejection, as I do not think the contributions of this work are significant enough to warrant publication at ICLR. Siamese recurrent networks are already well established techniques for distance learning ([Chopra et al. 05](http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf), [Pei et al. 16](https://arxiv.org/pdf/1603.04713.pdf)), so it seems the primary contribution here is a method of artificially adding noise to help distinguish expert and demonstrated behaviours in training, an idea already explored to a large extent by TCNs and D-REX. \n\n### Pros\n- The proposed approach performs well when compared against GAILfO and learns better distance functions than a TCN.\n- The results look good, and I really like the zombie walk.\n\n### Cons\n\n- The ideas in this work are very close to T-Rex [Brown et al. 19a](https://arxiv.org/abs/1904.06387) and D-Rex [Brown et al. 19b](https://arxiv.org/pdf/1907.03976.pdf) which uses a pairwise trajectory ranking formulation, which essentially learns the distance between trajectories, much like a siamese network. Brown et al. 19b also explore artificial trajectory ranking by adding noise to trajectories. \n- The proposed approach relies on a number of heuristics and connected neural components and seems to perform well in walking tasks, but makes no real theoretical contribution.\n\n### General comments and queries\n\n- The paper seems to have been quite rushed and in need of some refinement when it comes to presentation. Specifically, please use a spell checker - there are numerous typos where letters are missing from words alongside spelling errors (eg. Advasarial,  quadreped, etc...), and check figure sizes to make sure legends are legible. The accompanying website is also full of broken links and unfilled template information.\n- Some sweeping statements need toning down. Eg. in the introduction it is claimed that *The fundamental problem of imitation learning is how to align a demonstration in space and time with the agent’s own state.* which is arguable, I've always thought the fundamental problem in imitation learning is determining which part of a demonstration is essential and which is ancillary.\n- I'd recommend a discussion and comparison with [D-REX Brown et al. 19b](https://arxiv.org/pdf/1907.03976.pdf)\n- Equation 4 - how were the weights for these heuristic losses chosen?\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of learning to imitate behaviors from visual demonstrations, without access to expert actions. Consistent with recent approaches, the proposed method uses a neural network to measure the similarity between visual demonstrations and the agent's behavior, and employs this metric as a reward in RL. The primary contribution is the use of a recurrent siamese network that is trained to measure the distance between motions, as a means of better dealing with the challenges of imitation learning from a small number of (as few as one) noisy visual demonstrations. Experiments on a variety of simulated domains show that the proposed approach achieves reasonable results.

The paper was reviewed by four knowledgeable referees, who read the author responses and engaged in extensive discussion. The reviewers agree that learning to imitate behaviors from a small amount of noisy demonstrations is a challenging and important problem that is of significant interest. The proposed method nicely extends existing approaches to visual imitation learning, and the results reveal that the method performs well in a variety of continuous control domains. The reviewers raise several concerns regarding the clarity of the technical presentation and the sufficiency of the experimental evaluation. The authors have made a significant effort to address these concerns in their responses and updates to the paper, which the reviewers very much appreciate. However, some of the reviewers' primary concerns regarding clarity and the thoroughness of the experimental evaluation remain. This work has the potential to make a really nice contribution and the authors are encouraged to take this feedback into account for any future version of the manuscript.
Decision: Reject