Visual Imitation with Reinforcement Learning using Recurrent Siamese Networks
Authors: Glen Berseth
Christopher Pal
Venue:  (2020)
Keywords: imitation learning
reinforcement learning
imitation from video

Abstract: It would be desirable for a reinforcement learning (RL) based agent to learn behaviour by merely watching a demonstration.  However, defining rewards that facilitate this goal within the RL paradigm remains a challenge. Here we address this problem with Siamese networks, trained to compute distances between observed behaviours and the agent’s behaviours. Given a desired motion such Siamese networks can be used to provide a reward signal to an RL agent via the distance between the desired motion and the agent’s motion. We experiment with an RNN-based comparator model that can compute distances in space and time between motion clips while training an RL policy to minimize this distance.  Through experimentation, we have had also found that the inclusion of multi-task data and an additional image encoding loss helps enforce the temporal consistency.  These two components appear to balance reward for matching a specific instance of a behaviour versus that behaviour in general. Furthermore, we focus here on a particularly challenging form of this problem where only a single demonstration is provided for a given task – the one-shot learning setting. We demonstrate our approach on humanoid agents in both 2D with 10 degrees of freedom (DoF) and 3D with 38 DoF.

Under review as a conference paper at ICLR 2020
VISUAL IMITATION WITH REINFORCEMENT LEARN- ING USING RECURRENT SIAMESE NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
It would be desirable for a reinforcement learning (RL) based agent to learn be- haviour by merely watching a demonstration. However, defining rewards that fa- cilitate this goal within the RL paradigm remains a challenge. Here we address this problem with Siamese networks, trained to compute distances between observed behaviours and the agent's behaviours. Given a desired motion such Siamese net- works can be used to provide a reward signal to an RL agent via the distance between the desired motion and the agent's 's motion. We experiment with an RNN- based comparator model that can compute distances in space and time between motion clips while training RL policy to minimize this distance. Through ex- perimentation, we have had also found that the inclusion of multi-task data and additional image encoding loss helps enforce the temporal consistency. These two components appear to balance reward for matching a specific instance of a behaviour versus that behaviour in general. Furthermore, ticularly challenging form of this problem where only a single demonstration is provided for given task the one-shot learning setting. We demonstrate our ap- proach on humanoid agents in both 2D with 10 degrees of freedom (DoF) and 3D with 38 DoF.
focus here on a par-
INTRODUCTION
Imitation learning and Reinforcement Learning (RL) often intersect when the goal is to imitate with incomplete information, for example, when imitating from motion capture data (mocap) or video. In this case, the agent needs search for actions that will result in observations similar to the expert. However, formulating a metric that will provide reasonable distance between the agent and the expert is difficuut. Robots and people plan using types of internal and abstract pose representations that can have reasonable distances; however, typically when animals observe others performing tasks, only visual information is available. Using distances in pose-space is ill-suited for imitation as changing some features can result in drastically different visual appearance. In order to understand how to perform tasks from visual observation a mapping/transformatin is used which allows for the minimization of distance in appearance. Even with a method transform observations to similar pose space, each person has different capabilities. Because of this, people are motivated to learn transformations in space and time where they can reproduce the behaviour the best of their own ability. How can we learn a representation similar to this latent space?
An essential detail of imitating demonstrations is their sequential and causal nature. There is both an ordering and speed in which demonstration is performed. Most methods require the agent to learn imitate the temporal and spatial structure at the same time creating a potentially narrow solution space. When the agent becomes desynchronized with the demonstration, the agent will receive low reward. Consider the case when a robot has learned to stand when its goal to walk. Standing is spatially close to the demonstration and actions that help the robot stand, opposed to falling. should be encouraged. How can such latent goals be encouraged?
If we consider a phase-based reward function R(s,, Φ) where ó indexes the time in the demon- stration and and a is the agent state and action. As the demonstration timing d. often controlled by the environment, and agent diverge, the agent receives less reward, that exist elsewhere in the demonstration. The issue of determining an agent is displaying out- of of-phase behaviour understood trying to find the that would result in the highest reward
if it is visiting states

---

Under review as a conference paper at ICLR 2020
o' = max; R(s, a, o) and the distance d' - o is an indicator of how far away in time or out-of- form can be seen as a form of reward shaping. However,
phase the agent is. This this naive description ignores the ordered property of demonstrations. What is needed is metric that gives reward for behaviour that is in the proper order, independent of phase. This ordering mo- tivates the creation of recurrent distance metric that is designed to understand the context between two motions. For example, does this motion look like a walk, not, does this motion look precisely like that walk.
Our proposed Visual Imitation with Reinforcement Learning (VIRL) method uses Recur- rent Siamese Networks (RSNs) and has similarities both Inverse Reinforcement Learning (IRL) (Abbeel & Ng 2004) and Generative Advisarial Imitation Learning (GAIL) (Ho & Ermon 2016). The process of learning a cost function that understands the space of policies to find an optimal policy given a demonstration is fundamentally IRI. While using positive examples from the expert and negative examples from the policy similar to the method GAIL, uses to train a discriminator to recognize in distribution examples. In this work, we build upon these techniques by constructing a method that can learn policies using noisy visual data without action information. Considering the problem's 's data sparsity, include data from other tasks to learn a more robust distance function in the space of visual sequence. We also construct a cost function that takes into account the demonstration ordering as well pose using recurrent Siamese network. Our con- tribution consists of proposing and exploring these forms of recurrent Siamese networks as a way to address critical problem in defining reward structure for imitation learning from the video for deep RL agents and accomplishing this on simulated humanoid robots for the challenging single shot learning setting.
RELATED WORK
Learning From Demonstration Searching for good distance functions is an active research area (Abbeel& Ng. 2004; Argall et al.] (009). Given some vector of features, the goal is to find an optimal transformation of these features, such in this transformed space, there exists strong contextual meaning. Previous work has explored the area of state-based distance functions, but most rely on pose based metrics (Ho & Ermon 2016 M et al. (017)) that come from expert. While there is other work using distance functions, including for example [Sermanet et al. (2017); [Finn] et al. (2017); Liu et al.) (2017): Dwibedi al. (2018), few use image based inputs and none con- sider the importance of learning a distance function in time as well as space. In this work, we train recurrent Siamese networks (Chopra al. 2005) to learn distances between videos.
Partially Observable Imitation Without Actions For Learning from Demonstration (LfD) prob- lems the goal is to replicate the behaviour of expert Te behaviour. Unlike the typical setting for humans learning to imitate LfD often assumes the availability of expert action and observation data. Instead, in this work, we focus on the case where only noisy actionless observations of the expert are available. Recent work uses Behavioural Cloning (BC) to learn an inverse dynamics model to estimate the actions used via maximum likelihooo estimation (Torabi et al. 2018). Still, BC often needs many expert examples and tends to suffer from state distribution mismatch issues between the expert policy and student (Ross et al. 2011). Work in (Merel et al. 2017) proposes a system based on GAIL that can learn a policy from partial observation of the demonstration. In this work, the discriminator's state input is a customized version of the expert's state and does not take into account the demonstration's 's sequential nature. The work in (Wang et al., 2017) provides a more robust GAIL framework along with a new model to encode motions for few-shot imitation.
This model uses an Recurrent Neural Network (RNN) to encode demonstration but uses expert state and action observations. In our work, the agent is limited to only partial visual observation as a demonstration. Additional works learn implicit models of distance (Yu al., 2018 Pathak et al., 2018 Finn et al., (017): [Sermanet et al. (017), none of these explicitly learn sequential model considering the demonstration timing. An additional version of GAIL, infoGAIL (Li et al. 2017). included pixel based inputs, Goals can be specified using the latent space from Variational Auto Encoder (VAE) (Nair et al. 2018). J. Our work extends this VAE loss using sequence data to train a more temporally consistent latent representation. Recent work (Peng et al. (018b) has a 2D control example of learning from video data. We show results on more complex 3D tasks and additionally
model distance in time. In contrast, here we train recurrent siamese model that can be used to
2

---

Under review as a conference paper at ICLR 2020
able curriculum learning and allow for computing distances when the agent and demonstration are out of sync.
3 PRELIMINARIES
In this section, we outline the general RI. framework and specific formulations for RL that we rely upon when developing our method in Section
Reinforcement Learning Using the RL framework formulated with a Markov Dynamic Process (MDP): at every time step I, the world (including the agent) exists in state 8, E S, wherein the agent able to perform actions a, E A, sampled from a policy (() which results in a new state s1+1 E S and reward r, according to the transition probability function T(r,, S141 at) The policy is optimize to maximize the future discounted reward
T
(TT E Ero.....T
where 7' is the max time horizon, and is the discount factor, indicating the planning horizon length. Inverse reinforcement learning refers to the problem of extracting a reward function from observed optimal behavior Ng et al. (2000). In contrast, in our approach we learn distance that works across a collection of behaviours. Further, we do not assume the example data to be optimal. See Appendix 7.2 for further discussion of the connections of our work to inverse reinforcement learning.
GAIL VIRL is similar to the GAIL framework (Ho & Ermon 2016) which uses a Generative Advasarial Network (GAN) (Goodfellow al., 2014), where the discriminator is trained with posi- tive examples from the expert trajectories and negative examples from the policy. The generator is combination of the environment, policy and current state visitation probability induced by the policy Px (s).
min max F (log(D(s,, al0)))] + R.., [log(1 - Dss, alOa))]
0x 6e
In this framework the discriminator provides rewards for the RI. policy to optimize, as the probability of a state generated by the policy being in the distribution ri = [st,, a (0%). While this framework has been shown to work in practice, this dual optimization is often unstable. In the next section we will outline our method for learning a more stable distance based reward over sequences of images.
4 CONCEPTUAL WESTANCE-BASHR REINFORCEMENT LEARNING
Our approach is aimed at facilitating imitation learning within an underlying RL formulation over partially observed observations Unlike the situation in GAIL, we do not rely on having accces to state, 8 and action, a information - our idea is to minimize a function that determintes the distance between two sequences observations, o, one from the desired example behavior of, and another from the current agent behavior o". We can then define the reward used within an underlying RL framework in terms of distance function D, such that
re(0", -0) - o00000000)))
o0,
d(os,
(3)
t=
where in our setting here D(o 0 ,0, models distance between video clips from time / = 0 to i.
simple formulation of the approach above can be overly restrictive on sequence timing. While these distances serve RL rewards, they often provide insufficient signal for the policy to learn a good imitative behaviour, especially when the agent only has partial observations of the expert. We can see example of this in Figure Ta were starting is the agent (in red) begins to exhibit behaviour that is similar to the expert (in blue) yet the spatial distance indicates that this state is further away from the desired behaviour than at t4.
To encourge the agent to match any part of the expert behaviour we propose decomposing the dis- tance into two distances, by adding type of temporal distance shown in green. To compute time
3

---

Under review as a conference paper at ICLR 2020
independant distance can find the state in the expert sequence that is closest to the agent 's current state arg min iET d(og, of) and use in the following distance measure
d'(o", 0" i,t =... : 02 + d(o;. of)) d((
(4)
Using only single state time-alined may lead the agent fixating on mataching single state in the expert demonstration. To avoid this the neighbouring states given sequence timing readjustment are used in the distance computation. This framework allows the agent to be rewarded for exhibiting behaviour that matches any part of the experts demonstration. The better is learns to match parts of the expert demonstration the more reward it is given. The previous spatial distance will then help the agent learn to sync up its timing with the deomonstration. Next we describe how learn both of these distances.
Sanow
Updacnla) Upder fi!)
Sazpiid pagatoo
-aa an)
Updax
(a) State space
(b) System Diagram
Figure 1: A pair of sequences from the expert (blue) and agent (red). The spatial distance is shown via doted black lines and the temporal distance for d(o", ,o) is visulized in green. On the right the flow of control for the learning system shown.
Distance Metric Learning Many methods can be used to learn a distance function in state-space. Here Siamese network f o0)) with triplet loss time and task data (Chopra al., 2005). The triplet loss used to minimize the distance between two examples that are positive, very similar from the same class, and maximize the distance between pairs of examples that are known to be unrelated. For more details see supplementary document.
Sequence Imitation The distance metric is formulated recurrent style where the distance is computed from the current state and conditioned on all previous states d(o:/or-1:.... on). The loss function is a combination of distance Eq. and VAE-based representation learning objectives from Eq. 7 and Eq. g detailed in the suppiementary material. This combination of sequence- based losses assists in compressing the representation while ensuring intermediate representations informative. The loss function used to train the distance model on positive pair of sequences is:
1 LVIRL Op. =>oLsN 0) + A1
Σ
CsN(or, 01.
1=
1 Σ LvAR(Or)) Lv
A Ao(CAE(0)) LAE(O,)).
Where À = 00.100.1,0.10001) With a negative pair, the second sequence used in the VAE and AE losses would be the negative sequence.
The Siamese loss function remains the same in Eq. 9 but the overall learning process evolves to use an RNN-based deep networks. diagram of the full model is shown in Figure 2 This model uses time distributed Long Short-Term Memory (LSTM). A single convolutional network conv" is first used to transform images the demonstration o" to an encoding vector eç_ After the sequence of images is distributed through corv" there is encoded sequence ef. .,, >, this sequence is fed into the RNN Istm until a final encoding is produced hς- This same process is performed for copy of the RNN Istm" producing hf for the agent o². The loss is computed in a similar fashion to (Mueller & Thyagarajan] 2016) using the sequence outputs of images from the agent and another from the demonstration. The reward at each timestep is computed rı
4

---

Under review as a conference paper at ICLR 2020
||Istt" (cone, (s()) Istm" -Istm" (conua ||conv (sf) conva (s() | At the beginning of each episode, the RNN's internal state is reset. The policy and value function have 2 hidden layers with 512 and 256 units, respectively. The use of additional VAE-based image and Auto Encoder (AE))based sequence decoding losses improve the latent space conditioning and representation.
||ng - - -
mccde
deada
boma
Figure 2: Siamese network structure. The convolutional portion of the network includes convolution layers of filters with size 6 6 and stride 2 x 2. 16 filters of size 4 and stride 2 x 2. The features then flattened and followed by two dense layers activations except for the last layer that uses a sigmoid activation. Dropout used between convolutional layers. The RNN-based model uses LSTM layer with 128 hidden units, followed by dense layer of 64 units. The decoder model has the same structure in reverse with deconvolution in place of convolutional layers.
256 64 units. The majority of the network uses ReLU
Unsupervised Data labelling To construct positive and negative pairs for training we make use of time information in a similar fashion to (Sermanet et al. 2017). where observations at similar times in the same sequence are often cor- related and observations at different times will likely have little similarity. We compute pairs by altering one sequence and comparing this modi- fied version to its original. Positive pairs are cre- ated by adding noise to the sequence altering a few frames of the sequences. Negative pairs created by shuffling one sequence reversing it. More details are available in the supplemen- tary material. Imitation data for 24 other tasks are also used to help condition the distance met- ric learning process. These include motion clips for running, backflips, frontflips, dancing, punch- ing, kicking and jumping along with the desired motion. For details on how positive and negative pairs are created from this data, see the supple- mentary document.
Algorithm Learning Algorithm
Initialize model parameters 0, and 0a Create experience memory D + () while not done do
for i (0................. N) do
Ti f- 0
of) + env.reset() (sc, of
for t E {0, {O..... T) do
0 t T[1,s,,
{s1+1, 0f of t env.step(a.)
It +- -(00 (04)
Ti, {st, of, Tt,
{sr, of, of) t {8e+1 1441
end for end for
D DU ..... To........ .N} Update d(.) parameters 0g using D Update policy Oz using (70...., TN end while
Importantly the RL environment generates two different state representations for the agent. The first state 81+-1 is the internal robot pose. The second state 0r+ 1 is the agent's rendered view, shown Figure The rendered view is used with the distance metric to compute the similarity between the agent and the demonstration. We attempted using the visual features as the state input for the policy as well; this resulted in poor policy quality. Details the algorithm used to train the distance metric and policy outlined in the supplementary document Algorithm [
ANALYSIS AND RESULTS
The simulation environment used in the experiments is similar to the DeepMind Control Suite (Tassa et al. 2018). In this simulated robotics environment, the agent is learning to imitate given reference motion. The agent's 's goal is to learn a policy to actuate Proportional Derivative (PD) controllers at 30 fps to mimic the desired motion. The simulation environment provides hard-coded reward function based on the robot's pose that is used to evaluate the policy quality. The demonstration M the agent is leamning to imitate is generated from clip of mocap data. The mocap data is used to
5

---

Under review as a conference paper at ICLR 2020
animate a second robot in the simulation. Frames from the simulation are captured and used as video input to train the distance metric. The images captured from the simulation are converted to grey- scale with 64 64 pixels. We train the policy on pose data, link distances and velocities relative to the robot's Centre of Mass (COM). This simulation environment is new and has been created to take motion capture data and produce multi-view video data that can be used for training RI. agents or generating data for computer vision tasks. The environment includes challenging and dynamic tasks for humanoid robots. Some example tasks are imitating running, jumping, and walking, shown in Figure 3 and humanoid2d detailed in the supplementary material.
3D Humanoid Robot Imitation In these simulated robotics environments the agent is learning to imitate a given reference motion of a walk, run, jump or zombie motion. A single motion demon- stration is provided by the simulation environment as a cyclic motion. During learning, we include additional data from all other tasks for the walking task this would be: malking-dynamic-spee rumning, jogging, frontflips, backflips, dancing, jumping, punching and kicking) that are only used to train the distance metric. We also include data from a modified version of the tasks that has a randomly generated speed modifier E [0.5, walking-dynamic-sped. that warps the demon- stration timing. This additional data is used to provide a richer understanding of distances in space and time to the distance metric. The method is capable of learning policies that produce similar behaviour to the expert across a diverse set of tasks. We show example trajectories from the learned policies in Figure 3 and in the supplemental Video It takes 5 days to train each policy in these results on 16 core machine with an Nvidia GTX1080 GPU.
Algorithm Analysis and Comparison To evaluate the learning capabilities and improvements of VIRL we compare against two other methods that learn a distance function in state space, GAIL and using a VAE to train an encoding and compute distances between those encodings, similar to (Nair et al. 2018), using the same method as the Siamese network in Figure 4al We find that the VAE alone does not appear to capture the critical distances between states, possibly due to the decoding transformation complexity. Similarly, the GAIL. baseline produces very jerky motion or stands still, both of which are contained in the imitation distribution. Our method that considers the temporal structure of the data learns faster and produces higher value policies.
Additionally, we create a multi-modal version of VIRL. Here we replace the bottom conv net with a dense network and learn a distance metric between agent poses and imitation video. The results of these models, along with the default manual reward function provided by the environment, are shown in Figure 4b The multi-modal version appears to perform about equal to the vision-only modal. In Figure 4b we also compare our method
non-ssquenccebasee model that is equivalent to Time Contrastive Network (TCN). On average VIRI. achieves higher value policies. We find that using the RNN-based distance metric makes the learning process more gradual. We show this learning effect in Figure 4b where the original manually created reward with flat feedback leads to slow initial learning.
In Figure 4c we compare the importance of the spatial - and temporal ||n 1op--i? -1211 represen- tations learned by VIRL. Using the recurrent representation (temporal alone allows learning to progress quickly but can have difficulty informing the policy of how to best match the desired exam- ple. On the other hand, using only the encoding between single frames (spatial conv) slows learning due to limited reward for out-of-phase behaviour. We achieved the best results by combining the representations from these two models. The assistance of spatial rewards is also seen in Figure 4b where the manual reward learns the slowest.
Ablation We conduct ablation studies in Figure 5a to compare the effects of data augmentation methods, network models and the use of additional data from other tasks. For the more complex humanoid3d control problems the data augmentation methods, including Early Episode Sequence Priority (EESP), increases average policy quality marginally. The use of mutlitask data Figure &c] and the additional representational losses Figure Sa greatly improve the methods ability to learn. More ablation results are available in the supplementary material.
Sequence Encoding Using the learned sequence encoder collection of motions from different classes are processed to create a TSNE embedding of the encodings (Maaten & Hinton 2008). In Figure 5c we plot motions both generated from the learned policy and the expert trajectories

---

Under review as a conference paper at ICLR 2020
Figure 3: Rasterized frames of the agent's motion after training on humanoid3d walking (row 1,2), running (row 3-5), zombie (row 6,7) and jumping(row 80).). The multi-coloured agent a rendering of the imitation video. A video of these results is available here: https:/yootu.bes1iiiivVvyzz
Tg. Overlaps in specific areas of the space for similar classes across learned and expert mg data indicate a well-formed distance metric that does not sperate expert and agent examples. There is also separation between motion classes in the data, and the cyelic nature of the walking cycle is visible.
In this section, we have described the process followed to create and analyze VIRL. Due com- bination of data augmentation techniques, VIRL can imitate given only a single demonstration. We have shown some of the first results to learn imitative policies from video data using recurrent net-

---

Under review as a conference paper at ICLR 2020
Swidead Deats, toilie
baseline comparison
sequence-baser comparison
Spatial Temporal
Figure 4: Baseline comparisons between our sequence-based method, GAIL and TCN (4a) on the humanoid2d environment. Two additional baseline comparison between VIRI and TCN 4b In the benefit of combing spatial and temporal distances is shown. In these plots, the large solid lines the average performance of collection policy training simulations. The dotted lines of the same colour are the specific performance values for each policy training run.
Turkng
(a) Ablation humanoid3d walking
(b) Distance Metric Ablation
(c) TSNE embedding manoid3d)
(hu-
Figure 5: Ablation analysis of VIRL. We find that training RL policies is sensitive to the size and distribution of rewards. A few modifications assist in the siamese network's 's ability to compute useful distances. Including VAE and AE losses to assist in representation learning. The addition of multi-task training data is also important for learning better policies.
work. Interestingly, the method displays new learning efficiencies that are important to the method success by separating the imitation problem into spatial and temporal aspects. For best results, we found that the inclusion of additional regularizing losses on the recurrent siamese network, along with some multi-task supervision, was key to producing results.
DISCUSSION AND CONCLUSION
In this work, have created stration. The method uses a Siamese recurrent network to learn a distance function in both space and time. This distance function, trained on noisy partially observed video data, is used reward function for training an RL policy. Using data from other motion styles and regularization terms, VIRL produces policies that demonstrate similar behaviour to the demonstration.
method for learning imitative policies from single demon-
Learning a distance metric is enigmatic, the distance metric can compute inaccurate distances in areas of the state space has not yet seen. This inaccuracy could imply that when the agent explores and finds truly new and promising trajectories, the distance metric computes incorrect distances. We attempt to mitigate this effect by including training data from different tasks. We believe VIRL will benefit from more extensive collection of multi-task data and increased variation of each task. Additionally, if the distance metric confidence available, this information could used to reduce variance and overconfidence during policy optimization.
probable learning a reward function while training adds additional variance to the policy gra- dient. This variance may indicate that the bias of off-policy methods could be preferred over the added variance of on-policy methods used here. We also find it important to have a small learning rate for the distance metric. The low learning rate reduces the reward variance between data collec- tion phases and allows learning a more accurate value function. Another approach may be to use partially observable RI. that can learn better value function model given changing RNN-based
8

---

Under review as a conference paper at ICLR 2020
reward function. Training the distance metric could benefit from additional regularization such as constraining the kl-divergence between updates to reduce variance. Learning a sequence-basee based pol- icy well, given that the rewards are now not dependent single state observation is another area for future research.
method to GAIL, but found GAIL has limited temporal consisteney. This method led to learning jerky and overactive policies. The use of a recurrent discriminator for GAIL may mitigate some of these issues and is left for future work. It is challenging better than the carefully manually crafted reward functions used by the RL simulation environments that include motion phase information we have shown that our method that can compute distances in space and time has faster imitial learning. Potentially, combination of starting with our method and following with manually crafted reward function could lead to faster learning of high-quality policies. Still, as environments become increasingly more realistic and grow in complexity, we will need more robust methods to describe the desired behaviour we want from the agent.
We compare
produce results
the observations (Peng et al.) 2018a 2017). However,
Training the distance metric is a complicated balancing game. One might expect that the distance metric should be trained early and fast so that it quickly understands the difference between good and bad demonstration. However, quickly learning confuses the agent, rewards can change, which cause the agent to diverge off toward unrecoverable policy space. Slower is better, as the distance metric may not be accurate, it may be locally or relatively reasonable, which is enough to learn a good policy. As learning continues, these two optimizations converge together.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-first International Conference Machine Learning, ICML '04, pp- 1-, New York, NY, USA, 2004. ACM. ISBN 1-58113-838-5. doi: 10. 00.1145/11153301015544 URL http:://doo www.:ea.c.oooogg...ooo /tw.g/g...... 11
acm
http:/11.1111/11111
1015430
Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning
from demonstration. Robotics and Autonomous Systems, 57(5))469 483, 2009. ISSN 0921-8890.
doi: http://doi.org/10. robo
robot. 2008. 10.024.
URL http:/ http:://wwww../ www. sci wwweeciiiieeeeeeeeeeeeeeeeeiieee
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005.
IEEE Computer Society Conference on, volume 1, pp. 539-546. IEEE, 2005.
D. Dwibedi, Tompson, C. Lynch, and Sermanet. Learning Actionable Representations from Visual Observations. ArXiv e-prints, August 2018.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imi- tation learning via meta-learning. CoRR,
2017. URL attp:://axxii..rrr...
0bss1170.04900
Ian Goodfellow, Jean Abuget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 2672-2680, Curran Associates, Inc., 2014. URL
wwtpssccppppereeee 5423 generative adversarial wetteriialletet...ee nwws.ne......m pdf
K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9): 1904-1916, Sept 2015. ISSN 0162-8828. doi: (0.1109/TAAM.220155288888
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems 29, pp. 4575-4573. Curran Associates, Inc., 2016. learning..
6391 www.rllgeeeeeeieeeeeeeeee nips. :.Lps.cc//ppper//91-goooaaaaee www.:eeavvvverrrraaaeaa
pdf

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference onLearning Representations (ICLR), 2014.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from
visual demonstrations. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3812-3822. Curran Associates, Inc., 2017. URL httt::tppppppsaaaese//e nips www.:iiissp..ppp.occooeoooco
pdf
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1500.00971, 2015. URL
02971
Yuxuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. CoRR, abs/1707.03374 2017. URL
org wwt.:o//g/abbbbaoooo.ooo.. awtaaa/s/1110000 1777.0337
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-2665, 2008.
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. CoRR, 200/0200,,, 2017. URL
Jonas Mueller and Aditya Thyagarajan. Siamese recurrent architectures for learning sentence sim- ilarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI' 16, pp. 2786-2792. AAAI Press, 2016. URL http:///l. acm. wwtpoo/////tttttt 301 2016100.1 1016610..1000 301
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. CoRR, 007.00077.444, 2018. URL http:// arxiv.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo. Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. CoRR, abs/1 /ss/1880408606, 2018.
http://awxxvo..o///bs/a//////// 804. 08606
Xue Bin Peng and Michiel van de Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation, SCA '17, pp. 12:1-12:13, New York, NY, USA, 2017. ACM. ISBN 978- 1-4503-5091- doi: 10.1 10.1145/309956433095555 URL http:://d/oo actp:::/ddi..cm.commmmmmm acm. wttp///........r...0..1111.11
3099564. 3099567
Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph., 36(4): 41:1-41:13, July 2017. ISSN 0730-0301. doi: 10. 10.11445/307299933333600 URL hRLhhtpp///oi. acm. ccm.orrg/...1115/777293333 307602
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37
(4):143:11 July 2018a. ISSN 0730-0301. doi: 10. 10.1145/3197517:32011 //doi acm. org./////// 10. 1111/111115111 3201311
URLhttp
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. arXiv preprint arXiv: arXiv: 1810.0081, 2018b.
10

---

Under review as a conference paper at ICLR 2020
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. Geoffrey Gordon, David Dunson, and Miroslav Dudk (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Lauderdale, FI., USA, 11-13 Apr 2011. PMLR. URL v15/ ht15/rosslla.h.... html
presss/
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. ArXiv e-prints, July 2017.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05547,, 2015. URL http::/aaxx...org/abb//////
05477
Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastive networks: Self- supervised learning from multi-view observation. CoRR, abs/1704.06688,, 2017. URL http:
06888
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y.,,, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller. DeepMind Control Suite. ArXiv e-prints, January 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral Cloning from Observation. (July), 2018.
URL http: org/ abs 1wt///885...0 1100.01954......
Hado Van Hasselt. Reinforcement learning in continuous state and action spaces. In Reinforcement Learning, pp. 207-251. Springer, 2012.
Ziyu Wang, Josh Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Ro- bust imitation of diverse behaviors. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, nips cc/papex/
pp. 5329. Curran Associates, Inc., 2017. URL 2116-robust imiiatiiinniie. of of- diverse behaviors.ppff
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. CoRR, abs/1802.015577 2018.
org abs 1802.8880 01557
Zhuotum Zhu, Xinggang Wang, Song Bai, Cong Yao, and Xiang Bai. Deep learning representation using autoencoder for 3d shape retrieval. Neurocomputing, 204:41-50, 2016.
Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. Supervised represen- tation learning: Transfer learning with deep autoencoders. In Twenty-Fourt International Joint Conference on Artificial Intelligence, 2015.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI008, pp. 1433-1438. AAAI Press, 2008 ISBN 978-1-57735-368-3. URL http: acm. http://iaaaiooo.omdddeee-.-..
11

---

sequence Here we want to explicitly learn a latent variable =RNN that compresses the infor- Under review as a conference paper at ICLR 2020
7
APPENDIX
This section includes additional details related to VIRL.
7.1 IMITATION LEARNING
Imitation learning the process of training new policy to reproduce the behaviour of some expert policy. BC is fundamental method for imitation learning. Given expert policy possibly represented as a collection of trajectories < (s0. ao),...; (ST, aT) > new policy и can be learned to match this trajectory using supervised learning.
T
max E
log π (aels),,
(5)
t=0
While this simple method can work well, it often suffers from distribution mismatch issues leading to compounding errors as the learned policy deviates from the expert's behaviour.
7.2 INVERSE REINFORCEMENT LEARNING
Similar BC, Inverse Reinforcement Learning (IRL) also learns to replicate some desired be- haviour. However, IRL. makes use of the RL environment without a defined reward function. Here describe maximal entropy IRL (Ziebart et al., (008). Given expert trajectory (s0, .o.,..., (ST,ar aT) > a policy can be trained to produce similar trajectories by discovering distance metric between the expert trajectory and trajectories produced by the policy n
max min
cEC [e a)] H(T)) Exs [c(s.a)]
(6)
where e is some learned cost function and (T) is a causal entropy term. TE is the expert policy that is represented by a collection of trajectories. IRL is searching for a cost function that is low for the expert म and high for other policies. Then, a policy can be optimized by maximizing the reward function rtt -c(s,, a.).
7.3 AUTO-ENCODER FRAMEWORK
Variational Auto-encoders Previous work shows that VAEs can learn a lower dimensional struc- tured representation of distribution (Kingma & Welling 2014 A VAE consists of two parts an encoder 96 and a decoder Ps. The encoder maps states to a latent encoding and in turn the decoder transforms z back states. The model parameters for both & and 1b are trained jointly to maximize
AE 2b, s) - 3DD (go(=l())(lll))]l
(xlls)) Po (s|[2)
where DK is the Kullback- -eibblerr divergence, p(s) is some prior and a hyper- sepee-parameter to balance the two terms. The encoder 9R takes the form of diagonal Gaussian distribution Is = N (Hu(s), os)).. In the case of images, the decoder pe parameterized a Bernoulli distribution over pixel values. This simple parameterization akin training the decoder with cross entropy loss over normalized pixel values.
Sequence Auto-encoding The goal of sequence to sequence translation is to learn the condi- tional probability ..., yr (zo............................. .T), where x ........2 UT and y Yo...., '/r"
mation in TT. An RNN can model this conditional probability by calculating Ie (ro, ..... 2T of the sequence that can, in turn, be used to condition the decoding of the sequence y (Rumelhart et al. 1985).
T
p(y)
p (vo..... -1-11,10) t=
This method has been used for learning compressed representtiios for transfer learning (Zhu et al, 2016) and 3D shape retrieval (Zhuang al. 2015).
12

---

Under review as a conference paper at ICLR 2020
7.4. DATA
The mocap used in the created environment come from the CMU mocap database and the SFU mocap database.
Data Augmentation and Training We apply several data augmentation methods to produce ad- ditional data for training the distance metric. Using methods analogous the cropping and warping methods popular in computer vision (He et al. 2015) we randomly crop sequences and randomly warp the demonstration timing. The cropping performed by both initializing the agent random poses from the demonstration motion and terminating episodes when the agent's head, hands or torso contact the ground. As the agent improves, the average length of each episode increases and so to will the average length of the cropped window. The motion warping is done by replaying the demonstration motion at different speeds. Two additional methods influence the data distribution. The first method is Reference State Initialization (RSI) (Peng al.) 2018a). where the initial state of the agent and expert randomly selected from the expert demonstration. With this property, the environment agent to go back to random points in the demonstration as if replaying a remembered demonstration. The second is EESP where the probability a sequence x is cropped starting at i is p(i) [en(x) increasing the likelihood of starting earlier in the episode.
also be thought of as form of memory replay. The environment allows the
7.5 TRAINING DETAILS
The learning simulations are trained using Graphics Processing Unit (GPU)s. The simulation is only simulating the interaction physics of the world but also rendering the simulation scene to capture video observations. On average, it takes 3 days to execute a single training simulation. The process of rendering and copying the images from the GPU is one of the most expensive operations with VIRL. We collect 2048 data samples between training rounds. The batch size for Trust Region Policy Optimization (TRPO) 2048. The term 0.5.
The simulation environment includes several different tasks that are represented by collection of motion capture clips to imitate. These tasks come from the tasks created in the DeepMimic works (Peng al. 2018a). We include all humanoid tasks in this dataset.
In Algorithm we include an outline of the algorithm used for the method. The simulation envi- ronment produces three types of observations, 8141 the agent's proprioceptive pose, 6/-1 the image observation of the agent and m++1 the image-based oberservation of the expert demonstration. The images are 64 x 64.
7.6 DISTANCE FUNCTION TRAINING
Our Siamese training loss consists of
LSN (si, Sp: 8n) y* [1](s;) - [(p))1 + ((1 - (max(p (1](s::) ((s))l))))) (9)
where y is positive example Sp. pair where the distance should be minimal and y 0 is a negative example 8a, pair where the distance should be maximal. The margin is used as an attractor or anchor to pull the negative example output away from s; and push values towards a 0 to range. f-) computes the output from the underlying network. The distance between two states is calculated d(s, = [I](s) [(s')l and the reward as r(s, s') = -d(s, Data used train the Siamese network is combination of trajectories т (Su..... ST) generated from simulating the agent in the environment and the expert demonstration. For our recurrent model the same loss is used; however, the states 8p, 8n, 8; are sequences. During RI training we compute a distance given the sequence of states observed so far in the episode. This method allows us to train distance function in state space where all need to provide is labels that denote if two states, sequences, are similar or not.
show the training curve for the recurrent siamese network. The model learns smoothly, considering that the training data used is continually changing as the RL agent explores. In Figure ba the learning curve for the siamese RNN is shown after performing pretraining. We can see the overfitting portion the occurs during RL training. This overfitting can lead to poor reward prediction during the early phase of training.
In Figure 6b
13

---

Under review as a conference paper at ICLR 2020
Raration x160.0
Iteration x1.00
(a) siamese loss
(b) siamese loss with pretraining
Figure 6: Training losses for the siamese distance metric. Higher is better it indicates the distance between sequences from the same class are closer.
can be challenging to train sequenced based distance function. One particular challenge training the distance function to be accurate across the space of possible states. We found good strategy was to focus on the beginning episode data. When the model is not accurate on states it saw carlier in the episode; it may never learn how to get into good states later that the distance function understands better. Therefore, when constructing batches to train the RNN on, we give higher probability of starting earlier in episodes. We also give higher probability to shorter sequences. As the agent gets better average episodes length increase, so to will the randomly selected sequence windows.
7.7 DISTANCE FUNCTION USE
We find it helpful to normalize the distance metric outputs using = eer(r"**ux where wa 5.0 scales the filtering width. Early in training the distance metric often produces large, noisy values. Also, the RL method regularly updates reward scaling statisties; the initial high variance data reduces the significance better distance metric values produced later on by scaling them to small numbers. The improvement of using this normalize reward is shown in Figure 7a
(a) Reward smoothing
(b) Siamese update frequency
Figure 7: Ablation analysis of VIRL. We find that training RL policies is sensitive to the size and distribution of rewards. The siamese network benefits from several training adjustments that make more suitable for RI.
8 POSITIVE AND NEGATIVE EXAMPLES
We use two methods to generate positive and negative examples. The first method is similar to TCN, where we can assume that sequences that overlap more time are more similar. For each episode two sequences generated, one for the agent and one for the imitation motion. Here we list the methods used to alter sequences for positive pairs.
1. Adding Gaussian noise to each state in the sequence (mean 0 and variance 0.02)
2. Out of sync versions where the first state is removed from the first sequence and the last state from the second sequence
3. Duplicating the first state either sequence
4. Duplicating the last state in either sequence
We alter sequences for negative pairs
14

---

Under review as a conference paper at ICLR 2020
1. Reversing the ordering of the second sequence in the pair.
2. Randomly picking state out of the second sequence and replicating it to be as long as the first sequence.
3. Randomly shuffling one sequence.
4. Randomly shuffling both sequences.
The second method we use to create positive and negative examples is by including data for addi- tional classes of motion. These classes denote different task types. For the humanoid3d environment, we generate data for the same tasks are labelled as positive, and pairs from different classes are negative.
running, backflipping and frontflipping. Pairs from
8. ADDITIONAL ABLATION ANALYSIS
LSTM AE, Walk
Variance Comparison, Jump
MultiTask Data, Run
Figure 8: Additional dblation analysis of VIRL on humanoid3d.
8.2 RL ALGORITHM ANALYSIS
It is not clear which RL algorithm may work best for this type of imitation problem. number of RL algorithms were evaluated on the humanoid2d environment Figure Pa Surprisingly, TRPO (Schul- man et al. (013)) did not work well in this framework, considering has a controlled policy gradient step, thought it would reduce the overall variance. We found that Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al.) 2015) worked rather well. This result could be related to hav- ing changing reward function, in that if the changing rewards can be easier to learn. This can be seen in Figure Ob where DDPG is best at estimating the future discounted rewards in the environment. We also tried Continuous Actor Critic Learning Automa- ton (CACLA) (Van Hasselt 2012) and Proximal Policy Optimization (PPO) (Schulman et al.. 2017); we found that PPO did not work particularly well on this task; this could also be related added variance.
considered off-policy data,
Om Aradd wts Twenerieeeees
Smn Aelei Troking lnbere dibea
Shalsed Audeedddeeeee 30000 weppeeeessiio
(a) Average Reward
(b) Bellman Error
sequence- bequence-based based comparison DDPG
(c)
Figure RI. algorithm comparison on humanoid2d environment.
8.3 ADDITIONAL IMITATION RESULTS
Our first experiments evaluate the methods ability to learn complex eyclic motion for simulated humanoid robot given a single motion demonstration, similar to (Peng & van de Pannel 2017), but using video instead. The agent is able to learn robust walking gate even though only given noisy partial observations of demonstration Figure T0
15

---

Under review as a conference paper at ICLR 2020
Figure 10: Still frame shots from policy trained in the humanoid2d environment.
16


Reviews:
[{"review_id": "rJeK8ITJ5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an imitation learning method that deploys previously well-studied techniques such as siamese networks, inverse RL, learning distance functions for IRL and tracking.\n\n+ the paper studies an important problem of IL using visual data.\n+ I found the ablation studies in the appendix quite useful in understanding the efficiency of the proposed method.\n\n-In terms of novelty, the proposed approach is a combination of several past works so the technical novelty is limited. Additionally, it is not clear how impactful the proposed method can be given that it is only tested on a synthetic domain which is the same as the train domain. So, from the current experimental results it is not clear if this approach would be effective to be applied in a real system (e.g. robots) on the practical side. \n\n-There are not enough evaluation done to compare with the most updated state-of-the-art baselines. The evaluations are done on just a single synthetic domain with a single character. Therefore, the train and test videos are very similar. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syl0ilrb9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of the paper is to learn a distance function between observed and the agent’s behaviors. Once they have the distance function, they can learn the agent’s policy efficiently given a single demonstration of each task. In their formulation, the distance function and the policy are jointly learned. \n\nThe idea is reasonable and the performance outperforms baselines like GAIL and VAE. However, the paper is not-well written with many relevant equations defined in the supplementary material. The unsupervised data labeling part seems Adhoc with many details in the supplementary material. I wonder if the process stable or not. How many lower than the average performance of the proposed method as shown in F.g 4 are caused by unsupervised data labeling?\n\nIn Fig. 4b, the manual performance is very strong once converged. Although the proposed method initially reaches high reward, after twice many iterations the manual performance even outperforms the proposed method on average many times. Hence, I am not very convinced about the proposed method will be the best-picked method in practice.\n\nOverall, I think the idea is good. But the paper is poorly written and I concern the most about the stability of the unsupervised data labeling process. The experimental results are also not super convincing. Hence, I recommend for weak rejection.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgFxwxAFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents visual imitation with reinforcement learning (VIRL), an algorithm for learning to imitate expert trajectories based solely on visual observations, and without access to the expert’s actions.  The algorithm is similar in form to GAIL and its extensions, learning a reward function which captures the similarity between an observed behavior and the expert's demonstrations, while simultaneously using reinforcement learning to find a policy maximizing this reward, such that the learned policy will replicate the demonstrated behavior as well as possible.  A key feature of this method is that the learned reward function is defined by a learned distance metric, which evaluates the similarity between the agent's current trajectory, and the nearest demonstrated expert trajectory.\n\nThe network describing the distance metric is recurrent, such that the distance is defined between trajectories rather than individual states.  The distance function network is trained via a negative sampling approach, where expert trajectories are randomly reordered to produce examples that dissimilar to the expert trajectories.   The distance network also defines a variational autoencoder, and the reconstruction of the target trajectories is treated as an auxiliary task to help train better representations of the trajectory space.\n\nWhile previous work has considered the problem of visual imitation learning, the approach taken here is novel in its architecture and loss function, and significantly outperforms the baselines in terms of the similarity between the resulting behavior and the expert behavior.\n\nThe clarity of the technical presentation could be improved, however.  In particular, it would be helpful for the reader if the definitions of the negative sampling loss and the autoencoder losses were given before the combined loss, and if we saw the form of the loss for both positive and negative sequence pairs.  Equation 4 could also be made explicit, with the full summation term included.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main concern raised by reviewers is limited novelty, poor presentation, and limited experiments. All the reviewers appreciate the difficulty and importance of the problem. The rebuttal helped clarify novelty, but the other concerns remain.
Decision: Reject