Meta-Imitation Learning by Watching Video Demonstrations
Authors: Jiayi Li
Tao Lu
Xiaoge Cao
Yinghao Cai
Shuo Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Meta-imitation Learning
One-shot Learning
Learning by Watching
Generative Adversarial Networks

Abstract: Meta-Imitation Learning is a promising technique for the robot to learn a new task from observing one or a few human demonstrations. However, it usually requires a significant number of demonstrations both from humans and robots during the meta-training phase, which is a laborious and hard work for data collection, especially in recording the actions and specifying the correspondence between human and robot. In this work, we present an approach of meta-imitation learning by watching video demonstrations from humans. In comparison to prior works, our approach is able to translate human videos into practical robot demonstrations and train the meta-policy with adaptive loss based on the quality of the translated data. Our approach relies only on human videos and does not require robot demonstration, which facilitates data collection and is more in line with human imitation behavior. Experiments reveal that our method achieves the comparable performance to the baseline on fast learning a set of vision-based tasks through watching a single video demonstration.

Published as a conference paper at ICLR 2022
META- LEARNING BY WATCHING VIDEO DEMONSTRATIONS
Jiayi Li Tao Lu², Xiaoge Cao-.2, Yinghao Cai & Shuo Wang
1 Schooo of Artificial Intelligence, University of Chinese Academy of Sciences
State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences
"Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences Aljjaay2019 tao. lu, caoxiaoge2020, yinghao. caí, shuo. wang] en
ABSTRACT
Meta-Imitation Learning a promising technique for the robot learn a new task from observing one few human demonstrations. However, it usually requires a significant number of demonstrations both from humans and robots during the meta-training phase, which is laborious and hard work for data collection, espe- cially in recording the actions and specifying the correspondence between human and robot. In this work, present approach of meta-imitation learning by watching video demonstrations from humans. In comparison to prior works, our approach is able to translate human videos into practical robot demonstrations and train the meta-policy with adaptive loss based on the quality of the trans- lated data. Our approach relies only on human videos and does not require robot demonstration. which facilitates data collection and is more in line with human imitation behavior. Experiments reveal that our method achieves the comparable performance to the baseline on fast learning a set of vision- based tasks through watching single video demonstration.
INTRODUCTION
The demonstration provides skill guidance for specifying robotic tasks. Through it, robots can ac- quire many complex skills, including table tennis (Mälling et al. 2013). pouring water (Sermanet al.] 2018), and picking objects (Levine et al. (018)). Most prior work assumes that robots can receive demonstrations via kinesthetic teaching (Elliott et al. 2017 Ragaglia et al., 2018), teleoper- ation (Savarimuthu et al. 2017: Zhang et al.) 2018), or crowdsourcing platform (Mandlekar et al.] 2018 Wang et al.) 2021). which are distinct from the way how humans imitate others. Due to the mirror neurons Tranel et al. 2003 Molenberghs et al. 2009). humans are often able to watch others act, infer the intention, map to their
embodiment, expand skill set and enhance the representa- tions of the world. Motivated by this, we aim to endow robots with the ability to learn manipulation skills via video demonstrations from humans without access the actions of the demonstrator. Then the key challenge we would face is how to bridge the human-robot domain gap caused by the morphological difference and infer the performed skills from the raw video.
One way to overcome the domain gap is to translate the human videos to the robot domain using a generative modeling approach (Sharma et al. 2019 Smith et al., 2019 Yang al.) 2020). While this approach is intuitive, it is prone to capture task-agnostic pixel information so that requires task guidance and it is necessary evaluate the quality of translated images. The other way is to model visual imitation as shet learning problem (Duan et al. [017]. which can be solved with meta- learning algorithms (Finn et al., 20176] Yu et al.) 2018b). Domain gaps can be addressed with a supervised training process through the cross-domain dataset. However, meta-learning methods often require large amounts of demonstrations both from humans and robots during the meta-training phase, which is demanding and cine-connuming process for data collection, especially in recording the actions and specifying the correspondence between human and robot.
In this paper, we introduce new approach of meta-imitation learning by watching video demonstra- tions from humans. This approach automatically translates human videos to robot demonstrations using new- designee generative model and trains the meta-polis meta-policy in the imagined compact latent space learned during the translation process from human to robot. Empirical results suggest that the learned meta-policy achieved the comparable results to the baselines learning set of chal-

---

Published as a conference paper at ICLR 2022
lenging rision-based tasks under the condition of watching one video demonstration. This is the first time, to our knowledge, that meta-imitatee learning has been addressed only by using image obser- vation with need for demonstration information of robot states and actions. Our approach learns the correspondence between human and robot and trains end-to-end, which places the least demand on the imitation system and is natural learning way just like humans. The key contributions of this paper are summarized as follows:
Learning action-awarenes: compact latent representation from human to robot. We proposed novel CycleGAN structure, named A-C A-CycleGAN, that automatically translates the human domain data into the robot domain and learns parallel training of inverse dynamic model under the robot domain. This new structure helps us to achieve effective robot demonstration data from human videos.
aetion-awarrness compact latent representation with
Self-adaptive meta-imitation learning under the imagined latent space. The meta-policy is trained with the learned latent states achieved from A-CycleGAN and the actions predicted from the learned inverse dynamic model. All data utilized during training could be regarded being gen- erated by imagination based on human demonstrations. A self-adaptive mechanism is presented evaluate the quality of the translated data and prioritize high-quality data for better gradients update, ensuring that the meta-policy is properly trained even when the data does not come from actual robot actions.
Empirical performance on challenging rision-based tasks. We pair our approach with existing meta- inntaiiiitation learning methods which learns the meta-policy with both human and robot demon- strations on a set of challenging tasks. With the same human video demonstrations, our method achieved comparable results to the baselines in one-shot new task learning.
RELATED WORK
Learning from Demonstration (LID). LID is a rich and diverse research field that focuses on al- lowing robots to learn skills from humans or other expert demonstrations Ravichandar Raaioistat et al. 2020). We follow the problem of learning new task from one video demonstration of human, conforming to the reality that the robot can not receive the action information from the demonstration directly. The key point is how to specify the correspondence between human and robot. is usually resolved by two types of methods, either explicit implicit. Many explicit methods manually define how hu- man activities are translated into robot actions, with the help of specific visual recognition pipelines (Nguyen et al.) 2018 Yang et al., 2015). These techniques rely on the precise detection of hands and objects. Recent works are mainly concentrated on representation learning techniques implicitly. Many of these produce metric embeddings that focus on the interactions in the environment and are insensitive to nui puisance variables such as viewpoint and appearance in an unsupervised fashion (Ser-) manet et al., 2018: (cchmeckpeper et al., 2020; Zhou et al. [02I]. Other approaches train action or task classifiers from human and robot videos. The distance between the observed demonstration and the execution of the robot is used as a reward of reinforcement learning in learning the demonstrated task (Sermanet al. 2016 Sieb & Fragkiadaki 2018 Shao al. 2020aa] Pauly et al. [021). There are also ways to align the two domains using the generative model. Some work generates robot do- main images by integrating human-robot paired data with contexts (Liu al., 2018 Sharma al., 2019). while others employ pixel-level image translation with unpaired training data (Smith et al.
2019
Xiong al. 2021).
One-shot Imitation Learning. Humans naturally acquire the ability to execute a new task by wit- nessing it once performed by another individual. Prior research has enabled robots acquire similar ability. Some methods take advantage of the attention mechanism of the model or symbol planner to extract the operated items from a demonstration and transfer the skill directly (Huang al.. 2019 Shao et al.) 20206] Dasari & Gupta. 2020). Other approaches learn compact representation of task joint with the control policy and deduce information from unfamiliar tasks in task-embedded space (James et al.) 2018 Bonardi et al., 2020). In other work, the ohe-shot learning problem is solved with model-agnostic meta-learning (MAML) algorithm (Finn et al. 2017a). learns the task initial parameters by leveraging large amounts of prior meta-training data and optimize it via one or a few steps of gradient descent with new tasks (Finn et al. 20176). Domain Adaptation Meta- Learning (DAML) (Yu et al., 2018b) is an extension of MAML and could learn robotic manipulation skills from a single video of human. And it is also extended on multi-stage manipulation tasks (Yu et al. 2018a). Our approach is based on this technique, but recover the necessary information
2

---

Published as a conference paper at ICLR 2022
to complete the meta-policy learning from human videos without specialized access to the robot demonstration (such as action and state). This greatly lowers the difficulties in data collections for policy-training and places least demand on the imitation system, which is more natural way for robot imitation learning.
3 PRELIMINARIES
To learn from the human demonstration, our approach builds upon some prior work and presents an extension of algorithm that relaxes its implementation requirements. In this section, we will overview unsupervised image-to-image translation (Zhu et al. 2017) and domain-adaptive meta- learning (DAML) algorithm Yu et al. 2018b).
3.1 UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION
We treat the human-to-robot observation translation unsupervised image-to-image translation problem. That is, we aim to map images from a source domain (e.g., human domain) to a target domain Y (e.g., robot domain) in the absence of paired examples. CycleGAN (Zhu et al. 2017) is technique for learning mapping between two image domains from unpaired data. Mathematically, CycleGAN learns the translator G : X Y and inverse translator F Y + X given training samples {2} E X and {y} E Y. The image is embedded by the encoder of the translator into the latent space and then reconstructed by the decoder. The embedding in the latent space domain- agnostic. The generator needs to be jointly trained with two adversarial discriminators Dx and Dy, where Dx attempts to distinguish between images {ar ] and translated images { (F()) to encourages F to generate the outputs indistinguishable from domain X, and vice versa for Dy and G. The model is trained by minimizing adversarial losses LGAN:
LGAN(G, Dy, Dy, = Eyyrractos) (lo Dy Dy(y)) + Eu-pauca [log(1 - Dy (G()))) (I)
where aims to minimize this objective while an adversary 1 strives to maximize it. similar for F and Dx Further to avoid mode collapse that all input images map to the same output image, the cycle consistency loss Leye is introduced to incentivize the original image recovered after two translators:
Leye (G, F) = E (ea--paan.te)lIII(GGG æ||1] + Ey-pean (IG ylla].
(2)
Combining the two losses yields the full objective for translation:
Loo(G, F, Dx, Dy) LGAN (G, Dy, X, Y) LGAN (F, Dx X,Y) Meyo(G. F),
(3)
where À is the hyperparameter that controls the relative importance of the two objectives.
3.2 DOMAIN-ADAPTIVE META-LEARNING
Nomain-adaptive meta-Icarning (DAML) is an extension of the model-agnostic meta-learning algo- rithm (MAML), which could learn new task from one human video demonstration. Assuming that meta-training and meta-test tasks are drawn from the same distribution p(T), MAML is a meta- learning algorithm that intends to learn tasks using small amount of data. It extracts prior task knowledge using meta-training tasks, then inferences under learned priors from the meta-test demonstration (Grant et al. 2018). Specifically, consider supervised learning problem with loss function denoted as L(,, Dr), where 0 denotes the model parameters and Dy denotes the labeled data for task T. During meta- meta-training, MAML samples a task T and data from Dr, which are randomly partitioned into two sets, D,, and Deat. MAML optimizes the model parameters e such that one or a small number of gradient steps on Der will produce maximally effective behavior on Deat, corresponding to the following objective:
Σ L(0-a D4), D = min T~p(T)
Σ L(GT, Dyrt, T<p(T)
min
B
where a is the step size of the gradient update and the Ør is the updated parameters. The meta- optimization is performed over the model parameters 0, whereas the objective is derived using the updated parameters Φτ. At meta-test period, MAML executes gradient descent with to adapt the new task Trest using examples drawn from
@ = 0 a VLLO, Dtr...).
(5)
3

---

Published as a conference paper at ICLR 2022
A-CycleGAN
Self-Adaptive Meta-Imitation Learning
Urony Lever
Human Video
G
Inargined Vidro
Clear
Blur
Continuon
Beeanre Gradient
SSIM
Dean
Skared Conc Laver
Pass Bre SSIM Fea1)
d'
Lat
Latest Space
Latent States
to Teeeperal
Adtaptation objective
Kandom Video
y
Felare
Timppral
af
Fas
Xr+1
Predicted Action
Mete-objeeiive
y
Inverse namic Model
La--r
Meta-Imitation Learning
Figure Illustration of the Overall Architecture. Left: Action-awarenen CycleGAN (A- CycleGAN): The image translation model establishes the bidirectional mapping between the human demonstrations and the imagined videos of the robot. Then, inverse dynamic model takes as input the latent state to predict the corresponding robot action â", The CycleGAN part and Inverse Dynamic Model part share encoder layers of generator (shown in pink). Right: Self Addptivee Meta-Imitation Learning. The Imitttimtttionn Learning structure is similar to DAML and the meta- policy is trained with the data from the latent states in and predicted actions à". The translated imagined robot videos d' are evaluated by Byas to adaptively adjust the meta-objective.
DAML applies the MAML algorithm to the domain addapiiv one-shot imitation learning setting to solve the problem of learning from human video. Unlike the supervised meta-learning issue, can not use the standard imitation learning loss for the inner adaptation objective computed using Drr without the human's 's actions. To handle the domain shift between the human and robot, DAML adopts the multiple layers of ID convolutions over time to donate the adaptation objective L which can be interpreted as simply directing the policy parameter update to modify the policy to pick up on the right visual cues in the scene. During meta-training, the meta-objective for A is the behavioral cloning loss L
min ΣΣ Σ T D' d'eDy
LBC(f (V)),(0,
(6)
where d is the human demonstration and d' is the robot demonstration. After training, DAML runs the gradient descent using the adaptation loss Ly based on the human demonstration for a new task.
4 METHOD
Our goal is to learn the task prior knowledge and rapidly adapt to new tasks using only human video without robot demonstration. Our method comprises two main modules: domain adaptation A-CycleGAN and self-adaptive meta-imitation learning module with adaptive loss. In this section, we will systematically describe them.
4.1 A-CYCLEGAN: ACTION-AWAREIE CYCLEGAN FOR DOMAIN ADAPTATION
To extract the available states of the robot domain from human videos dh D4, including the domain-agnostic but action-awareness representations of scenes and corresponding actions for robot, we propose new domain- adaptaaionn method, named Action- aattennes CycleGAN (A-CycleGAN) to establish the bidirectional mapping between the human demonstrations (human/source domain) and the corresponding generated robot videos (robot/target domain). In meta-imitation phrase, we refer to the robot videos translated from the human demonstrations by the A-CycleGAN the imagined robot videos d' E Dr. Given that the original CycleGAN only supervise the overall
4

---

Published as a conference paper at ICLR 2022
content and style in the scene which may be inclined to reconstruct fixed task-agnostic background, we combine it with a coupled inverse dynamic controller ()(-) to predict the awareness im- ages of human behavior in the robot domain to propel the task-related translation. So in general, A-CycleGiAN includes two parts: the CycleGAN part and the Inverse Dynamic Model part. For CycleGAN part, due to the multi-task experiment setting, the original objective of CycleGAN stated in Section [.1] can not guarantee the mapping consistency within the scene, for which we collect limited number of pairwise human and robot random video p", p" (these pairwise videos do not have to be perfectly aligned and easy to collect) and add an auxiliary objective as follows:
Lauz (G, P) (lG + - [|P")
That is, the output of the generator is constrained by the other domain image, improving the certainty of the content in the scene.
As for inverse dynamic model, it takes input current visual observation of the robot x along with the observation for the next time step o to predict the actions a = (y +{+1) the robot should take to make the transition to its next state. The training data is sampled from robot random move video (xf, aj ... . af)), ), in which the robot moves around in the scene but does not specifically attempt the task. So the full objective of A-CycleGAN is:
LAC-C = LGAN + Lauz + XLeyer
(8)
LAC-1 || ([T a,,l?
The two parts of A-CycleGAN share the network structure of encoding layers of generator F, as depicted in Figure [ During training, they are trained parallel. Through the shared network, the gradient update of the inverse dynamic model would guide the CycleGAN to retain the action- awareness information in the translated images, which is critical for ensuring the translated data is useful for the subsequent meta- meta-imitation learning.
After A- A-CycleGAN training, the human images could be translated into robot domain, and the gen- erated robot images are used to predict the corresponding actions through inverse dynamic model. These pairs of robot images and actions could be the training data for meta-policy learning. Here we refer to the generated robot images as Imagined Robot Images j", the sequences of ø" as Imagined Robot Video d', and the corresponding predicted actions as Imagined Robot Action â". Considering the quality of the translated images and the high- digg--dmension property, in our work we do not directly employ 5" for the following meta-learning but rather the latent states é" outputted from the encoding layers of generator F. Here we refer 2" as Imagined Latent States and the sequences of 2n as Imag- ined Latent Video i (as stated in Figure I.. Latent states are domain-agnostic and provide a more stable representation of image demonstrations. Our empirical results verified that the meta-policy achieved better performance based on S" than ÿ".
4.2 ADAPTIVEE META-IMITATION IMITATION LEARNING
Through Imagined Robot Video d' and Imagined Robot Action a", our algorithm can work on the imagined data and does not require dataset of real robot demonstration during training, which would greatly reduce the burden on kinesthetic teaching or teleoperation.
However, we can not blindly take these data as a precise input to the model, as the quality of the translated images produced by the generative model in different scenes varies, and some may pro- duce artifacts. Thus we suggest evaluating the data quality to receive the learning worth adaptively. In practice, :" are encoded by 5" translated through the human demonstration, so we evaluate based on the quality of i"". With the principle that the imagined video should be as clear and contin- uous as the real video, we adopt the Brenner gradient to detect edges in the image (Ali & Mahmood, 2018) and the structural similarity index measure (SSIM) assess the continuity of adjacent images in the video (Wang et al., 2004). Mathematically. the Brenner gradient Bre(ū") computes the first difference between pixel and its neighbor with the horizomtal and vertical distance of 2. The SSIM considers comparison measurements of luminance, contrast and structure between the two images of of Di+1:
5

---

Published as a conference paper at ICLR 2022
SSIM (07,011)) = (2 + +a)) g2 + +02) (ug: + fb +aa (oz)
(9)
+ 02 Dit1
+ +02))
where Df is the image at step & in d" Mgr is the average of the image. of is the variance. og 84+ is the covariance of two images. and 02 are two variables to stabilize the division with a weak denominator. SSIM index a decimal value between- - and I, with 1 being reached only when two identical sets of data present. We normalize both the Brenner gradient and the SSIM to the range of O to 1. In this way, the quality the image is denoted as Bquai
Pqua (vE) = Bre(ū() SSIM D[+1).
(10)
This metrie is applied adaptively adjust the learning loss of the imitation module.
As discussed in Section 3.2, we separate the 'r into two sets, fty and i Considering the quality loss of i" and predicted actions â" to the real demonstrations, we do not directly use MAML but DAML structure for meta-imitation learning and keep the inner objective Lap. in the form of ID convolutions over time during meta-training phase to lower the error induced by â". Thus, iT are the sequences of states ($f. ... . and ir includes sequences of states and actions (2ī, af. 27., az)- The meta-objective is the normal behavior cloning loss [.cC with the adaptive adjustment:
min 8,10
Bqua LB (6- ooo (ir) Peat).
(1I)
Compared to the common meta-imitation learning module discovering the visual cues to accomplish the task only through action guidance, which may result in insufficient adaptation to the new scene, our method based on the latent states and adaptive loss maintains more task-related and high-quality information to facilitate the meta-policy learning. The whole procedure is illustrated in Figure
5 EXPERIMENTS
The aims of our experimental evaluation are to answer the following questions:
1. Can our approach allow the robot to learn new skills by observing only one human demon- stration without any robot instruction during training?
2. How does our approach compare to the mainstream meta-imitation baseline that requires robot demonstrations?
3. In A-CycleGAN, what is the impact of coupled training the generative model and the in- verse dynamic model?
4. How important is it that the meta- intta-mmtatiion module takes the latent states as input?
5. What benefits do gain from the adaptive loss?
Our task is challenging: it lacks a significant amount of accurate information about the skills throughout the training. Especially the latent states and the corresponding actions used for meta- policy training are all estimated. In our evaluation, we design two robot manipulation skills: shape- drawing and pushing. These two skills are also challenging: The shape-drawing skill needs the robot to follow the designated shape lines accurately, which has not appeared in prior meta- meta-learning exper- iments. The pushing skill needs robot to push the target object to the goal under the new distractor (different from demonstrations) in the meta-test phase, which would be more effective in verifying the generalization of the meta-policy. 2018b) in which the robot only needs to push the target away under the same distractor in demon- strations. To our knowledge, we are the first to propose the meta-imitation learning method using only human video demonstrations during meta-tran training (referred to MILV). Consequently, compare our method with the meta-imitation learning baseline DAML which could learn the new task from single video but require robot demonstration during meta-training. We also compare with our ablations to understand the complementary performance of each module, which includes decoupled training of the generative model and the inverse dynamic model (referred as DeGI); di- rectly feeding the meta-imitation module with the imagined robot images (FeedImg), and updating the meta-imi veta-imitation module with the stable loss (WStable).
is also harder than the assignment in DAML (Yu et al.)

---

Published as a conference paper at ICLR 2022
Simulation Fasironment
Real-werld Eavtiromment
Chau
Source Domain
La Borar
Target Domain
distraciie
Figure 2: Examples of simulated drawing (left), pushing (middle), and real-world pushing (right) tasks. The top row shows the source domain demonstration, while the bottom shows policy execu- tion in target domain.
Simulated Pushing Objects
Real-world Pushing Objects
Training
Testing
Training
Testing
Figure 3: All of the objects used for training and evaluation for pushing task.
5.1 SIMULATED TASKS
To facilitate the evaluation, we first consider performing the method entirely in the simulated en- vironment where one robot learns skills from another. As illustrated in Figure [ The experiment involves the shape-drawing task, depicting the shape following the trajectory instructions; object pushing task, pushing the specified object into the target amid one distractor. We design the environ- ment starting from the OpenAl Gym environment (Brockman et al. 20166), based on the simulated MuloCo physics engine (Plappert et
2018 Todorov al. 2012).
The demonstrator is a 7-DF Sawyer robot (source domain) with a two-finger gripper, and the learner a 7-DoF Fetch robot (target domain) with the different appearance. The camera records sequences of 128 x 128 RGB pixels images from a different perspective. The policy will control the grip- per incremental movement. For the drawing task, the goal to depict the shape according to the red trajectory indication with random start position. We selected 11 different shapes, including triangle, circle, and square, et al. with various sizes for meta-training of 1,320 Sawyer robot demonstrations (source domain), an equal number of Fetch robot random videos (target domain), and 66 Sawyer- Fetch paired videos. For DAML,, Fetch robot random videos replaced by the equal number of demonstrations corresponding to the Sawyer (including images and actions), and does not require paired data. For evaluation, we used 12 new shapes including scale and rotation variants of shapes in training set, and unseen new shapes. See Appendix A for more details about the experimental setup and choices of hyperparameters. At test time, the Fetch robot receives one video demonstration per task and evaluates 50 trials. Each trial is 50 steps and we define success when the longest common subsequence length (Bergroth et al.) 2000) between the shape drawn by the robot and the real path are longer than the threshold.
The training set consists
As for the pushing task, the goal is to push particular object with random start position into the random red target amid the other distractor. The experimental objects have a wide range of shapes, sizes, frictions and masses. Note that the target object in demonstrations is the same in scenes for the adaptation policy, but the distractor is not always consistent. There are objects for the meta- training and 8 for evaluation, including tetrahedron, kettle, pawn, and 1,000 Sawyer demonstrations, an equal number of Fetch random videos, and 60 Sawyer-Fetch paired videos. For DAML, the Fetch random videos replaced by the equal number of demonstrations (including images and actions) and it does not require paired data. All of the objects are shown in Figure 3
The datasets contain
The testing result rendered in Figure 4 and the success rates of all experiments are summarized in Table I which confirms that proposed method achieves compelling performance in both tasks.
7

---

Published as a conference paper at ICLR 2022
Pollcy
Draming
Pushing
Figure 4: All simulated testing visualization results. From left to right: demonstration from the source domain, the policy execution target domain. Note that the distractor is inconsistent for demo and executed scenes in pushing tasks.
Table 1: One-shot success rate in all experiments
Method
Simulated Drawing Simulated Pushing
Real-World Pushing
DAML
43.3%
67.3%
58.8 %
DeGI
24.2%
58.5%
32. 1
Feedlmg
27.1
57.6%
38.1% 1%
WStable
40.0%
64.6%
51.3% 56.3%
MILV((Our)
46.3% %
69.2 %
Simulated Draning
Starolated Pushring
Demonstartion
DeGI
Figure 5: Translated images comparison on the simulated drawing and pushing task.
In simulated drawing task, all methods have the success rate under 50% and are not very high. The main reason that the image observation not efficient enough the robot state, such as joint angles, and the task is also tough one to complete. In this condition, Ours (46.3%) has the similar performance with DAML (43.3%), even better. The success rate of WStable (40.0%) slightly lower than ours, indicating that the quality evaluation of the data contributes to the accuracy. DeGI (24.2%) has only half of ours, which means the A-CycleGAN critical in our methods in translating human videos into the effective action- awareness robot demonstration data. As shown in Figure the translated images obtained by the coupled model (Ours) concentrate more on reconstructing the robotic arm motion and objects in the scene forcing to preserve information required for the meta- imitation module. On the contrary, the images outputs by the decoupled model (DeGI) have no obvious preference and it performs inferior ours. FeedImg (27.1%) also performs poorly, which means the imagined latent states are more useful features than the imagined robot images for the meta-policy training.
In the simulated pushing task, the whole performance is higher compared to the shape-drawing task. Mainly because the pushing task only cares about whether the goal is the target area, no matter what kind of path the robot experienced. Our method (69.2%) performs slightly superior than the
Table 2: Breakdown of the failure modes
Method Identify Objects
Control
DAML
76.9%
23.1% %
FeedImg
48.9%
51.1%
MILV(Ours)
43.4%
56.6%
8

---

Published as a conference paper at ICLR 2022
Demonsication
Policy Execation
Figure 6: The real-world testing visualization results.
DAML (67.3%). DeGl (58.5%) and FeedImg (57.6%) worse than WStable (64.6%), which also indicates that A-CycleGAN and the latent states for meta-training are the key modules in our method, and the adaptive loss further improve the performance through high-quality data selection.
In order to further understand the effect of our method, we count the failure mode and its proportion in Table 2 which includes incorrect object identification and control failures. It is apparent that algorithms with image imputs, such as DAML. FeedImg, have greater likelihood of misidentify- ing objects than ours, implying that they share the critical flaw: meta-policy supervised only by executed action may be unable to comprehend the correct visual information in the scene, while the latent states learned by A-CycleGAN incorporate more visual information except corresponding to the actions. That means our method could be scaled to broader distributions of tasks. We can also see that compared to DAML, our method has higher control failures (56.6% vs. 23.1%). For control accuracy, we could improve the performance by discovering more features to assist domain alignment in the next-step work.
5.2 REAL.-WORKD PUSHING
We designed the real-world pushing environment using a URS robotic arm and RGB camera. The goal is to push the correct object to the pink target area. The sample videos are shown in Figure 2] Images taken from the human and rohot domains have unique perspectives. For meta-training, collected dataset with 11 objects, consisting of 550 human demonstrations, 550 robot random videos and 110 human-robot paired videos. For DAML, random videos are replaced by 550 URS demonstrations corresponding to humans (include images and actions). During the evaluation, provided 8 novel objects. The task is considered successful if the robot pushes the target object within the center of pink region. We tested each of the objects 20 times.
The testing results in the third column of Table D report the success rate of our real-world exper- iments. The whole success rate (lower than 60%) is lower than in simulation environment, which is mainly influenced by the observation and action noise in real-world. Our method (56.3%) still achieves the comparable results to DAML (58.8%). The performance of DeGI(32.19 1%) and FeedImg (38. (3))%% is down greatly and again verifies the two modules of our method are indispensable. Figure 6 shows the real-world testing visualization results. Additional experiments testing results refer to Appendix B
CONCLUSIONS
We provide meta-imitation approach that enables robots to learn the meta-policy by only using video demonstrations from humans and have the ability of one-shot learning new tasks from watch- ing single video. Our approach automatically translates human videos into robot demonstrations using new-designed generative model A-CycleGAN and train the meta-policy in the imagined compact latent space with the proposed adaptive loss. Empirical results suggest that the learned meta-policy achieves compelling results the baselines in one-shot vision task Icarning both in simulation and real-world. Our approach relaxes the demands for training data, requiring only hu- man video demonstrations and gives try to the natural imitation learning of robots.
In the future, we hope to improve our task performance by incorporating reinforcement learning techniques for online trial and error. Since our method requires some Nomain-alignee data, expect that the content translation model, such as UNIT (Liu et al.) (017). would likely extend our method to adopt human robot demonstration data from different sources.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We'd like to begin by acknowledging the students and collaborators at CASIA who gave valuable feedback which made the final paper much stronger. In particular, we thank Weiyan Guo, Congjia Su, Junhang Wei, Shaowei Cui and Yaozhong Cao. This work was supported by grants from the National Key R&D Program of China(grant 2019YFB1311901
REFERENCES
Usman Ali and Muhammad Tariq Mahmood. Analysis of blur measure operators for single image blur segmentation. Applied Sciences, 8(5):807, 2018.
Lasse Bergroth, Harri Hakonen, and Timo Raita. A survey of longest common subsequence algo- rithms. In Proceedings Seventh International Symposium String Processing and Information Retrieval. SPIRE 2000, pp. 39-48. IEEE, 2000.
Alessandro Bonardi, Stephen James, and Andrew Davison. Learning one-shot imitation from humans without humans. IEEE Robotics and Automation Letters, 5(2):3533-3539, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:2011.05970. 2020.
Yan Duan, Marcin Andrychowicz. Bradly C Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. arXiv preprint arXiv:1703.07326, 2017.
Sarah Elliott, Zhe Xu, and Maya Cakmak. Learning generalizable surface cleaning actions from demonstration. In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 993-999. IEFE, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine One-shot visual imita- tion learning via meta-learning. In Conference on Robot Learning, pp. 357-368 PMLR, 2017b.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hicrarchical bayes. arXiv preprint arXiv: 08930, 2018.
De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, and Juan Carlos Niebles. Continuous relaxation of symbolic planner for one-shot imitation learning. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2635-2642. IEEE, 2019.
Stephen James, Michael Bloesch, and Andrew J Davison. Task-embedded control networks for few-shot imitation learning. In Conference on Robot Learning, pp. 783-795. PMLR, 2018.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):421 1436, 2018.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, 2017.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference Robotics and Automation (ICRA), pp. 1118-1122 125. IEFF, 2018
10

---

Published as a conference paper at ICLR 2022
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: crowdsourcing platform for robotic skill learning through imitation. In Conference Robot Learning, pp. 879 893. PMLR, 2018.
Pascal Molenberghs, Ross Cunnington, and Jason B Mattingley. Is the mirror neuron system in- volved in imitation? a short review and meta-analysis. Neuroscience & biobehavioral reviews, 33 (7):975-980, 2009.
Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3): 263-279, 2013.
Anh Nguyen, Dimitrios Kanoulas, Luca Muratore, Darwin G Caldwell, and Nikos Tsagarakis. Translating videos to commands for robotic manipulation with deep recurrent neural networks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3782-3788. IEEE, 2018.
Leo Pauly, Wisdom C Agboh, David C Hogg, and Raul Fuentes. O2a: One-shot observational learning with action vectors. Frontiers in Robotics and AI, 8, 2021.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint 18f.:08802.99444 2018.
Matteo Ragaglia et al. Robot learning from demonstrations: Emulation learning in environments with moving obstacles. Robotics and autonomous systems, 101:45-56, 2018.
Harish Ravichandar, Athanasios Polydoros. Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
Thiusius Rajeeth Savarimuthu, Anders Glent Buch, Christian Schlette, Nils Wantia, Jürgen Roß- mann, David Martínez, Guillem Alenyà, Carme Torras, Ales Ude, Bojan Nemec, et al. Teaching robot the semantics of assembly tasks. IEEE Transactions Systems, Man, and Cybernetics: Systems, 48(5):670-692, 2017.
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Rein- forcement learning with videos: Combining offline observations with interaction. arXiv preprint MrXiv:2011.06507 2020.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint 16X1v:1612.06699, 2016.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concep2rooott: Learn- ing manipulation concepts from instructions and human demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020a.
Quanquan Shao, Jin Qi, Jin Ma, Yi Fang, Weiming Wang, and Jie Hu. Object detection-based one-shot imitation learning with an rgb-d camera. Applied Sciences, 10(3):803, 2020h.
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
Maximilian Sieb and Katerina Fragkiadaki. Data dreaming for object detection: Learning object- centric state representations for visual imitation. 2018 JEEE-RAS 18th International Confer- on Humanoid Robots (Fumanoidds, pp. 1-9. IEFF, 2018
11

---

Published as a conference paper at ICLR 2022
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint arXiv: 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Daniel Tranel, David Kemmerer, Ralph Adolphs, Hanna Damasio, and Antonio R Damasio. Neural correlates of conceptual knowledge for actions. Cognitive neuropsychology. 20(-6))409 432, 2003.
Chen Wang, Rui Wang, Danfei Xu, Ajay Mandlekar, Li Fei-Fei, and Silvio Savarese. Generaliza- tion through hand-eye coordination: An action space for learning spatially-invariant visuomotor control. arXiv preprint arXiv:2103.00375. 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj. Samarth Sinha, and Animesh Garg. Learning by watching: Physical imitation of manipulation skills from human videos. arXiv preprint arXiv:2101.0 007241, 2021.
Shuo Yang, Wei Zhang, Weizhi Lu, Hesheng Wang, and Yibin Li. Cross-context visual imitation learning from demonstrations, In 2020 IEEE International Conference on Robotics and Automa- tion (ICRA), pp. 5467-5473, IEEE, 2020.
Yezhou Yang, Yi Li, Cornelia Fermuller, and Yiannis Aloimonos. Robot learning manipulation action plans by" watching" unconstrained videos from the world wide web, Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
Tianhe Yu, Pieter Abbeel, Sergey Levine, and Chelsea Finn. One-shot hierarchical imitation learning of compound visuomotor tasks. arXiv preprint arXiv: 1810. 1043, 2018a.
Tianhe Yu, Chelsea Finn, Sudeep Dasari, Annie Xie, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain domain-adaptive meta-learning. In Pro- ceedings of Robotics: Science and Systems, 2018b.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5628-5635. IEEE, 2018.
Yuxiang Zhou, Yusuf Aytar, and Konstantinos Bousmalis. Manipulator-nndependen representations for visual imitation. arXiv preprint a1X1::2103.09016, 2021.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A ADDITIONAL EXPERIMENTAL DETAILS
We provide additional experimental details for all experiments, including data collection, evaluation, and training hyperparameters.
12

---

Published as a conference paper at ICLR 2022
2
THE
Figure 7: The display of all training and testing shapes for the drawing task. For testing shapes, the first four are scale variants of the training set, the middle three are rotation variants of the training set, and the last five are novel unseen shapes.
A.1 EXPERIMENTAL SETUP
In the experiment, the imput to the policy is only sequences of 128 x 128 RGB pixels images, without any information the robot joint or end-effector. The policy output is the incremental movement of the robot end-effector in 3D space, as (Ar, Ay, Az). where Ar E [0, 5cm], Ay E [0, Scm], and Az = Oem (z is set as constant 0.3cm above the table). For shape-drawing tasks, all shapes to be depicted are shown in Figure 7 The policy roll-out is considered successful if the longest common subsequence lengths of the passed trajectory and the real shape are greater than 30 (60% of the total length). For the pushing task, as illustrated in Figure 4 only the target object is identical across the robot demonstrations and the actual execution scenes; the distractor not necessarily the same. The push is considered successful if the target object lands on the target area at the last time step. In simulation, the target area the red rectangle area(as indicated in Figure 2 with 12cm width and 12cm height. In real-world, the target area is the pink rectangle area(as indicated in Figure 2 with 12cm width and 12cm height. The expert policy inputs include the position and size of the shape for the drawing task, well as the position of the target object, the robot end-effector and the target for the pushing task. We develop the corresponding solutions for each task using artificially formulated rules.
The policy is fed just the sequence of 128 x 128 RGB pixels images from a RealSense D455 camera, and the length of each trajectory is limited to 50 steps. A trial is considered success if the center of the target object lands on the target within 50 step. To collect demonstration data for one task, we randomly assign target location for each trial and select target object and random distractor a random location in front of the robot. The robot demonstrations provided to DAML for training generated by rules.
A.2 HYPERPARAMETER FOR NETWORKS
For all experiments, our image translation model uses a convolutional neural network where the en- coder uses 3 convolutional layers with 7x7, 7, 32 3x3, 64 3x3 filters respectively, followed by four residual modules (each includes two convolutional layers), and the decoder contains deconvolutional layers restoring the image the original size. The discriminator Dx and Dy uses 3 strided volutions and non-strided convolutions, which aim to classify whether overlapping image patches are real or fake. Its architecture can work on images of any size in fully convolutional fashion (Zhu et al. 2017). We set À = 10 and the batch size is in all experiments. These networks are traindd from scratch with a learning rate of 0.0002.
The inverse controller takes latent states from the generator F of CycleGAN as the input, using a strided convolution and a non-strided convolution with 16 5x5 filters, followed by spatial softmax and 2 fully-connected layers with 200 hidden units. The training loss is the mean squared error between the predicted actions and the ground truth robot command. Throughout the training, the learning rate remains constant at 0.001.
The meta-imitation module also takes latent states the translation model as the input, connecting with 2 strided convolutions and non-strided convolution with 16 5x5 filters, followed by spatial softmax and fully-connected layers with 200 hidden units. The last layer applies the two-head architecture, with one head used for the pre-update demonstration using fully-connected layer and one head used for the post-upun post-update policy consisting of three layers of temporal convolutions. the first with x0x1 1 filters and the third with 1x1 filters. Our ablation experiments with image input
13

---

Published as a conference paper at ICLR 2022
DAML
A
Ours
A
Figure The visualization testing results of our method and DAML in the shape-drawing task.
take human and translated images as inputs. The network structure of the meta-imitation module comprises 6 convolutional layers, as well as the same fully-connected layers and others as described above. The policy used I meta-gradient update with the step size 0.01.
B ADDITIONAL EXPERIMENTAL RESULTS
B. SIMULATED DRAWING
The visualization testing results of our method and DAML in the shape-drawing task are shown in Figure 8 with blue trajectory representing the groundtruth and red representing the policy execu- tion. It can be seen that both methods effective for the majority of the novel shapes. However, for some shape, such as the first row fifth and the second row first, our method do not perfectly fit the groundtruth but close to while the DAML is considerably different from it, preferring draw these two as arrow shapes in the training dataset. The same condition appears in the second row second which is likely to be drawn square. As result, suggest that DAML falls short of generalization for new shapes, whereas errors introduced by the translation and inverse dynamic model in our technique prevents model over-fitting to certain extent.
B.2 REAL- WORLD PUSHING
Failures in real-world pushing tasks are presented in Figure 9 Due to the more complex environ- ment and misaligned paired data, the task is more likely to fail in real-world conditions than in the simulation settings.
B.3 ADDITIONAL REAL- WORLD PUSHING EXPERIMENT
Table 3: Results on real-world pushing forward task
Failure Modes
Method
Success Rate
Identity Object Control
DAML MILV(Ours)
81.3%
86.7%
13.3%
77.5%
63.9%
36.1%
To further validate the performance of our method compared with DAML, we run an additional real-world pushing experiment designed in the DAMI. paper (Yu et al. 201 As illustrated in
14

---

Published as a conference paper at ICLR 2022
Demonstration
Incorrect Identification
Control Failure
Figure 9: Instances of failure in real-world pushing tasks.
Demonstratton
Policy Execation
distrur
detricc
Figure 10: The demonstration and testing results on pushing forward task.
Figure 10 the goal of task is just to push the target object forward for more than 5cm amid the same distractor that in the demonstration. Apparently, this task is much easier than the pushing task that we designed our work. Other experimental settings are the same as ours, including the type of objects, the perspectives of human and robot, the number of training data and
Figure 10 shows the visualization results. As demonstrated in Table 3 our testing success rate is 77.5% and DAML is 81.3%, which reveals the promising performance of method. We count the failure mode and again find our method provides a more accurate identification of the target object.
B.4 ADDITIONAL TRANSLATED RESULTS
Here provide some qualitative examples of the translated videos using our translation model A- CycleGAN in Figure II
The odd lines are demonstrations from the source/human domain, even are translated videos.
B.5 THE CORRELATION BETWEEN THE ADAPTIVE METRIC AND THE TRANSLATION ERROR
Here measure the actual translation enror by computing the SSIM of the paired corresponding video and analyze the correlation between the adaptive metric 3quo and e on simulated pushing and real-world pushing tasks.
As shown in Figure 12 the 8rua is positive correlated with e, which consistent with our intuition.
B.6 THE EFFECT THE NUMBER THE PAIRED VIDEO
We evaluate method range of paired video numbers and show the test success rate simulated pushing task Figure [3] Apparently, the increase in the number paired data means that the increase of supervised data for translation model improve the test success rate. When
15

---

Published as a conference paper at ICLR 2022
Simulated Drawing
A
A
A
C
Simulated Pushing
&
Real- World Pushing
Real-World Pushing Forward
Figure 11: The examples of the translated videos on all tasks.
16

---

Published as a conference paper at ICLR 2022
Simulated Pushing Task
Real-world Pushing Task
0.85
1.00
Flineg nith Bscar funshie
Fiiting vōh lhsar fuaction
Original data
Original dats
0.86
0.80
0.72
Bqus
Bque
0.75
0.58
0.70
0.44
0.80
0.84
0.60
0.64
0.68
SSIM
SSIM
Figure 12: The correlation between the adaptive metric Sque and translation error c.
Simulated Pushing Task
Test success rate (%)
66
64
62-
60
20
30
70
Total number of paired videos
Figure 13: The success rate on simulated pushing task a function of the paired video numbers.
the number ups to more than 60, the growth of success rate substantially slows, which is why the amount of data utilized in experiment is 60.
17


Reviews:
[{"review_id": "q93u8q4uwas", "reviewer": "Reviewer_EwC5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This method extends the previously proposed DAML algorithm, requiring only human video demonstrations (and also a small amount of paired data). This is appealing especially when it’s difficult to collect large amounts of demonstrations in the robot domain, e.g., through teleoperation. The resulting method performs comparably to the DAML algorithm, which requires more robot demonstration data.\n\n- The experiments study the relevant ablations of the method, including using a decoupled CycleGAN and inverse dynamics model, using the translated images instead of the latent representation, and using a non-adaptive loss in DAML. The results show that all of the choices in their method make it stronger.\n\nWeaknesses:\n- The experiments lack comparisons to other methods besides DAML, which operates on a different set of assumptions. Since the success rates for all methods are quite low, it would be good to understand where the upper bound is. The method proposed in [1], like DAML, requires robot demonstrations as well, but may perform better than DAML. Another potential comparison is AVID [2]. Specifically, we can use all of the random robot data to train a visual dynamics model, and the rest of the data to train CycleGAN. One difference is that AVID solves multi-stage tasks and assumes the beginning of each stage is specified by the user; their reward function for planning is then defined by learned success classifiers. Instead, we could derive a simple dense reward function from the translated video, e.g., to match the translated video frame-by-frame or just to match the final frame as measured by the MSE. \n\n- The SSIM and Brenner gradients are heuristic measures of the quality of the translated videos, and do not directly measure how accurate they are with respect to the true corresponding video in the robot domain. Since the meta-IL component operates on the latent representation and not the generated images themselves, it is an even more indirect measure. However, the non-adaptive loss does lead to worse results, so it empirically seems to work better. I’m curious how well these two heuristics actually correlate with the translation error. This could be evaluated on the small dataset of paired videos -- compute the MSE of the translated video and true robot video pair, and then measure the correlation with \\beta_qua.\n\n- It would also be nice to include some qualitative examples of the translated videos.\n\nOther questions:\n- The method still requires a small amount of paired human and robot data, which could consist of random actions (in their experiments, this is <= 100 trajectories). How does varying this number affect the performance of the method?\n\n- The success rates for all methods are quite low. It’s also surprising that DAML, which leverages paired demonstrations, performs similarly to/worse than A-CycleGan + DAML, which doesn’t. How does the meta-policy learned by the different methods perform on the training shapes? \n\n- Compared to DAML, the main mode of failure is due to control, i.e., not executing the right sequence of actions to complete the task, which may be explained by the prediction error in the inverse dynamics model. The chosen action space in the experiments is 3D - I’m curious how well it scales to higher dimensional action spaces, e.g., if the tasks required joint space control. \n\n\n[1] Dasari et al. Transformers for One-Shot Visual Imitation. CoRL 2020.\n\n[2] Smith et al. AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos. RSS 2020.\n", "summary_of_the_paper": "This paper studies the meta-imitation learning from human videos problem, which aims to learn a new task by watching a few demonstrations of a human performing the task. While DAML [1], one of the relevant prior approaches to this problem, requires paired demonstrations in the human domain and robot domain for training, this approach only requires human videos and a small amount of paired data. The approach proposed in this paper translates human videos into the robot domain with a CycleGAN. Simultaneously, an inverse dynamics model is trained in the robot’s observation domain with randomly collected robot data to recover the actions from the translated demonstrations. Because the quality of the translations can be quite varied, higher-quality translations, measured based on the SSIM and Brenner gradient, are prioritized over lower-quality ones in the meta-imitation learning procedure, implemented by the DAML algorithm.", "summary_of_the_review": "The paper tackles a challenging and compelling problem: meta-imitation learning of human demos with a small amount of paired robot demos. However, the experiments lack relevant comparisons to contextualize the significance of the proposed method. The heuristics used in the adaptive loss could also be better analyzed, e.g., by looking at their correlation with the MSE between the translated videos and true robot video.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UHHvt-7lrg6", "reviewer": "Reviewer_tm35", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The task is very relevant in the robotics community\n+ The paper is well structured\n+ The experimental analysis is convincing\n\n- The reviewer finds some part a bit unclear. Here is a tentative summary of the main concerns:\n   * Not fully clear to the reviewer how supervision works in A-CycleGAN (on the robot side)\n   * Not fully clear to the reviewer how information on action and on environment is disentangled and exploited (Sec. 4.1)\n   * The reviewer fails in gathering an intuition about the interplay between the different modules in the architecture. In particular, the Inverse \n      Dynamic Model lacks details, and the Meta-Imitation Learning, one of the core parts of the work, has a description which is a but \n      compressed. ", "summary_of_the_paper": "The paper considers the problem of robot imitation learning from video demonstration, without the need of accessing to an explicit representation of the action of the demonstrator. \nThe authors introduce an approach for meta-imitation learning that aims to traslate human videos to robot demonstrations from a single video demonstration. The correspondence between human and robot is learned end-to-end exploiting a generative architectures, A-CycleGAN, extension of CycleGAN. The method is experimentally assessed on different visual tasks.", "summary_of_the_review": "I find the topic very interesting and the video-based approach is certainly among the most relevant in robotics. However, I find the clarity of the presentation could be improved to fully appreciate the essence of the approach ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Pk9EQV5TM8K", "reviewer": "Reviewer_337g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- This paper tackles a very challenging problem and accordingly presents an elaborately designed approach that brings together components from several state-of-the-art methods from slightly different areas. As a result, the overall approach is pretty novel.\n\nWeaknesses\n\n- The motivation of this paper is to reduce data collection complexity by only requiring human demonstrations, since robot demonstrations may require kinematic teaching / teleoperation which can be difficult to collect. However, while the proposed method doesn’t require robot demonstrations, it does require pairwise human and robot videos for training the CycleGAN component. My assumption is that this is also difficult to collect in practice, so I think the value of the proposed method is significantly diminished. \n\n- In terms of the experiments, three tasks are presented, with two in simulation, where demonstrations are from a Fetch robot and the learner is a Sawyer robot, and one task in the real world that learns from human videos. While the simulated robot-to-robot experiments demonstrate some aspects of the algorithm, they are less powerful results. With only one task in the target setup, I feel the overall experimental results are not sufficiently rigorous. \n\n- In addition, the final success rate achieved by the algorithm for the real-world pushing task is 56.3%, which is a lot lower than what DAML achieves as shown in the DAML paper (88.9%) on a highly similar task. While in the present paper, the proposed algorithm matches DAML’s performance (56.3 vs 58.8%), both success rates are low. I wonder if there are any structural limitations of the proposed approach that prevents it from achieving a higher rate. Consider running experiments with more training data; would the proposed algorithm still be able to match DAML’s performance?\n\n- The writing of the paper needs significant polishing.", "summary_of_the_paper": "This paper studies the problem of meta imitation learning of robot control policies from only video demonstrations of humans performing the tasks. The goal is to make data collection simpler by not requiring robot demonstrations which can be expensive to acquire. The method proposed by the authors combines multiple components. Specifically, a CycleGAN which translates human demo videos into robot demo videos, an Inverse Dynamics Model which infers robot action from consecutive video frames, a method to weigh the training loss according to the quality of translated images, and finally the same learning algorithm from DAML is used. ", "summary_of_the_review": "This paper studies a very challenging problem and presents a novel method, but the experiments are not convincing enough.\n\n----\n\nUPDATED the overall recommendation after authors' response. see details in thread.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1lhfWhx4Uyp", "reviewer": "Reviewer_9GZg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Paper is well-written.\n- The paper only uses human video demonstration and no robot demo/states and actions to learn a task seems unique to me.\n- Paper situates the problem very well and presents good series of prior work to solve the problem addressed in the paper.\n- Good range of experiments\n\nWeakness:\n- In the introduction section, the authors mention about challenging tasks, could you please explain in one line what is the difficulty of the tasks that you are attempting the robot to learn?\n- It will be good add a description of \"imagined videos\" that has been reference in the paper multiple times. Maybe add in the Method section. \n- I encourage the authors to submit a video of experiment results for both simulated tasks and real-world pushing task.\n- A question asked in the Introduction section para 2, is not clear to me. Could you please re-write the sentence with more clarity?", "summary_of_the_paper": "The paper talks about solving an interesting problem of meta-imitaion learning from watching human video demonstration. The paper proposes a A-CycleGAN approach to learn a latent representation for human-robot video correspondence and a self-adaptive meta imitation learning under imagined latent space to evaluate the quality of translated data. The proposed methods in the paper achieves comparable performance to the state-of-the-art method.", "summary_of_the_review": "Based on paper methodology contributions and above merits (explained in Main Review section).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper working on the difficult problem of learning from video demonstration. The authors provided convincing experimental solutions for visual representation, domain adaptation, and imitation. It would be a nice ICLR paper.
Decision: Accept (Poster)