XIRL: Cross-embodiment Inverse Reinforcement Learning
Authors: Kevin Zakka
Andy Zeng
Pete Florence
Jonathan Tompson
Jeannette Bohg
Debidatta Dwibedi
Venue: CoRL2021 Oral (2021)
Keywords: inverse reinforcement learning
imitation learning
self-supervised learning

Abstract: We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -- shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings typically required alignment with a reference trajectory, which may be difficult to acquire under stark embodiment differences. We show empirically that if the embeddings are aware of task-progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. Additionally, when transferring real-world human demonstrations to a simulated robot, we find that XIRL is more sample efficient than current best methods.

XIRL:
Cross-embodimenn Inverse Reinforcement Learning
Kevin Zakka! Andy Zeng', Pete Florence", .Jnaathan Tompson", Jeannette Bohg', and Debidatta Dwibedi Stanforr University, *Robotics at Google, UC Berkeley
Abstract: We investigate the visual ennodimmodd imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments shape, actions, and-effector dynamics, etc. In this work, we demonstrate that is possible to automatically discover and learn vision-basee -boood reward functions from erbsdemmodi demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross- embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle- consistency constraints learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing re- wards from self superviidd embeddings typically required alignment with reference tra- jectory. which may be difficult to acquire under stark embodiment differences. We show empirically that if the embeddings distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes entirely new embodiments. Additionally, when transferring real-world human demon- strations to a simulated robot, find that XIRL is more sample efficient than current best methods. Qualitative results, code, and datasets are available at hups://x-irl .it.huuu...
aware of task progress, simply taking the negative
Keywords: inverse reinforcement learning, imitation learning, self-supervised learning
1 Introduction
The ability to learn new tasks from third-person demonstrations holds the potential to enable robots to leverage the vast quantities of tutorial videos that can be gleaned from the world-wide web (c.g.. YouTube videos). However, distilling diverse and unstructured videos into motor skills with vision-based policies can be daunting in different environments, but also with different experts that may use different tools, objects, or strategies to perform the same task. Perhaps most critically, there often exists a clear embodiment gap between the human expert demonstrator, and the robot hardware that executes the learned policies. One approach to close this gap is to learn a mapping between the human and robot embodiment [1], which is a non-trivial intermediate problem in itself
the videos themselves
often not only captured from different camera viewpoints
Despite considerable progress in learning policies with paired observations and actions (e.g., collected via telcoperation) [2, 3], much less work has been done in getting robots to learn policies from tasks defined only by third-person observations of demonstrations [4,5, 5, 6). This is a surprisingly challenging problem, as different embodiments are likely use unique strategies that suit them and allow them to make progress on a given task. For example, asked to "place five pens into a cup", a human hand is likely to scoop up all pens before slipping them into the cup, whereas two-fingered gripper might instead need to pick and place each pen individually. Both strategies complete the task, but generate different state-action trajectories. In this setting, it may be difficult to acquire labeled frame-to-frant correspondences between expert demonstration videos and learned embodiments [6]. particularly across multitude of embodiments or experts. The approach we investigate instead is whether we can successfully learn tasks through a learned notion of task progress that is invariant to the embodiment performing the task.
"Work done intern Google.
Sth Conference Robot Leaminng 2021), London, UK.

---

Demtce Tratedieeee
Denroe of Tichedineet
Deroo Firbudiment
Reward
Temporal cedde-Consstenng Leaning
Tusk Progress
Ensndor
apous
Beward Function
Eneodes
Eneoder
Encoder
Goal
Nacnzoc stgreesi acrin dema muzlip eododinero
Using TOC we learn progress aware, embodiment- agnostic encoder
Distance from a single goal state in the TOC embedding space used reward to train policies for multi-step
tasks on novel embodiments
Figure Cross embodinent Inverse Reinforcement Learning. learn embolimment invariant visual representations from ollliee video demonstra- tions (stick agents on the left) using TCC [7]. then use the trained encoder generate emhodimenti invariant visual reward functions that can be used learn policies new emboliments (gripper the righ) with reinforcemmett leaminng
In this work, we propose to enable agents to imitate video demonstrations of experts - including ones with different embodiments by using a task-specific, embodiment- invariant reward formulation trained via temporal cycle-consistency (TCC) [7]. We demonstrate how we can leverage an encoder trained with TCC to define dense rewards for downstream reinforcement learning (RL) policies via simple distances in the leamed embedding space. Simulated experiments across four different embodiments show that these learned rewards are capable of generalizing to new embodiments, enabling unseen agents to learn the task via rein- forcement, and surprisingly in some cases, exceeding the sample efficiency of the same agent leamed with ground truth sparse rewards. We also demonstrate the effectiveness of our approach for learning robot poli- cies using human demonstrations on the State Pusher environment from [6]. where our reward is first learned on real-world human demonstrations, then used to teach a Sawyer arm how to perform the task in simulation.
Our contributions are follows: (i) We introduce Cross eooos-embodiment Inverse Reinforcement Learning (XIRL, an effective, label-free framework for tackling cross-embodiment visual imitation learning. Our core contribution is to use self-supervisee learning third-person demonstration videos to define dense reward functions amenable for downstream reinforcement learning policies, (ii) Along with XIRL, we re- lease a cross-emben cross-embodiment imitation learning benchmark, X-MAGICAL, which features multiple simulated agents with different embodiments performing the same manipulation task, including one thousand expert demonstrations for each agent, (iii) We show that XIRL significantly outperforms alternative methods on both the X-MAGICAL benchmark and the human-to- to-robot transfer benchmark from [6]. and discuss observations, which point to interesting areas for future research, (iv) Finally, introduce werl-world dataset, X-REAL Cross-embodimees Real demonstrations), of a manipulation task performed with nine different embodiments, which can be used to evaluate eossdiembo reward learning methods".
2 Related Work
Traditional formulations of imitation learning [8, 1, 9] assume access to a corpus of expert demonstration data which includes both state and action trajectories of the expert policy. In the context of third third-person imitation learning, including when learning from expert agents with different embodiments, obtaining access to ground-trutt -round-truth actions is difficult or impossible'.
Inferring expert actions. To address this issue, several approaches either try to infer expert actions [10, 11, 12] for example by training an inverse dynamics model on agent interaction data [10] employ forward prediction on the next state to imitate the expert without direct action supervision [13]. While these methods successfully address learning from observation-only demonstrations, they either do not support skill transfer different policy embodiments at all, or they cannot take advantage of multiple embodiments in order to improve generalization to unseen policy configurations. We explicitly address these problems in this work.
Imitation via learned reward functions. In contrast to imitation via supervised methods, such as BCO [10], recent body of work [4, 5, 14, 6, 15] has focused on learning reward functions from expert video data and then training RL policies to maximize this reward. In [4], the authors combine ImageNet pre-trained visual features with an L2-norm distance reward to match policy and expert observations
For more details regarding X-REAL, please see Appendix C. For more comprehensive related work, please see Appendix A.

---

in latent feature space. In their follow-up work [5], the reward is computed in viewpoint-invarian representation that self-supervised video data. While both these methods are compelling in their use of cheap unlabeled data to learn invariant rewards, the use of a time index as a heuristic for defining weak correspondence is a constraining limitation for tasks that need to be executed at different speeds, or are not strictly monotonic (e.g., have ambiguous sub-task ordering).
[14] a dense reward learned via unsupervised learning on YouTube data and the authors make no assumption about time alignment. However, in their work, the expert and leameed policy are executed in the same domain and embodiment, an assumption we relax in our work. Framed in a multi-task learning setting, [16] propose training policies with morphologically different embodiments first on a similar set of proxy tasks, in order to learn latent space to map between domains, and then sharing skills on held-out task from one policy another. A time-index heuristic is used to define a metric reward when performing RI. training of the new task. In work, the learned embedding finds corresoondenees in a uully--nsuperviieee fashion, without the need for such striet time alignment. In [17], a small sub-set of states is human labeled for goal success and a convolutional network is then trained to detect successful task completion from image observations, where on-policy samples are used negatives for the classifier. By contrast, our learned emhedding encodes task progress in its latent representation without the use of expensive human labels.
Imitation via domain adaptation. An additional category of approaches to third-person imitation learning are those that perform domain adaptation of expert observations [18, 19, 20, 21]. For instance, in [18] translation between policy domains, which is then used to construct a reward function for model-basee RL algorithm. A similar model-free approach is proposed in [19]. In [20]. a generative model is used to predict robot high-level tub-goals conditioned on a third-person demonstration video, and a lower-level control policy is trained in task-agnostic manner. Similarly, [21] uses high level task conditioning from zero-shot transfer of expert demonstrations, but they use KL matching to perform both high and low-level imitation. In contrast these methods, the unsupervised TCC alignment in this work avoids performing explicit domain adaptation or pixel-level image translation by instead learning robust and invariant feature space in fully offline fashion.
a CycleGAN [22] architecture is used to perform pixel-level image-to-inan
Reinforcement learning with demonstrations. Recent work in refline-reifforcemen learning [6] explicitly tackles the problem of policy embodiment and domain shift. Their method, Reinforcement Learning from Videos (RLV), uses labelled collection of expert-policy state pairs in conjunction with adversarial training to learn an inverse dynamics model jointly optimized with the policy. contrast, we dense state correspondences by using supervupe avoid the limitation of collecting cycle-consistency automatically learn the corresoondecce between algorithm (i.e., TCC (7)) which uses states of two domains. We also show that this formulation improves generalization to unseen embodiments. Since the problem setup is similar to ours, we also compare to their method a baseline.
3 Approach
Our overall XIRL framework (Figure 1) addresses the crass-embodiment visual imitation problem (Section 3.1). The framework consists of first using TCC to self-supervise ceabodiment-invariien visual representations (Section 3.2), then using a novel ceaboddimen-iivvarian visual reward function (Section 3.3) to perform cross-embodiment visual imitation via reinforcement learning (Section 3.4).
3.1 Problem Formulation
Our objective is to extract an agent-invariant definition of a task, via a learned reward function, from a dataset of videos of separate agents performing the same task. In particular, we are interested in agents that may solve the task in entirely different ways due to differences in their end-effector, shapes, dynamics, etc., which we refer to as embodiment differences. For example, consider how differently vacuum gripper and parallel-jaw gripper will grasp an object result of their respective end-effectors. Such setup is quite common in robotic imitation learning, where we might have access to observation data of humans demonstrating a task, but want to teach robot to perform it. It is very likely that the way the human executes the task will diverge from how the robot would execute it.
We define a datasct of multiple agents performing the same task T as D D= U"_ D.. where D; is an agent-specific dataset containing observations of only agent performing task T. Each agent's dataset D; is a collection of videos defined as D; [u] vf.... }. where of represents the video of the jth demonstration of agent i successfully performing the task T. We would like to highlight that D only contains observation data, i.e., it does not store the actions taken by the respective agents. We use self supervised representation leaming techniques to leam task-specific representations from this dataset.
3

---

long-stic
medium-stick
medium-stick
long Jong-stick
medium-stick
00
00
0.0
-05
-05
short.stick
shortatick
short-stick
10
10
0.5
0.0
-05
-05
-10
-10
-45
05
-2.
0.0
Agent distribtioon
Debris distrbbution.
Agent Injectury randumlly sampled
Figure 2. Difference in state distributiurs of the across different embodirments performing the same lask in M-MICCL (Section 4.1).
3.2 Representation Learning
In this work, we use TCC [7] to learn tpsc-fpe representations in self-supervised way. The method has been shown to learn useful representations on videos of the same action for temporally fine-grained downstream tasks. In their paper, the authors show that TCC representations can predict frame-level task progress, such predicting how much water is in cup during pouring, without requiring any human annotations. Task progress can provide dense signals for learning a new task and we would like bake this property into our learned reward. Another advantage of TCC is that it does not require supervision for frame-level alignment (i.e., which frames in two videos correspond each other). Such frame-to-fin to-frame correspondences are required by a prior method, RLV [6], to achieve successful reinforcement learning on the considered tasks, but we would like to avoid this type of manual supervision.
We train an image encoder o, that ingests image I and returns embedding vector ((T), using TCC. TCC assumes that there exist semantic temporal correspondences between two video sequences, even though they are not labeled and the actions may be executed at different speeds. Note that in our case, the assumption holds since all sequences in our dataset originate from the same task. Furthermore, even if two different agents accomplish the task very differently, there will be a common set of states or frames that will temporally correspond. When apply the TCC loss, we compare frames of one agent performing the task with frames from another, and search for temporal similarities in how both execute the task. By performing multiple such comparisons and cycling-back (described in the next paragraph), TCC encodes task progress in its latent representation, property that is useful for learning task-speciie yet embodiment-invariann reward functions. For completeness, we describe the training technique below.
We define a dataset Dan = (u,, t2 that contains the videos all agents executing the same task T. We are able to merge the datasets of different agents since TCC does not require embodiment labels (i.e., IDs) during training. We first sample a random batch of videos and embed all their frames using the aforementionedd image encoder 6. For each video e, this results in sequence of embeddings (o{ ((((())) (e?)"...(((!""))) }}. where L, is the length of the 7th video, From this mini-batch of sequences of video frame embeddings, we choose a pair of sequences V, and Vj and compute their TCC loss. In particular, randomly sample a frame embedding from sequence Vi - say V, corresponding the 1 frame of video 1; and compute the soft neagest-n of V in sequence V, in the embedding space as follows:
L
e' || 1//1 E e
Og V
where
V7 VI
k
We then cycle-back [7] to the first sequence Vú by computing the soft-nearest neighbor of V with all the frames in V. The probability of cycling-back to the 4th frame in VÚ can be computed as:
114 1/
Ihe -1V -V-111
The expected frame index we cycle-back to is then 1hy - =K Btyk: Since we know the index of the frame that started the cycle, in this case we minimize the mean-squared (MSE) error loss between t and the closest index retrieved via soft-nearest neighbor, i.c.. 13" The loss for a single frame is thus: Li - (uly 2) Finally, we minimize the average loss L over all frames in video with all other videos in the dataset vg. defined L -Enr

---

Table I. Statistics of Demos of the X MAGICAL Emboddments
Embodiment
Mean + Std. Dev. Demo Length (no. of frames)
Long-stick Medium-stick
48.0 28.5
59.9
1.7..9
Short-stick
81.7 3.30
Gripper
110.8 39.8
3.3 Reward Function
Once the encoder Φ has been trained on demonstrations of different agents performing the same task T, we want to use it to transfer information about the task from one agent to another. We do this by leveraging @ to generate rewards via distances to goal observations in the learned embedding space. Specifically, we define the goal embedding g as the mean embedding of the last frame of all the demonstration videos in our offline dataset Danl- Concretely, g g= I
((v 1/N, where L, is the length of video U; Our reward r then is the scaled negative distance of the current state embedding to the goal embedding g i.e., r(s) - 9|.3. where ((s) is the state embedding at the current timestep and k. is a scale parameter that ensures the distances are in a range amenable for training reinforcement learning algorithms [23]. We found it effective to set K to be the average distance of the first frame's embedding to the goal embedding for all the demonstrations in the dataset. Defining/ r in this manner gives us several advantages: (a) it is dense, encoding both task completion and task progress, (b) it does not require any correspondence with a reference trajectory [5. 14] and (c) sidesteps the need for a finite library of reference trajectories, unlike prior work [5, 16] that define time-indexed rewards relative to some reference trajectory. Thus, agents with trajectories of varying lengths (due to embodiment-speede constraints) can efficiently leverage this reward because the learned encoder can map different strategies to a common notion of task progress.
3.4 Reinforcement Learning
Using the pre-trained frozen encoder Φ we define a Markov Decision Process (MDP) for any agent as the tuple (S, .A. P. r) where S is the set of possible states, A is the set of possible actions, P is the state transition probability matrix encoding the dynamics of the environment (including the agent) and is the learned reward function (defined in Section 3.3). Notice how we are able to use the same reward function for any agent even ones that the encoder may not have seen during training. Furthermore, this reward function solely depends on the learned encoder. Hence, the task represented by the MDP now depends solely on how well the encoder has learned task- task-specific representations since we do not use the environment reward to either define the task or learn the policy. This is an important distinction because we are expecting the encoder to generalize to new states that an agent might encounter during training, as the expert demonstrations in our dataset only contain successful trajectories, is also possible to augment sparse rewards (which only define task success failure) with our learned dense reward while training policies.
4
Experimental Setup
4.1 X-MAGICAL Benchmark
We introduce cross-embodiment imitation learning benchmark, X-MAGICAL, which is based on the imitation learning benchmark MAGICAL [24], implemented on top of the physics engine PyMunk [25]. In this work, we consider a simplified 2D equivalent of a common household robotic sweeping task, wherein an agent has to push three objects into a predefined zone in the environment (colored in pink). We choose this task specifically because its long-horizon nature highlights how different agent embodiments can generate entirely different trajectories. The reward in this environment defined the fraction of debris swept into the zone at the end of the episode.
Multiple Embodiments in X-MAGICAL. We create multiple embodiments by designing agents with different shapes and end-effectors that induce variations in how each agent solves a task. In Figure I, we show three of these embodiments and some sample trajectories that solve the sweeping task. Please see Appendix B for detailed description of the benchmark. Three agents are shaped like stick and they differ only in length. We call them short-stick, medium- seedium-stick and long-stick based on the length of their body. The agent in the last row is called gripper: it is circular in shape and has two arms that can actuate. All agents capable of two actions rotation around their axis and translation in forward/hackwan direction along this axis (similar to the agent in the MAGICAL benchmark). All agents have a two-dimensional action space and use force and torque control to change their position and orientation respectively. The
5

---

gripper agent has an additional degree of freedom for opening or closing its fingers. The default state of the gripper's fingers is open. For all agents, the state representation is 16-dimensional vector with the following information: (z. y) position of the agent, (oos 8, sin f) where @ is the agent's 2D orientation, and for each of the three debris: its (I, y) position, its distance to the agent and its distance to the goal zone. We ftame-stack [26] three consecutive state vectors to encode temporal and velocity information, resulting in final state dimension of 48.
Demonstrations and Different Embodiment Strategies in X-MAGICAL. To learn task-specific repre- sentations for this task, we collect 1000 demonstrations per agent, where each demonstration consists in sweeping all three debris, initialized with random positions, into the target zone. This is the dataset Dall (de- scribed in Section 3.1) containing observation-oni agent-specific demonstrations. In Figure 2. we highlight the differences that exist between the trajectories taken by these agents. Figure 2a shows a heat map of the frequency of visits (state visitation count) of each agent at every 2D position in the grid, across all demon- strations in the dataset. We plot the 2D projection of the state visitation count onto the XY plane with a bin width of 0.1. Yellow encodes higher state visitation whereas blue encodes lower state visitation. We observe that agent long- -tick has less coverage of the environment as opposed to agent gripper, which has signifi- cantly more coverage. Similarly, we show the distribution of debris locations for all agents across all demon- strations in Figure 2b. In Figure 2c, we plot randomly sampled trajectory from each agent's 's demonstration pool. We use transparency to encode the start (lighter) and end (darker) of the trajectory. It is clear from this figure that each agent solves the task in a different manner. Additionally, there is significant difference in the time taken by each agent to execute this task, as shown in Table 1. Agent long-stick is able to finish the task the quickest because of its long shape that can sweep all the debris once, while agents short-stick and gripper take longer because they have to frequently push or grasp one debris at a time. These differences are the types of challenges that representation must overcome to successfully generalize across embodiments. As such, X-MAGICAL serves to create a highly simplified version of a real- world scenario where we might want to learn new tasks from an observation dataset of humans performing these tasks in highly diverse ways.
4.2 Baselines
Here, we describe the alternative reward functions we baseline our method against, color coded to match their appearance in Figs. 3, 4, and 5. 1) ImageNer: We use ImageNet pre-trained ResNet- 18 encoder with no additional self-supervised training, i.e., we load the pre-trained weights, discard the classification head, and use the 512-dimensional embedding space from the previous layer. 2) Goal classifier: We follow [27] and train goal frame classifier on a binary classification task where the last frame of all the demonstrations is considered positive and all the others are considered negatives. We use the output probabilitics of the classifier as the reward function. 3) IJFSS: We implement the method from [16] which learns feature space that is invariant to different embodiments using a contrastive loss function paired with an autoencoding loss. 4) TCN: single-view Time-Contrastive Network (TCN) [5] with positive and negative frame windows of and 4 respectively. For more details regarding baseline implementations, see Appendix D.I.I.
4.3
Implementation Details
Our encoder for all experiments and methods is a ResNet- 18 [28] initialized with ImageNet pre-trained weights We replace the classification head with an embedding layer outputting 32-dimensional vector. The encoder is trained on images of resolution 224 x 224 with ADAM [29] and learning rate of 10-5, Note that our learned reward is agnostic to the RL algorithm used in this work, opt for Soft-Actor Critic (SAC) [30], which a reinforcement learning algorithm that has been successfully used to train policies for continuous control tasks [31]. Once the TCC encoder is trained, we use it to embed the observation frames as the agent interacts with the environment.
5 Experiments
We execute a series of experiments to evaluate whether the learned reward functions are effective at visual imitation. Specifically, our experiments seek to answer the following questions: first (Section 5.1), in the same-embodiment case, where the demonstration dataset D contains the embodiment of the learning agent, does our method enable successful reinforcement learning for that agent? Next (Section 5.2), we investigate our primary interest, the cross-embont pross-embodiment case, where the demonstration dataset D does not contain the embodiment of the learning agent. To additionally test our approach using real-world data (Section 5.3), we use the dataset from [6] to leverage real-world human demonstrations to learn policies in simulation. Note that each embodiment's 's performance is evaluated over 50 episodes and all figures plot the mean performance over 5 random seeds, with a standard deviation shading of +.0.5 Videos of our results are the supplementary video.
6

---

mageNect XIL
Goal Classifor
LG
Success hyre
Succues
2
(houssace)
Iffourande
Figure 3. Same-embodiinntt setting: Comparison of XIRL with other baseline roward functions, using SAC [30] for RL policy learning on the X-MAGICAL aweeping uakk.
XPL
TCN
Goal Classifee
LIFS
300
Stago (hrcuaanc))
Sheps (thosmandi)
Stape (hozseno))
Steps (thoosardds)
Figure Cross-embodiment setting: XIRL performs Eavorably wheen compured with other baseline reward functions, trained observation- onll demonstrations from different emboodiments, Each agent long-stick, medium-stick, short-stick, gripper) is shown using demoestrations from the other 3 emboddments, with SAC [30] for RL. policy learning the X-MAGICAL sweeping task.
5.1 Results on Learning from Embodimmbdd Demonstrations
In this experiment, we want to validate whether our approach of using a reward function trained with TCC is good enough to train agents to perform the task defined in a dataset of expert demonstrations. Note that the learned reward function has to be robust enough such that it can provide a useful signal for new states an agent might encounter while learning a policy and interacting with the environment. In Figure 3, we compare our method XIRL with baselines described in Section 4.2. We find XIRL is more sample-efficient than the other learned reward baselines. We attribute this sample efficiency to the fact that the TCC embeddings encode task progress which helps the agent learn to reach for objects and goal zones while interacting with the environment, rather than exploring in a purely random manner. This experiment provides evidence that XIRL's reward function is suitable for downstream reinforcement learning.
5.2 Results on Learning from Eooss-Embodiment Demonstrations
After verifying that XIRL works on the embodiments it was trained with, we move to the experiments that answer the core question in our work: can XIRL generalize unseen embodiments? In this section conduct experiments where the reward function learned using embodiments that are different from the ones on which the policy is trained. As noted in Table 1, the timescales with which the agents execute the task can vary significantly. We conduct four experiments, each corresponding to holding-out one agent from the expert demonstration set. In each such experiment, we train an encoder on demonstrations from the remaining three agents. We compare with reward functions learned using TCN, LIFS, and goal frame classifiers. While both TCC and TCN are contrastive losses, the former makes explicit comparisons across different embodimmnts, whereas the latter implicitly relies on an encoder shared across embodiments to learn the cross-embodiment representation. In Figure 4, we show that XIRL. generalizes new agents significantly better than the other learned reward baselines.
5.3 Results on Learning from Real-World
Demonstrations
In this experiment, we test how well we can learn rewards from more challenging real-world human demonstration videos. To do so, we use the dataset and State Pusher environment introduced in [6]. We train two XIRL encoders: XIRL (sim only) trained on teleoperated simulated trajectories (i.e., no domain shift) and XIRL (real only) trained solely on the real-world human demonstrations without using any form of human labeling of paired frames. We compare our results to training the policy on the sparse reward from the environment and the RLV method presented in [6]. As demonstrated in Figure 5, we find our approach can improve the sample-efficieng of learning in the environment, compared to using RLV or solely the environment reward.
5.4 Qualitative comparison between learned reward functions vs. handcrafted rewards
In Figures 6 and 7, we visualize and compare the XIRL reward function with the environment's reward (ground truth) for the Sweeping task from X-MAGICAL (see Sec. 4.1 for details) and the State Pusher and Drawer Opening tasks from [6]*. We find that the leamed reward is highly correlated with the ground truth
For additional qualitative visualizations real- world dataset with multiple embodiments, see Appendix C.
7

---

Pusher
Eploode Retum
-00
20
Sparie Renard XBI clm enly)
(nea ango
(thousands)
Figue 5. Renl-world-demo cross-embodiment setting: Comparison of XIRL with beselines using the simullteed State Pusher environment from (6]. XIRL (real only) leverages real-world demonstration vidcos of humans (left, teach robot in sim (lft, not use naman-lahelld data of paired frame corresrondlenees RIV demotes results taken verharim from [6] which uses a different implementation SAC policy leaming.
unlike [6], do
reward from the environment for both successful demos (first column in Figures 6 and 7) and unsuccessful demos (second column in Figures and 7). It is especially encouraging environment like the Drawer Opening task, XIRL. provides a dense signal that should allow the agent to see that in the example of the failed collision trajectory
that in sparser reward
leam the task more efficiently. Additionally, (i.e., second row third column Fig. 7), where the arm collides with the drawer rather than opening XIRL is able to provide it with a partial reward (i.c., for correctly moving towards the drawer) as opposed the environment reward which remains zero.
Rowerd
2 Rewand
Figue X-MAGICAL demo coos-cmbodlmmnt setting: Visualizing our Ieamndd reward function XIRL vs. the environment's sparse reward on a
successial and unsuccsstul demonstration. for the short-atick agent the X- X-MAGICL sweeping task. The learned reward trained the three other agents.
Damo
Negativa Dama
Osamatoo
EI 3HG Rowalr
J0TL
limestey
Noward SOOATE
Figure Real-world-demo crast-embodiment setting: Visualizing our leamned XIRL. reward function vs, the envirooment' is sparse reward on the State Pusher (sap) and Drawer Opening (bottom) tusks from
6 Conclusion
This paper presents XIRL, a framework for learning vision-based reward functions from videos of expert demonstrators exhibiting different embodiments. XIRL uses TCC self-supervise deep visual encoder from videos, and uses this encoder to generate rewards via simple distances to goal observations the embedding space. XIRL. enables unseen agents with new embodiments to learn the demonstrated tasks via IRL. Reward functions from XIRL are fully self-supervised from videos, and we can successfully learn tasks without requiring manually paired video frames [6] between the demonstrator and learner. In this sense, method presents favorable scalability an arbitrary number of embodiments experts with varying skill levels. Experiments show that policies learned via XIRL are more sample efficient than multiple baseline alternatives, including TCN [5], LIFS [16], and RLV [6]. While our experiments demonstrate promising results for learning policies in simulated environments using rewards learned from both simulated and real-world vidcos, we have yet to show policy learning on a real robot, which we look forward to trying Post-COVID.

---

Acknowledgments
We would like to thank Alex Nichol, Nick Hynes, Sean Kirmani, Brent Yi and Jimmy Wu for fruitful technical discussions, Sam Toyer for invaluable help with setting up the simulated benchmark, and Karl Schmeckpeper for discussions and help related to RLV.
References
[ B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5), 2009. 1, 2, 11
[2] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen. K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA. IEEE, 2018. 1
[3] P. Florence, I.. Manuelli, and R. Tedrake. Self-supervised comreponndncee in visuomotor policy learning. IEEE RAL. 5(2), 2019.
[4] P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXi: 2016, 2,
[5] Semmanet, C. Lynch, Y. Chebotar, Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pages 1134-1141, IEEE, 2018. 2, 3, 6, 8, 11, 16
[6] K. Schmeckpeper, O. Rybkin, K. Danillidis, Levine, and C. Finn. Reinforcement leaming with videos: Comhining offline observations with interaction, 2020. 1, 2, 3, 4, 6, 7, 8, 11, 20
D. Dwibedi, Aytar, Tompson, Sermanet, and A. Zisserman. Temporal cycle-consistency Jearning. In
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 2. 4. 12, 16
[8] C. Atkeson and Schaal. Robot learning from demonstration. In ICML, 1997. 2, 11
[9] T. Osa, J. Pajarinen, G. Neumann, J. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. Found. Trends Robotics, 7:1-179, 2018. 2, 11
[10] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In ICCAI, 2018. 2, 11
[11] A. D. Edwards, H. Sahni, Y. Schroecker, and C. L. Isbell. Imitating latent policies from observation. arXiv
preprint 100Xvv:8805009141 2018. 2, 11
[12] Radosavovic, X. Wang. I.. Pinto, and J. Malik. State-only imitation leamning for dexterous manipulation. arXiv
preprint 2rX0v:2004.046500 2020. 2, 11
[13] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Shentu, Shelhamer, Malik, A. Efros, and Darrell. Zero-shot visual imitation. Proceedings of the IEEE conference on computer vision and pattern
recognition workshops, pages 2050-2053, 2018. 2, 11
[14] Y. Aytar, Plaff, Budden, Paine, Wang. and N. de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pages 2930-29902900 2941, 2018. 2, 3. 11
[15] O. Mees, M. Merklinger, G. Kalweit, and W. Burgard. Adversarial skill networks: Unsupervised robot skill learning from video. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages
4188 4194. IEEE, 2020. 2, 11
[16] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine. Leaminng invariant feature spaces to transfer skills with reinforcement leamning 2017. 3,5, 6, 8, 11, 16
[17] A. Singh, L. Yang. K. Hartikainen, C. Finn, and S. Levine. End-to-ene robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904 1904407554, 2019. 3. 11
[18] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint 19Xiv:1912.04442 2019, 3, 11
[19] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In ICRA. IEEE, 2018 3, 11
[20] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller.
In Advances in Neurul Information Processing Systems, pages 2597-2607, 2019. 3, 11
[21] D. Hejna, L. Pinto, and P. Abbcel. Hierarchically decoupled imitation for morphologial transfer. International
Conference on Machine Learning, pages 4159-4171. PML. 2020. 3, 11
[22] J. Zhu, T. Park, P. Isola, and A. A. Eiros. Unpaired image-to-image translation using cycle-consistent
adversarial networks. In ICCV, 2017. 3, 11
[23] Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement leaming that
matters. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 5
[24] S. Toyer. R. Shah, A. Critch, and S. Russell. The magical benchmark for robust imitation. arXiv preprint
amXiv:2011.00401 2020. 5, 12
[25] V. Blomqvist. Pymunk: A easy-to-use pythomic rigid body 2d physics library (version 6.0.0), 2007. URL
attps://wwet pymunk.. org. 5
[26] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement leaming. nature, 518(7540):
529 533, 2015.

---

[27] M. Vecerik, O. Sushkov, D. Barker. T. Rothöri, T. Hester, and J. Scholz. A practical approach to insertion with
variable socket position using deep reinforcement lcarning. In 20/9 International Conference Robotics and Automation (ICRA), pages 754-760. IEEE, 2019. 6, 15, 16
[28] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[29] D. P. Kingma and Ba. Adam: method for stochastic optimization. arXiv preprint:1412.66 2014,
[30] T. Haamoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep
reinforcement learning with stochastic actor. arXiv preprint 1r0Xvv1801.012995 2018. 6, 7, 17
[31] Haarnoja, A. Zhou, Hartikainen,
Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv:18/2.05400 2018. 6
[32] C. Yang, X. Ma, W. Huang, F. Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. ArXiv, abs/1910.00417,, 2019. 11
[33] L. Shao, T. Migimatsu, Q. Zhang, K. Yang. and J. Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. Robotics: Science and Systems, 2020. 11
[34] G. Berseth and C. Pal. Visual imitation learning with recurrent siamese networks. ArXiv, abs/1901.07 076 2019. 11
[35] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning,
In Aaai, volume 8, pages 1433-1438. Chicago, II., USA, 2008. 11
[36] B. D. Ziebart, J. A. Bagnell, and A. K. Dey. Modeling interaction via the principle of maximum causal entropy.
In ICML, 2010. 11
[37] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.
Generative adversarial networks. arXiv preprint arXiv:1406.2601 2014. 11
[38] J. Ho and S. Ermon. Generative adversarial imitation leaming. In NeurlPS, 2016. 11
[39] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint
18Xivv1807.06658, 2018.
[40] B. C. Stadie, P. Abbeel, and Sutskever. Third-person imitation leaming, arXiv:1003.0100: 2017. 11
[41] Y. La and J. Tompson. Adail: Adaptive adversarial imitation learning. NewrIPS Workshop on Learning
Transferable Skills, 2019. 11
[42] Liu. Z. Ling, T. Mu, and H. Su. State alignment- imitation learning. arXiv preprint arXiv:191 109477
2019. 11
[43] W. Sun, A. Vemula, B. Boots, and J. Bagnell. Provably efficient imitation learning from observation alone. ArXiv, abs/1905, 10948, 2019. 11
[44] K. Kim, Gu, J. Song. Zhao, and S. Ermon. Domain adaptive imitation learning. arXiv, 2020. 11
[45] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein. L. Antiga,
A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural
Information Processing Systems 32. Curran Associates, Inc., 2019. 14
[46] A. Buslaev, A. Parinov, E. Khvedchenya, V. Iglovikov, and A. A. Kalinin. Albumentations: fast and flexible
image augmentations. arXiv e-prints, 2018. 16
[47] D. Yarats and I. Kostrikov. Soft actor-critic (sac) implementation in pytorch. https://github: com/
denisyarate/ppa paatttepptoccccheeee 2020. 17
[48] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. arXiv e-prints,
2015. 17
[49]
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation ervor in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICMI, 2018, Stockholmsmassson Stockholm, Sweden, July 10-15, 2018, 2018. 17
[50] A. Saxe, J. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics learning in deep linear neural networks. arXiv e-prints, 2013. 17
[51] T. Chen, S. Korbllihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv: arXiv:2002.00 05709, 2020. 17, 18
10


Reviews:
[{"review_id": "rS18VyBHPp9", "reviewer": "Reviewer_Fbyh", "summary": "This work introduces the use of Temporal Cycle-Consistency (TCC) for imitation learning. By training an encoder via TCC to embed similar states across different embodiments into similar latent variables, the authors can use proximity in latent space to a goal state as reward for imitation. The authors show this to great success in an artificial sim-to-sim 2D task and to decent success in a real-to-sim task.\n\nTo me, this paper is pretty water-tight. The method isn't groundbreaking because it's mostly using TCC, but the idea of using that as reward is novel enough to warrant publication. The experiments are all solid, the baselines are solid, and I'm honestly failing to see any major weaknesses. So I'm recommending acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- Great experiments. 2D sim2sim environment is a great illustration and quick turnaround cross-embodiment playground.\n- Writing is great and clear. The background section is just right for my personal taste in terms of explaining the basics but not overexplaining anything.\n- In the appendix there is a bonus dataset that should be put into the main body of the paper, X-REAL, that looks fun.\n- Very specific about the implementation and training details (in main paper and appendix), great work! \n- Using 5 seeds for all RL experiments, good job!\n- The smooth (learned) rewards in Fig. 6 and 7 are excellent.\n\n### Weaknesses:\n\n(None of these are major, just little things that I think might improve the paper)\n\n1. Maybe also include or at least mention VIRL (https://arxiv.org/abs/1901.07186), since it's similar to TCN and needs fewer expert samples. Compared to your method, it takes an embedding of the entire sequence so far and compares it to the expert.\n2. Fig.1 left side, bottom - what's that extra encoder at the bottom, and what's its output? How's it trained? Isn't that encoder part of the TCC training process? If that's the case, just remove it.\n3. In line 123 you refer to Fig.2c but the task wasn't really introduced yet. So to the average reader, up until then, Fig.2c is just squiggly lines.\n4. line 169 - you're using the mean embedding of all agents' last state as goal? Couldn't that potentially lead to an unreachable goal? If I look for example at the t-SNE embedding of the real videos, some of the final states are very far apart. Maybe as an analogy, when generating new images with a VAE by interpolating between two latents, when moving linearly through the state space, the results are usually unrealistic-looking, but when moving across a hypersphere between both samples, the results look much better (see here for example https://arxiv.org/abs/1609.04468). Maybe it'd be better to take the middle point on a hypersphere instead of the euclidean mean?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As mentioned above, the paper is very solid.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see weakness section", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "mMvSu_CXsrr", "reviewer": "Reviewer_tUkm", "summary": "Paper proposes to use TCC (temporal cycle-consistency learning) for learning representations in cross-embodiment visual imitation learning tasks. The authors empirically demonstrate that the learned representations can deliver a simple reward function (the distance between the current state and goal representations) that is sufficient for RL. A cross-embodiment benchmark (X-MAGICAL) is also presented and the proposed approach attains superior performances on both the simulation and real-world data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n[+] The paper is overall clear and well-written. I like how the authors approach their main results in an illustrative fashion and I believe this paper can also be easy to follow for readers without dense imitation learning background.\n\n[+] The proposed method is well-motivated. I agree that cross-embodiment imitation learning can effectively be a representation learning problem and TCC seems to be a reasonable inductive bias to produce the needed representations. I'm a little bit surprised that the resulting representations can even provide a straightforward reward solution. \n\n[+] I particularly like the real-world experiments and the results are truly impressive.\n\n[Weaknesses]\n\nI don't have major concerns at this point but I do encourage the authors to address the following issues in a rebuttal:\n\n[-] It is still unclear how the agent can benefit from cross-embodiment pretraining. Indeed I think the crucial question is: does the agent learn from other embodiments or just learn from visual pretraining. The following experiments may help clarify this question:\n\n- Add a baseline that just leverages general-purpose self-supervised visual pretraining, e.g. MoCo, SimCLR.\n- Try to learn from other embodiments only in the pretraining stage.\n\n[-] Some papers on imitation learning from observations (somehow shares the goal of cross-embodiment imitation learning) should be cited:\n\n[1] Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement\n\n[2] Learning and Inferring Movement with Deep Generative Model\n\n[3] Third-Person Imitation Learning\n\n[4] Provably Efficient Imitation Learning from Observation Alone", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given the overall technical quality, I recommend acceptance but I'm open to other assessment if there is something I missed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "I may consider further increasing my score if the following points can be addressed:\n\n1. Citations to the aforementioned papers.\n\n2. Extra results as requested above.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "F2hUjMasllC", "reviewer": "Reviewer_dDLT", "summary": "This paper presents a method of learning cross domain representations.  These representations can be used as a reward signal to speed the training of RL policies.  ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is mostly clearly written and presents an interesting new method.\n\nThe performance using human data is impressive, both on the X-Real dataset and on the RLV tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a novel method for learning cross domain embeddings and reward signals and provides extensive experimentation to show that the proposed method is effective in a variety of environments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "The paired data in RLV is in the expert domain, but is off policy. \n\nWhat is the difference between Env Reward in table 2 and Env Reward Only in Figure 5?\n\nI don't see the training curves for the Drawer Opening task, despite Figure 7 including it.\n\nThe X-Real tasks would be better in the main paper, rather than the supplemental."}}]

Author rebuttal:

Meta-review: This paper proposed to learn cross domain representations as a reward signal to achieve RL policies, and present a self-supervised method to learn deep visual embeddings considering embodiment differences. 

I agree with reviewers that this paper is well organized and the idea is quite novel, the extensive experiments demonstrated the proposed method is effective in a variety of environments.

I believe this is a good work, I agree with reviewers on some minor points they mentioned, and suggest authors carefully address these comments to further improve this paper. 

More added: 
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers, I really believe the revised version will be more better. Congratulations!
Decision: Accept (Oral)