Graph Inverse Reinforcement Learning from Diverse Videos
Authors: Sateesh Kumar
Jonathan Zamora
Nicklas Hansen
Rishabh Jangir
Xiaolong Wang
Venue: CoRL 2022 Oral (2022)
Keywords: Inverse Reinforcement Learning
Third-Person Video
Graph Network

Abstract: Research on Inverse Reinforcement Learning (IRL) from third-person videos has shown encouraging results on removing the need for manual reward design for robotic tasks. However, most prior works are still limited by training from a relatively restricted domain of videos. In this paper, we argue that the true potential of third-person IRL lies in increasing the diversity of videos for better scaling. To learn a reward function from diverse videos, we propose to perform graph abstraction on the videos followed by temporal matching in the graph space to measure the task progress. Our insight is that a task can be described by entity interactions that form a graph, and this graph abstraction can help remove irrelevant information such as textures, resulting in more robust reward functions. We evaluate our approach, GraphIRL, on cross-embodiment learning in X-MAGICAL and learning from human demonstrations for real-robot manipulation. We show significant improvements in robustness to diverse video demonstrations over previous approaches, and even achieve better results than manual reward design on a real robot pushing task. Videos are available at https://sateeshkumar21.github.io/GraphIRL/.

Graph Inverse Reinforcement Learning from Diverse Videos
Sateesh Kumar
Jonathan Z
Nicklas Hansen *
Rishabh Jangir
Xiaolong Wang
UC San Diego
Demo
Real
Reach
Push
Peg in Box
Figure 1: GraphIRL. We propose an approach for performing inverse reinforcement learning from diverse third-person videos via graph abstraction. Based on our learned reward functions, successfully train image-based policies in simulation and deploy them on a real robot.
Abstract: Research on Inverse Reinforcement Learning (IRL) from third- person videos has shown encouraging results removing the need for manual reward design for robotic tasks. However, most prior works are still limited by training from relatively restricted domain of videos. In this paper, argue that the true potential of third-person IRL lies in increasing the diversity of videos for better scaling. To learn reward function from diverse videos, we propose perform graph abstraction on the videos followed by temporal matching in the graph space to measure the task progress. Our insight that task can be de- scribed by entity interactions that form graph, and this graph abstraction can help remove irrelevant information such as textures, resulting in more robust reward functions. We evaluate our approach, GraphIRL, on cross-embodiment learning X-MAGICAL. and learning from human demonstrations for real-robot manipulation. We show significant improvements in robustness to diverse video demonstrations over previous approaches, and even achieve better results than manual reward design on real robot pushing task. Videos are available at https: ftaps:///sateeskkuma211. tithub....... io/ /GraphIRL/.
Keywords: Inverse Reinforcement Learning, Third-Person Video, Graph Network
Introduction
Deep Reinforcement Learning (RL) powerful purpose framework for learning behavior policies from high-dimensional interaction data, and has led to a multitude of impressive feats in application such game-playing [1] and roboties 3]. Through interaction with unknown environment, RI. agents iteratively improve their policy by learning to maximize reward signal, which has the potential be used in lieu of hand-crafted control policies. However, the performance of policies learned by RL is found to be highly dependent on the careful specification of task-specific
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

reward functions and, as result, crafting good reward function may require significant domain knowledge and technical expertise.
As an alternative to manual design of reward functions, inverse RL. (IRL. has emerged as a promising paradigm for policy learning. By framing the reward specification as a learning problem, operators can specify a reward function based on video examples. While imitation learning typically requires demonstrations from first-person perspective, IRL can in principle learn reward function, i... a measure of task progression, from any perspective, including third-person videos of humans performing a task. This has positive implications for data collection, since is often far easier for humans to capture demonstrations in third-pesoon.
Although IRL from third-person videos is appealing because of its perceived flexibility, learning a good reward function from raw video data comes with a variety of challenges. This is perhaps unsurprising, considering the visual and functional diversity that such data contains. For example, the task of pushing an object across a table may require different motions depending on the embodiment of the agent. A recent method for cross-embodiment IRL, dubbed XIRL [4], learns to capture task progression from videos in a self-supervised manner by enforcing temporal cycle-consistency constraints. While XIRL can in principle consume any video demonstration, we observe that its ability to learn task progression degrades substantially when the visual appearance of the video demonstrations do not match that of the target environment for RL. Therefore, it is natural ask the question: can we learn to imitate others from (a limited number of) diverse third-person person videos?
In this work, we demonstrate that it is indeed possible. Our key insight is that, while videos may be of great visual diversity, their underlying scene structure and agent-object interactions can be abstracted via graph representation. Specifically, instead of directly using images, we extract object bounding boxes from each frame using an off-the-shelf detector, and construct graph abstraction where each object represented as a node in the graph. Often- in robotics tasks the spatial location of an object by itself may not convey the full picture of the task at hand. For instance, to understand a task like Peg in Box (shown in Figure 1), we need also take into account how the agent interacts with the object. Therefore, we propose to employ Interaction Networks [5] on our graph representation to explicitly model interactions between entities. To train our model, we follow [4, 6] and apply a temporal cycle consistency loss, which (in our framework) yields task-specific yet embodiment- and domain-agnostic feature representations.
We validate our method empirically on a set of simulated cross-domain cross-embodiment tasks from X-MAGICAL [4], well three dision-based robotic manipulation tasks. To do so, we collect diverse set of demonstrations that vary in visual appearance, embodiment, object categories, and scene configuration; X-MAGICAL demonstrations are collected in simulation, whereas our manipulation demonstrations consist of real-world videos of humans performing tasks. We find our method to outperform a set of strong baselines when learning from visually diverse demonstrations, while simultaneously matching their performance in absence of diversity. Further, we demonstrate that dision-based policies trained with our learned reward perform tasks with greater precision than human- designed reward functions, and successfully transfer to a real robot setup with only approximate correspondence to the simulation environment. Thus, our proposed framework completes the cycle of learning rewards from real-world human demonstrations, learning policy in simulation using learned rewards, and finally deployment of the learned policy on physical hardware.
2 Related Work
Learning from demonstration. Conventional imitation learning methods require access to expert demonstrations comprised of observations and corresponding ground-truth actions for every time step [7, 8, 9, 10], for which kinesthetic teaching or telcoperation are the primary modes of data collection in robotics. To scale up learning, video demonstrations are recorded with human operating the same gripper that the robot used, which also allows direct behaviro cloning [11, 12]. More recently, researchers have developed methods that instead infer actions from data via a learned forward [13] or inverse [14, 15] dynamics model. However, this approach still makes the implicit assumption that imitator and demonstrator share a common observation and action space, and are therefore not directly applicable to the cross- dross-n cross-embodimenn problem setting that we consider.
Inverse RL. To address the aforementioned limitations, inverse RL has been proposed [16, 17, 18, 19, 20, 21] and it has recently emerged as a promising paradigm for cross-embodiment imitation in particular [22, 23, 24, 25, 26, 27, 28, 4, 29]. For examplc, Schmeckpeper et al. [22] proposes

---

Bounding Bozes
Graph Abstraction
Video
Video Seq #2
Video Suq Video
Encode Graph
Alignment
Eebadding Video Sequence fout
Giggt
Emoodring Space
Graph
Videe
Sustel
Embeddiing Video Sequence
RL. w/ Learned Reward
Video Sequence
Temporal Alignment
(4)
Makeingg
(b)
9
(4)
01
Time
Figure 2: Overview. We extract object bounding boxes from video sequences using an off-the-stie sfffthe-shelf detector, and construct a graph abstraction of the scene. We model graph-abstracted object interactions using Interaction Networks [5], and learn reward function by aligning video embeddings temporally. We then train image-base based RL policies using our learned reward function, and deploy on a real robot.
method for integrating video demonstrations without corresponding actions into off-policy RL algorithms via a latent inverse dynamics model and heuristic reward assignment, and Zakka et al. [4] (XIRL) learns a reward function from video demonstrations using temporal cycle-consistency and trains an RL agent to maximize the learned rewards. In practice, however, inverse RL methods such as XIRL are found require limited visual diversity in demonstrations, Our work extends XIRL to the setting of diverse videos by introducing a graph abstraction that models agent-object and object-object interactions while still enforeing temporal cycle-consistency.
Object-centric representations. have been proposed in many forms at the intersection of computer vision and robotics. For example, object-centric scene graphs can be constructed for integrated task and motion planning [30, 31, 32), navigation [33, 34., relational inference [35, 36], dynamics modeling [5, 37, 38, 39, 40, 41]. model predictive control [42, 43, 44], grasping [45, 46] or visual imitation learning [47, 48, 49]. Similar to our work, Sieb et al. [49] propose to abstract video ensoddmebod imitation, and act by minimizing the difference between the demonstration graph and graph constructed from observations captured at each step. As such, their method is limited to same-domain visual trajectory following, whereas we learn general alignment function for cross-domain cross-embodiment imitation and leverage Interaction Networks [5] for modeling graph-abstracted spatial interactions rather than relying on heuristics.
demonstrations as object-centric graphs for the problem of single-vides
3 Our Approach
In this section, we describe our main contribution, which is self- method for learning visually invariant reward function directly from a set of diverse third-person video demonstrations via graph abstraction. Our Graph Inverse Reinforcement Learning (GraphIRL) framework. shown in Figure 2, consists of building an object-centric graph abstraction the video demonstrations and then learn embedding space that captures task progression by exploiting the temporal cue in the videos. This embedding space then used to construct a domain invariant and embodiment invariant reward function which be used to train any standard reinforcement learning algorithm.
Problem Formulation. Given task T, our approach takes dataset video demonstrations D (Vi, V2,..., V.). Each video consists of image frames { .z.... T& where i denotes the video frame index and denotes the total number of frames in V.. Given D, our goal is learn reward function that can be used to solve the task T for any robotic environment. Notably, do not

---

assume access to any action information of the expert demonstrations, and our approach does nor require objects or embodiments in the target environment to share uppearance with demonstrations.
3.1 Representation Learning
To learn task-specific representations in self-superviste manner, we take inspiration from Dwibedi et al. [6] and employ a temporal cycle consistency loss. However, instead of directly using images, we propose novel object-centric graph representation, which allows us learn an embedding space that not only captures task-specific features, but depends solely on the spatial configuration of objects and their interactions. We here detail each component of our approach to representation learning.
Object-Centrif Representation. Given video frames { Ii..................
..................... .,,,,,.. }. we first extract object bound- ing boxes from each frame using an off-the- detector. Given N bounding boxes for an image, we represent each bounding box as 4 + m dimensional vector oj = {z1: y1 T22 32, d1, d2..... dm h- where the first dimensions represent the leftmost and rightmost corners of the bounding box, and the remaining m dimensions encode distances between the centroids of the objects. For each frame T we extract object-centric representation T {01, 02,..., 0m } such that we can represent our dataset of demonstrations D' (V7, V2 ..... VM') where V is the sequence of bounding boxes corresponding to video V,. Subsequent sections describe how we learn representations given D'.
Spatial Interaction Encoder. Taking inspiration from recent approaches on modeling physical object dynamics [5, 37J, we propose a Spatial Interaction Encoder Network to explicitly model object-object interactions. Specifically, given a sequence V; from D', model each element I' as a graph. G = (O, where O is the set of objects {01, O2.... 3m m is the total number of objects in I', and R denotes the relationship between objects (i.e., whether two objects interact with each other). For simplicity, all objects are connected with all other objects in the graph such that R= ((. i * jÁi m A S m). We use fully-connected graph because this makes the least assumption about the problem and task specific object interaction structure could then be learned directly from the data. We compose object embedding for each of o, E O by combining self and interactional representations as follows:
m
fo(o,) &ugg (fs + fin) with fa(o;) = (o)),
finlo)) = = Smlo, o,) (i.j) ER, (1)
where fa(o.) represents the self or independent representation of an object, fm represents the interactional representation. i.e., how it interacts with other objects in the scene, fo is the final object embedding, and (. represents concatenation. Here, the encoders a., oi and &ugg denote Multi layer Perceptron (MLP) networks respectively. We emphasize that the expression for fin(-) implies that the object embedding fo).) depends on all other objects in the scene; this term allows us to model relationships of an object with the others. The final output from the spatial interaction encoder (+(-) for object representation I' is the mean of all object encodings:
1
(1'))
f(o.).
(2)
m
The spatial interaction encoder is then optimized using the temporal alignment loss introduced next.
Temporal Alignment Loss. Taking inspiration from prior works on video representatoin learning [6, 50, 51], we use temporal alignment as a proxy for video representation learning. Given pair of videos, the task of self- supersuper alignment implicitly assumes that there exists true semantic correspondence between the two sequences, i.e., both videos share common semantic space. These works have shown that optimizing for alignment leads to representations that could be used for tasks that require understanding task progression such as action-classificate order to solve for alignment, learning model has to learn features that are (1) common across most videos and (2) exhibit temporal ordering. For a sufficiently large dataset with single task, the most common visual features would be distinct phases of a task that appear in all videos and if the task has small permutations, these distinct features would also exhibit temporal order. In such scenarios, the representations learned by optimizing for alignment are task-specific and invariant to changes in viewpoints, motions and embodiments.
This is because in
In this work, we employ Temporal Cycle Consistency (TCC) [6] loss to learn temporal alignment. TCC optimizes for alignment by learning an embedding space that maximizes one-to-one nearest neighbour mappings between sequences. This achieved through a loss that maximizes for cycle- consistent nearest neighbours given a pair of video sequences. In our case, the cycle consistency

---

is applied on the graph abstraction instead of image features as done in the aforementioned video alignment methods. Specifically, given D', we sample pair of bounding box sequences V ( ..... I } and V = jn, Th ). we extract embeddings by applying our spatial interaction encoder defined in Equation 2. Thus, we obtain the encoded features S; = (v' (b and Sj = -+(("))... ..... (( )}. For the nth element in St, we first compute its nearest neighbour, vg) in Sy and then compute the probability that it cycles-back n, 3ijn as
m
--11s--535
-llv 11
2
84 kjn
(3)
Σx m
11 6 11S
Σx e -11o, S
The eycle consistency loss for rith element can be computed D (1²) n) where [4) Σx f is the expected value of frame index n as we cycle back. The overall TCC loss is then defined by summing over all pairs of sequence embeddings SS) in the data, i.e., L.j = Eijn L.
3.2 Reinforcement Learning
We learn task-specific embedding space by optimizing for temporal alignment. In this section, we define how to go from this embedding space to a reward function that measures task progression. For constructing the reward function, embedding space, we can use euclidean distance as a notion of task progression, i.e., frames far apart in the embedding space will be far apart in terms of task progression and vice versa. We therefore choose to define our reward function as r(o) - _ lob(() - 0|ا² with = Σi=1 16 ( m.) ) where is the current observation, tp is our Interaction
leverage the insight from Zakka et al. [4] that in a task-
setworks-based encoder from Section 3, g is the representative goal frame, m, is the length of sequence V and c is a scaling factor. This definition gives dense reward because as the observed state gets closer and closer the goal, the reward starts going down and approaches zero when the goal and current observation are close in embedding space; refer supplementary material for qualitative analysis of the learned reward function. After constructing the learned reward, we can use it to train any standard RL algorithm. We note that, unlike previous approaches [22, 4], our method does not use any environment reward improve performance, and instead relies solely on the learned reward, which our experiments demonstrate is sufficient for solving diverse robotic manipulation tasks.
4 Experiments
In this section, we demonstrate how our approach uses diverse video demonstrations to learn reward function that generalizes to unseen domains. In particular, we are interested in answering the questions: (1) How do vision-based methods for IRL perform when learning from demonstrations that exhibit domain shiff? (2) Can our approach learn stronger reward signal under this challenging setting? To that end, we first conduct experiments on the X-MAGICAL benchmark [4]. We then conduct experiments on multiple robot manipulation tasks using a diverse set of demonstrations.
Implementation Details. All MLPs have hidden layers, and the embedding layer outputs features of size 128 in all experiments. We use Soft Actor Actor-Critic (SAC) [52] as backbone RL algorithm for all methods. For experiments on X-MAGICAL, we follow Zakka et al. [4] and learn state-based policy. For robotic manipulation experiments, we learn a multi-view image-based SAC policy [53]. For fair comparison, we only change the learned reward funetion across methods and keep the RL. setup identical. Refer to the supplementary material for further implementation details.
Baselines. We compare against multiple vision-based approaches that learn rewards in self- supervised manner: (1) XIRL [4] that learns a reward function by applying the TCC [6] loss on demonstration video sequences, (2) TCN [54] which is self-supervised contrastive method for video representation learning that optimizes for temporally disentangled representations, and (3) LIFS [55] that learns an invariant feature space using dynamic time warping- based contrastive loss. Lastly, we also compare against the manually designed (4) Environment Reward from Jangir et al. [53]. The environment reward baseline is an oracle method since it is dense reward and is carefully designed for the task under consideration. For learning-bass baselines, we use a ResNet-18 encoder pretrained on ImageNet [56] classification. See supplementary material for details.
5

---

Standard Environment
Diverse Environment
Gripper
S-stick
M-stick
L-stick
Gripper
S-stick
M-stick
L-stick
Figure 3: Overview of X-MAGICAL task variants. We consider two environment variants and four embodiments for our simulated sweeping task experiments. We evaluate IRL algorithms both the Diverse and Standard environments across four embodiments in the moss-Embodimenl settings.
0.8
gripper
shortstick
1.0
mediumstick
1.0
longstick
0.6
Succets Rate
0.6
0.8
0.8
0.4
0.6
0.6
0.4
0.4
0.0
100
300 Steps (thousands)
200
500
0.00
100
400
500
0.0g
100
300
400
500
0.0
Steps (thousands)
Steps (thousands)
Steps (thousands)
GraphIRL (Durs)
LIFS
gripper
shortstick
mediumstick
longstick
0.6
Success Rate
0.6
0.4
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0
100
200
300 400 (thousands)
500
0.0
100
200
300 400 (thousands)
500
0.0
100
200
300 (thousands)
400 500
0.0 100 200
300
400
500
Steps (thousandss
GraphIRL (Ours)
XIRL
TCN
LIFS
gripper
medlumstick
longstick
0.6
0.8
Success
0.4
0.4
0.6
0.4
0.4
0.2
0.2
0.2
0.2
0.0
100 200 300
400
500
0.0
100
200
400 (thousands]
300
500
0.00
100
200
300 400
500
0.0
100
200
400 500
Steps (thousands)
(thousands)
GraphIRL (OOD)
XIRL (OOD)
TCN (OOD)
LIFS (OOD)
Figure 4: Cross-Embodiment Cross-Enviromment. Success rates of our method GraphIRL and baselines (top) Standard Environment Pretraining Diverse Environment RL, (middle) Diverse Environment Pretraining > Standard Environment RL and (bottom) Diverse (OOD) Environment Pretraining Standard Environment RL. All reported numbers averaged over 5 seeds. Our approach performs favorably when compared to other baselines on all three settings.
4.1 Experimental Setup
We conduct experiments under two settings: the Sweep-to- Goal task from X-MAGICAL [4], and robotic manipulation tasks with xArm robot both in simulation and on real robot setup. We describe our experimental setup under these two settings in the following.
X-MAGICAL. We choose to extend X-MAGICAL [4], a 2D simulation environment for cross- embodiment imitation learning. On this benchmark, we consider a multi-object sweeping task. where the agent must push three objects towards a static goal region. We utilize two variants of the X-MAGICAL benchmark, which we denote as Standard (original) and Diverse (ours) envi- ronments, shown in Figure 3. Standard only randomizes the position of objects, whereas Diverse also randomizes visual appearance. We consider a set of four unique embodiments {gripper, short- stick, medium-stick, long-stick). In particular, we conduct experiments in the cross-envirommen and mousseembodiment setling where we learn reward function the Standard environment on held-out embodiments and do RI. in the Diverse environment on target embodiment, or vice-versa. This provides an additional layer of difficulty for the RL. agent as visual randomizations show the brittleness of vision-based IRL methods.

---

XArm Reach
xArm Push
XArm Peg in Box
1.0
1.0
1.0
Success Rate
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
200 Steps (thousands)
100
300
200
400
600
800
250
500
750
1000
Steps (thousands)
Steps (thousands)
GraphIRL (Ours)
XIRL
TCN
LIFS
Env. Reward
Figure 5: Robotic Manipulation. Success rates of our method GraphIRL and baselines on the tasks of Reach, Push and Peg in Box. All results are averaged over 5 seeds. We observe significant gains in performance specially over vision-based baselines due to large domain-gap
Real
XIRL
Env. Reward
GraphIRL (Ours)
Push
0.27
0.47
0.60
Reach
0.26
0.93
0.86
Peg in Box
0.06
0.60
0,53
Table 1: Real robot experiments. Success rate on robot manipulation tasks on physical hardware. We evaluate each method for 15 trials using a fixed set of goal and start state configurations.
Robotic Manipulation. Figure shows initial and success configurations for each of the three task that we consider: (1) Reach in which the agent needs to reach goal (red disc) with its end-effector, (2) Push in which the goal is to push cube to a goal position, and (3) Peg in Box where the goal is to put a peg tied to the robot's end-effector inside box. The last task is particularly difficult because it requires geometric 3D understanding of the objects. We collect a total of 256 and 162 video demonstrations for Reach and Peg in Box, respectively, and use 198 videos provided from Schmeckpeper et al. [22] for Push. The videos consist of human actors performing the same tasks but with number of diverse objects and goal markers, well as varied positions of objects. Unlike the data collected by Schmeckpeper et al. [22], we do not fix the goal position in our demonstrations. Additionally, we do not require the demonstrations to resemble the robotic environment in terms of appearance or distribution of goal location. In order to detect objects in our training demonstrations, we use trained model from Shan et al. [57]. Refer supplementary for details robot platform.
4.2 Results
X-MAGICAL. Results for the cross-embodiment and cross-enviromment setting are shown in Figure
4. When trained on Standard, our method performs significantly better than vision-based baselines (e.g., 0.58 GraphIRL for gripper vs 0.35 for XIRL and 0.99 GraphIRL for longstick vs 0.56 XIRL). We conjecture that vision-based baselines struggle with visual variations in the environment, which our method is unaffected by due to its graph abstraction. Additionally, we note that XIRL performs strongly when trained on Diverse. This could be because the Diverse environment contains some examples that are visually the same as Standard environment. To verify this, we conduct an experi- ment where we remove the shape (Square) and color (Red) used in Standard environment from the random configurations for Diverse environment and then construct new set of demonstration. We refer to this environment as Diverse (Out Of Distribution). In this setting, XIRL's performance drops significantly for out of embodiments.
Robotic manipulation in simulation. In this section, we answer the core question of our work: can we learn to imitate others from diverse third-person videos? In particular, we collect human demonstrations for manipulation tasks as explained in Section 4. and learn a reward function as explained in Section 3. This is challenging setting because as shown in Figure 1, the collected data and robotic environments belong to different domains and do not share any appearance characteristics. Further, unlike previous works [22, 4], we do not use any environment reward an additional supervision to the reinforcement learning agent. Figure presents our results. For the Reach task, GraphIRL and environment reward achieve success rate of 1.0, while other baseline methods are substantially behind GraphIRL (e.g. 0.477 XIRL and 0.155 TCN). In the Push setting, vision-based baseline methods still perform poorly. Similar to Reach, XIRL. performed the best out of the vision- based basclines with success rate of 0.187, and GraphIRL performed better than environment
7

---

XArm Reach
xArm Push
XArm Peg in Box
1.0
1.0
1.0
Success Rate
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
100
200
300
200
400
600
800
500
1000
1500
Steps (thousands)
Steps (thousands)
Steps (thousands)
GraphIRL (10% Demos)
GraphIRL (25% Demos)
GraphIRL (100% Demos)
Figure 6: Ablation Study on Number of Demonstrations. Results averaged over 5 seeds. We use a total of 256, 198 and 162 videos for Reach, Push and Peg Box respectively.
reward with success rate of 0.832. The Peg in Box task is rigorous to solve since it requires 3-d reasoning and a precise reward function. Here, while all vision-based methods fail, GraphIRL solves the task with 55.2% success rate. Overall, our GraphIRL method is able solve 2D and 3D reasoning tasks with a real-robot without a hand-designed reward function or access to 3D scene information.
Real robot experiments. Finally, we deploy the learned policies we conduct 15 trials per method and report the average success rate. Results are shown in Table 1. Interestingly, we find that GraphIRL outperforms XIRL in all three tasks the real robot setup, and on Push, GraphIRL performs better than the environment reward specifically designed for the task (e.g. 0.93 Environment Reward vs 0.86 GraphIRL) which is in line with our findings in simulation.
real robot. For each experiment,
4.3 Ablations
Variant
Success Rate
Impact of Modelling Spatial Interactions. We study the impact of modeling object-object spatial interactions using Spatial Interaction Encoder Network (IN) described in Section 3.1. Specifically, replace our proposed encoder component with an Multi-Layer Perceptron (MLP) by concatenating rep- resentations of all objects into a single vector and then feeding it to a 3-layer MLP network. Results in Table 2. We observe 20% improvement
MLP IN
0.61 10.116
0.80410.054
Table 2: Impact of modelling object- object interaction Push task. MLP: Multi-layer perceptron and IN: Spatial Interaction Network En- coder. Results over 5 seeds.
that modeling object interactions leads in the RL success rate (i.e. 0.61 for MLP vs 0.804 for IN).
Impact of Decreasing Number of Demonstration Videos. Results in Figure 6. We find that our approach is very data efficient and can learn meaningful rewards even from a small number of videos. It achieves decent RL success rate with only 10% of total videos used (e.g. 72% for Push).
5 Conclusions and Limitations
Conclusion. We demonstrate the effectiveness of our proposed method, GraphIRL, in a number of IRL settings with diverse third-person demonstrations. particular, we show that our method successfully learns reward functions from human demonstrations with diverse objects and scene configurations, that we are able to train image-based policies in simulation using our learned rewards, and that policies trained with our learned rewards are more successful than both prior work and manually designed reward functions on a real robot. We also hereby commit to release our complete code and data to the public.
Limitations. While our method relaxes the requirements for human demonstrations, collecting the demonstrations still requires human labor; and although our results indicate that we can learn from relatively few videos, climinating human labor entirely remains an open problem. Moreover, our approach assumes access to object bounding boxes. This implics that our method's performance is dependent on the performance of the object detector. Fortunately, 2d object detectors have become very reliable as we show in our experiments, we are able to use an off-the-shelf object detector to extract the bounding boxes without having to perform any manual labeling. Finally, the proposed graph abstraction allows us to solve tasks despite the large domain gap but it has some potential disadvantages too. In particular, we lose fine grained information such as object poses and precise object interactions which could be useful for complex tasks such as medical procedures. However, we conjecture that images might also not be sufficient for such task since inferring accurate object pose, 3D geometric information from 2D image itself is challenging problem.
8

---

Acknowledgments
This work was supported, in part, by grants from NSF CCF-21 (TILOS), NSF 1730158 CI- New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 DC* Pacific Research Platform, and gifts from Meta, Google, Qualcomm.
The authors would like thank Sanjay Haresh, Kevin Zakka, Jianglong Ye and Mohit Jain for helpful discussions.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou,
H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.
[2] S. Levine, P. Pastor, A. Krizhevsky, Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421 -46,6436,,, 2018
[3] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3-20, 2020.
[4] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pages 537-546. PMLR, 2022.
[5] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances neural information processing systems, 29, 2016.
[6] D. Dwibedi, Y. Aytar, Tompson, P. Sermanet, and Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1801-1810, 2019.
[7] A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
[8] C. G. Atkeson and S. Schaal. Robot learning from demonstration. In ICML, volume 97, pages 12-20. 1997.
[9] B. D. Argall, S. Chermova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-4)) 4833 2009.
[10] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, roboties, and autonomous systems, 3:297-330, 2020.
[11] S. Song, A. Zeng. Lee, and Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3): 4978 4985, 2020.
[12] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. In Conference on Robot Learning, pages 1992-2005. PMLR, 2021.
[13] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and Darrell. Zero-shot visual imitation. 2018 IEFF/CVE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2131-21313, 2018
[14] Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. ArXiv, 0bs//805.0199/ 2018.

---

[15] I. Radosavovic. X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. 2021 IEEE/RSI International Conference on Intellieet Robots and Systems (IROS), pages 7865-7871, 2021.
[16] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, page 2, 2000.
[17] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 2004.
[18] Ho and Ermon. Generative adversarial imitation learning. Advances neural information processing systems, 29, 2016.
[19] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv: 1710. /248, 2017.
[20] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. De Freitas. Playing hard exploration games by watching youtube. Advances in neural information processing systems, 31, 2018.
[21] Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.
[22] K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. arXiv preprint arXiv:2011.06550 2020.
[23] Jin, L. Petrich, Zhang. M. Dehghan, and M. Jagersand. Visual geometric skill inference by watching human demonstration. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8985-8991. IEEE, 2020.
[24) H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 202/ IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[25] Y. Lee, A. Szot, S.-H. Sun, and J. J. Lim. Generalizable imitation learning from observation via inferring goal proximity. Advances in Neural Information Processing Systems, 34, 2021.
[26] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in-the-wild" human vidcos. arXiv preprint arXiv:2103.168177 2021.
[27] A. Fickinger, S. Cohen, S. Russell, and B. Amos. Cross-domain imitation learning via optimal 2021.
transport. arXiv preprint
[28] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570-587. Springer, 2022.
[29] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning- based framework for efficient dexterous manipulation. arXiv preprint arXiv:2203.13255 2022.
[30] G. E. Fainckos, A. Girard, H. Kress-Gazit, and G. J. Pappas. Temporal logic motion planning for dynamic robots. Automatica, 45(2):343-355 2009.
[31] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible iadneee-ndeependent interface layer. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 639-646, 2014.
[32] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu. Hierarchical planning for long-horizon manipu- lation with geometrie and symbolic scene graphs. In 2021 1 IEEE International Conference on Robotics and Automation (ICRA), pages 6541-6548, IEEE, 2021.
[33] S. Gupta, V. Tolani, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. International Journal of Computer Vision, 128:1311-133 2019.
10

---

[34] W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. arxiv preprint arXiv:1810.06533, 2018.
[35] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3097-3106, 2017.
[36] Y. Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang. Scene graph generation from objects, phrases and region captions. 2017 IEEE International Conference on Computer Vision (ICCV), pages
2017.
[37] N. Watters, D. Zoran, Weber, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks: Learning physics simulator from video. Advances in neural information processing systems, 30, 2017.
[38] J. Materzynska, T. Xiao, R. Herzig, H. Xu, X. Wang, and T. Darrell. Something-else Compo- sitional action recognition with spatial- temporall interaction networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1049-1059, 2020.
[39] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani. Compositional video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10353-10362, 2019.
[40] H. Qi. X. Wang, D. Pathak, Y. Ma, and J. Malik. Learning long-term visual dynamics with region proposal interaction networks. arXiv preprint (iXiv:2008.022653 2020.
[41] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint wrXiv:2009.05085 2020.
[42] A. Sanchez-Gonzalez, N. Heess, T. Springenberg, Merel, M. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pages 4470-4479, PMLR, 2018.
[43] Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba. Learning compositional koopman operators for model-based control. arXiv preprint arXiv:
2019.
[44] Y. Ye, D. Gandhi, A. Gupta, and S. Tulsiani. Object-centric forward modeling for model predictive control. In Conference Robot Learning, pages 100-109. PMLR, 2020.
[45] E. Jang, C. Devin, V. Vanhoucke, and S. Levine. Grasp2vec: Learning object representations from self-supervised grasping. arXiv preprint
2018.
[46] Y. Yang. Y. Liu, H. Liang, X. Lou, and C. Choi. Autribute-base robotic grasping with one-grasp adaptation. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6367-6363. IEEE, 2021.
[47] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[48] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz. S3k: Self-supervised semantic keypoints for robotic manipulation via multi-view consistency. arXiv preprint arXiv:2009. /47/ 2020.
[49] M. Sieb, 7. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imitation. In Conference on Robot Learning, pages 979-989, PMLR, 2020.
[50] S. Haresh, S. Kumar, H. Coskum, S. N. Syed, A. Konin, Z. Zia, and Q.-H. Tran. Learning by aligning videos in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3548-5558, 2021.
[51] W. Liu, B. Tekin, H. Coskun, V. Vineet, P. Fua, and M. Pollefeys. Learning to align sequential actions in the wild. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 2181-2191, 2022.

---

[52] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[53] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):3046-3055 2022.
[54] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018.
[55] A. Gupta, C. Devin, Y. Liu, Abbeel, and S. Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv:/73..0230 2017.
[56] O. Russakovsky, J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, et Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-255 2015.
[57] D. Shan, J. Geng, M. Shu, and D. F. Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1869-98788 2020.
[58] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference on Learning Representations, 2020.
[59] N. Hansen, Su, and X. Wang. Stabilizing deep q-learning with convnets and vision trans- formers under data augmentation. Advances in Neural Information Processing Systems, 34: 3680-3693, 2021.
12


Reviews:
[{"review_id": "z8EIJnL22cv", "reviewer": "Reviewer_NhHS", "summary": "This work proposes a framework for robot learning from third-person human demonstration videos. To learn from a diverse set of videos that have different viewpoints, motions, and embodiments, the IRL is performed at an abstract, graph level. They show that the reward function, indicating task progression, that is learned with this abstraction can be used to train an RL agent and the learned policy can be successfully transferred to a real robot. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Plus\n- The research problem is well motivated. The components in the proposed pipeline are designed well. There are no major flaws in experiments and evaluations.\n- Choosing the right representation for an IRL problem is a fundamental research question. Scene graphs have a lot of potential in this direction and this work has demonstrated its effectiveness.\n- No environment nor additional hand-crafted reward is required, which sets this paper apart from some previous works.\n\nMinus\n- This work assumes an object detector is given, which is a reasonable assumption. But there are several unanswered questions. What off-the-shelf detector is being used here? What’s the detection accuracy in the tasks presented? More critically, how accurate the perception module needs to be for the entire pipeline to be effective?\n- Graph abstraction has all the advantages mentioned in the paper, but like any abstraction, some information is lost. What is some critical information that is lost in the abstraction? Does this prevent the proposed approach to generalize to tasks that heavily rely on, for example, object pose information, 3D geometric information, etc. This should be at least discussed in the limitations section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall this is solid work. See strengths for the reasons for acceptance. Although the concerns raised should be addressed and discussed in detail. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- How is occlusion handled? Do we always assume that all objects are visible?\n- What is the action space for robotic tasks?\n- Sim2real results: we need more information on what causes the performance drop here. What are some typical failure cases? Is that the policy? Perception? Action execution?\n- Sim2real results: for Peg-in-Box, the performance in the real world is better than the one in simulation? \n- Ln73-75: can you explain why approaches like BCO [14] assume shared observation / action space? At least not according to this review paper: Recent Advances in Imitation Learning from Observation (Torabi et al., 2019) see section 3.1. \n"}}, {"review_id": "yw_MYe5lqzY", "reviewer": "Reviewer_J5yc", "summary": "This paper introduces an inverse reinforcement learning approach for learning from diverse third-person videos. The key component of the method is representing the frames of videos with graphs, which capture scene semantics and object interactions while leaving out visual details such as textures. They generate the graphs by running a pre-trained object detector then computing geometric relationships between each pair of detected objects; then, they apply temporal cycle consistency-based self-supervised learning on the graph representations, which yields an embedding space from which they can generate a dense reward signal given an observation and a goal. They show much improved performance with their method on a simulated sweeping task and three robotic manipulation tasks on a real robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Graph-based method is intuitive, well-explained, and shown to be very effective.\n2. Experiments are thorough.\n3. Writing is very clear.\n\nWeaknesses:\n1. The selection of architecture for processing the graph and building the graph could be better motivated. For instance, were other graph convolutional architectures considered? Why use a fully-connected graph? etc.\n2. Regarding the graph representation, it is unclear exactly how exactly it differs from prior work in object-centric representations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper shows that graph representations can be used to greatly improve the performance of inverse RL when learning from diverse data. The method is elegant, clearly explained, and highly effective, and the experiments are thorough, with results shown on a real robot. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "no specific issues, addressed in weaknesses", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "cgS7tAUb4xhB", "reviewer": "Reviewer_77Z4", "summary": "The authors present a technique they call GraphIRL, for learning reward abstractions across domains with diverse embodiment gaps. The method decomposes the underlying scene structure and agent-object interaction into a graph, which the authors claim is invariant to distractor features (like texture, agent embodiment, etc). The learned reward is then used in an IRL setting to learn policies from third-person demonstrations and they show successful cross-embodiment transfer on both simulated and real-world setups.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is highly polished and easy to read. The videos are high quality and the experimental results are quite thorough (with some details missing… see below).\n\n- Real robot experiments are appreciated.\n\n- This is an incredibly challenging problem domain so any improvement in performance could be impactful for the community.\n\nWeaknesses\n\n- Some references are missing. There have been quite a few papers that use object-centric representations to define (by definition) invariant state representations. One example would be Manuelli., “Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning.” I suggest the authors perform an additional literature search.\n\n- The reward is not conditioned on policy actions and so implies some limitations in the form of reward that can be learned. e.g. learned rewards cannot be directly action dependent - or assumes the learned reward can infer the policy action if this is what is needed (concrete example: rewards that introduce penalties or costs for certain actions). Please add details discussing the fundamental limitations of the assertions made, especially since you have a “Limitations” section.\n\n- Similarly, the reward is defined on a “representative goal frame” which also limits the rewards that can be learned (effectively, you can only represent rewards where some individual goal state is sufficient to solve the task). This should be discussed to avoid overselling the method.\n\n- I’ve read the “Our Approach” section a few times and while I think I understand how the graph is constructed, a visual aid would be immensely helpful. I understand the component parts (psi, f_o, f_s, f_in), and I can follow these into S_i and S_j, but it would be nice to see the flow from image to bounding box, to embeddings to construction of the graph. Figure 2 is a good start, but doesn’t actually contain any of the terms from the “Our Approach” section so is somewhat hard to follow.\n\n- Table 1: please explain why the environment reward underperforms your method? Is this a sparse reward (and if so why did you compare a dense reward with a sparse one)? Otherwise, I’d expect a well crafted true reward to be an upper bound so this requires some explanation.\n\n- For the real robot experiments, did you train on any real data? Or did you train the reward only in sim and then deploy in real? Likewise, did you train the policy in real (or reward + policy trained in sim and deployed in real)? Sorry if this is spelled out, but I wasn’t able to confirm. If it’s the case that the reward was trained in sim only, it would make sense to me that a method that first crops objects learns a reward function that can better transfer sim to real, while a method like XIRL, that uses full image embeddings, won't perform as well (since background distractor features are going to result in generalization issues). This isn’t necessarily a ding against the method at all (and in fact using bounding boxes is a reasonable contribution), but an experiment that also trains the full stack in real would be necessary to make sure the graph representation accounts for the claimed performance gains (and it’s not just because of cropping out of the background).\n\n- Nit: It would be helpful if the number of videos was in the Table 3 caption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I’m on the fence between weak reject and weak accept. The proposed method is sound, however I have some questions pertaining to the real-robot experiments so that readers can better understand the performance implications of the designed reward. Likewise, an honest and comprehensive review of the method's limitations is needed (the discussion around requiring human demonstrations seems like a throwaway statement), however this seems easily addressable and go a long way to help frame the paper within the context of solutions to the problem.\n\nUpdate: Bumping up to weak accept after rebuttal. The authors did a great job addressing some of my concerns in the updated version.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Issues listed in the summary above.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "4HnpR0-E10Y", "reviewer": "Reviewer_A9wW", "summary": "This paper proposes a method for learning a reward function which is both cross-embodiment and cross-domain. This allows for the reward function to be learned from human videos, for the learned reward to be used for learning in simulation, and for the learned policy to be deployed in the real world again on a physical robot. A graph representation is used to cross the domain gap, and temporal cycle consistency loss is used to cross the embodiment gap. Experiments show improved performance on a cross-embodiment benchmark, and successful deployment on a physical robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The combination of graph representations and IRL appears to be novel and is likely to have significant impact, as this general approach can be used for many tasks.\n1. The experimental finding that learned reward functions can be better than hand-designed rewards for these tasks is intriguing, and may be useful for the community.\n1. The paper does well at proposing a vision for this learning pipeline: “learning rewards from real-world human demonstrations, learning a policy in simulation using learned rewards, and finally deployment of the learned policy on physical hardware”. This seems like a promising way forward for robot learning, and therefore raises the impact potential of this work.\n1. Evaluation on a real robot shows that this method can be readily applied in the real world, supporting the vision for the full pipeline.\n1. Data efficiency. As suggested by an ablation, the graph abstraction is much more data-efficient. A comparison using this metric to other methods may be interesting.\n\nLimitations:\n1. The node features only use the bounding box and centroids of each object, in the current draft of the paper. Other visual information is discarded. This is what helps the method cross the domain gap, but also drastically cuts the number of tasks that the method can be applied for, if it does not use visual features beyond the sizes of objects. For example, if the object to be pushed and the goal are the same size, then it is unclear how the network would be able to tell them apart.\n1. X-MAGICAL is a useful benchmark, as it is designed specifically for evaluating cross-embodiment methods, but it is not very clear why strong performance in this benchmark would necessarily translate well to strong performance in real robot experiments. The policy is state-based rather than image-based. It seems like objects do not collide or interact much, so this might not show the full power of the Interaction Networks used in this method. The results on this benchmark suggest that when trained on not diverse videos but one visual style of videos, the proposed method generalises better to videos with different visual features. This makes sense because these visual features are all discarded by the representation, keeping only bounding boxes. However, in the second row of Figure 4, it seems like the baseline method XIRL performs just as well as the proposed method when the training dataset contains visually diverse videos, presumably because the visual encoder learns which features to focus on. This might lessen the impact of the current work, because it is also motivated by the context of learning from diverse videos.\n1. The real-world experiments address a scenario which also might not show the full potential of Interaction Networks, as there are only 2-3 objects involved. The results are not very convincing for the proposed method unfortunately, in the current draft. The number of demonstrations is in the range of 162-256, which is high considering the simplicity of the task. For the peg-in-hole experiment, it does not seem as though 3D geometric understanding is needed to complete the task. In the video on the website, simply moving in a 2D line without raising peg allows peg to be dragged over wall and into box, due to compliance of string. In the demo videos, the peg is lifted over the wall -- it seems like the robot does not succeed in learning this behaviour. Additionally, it is not clear how the limited visual features in the graph representation would allow this 3D information to be captured. On Line 250, it is claimed that the proposed method outperforms the oracle method. This conclusion is not fully proven by the data as it stands. This is because (a) of the variance of the reward curves and (b) because this conclusion seems highly dependent on this specific termination point for the reward curve, when before that SAC was significantly out-performing GraphIRL, and after this point it is unclear which method will perform better, since neither curve looks like it has reached convergence.\n1. The limitations section does not explore the limitations of the method in detail, such as those mentioned in the review.\n\nQuestions:\nIn Eq (4), why is g not just the final frame, but an aggregation across frames?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel idea and with high impact potential. However, there are some limitations introduced as a result of design decisions, and experimental evidence does not yet conclusively show the potential of this work. I will happily reconsider my weak reject recommendation if the visual features limitation is addressed and additional experiments demonstrate cleanly the advantages of this method.\n- - -\nEDIT: thank you to the authors for their hard work in completing the rebuttal. Taking the paper and discussion into consideration, I have now updated my recommendation to **Weak Accept**. The main reasons for this are as follows:\n* The authors have clarified how 3D reasoning can still be performed by their method. Thank you for this: I now understand and agree.\n* The authors have clarified that the object identity is included in the representation, albeit without visual features. With object identity and size, this is sufficient to solve some tasks already which are useful in the real world, and so it is sufficiently impactful as a method to merit acceptance. However, the fact remains that removing visual features is what allows this method to cross the sim2real gap, so extending this method to tasks which require the robot to use visual features will require significant changes, therefore this remains a limitation.\n* The new experiment on OOD performance shows that the prior XIRL work does struggle with overcoming these visual distractor features, so the value of the proposed method has been made clearer.\n\nAdditional comments on the rebuttal:\n* I appreciate the data efficiency experiment. This is a promising hypothesis to demonstrate an advantage of the graph representation. However, a comparison to other methods is required to demonstrate this advantage, since it is possible that the performance of other methods changes similarly when the training dataset size is changed. I understand that the authors have promised to run this later and include it in the final version if the paper is accepted, presumably due to time constraints.\n\nOverall, graph representations seem like a useful and novel direction for IRL, and so this paper will be a valuable addition to CoRL 2022.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. Line 32: another important advantage of third-person videos, such as those used in this approach, compared to demonstrations involving a robot is the availability of in-the-wild data.\n2. The description of TCC was quite concise, but it may be worth spending more space on this to explain it in more detail as it is quite key to the paper. The diagram from the original TCC paper conveys the intuition well.\n3. In Eq (3), lowercase s is not defined, it seems. Probably it should be the same as uppercase S.\n4. It may be helpful to emphasise that SAC is an oracle method in the real world experiments, in that it has access to reward function information"}}]

Author rebuttal:

Decision: Accept (oral)