VIRL: Self-Supervised Visual Graph Inverse Reinforcement Learning
Authors: Lei Huang
Weijia Cai
Zihan Zhu
Chen Feng
Helge Rhodin
Zhengbo Zou
Venue: CoRL 2024 (2024)
Keywords: Inverse Reinforcement Learning
Learning from Video
Graph Network
TL;DR: An inverse reinforcement learning method that leverages visual features and graph abstractions from videos to learn dense reward functions for reinforcement learning, demonstrating generalization to extrapolation tasks and unseen domains.

Abstract: Learning dense reward functions from unlabeled videos for reinforcement learning exhibits scalability due to the vast diversity and quantity of video resources. Recent works use visual features or graph abstractions in videos to measure task progress as rewards, which either deteriorate in unseen domains or capture spatial information while overlooking visual details. We propose $\textbf{V}$isual-Graph $\textbf{I}$nverse $\textbf{R}$einforcement $\textbf{L}$earning (VIRL), a self-supervised method that synergizes low-level visual features and high-level graph abstractions from frames to graph representations for reward learning. VIRL utilizes a visual encoder that extracts object-wise features for graph nodes and a graph encoder that derives properties from graphs constructed from detected objects in each frame. The encoded representations are enforced to align videos temporally and reconstruct in-scene objects. The pretrained visual graph encoder is then utilized to construct a dense reward function for policy learning by measuring latent distances between current frames and the goal frame. Our empirical evaluation on the X-MAGICAL and Robot Visual Pusher benchmark demonstrates that VIRL effectively handles tasks necessitating both granular visual attention and broader global feature consideration, and exhibits robust generalization to $\textit{extrapolation}$ tasks and domains not seen in demonstrations. Our policy for the robotic task also achieves the highest success rate in real-world robot experiments.

VIRL: Self-Supervised Visual Graph Inverse Reinforcement Learning
Lei Huang', Weijia Cai', Zihan Zhu?, Chen Feng Helge Rhodin Zhengbo Zou CColumbia University "Brown University "NYU 'BC 'Bielefeld University
Abstract: Learning dense reward functions from unlabeled videos for reinforce- ment learning exhibits scalability due to the vast diversity and quantity of video resources. Recent works use visual features or graph abstractions in videos to measure task progress as rewards, which either deteriorate in unseen domains or capture spatial information while overlooking visual details. We propose Visual- Graph Inverse Reinforcement Learning (VIRL.), self-supervised method that synergizes low-level visual features and high-level graph abstractions from frames graph representations for reward learning. VIRL utilizes visual encoder that extracts object-wise features for graph nodes and graph encoder that derives properties from graphs constructed from detected objects in each frame. The encoded representations are enforced to align videos temporally and reconstruct in-scene objects. The pretrained visual graph encoder is then utilized to con- struct a dense reward function for policy learning by measuring latent distances between current frames and the goal frame. Our empirical evaluation on the X-MAGICAL and Robot Visual Pusher benchmark demonstrates that VIRL ef- fectively handles tasks necessitating both granular visual attention and broader global feature consideration, and exhibits robust generalization to extrapolation tasks and domains not seen in demonstrations. Our policy for the robotic task also achieves the highest success rate in real- world robot experiments. Project website: https://leihhhuuan github. io/VIRL/.
Keywords: Inverse Reinforcement Learning, Learning from Video
Introduction
Intelligent agents have mastered complex tasks by learning policies through reinforcement learning (RL), particularly in the gaming domain, where their performance can match or even surpass cham- pion level [1, 3]. Constructing precise dense reward function fundamental and indispensable for appropriately training agent for task, it succinctly defines the task master [4] and provides dense signals for faster policy learning [5, 6].
However, hand-designing such functions requires substantial domain knowledge and extensive en- gineering efforts, making it unscalable given the myriad tasks in the real world [7]. Moreover, the process of fine-tuning reward functions [8] susceptible to overlooking corner cases, leading to agent misbehavior due to unintended sub-optimal rewards [9]. Recently, large language models (LLMs) have been leveraged assist in generating executable reward functions for RL [10, 11], but they necessitate appropriate prompts and function templates from domain experts and involve considerable training time due to the evolutionary search process.
On the other hand, readily available video demonstrations contain implicit dense rewards the form of continuous progress towards goals. After watching videos, one can easily estimate how much progress is achieved given frame and the goal. To this end, inverse reinforcement learning (IRL) has been proposed [12, 13, 14, 15, 16] to learn reward functions from demonstrations by utilizing features of environments and agents. Compared to those requiring ground-truth actions [17, 18) or
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Demonshater
Desasti
tasks due to its (a) Extrapolation Task
(b) deppeeeee Task
(c) Cross-domain Task
Figure 1: Capabilities of VIRL: (a) Generalizable to task extrapolations; (b) Adaptable to visual- reature-dependent task, such pushing debris with identical shapes; (c) Robust to domain shift. Figures on the top are unlabeled video demonstrations and those on the bottom are tasks for imitators to learn. X-MAGICAL benchmark contains 4 embodiments: short-stick, medium-stie stsik,, long-stick as shown in (a) top from left to right, and gripper in the bottom. All tasks are set to be cross- embodiment. Note: Long-stick agent is unsuitable for (b) excessive length, which hinders task execution.
inferred actions from dynamic models [19, 20], IRL algorithms for action-free videos [21, 22, 23] have garnered greater attention recently due to their scalability and lower cost of data collection.
Recent IRL works [23, 24] achieve impressive performance in building task-specific embodiment- agnostic reward functions using encoders pretrained by temporally aligning videos of varied lengths [25]. Agents are encouraged to reach the provided goal in the latent space by receiving rewards based on the distance between the embeddings of the current state and the goal state. However, employing visual encoder [23] suffers from performance deterioration in unseen domains; and utilizing graph abstractions of object coordinates [24]. while robust domain shifts, sacrifice visual features, which task-specific, prior works have not been tested beyond demonstrated tasks, such as extrapolations, which may pose challenges even with a minor change in object counts [26]. Therefore, we are interested in two questions: (i) How can we leverage both low-level visual features and high-level graph abstractions to adapt to both scometry-awar and -unaware tasks? (ii) Can a single pretrained encoder be used to build reward functions of tasks beyond those it is trained on, such as task extrapolations?
be crucial for certain tasks. Additionally, since their representations
To this end, propose visual graph encoder that combines visual encoder for capturing object- centric low-level features with graph encoder for extracting scene-level abstractions. By formu- lating abject-centric graphs that are generalizable across various object counts, we demonstrate that the pretrained visual graph encoder can construct reward functions for tasks of manipulating an un- seen number of objects. This capability is inspired by the adaptability and ficxibility of graph neural networks encode graph structures with varying numbers of nodes. Figure illustrates the main properties of our approach, and Table 2 in Appendix A.1 compares advantages over related works.
Related Work
For more comprehensive related work, please see Appendix A.2.
Imitation learning with actions. Imitation learning from expert demonstrations with state and action pairs has demonstrated successes in various approaches [27]. from behavioral cloning [28, 29, 30] to inverse reinforcement learning [17, 31, 32). However, its reliance on ground-truth actions renders data acquisition at scale both costly and challenging, thereby limiting its wider applicability. To leverage unlabeled video demonstrations that are easier to obtain, methods have been developed to infer actions in videos [33, 34, 35, 36, 37, 38, 39]. For example, VPT [39] trains an inverse dynamics model (IDM) on small labeled dataset and uses the trained IDM to augment a large number of unlabeled videos with actions, which used for the subsequent policy learning via behavioral cloning. These methods have successfully trained agents using unlabeled videos, but

---

subject to well-defined morphologies and dynamics of demonstrators and imitators, making it hard to leverage demonstrations with diverse embodiments.
Domain adaption for imitation learning. To overcome the domain gap between varied agent em- bodiments, several works [40, 41, 42, 43, 44, 18] attempt translate the context in demonstrations to the imitators' context in terms of agents and viewpoints, using methods such as generative adver- sarial networks [45, 46] and context translation. Furthermore, another way to address the domain gap between agents is by inpainting them from video demonstrations and online visual states [47].
Inverse reinforcement learning from videos. On the other hand, a group of works [21, 22, 48, 49, 50, 44, 51, 52, 53, 24] has focused on learning reward functions from videos by extracting latent features that indicate intermediate steps measure task progress, which are naturally and implicitly present in demonstrations. Learned reward functions then used for policy training in the regular RL regime. To train on videos with varied lengths and paces, XIRL [23] adopts temporal cycle-consistency loss [25] to learn visual embeddings that measure task progression and then uses distances between current state embedding and the goal embedding as reward signal for policy learning. They demonstrate the pretrained encoder benefits from the diversity of demonstrators' embodiments and generalizes to unseen embodiments. Building on XIRL, our work seeks to train an encoder that generalizes constructing reward functions for task extrapolations by leveraging graph abstractions, while maintaining the flexibility for tasks requiring attention to low-level visual features by encoding object features.
Object-centric scene graphs. Abstracting visuals into object-centric scene graphs is beneficial for agents to understand environments and make decisions various tasks [54], such as dynamies mod- cling [55, 56], navigation 58, 59], and robotic manipulation [60, 61, 62, 43, 63, 64]. Following up XIRL, GraphIRL [24] enhances robustness to diverse object appearances by abstracting frames into graphs, where each node represents object and contains features including its bounding box coordinate and distances to other objects. However, limitations arise when tasks require attention to object geometries or when dealing with different numbers of objects. Motivated by their work and aiming to address the limitations, we encode object visual features and locations into nodes in a unified way and build graphs such that the encoder can generalize scenes with arbitrary num- bers of objects. Our work can be considered generalized version of GraphIRL, suitable for both geometry-aware and -unaware tasks, well unseen task extrapolations.
3 Method
Our objective is to learn reward functions from video demonstrations showcasing a specific task, enabling an agent with a different embodiment to learn the task and its extrapolations where there unseen number of objects manipulate (Section 3.1). Our approach involves phases: Firstly, pretraining visual graph encoder (Section 3.2) capture object-wise visual features and global scene features in a self-supervised manner (Section 3.3); Secondly, cross-embodiment policy learning for tasks by training RL algorithms with dense reward functions built on the pretrained encoder (Section 3.4). The overall framework of VIRI. is depicted in Figure 2.
3.1 Problem formulation
VIRL takes as input a dataset of action-free videos DJ = U VeT of the identical task T, where vee is the ttth video demonstration conducted by agent with embodiment A vidco is defined to be a sequence of image frames ye,T = { 12, I where L3 is h*h video's length. We design visual graph encoder dug to transform DT into dataset of graph sequences GT U G where G = { "h.,.., Gh. (h,,), }, and extract graph encodings. We pretrain the encoder via tem- poral alignment of graph encoding sequences and pixel reconstruction of manipulable objects. The pretrained encoder then utilized construct general-form dense reward functions that gauge task progress through lowelevel visual and high-level structural features in graphs. It merits attention that the visual graph encoder's versatility allows it to not only formulate reward function for the
3

---

Pretraining Visual Graph Encoder
RL with Learned Encoder
Crops
Merge
Reconstruction Targel
Reconstruction from Graph Embedding
Goal Frame
Delection
ERooss
VS
Features
Object
Construct
Graph
Reconstruction
Decoder
Frame
Loss
Action
Visual Graph Encoder
Video Video
Embedding Space
Visual
Graphify
Graph
Module
Encoder
Temporal Cycle- Consistency Loss
Reward (Embedding
Visual Graph Encoder
Distanos)
Figure 2:
Overview of Visual Graph Inverse Reinforcement Learning. We transform videos to graph sequences containing both low-level visual features and high-level global features, pretrain visual graph encoder via temporal alignment and object-centric scene reconstruction, and build reward functions using the visual graph encoder for unseen embodiment to learn the demonstrated task and task extrapolations with reinforcement learning.
demonstrated task T but also for its extrapolation tasks T" in zero-shot fashion, attributable to the encoder's 's capability process variable-sized graphs. We define extrapolation tasks T" to be those where agents manipulate more objects than demonstrated, which are unseen in videos.
3.2 Visual graph encoder
We propose a visual graph encoder composed of visual graphify module and graph encoder. The visual graphify module builds object-centric graphs from images by encoding the spatial and visual features of each object in an image into graph node features and objects interactions into edge features. Then, the graph encoder takes graphs and outputs the representations, which will be trained via temporal alignment in the latent space and reconstruction of object-centric frames.
Visual graphify module. We use the proposed visual graphify module to transform every quence of frames vel = {It I .i..... 12. in the dataset D to a sequence of graphs G+." (gf, C..., G get GT. A graph is defined as G. (V,E), where V is the node set sponding objects in the frame and & is the edge set indicating the relationships between objects. For every frame 1 E R xxxxxx xC an off-the-shelf detector is employed to get N bounding boxes of N objects in the scene. Contrary to approaches that extract visual and positional features using CNN-based encoder and an MLP separately [65, 66], we encode both features from regions defined by bounding boxes using shared CNN-based encoder an unified way. Specifically, for each ob- ject in image, the pixels within bounding box is kept while all the other pixels being masked so that the positional information is encoded implicitly. After this operation, an image is transformed to stack of images I E N each of which contains only one object and is then encoded node using the shared ResNet-based [67]
into an object feature fo, E Pobi as the feature visual encoder. The distance between two objects da.; E Ih represents the edge feature connect- ing the node pair. We consider all graphs to be fully connected and undirected, allowing for direct interactions between all objects.
Apart from the transformation from images to graphs, we obtain the reconstruction target from the stack of object images T ensure that the embedding after the graph encoder can recover sufficient low-level visual features of objects. It contributes our work's advantage of only imitating the spatial interactions between objects but also capturing the low-level visual features of manipulative objects. For example, in the context of the Sweep-to-Goal task in X-MAGICAL. [23], variety of

---

task variants can be generated, such as pushing multiple objects irrespective of their appearance and selectively pushing duplicates out three objects present a scene. With this motivation, we merge all non-agent images in stack, i.e., I5, E (R xXx into single RGB image to generate the reconstruction target I for the original frame I
Graph transformer encoder. Subsequent to obtaining G". we use graph network to extract frame embeddings for temporal alignment and reconstruction. Considering the impressive perfor- mance of Transformer models in computer vision and natural language processing [68, 69, 70], we employ Graph Transformer [71] network as the encoder. Given a graph with node features H = (hi, h/. hyy }or and edge features E = j E [N] j}, using the graph transformer operator by:
we update the node features
anj (ugWW))) Wyd,)), where Oij = a (W/D)) (W vd
+ Wydde)
h h2+1 = w/nh
Σ JEN.
All Ws are different trainable parameters. di is the attention coefficient and a is the softmax function. After the graph transformer operations, we apply a global mean pooling layer to extract the average of node encodings as the final embeddings for temporal alignment and reconstruction.
3.3 Loss function for pretraining
The visual graph encoder is trained in self-supervised way. Given each video V, we obtain sequences of graph embeddings (V) using the proposed visual graph encoder. For simplicity, we omit subscripts and superscripts for video index k, embodiment and task T. We adopt TCC loss [25] to temporally align the graph embeddings between sampled videos for measuring task progress. Meanwhile, we enforce the embeddings to keep objects' low-level visual features by adopting reconstruction loss, which is inspired by He et [72] and Ye et [62].
Temporal alignment. Given a pair of randomly sampled videos U and V. the visual graph en- coder computes their embeddings (U') = (u1, U2,.... "Lg) ) and Φeg(V) To compute the TCC regression loss, we randomly select an embedding u, in &ug (U) soft nearest neighbor in (V) by estimating the similarity between uy and every embedding in (V) and conducting weighted average over (Vy(V):
(e1, U2, UL
(U) and find its
Ly where aj
||:
Σ
ajuj,
Σ J.v 1144-1/1
We then operate cycle back from and find the index of U's soft nearest neighbor in Φ (U):
Ly Σ Baki where Be 110--1111
Σj e-11--ay112
Since the index of the starting point u; is known to be i, the loss can be computed using the squared distance (īi 10-1] i)?. Additionally, variance o² = Et"" 3x (I: - 72)2 is added to the loss function as a regularization term make predictions less dispersed [25]. Thus, the final objective function for temporal alignment is Ltoe (i-7)² Alog(a), where À is constant weight.
Object-centric reconstruction. When coming to tasks that require reasoning about low-level fea- tures, such as pushing duplicates among several objects, we want the visual graph encoder to keep as much object-centric low-level information as possible [73]. Thus, we define decoder Dec (og (V)) that maps from graph embeddings back frames. We would like highlight that, instead of re- constructing the original frame IA, our reconstruction target is It¹ generated in the visual graphify module as introduced in Section 3.2. which keeps pixels within the bounding boxes of - objects, excluding the agent. We define the reconstruction loss as Lyee = !| Dec( (0gg (1/)))
Finally, we combine the two losses to get the overall loss function for pretraining the visual graph encoder: L = Lice d Lrecs where d is a weight term.
5

---

3,4 Reinforcement learning with learned reward
We use the visual graph encoder &ug pretrained on videos of task T to build reward functions for and its extrapolations T", which are then used for an agent with an embodiment different from demonstrators to learn the tasks with RL algorithms. Since the embedding sequences are tempo- rally aligned in the embedding space, can build reward function for a task by measuring the current progress in the embedding space [23), i.e., the negative distance between the current frame embedding (1) and the goal embedding g: r(1) - -1/s||0e,(()) =llă. where g is defined by the average embedding of all last frames in videos, and is parameter that rescales the reward to reasonable range for policy learning [74]. For the most challenging extrapolation task where agents learn to push 3 debris from videos of pushing debris, combining our approach with a reformated reward function: r(/) = 1/s x [gg(loog (1) - g 12 + 1). can achieve better performance.
Compared to the most related prior works [23, 24). the main advantages of building reward functions using our approach are twofold: (i) it reserves the ability to learn tasks that require attention to low- level features while maintaining generalization to unseen domains by leveraging graph abstraction, (ii) apart from the demonstrated task 7, it generalizes better to task extrapolations 7' because of the flexibility of the encoder to consume graphs consisting of various numbers of objects.
Experiments
We conduct experiments on the Sweep-to-God task in X-MAGICAL [75, 23] and Robot Visual Pusher [44, 24] benchmark to answer following questions: (i) Can pretrained visual graph encoder construct reward functions for extrapolations of the demonstrated task in zero-shot? (ii) Does VIRL retain low-level visual features for learning? (iii) How does VIRL perform in shifted domains !? We also deploy policies trained in simulation real robot for robotic tasks. For implementation details, please see Appendix A.3.
4.1 Experiment setup
Baselines. We compare our work against self-supervised approaches that learn reward functions from unlabeled videos: (i) XIRL [23] pretrains task-specific visual encoder using TCC loss on videos and builds reward functions using the pretrained encoder. (ii) GraphIRL [24] transforms videos to graphs of object coordinates, uses a spatial interaction graph encoder extract features, and follows similar pipeline of XIRL. (iii) TCN [22] uses triplet loss pretrain representations on videos such that frames the same time are close in the embedding space while those that are temporally different are far apart. (iv) LIFS [73] trains an invariant feature space to transfer skills between agents with different embodiments using a contrastive loss and a reconstruction loss.
Task descriptions. Sweep-to-Goal in X-MAGICAL. All tasks are set to be cross- eooss-mmbod,eet, i.e., an agent learns from videos of other 3 agents. Task (Figure la): learning to sweep 3 debris the colored goal region the top. given videos of sweeping I or debris. Task II (Figure lb): learning to sweep 2 debris with the same shape to the goal region while leaving the rest outside, given videos of the same task. Task III (Figure 1c): learning to sweep 3 debris regardless of shapes and colors, given videos of sweeping squares. Robot Visual Pusher tasks. Task (Figure 5): xArm learning to push an object to the goal, given videos of the same task by human. Task II: xArm learning to push specific object out of objects in-scene to the goal, only given the set of human videos Task
4.2 Experiment results on X-MAGICAL
Results in extrapolation tasks. We validate a key feature of our method: the visual graph encoder's ability to construct reward functions for extrapolations of the demonstrated tasks and train agents to manipulate an unseen number of objects. We hegin by testing the results of policy learning from videos showcasing cross-embodiment demonstrations of pushing 2 debris. We then increase the
For experiment results of X-MAGICAL in domain shift environment, please see Appendix A.4

---

VEL OOrr
XPL
TOM
2001
2000
Laggor
Figure 3: Cross-embodiment extrapolation tasks. Learning push 3 debris given videos of push- ing 2 debris (top), and videos of pushing debris (bottom). Note: GraphIRL is not applicable when agents are learning to manipulate different number of objects than in the video demonstrations.
Figure 4: Cross-embodiment risual-feature-deeeedett task. Our approach can be applied for visual-featus seaal-ffatue--eppeee dependent tasks. For example, pushing debris with the identical shape out of 3. Note: long-stick is not included for this task because its excessive length hinders the task execution.
difficulty by providing demonstrations of pushing only debris, as shown in Figure Ia (top). In Figure (top), we illustrate that in the first scenario, our approach outperforms baselines for out of 4 embodiments and is on par with XIRL for medium-stick. In the more challenging scenario, where demonstrators push debris, policy learning becomes significantly harder, is not demonstrated that returning to fetch additional debris leads to greater rewards in the long term. Figure (bottom) shows that our approach generalizes to manipulate the unseen number of objects with higher success rates than others. Surprisingly, TCN outperforms XIRI. in the most challenging setting and delivers performance comparable to our approach for the gripper and long-stick configurations.
Results in fisual-featrre-deeeeeee task. We evaluate the second key feature of our method: the adaptability
tasks by encoding visual features of objects into nodes. The task for imitators to learn is identical to what is demonstrated, can be anticipated that the performance of XIRL should be comparable to our approach. Figure tasks, achieving similar
shown in Figure 1b. Hence, it
reveals that our approach is capable of policy learning for geometry-awan performance to XIRL for two embodiments, yet slightly lower for medium-stick. On the other hand, GraphIRL shows inferior performance in this task because of the lack of visual information.
4.3 Experiment results on Robot Visual Pusher
Learning robotic manipulation from human videos. We build reward functions on encoders pretrained on human videos and use them for policy learning of demonstrated task. Task in Figure 5 shows the result. Although XIRL can improve the sample efficiency when combined with sparse
7

---

Human video
Task!
Deria Paward
XPL
Canoe Firenn) Qupet, apficiii
Figure 5: Robot Visual Pusher. Task robot learns the demonstrated task. Task II: robot learns variant task push a specific object among objects to goal. Thresholds for success, measured by distance between positions of object and goal, are 5cm for Task and 10cm [24] for Task II.
environment reward [23], we notice that its Iearned rewards alone are insufficient for policy learning. In contrast, our approach provides a learned reward function closest dense environment rewards.
Learning visual-featur--eeeedenn task variant. We the same encoders in the first task to build reward functions for the task variant: pushing a specific object among two objects to the goal. It merits attention that GraphIRL is not applicable for this task since it encodes object counts in demonstrations into node features and cannot generalize to scenarios with unseen numbers of objects. The result of learning Task in Figure 5 shows that our approach VIRL surpasses the performance of using handerafted dense environment rewards, while hard for agents learn functional policies given reward signals from the baseline XIRL.
Additionally, we ablate the detector's s accuracy to study the impact of missing detection in both robot tasks. For experiment details and results, please see Appendix A.5.
Sim2Real transfer. We evaluate policies trained in simulation, without domain randomization, the Robot Visual Pusher Task II. Each policy runs for 20 trials: the green object is placed on the left and the red on the right for the first 10 trials, then their positions are switched. Results are in Table 1. The performance gap between simulation and real robot likely stems from unaligned physics properties of objects and lack of domain randomization. For details of robot setup and episode visualization, please see Appendix A.6.
real-world xArm 6 robot in
Method
Success rate
VIRL (ours)
0.55
XIRL
0.05
Env. reward GraphIRL
0.2
Table 1: Real robot test results.
Discussion
Conclusion. We present VIRL, an IRL method utilizing visual graph encoder to construct reward functions for demonstrated tasks in videos and their extrapolation variants. VIRL integrates object- level spatial and visual features unifiedly into nodes for graph construction from frames. Pretraining the visual graph encoder is self-supervised using TCC loss and reconstruction loss. The design of the visual graph encoder enables effective reward function generation for the demonstrated task and extrapolations involving unseen object numbers, which is challenging in robotic tasks [26]. Additionally, our approach is flexibly adaptable for risual-feature-deeenee
tasks.
Limitations. Our approach depends on reliable object detectors. While occasional detection failures have minimal impact, missing detection in keyframes can be critical. Additionally, the tested tasks don't fully capture the complexity of kinematics and scene variations in benchmarks like RoboSuite [76], Meta World [77]. and BridgeData V2 [78]. Third, remains unclear how VIRL would general- ize to deformable object manipulation [79, 80]. Lastly, focusing only on object-centric information may limit performance on tasks requiring broader scene context."
For comprehensive discussion of limitations, please Appendix A.9

---

Acknowledgments
We would like to thank the reviewers for the insightful comments and thank Omar Swei, Renjie Liao, Tengda Han, Qi Zhu, Runsheng Xu, and Binghao Huang for the fruitful discussions.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-533, Feb 2015. ISSN 1476-4687.
[2] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche. J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587)4444488) Jan 2016. ISSN 1476-4687.
[3] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T.. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782)350-334 Nov 2019. ISSN 1476-4687.
[4] Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, Z. Zhao, et al. Toward general-purpose robots via foundation models: A survey and meta-analysis. arXiv preprint arXiv:2312.08772. 2023.
[5] A. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, page 278-287, San Francisco, CA, USA, 1999. ISBN 1558606 122.
[6] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
[7] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06177 2022.
[8] S. Booth, W. B. Knox, J. Shah, S. Niekum, P. Stone, and A. Allievi. The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifications. Proceedings of the AAAI Conference on Artificial Intelligence, 3765):5920-5925) 5929, Jun. 2023.
[9] J. Clark and D. Amodei. Faulty reward functions in the wild. Internet: https://blog. openai. com/faulty- fennrdiitttttiitnoooonn 2016.
[10] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08666 2023.
[11] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models, arXiv preprint arXiv:2310.1299: 2023.
[12] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, page 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
[13] Abbeel and A. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning. ICML '04, page New York, NY, USA. 2004. Association for Computing Machinery. ISBN 1581 138385. 15...1114//111130.101111
[14] U. Syed and E. Schapire. game-theoretic approach apprenticeship learning. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Sys- tems, volume 20. Curran Associates, Inc., 2007.

---

[15] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforce- ment learning. In Proceedings of the Twenty-Thrr AAAI Conference on Artificial Intelligence volume 8, pages 1433-1438, 2008.
[16] D. Hadfield-Menell, Milli, P. Abbeel, S. Russell, and A. D. Dragan. Inverse reward design. In Proceedings of the 31st International Conference Neural Information Processing Sys- Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN
tems, NIPS' 17, page 9781510860964.
[17] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learn- ing Research, pages 49-58, New York, New York, USA, 20-22 Jun 2016. PMLR.
[18] A. Fickinger, S. Cohen, S. Russell, and B. Amos. Cross-domain imitation learning via optimal transport. International Conference on Learning Representations, 2022. URL https: /ooenreview http://oroummidxxppppppe2
[19] A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell. Imitating latent policies from observation. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Confer- ence on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1755-1763 PMLR, 15-15/ Jun 2019.
[20] Radosavovic, X. Wang. L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7865-7871, 2021.
[21] P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. In Proceedings of Robotics: Science and Systems, Cambridge, Massachusetts, July 2017. doi:
[22] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141, 2018.
[23] K. Zakka, A. Zeng, Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 537-546. PMLR, 08-11 Nov 2022.
[24] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang. Graph inverse reinforcement learning from diverse videos. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 55-66. PMLR, 14-18 Dec 2023.
[25] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), June 2019.
[26] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In arXiv,, 2024.
[27] т. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
[28] D. A. Pomerleau. Alvinn: an autonomous land vehicle in a neural network. In Proceedings of the 1st International Conference Neural Information Processing Systems, NIPS 88, page 305-313, Cambridge, MA, USA, 1988. MIT Press.
10

---

[29] Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In G. Gordon, D. Dunson, and M. Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Pruceedings of Muchine Learning Research, pages 627-635, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: /tt.ttss://pproeeeeeee http:///poceeeen..ce/e . mlr pptsss/11/rossstaa....t html.
[30] A. 7hou, M. J. Kim, L. Wang, P. Florence, and C. Finn. Nerf in the palm of your hand: Cor- rective augmentation for robotics via novel-view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17907-17917, June 2023.
[31] J. Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[32] Haldar, Mathur, D. Yarats, and L. Pinto. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning, pages 32-43. PMLR, 2023.
[33] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, J/,A--8,, pages 4950-4957. International Joint Conferences on Artificial Intelligence Organization, 7 2018. doi: 1oi:10024963/jjca..2201 2010///0///...2//3.033 2018/687. URL atttps://don org/10. 24963/ ijcai. 2018/687.
[34] D. Pathak, Mahmoudieh, M. Luo, P. Agrawal, Chen, F. Shentu, E. Shelhamer, J. Ma- lik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In International Conference on Learning Representations, 2018. URL https: http///ppnnreiiieeee http://perrreeeeeeeeeereeeeeee httpforrrumiddBiiisuuuWWnW
[35] C. Yang, X. Ma, W. Huang, F. Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Advances in neural information processing systems, 32, 2019.
[36] A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell. Imitating latent policies from observation. In International conference on machine learning, pages 1755-1763. PMLR, 2019.
[37] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2Robot, Learning Manip- ulation Concepts from Instructions and Human Demonstrations. Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. 200:10..6677/RSSS200000111
[38] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7865-7871. IEEE, 2021.
[39] B. Baker, Akkaya, P. Zhokov, Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639- 24654, 2022.
[40] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Roboties and Automation (ICRA), pages 1118-1125 IEEE, 2018.
[41] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hier- archical controller. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[42] L. Smith, N. Dhawan, M. Zhang, Abbeel, and Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. In Proceedings of Robotics: Science and Systems, 07 2020. 0000:0...577RSS.2200001222.2

---

[43] M. Sieb, Z. Xian, A. Huang. O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. In I.. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 979-989. PMLR, 30 Oct-01 Nov 2020.
[44] K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 339-354. PMLR, 16-18 Nov 2021.
[45] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE international conference on com- puter vision, pages 2223-2232, 2017. doi: 110100111///CC22211...22
[46] Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and Darrell. Cy- CADA: Cycle-consistent adversarial domain adaptation. In Dy and A. Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pages 1989-1998. PMLR, 10-15 Jul 2018.
[47] S. Bahl, A. Gupta, and D. Pathak. Human-to-BR Robott Imitation in the Wild. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. 1oi:10.15607/RSS 2022.XV111.06.
[48] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas. Playing hard explo- ration games by watching youtube. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[49] A. Singh, L. Yang, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgauu Germany, June 2019. doi: 10.116677/RSSS.2191..0000
[50] O. Mees, M. Merklinger, G. Kalweit, and W. Burgard. Adversarial skill networks: Unsuper- vised robot skill learning from video. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4188-4194, IEEE, 2020.
[51] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj. S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page 7827-7834. IEEE Press, 2021. 11099100100/1 0.666.60..
[52] Y. Lee, A. Szot, S.-H. Sun, and J. J. Lim. Generalizable imitation learning from observa- tion via inferring goal proximity. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 11-16130.. Curran Associates, Inc., 2021.
[53] A. S. Chen, S. Nair, and C. Finn. Learning Generalizable Robotic Reward Functions from "In-The-Wilde Human Videos. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi:10. 1ww.1..1.......2000 202.I10000....
[54] D. Bear, C. Fan, D. Mrowca, Y. Li, S. Alter, A. Nayebi, J. Schwartz, L. F. Fei-Fei, J. Wu, Tenenbaum, and D. L. Yamins. Learning physical graph representations from visual scenes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6027-6033 6039. Curran Associates, Inc., 2020.
[55] Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, and k. kavukcuoglu. Interaction net- works for learning about objects, relations and physics. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gamett, editors, Advances in Neural Infommtion Processing Systems, vol- ume 29, Curran Associates, Inc., 2016.
12

---

[56] H. Qi, X. Wang, D. Pathak, Y. Ma, and J. Malik. Learning long- tenm visual dynamics with region proposal interaction networks. arXiv preprint arXiv:2008 2020.
[57] W. Yang. X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. CoRR, abs/1810.06543, 2018, URL http: http::tttt/sssete wwtps/ivw.rrg/b////// 201./.111.00533.
[58] S. Zhang, X. Song, Y. Bai, W. Li, Y. Chu, and S. Jiang. Hierarchical object-to-zone graph for object navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15130-15140, October 2021.
[59] N. Hughes, Y. Chang, and L.. Carlone. Hydra: A real-time spatial perception system for 3D scene graph construction and optimization. Robotics: Science and Systems (RSS), 2022.
[60] A. Sanzhez-Gonzalez, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pages 4470-4479. PMLR, 10-15 Jul 2018.
[61] Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba. Learning compositional koopman operators for model-based control. In International Conference on Learning Representations, 2020.
[62] Y. Ye, Gandhi, A. Gupta, and Tulsiani. Object-centric forward modeling for model predictive control. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 100-109, PMLR, 30 Oct-01 Nov 2020.
[63] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu. Hierarchical planning for long- long-horizon manip- ulation with geometric and symbolic scene graphs. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6541-6548 IEEE, 2021.
[64] H. Jiang, B. Huang, R. Wu, Z. Li, S. Garg, H. Nayyeri, S. Wang, and Y. Li. Roboexp: Action- conditioned scene graph via interactive exploration for robotic manipulation. arXiv preprint 240. 15487, 2024.
[65] P.-E. Sarlin, D. Donone T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2020.
[66] C. Elich, I. Armeni, M. R. Oswald, M. Pollefeys, and J. Stueckler. Learning-based relational object matching across views. In 2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pages $999 6005, 2023. doi:
[67] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[68] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[69] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pages 4171-4188 Minneapolis, Minnesota, jun 2019, Association for Computational Linguistics.
13

---

[70] A. Dosovitskiy, L. Beyer. A. Kolesmikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x words:
Transformers for image recognition at scale. ICLR, 2021.
[71] Y. Shi, Z. Huang, S. Feng, H. Zhong, W. Wang, and Y. Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Z.-H. Zhou, editor, Proceed- ings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
[72] K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, June 2022.
[73] A. Gupta, C. Devin, Y. Liu, Abbeel, and S. Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In International Conference Learning Repre- seniations, 2017.
[74] Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. Proceedings of the AAAI Conference Artificial Intelligence, 32(1), Apr. 2018.
[75] S. Toyer, R. Shah, A. Critch, and S. Russell. The magical benchmark for robust imitation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18284-18295. Curran Associates, Inc., 2020.
[76] Y. Zhu, J. Wong, A. Mandlekar, R. Martín- A. Joshi, S. Nasiriany, and Y. Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[77] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019, URL https: / //a///aaaa /twxxv.xvvv////////////a/////aa/ . httgg//b//9990.. 10897.
[78] H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, Hansen-Estruch, Q. Vuong, A. He, V. Myers, K. Fang, Finn, and Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[79] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for ma- nipulating rigid bodies, deformable objects, and fluids. In International Conference on Learn- ing Representations, 2019. URL https: http:://pppeeeveeeeeeee
[80] Weng, S. Bajracharya, Y. Wang, K. Agrawal, and Held. Fabricflownet: Bimanual cloth manipulation with flow-based policy. In Conference on Robot Learning, 2021.
[81] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[82] D. P. Kingma and Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[83] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[84] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
14

---

[85] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2024.
[86] G. Jocher, A. Chaurasia, and J. Qiu. Ultralytics YOLO, Jan 2023, URL https://github. cow//ultrralyyylc///ttaaaatttee
[87] N. Watters, D. Zoran, T. Weber, Battaglia, Pascanu, and A. Tacchetti. Visual interaction networks: Learning physics simulator from video. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceeete neurips @cab0oasSccd2111cc3300000eee-eed pdf.
[88] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenbom, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, 7. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby. Simple open-vocabulary object detection. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassmer, editors, Computer Vision - ECCV 2022, pages 728-755, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20080-9)
[89] T. Cheng, L. Song, Y. Ge. W. Liu, X. Wang, and Y. Shan. Yolo-world: Real-time open- vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16901-16911, 1, June 2024.
15

---

A Appendix
A.1 Property Comparison between VIRL and Baselines
Method
Extrapolation
-eatur-e dependent task
Task in shifted domain
Cross- embodiment
task
XIRL [23]
GraphIRL [24]
X
VIRL (ours)
Table Property comparison different task settings between VIRL and the most related works.
A.2 Extended Related Work
Imitation learning with actions. Imitation learning from expert demonstrations with state and action pairs has demonstrated successes in various approaches [27], from behavioral cloning [28, 29, 30] inverse reinforcement learning [17, 31, 32]. However, its reliance on ground-truth actions renders data acquisition at scale both costly and challenging. thereby limiting its wider applicability. To leverage unlabeled video demonstrations that easier to obtain, methods have been developed to infer actions in videos [33, 34, 35, 36, 37, 38, 39]. For example, Pathak et al. [34] utilize forward model so that, given the current state and the policy's predicted action, it outputs the next state. With the state sequences in state-only demonstrations, both the model and the policy can be trained using the difference between the forward model's 's output and the ground truth of the next state. A similar forward dynamics model also used in Edwards et [36]. another example, VPT [39] trains an inverse dynamics model (IDM) on a small labeled dataset and then uses the trained IDM augment large amount of unlabeled videos with actions, which are used for the subsequent policy learning via behavioral cloning. These methods have successfully trained agents using unlabeled videos, but are subject to well-defined morphologies and dynamics of demonstrators and imitators, making it hard to leverage demonstrations with diverse embodiments.
Domain adaption for imitation learning. To overcome the domain gap between varied agent em- bodiments, several works [40, 41, 42, 43, 44, 18] attempt translate the context in demonstrations to the imitators' context in terms of agents and viewpoints, using methods such as generative adver- sarial networks [45, 46] and context translation. Furthermore, another way to address the domain gap between agents is by inpainting them from video demonstrations and online visual states [47].
Inverse reinforcement learning from videos. On the other hand, a group of works [21, 22, 48, 49, 50, 44, 51, 52, 53, 24] has focused on learning reward functions from videos by, instead of us- ing any additional models for action prediction or domain translation, extracting latent features that indicate intermediate steps measure task progress, which are naturally and implicitly present in demonstrations. These learned reward functions are then used for policy training in the regular RL regime. For example, Aytar et al. [48] pretrain visual encoder on classifying temporal distances between frames in a single video, retrieve a sequence of embedding checkpoints in the video, and then generate reward signals by comparing observations and embedding checkpoints. For learning robotic tasks from human videos, Sermanet al. [22] train viewpoint-invarre pairs of simultaneous videos with an objective of attracting positive frame pairs while repulsing neg- ative frame pairs in the latent space. They demonstrate that the agent-invarian representations can directly provide reward signals for robot arms. To handle video demonstrations with varied lengths and paces, XIRL [23] adopts temporal cycle-consistency loss [25] to learn visual embeddings that measure task progression and then uses distances between current state embeddings and the goal embedding as reward signal for policy learning via RL. They demonstrate the pretrained encoder benefits from the diversity of demonstrators' embodiments and generalizes to agents with unseen embodiments. Building on XIRL, our work seeks to train encoder that generalizes construct- ing reward functions for task extrapolations by leveraging graph abstractions, while maintaining the flexibility for tasks requiring attention to low-level visual features by encoding object features.
representations on
16

---

Object-centric scene graphs. Abstracting visual scenes into object-centric scene graphs is benefi- cial for agents to understand environments and make decisions in various tasks [54], such as dynam- ies modeling [55, 56], navigation [57, 58, 59], and robotic manipulation [60, 61, 62, 43, 63, 24, 64]. As follow-up work to XIRL, GraphIRL [24] enhances robustness to diverse object appearances by abstracting frames into graphs. where each node represents an object and contains features including its bounding box coordinate and distances other objects. However, limitations arise when tasks require attention to object geometries or when dealing with different numbers of objects. Motivated by their work and aiming to address the limitations, we encode object visual features and locations into nodes in unified way and build graphs such that the encoder can generalize to scenes with ar- bitrary numbers of objects. Our work can be considered a generalized version of GraphIRL, suitable for both fesual-featureedeeeeeent and -independent tasks, as well as unseen task extrapolations.
A.3 Implementation Details
The visual encoder borrows from a pretrained ResNet-18 18 [81]. We take 512-dimensional embed- dings as node features construct complete graphs and use graph transformer [71] to extract 512- dimensional graph embeddings. We use the graph embeddings for reconstruction from decoder and temporal alignment. We adopt ADAM [82] optimizer and Soft Actor-Critic (SAC) [83] RI. algo- rithm. Hyperparameters are from Zakka et al. [23] with minimum fine-tuning on the number of frames per sequence in pretraining due to memory limitation. Learning rates are 10-6 for pretrain- ing and 10 for RL. Our approach does not rely on any data augmentation. All results represent the mean performance over random seeds.
A.4 Additional Experiment Results of X-MAGICAL
Results in domain-shifted environment. We assess VIRL's performance in shifted domains, specifically with respect unseen goal region colors and debris shapes, depicted in Figure 1c. In contrast to GraphIRL, which enhances robustness to unseen environments by disregarding visual features and focusing solely on graph abstractions of coordinates of bounding boxes, we encode both spatial and visual features into embeddings. Our aim is to validate that graph embeddings remain effective for measuring progress despite the incorporation of unseen visual features. Fig- ure shows that our approach significantly outperforms baselines in the short-stick scenario. For gripper, our approach and GraphIRL achieve similar performance, outperforming the rest, which expected considering that GraphIRL's abstracted spatial information of objects is sufficient for learning this task and is robust to shifted domains. For medium-stick and long-stick, results show that our approach is on par with the best baselines.
Figure 6: Cross-embodiment cross-enviromment. Encoder is pretrained on demonstrations as shown in Figure Ic (top). Policy learning in diverse environments with randomized goal region color and debris shape as shown in Figure Ic (bottom). Each agent (gripper, short-stick, medium- stick, long-stick, learns from videos of agents with the other 3 embodiments. For example, gripper learns from videos by short-stick, medium-stick, and long-stick.
17

---

3033
(a) Robot Visual Pusher Task
(b) Robot Visual Pusher Task II
Figure 7: Impact of object detection failures. Each object set to not detected with a probability. We test with probabilities being 0.1, 0.3, and 0.5 for both Task I and Task II.
A.5 Ablation Study
Impact of missed object detection. Our approach requires reliable object detector in both pre- training and policy learning. The detection performance on collected demonstrations can be exhaus- tively evaluated, while that on scenes during policy learning is prone to uncertainty. Therefore, we study the effect of object detection failures during policy learning. We randomly mask an object's visual and positional features setting its crop to be zeroed out with probability. The objects subject to mask include the robot end effector, the red and green cubes, and the goal. We test with probabilities being 0.1, 0.3, and 0.5 for Robot Visual Pusher Task I and Task II, and the results are shown in Figure 7. The effect brought by failing to detect all objects in scene at step that the corresponding reward signal may not accurately reflect the actual task progress. Interestingly, performance drop by detection failure varies between tasks.
A.6 Real- World Robot Setup
In the real-world robot experiment, we apply the trained policies of the variant task Robot Visual Pusher, i.e., Task II in Figure 5, Ufactory xArm 6 robot arm. The observation is captured by fixed Intel RealSense D435 camera. The robot setup is shown in Figure 8. We utilize the codebase of Diffusion Policy [84, 85] for the policy deployment on the real robot. We also visualize successful trial and a failed trial as shown in Figure 9a and Figure 9b, respectively.
Figure 8: Real-world robot experiment steup.
A.7 Details of Object Detector
Detection for X-MAGICAL. We use YOLO-v8 [86] the object detector for X-MAGICAL exper- iments. For encoder pretraining, we train the detector small set of images from demonstration
18

---

(a) Real-world robot successful trial.
(b) Real-world robot failed trial.
Figure 9: Real-world robot experiment visualization.
data by XIRL [23]. and evaluate it to ensure that detection is successful for all demonstration videos. For policy learning, adopt the trained detector from the pretraining step. Given an image, ap- ply the detector detect objects and build the corresponding graph of the image. However, the detection is not always successful during policy learning. We use the image from the previous step when detection failure happens.
Detection for Robot Visual Pusher. We also use YOLO-v8 [86] as the object detector for Robot Visual Pusher experiments, but only in pretraining. Similarly, for encoder pretraining, we train the detector small set of images from demonstration in [44], and evaluate it for successful detection of the object and the robot end effector over all demonstration videos. For scenes where the goal is occluded by the object and cannot be detected, we set the goal bounding box to be the summation of the object bounding box and Gaussian noise. For policy learning, we do not use any object detector. Instead, we follow GraphIRL [24] and compute objects bounding boxes in 2D image.
A.8 Technical Comparison between VIRL and GraphIRL
VIRL differs from and improves upon GraphIRL [24] mainly in three aspects: (i) the construction of graphs from images, (ii) the choice of graph encoder, and (iii) the loss function for pretraining.
Graph construction. GraphIRL concatenates object's bounding box coordinate with its dis- tances to other detected objects as the node feature {I1, Mr. 22, y2, d1 d2.. d Although robust to domain shift and visual distraction, their design has two downsides: First, it keeps ob- ject spatial information via coordinates while disregarding the visual features; Second, a pretrained graph encoder cannot generalize to policy learning for tasks where the number of objects different from that in demonstrations because of the change node feature length.
To resolve the first issue, VIRL encodes object-wise visual features into graph nodes. Specifically, VIRL creates an image for each detected object by zeroing out pixels outside of the object bounding box, keeping images shape identical to the original scene image. We then use convolutional neural network to extract the object's visual feature f,, ResNet(Lo). applicable tasks dependent on visual features. To resolve the second issue, we simply make the distance between two nodes the feature of the edge connecting them.
this way, we make our approach
Graph encoder. GraphIRL proposes Spatial Interaction Encoder for their graphs following [55] and [87] to extract abstract information. In contrast, we adopt Graph Transformer [71] as the graph encoder. We argue that any graph neural network that is capable of extracting decent global features from graphs with edges should applicable in our approach.
Loss function for pretraining, GraphIRL follows XIRL [23] and uses Temporal Sycle-Consistenge (TCC) loss as the loss function for pretraining their graph encoder. In contrast, follow [73] and
19

---

add reconstruction loss apart from TCC loss to ensure that sufficient object- low-level visual features are kept in the graph embedding for learning policies of visual-feauure-deeeddent tasks.
A.9 Extended Limitations and Opportunities
First, our approach relies on reliable object detectors. Although occasional nonconsecutive detec- tion failure may result in limited influence, detection failure keyframes can be influential. The opportunity brought by this limitation is that approach may be applicable for more complex tasks by leveraging more powerful open-vocabulary detectors, such OWL-ViT [88] and YOLO-World [89]. By incorporating object tracking methods, the occlusion cases may be also improved. Second, while our approach is tested on X-MAGICAL, Robot Visual Pusher, and real-robot, which provided valuable initial insights, we acknowledge that does not fully capture the complexity of kinematics and scene variations present in more challenging benchmarks, such tasks in RoboSuite [76], Meta- World [77], and BridgeDataV V2 [78]. Third, our approach proposed for rigid object manipulation in the first place. Thus, it is unclear how would generalize tasks of deformable object manipu- lation [79, 80]. Fourth, since approach focuses object-centric information while disregarding non-object-centriy information, it might struggle with certain tasks that require an understanding of the scene context.
20


Reviews:
[{"review_id": "oUEj00WoTO", "reviewer": "Reviewer_avVr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces VIRL, a novel graph object-centric visual representation that can be used for inverse RL. The main advantage of VIRL is its ability to generalize to unseen number of objects. Simulation results are shown on the X-Magical benchmark as well as a visual pushing task.", "summary_of_recommendation": "Weak Accept because this paper is technically correct but moderate in impact.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "My concerns are stated above. Here are some additional minor comments:\n1. The texts in Figure 2 and 3 are hard to read; consider re-sizing.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NFWs0oI7zA", "reviewer": "Reviewer_4TrT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an approach for learning reward functions with little supervision using a graph-based encoder network and temporal consistency and reconstruction objectives. The final reward function can generalize to computing meaningful distance rewards in scenes with more objects or even from human videos to robot scenes.", "summary_of_recommendation": "Overall, I like the paper and I recommend accepting it. I mentioned multiple downsides but I don’t think they outweigh the interestingness of the approach. If possible, I would encourage the authors to evaluate their approach on some messy real robot data like BridgeDataV2 with a lot more distractors and visual diversity.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- it would be great to add an experiment that evaluates the method on a less clean, real world dataset like BridgeDataV2", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27bqZyCMKR", "reviewer": "Reviewer_M2rM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a generalized version of GraphIRL, named VIRL,  which pre-trains a graph encoder on videos and then applies that encoder to extract reward functions for downstream tasks.  VIRL shows better results over previous methods such as XIRL on in-domain tasks and the extrapolation ability.", "summary_of_recommendation": "I would recommend the rejection for the current version. I think the idea of generalizing reward function learning is good, but the current paper seems to not propose too much new insights/techniques compared to previous methods. Also, the experiments such as ablation studies and real robot experiments are lacking.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. The curves in Figure 6 are not converged. It is good to see a better sample efficiency, yet it would be also interesting to see the converged results. Could authors run experiments to converge and show the results?\n2. Showing experiment results on real robots would give better insight into how VIRL helps real-world robot learning.\n3. Detailed difference between GraphIRL. What is the actual technical contribution that differs VIRL from GraphIRL?\n4. No ablation studies on the key factors that make GraphIRL perform better than previous methods.\n5. Figure 2 could be largely improved for better visualizations.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V30WaauWdC", "reviewer": "Reviewer_SzPu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an approach to inverse reinforcement learning from video demonstrations.  The approach first encodes the video demonstrations into an object-centric graph representations.  The graph embedding is pretrained with a cycle-consistiency loss and an object-centric reconstruction loss.  The resulting embedding can be used as a reward function by calculating the distance between the current embedding and the goal embedding.", "summary_of_recommendation": "Weak Accept:  This paper is interesting and proposes a novel method, but it is unclear how well it will work on more complicated tasks.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "How well does the method handle objects with overlapping bounding boxes?\n\nWhat off-the-shelf object detector is used by this method?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission proposes an extension of graph IRL to pretrain on visual data with detailed experiments and analysis.

The submission is well-written and presents a clear and simple, general method. It demonstrates promising results in generalizing learned distance functions to new embodiments and scenes. The integration of visual features and graph abstractions is a significant strength, and the self-supervised approach for pretraining is a good idea.

The proposed method seems to struggle with moving distractors and is strongly affected by noise in perception and detection. This limits its application wrt complex scenarios with occlusions or mis-detections. The lack of real-world robot experiments and exploration of more manipulation-centric benchmarks limit its convincingness. Additionally, the approach discards non-object-centric information, which might be important for certain tasks. The paper could benefit from including ablations and addressing the scalability of the method to more complex tasks.

The paper has improved on the above description during the rebuttal via detailed communication between reviewers and authors. Based on the above the paper is recommended for acceptance. Please follow up on the remaining open points for a potential camera ready version.
Decision: Accept