Task-Oriented Hierarchical Object Decomposition for Visuomotor Control
Authors: Jianing Qian
Yunshuang Li
Bernadette Bucher
Dinesh Jayaraman
Venue: CoRL 2024 (2024)
Keywords: Visual Representations
Entities
Imitation
Manipulation
TL;DR: We show how a visual scene can be represented by task-oriented entity tree structure, and enables efficient and generalizable imitation policy learning.

Abstract: Good pre-trained visual representations could enable robots to learn visuomotor policy efficiently. Still, existing representations take a one-size-fits-all-tasks approach that comes with two important drawbacks: (1) Being completely task-agnostic, these representations cannot effectively ignore any task-irrelevant information in the scene,  and (2) They often lack the representational capacity to handle unconstrained/complex real-world scenes. Instead, we propose to train a large combinatorial family of representations organized by scene entities: objects and object parts. This hierarchical object decomposition for task-oriented representations (HODOR) permits selectively assembling different representations specific to each task while scaling in representational capacity with the complexity of the scene and the task. In our experiments, we find that HODOR outperforms prior pre-trained representations, both scene vector representations and object-centric representations, for sample-efficient imitation learning across 5 simulated and 5 real-world manipulation tasks. We further find that the invariances captured in HODOR are inherited into downstream policies, which can robustly generalize to out-of-distribution test conditions, permitting zero-shot skill chaining. Appendix and videos: https://sites.google.com/view/
hodor-corl24

Task-Oriented Hierarchical Object Decomposition for Visuomotor Control
Jianing Qian', Yunshuang Li², Bernadette Bucher Dinesh Jayaraman' University of Pennsylvania', University of Southern California", University of Michigan"
(jianingg, dinesh @ upenn edu,, yunshuandusc. bucherbe www.emmicchee.....oo edu
Abstract: Good pre-trained visual representations could enable robots to learn visuomotor policy efficiently. Still, existing representations take one-size-fits- all-tasks approach that comes with two important drawbacks: (1) Being com- pletely task-agnostic, these representations cannot effectively ignore any task- irrelevant information in the scene, and (2) They often lack the representational capacity to handle unconstrained/complex real-world scenes. Instead, we pro- pose to train a large combinatorial family of representations organized by scene entities: objects and object parts. This hierarchical object decomposition for task- oriented representations (HODOR) permits selectively assembling different repre- sentations specific each task while scaling in representational capacity with the complexity of the scene and the task. In our experiments, we find that HODOR outperforms prior pre- pre-trained representations, both scene vector representations and object-centric representations, for sample-efficient imitation learning across 5 simulated and 5 real-world manipulation tasks. We further find that the invariances captured HODOR are inherited into downstream policies, which can robustly generalize out-of-distribution test conditions, permitting zero-shot skill chain- ing. Appendix, code and videos: sttps://sites google com/vel view/ modor-cor124
Keywords: Visual Representations, Entities, Imitation, Manipulation
Introduction
How should an embodied agent internally represent its visual observations? The role of a repre- sentation is to supply task-relevant information to the decision- makiing procedure, such as neural network policy. This implies two desirable properties for any representation. First, a representa- tion should highlight task-relevant aspects of the scene and de-emphasize irrelevant information. This means that good representations are inherently task-specific: driving on the road relies on very different aspects of the scene than walking on the pavement. Or, in our experimental settings, the same kitchen robot might need attend to different scene aspects when executing different sub- tasks: boiling water, turning on faucet, putting vegetables into pot, and so This necessary task-specificity was an important motivation for "end-to-end" approaches in the last decade [1, 2]. Rather than hand-crafted "off-the-shelf" representations (e.g., bags of visual words [3], SIFT [4), HoG features [5]), they aimed to optimize the entire control pipeline for one or few tasks. Second, representations must conveniently organize scene information so that the policy can be represented in simple function. For robot learning, such simplicity in optimal policies translates ease of discov- ering these policies by optimization procedure operating over limited experience. For example, representation could aim to disentangle the latent factors of variation in the inputs [6. 7, 8, 9, 10].
In recent years, motivated by efforts to improve data efficiency, off-the-shelf representations have risen in popularity again, this time in the form the activations of pre-trained vision models [11, 12, 13, 14, 15, 16, 17, 18]. While these pre- pre-trained representations are longer hand-crafted, they retain another critical drawback of off-the-shelf representations: offftteessheerreeeeeenttttoos they are agnostic to the downstream
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Kettle on Stove
Turn on Faucet
Put Uppplennt
Tasks Descriptions
"Put on Stove"
Object Represennaatons
HODOR
Part Segmentations
Figure 1: HODOR embeds 2D input image into an object-centric, multi-resolution, task-specific
representation for efficient policy learning
task. As such, they are trained to capture all scene information, even though most of this information might be irrelevant to any given downstream use case. Furthermore, in complex, cluttered scenes, these standard ixed-dimensionality "scene vector" representations are limited in their ability to comprehensively capture all fine-grained details.
How might we enable task specificity alongside representation pre-training? Rather than train single representation, we propose to generate a menu of representations that can be combinatori- ally assembled into the right platters suited for each downstream task. In particular, we propose hierarchical object decomposition for fask-oriented representations (HODOR). HODOR recognizes that scene entily trees, i.e., trees of objects and object parts, provide a convenient organizing princi- ple for representation menu: different objects are relevant at different levels of detail to different tasks or task phases. Further, HODOR exploits advances in semantic visual understanding to select task-relevant scene entity trees for any given task. HODOR involves no new pre-training generate representations; instead, it mines these representations from existing vision and language foundation models. See Fig 1.
Our main contributions are: 1) We introduce pre-trained approach generate multi-level de- composition of given scene and then "filter" based on language-based task descriptions into task-specific tree; 2) we present approach for downstream policy networks learn from this, and structured HODOR representation, and 3) We empirically show that HODOR is better than prior pre-trained representations at downstream manipulation tasks.
Related Work
Pre-trained representations in computer vision and robotics Rather than learn from scratch for each new task, pre-trained representations are attractive for their potential to bootstrap task learning, reducing task-specific training data requirements. Pre-trained visual representations have demon- strated their potential utility for many downstream visual recognition tasks [11, 12, 19, 20, 21, 22]. robotic
More recently, robot learning researchers applied these pre-trained representations tasks [23, 13, 14, 15, 24). Several works have recently studied whether, when, and how these representations improve robot policy learning [25, 26, 27). The above approaches all focus what we call "scene vector" representations i.e., summarizing image in monolithic, unstructured, fixel-dimensionality vector. HODOR instead structures the representation along scene entities.
Object-centric embeddings (OCE) With constrained task setups, roboticists have long used ob- ject bounding boxes, poses, and shapes to represent state, usually from bespoke domain -spciffc supervised models. A recent wave of object-centric representation learning approaches have pro- posed modified neural network architectures that encouraged self-supervised learning to generate discrete objects and entities [28, 29, 30, 31, 32, 33, 34, 35, 36, 37] [38, 39]. While they have yielded promising results on small domains with large datasets and few objects, has proven difficult to scale such approaches to train effective object-centric embeddings on mn-the-wild images. Mean- while, advances in foundation models for open-world visual recognition have made possible once more to assemble object-centric representations for robotic cases without needing train such representations from scratch [40, 41, 42]. Closely related to our method, POCR [41] is recent

---

object-centric scene encoder that leverages the zero-shot segmentation ability of vision foundation models such as SAM [43] and LIV [15]. Like POCR, HODOR also assembles object-centric "off- the-shelf" embeddings but with two important new components: hierarchical scene entity trees and task-conditioned object selectivity.
Hierarchical coarse-to-fine representations Recognizing that different levels of detail are ap- propriate for different downstream use cases, several methods aim build multi-resolution scene vector representations within neural networks [44, 45, 46]. Similar to HODOR, others have also rep- resented objects at multiple resolutions [47.48, 48,49, 49, 34, 50, 51]. In robotics, Ravichandran et al. [50] build 3D scene graphs for representing multi-resolution scenes in a SLAM framework; in this con- text, they represent full metrie-semantic 3D meshes at the levels of "objects", "places", "rooms", and "buildings". Ok et [51] represents object shapes for navigation agent more coarsely for farther away objects and more finely for nearby ones. Liu et al. [52] builds 3D diffusion models for robotic manipulation skills by leveraging object-part decomposition. Geng al. [53] builds large-scale 3D manipulation benchmark that enables cross-category generalization by annotating object-part decomposition. In contrast, we work with 2D image observations, construct task coss-conditioned repre- sentations, and further decompose objects into parts to facilitate fine-grained manipulation tasks.
Task conditioning in policies and representations For conditioning policies on task identity in multi-task learning settings, most prior works [54, 15] propose append embedded language- task descriptions alongside sensory observation embeddings. Rather than filtering task-ittelevant information within the representations input the policy, such approaches rely the downstream policy to separate relevant from irrelevant information. More recently, many works suggest that obtaining task-relevant scene representations benefits downstream manipulation tasks. Sundaresan et al. [55], Parashar et al. [56] map task descriptions to interaction points 3D space. Marza et al. [57]. Chen al. [58] propose to process visual observations into task-relevant embeddings before passing them as input the policy. Ma al. [59]. Eftekhar al. [60] use task descriptions to identify task-relevant regions in 2D observation guide explorations for reinforcement learning- Zhu et al. [40] ask users to scribble over task-relevant objects on image observations to build task- relevant scene representations. With HODOR, we build scene representations tied to entity trees in the scene, which are automatically filtered into task-relevant representations by selecting the appropriate entities as dictated by task semantics.
To sum up. HODOR presents a new visual representation framework for robotics that is object- centric, cask-condiiioned, and multi-resolution. In our systematic experimental study, find that the combination of these properties in HODOR yields important gains in task performance across variety of robot manipulation tasks.
3 Inferring HODOR Representations From Foundation Models
As motivated above, we propose to assemble image representations that are object-centric, task- conditioned, and multi-resolution. Below, describe the step-by-step procedure for inferring the HODOR representation for an input image.
3.1 Preliminaries: Object-Centric Image Embeddings from Pre-trained F Foundation Models
Following common convention [28, 29, 30, 31, 32, 33, 34, 35, 36, 37], we define object-centric embedding (OCE) of of an image observation at time step t as a set of "slots" sf that summarize the entities (e.g., objects, parts, sub-parts) in the image. Each entity slot sf = (1), 2f) containing of its location 7 E 1R (e.g. axis-aligned bounding boxes denoted by four coordinates) and content embedding 2f D.
We start from successful recent OCE approaches [40, 41] that leverage the zero-shot segmentation ability of vision foundation models such as SAM [43]. Specifically, we follow Zhu et al. [40] to

---

obtain SAM object segment "slot" masks m' = }m, on the first frame = 0 of a policy rollout or demonstration, and then track each mask for the rest of the rollout using XMem [61].
Now that we have set of entity masks m, of the scene, how can we assemble them into a hierar- chical structure that best describes the task at hand? In Section 3.2, we describe our approach that selects task-relevant entities conditioning the task descriptions. In Section 3.3, describe algorithm to discover and group parts into the task-relevant objects, and finally, in Section 4, we describe our policy design to best utilize our representations for downstream robotics tasks.
3.2 Lask-Conditionee Entity Selection
Consider a robot kitchen environment, as in our experiments in Sec 5. This environment has a large task space, including boiling water, putting vegetables in a pot, or turning on the faucet. Each task requires the robot to pay attention different parts of the scene. Most prior approaches naively use constant scene representation for all tasks. Instead, we propose to construct bespoke task-relevant representations. We exploit recent advances in vision and language foundation models to permit specifying task-relevant parts of the scene using natural language task descriptions. Given the task description d, we queried large language models (GPT-4 [62]) to identify the entities in d. For example, the task-relevant object for "open the microwave" is "microwave." After identifying the noun entities, we prompt Grounded SAM [63], an open-set object segmentation model with these entity names, to produce a of task-relevant segmentation masks Mer {m:}.
3.3 Hierarchical Object Decomposition
Given the image observation / and the selected task-relevant object masks Mtr. we can assemble the hierarchical OCE. For our settings, we set up three levels in the hierarchy. The first level consists of the root node, which is the scene-level slot Swmme (Iweees Zscene): )- Similar to [16]. we use the CLS token of DINO-v2 ViT as the scene-level representations Zsceme. We define Leeene = [0., 1., O., 1.] in normalized scene coordinates, representing bounding box containing the entire image. The second level contains an ordered sequence of task relevant object slots [st,aty] = [(i 21, Zi is computed by average-poolingg pooling DINO-v2 features of the last attention layer the corresponding segmentation mask m; E My,, and oiy is the tightest bounding box around m.-
Finally, the third level contains an unordered set of part slots {Sipur] = {(1i,oms 2j.par) corre- sponding to each parent object. To discover the object parts, we input image observation into SAM obtain all the segmentation masks at different levels of granularity. We assign part to an object parent if their intersection greater than 0.5 with object We calculate the slot vectors of parts pjrr in the same way as for the objects.
The resulting HODOR representation [Sscone; [Siobg] [jjpprr) contains three layers of slots in coarse-to-fine order: the full scene, task-relevant objects, and their corresponding object parts. HODOR conveniently organizes the scene information into entities and limits the finer levels of the representation to directly task-relevant objects, while still retaining sufficient coarse information to, say, avoid collisions with the rest of the scene. Further, the scene and the task grow in complexity, the size of HODOR representations grows in tandem to accommodate the increased policy needs for task-relevant scene information. Finally, this representation involves redundancies that may improve robustness to failures in object segmentation or tracking: coarse details of task-relevant objects are still retained the scene-level slot.
Policy Architecture and Training
We have seen above that HODOR produces a structured representation containing nested ordered sequences and unordered sets of slots. For the policy to flexibly process this representation, we use a transformer-basee policy architecture, illustrated in Fig 2.The input self-attention layer naturally processes variable-sizer inputs as well as unordered sets without any special modifications. We feed HODOR representations into the self-attention layer as follows. Each individual slot at each level

---

of HODOR is represented as an imput token. To represent the level-wise ordering, we attach level emhedding to the slot vectors. Similar to positional encodings common transformer architectures, these level embeddings & are learnable tokens that are designed to encode hierarchical graph infor- mation. We learn one token @ceene for the scene and one token @obj(i) for each object. All parts (S).prr) that belongs to the same parent object share the same embedding part(() Like DINO- v2, we append learnable CLS token the input so that the self-attention output accommodate an input summary in the output CLS token. We can either feed the CLS token directly into the downstream MIP or take in the concatenation of all tokens output from the attention layer. More details are in the Appendix A.I. In our experiments, we train the above policy architecture on expert demonstrations with a standard behavior cloning objective for each target manipulation task.
5 Experiments
Our method assembles task-oriented entity-based scene representations for manipulation tasks. With simulated and real robot experiments, we aim to answer: (1) Do imitation learned policies for robot manipulation tasks benefit from HODOR compared to alternative pre- trained visual representations? (2) Do policies trained on HODOR representations acquire more invariance to task-irrelevant information compared others? (3) To what extent HODOR's s gains attributable to its task-oriented, multi-resolution, and object-centric properties?
5.1 Simulated Experiments
ltterachical Eibbolding Embedding
In order to evaluate the performance of our hierarchical scene representations, we first test our method and baselines on five simulated Franka Kitchen [64] tasks: SlideCabinelDoor, OpenMicrowave, OpenCabinetDooor TurnOnLight and Turn- Knob. Fig. 4 shows these tasks. The 9-DoF Franka robot is controlled through joint velocities. Following the represen- tation evaluation paradigm in [16], we collected 40 demon- strations for each task and evaluated the behavioral cloning (BC) policy performance as a function of varying numbers of demonstrations. Each method is trained with 3 random seeds. We report the success rate in Fig. 5, which is calculated on 100 online rollouts, and take the maximum success rate over the course of training. In practice, prompting Grounded SAM with the object name in the simulated environment does not reliably identify the objects due to the fact that objects in the simulated environment could look ambiguous. Instead, we simplify this by drawing tight bounding boxes around target objects. In experiments, hand-annotate task-relevant object locations for TurnKnob, TurnOn- Light, and OpenCabinetDoor; more details provided in the Appendix A.2.
CLS
Alsertion Layers
Iegel
Figure 2: Policy Architecture. We illustrated the resulting HODOR representations for task Put Eggplant in Pot. There two relevant objects in this task: eggplant and pot. Their corre- sponding parts are also visualized.
Baselines: We compare HODOR against four prior pre-trained image representations for robots: DINOv2 [12] representations are widely used for downstream vision and robotics tasks; similar to [41], we use the last-layer CLS token. R3M [13] and LIV [15] are popular robotics-targeted scene vector representations. Finally, as described in Sec. 2. POCR [41] builds 1-level object- centric representations using SAM [43] and LIV.
Results: Fig. 5 shows success rates for HODOR and baselines. Please see the videos of the policy rollout in the URL above. R3M and LIV, which are trained on robot-specific data, outperform DINOv2, which is trained on standard vision tasks. HODOR outperforms all these methods with nearly all demonstration set sizes on all tasks besides OpenCabinet- Door, with higher average performance and lower standard errors (shaded region) throughout. The comparison with DINOv2 is particularly instructive since HODOR itself relies DI- NOv2 encodings of each slot. Here, HODOR is vastly superior, with particularly large gains
5

---

Xicronore
Cabinet
Knob
Light
Cabinet Door
Figure 4: Visualization of simulated Franka Kitchen tasks.
OpenMicrowave
SlideCabinetDoor
TurnKnob
TurnÖnLight
OpenCabinstDoor
1.0
1.0
1.0
Succoss Rate
0.0
0.8
0.5
0.4
DINOV2 R3M
0.4
0.2
LIV
HODOR((UrR)
0.4
0.2
0.1
Number Demos
Number Demos
Number Demos
Number Demos
Number Demos
Figure 5: BC performance of HODOR and baselines function of the number of demonstrations measured by success rate on five tasks from the FrankaKitchen benchmark. Shaded regions show the standard error across seeds.
in low-data settings. Likewise, POCR, which builds a task-agnostic one-level object-centric representation that includes distractor objects too, and suffers with few demos. We see that HODOR achieves data efficiency benefits from focusing its representation on task-relevant objects. Overall, these results validate that HODOR's carefully structured representations tations of DINOv2 embeddings to match and outperform the best pre-trained image representations, such as R3M and LIV. On one outlier task, OpenCabinetDoor, HODOR does not achieve the best results. This task poses particu- lar difficulties for HODOR because the target objects are always visible, either occluded by the robot arm outside the image field of view. This trips up HODOR's object detection and tracking pipeline, affecting represen- tations and policies trained with few demos.
overcome the relative limi-
Keadl
The HODOR framework confers two impor- tant new properties an object-centric representation: multi-resolution and task-orientedness. To what extent do these contribute to HODOR's success? We design two ablation experiments: 1) Ours - multi-eevel is the variant of our method without the object decom- position: instead of the full HODOR representations [Sicenes [Stowy], I Hil]. we only input scene and object representations (no parts) [Sscene: {St H to the downstream policy. Note that this is dif- ferent from the POCR baseline above in subtle implementation details, described in Appendix A.9. 2) Ours-tars conditioning (Ours-TC) is the variant of our method without task conditioning. This means instead of selecting task-relevant objects, include all objects and their parts in the image representation. Fig. 6 shows these results. In general, Ours taans-muli--eeel performs comparable to R3M and LIV, but still worse than HODOR, especially when number of demonstrations is low. This shows that our hierarchically structured representations help agents to learn more efficiently from demonstrations. On the other hand, Ours-task conditioning performs much worse than HODOR. Without task-conditioned entity selection, the representation space grows too large.
A Ablations:
NarVia foncketk FRUNI
Figure Task Visualization.
5.2 Real Robot Experiments
We now evaluate HODOR on set real robot tasks a tabletop kitchen manipulation environment. Our tasks involve a variety of skills, such as picking, placing, twisting, and pouring. Task Design: We create kitchen environment with various objects the countertop. We use 7-DoF Franka robot arm with continuous joint-control action space. Details of hardware setup are

---

Openklicrowave
SlideCabinetDoor
TurnKnob
TurnOnLight
OpenCabinetDoor
1.0
1.0
0.7
0.8
0.8
6.8
0.9
0.5
0.6
0.6
Succoss
0.4
0.4
0.4
0.4
0.7
Ours-TC
0.2
murt-maltileel
0.6
HOBORIOUm)
0.0
Number Demos
Number Demas
Number of Demos
Number of Demos
Number of Demos
Figure 6: BC performance of HODOR and ablation experiments as a function of the number of demonstrations measured by success rate tasks from the FrankaKitchen benchmark. Shaded regions show the standard error across seeds.
Task
Eggplant Sink
Kettle-Stove
Faucet
Eggplant Pot
Water-Pot
Overall
Method
# of Trials
15
15
15
15
15
15
15
15
75
75
IND
OoD
[ND
OoD
IND
OoD
IND
OoD
IND
OoD [ND OoD
HODOR (Ours)
12
12
8
13
8
12
9
64
49
DINOv2 [12]
5
13
15
5
11
7
11
58
18
LIV (ResNet50) [15]
1
6
8
2
8
0
5
0
36
3
R3M (ResNet50) [13]
0
3
2
10
33
Table IND and OoD BC Results on Real Robot Tasks. We report the number success each task out of 15 trials.
in the appendix. For each task, we collect 100 demonstrations via teleoperation; for each trajectory, the positions of relevant objects the scene are randomized within fixed distribution. We design suite of tasks that progressively harder, in terms of manipulation difficulty and complexity of the environment. Fig 3 shows an illustration of each task:
1. Put Eggplant in Sink (Eggplant-Siin and placing soft plush toys. We create more challenging visual scenario by cluttering the sink with other fruits.
This is the easiest task that only involves picking
Place Kettle on Stove (Kett "Kettle-Sae e-Stove) : Pick up the kettle by the handle, and put onto the stove. The initial position and orientation of the kettle randomized.
Open Faucet (Faucet) Twist the knob of the faucet that placed over the sink. Since the knob is small, it requires the agent to learn precise actions in order to turn it on.
Previous tasks only involve one task-relevant object. To further test the ability of our representation when multiple task-relevant objects are involved, designed the two following tasks:
4. Put Eggplant in Pot (Eggplant P Pick up the eggplant from the sink and put it in the pot. This task involves two objects: "eggplant" and "pot." The initial positions of both objects are randomized.
5. Pour Water from Kettle into Pot (Water-Pot): Pick up the kettle by the handle, move it over to the pot, and pour water out of the kettle (simulated by small orange beads for better visualization) into the black pot. This is by far the hardest task since grasping and pouring out of a heavy kettle requires a firm and secure grasp of its handle. Both the position of the kettle and the pot are randomized.
Evaluation: Aside from the graded difficulty of these tasks, we also further study policy perfor- mance for each task under an easy "in-distribution" (IND) setting and hard disttof-disttrbuution (OoD) setting. In IND setting, task-irrelevant objects are kept at the same locations as in the collected demonstrations. We report the number of successful runs out of 15 trials for each task. To evaluate the systematic generalization abilities of HODOR, design three OoD scenes for each task and each OoD scene is evaluated over 5 trials (15 trials in total). In the OoD case,

---

lask-irrelevant objects are either placed at unseen locations or removed. Fig. shows illustrations of the OOD setup.
Table shows the results of all meth- ods. Please see URL for videos. HODOR beats all three baselines even in IND set- tings, and its gains are particularly large in OoD, with LIV and R3M faring particularly poorly. These baselines suffer from both flat scene vector representation that struggles capture fine-grained object details needed in tasks and sre-size-fits-alltaae approach to building visual representations when many distractor objects are present. To take some examples, Kettle on Stove and Open Faucet demand fine-grained representation of object parts, such as kettle handle and faucet knob, and HODOR enjoys the benefit of rep- resenting the relevant entity trees. Next, Put Egsplant in Pot in the OoD setting in- volves large distractor objects that are unseen during training. HODOR, being inherently invariant to task-irrelevant scene objects, transfers best Eggplant in Pot and Pour Water from Kettle into Pot, HODOR grows its representation to accommodate multiple task-relevant objects and outperforms baselines.
Results:
Egppplantin
Kettle Stove
PIA Egpp PNE
Figure 7: OOD scene settings in real robot exper- iments. Red circles mark the changes the OoD scenarios compared to the demonstrations.
to such settings. In the last two tasks, Put
To capitalize our promising OoD results above showing system- atic generalization to distractor objects, we at- tempt to execute the five skills one after an- other in a pre-determined sequence resembling an eggplant preparation recipe: pick up the ket- tle and put on the stove, put the eggplant in the sink, turn on the faucet, put eggplant in pot and finally pick up kettle and pour water into the pot. Note that each skill at training time is trained from the same starting state distribution, so each of the second, third, fourth, and fifth skills here must handle very ffr-out-of-domain initial states produced by the previous skills in the sequence. For example, since the eggplant is in the sink while the agent tries turn on the faucet, alters the appearance of the scene large enough that our baselines fail to locate the faucet. After many trials, baseline goes past the second skill in this skill-chaining setting, but HODOR successfully chain all five, demonstrated in Fig 8.
Zero-shot Skill Chaining:
Figure 8: Illustration zero-shot skill chaining. We demonstrate executing five skills in sequence, creating hard OoD setups.
6 Conclusions and Limitations
hierarchical task-oriented scene representation generated through object decompositions. Instantiated from vision foundation models, HODOR outperforms state- of-the-art pre-trained representations for robotic policy learning. It also demonstrates robustness to variations in task-irrelevant scene aspects enabling impressive new use cases such as zero-shot chaining of separately trained skills. While these empirical results are impressive, combining mul- tiple pre-trained models creates several potential failure points for HODOR; we analyze these in detail in Appendix A.7. In short, find that the overall system is often capable of correcting for component failures, and/or such failures rare in the types of tasks most frequently studied in imitation learning for manipulation today. This is reflected in HODOR's ss strong results across our extensive evaluations.
We have presented HODOR,

---

Acknowledgments
This project was supported in part by NSF CAREER Award 2239301, NSF Award 2331783, and ONR award N00014-22-12677.
References
[1] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-137) 1373, 2016.
[2] M. Bojarski, D. Del Testa, D. Dworakowski, B. Fimer, B. Flepp. P. Goyal, L. D. Jackel,
M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: 2016.
[3] Sivic and A. Zisserman. Efficient visual search of videos cast as text retrieval. IEEE trans- actions on pattern analysis and machine intelligence, 31(4):591-606 2008.
[4] D. G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150-1157. leee, 1999.
[5] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S), 1: 886- 893 vol. 1, 2005. URL https:/ /taps://api.. semanticscholart......ccccccccce httpp:///pa...eemnnncccccooooooo htt.://chhh..a...ooorroosps.ss/a 206590483.
[6] S. Liu, Y. Hu, Y. Zeng, Q. Tang, B. Jin, Y. Han, and X. Li. See and think: Disentangling semantic scene completion. Advances Neural Information Processing Systems, 31, 2018.
[7] S. Benaim, F. Warburg, P. E. Christensen, and S. Belongie. Volumetric disentanglement for 3d scene manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8667-8677, 2024.
[8] D. Raposo, A. Santoro, D. Barrett, R. Pascanu, T. Lillicrap. and P. Battaglia. Discov- ering objects and their relations from entangled scene representations. arXiv preprint 1721v:170..05068, 2017.
[9] K. Do and T. Tran. Theory and evaluation metrics for learning disentangled representations. arXiv preprint
2019.
[10] Van Steenkiste, F. Locatello, Schmidhuber, and Bachem. Are disentangled repre- sentations helpful for abstract visual reasoning? Advances in neural information processing systems, 32, 2019.
[11) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Confer- ence on Computer Vision (ICCV), 2021.
[12] M. Oquab, Т. Darcet, Moutakanni, H. Q. Vo, M. Szafraniec, Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-.. B. Huang, S.-W. Li, I. Misra, M. G. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision. ArXiv, abs/2304.07193, 2023, URL https: ttaps://apiis /tpp.ppp.../// seranteemnniiooo icschhlaa.. org/Corpus : 258170077.
[13] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual repre- sentation for robot manipulation. In Conference Robot Learning, 2022. URL https: //api. seapitssmatticccoooo Ditp://ccci.aaoooooooooooeeaiiaa
wttg:/CorpussD- 24:21118844

---

[14] Y. J. Ma. S. Sodhani, D. Jayaraman. O. Bastani, V. Kumar, and A. Zhang. Vip: Towards uni- versal visual reward and representation via value-implicit pre-training. ArXiv, abs/2210.000300, 2022. URL https::///// //pp./ap....//// semanticscholar htt.aaccccaaa..rocccooaaaaaaaaaa
[15] Y. Ma, W. Liang. V. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representations and rewards for robotic control. International Con- ference on Machine Learning, 2023. URL https: hess://app. /api. http://aap....aatsssesseeeeeeese semanticscholes
CorpusID: 22:.:I:::229999999
[16] Y. Hu, R. Wang, L. Li, and Y. Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. arXiv preprint arXiv:2304.04591 2023.
[17] N. Hansen, Z. Yuan, Y. Ze, Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre-training for visuo-motor control: Revisiting leaming-from-ssrenn fcom-scratch baseline. ArXiv, abs/2212.05749, 2022. URL https::///// / /app//ap../////// hetp::::/ap.aaaaaamaa.mmmmmsm semanticssssseeeeeeee centtiiccchlees ar. org DD 25442228444
[18] K. Burns, Z. Witzel, J. I. Hamid, T. Yu, C. Finn, and K. Hausman. What makes pre-trained visual representations successful for robust manipulation? ArXiv, abs/2312.12444 2023. URL
www.ntemaiii...m ieeamntccscholar. httpss//app..eemaniicchhhooe
Cttp//rrrsssDDD:D ID:
[19] Chen, S. Korbblith, M. Norouzi, and E. Hinton. simple framework for contrastive
learning of visual representations. ArXiv, abs/2002.05709, 2020. URL /ttps://apy)
semanticscholes org/ :
[20] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning, arXiv preprint arXhv:1911.05722, 2019.
[21] X. Chen, H. Fan, R. Girshick, and K. He. Improved basclines with momentum contrastive learning. arXiv preprint WrXiv:2003.04297 2020.
[22] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv, abs/2000600988,, 2020. URL https: /ttpss:/apl.. http://.....ss...ii..mmmmmmm
semant wcwchollcloooccooooo ar. org/ Corp/orruusssc ID: 21 9721240.
[23] Sermanet, C. Lynch, Y. Chebotar, Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141 IEEE, 2018.
[24] W. Shen, G. Yang, Yu, J. Wong, Kaelbling, and Isola. Distilled feature fields enable few-shot language-guidee manipulation. arXiv preprint arXiv:2308.0799. 2023.
[25] N. Hansen, Z. Yuan, Ze, T. Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre- training for visuo-motor control: Revisiting a learning-froom fcratsc baseline. arXiv preprint arXiv:2212.057499 2022.
[26] K. Burns, Z. Witzel, J. I. Hamid, T. Yu, C. Finn, and K. Hausman. What makes pre-trained visual representations successful for robust manipulation? ArXiv, 2023.
[27] Y. Hu, R. Wang, L. E. Li, and Y. Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. In A. Krause, E. Brunskill, K. Cho, B. Engel- hardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 13628- 13651. PMLR, 23-29 Jul 2023.
[28] т. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world mod- els. ArXiy, abs/1911.11247,, 2019. URL https://apps //appi semant iestaantcccholla org/
10

---

[29] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. P. Burgess, D. Zoran, L. Matthey, M. M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational in- ference. ArXiv, abs/1 htss/1903.000 903.00450, 2019. URL https: http://pp..smmatccccchlelleeeeee orgg
CorpusID: 67855876.
[30] M. Engelcke, A. R. Kosiorek, O. Jones, and I. Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. ArXiv, abs/1907.13052, 2019. URL 198986015.
https: /ppi. semanticschon wtwpcchoow.....ooooouooooeeeeeee
[31] T. D. Kulkamni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. ArXiv, abs/1906. 18833, 2019. URL https:t/p///. / //ppi semanticschon ar 1
[32] Locatello, D. Weissenborn, Unterthiner, A. Mahendran, G. Heigold, J. Uszkor- eit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. ArXiv, abs/2006.15055 2020. URL https: //api: semanticscho wttpa/a.oorooooooe.oeeoeeeeeeooo orgp/o/pss/rrs
220127924.
[33] G. Singh, F. Deng, and S. Ahn. Illiterate dall-e learns to compose. In International Con- ference on Learning Representations, 2022. URL https: /oppnreeeee/...ii....// net forum? id=h0oyvOWe3oh.
[34] Qian, A. Panagopoulos, and D. Jayaraman. Discovering deformable keypoint pyra- mids. In European Conference on Computer Vision, 2022. URL https://api.
semanticscholat wttp:oooous...2252200...0
[35] W. Yuan, C. Paxton, K. Desingh, and D. Fox. Sornet: Spatial object-centric representa- tions for sequential manipulation. ArXiv, abs/2109.03891, 2021. URL /ttps://api..
semanticscholes Crtp/oorruu.:::23345522222
[36] B. Sauvalle and A. de La Fortelle. Unsupervised multi-object segmentation using attention and soft-argmax. 2023 JEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3266-3275, 2022. URL https:: /taps://api.. wee.atticscchit 249097913.
[37] M. Chang, T. L. Griffiths, and Levine. Object representations fixed points: Training iterative refinement algorithms with implicit differentiation. ArXiv, abs/2207.00787, 2022. URL https:://// /api........ wwwanticschont
org/ Corpps/oppsiioccee :I..puu4ID:20224555..
[38] K. Fan, Z. Bai, T. Xiao, D. Zietlow, M. Horn, Z. Zhao, C.-J. Simon-Gabriel, M. Z. Shou, F. Locatello, B. Schiele, T. Brox, Z. Zhang, Y. Fu, and T. He. Unsupervised open-vocabulary object localization in videos. 2023 IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 13701-13709, 2023. URL.ttps: http://ppp...mmt.t.ccccalaaaaaae .org/
CorpusID: 262045019.
[39] M. Seitzer, M. Horn, A. Zadaianchuk, D. Zietlow, T. Xiao, C.-. Simon-Gabriel,, T. He, Z. Zhang, B. Scholkopf, T. Brox, and Locatello. Bridging the gap to real-world object-centric learning. ArXiv, abs/2209. 10099...466666 14860, 2022. URL https /api. semanticschen .org/ Corpus ID: 252596292.
[40] Y. Zhu, Z. Jiang, P. Stone, and Y. Zhu. Learning generalizable manipulation policies with object-centric 3d representations. ArXiv, Abs/2310.14386. 2023. URL https://api. semanticscho
[41] Shi J. Qian*, Y. J. Ma, and D. Jayaraman. Plug-and-play object-centric representa- tions from "what" and "where" foundation models. ICRA, 2024. URL https: sites. google.. www/cwewwwvvcceeoo.o htnn/vie/pocr.

---

[42] J. Qian, A. Panagopoulos, and D. Jayaramnn. Recasting generic pretrained vision transformers as object-centric scene encoders for manipulation policies. ICRA, 2024. URL https:// sites google. www.vcimiii/eeee...ooe.........w www.otooooof...o.
[43] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y Lo, Dollár, and R. B. Girshick. Segment anything. 2023 TEEE/VVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023. URL https:
//api. www.sstttannnncoo.e.eeeo
org/ wtt/Corpus 257952310.
Corpus
[44] T.-Y. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and Belongie. Feature pyramid networks for object detection. 20/7 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 936-944, 2016. URL https:/ /tt///api... www.nticscoolarrccoomm
Corpus ID: 10716717
[45] A. Kusupati, G. Bhatt, Rege, M. Wallingford, A. Sinha, Ramanujan, W. Howard-Snyder, K. Chen, S. M. Kakade, Jain, and A. Farhadi. Matryoshka representation learning. In Neural Information Processing Systems, 2022. URL https: /api semanticscholas .rrg/ CorpusID: 252683450.
[46] S. Saxena, M. Sharma, and O. Kroemer. Mrest: Multi-resolution sensing for real-time control with vision-language models. 2024. URL
http/ppp..smmmicccccooll.eeeeet. org
Corpus C
267301353.
[47] D. Mrowca, C. Zhuang, E. Wang, N. Haber, L. Fei-Fei, B. Tenenbaum, and D. Yamins. Flex- ible neural representation for physics prediction. In Neural Information Processing Systems, 2018. URL attps://apt http://pp...aaatiiiscccm semanticscholat www.aaa..rggcoopoooooeeea : 49348777.
[48] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. ArXiv, abs/ 1ai.01880.1166,,,, 2018. URL https:// //api. semanticschol ar. org. Corpus ID: 52917627.
[49] Y. Wang, Y. Li, K. R. Driggs-Camphell, L. Fei-Fei, and J. Wu. Dynamic-resolution model learning for object pile manipulation. ArXiv, abs/2306.16700 2023. URL https://api. remanticscholar: ort/CorrpsID::22000.
[50] Z. Ravichandran, L. Peng. N. Hughes, J. D. Griffith, and L. Carlone. Hicrarchical represen- tations and explicit memory: Learning effective navigation policies on 3d scene graphs using graph neural networks. In 2022 International Conference on Robotics and Automation (ICRA), pages 9272-9279. IEFE, 2022.
[51] K. Ok, K. Liu, and N. Roy. Hierarchical object map estimation for efficient and robust nav- igation. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1132-1139, IEEE, 2021.
[52] W. Liu, Mao, Hsu, Т. Hermans, A. Garg, and Wu. Composable part-based manipu- lation. ArXiv, abs/2405.05876 2024. URL https:/ /api...... semanticscholar .orgoorrlrrrrrroooooooooooo
CorpusID:2551544
[53] H. Geng, Z. Li, Y. Geng, J. Chen, H. Dong, and H. Wang. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. 2023 IEEE/CVF Con- ference on Computer Vision and Patlern Recognition (CVPR), pages 2978-2988, 2023. URL https: //pp./app..// semant icscholar. org/ wttp/oorpsII::25755555
[54] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pages 9767-9777, PMLR, 2021.
12

---

[55] P. Sundaresan, S. Belkhale, D. Sadigh, and J. Bobg. Kite: Keypoint-conditionee poli- cies for semantic manipulation. ArXiv, abs/2306. 16605, 2023. URL https://api. gemanticschole wtt.:oor///ooooo. 2592872399288888
[56] P. Parashar, V. Jain, X. Zhang. J. Vakil, S. Powers, Y. Bisk, and C. Paxton. Slap: Spatial- language attention policies. In Conference on Robot Learning, 2023. URL https::/aaaaat
semanticscholat org httg:/CoruII:::52987777
[57] P. Marza, L. Matignon, O. Simonin, and C. Wolf. Task faskiconditioned adaptation of visual fea- tures in multi-task policy learning. ArXiv, abs/2402.07739, 2024. URL https://apt
intannicscholaz
Segg/CoopssD::: : 22T.:ID::2766699990
[58] W. Chen, O. Mees, A. Kumar, and S. Levine. Vision-language models provide promptable representations for reinforcement learning. ArXiv, abs/2402.02651, 2024. URL https:// api. www.stiemacseaeeeeeeeee
[59] R. Ma, J. Luijkx, Z. Ajanovic, and J. Kober. Explorlim: Guiding exploration in reinforcement learning with large language models. ArXiv, abs/2403.09583, 2024. URL https:://// //api. cemanticsches
org :atpp/Crri::D/000 268385552222222
[60] A. Eftekhar, K.-H. Zeng, J. Duan, A. Farhadi, A. Kembhavi, and R. Krishna. Selec- tive visual representations improve convergence and generalization for embodied ai. ArXiv, a.0/231,.04193, 2023. URL https: //ap:://ap.... semanticscholar: peg/CorppsID: 265043632.
[61] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, 2022. URL https: / //api. semantettiii.. aschol ar. /aarg//eruuu////eeeeeeeeeeee ID: 250526250.
[62] O. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin. S. Balaji, V. Balcom, Baltescu, H. Bao, M. Bavarian, Belgum, I. Bello, J. Berdine, G. Shrnprot C. Berner, L. Bog- donoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. But- ton, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, Gu, Y. Guo, C. Hallacy, Han. J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, L. Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, H. Kirchner, J. R. Kiros, M. Knight, D. Kokotajlo, L. Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Le- ung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, Lue, A. A. Makanju, K. Malfacini, S. Manning, Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. P. Mossing, T. Mu, M. Murati, O. Murk, D. M ely, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, O. Long, C. Keefe, J. W. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Paras- candolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Bel- bute Peres, M. Petrov, H. de Oliveira Pinto, M. Pokorny, M. Pokrass, V. H. Pong T. Powell, A. Power, Power, F. Proehl, R. Puri, Radford, J. Rae, A. Ramesh, C. Raymond, Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. D. Saltarelli, T. Sanders. S. San- turkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shich, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. D.
13

---

Sokolowsky. Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. A. Tezak, M. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. L. Wainwright, J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, Weng, L. Weng, M. Wi- ethoff. D. Willner, C. Winter, Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. Gpt-4 technical report. 2023. URL https: / //ap:.///p..... http:/pp.aaaaaaaaaee..eeeee semant icscho ar. or..or//ooopsstDD///
II: 257532815..
[63] Ren, S. Liu, A. Zeng, J. Lin, K. Li, Cao. J. Chen, X. Huang, Y. Chen, F. Yan, Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L.. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024.
[64] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. ArXiv, abs/1910. 1956, 2019. URL https: / /api. semanticschol ar. otrg//orrpsSTT::24455555
14

---

A Supplementary Material
We first go policy architecture details in Section 4. We then present additional descriptions of the simulated experiment setup in Section 5.1 and the real robot experiment setup in Section 5.2. Lastly, we show results for one additional ablation experiment for the real robot experiments in Section 5.2. For visualization of the real robot experiments as well as the zero-shot skill chaining experiments mentioned Section 5.2, please see videos attached.
A.1 Architecture Implementation Details
In Section of the main paper, overviewed the architectural choices. Here, we provide more detailed description of the implementation details. In both simulated and real robot experiments, we train a separate policy for each task. For both environments, we use DINO-v2 with ViT-B/16 backbone to encode objects and parts. In simulated experiments, the policy network is implemented as a 4-layer MLP with hidden sizes [5 1512,256,128 128], and the concatenation all token outputs from the attention layer is taken in. In real robot experiments, we use 3-layer MLP with hidden sizes [1024, 1024] instead. Under the robot's hardware constraint, we only input the CLS token into the policy MLP reduce the number of parameters. All methods share the same policy network architecture.
A.2 Simulated Experiments Setup
In Section 5.1 of the main paper, we briefly describe the five simulated tasks. Now we will go over a detailed description of each task and how the task-relevant objects are selected: In OpenMierowave, the goal is to open microwave sitting a kitchen counter. It requires the agent to locate the microwave and its handle. In SlideCabinetDoor and OpenCabinetDoor, agents need to locate the handle of cabinet doors and open them. In TurnOnLight and TurnKnob, agents need to turn the perspective knobs on panel. In Mpennicrowvve, the task-relevant object is selected by prompting GroundedS with "microwave." " In SlideCabinetDoor, the task-relevant object is selected by prompting GroundedSAM with "cabinet." For the rest of the tasks, we annotate the task-relevant object locations. Note that since the positions of objects in the environments are fixed, we only need to annotate the position of the task-relevant objects once.
A.3 Real Robot Experiment Setup
We use 7-DoF Franka robot arm with continuous joint-control action space at 15 Hz. A Zed 2 camera is positioned on the table's right edge, and only its RGB image stream-excluding depth information-2a employed for data collection and policy learning. Another Zed mini camera is affixed the robot's wrist. We encode the wrist image with DINO-v2 and pass the CLSS token as an additional token the policy during training. Operating under velocity control, robot's action space encompasses 6-DoF joint velocity and singular dimension of the gripper action (open close). Consequently, the policy produces 7D continuous actions.
A.4 Additional Ablation Experiment for Real Robot Setup
Similar to the simulated experiments, we perform the ablation experiments Ours Sisi-mmuli-eeeee where we remove object decomposition. Our main observation is that compared to this ablation, our method performs more robustly in more complicated tasks where identifying parts crucial to the task's success. Especially in Pout Water From Kettle into Pot. where firm and secure grasp is needed to pick up the kettle and precise location of the pot is needed, mulsi- level succeed 11 times in IND setup and only 6 times in OoD setup, proving that having the ability to identify and locate the parts greatly improves the task success rate in both IND and OoD cases. Full results are Table
15

---

Task
Eggplant-
Kettle-Stove
Faucet
Eggplant-Pot
Watcr-Pot
Overall
Method
# #of Trials
15
15
15
15
75
IND
OoD
|IN
OoD
|IND
OoD
IND
OoD
[IND
OoD]
IND OoD
HODOR (Ours)
14
12
13
12
12
8
13
12
9
64
49
Ours-multi-Ieve:
14
12
11
12
11
45
Table 2: IND and OoD BC Results on Real Robot Tasks. task out of 15 trials.
We report the number of success of each
A.5 LLM-based task-relevant object name identification
In this paper, we focus on evaluating short-horizon tasks, where natural task descriptions typically make the relevant objects immediately transparent. This is true for our tasks: [Put Eggplant in Sink, Place Kettle on Stove, Open Faucet, Put Eggplant in Pot, Pour Water from Kettle into Pot]. Identifying the right entities (e.g. "eggplant" and "sink") in these task names is very easy for modern LLMs, so we used very simple prompts: "Imagine you are robot performing the task "(insert-task- name)", " what are the task- objects?" With GPT-4, this works with 100% accuracy across all our tasks. We include the results from GPT-4 for all tasks in Fig (a)-(e).
tnopihe you rnbbtt performing Maac Nottle stowe. what taclenoloven obgects?
tackrodeenn
par bask "The fapplent Sink," rolevant objents are:
the "Pare Keltle en Stowe the trnk onlovent okients
1. Oegplent that slaced thwairnk.
1. Kettle The tte tat seeds be ploced stowe
locatizo whwre agga reech to bew
2. Stowe acation where the sottle noeds to be aced.
Addi onaily sink alreadyss in zae obstruciad. objecta reldted the use dish ioes, frunei, or other disthes might ind relevant
Add denallyylly, stove siturned off other me it those mn git beneevenn considor for propar ssccutio the tack.
(a) Put Eggplant in Sink
(b) Place Kettle on Stove
Imegine are robot performinn thet "apnn what the Lask walaaeaaaantt ofjects?
Imagine
C
For the tzsk "Opas Roxeet" "he obl oby are:
Reacet The ten needs to ooenod.
sggalartin in Pot." toderolenant ofjects
Sisk Tap: Thetiature where the neanet in borateed conateer foe teareth. functon
Espplant be sloced thepst.
aggs be pioeed.
T'e taceth pf
e tree handles, abso relewant
pats relevont cosderattiee
night
la comidee snaculing the
(c) Open Faucet
(d) Put Egeplant in Pot
Imagin nabbot perforr rng the lask Your waber from Cattle in Tot, what busk relewant objects?
the Mettettiiin Pot." "Unclank look ohiints
Weder, selecent odjento are.
Kettle
Wuter:: The andouaooonnn wood
tue thersstanerrie which water from
Heat be stowe kettle, of other appikinc to generate
Water substance belng transferred from
ner: The bom (such as kkttll er pot) holds water baingg hested.
Adsitioally if thene conetra lie the lettle or pal, the pot slraady these co relevant meytty performing task.
Add precess invlves any contreis sectings adjusin y semporcturd vode abe be relevent seccosslllly compo taok.
(e) Pour Water from Kettle into Pot
(f) Boil Water
Figure GPT-4 responses for task-relevan object name identification
A.6 Technical Details for Generating Part Masks and Training
We first identify masks for the task-relevant objects by prompting GroundedSAM. Next, we ask SAM generate all possible masks in the scene and keep the ones that have an intersection 0.5 with the task-relevant object masks. In this way, generate comprehensive list of part masks for every task-relevan object. We provide visualization of objects and part masks for each real-
16

---

(a) Object-part segmentation masks for faucet.
(b) Object-pate part segmentation masks for pot.
(c) Object-pan segmentation masks kettle.
9330 onT
(d) Object-pan part segmentation masks for eggplant.
Figure 10: GroundedSAM results for detecting task-relevant objects in real-robot setting. The first image for each row shows GroundedSAM's detection and segmentation results. The following frames show the part masks for that task-relevant object.
robot task in Fig. 10. During training time, we precompute the segmentation masks and build the representation for each frame. So no pre-trained model is occupying the GPU during training time. We use one Nvidia 3090 GPU to train each mode and training one model roughly takes 6-8 hours.
A.7 Limitations and Failure Case Analysis
While combining multiple pre-trained models might priori carry the risk of error aggregation and brittleness compared to other approaches, we do not observe this to be the case empirically for HODOR in our extensive evaluations across 10 simulated and real-robot tasks. However, still valuable to understand how component failures affect task performance, and we present this analysis below.
Failures of LLM-based task-relevant object name identification: It is plausible that natural task names sometimes do not reveal the relevant objects so easily, particularly for long-horizon / high- level tasks. To study what might happen in such settings, we have now paraphrased the "Place Kettle on Stove" task to "Boil Water". GPT-4 still succeeds in generating the suggestions that contain the correct object names, as shown in Fig. 9(f). GroundedSAM will still return the same results this case, since it will simply return empty list and if the suggested objects are in the scene. For still more challenging settings e.g. long multi-step task like "make sandwich", careful prompt engineering may be necessary, together with the use of VLM rather than pure language model.
17

---

Failures of segmenting/tracking the objects of interest: Recall that we prompt Grounded-SAl with the list of objects generated by GPT-4 to segment those objects the first frame, and afterwards propagate the masks of task-relevant objects to all following frames using XMem tracker. In our experiments, first frame segmentation failures are rare: this happened in out of 75 of the real-robot evaluation episodes, and the task policy fails when this happens. This failure case is visualized in Fig. 11. Tracking failures ing the episode due to occlusions or failures in the tracking model, illustrated in Fig. 12. While common, this does not usually result in task failure. Recall that our representation is structured as a coarse-to-fine representation, and the coarsest level is a representation of the full scene, indepen- dent segmentationn/trakingg failures. In this sense, HODOR representations have some built-in redundancy that can aid in robustness. Occasional tracking failures are typically easily handled, and sometimes even segmentation failures.
common: task-relevant objects their parts may be lost dur-
Our approach could be further robustified against first frame segmentation failures by running seg- mentation on multiple keyframes throughout the episode, e.g, once every k frames, rather than only on the first frame. This inplementation-level improvement might improve HODOR performance in more challenging tasks in the future, particularly objects of interest are not visible in the first frame. sum up, while each component model the HODOR system does indeed offer a point of failure, the overall system is often capable of correcting for these errors, and/or these points of fail- are rare in the types of tasks most frequently studied in imitation learning for manipulation today. This is backed up by the fact that HODOR outperforms all baselines in our extensive evaluations
pot. 0.56
Figure 11: GroundedSAM error. In this task, the task-relevant object should be the black pot in the background as well as the kettle. However GroundedSAM outputs false positive mask for the pan in front.
A.8 Additional Ablation Experiments
In addition to the baselines Sec 5.1, we show results on two additional baselines: 1) dense DINO the variant of Ours-mult--level, but instead using the last-layer CLS token as the scene- level representation, we use the concatenation of the dense DINOv2 features. 2) cos-coondttioned dense DINO(te dense DINO) is baseline that, instead of selecting task-relevar information using GroundedS we concatenate CLIP embedding on the task description with the dense DINOv2 representations, and use the same number of self-attention layers as in HODOR to process this dense representation. AS Fig. 13 shows, both of these baselines performs poorly-much worse than HODOR on all five Franka kitchen tasks, ranking near the bottom of the baselines on most tasks. This is not surprising for several reasons. First, using dense DINO features instead of the CLS token would increase the policy complexity. Assuming patch size of 16 by 16 and an image size of 224 by 224, concatenating the dense features would result in 14 14=11 feature tokens instead of 1 (CLS) token. This increases the policy complexity and thus decreases the sample efficiency:
18

---

t1 t2 t3 t4
Tracking errors due to occlusions are fairly common during policy execution. In this evaluation episode from the Pour Water Pot task, XMem loses track of the kettle lid after the robot picks up the kettle, and the handle becomes heavily occluded.
Figure 12:
would need lot more data to train this dense DINO policy. Second, since CLIP features are not trained be aligned with DINO, conditioning on these CLIP features won't help the policy focus on task-relevant features. In contrast, HODOR focuses on rask-relevant objects to form a more compact yet sufficient representation, resulting in sample-efficient policy learning.
OpenMicrowave
SlideCabinetDoor
TurnKnob
TurnOLLight
OpenCabinetDoor
1.0
0.5
0.6
Success Rate
0.5
0.4
0.6
0.6
LIV
0.3
HOBORLDurS)
POCR
0.2
0.1
20
Figure 13: Results for additional baselines: dense DINO and te dense DINO. Both of them are perform poorly on FrankaKitchen tasks. This likely due to lack sample efficiency when using dense DINO features.
A.9 Differences between Ours-Multilevel and POCR
there are subtle differences that are never- theless important. 1) POCR requires that the set of object slots be ordered in the same way across all episodes for a task. This causes brittleness when an object is occluded or less visible in some episodes relative others. HODOR instead orders object slots through tracking only within an episode, not across them. 2) POCR constructs object slot embeddings by using each object mask to create masked object-specific version of the RGB image. before encoding that image with LIV. For small object, the masked image is almost entirely blank, with only few pixels occupied by the object; the LIV encoding can fail to capture sufficient details of the object. To avoid such failure points, HODOR uses the masks directly to pool DINO features within each object region. These differences are proven to be important, as when we compare Fig 5 with Fig 6, Ours-multileva outperforms POCR in most cases.
POCR is not just 1-level implementation of HODOR:
19


Reviews:
[{"review_id": "ghhVdxZU96", "reviewer": "Reviewer_U4ZG", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel approach based on hierarchical object-part based representations for imitation learning in robotic manipulation tasks.", "summary_of_recommendation": "See my questions and weakness reviews. I am happy to raise my score if the limitations of the system are clearly studied (experimentally) and discussed in revisions.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. In Figure 4 and Figure 5, is POCR essentially Ours-TC-Multilevel? Based on my understanding, POCR = object-level representation only, and no task conditioning. Then, the results are quite interesting because the performance comparison becomes:\n\nOurs > Ours-multilevel > Ours-TC-multilevel > Ours-TC.\n\nIs this correct? Or are there any other differences between Ours-TC-Multilevel and POCR?\n\n2. Can you provide details of the LLM promoting? Specifically, can you provide the raw prompts, queries, and LLM responses? How are these prompts built and how do you handle failures/stochasticity in LLM-generated outputs?\n\n3. There is related work on part-based manipulation policy learning that should be cited and discussed.\n\nOne-Shot Transfer of Affordance Regions? AffCorrs!: https://arxiv.org/pdf/2209.07147\nComposable Part-Based Manipulation: https://arxiv.org/pdf/2405.05876\nPartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations: https://arxiv.org/abs/2303.16958", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pU8b7bwTfb", "reviewer": "Reviewer_cTv2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Paper proposes 3 levels object-centric representation for more efficient and robust BC. The paper uses ChatGPT + GroundingSAM to filter non-important object representations and represent the scene with only necessary objects for a particular language conditioned task", "summary_of_recommendation": "While promissing current version of the paper is missing some imporant simple baselines and are not adressing some of the limitations.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "# Questions:\n\n1. Parts are defined heuristically as anything that intersects with the object sufficiently (>0.5). It is not clear why IoU is used. The definition of a part is that the intersection is the size of the smaller mask, whereas the union is the size of the larger mask. Thus, this criterion (with some thresholds, e.g., [0.9, 1.1] of the original size) seems like a natural definition of a “part.”\n2. Why is the object-level “positional” encoding unique for each object? How is it determined how many of these encodings would be learned? Are they based on the language description of the objects?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBt52Yd5F3", "reviewer": "Reviewer_fKHY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A novel framework for robot visuomotor control by creating task-specific visual representations. Organizes scene entities hierarchically into objects and parts, allowing robots to focus on task-relevant information. This method improves sample efficiency and performance in complex scenes by filtering out irrelevant details based on language-based task descriptions. Key contributions include a multi-level scene decomposition and a task-conditioned entity selection.", "summary_of_recommendation": "Good paper, some additional results to support the robustness of this approach would be good to help support the applicability of the method to more general settings. I believe the idea is good and the paper could be improved based on the questions above.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "I summarise the questions raised above below:\n\n- Do you have to keep segmenting the parts at each step across the policy execution?\n- What is the computational overhead for segmentation at each step?\n- How does the method handle scenarios where parts of objects are occluded during task execution?\n- If an object's part, such as the handle of a kettle, gets obscured from the camera mid-execution, does the segment for that part disappear? How does this impact policy performance?\n- Do you assume that all parts and the full representation for an object remain consistent across the policy execution, or can some parts disappear? If parts can disappear, how is this managed?\n- How sensitive is your approach to poor segmentation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cmJ7eYWoSX", "reviewer": "Reviewer_9qHC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method (HODOR) that is built upon the idea of several previous works (and notably POCR) – a pipeline of several large pre-trained models (SAM, XMem, GPT, DINOv2) to supply a decomposed representation of the scene for the downstream task of object manipulation via imitation learning. The idea is to provide a hierarchical tree of object parts instead of just the entire segmented object. The relevant entities are extracted by prompting a large network to identify objects of interest in a textual description of the task.", "summary_of_recommendation": "Overall, I see very moderate contribution and novelty in this paper. I have raised several concerns in the “Weaknesses” and “Questions” parts of my review that I would like the authors to address. As such, my current recommendation is “Weak Reject”, but I’m willing to increase my score given convincing answers to my concerns and the concerns that might be raised by the other reviewers. **Post-Rebuttal**: I have read the other reviews and it seems that most of the reviewers raised the same concerns regarding the limitations of this method of relying on a myriad of pre-trained models and the moderate novelty. The paper is well-positioned in the \"using large pre-trained models for robotics\" literature, and while I still have my concerns regarding this pipeline, I appreciate the authors' response and increase my score, but I will not argue against rejecting it if it is the general consensus amongst the other reviewers.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "* Concatenation vs. [CLS] token for pooling the output of the Transformer: I understand the memory reasoning for the real-robot experiments, but do you have actual ablation of the pooling type (even if simulation)?\n* Missing technical details: [lines 150-151] “...SAM to obtain all the segmentation masks at different levels of granularity..”: there is no explanation, even in the appendix, of what exactly is done in this step. Can the authors explain what are the “different levels of granularity” and what is done in practice? Objects can differ in size which might affect how they are further decomposed to parts. As this is the main contribution of the paper, I’d expect a deeper discussion/explanation of this.\n* R3M seems very competitive to HODOR, and significantly outperforms it for the “Open Cabinet Door” task. Why do you think R3M fares better than HODOR in this case?\n* Additional training details: what kind of hardware and computational resources were used to train the models? How long does it take? How do you deal with the memory required for all the large pre-trained models?\n* Why is “skill-chaining” an interesting feature? It seems that the robot state is reset after each task, so it is like solving each task separately. If the authors argue that the previous execution altered the environment state, this is just like testing if the method generalizes to the same task with different initial environment states.\n* Do the authors plan to release an open-source code?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers broadly agree that it is important to find representations that enable learning from a small number of examples.  The proposed method is sensible, but there is concern about its generality, in the face of situations where other objects are important (e.g., something needs to be moved out of the way) or there is substantial occlusion.  An additional concern is the robustness of the system:  what happens in the face of (inevitable) failures of the component foundation models?  In the final revision, it will be important to be as clear as possible about the degree to which this method generalizes (to different layouts, sets of parts, especially when substantially different action sequences are required) and to talk about strategies for improving in that direction.
Decision: Accept