HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation
Authors: Yi Li
Yuquan Deng
Jesse Zhang
Joel Jang
Marius Memmel
Caelan Reed Garrett
Fabio Ramos
Dieter Fox
Anqi Li
Abhishek Gupta
Ankit Goyal
Venue: ICLR 2025 Poster (2025)
Keywords: vision language model; cross-domain generalization; sim-to-real transfer; robot manipulation; vision language action model
Primary area: applications to robotics, autonomy, planning
TL;DR: Hierarchical VLA architectures can enable robotic manipulation with semantic, visual, and geometric generalization after trained on cheap off-domain data

Abstract: Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is the lack of robotic data, which are typically obtained through expensive on-robot operation. A promising remedy is to leverage cheaper, *off-domain* data such as action-free videos, hand-drawn sketches, or simulation data. In this work, we posit that *hierarchical* vision-language-action (VLA) models can be more effective in utilizing off-domain data than standard monolithic VLA models that directly finetune vision-language models (VLMs) to predict actions.
In particular, we study a class of hierarchical VLA models, where the high-level VLM is finetuned to produce a coarse 2D path indicating the desired robot end-effector trajectory given an RGB image and a task description. The intermediate 2D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Doing so alleviates the high-level VLM from fine-grained action prediction, while reducing the low-level policy's burden on complex task-level reasoning.
We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios, including differences in embodiments, dynamics, visual appearances, and task semantics, etc.
In the real-robot experiments, we observe an average of 20% improvement in success rate across seven different axes of generalization over OpenVLA, representing a 50% relative gain.
Visual results are provided at: [https://hamster-robot.github.io/](https://hamster-robot.github.io/)

Published as a conference paper at ICLR 2025
HAMSTER: HIERARCHICAL ACTION MODELS FOR OPEN WORLD ROBOT MANIPULATION
Yi 1. Yuquan Deng"?. Jesse Zhang* Joel Jang' Marius Memmel Caelan Garreft', Fabio Ramos', Dieter Fox Anqi Liti, Abhishek Gupta Ankit Goyalt NVIDIA PUniversity of Washington SUniversity of Southern California
ABSTRACT
Large foundation models have shown strong open-world generalization to com- plex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is the lack of robotic data, which are typically obtained through expensive on-robot operation. A promising remedy is to leverage cheaper, "off-domain" data such as action-free videos, hand- drawn sketches or simulation data. In this work, we posit that hierarchical vision- language-action (VLA) models can be more effective in utilizing off-domaii data than standard monolithic VLA models that directly finetune vision-language mod- els (VLMs) to predict actions. In particular, we study class of hierarchical VI.A models, where the high-level VLM is finetuned to produce a coarse 2D path in- dicating the desired robot end-effector trajectory given an RGB image and a task description. The intermediate 2D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Doing so alleviates the high-level VLM from fine-graine action prediction, while re- ducing the low-level policy's 's burden on complex task-level reasoning. We show that, with the hierarchical design, the high-level VLM can transfer across signif- icant domain gaps between the off-domain finetuning data and real-robot testing scenarios, including differences on embodiments, dynamics, visual appearances and task semantics, etc. In the real-robot experiments, we observe an average of 20% improvement in success rate across seven different axes of generalization over VLe, representing 50% relative gain. Visual results are provided at: https: pettps:/hamsterr robot github...... /o/
INTRODUCTION
Developing general robot manipulation policies has been notoriously difficult. With the advent of large vision-language models (VL.Ms) that display compelling generalization capabilities, there is optimism that the same recipe is directly applicable robot manipulation. A line of prior work (Bro- han et al., 2023a; Kim et al., 2024; Black et al., 2024) builds open-world vision-language-ccioon models (V (VLAs) by finetuning off-the-shelf pretrained VLMs to directly produce robot actions. These VLA models, which we refer to in this work as monolithic VLA models, rely crucially on large robotics datasets, complete with on-robot observations, e.g., images and proprioceptive states, and actions. However, on-robot data is expensive, since end-to-end observation-action pairs are typically collected the robot hardware through, e.g., teleoperation. Despite recent community-wide efforts in building large-scale robotics datasets (Collaboration et al., 2023; Khazatsky et al., 2024), the size. quality, and diversity of existing robotics datasets are still limited, and monolithic VLA models have yet to demonstrate emergent capability comparable to VLMs and LLMs in other domains of study. Moreover, monolithic VLA models are constrained by their inference frequency to achieve dexterous and dynamic manipulation tasks (Brohan et al., 2023a; Kim et al., 2024).
On the other hand, relatively small robot policy models have shown impressive dexterity and robust- ness. Such models have demonstrated promise across a range of complex tasks involving contact- rich manipulation and 3D reasoning, spanning domains from tabletop manipulation (Shridhar et al.,
co-first authors
project lead equal advising

---

Published as a conference paper at ICLR 2025
Imitation Learning
VLA Models
HAMSTER: VLM Imitation Learning
Data: In- Ro-omain Robot Teleop
Data: Large-Scale
Robot Teloop
Data: In-
Robot Teleop
Data: Easy to Obtain, Off-Domain Data
Web Duta
D2wr Appit
HAMSTER VLM
A Drow 20
Low-Level Policy
Vision Language Action Model
Data: In RobiE Twleap
Low-Level Policy
Generalization: Low
Generalization: Medium
Generalization: High
Overview of HAMSTER, VLAs and "smaller" imitation learning methods. HAMSTER's 's hicrarchi- cal design results in better generalization with small amount of in-domain data. HAMSTER is able utilize cheap training sources such as videos or simulations for enhanced generalization.
Figure 1:
2023; Goyal et al., 2023; 2024; Ke et al., 2024) to fine dexterous manipulation (Chi et al., 2023; Zhao et al., 2023). Trained on relatively small datasets, these models show local robustness, and can achieve dexterous and high-precision control. However, they are often brittle to drastic changes in the environment or semantic description of the tasks (Pumacay et al., 2024). These models also can struggle to effectively leverage simulation data for real-world manipulation tasks due to sim-to-real gaps in visual appearances and system dynamics (Li et al., 2024; Mandlekar et al., 2021).
In this work, we ask how can we marry the generalization benefits of large VLMs, with the ef- ficiency, local robustness, and dexterity of small policy models? Our key insight is that, instead of directly predicting robot actions, VLMs can be fine-tuned to produce intermediate representations as high-level guidance on solving the robot manipulation task. The intermediate representation can then be consumed by the low-level policy model to produce actions, alleviating the low-level policy from the burden of long-horizon planning and complex, semantic reasoning. Further, if the inter- mediate representations are chosen such that they are 7) easily obtainable from image sequences; 2) largely embodiment agnostic; and 3) sufficiently robust to subtle changes in dynamics, the VLM can be fine-tuned with off off-domain data where robot actions are unavailable or inaccurate. Such off- domain data does not need to be collected on the actual robot hardware. Examples of off-domain data include action-free video data, simulation data, human videos, and videos of robot with differ- ent embodiments. These off-domain data are generally easier to collect and may already be abundant in existing datasets. We hypothesize, and show experimentally in Fig 7, that this hierarchical sepa- ration can allow VLA models to more effectively bridge the domain gap between off-domain data and in-domain robotic manipulation.
To this end, propose hierarchical architecture for VLAs. HAMSTER (Hierarchical Action Models with SeparaTEd Path Representations), where large fine-tuned VLMs are connected to low- level policy models via 2D path representations! A 2D path is coarse trajectory of the 2D image- plane position of the robot end-effector", as well as where the gripper state changes, i.e., opens and closes (see Fig. 2). These 2D paths can be obtained cheaply and automatically from data sources such as action-free videos or physics simulations, using point tracking (Doersch et al., 2023; Karaev et al., 2025), sand-sketching (Gu et al., 2023), or proprioceptive projection. This allows HAM- STER can effectively leverage these abundant and inexpensive off-domain data when fine-tuning the high-level VLM. The hierarchical design presented in HAMSTER also offers additional advan- tages through the decoupling of VLM training and low-level action prediction. Specifically, while the leeeer-level VI.M is predicting semantically meaningful trajectories from monocular RGB cam- inputs, the lower-level policy models can additionally operate from rich 3D and proprioceptive inputs. In doing so, HAMSTER inherits the semantic reasoning benefits of VLMs along with the 3D
'Representations similar to 2D paths has heen explored in the robot learning literature (Gu et al., 2023), primarily as a technique for flexible task specification. We refer readers to section 2 for a detailed discussion. Forr human vidco, this corresponds the position of the palm center or fingertips.
2

---

Published as a conference paper at ICLR 2025
reasoning and spatial awareness benefits of 3D policy models (Goyal et al., 2024; Ke et al., 2024).
Moreover, the high-level VLM and low-level policy model can be queried at different frequencies
In summary, we study a family of hierarchical VLA models HAMSTERs, where finetuned VLMs are connected to low-level 3D policy models (Goyal et al., 2024; Ke et al., 2024). The 2D paths produced by high-level VLMs serve as guidance for a low-level policy that operates rich 3D and proprioceptive inputs, allowing low-level policies to focus on robustly generating precise, spatially- aware actions. In our experiments, we observe an average of 20% improvement in success rate over seven different axes of generalization over OpenVLA (Kim et al., 2024), which amounts to 50% relative gain, as shown in Table 6. Since HAMSTER is built on both open-source VLMs and low-level policies, it can serve as a fully open-soureed enabler for the community- builddig vision- language-action models.
important to note that while we are certainly not the first to propose hierarchical VLA models (Gu et al., 2023; Nasiriany et al., 2024a), we propose the novel insight that this type of hierarchical decomposition allows for these models to make use of abundant off- domain data for improving real-world control. This opens the door to alternative ways of training large vision-language-actiin models using cheaper and more abundant data sources.
2 RELATED WORK
LLMs and VLMS for robotics. Early attempts in leveraging LLMs and VLMs for robotics are through pretrained language (Jang et al., 2022; Shridhar et al., 2023; Singh et al., 2023) and vi- sual (Shah & Kumar, 2021; Parisi et al., 2022; Nair et al., 2023; Ma et al., 2023) representa- tions. However, these are insufficient for complex semantic reasoning and generalization to the open world (Brohan et al., 2022; Zitkovich et al., 2023). Recent research has focused on directly leveraging open world reasoning and generalization capability of LLMs and VLMs, by prompting or fine-tuning them to, e.g., generate plans (Duan et al., 2024; Huang et al., 2023b; Lin et al., 2023; Liang et al., 2023; Singh al., 2023; Brohan al., 2023b) construct value (Huang et al., 2023a) and reward functions (Kwon et al., 2023; Sontakke et al., 2023; Yu et al., 2023; Ma et al., 2024; Wang et al., 2024). Our work is more closely related to VLA models, summarized below.
Monolithic VLA models as iinguage-conditione robot policies. Monolithic VLA models have been proposed to produce robot actions given task description and image observations directly (Bro- han et al., 2022; Jiang et al., 2023; Zitkovich et al., 2023; Team et al., 2024; Kim et al., 2024; Radosavovic et al., 2023). Monolithic VLA models are often constructed from VLMs (Liu et al., 2024d; Bai et al., 2023; Driess et al., 2023; Lin et al., 2024), and are trained on large-scale on-robot data (Brohan et al., 2022; Collaboration et al., 2023; Khazatsky et al., 2024) to predict actions as text or special tokens. However, due to the lack of coverage in existing robotics datasets, they must be finetuned in-domain on expensive on-robot data. Their action frequency is also constrained by inference frequency, limiting their capability to achieve dexterous and dynamic tasks. The most rele- vant monolithic VLA model to our work is LLARVA (Niu et al., 2024), which predicts end-effector trajectories in addition to robot actions. However, LLARVA only uses trajectory prediction as an auxiliary task to improve the action prediction of a monolithic VLA model. In contrast, our work takes hierarchical approach, enabling us use specialist lower-level policies that take in additional inputs the VLMs cannot support, such as 3D pointclouds, to enable better imitation learning. Our predicted paths then enable these lower-level policies to generalize more effectively.
VLMs for predicting intermediate representations. Our work bears connections to prior meth- ods using vision-language models predict intermediate representations. These methods can be categorized by the choice of predicted representations:
Point-based predictions: A common intermediate prediction interface has been keypoint affor- dances (Stone et al., 2023; Sundaresan et al., 2023; Nasiriany et al., 2024b; Yuan et al., 2024b; Kuang et al., 2024). Keypoint affordances can be obtained through using apcabuocabylary detec- tors (Minderer et al., 2022), iterative prompting of VLMs (Nasiriany et al., 2024b), or fine-tuning detectors to identify certain parts of an object by semantics (Sundaresan et al., 2023). Perhaps most related to our work, Yuan et al. (2024b) finetune a VLM to predict objects of interest well as free space for placing an object, and Liu et al. (2024b) propose a mark- visual prompting procedure to predict keypoint affordances work finetunes a VLM model to not just predict points but rather entire 2D paths, making it more broadly applicable across robotic tasks.
well
fixed number of waypoints. As opposed to these, our
3

---

Published as a conference paper at ICLR 2025
Trajectory-based predictions: The idea of using trajectory-based task specifications to condition low-level policies was proposed in RT-trajectory (Gu et al., 2023), largely from the perspective of flexible task specification. This work also briefly discusses the possibility of combining trajectory- conditioned model with trajectory sketches generated by a pre-trained VLM. Complementary to RT-Trajectory, the focus of this work is less on the use of trajectory sketches for task specifica- tion, but rather a hierarchical design of VLAs such that the high-level VLM can be fine-tuned with relative cheap and abundant data sources. This could include data such as action-free videos, or sim- ulation data that look very different from the real world. We show that the emergent generalization capability of VLMs from its web-scale pretraining allows it transfer to test scenarios of interest with considerable visual and semantic variations. While RT-trajectory uses human effort or off-the-shelf pre-trained VLMs to generate trajectories, we show that fine-tuning VLM models on cheap data sources can generate significantly more accurate and generalizable trajectories (see Table. 5). More- over, our instantiation of this architecture enables the incorporation of rich 3D and proprioceptive information, compared to monocular 2D policies (Gu et al., 2023).
Similarly, the emergence of rrak-any-poiit (TAP) models (Doersch et al., 2023; Wang et al., 2023) has enabled policies conditioned on object trajectories (Yuan et al., 2024a; Xu et al., 2024; Bharad- hwaj et al., 2024) or points sampled from a fixed grid in the image (Wen et al., 2023). While our current formulation focuses on end-effector trajectories, this framework can naturally extend to pre- dicting object trajectories or other motion cues. By leveraging the predictive capabilities of VLMs, such an extension could further enhance the model's ability to generalize across diverse scenarios and improve its capacity for fine-grained motion reasoning.
Leveraging simulation data for training robot policies. There has been extensive work on lever- aging simulation for robot learning. Simulation data is popular in reinforcement learning (RL), as RL on real robotic systems often impractical due to high sample complexity and safety con- cerns (Lee et al., 2020; Handa et al., 2023; Torne et al., 2024). Recently, simulation has been also exploited to directly generate (Fishman et al., 2022) or bootstrap (Mandlekar et al., 2023) large-scale datasets for imitation learning, to reduce the amount of expensive robot teleoperation data needed. Our work takes a different approach using simulation data to finetune a VLM, and showing that VLM is able to transfer the knowledge learned from simulation data to real robot systems, despite considerable visual differences. A related observation is recently made by (Yuan et al., 2024b), but they use keypoint affordances the interface between the VLM and the low-level policy opposed to more general expressive 2D path representations.
BACKGROUND
Imitation Learning via Supervised Learning. Imitation learning trains a policy Te(a 8, 0, 2) from expert demonstrations, where denotes proprioceptive inputs, o includes perceptual obser- vations (e.g., RGB images, depth), and provides task instructions. Given an expert dataset D {s,, o,, 2,, a,)))) ) the policy is optimized via maximum likelihood estimation, maximizing E E(s,,o, =,))),,; [log To (a: s; 0;, 2.]. Despite advancements in architectures such as 3D policy representations (Goyal et al., 2023; Ke et al., 2024), generalizing to novel semantic or visual vari- ations remains challenging. In this paper, we explore how VLMS can enhance imitation learning models for better generalization.
Vision-Language Models. VLMs (Liu et al., 2024a; Lin et al., 2024; Liu et al., 2024d) are large transformer models (Vaswani et al., 2023) that accept both vision and text tokens to generate text responses. They are pre-trained on extensive multimodal datasets (Zhu et al., 2023; Byeon et al., 2022) and later fine-tuned on high-quality, task-speeific data (Shen et al., 2021; Lu et al., 2022). By tokenizing each modality into shared space, these models autoregressively produce sequences of text tokens conditioned on an image and prior tokens. In our work, we assume access to such a pre- trained, text-and-image VL.M (Lin al., 2024; Liu al., 2024d), further fine-tuned via supervised loss that minimizes the negative log- likeliool of the target tokens.
HAMSTER: HIERARCHICAL ACTION MODELS FOR ROBOTIC LEARNING
In this work, we examine how VLA models can leverage relatively abundant data and demonstrate cross-domain transfer capabilities, as opposed relying purely on expensive observatioo-languaae--
4

---

Published as a conference paper at ICLR 2025
VLM Input
Policy Input
Low-Level 3D Policy
Instruction: "Put the spicy food in left bowl."
HAMSTER VLM
Proprio: (REF pos, f...]
Instruction: "Puc the object in the bowl."
VLM response:
[:0.25, 0.1. 0).
(0.29, 0.3, 0% (0.31, 1% [0.33, 0.5,
20 Path
Draw
First Image
Image + Path
Observation
Depth/30
(a) VLM Path Prediction
(b) Low-Level Action Execution
Figure 2: Depiction of HAMSTER's execution. The high-level VLM called once generate the 2D path. The low-level policy is conditioned on the 2D path and interacts with the environment sequentially to execute low-level actions. The path predicted by the VLM enhances the low-level policy generalization capability.
action data collected a robot. HAMSTER is f'amily of hierarchical VLA models designed for this purpose, exhibiting generalizable and robust manipulation. It consists of two interconnected models: first, higher-level VLM that is finetuned on large-scale, off-domain data to produce in- termediate 2D path guidance (detailed in Section 1), and second, a low-level policy that produces actions conditioned on 2D paths (detailed in Section 4.2).
The primary advantages of finetuning such a hierarchical VLM that produces intermediate repre- sentations opposed to directly producing actions with a monolithic model (Kim et al., 2024; Zitkovich et al., 2023; Black et al., 2024) are threefold: 1) our hierarchical VLM can leverage off- domain datasets lack of precise actions, e.g., simulation and videos; 2) we find empirically that hierarchical VLMs producing 2D paths generalize more effectively cross-domain than monolithic VLA models; and 3) the hierarchical design provides more flexibility on the sensory modality, and allows for asynchronous query of large high-level VLA models and small low-level policy models.
4.1 HAMSTER'S 's VLM FOR PRODUCING 2D PATHS TRAINED FROM
DATA
The high-level VLM of HAMSTER predicts a coarse 2D path p to achieve the task given a monoc- ular RGB image img and language instruction i.e., f VLM img, 2). The 2D path p describes coarse trajectory of the robot end-effector, or human hand in the case of human videos, on the input camera image. It also contains information about the gripper state. Formally, the 2D path is defined as p = [are, yt. g :eipper- r_open,)) where Zt, Vt E [0, are normalized pixel locations of the end effector's (or hand) position at step t, and gripper.open, is binary value indicating the gripper state, i.c., open and close.
Although, any pretrained art-and-image-inpu VLM (Lin al., 2024; Liu et al., 2024d; Achiam et al., 2023) can be used to predict such a 2D path by casting an appropriate prompt, we find that pre- trained VLMs struggle with predicting such a path in a zero-shot manner (see Table 5). Therefore, we finetune pre- pre-trained VLMs on datasets that ground VLMs to robot scenes and path predictions collected from easier-to-obtain sources, i.e., internet cataull-uusttionnnneeeeee other modalities, and simulation data. This is in contrast to work such Gu et al. (2023), where pre-trained VLMS are tasked with directly performing spatially relevant path generation.
data, robot data from
We use VILA-1.5- 133 (Lin et al., 2024) as our base VLM, [3-billion-parametee vision language model trained on interleaved image-text datasets and video captioning data. Although it is possible to curate dataset on path prediction {(img, 22, P.)); and train the VLM only on the dataset, the literature (Brohan et al., 2023a; Yuan et al., 2024b) has shown that co-training the VLM on a variety of relevant tasks, all framed VQA tasks, can help retain the VLM's generalization capability. To this end, we curate a multi-domain dataset to finetune this model for effective 2D path prediction.
4.1.1 FINETUNING OBJECTIVE AND DATASETS.
Predicting the 2D path of the end-effector requires understanding what objects to manipulate a given task in terms of their pixel positions, but also reasoning about how robot should perform the task. To enable this understanding, we collate a diverse off-domain dataset Don from a wide range of modalities, including real-world data, visual question-answerte data, and simulation data. Impor- tantly, none of this off-domain data used to train the VLM comes from the deployment environment, thereby emphasizing generalizability.
5

---

Published as a conference paper at ICLR 2025
Pixel Point Prediction
Simulated Robot
Off Domain Robot
Instr.
Find all instances of cushions
Locate objeet between the marked itmss
Find spaces above the bordered Item
Screw the light bulp
Cover bowl with the towe!
Put the marker Inside the Silver
Image
Patha
(10.49, 0.38, 08, (9.53, 0:42,
[10.57, 0.48),
100.50 0,601,
[00.4, 0.6, ciose). (0.4, 0.6, dose), (0.8, 0.7, open))
100.2, 0.2, dese), 10.3,
t00.7, 0.5, dose), (0.5. 0.6, close), (0.6 0.7, dlsee, (0.7, 0.6, open))
0,49),10
0.70 10.43, 0.67),
0.2, dloo), (0.1, 0.2. dove), (0.1, 0.3, open))
0.05)....
(0.55, 0.47),
Figure 3: Off Domain Training Data: Dur contains (a) Pixel Point Prediction: 770k object location tasks from RoboPoint. (h) Simulated Rohot Data: 320k 2D end-effector paths from RIBeench environment. (c) Real Robot Data: 1 110k 2D end-effector paths from Bridge and DROID trajectories.
We assemble dataset Dunr = {img,, 2,, ) of image inputs img,, language prompts 2;, and answer ans; consisting of three types of off off-domain data: (1) pixel point prediction tasks (whar); (2) simulated robotics tasks (what and how); (3) a real robot dataset consisting of trajectories (what and how). We detail each dataset below; see Figure 3 for visualization of each dataset's prompts and corresponding answers.
Pixel Point Prediction. For pixel point prediction, use the RoboPoint dataset (Yuan et al,, 2024b) with 770k pixel point prediction tasks, with most answers represented as list of 2D points corresponding to locations Locate ob ject between the marked items, an input image img and answer ans like (0.25, 0.11), (0.22, 0.19), (0.53, 0.23)]. See the left of Figure for example. This dataset sists of data automatically generated in simulation and collected from existing real-world datasets; its diverse tasks enable the HAMSTER VLM to reason about pixel-object relationships across di- verse scenes while retaining its semantic generalization capabilities.
the image. A sample consists of a prompt like
Simulated Robot Data. We additionally generate a dataset of simulated roboties tasks from RL- Bench (James et al., 2020), a simulator of Franka robot performing tabletop manipulation for wide array of both prehensile and non-predn non-prehensile tasks. We use the simulator's built-in planning al- gorithms to automatically generate successful manipulation trajectories. Given a trajectory, we use the first frame from the front camera as the image input img. We construct prompt 2 to instruct the VLM to provide sequence of points denoting the trajectory of the robot gripper to achieve the given language instruction (see Figure 2). The ground-truth 2D path = [(=re Vt: gr gripper..r. pper_open,)] is given by propriceptive projection using forward kinematics and camera parameters.
We generate 1000 episodes for each of 81 robot manipulation tasks in RLBench, each episode with -4 language instructions, for a total of around 320k (img, 2, ans) tuples, where ans p. See the middle of Figure for an example.
Real Robot Data. Using real robot data allows us to ensure the VLM can reason about objects and robot gripper paths when conditioned on scenes, including real robot arms. We use existing, online robot datasets not from the deployment environment enable this VLM ability. We source 10k trajectories from the Bridge dataset (Walke et al., 2023; Collaboration et al., 2023) consisting of WidowX arm (different embodiment from test robot) performing manipulation tasks and around 45k trajectories from DROID (Khazatsky et al., 2024). We covert both datasets to VQA dataset in as similar way the simulated RL-Bench data, where the 2D paths are extracted from proprioception and camera parameters (see the right of Figure 3 for an example). Note that we essentially utilize the robot data as video data, where the end effector is tracked over time. principle, this could be done with any number of point-tracking methods (Doersch et al., 2023) raw video as well, with no action or proprioceptive labels.
We finetune the HAMSTER VLM on all three types of data by randomly sampling from all samples in the entire dataset with equal weight. We also include a 660k-sample VQA dataset (Liu et al., 2024c) for co-training to preserve world knowledge. We train with the standardized supervised prediction loss to maximize the lig-likelihood the answers ans: Eing, 2, a log VLM (ans; img,, 21).
Note that this is temporally ordered path, but rather set of unordered points of interest in image.

---

Published as a conference paper at ICLR 2025
Remark. One issue with simulation and real robot data is that the extracted 2D paths p can be extremely long, e.g., exceeding one hundred steps. Since we want the HAMSTER VLM to reason at high level instead of the same scale the low-level control policy, simplify the paths pd with the Ramer- Dww.-oo-oggllleeeeeeee curves composed of line segments to similar curves composed of fewer points. We refer readers to Appendix G for an ablation study.
algorithm (Ramer, 1972; Douglas & Peucker, 1973) that reduces
4.2 PATH GUIDED LOW-LEVEL. POLICY LEARNING
The low-level policy of HAMSTER (a 8, O, 2) is conditioned on proprioceptive and perceptive observations, (optional) language instruction and, importantly, 2D path. While a low-level control policy can learn to solve the task without 2D path, the paths allow the low-level policy to forgo long-horizon and semantic reasoning and focus on local and geometric predictions to produce robot actions. As find empirically (see Figure 4), 2D paths allow for considerably improved visual and semantic generalization of low-level policies.
HAMSTER's general path-conditioning framework allows lower-level policies take in proprio- ceptive and perceptual (e.g., depth images) observations, that are not input to the high-level VLM. We consider low-level policies based on 3D perceptual information, i.e., o (img, pointcloud), available at test time on a robotic platform with standard depth cameras. We study two choices of policy architecture, RVT-2 (Goyal et al., 2024) and 3D-DA (Ke et al., 2024) which has shown state-of-the-art results on popular robot manipulation benchmark (James et al., 2020).
Conditioning on Paths. Most policy architectures use the form ao 8, 0, without 2D path inputs. One naïve option is to concatenate the path with proprioceptive or language inputs. However, because 2D paths vary in length, the architecture must handle variable-length inputs. To incorporate the 2D path ý from the VLM without major modifications, we alternatively overlay the 2D path onto the image observation (Gu et al., 2023). Our implementation follows this approach by drawing colored trajectories all images in the trajectory of ..... of= points at each (It,, yr connected with line segments using a color gradient to indicate temporal progression (see Figure 2(b)), and circles mark changes in gripper status (e.g., green for closing, blue for opening). If the policy architecture allows images with more than three channels, we can also include path drawing as separate channels, instead of overlaying on the RGB channel. We empirically study both drawing strategies, overlay and concatenating channels, in section 5.3.
Policy Training. To train the policy, we collect a relatively small-scale task-specific dataset D = {(s, o,, 24, a,)) on the robot hardware. During training, we use oracle 2D paths con- structed by proprioception projection, similar to how the 2D paths are constructed for the VLM training data, and construct path-labeled dataset Pauhh = {(s;, 0z, Zi, Pi, (.))),;))))) [ We train pol- icy (a(a 8, 0, 2, p) with standard supervised imitation learning objectives on Dpat to maximize the log-likelihood of the dataset actions: (. log (a; 3i, Oj, 2, Pt). For further implementation details, see Appendix B.
Inference Speed. Monolithic VLAs query the VLM every action step (Kim et al., 2024; Bro- han et al., 2023a), which can be very expensive with large VLMs. For example, VLen'LA's 7B- parameter VLA only runs at 6Hz an RTX 4090 (Kim et al., 2024). Instead, HAMSTER' hicrar- chical design allows us to query the VLM only one or few times during an episode to generate 2D) paths á that can be followed by low-level policy for multiple steps. Therefore, HAMSTER can be scaled to large VLM backbones without needing end-users to be concerned about inference speed.
5 EXPERIMENTAL EVALUATION
We evaluate our approach in both simulation and real-world experiments to the following key ques- tions. Do hierarchical VLAs:
Q1 Generalize behaviors to unseen scenarios with significant visual and semantic variation?
Q2 Achieve stronger cross-domaie generalization than monolithic architectures?
Q3 Facilitate learning of non- non-prehenee prehensile and long- horghoorzon tasks?
Q4 Exhibit strong demonstration efficiency?
Q5 Have improved visual + semantic reasoning due to hierarchy and VLM fine-tuning?
7

---

Published as a conference paper at ICLR 2025
RVT2
HAMSTER-RVT2
HAMSTERY SODA
100
Success Rale
80
20
Task Vartation Basic
Object
Visual
Language
Spatial
Novel Object
Muttiple
Average
puth down green LooBs
2At 4. blokk powd
pue
bowl
WET Nather
prect Sowt the
E pan
Figure 4: Depiction of quantitative real-work policy execution results on a world robot, evaluated across different axes of generalization and across both prehensile and non- prehensile tasks. Across all generalization axes, HAMSTER outperforms monolithic VLAs and the base 3D imitation learning policies.
5.1 REAL WORLD EVALUATION ON TABLETOP MANIPULATION
To answer Q1, our real-world evaluation experiments aim test the generalization capability of hierarchical VLA models actoss significant semantic and visual variations. In particular, we consider a variant of HAMSTER that uses a VLM VILA--1.-133 (Lin et al., 2024)) finetuned on the data mixture in Section 4.1 as the high-level predictor, with two low-level 3D policy architectures RVT-2 (Goyal et al., 2024) and 3D Diffuser Actor (3D-DA) (Ke et al., 2024) as choices of the low- level policy, described in Section 4.2. The low-level 3D policies collected via teleoperation shown in Fig. 3. Importantly, the high-level VLM has not seen any in- domain data and is only finetuned the off-domain data described in Section 4.1. This suggests that any generalization that the VLM shows result from cross-domain transfer.
trained with 320 episodes
Baseline comparisons. To answer Q2, compare HAMSTER with a state-of-the-ar monolithic VLA, Open (Kim et al., 2024) as well as non-VLM 3D policies, RVT-2 (Goyal et al., 2024) and 3D-DA (Ke et al., 2024). For fair comparison, we finetune OpenVLA on the collected in-domain data described above since Open VLA showed poor zero-shot generalization. The 3D policy (RVT-2, 3D-DA) baselines are trained with the same teleoperation data used train the low-level policy in HAMSTER but without the intermediate 2D path representation from HAMSTER's VLM.
Finetuning Open VLA with RL RLBench. To ensure our method's advantage over Open VLA (Kim et al., 2024) is not solely due to RLBench data, we fine-tuned Open the same RLBench dataset used for HAMSTER's s VLM-1,000 episodes per task across 81 tasks (using only episodes with good front-camera visibility) until achieving over 90% token accuracy (Kim et al., 2024). We then fine-tuned this model on our tasks following the procedure Appendix C.2. In real-world pick-and-place experiments (6 trials over 6 "Basic" tasks as shown in Table 4), RLBench finettned OpenVLA VLA averaged a success score of 0.54 versus 0.58 for the model without RLBench fine-tuning. This suggests that monolithic VLA architectures like Open VLA gain little benefit from RLBench data, likely due to mismatches in action and observation spaces relative to the real-world setup.
Quantitative Results. Figure 4 summarizes our real-world results. To answer Q3, we evaluate across multiple task types, including 'pick and place,' and nonprehensile tasks such as 'press but- tons' and 'knock down objects.' We also test generalization across various axes (Q1) obj and goal: unseen object- goal combinations; visual: visual changes in table texture, lighting, distractor objects; language: unseen language instructions (e.g., candy - sweet object); spatial: unseen spatial object relationships in the instruction; novel object: unseen objects; and lastly, multiple: combination of multiple variations. In total, we evaluate each model 74 tasks for 222 total evaluations. Detailed results and the success score metric provided in Appendix Table 4.
Qualitative Eval on Various Tasks. In addition to the quantitative evaluation conducted for com- parison with VLe we also present qualitative results that demonstrate how HAMSTER's hi- erarchical structure enables low-level policy models to generalize to more complex tasks. Figure illustrates the diverse tasks HAMSTER can handle, including unfolding towel, opening and closing drawers, pressing buttons, wiping surfaces, and cleaning tables. These tasks present challenges such varying lighting conditions, cluttered backgrounds, and semantie understanding requiring exter-
8

---

Published as a conference paper at ICLR 2025
HAMSTER
Low-Level Policy
beck op the (roon
eng bon
sicc up the anad object the nol bool
gadie and pan
Figure 5: Example real-world HAMSTER rollouts demonstrate its strong performance in novel scenes
achieved by leveraging VL.Ms' generalization capahilities and the robust execution of low-level 3D policies.
Method
Success
Method
Original Camera
Novel Camera
Success
Complete
Success
Complete
3D-DA
0.18 0.10
HAMSTER+3DDDA DA (50%)
0.36 0.04
VLA
0.60
0.30
0.23
0.00
HAMSTER+RVT2
0.83
0.70
0.73
0.40
HAMSTER+3D-DA
0.43 0.05
HAMSTER (Concat)
1.00
1,00
0.98
0.90
Table 1: Results on Colosseum demon- strate that HAMSTER is data efficient, achieving 2X the success score of 3D-DA with just 50% the data.
Table 2:
Real world results demonstrate HAMSTER general-
izes to better to novel camera views (see Fig.Figure 6). We ran 10 trails and report averaged success score (success) described in Table 4 and number of successful executions (complete).
nal world knowledge. Additionally, HAMSTER demonstrates the ability to perform long-horizar tasks-none of which are part of the in-domain training set used to train the policy model.
Overall, we find that HAMSTER significantly outperforms monolithic VLA models and (non-VL.M) 3D policies by over 2x and 3x, respectively, on average. This is significant because this improved performance is in the face of considerable visual and semantic changes in the test setting, show- ing the ability of HAMSTER to generalize better than monolithic VLA models non-VLM base models. We further group results by task type in Table 6, where we see HAMSTER outperforms Open VLA across all task types (pick and place, press button, and knock down). See Appendix C for evaluation conditions, a task list, and other experiment details, and Appendix for failure modes.
5.2
SIMULATION EVALUATION
Overall Results. For further investigation into Q1, Q2, and Q3, we conducted a controlled sim- ulation evaluation using Colosseum (Pumacay et al., 2024), which provides significant visual and semantic variations across pick-place and non-prehensile tasks. Pairing our high-level VLM with the state-of-the-ard 3D-DA (Ke al., 2024) policy on RLBench, we compared HAMSTER against vanilla 3D-DA implementation without path guidance. As shown in Table 3 over 5 seeds, HAM- STER outperforms the vanilla approach by an average of 31%. This improvement stems from train- ing with path-drawn images, which encourages the policy to focus on the path rather than extraneous visual features, thereby enhancing robustness to visual variations. We refer readers to Pumacay et al. (2024) for details on the variations and Appendix F for further simulation experiment details.
HAMSTER with Fewer Demonstrations. We also test HAMSTER's ability to work well with limited demonstrations to answer Q4. We test on subset of 5 Colosseum tasks, namely, PLACE_ PLACE.WINE_ME WINE_AT LOCATTOTTIIO,, INSERT_ONEE SOTTOSSQQARREEEGOOO
SLIDE_BLOCK_TO TOL.LCCKTTTTTAGEET
STACK_CUPS, SETUP-CHESS Results in Table 1 demonstrate that HAMSTER+3D-DA with just 50% of the data still achieves 2x the success rate of standard 3D-DA, demonstrating that HAM- STER is demoonstaaiion-fffiiien for the downstream imitation learning tasks.
5.3 VLM GENERALIZATION STUDIES
Finally, we answer Q5: can HAMSTER's hierarchy enable superior visual and semantic reasoning?
VLA from a new camera an- gle (Figure 6) across 10 pick-and-place trials using 6 training objects and 3 training containers to
Camera View Invariance. We test HAMSTER+RVT2 against VLe

---

Published as a conference paper at ICLR 2025
Arg.
no var
bac tox
cum pos
distractor
lig col
man obj col
man obj siz
3D-DAI Ke et a
0.35 0.0.
0.13 0,006
0.34 0.007
0.35 0.1.1
0.39 #0111
0.44 4.4+0.11 0.13
0.41 + 0,01
0.11
HAMSTER (w 3D-DA)
man obj tex
rec obj
rec obj
rec obj
and
lex
3D. Ke ]
0.27
01.04
0.34
0.30
0,36 0.05
0.36
0.12
0.07
1 0.03
0.45
1. 0.12
0.42
0.06
0.23
0.04
HAMSTER 30-DA)
0.48
0.06
0.48
0.05
0.40
0.05
0.56
0.00
0.11 0.10
0.58
0.04
0.56
0.03
0.38 0.07
Table 3:
Simulation evaluation of HAMSTER across different visual variations. We test vanilla 3D Diffuser Actor and HAMSTER across variations in Colosseum (Pumacay al., 2024) and find that HAMSTER gener- alizes more effectively than 3D Diffuser Actor. Avg. indicates mean across variations, including no variation.
Move the
toy cup holder
light bulb in lamp
Push color prees button
Jensen Huang
(b)
Figure 7: HAMSTER's VLM demonstrates strong generalization to unseen scenarios. From left to right: (a) leveraging world knowledge for user-specified tasks, (b) handling out-of-domain inputs like numan-drawy sketches, and (c) transferring from diverse simulations to visually distinct real-world tasks. Blue-to-red lines indicate motion, with blue and red circles marking grasp and release points, respectively.
check HAMSTER's visual spatial reasoning. The results in Table 2 show that HAMSTER signifi- cantly outperforms OpenVLA and remains robust to new camera angles, benefiting from its VLM trained on diverse off-domain tasks across various viewpoints. Additionally, we compare HAM- STER+RVT2 (Concat), where instead of overlaying the path on the input RGB image, we modify RVT-2 to accept 6-channel input by concatenating the original RGB image with separate RGB image containing only the drawn path. We can easily apply this due to HAMSTER's hierarchical nature. Concatenated paths actually achieve the best performance, demonstrating the effectiveness of this path representation, though it is less general and not compatible with all imitation learning policy architectures (such as 3D-DA it uses pre-trained image encoder expecting 3 input chan- nels). One possible explanation is that T' virtual reprojection can fragment the 2D path when is directly drawn on the image, making harder for RVT2 to decode. By providing dedicated path channel (via concatenation), path guidance is preserved more effectively.
VLM Generalization. We further demonstrate the benefit of HAMSTER's hi- erarchy by demonstrating that the VLM generalizes well to visually unique and semantically challenging tasks due to off-domain fine-tuning. We visualize example HAMSTER path drawings in Figure 7, demonstrating HAMSTER' VLM itself effectively reasons semantically and visually for unseen tasks. We further investigate VLM performance in Appendix D.1, where we find that (1) HAMSTER outperforms zero-shot path generation from closed-source VLMs (Gu al., 2023; Liang et al., 2023) and (2) that inclusion of simulation data im- proves HAMSTER'S real-world performance. Both results point to the henefit of explicit hierarchy: off-domain VLM fine-tuning that improves its perfor- mance. See Appendix D.I for further details.
Figure 6: Camera pos. for view in- variance: old (right) and new (left).
CONCLUSION AND LIMITATIONS
In summary, we study hierarchical VLA models that achieve robust generalization in robotic manip- ulation. We introduce HAMSTER, consisting of a finetuned VLM that accurately predicts 2D paths and low-level policy that learns to generate actions using the 2D paths. This two-step architec- ture enables visual generalization and semantic reasoning across considerable domain shifts while enabling specialist policies, like ones conditioned on 3D inputs, execute low-level actions.
This work represents an initial step towards developing versatile, hierarchical VLA methods. The proposed work only generates points in 2D space, without making native 3D predictions. This prevents the VLM from having true spatial 3D understanding. Moreover, the interface of just using 2D paths is bandwidth limited one, which cannot communicate nuances such force rotation. In the future, investigating learnable intermediate interfaces is a promising direction. Moreover, training these VLMs directly from large-scale human video datasets would also be promising.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Wentao Yuan for generously providing the Robopoint dataset. We also acknowledge Entong Su and Yunchu Zhang for their assistance in setting up the robot environment. We are grateful for the support from the Army Research Lab through sponsored research, as well as the Amazon Science Hub for Yi and Marius. We also thank Animesh Garg for many helpful discussions. Finally, we extend our gratitude to Yao Lu, Hongxu Yin, Ligeng Zhu, Borys Tymchenko, and Zhijian Liu from NVIDIA's VILA group for their valuable support throughout this work.
REFERENCES
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Bel- gum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shapiroo Christopher Berner, Lenny Bog- donoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim' on Posada Fish- man, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufci Guo, Chris Hallacy, Jesse Han, Jeff Har- ris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Ni- tish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kon- draciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M' ely, Ashvin Nair, Retichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Ouyang Long, Cullen O' Oeeef, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rim- bach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers. Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tian- hao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. In arxiv preprint, 2023. https: /ttttss//arxivie htttt:ps///xxx..ooo///fff.o.. wwtgg/of///ooo.ooooooooooo.oo. /2303.23.000 08774.
11

---

Published as a conference paper at ICLR 2025
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large lainual model with versatile abilities. arXiv preprint ar arXiv:2308.129000 2023.
Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Pre- dicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint 1:Xiv22405.01522 2024.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi_O: A vision- sasoon-lngaag--ettee flow model for general robot control. arXiv preprint arXiv:2410.24164 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Ri-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:2212.06811 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Her- 70g, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang. Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Hen- ryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt- 2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint 130. 2023a.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287-318. PMLR, 2023b.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Saehoon Kim. Coyo-700m: Image-text pair dataset. htttp:://itthbb.omm/ wwt..///oaaaaabaairaroa coyo-dataset, 2022.
Cheng Chi, Siyuan Feng Yilun Du Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Kostas E. Bekris, Kris Hauser. Sylvia L. Herbert, and Jingjin Yu (eds.), Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: 10.15607/RSS.2022 202.XXXX226....0 iidoi.or//00 00.0000.100 15567/R55 2023. 026.026.
URL https:
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Mad- dukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anchit Gupta, Andrew Wang. Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Lmmerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian lchter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen. Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ccola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang. Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Ra- dosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu. Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan
12

---

Published as a conference paper at ICLR 2025
Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang. Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yun- liang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip. Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma. Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Nor- man Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong. Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart Mart'in-Mart in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti. Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suncel Belkhale, Sungjae Park. Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding. Todor Davehev, Tony Z. Zhao, Travis Armstrong. Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vin- cent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng. Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang. Yao Lu, Yecheng Jason Ma. Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yuch-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang. Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma. Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Fmbodiment: Robotic learning datasets and RT-X models. https: a aariiii org/ abs/ 20os/23:1 2310. 08864, 64, 2023.
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp-
10061-10072, 2023.
David H Douglas and Thomas K Peucker. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica, 10(2):112- doi: 103318/7MM5--6770-777377777 122, 1973.
URL
FW57-6770-0750-772
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied mul- timodal language model. In International Conference Machine Learning, pp. 8488. PMLR, 2023.
Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ran- jay Krishna. Manipulate-anything Automating real-world robots using vision-language models. arXiv preprint arXiv:2406. 18111 2024.
Adam Fishman, Adithyavairavan Murali, Clemens Eppner, Bryan Peele, Byron Boots, and Dieter Fox. Motion policy networks. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski (eds.), Conference Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 967-977. PMLR, 2022. URL https:// f shman2 html.
proceedings, ngs . press/
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pp. 694-710. PMLR, 2023.
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Kei Chao, and Dieter Fox. Rvt2: Learning precise manipulation from few demonstrations, RSS, 2024.
13

---

Published as a conference paper at ICLR 2025
Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong. and Ted Xiao. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches, 2023.
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Alcksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5977-5984. IEEE, 2023.
Wenlong Huang, Chen Wang, Ruohan Zhang. Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learming, pp. 540-562. PMLR, 2023a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pp. 1769-1782. PMLR, 2023b.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):0019 3026, 2020.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Confer- ence on Robot Learning, pp. 991-1002. PMLR, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In International Conference on Machine Learning, 2023.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference Computer Vision, pp. 18-35. Springer, 2025.
Tsung-War Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. First Workshop on Vision-Language Models for Navigation and Manipulation ICRA 2024, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Jocy Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha. Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mereat, Abdul Rehman, Pan- nag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Bai- jal, Mateo Guaman Castro. Daphne Chen, Qiuyu Chen. Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jack- son, Charlotte Lc, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mir- chandani, Daniel Morton, Tony Nguyen, Abigail O' Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Marte Sub- ramanian Ramamoorthy, Dorsa Sadigh, Shuran Song. Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manip- ulation dataset. 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint 004444400.09246, 2024.
14

---

Published as a conference paper at ICLR 2025
Yuxuan Kuang, Junjie Ye, Haoran Geng, Jiageng Mao, Congyue Deng, Leonidas Guibas, He Wang, and Yue Wang. Ram: Retrieval-based affordance transfer for generalizable zero-shot robotic manipulation. arXiv preprint arXiv:2407.04688, 2024.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023,
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xcabc5986, 2020.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Sean Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.cm 00040. 2024.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493-9500, IEEE, 2023.
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre- training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 26689-26699, June 2024.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. Autonomous Robols, 47(8):1345-1365, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Fangehen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03172 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024d.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, 2023.
Yecheng Jason Ma. William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representa- tions, 2024.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Marthn What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Leurning (CoRL), 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang. Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference Robot Learning, pp. 1264-1864. PMLR, 2023.
15

---

Published as a conference paper at ICLR 2025
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple spen-vocabulary object detection. In European Conference Computer Vision, pp. 728-755. Springer, 2022.
Matthias Minderer, Alexey A. Gritsenko, and Neil Houlsby. Scaling open-vocabulary object de- tection. In seirty-eevntth Conference on Neural Information Processing Systems, 2023. URL https: /ttpss//ppeeevviiie net/ forum? dommmnccemnnNn
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: univer- sal visual representation for robot manipulation. In Conference on Robot Learning, pp. 892-909.
PMLR, 2023.
Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. Rt-affordance: Affordances are versatile intermediate representations for robot manipulation. arXiv preprint arXiv:2411002704, November 2024a. URL https:
/ //aaxiv.org/aan abs/
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowl- edge for vlms. In International Conference on Machine Learning. 2024b.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. LLARVA: Vision-action instruction tuning enhances robot learning. In 8th Annual Conference on Robot Learning, 2024. URL http::://ppeeeeiiie net// BerrumiddQ21GGMECCC
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In international conference machine learning, pp. 17359-17371. PMLR, 2022.
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum: benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191 2024.
Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training In Conference on Robot Learning, pp. 683-693, PMLR, 2023.
Urs Ramer. An iterative procedure for the polygonal approximation of plane curves Computer Graphics and Image Processing, 1(3):244 256, 1972. ISSN 0146-664X. doi: https://doi.or// 1016/S0146 008800444666664772880000744 URL https: /wwp///www/////////////////tt... www.scedireeeteeeecceeeeeeeeeeee com/ science/
artiicle/p1//5001666622888
Rutav M Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. In
International Conference on Machine Learning, pp. 9465-9476. PMLR, 2021.
Zejiang Shen, Kyle Lo, Lucy Lu Wang, Bailey Kuehl, Daniel S. Weld, and Doug Downey, Incor- porating visual layout structures for scientific text classification. ArXiv, abs/2106.00676, 2021. URL htthtss:
/ttp:////aivve// org/ abs /2106.216.... 00676.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robol Learning, pp. 785-799. PMLR, 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11530 IEEF, 2023.
Sumedh Anand Sontakke, Jesse Zhang, Séb Amold, Karl Pertsch, Erdem Biyik, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. In NeurlPS, 2023.
16

---

Published as a conference paper at ICLR 2025
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang- Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open-world object manipulation using pre-trained vision-language models. In Conference Robot Learning, pp. 3397-3417. PMLR, 2023.
Priya Sundaresan, Suneel Belkhale, Dorsa Sadigh, and Jeannette Bohg. Kite: Keypoint-conditioned policies for semantic manipulation. In Conference on Robot Learning, pp. 1006-1021. PMLR, 2023.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint 24Xhv:2405.12213, 2024.
Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust ma- nipulation. Robotics: Science and Systems, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL org/abs/ /1w0b///76..... 03762.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
Qianqian Wang, Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19795-19806, 2023.
Yufei Wang, Zhanyi Sun, Jesse Zhang. Zhou Xian, Erdem Biyik, David Held, and Zackory Erick- son. Rl-vim-f: Reinforcement learning from vision language foundation model feedback. In International Conference on Machine Learning, 2024.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Weizstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In 8th Annual Conference on Robot Learning. 2024.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, al. Language to rewards for robotic skill synthesis. In Conference on Robot Learning, pp. 374-404. PMLR, 2023.
Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. arXiv preprint arXiv:2401. 1439, 2024a.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: Dision-languse model for spatial affordance prediction in robotics. In 8th Annual Conference Robot Learning, 2024b. URL https: / //openrevies www....... net. forum? d=GVx66p700t
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-graine grained bimanual manipulation with low-cost hardware. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu (eds.), Roboties: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: (0.15507/SSS.003 202..22222.26666666 URL URLhttps:// httpttttp///.......tttt/t/tttttt http//ii.oooo/////// org/10...000 15..0.1517/R... 2023. XTX. 61.6.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: An open, billion- scale corpus of images interleaved with text. arXiv preprint arXiv:2304.0699 2023.
17

---

Published as a conference paper at ICLR 2025
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference Robot Learning, pp. 2165-2183 PMLR, 2023.
18

---

Published as a conference paper at ICLR 2025
Figure 8: Examples of various robot tasks and environments that HAMSTER can handle. See more details in teaser video at https:/hammsterrrboi,,, github.i.....o....
Point Data
Sim Data
Robot Data
In- In-Domain Data
[(0.49, 0.38,
[(0.1, 0.5, close),
[[0.2, 0.2, close),
Paths
0.08, 0. 06)
(0.1. 0.5, dose),
[0.3, 0.2, close),
(0.53, 0.42,
(0.7, dose),
[0.1, 0.2, close).
0.07, 0.05]]
(0.8, open)]
[0.1, 0.3, open]]
Hamster VLM: VILA-1.5-13b
Low-Level 3D Policy
Instr. Image
pos, depth
Put object
Find instances cushions
wine bottle in the wine rack
Cover with the towel
Image and Path
Proprio/Sensor
Instr.
(a) VLM Training on Roff
(b) Low-level Policy Training on 9
Figure 9: (a): Examples of training data Don used train HAMSTER's VLM. (b): The data used to train HAMSTER's low-level policies.
A VLM FINETUNING DATASET DETAILS
Pixel Point Pred Data. Our point prediction dataset comes from Robopoint (Yuan et al., 2024b). 770k samples in point prediction dataset contain labels given as nd [0.25, 0.11), (0.22, 0.19), (0.53, 0.23)]. or bounding boxes in [[Cx, cy, w, h)] style. Other than that, following Robopoint (Yuan al., 2024b), we use the VQA dataset Liu al. (2024c) with 660k samples which answer VQA queries in natural language such as "What is the person feeding the cat?" We keep these data as is because these VQA queries are likely benefit VLM's semantic reasoning and visual generalization capabilities; we fine-tune HAMSTER's 's VLM on the entire Robopoint dataset as given.
of unordered points such
Simulation Data. with poor visibility on the front_cam view in RLBench. We use the first image each episode combined with each language instruction. The final dataset contains around 320k trajectories.
We selected 81 RLBench tasks of 103 generate data by removing tasks
Real Robot Data. For the Bridge (Walke et al., 2023) dataset, which only provides RGB images, we extract trajectories by iteratively estimating the extrinsic matrix for each episode. each scene, randomly sample few frames and manually label the center of the gripper fingers. Using the corresponding end-effector poses, we compute the 3D-2D projection matrix with PnP (Perspective- n-Point) approach. We then apply this projection matrix to the episodes and manually check for any
19

---

Published as a conference paper at ICLR 2025
misalignments between the projected gripper and the actual gripper. Episodes exhibiting significant deviations are filtered out, and a new round is started to estimate their extrinsic matrix.
For DROID (Khazatsky et al., 2024), large portion of the dataset contains noisy camera extrinsics information that do not result in good depth alignment. Therefore, we filter out trajectories with poor-quality extrinsics measured by the alignment between the projected depth images and the RGB images. This results r-45k trajectories (~22k unique trajectories as trajectories each have 2 different camera viewpoints) which for constructing the VLM dataset Doff described in Section 4.1.
B IMPLEMENTATION AND ARCHITECTURE DETAILS
HAMSTER Prompt
In the image, please execute the command described in (quest) (quest] (/quest). Provide sequence of points denoting the trajectory of robot gripper to achieve the goal. Format your answer as list of tuples enclosed by (ans) and (/ans) tags. For example: (ans) [ (0. 25, 0.32), (0 32, 0.17), (0.13, O. 0.24), (action)Open (0 74, 0 21)1,, (action)Close Grippez(/actior))
Gripper(/action), ((.an)))))
The tuple denotes the x and location of the end effector of the gripper in the image. The action tags indicate the gripper action.
The coordinates should be floats ranging between and I, indicating the relative locations points in the image.
Figure 10: The full text prompt we use to train HAMSTER with on simulation and real robot data (Section 4.1). We also use this prompt for inference.
B. VLM IMPLEMENTATION DETAILS
VLM Prompt. We list the prompt for both fine-tuning on sim and real robot data and evaluation in image and the prompt, except when training Pixel Point
Figure 10. We condition the model Prediction data (i.e., from Robopoint (Yuan et al., 2024b)) where we used the given prompts from the dataset. Note that we ask the model output gripper changes separate language tokens, i.c., Open Gripper/ / Close Gripper, as opposed to as numerical value as shown in simplified depictions like Figure
VLM Trajectory Processing. As mentioned in Section 4. 1, one problem with directly training on the path labels pd is that many paths may be extremely long. Therefore, we simplify the paths p" with the Mamner-Doulls-Peett composed of line segments to similar curves composed of fewer points. We run this algorithm on paths produced by simulation and real robot data to generate the labels yf for Dor- We use tolerance e 0.05, resulting in paths that are around 2-5 points for each short horizon task.
algorithm (Ramer, 1972; Douglas & Peucker, 1973) that reduces curves
VLM Training Details. We train our VLM, VILAI.5-13B Lin al. (2024), on node equipped with eight NVIDIA A100 GPUs, each utilizing approximately 65 GB of memory. The training process takes about 30 hours complete. We use an effective batch size of 256 and learning rate of 10-5, During fine-tuning the entire model includdng the vision encoder- is updated.
B.2 LOW-LEVEL POLICY TRAINING DETAILS
We train RVT2 (Goyal et al., 2024) and 3D-DA (Ke al., 2024) as our lower-level policies. We keep overall architecture and training hyperparameters the about how the inputs were modified other than the 2D path projection follow.
paper settings. Specific details
For low-level policy training, train the policies ground truth paths constructed by projecting trajectory end-effector points to the camera image. In order to also ensure the policies are robust possible error introduced by HAMSTER VLM predictions during evaluation, we add small
20

---

Published as a conference paper at ICLR 2025
RI-Trajectory GPT-4o Prompt
In the image, please execute the command described 'qquestt'. Provide a sequence of keypoints denoting a trajectory of robot gripper to achieve the goal. Keep in mind these are keypoints,
you do not need to provide too many points.
Format your answer list of tuples enclosed by <ans> and </ </ans> tags. For example:
<ans> [ (0.25, 25, 0.32), (0.32, 0.17), (C.13, 0 0.24), <action>Open
Gripper</actions (0 74, 0. 21), <action>Clos ...Ins/ns>>>>>>
Grippers/action:
The tplle denotes point and location of the end effector of the gripper in the image. The action tags indicate the gripper action.
The coordinates should be floats ranging between and I, indicating the relative locations the points in the image.
The current position of the robot gripper is: (current position). Do not include this point in your answer.
Figure 11: The full text prompt we use to prompt RT-Trajectory with GPT4-o.
amount of random noise (N(0, 0.01)) to the 2D path (æ; y) image points during training to obtain slightly noisy path drawings. No noise was added to the gripper opening/closing indicator values.
RVT2 (Goyal et al., 2024). We remove the language instruction for RVT-2 when conditioning on HAMSTER 2D paths.
3D-DA (Ke al., 2024). simulated experiments Colosseum, no changes were needed. In fact, we saw performance drop for HAMSTER+3D-DA when removing language for Colosseum tasks and a small drop in performance when using simplified language instructions. This is likely due to 3D-DA's visual attention mechanism which cross attends CLIP language token embeddings with CLIP visual features, therefore detailed language instructions are beneficial.
In real- world experiments, simplify the language instruction in the same way as for RVT2 when conditioning on HAMSTER 2D paths to encourage following the trajectory more closely with lim- ited data. In addition, we reduced the embedding dimension of the transformer 60 from 120, removed proprioception information from past timesteps, and reduced the number of transformer heads to 6 from 12 in order prevent overfitting.
C REAL WORLD EXPERIMENT DETAILS
C.I TRAINING TASKS AND DATA COLLECTION
For our real-world experiments, we collected all data using Franka Panda arm through human teleoperation, following the setup described in Khazatsky et al. (2024). Below, we describe the training tasks:
Pick and place. We collected 220 episodes using 10 toy objects. most the training data, bowls were placed closer to the robot base, while 3 objects were positioned nearer to the camera. The language goal for training consistently followed the format: pick up the {object] and put it in the (container).
Knock down objects. We collected 50 episodes with various objects of different sizes. Typically, 3 objects were arranged in row, and followed the format: push down the {object).
knocked down. The language goal for training
Press button. We collected 50 episodes with 4 colored buttons. In each episode, the gripper was teleoperated to press one of the buttons. The language goal followed the format: press the (color) button.
21

---

Published as a conference paper at ICLR 2025
RI-Trajectory Code as Policies Prompt
Task Instruction: (task instrrcciion]
Robot Constraints:
The rohot takes input 2D poses with gripper open/closing status of the form (a, ys gripper open == 1)
The gripper open and close with only binary values
workspace a 1 square centered (0.5, 0.5)
The x-axis points rightward and y-axis points downward.
Please write Python code that generates list 2D poses and gripper statuses for the robot to follow. Include Python comments explaining cach step. Assume you can use numpy or standard Python librarics, just make sure to import them.
Enclose the start and end of the code block with <code> and </code> that it be parsed. Make sure that is self-contained script such that when executing the code string, there variable named bobossposes which is list poses of the form: [ (x, Y, gripper), (x, Y, gripper),
...].
Scene Deseription:
<code>
</code>
Figure 12: The full text prompt for RT-Trajectory with Pode-as-Policies top of GPT4-o. The scene description at the bottom comes from an open-vocabulary object detector describing each detected object and its bounding box in the image based on the task instruction.
When training RVT2, which requires keyframes as labels, in addition to labeling frames where the gripper performs the open gripper and close gripper actions, we also included frames that capture the intermediate motion as the gripper moves toward these keyframes.
C.2 BASELINE TRAINING DETAILS
VLe (Kim et al., 2024). Following Kim al. (2024), only utilize parameter efficient fine-tuning (LoRA) for all of our experiments, since they showed that it matches full fine-tuning performance while being much more efficient. We follow the recommended default rank of r=32. We opt for the resolution of 360 x 360 to match all of the baseline model's resolutions. We also follow the recommended practice training the model until it surpasses 95% token accuracy. How- ever, for some fine-tuning datasets, token accuracy converged near 90%. We selected the model checkpoints when we observed that the token accuracy converged, which usually required 3,000 to 10,000 steps using global batch size of either 16 or 32. Training was conducted with 1 or A6000 gpus (which determined the global batch size 16 or 32). Emprically, we observed that checkpoints that have converged showed very similar performance in the real world. For example, when we evaluate checkpoint that was trained for 3,000 steps and showed convergence, evaluating on checkpoint trained for 5,000 steps of the same run resulted in very similar performance.
TT-arajecy (Gu et al., 2023). We implement two versions of RT-Trajectory for the comparison in Table 5, The first (0-shot GPT-4o) directly uses GPT-4o to generate 2D paths with prompt very similar to the one we use for HAMSTER, displayed in Figure 11.
The second version implements RT-Trajectory on of Mode-as-Policies (Liang et al., 2023), as described in RT-Trajectory. We use OWLv2 (Minderer et al., 2023) to perform pocauuacybylary object detection the image to generate list of objects the scene description and then prompt RT- Trajectory with the prompt shown in Figure 12. We also use GPT-4o as the backbone for this method.
22

---

Published as a conference paper at ICLR 2025
Category
OP
RVT2-Skuech
3DDA
3DDA-Sketch
pek up com and put in Work howl
Basc
Pick up the gaps and in the White bowl
0:75
Basic
pock up lie milk and pir in the white bowl
0
0.25
Bask
pick the salt bottle and put in the white bowl
0.75
0.5
Brsse
pick liee shrimp aul it the roll howl
0.75
11.
0
Bask
the
pot
red bowl
0.25
Naoe
prees
Bask
pesss down green botton
Hisic
piuss dowen th: yollow butlin
Basic
peess down the blue button
pied dosen the gremn borile
05.
0.5
Bank
Pish down the pocky
01.5
Bosic
jush cloen the real lug.
0.5
Bask
pusli dows hind boy
Plich
yellow
Object and Gra)
pick up bottle and put
white boul
Object and Gral
pick up haraua and pat in Mack bowl
Object and Goal
pick up the grage and put it the black bowl
0.25
0.5
Obhectt and Gral
picc up the zimmi and put it the nnd bowl
0.75
0.5
Object and Goal
pick up the milk and pit in the while bowl
0.25
0
0.25
Object and Geal
pick up the strimp and put the white bowl
022
0.25
Goal
pick up the cupcake
black
0.5
Obhect (hoi
put pick up the com and put in the rod lovel
bowl
Object and Cral
Object and Gral
pick up the green pepper and pot in the red bowl
0.75
0:25
Object and Gra)
plak up the unange and put in the white bowe
0.25
Visni Ichl: Textur))
pick up the sall bittle and put it in the white hool
Viaral; Table Testuee)
pick up the barara and pat in the bacc bowl
0.25
0.25
01.75
Visun(lighing)
the gripe and it the black howl
025
0.5
0.25
bowl
Visualichutter
pock up
and pir
while
Clutal)
pock up shrmmp and put it the bowl
075
Viual(miss
pick upe green pepper and pot in the mil bowl
0.2
0.25
Visullimis)
pick up the sali bittle and put in the white toul
025
01.25
Vissallsppeanancc change)
pick up the green pepper and pot it the black howl
0.5
0
Visual repecee snce change)
pick up the valf botti and put in the Back bowl
Vieral Table Textrr)
pros down the rnd bution
0.5
Lighini
jecss
zrom
Vissalichutter
press down yellow Sution
Visualimix)
puess down Bluc button
Visuali Table Vexurr)
push dowp the pocky
Viesalichatte))
punli disen lhe greze borile
Vieialichatter
push dows the chocolate box
Visuslonia)
pusl dows the grees horde
0.3
pick up
pick up spoy objezt jui the bowl
object and
Languag
howl
Language
pick up silly object
bool
Lanpugge
pios up the abjozt with maler of encumber and put in hom!
0:25
Language
pick up the object with color of Javender and pot in the black bowl pick up The object with the color sky
Language
and and put it in the contomne with the color cool
0.25
Language
upt Nick with
and put in the container
the eutheslaes
Language
jross
Languggs
pros tle bution
color of cusumber
Langunge
gress the bytion with the color day
0.5
Language
pros Die button with the solor of bunana
angrage
puch down the obyect eolor of leaf
Language
push down the hox comain enchy biseni
Language
push down the big with coler fine
pick up left object and put with
Sponal
pick up the midkle objest and it left pick up the right object and put it the howl
Special
hond
Sperial
0.5
0.25
Syeria
pick up the left abject and put it the right houd
003
01.35
Sponal
pick up the midile object and put it the right howl
0
Syuzil
tiee abject and pur in nightght bowl
03
Sperial
pros down the left button
Systul
jecss
Sponia
press down right button
Spocial
puzhi down lhe object
Spucial
push down the middle object
0.5
9
Sperial
doon the right object
03
0.5
0.5
Nevel Object
pick up the "R" and pot in the rod bowl
0
Novel Ubjject
pik up the boxed juice and put it aod bowt
0.75
0:75
1
Novel Object
the cholate
white bowl
0.55
0.5
Ulbyect
pack up smill
put
Object
Fick up mouse and in
bowl
Nowil Clhject
Pics the and put the while howl
Maltagle
Pick up the lays and put it the pan
0.25
0.35
Mulliple
pick up lle garlic and put it ilacn pin
035
Miltipie
pick up "K" and it the pan
0.5
05
Malriple
jact the pocky and in The
0.35
0.25
0.25
Table 4: Detailed results of real-world evaluation. The first column indicates the variation category, while the second column presents the language instruction. For the pick ck and pl alae task, 0.25 points are awarded for each successful action: reaching the object, picking it up, moving it to the target container, and placing it inside. For the knock down task, 0.5 points are awarded for touch- ing the correct object and successfully knocking it down. For the press but t on task, 0.5 points are awarded for positioning the gripper above the correct button and successfully pressing it.
23

---

Published as a conference paper at ICLR 2025
Original Hamster
Hamster
RT-Trajectory (CaP) RT-Trajectory (GPT4-0)
Screw light on
block the Teylor Swift
OCoo
Instr: Press with banana
Figure 13: Human VLM evaluation example images and instructions along with corresponding trajectories from HAMSTER without any finetuning (RLBench) simulation data, HAMSTER finetuned on all the data in Section 4,1, RT-Trajectory (Gu et al., 2023) with Code-as-Policies (Liang al., 2023) powered by GPT-4o (Achiam al., 2023), and RT-Trjaectory powered by GPT-4o directly.
C.3 EVALUATION TASKS
We evaluate our method on the tasks of pick and place, knock down object, and press button across various generalization challenges, as illustrated in Figure 4. Detailed results are available in Table 4. Following (Kim et al., 2024), we assign points for each successful sub-action. For VLM, human experts are employed to assess the correctness of the predicted trajectories.
D EXTENDED RESULTS
D.1 IMPACT OF DESIGN DECISIONS ON VLM PERFORMANCE
To better understand the transfer and generalization performance of the proposed hierarchical VLA model, analyze the impact of various decisions involved in training the high-level VLM. We con- duct human evaluation different variants of a trained high-level VLM on a randomly collected dataset of real-world test images, shown in Figure 7. We ask each model generate 2D path traces corresponding to instructions such as "move the block on the right to Taylor Swift" or "screw the light bulb the lamp" (the full set Appendix D.2). We then provide the paths generated by each method to human evaluators who have not previously seen any of the models' predictions. The human evaluators then rank the predictions for each method; we report the average rank across the samples in Table 5.
We evaluate the following VLM models: (1) zero-shot state-of-the-ee closed-soures source models such as GPT-4o using a similar prompt to ours (shown in Figure 11), (2) zero-shot state-of-the-ae closed- source models such as GPT-4o but using Code-as- Poliicccs (Liang al., 2023) to generate paths as described in Gu et al. (2023) (prompt in Figure 12), (3) finetuned open-source models (VILA-1.55 13b) on the data sources described in Section 1. but excluding the simulation trajectories from the RLBench dataset, (4) finetuned open-source models (VILA-1.5-13b) on the data sources described in Section 4.1, including path sketches from the RLBench dataset. The purpose these evaluations to first compare with closely related work that generates 2D trajectories using pretrained closed source VLMs Gu et al. (2023) (Comparison (1) and (2)). The comparison between (3) and (4) (our complete method) is meant isolate the impact of including the simulation path sketches from the
24

---

Published as a conference paper at ICLR 2025
Method
VLM
Finetuning
Rank Exc. Real RLB.
Rank Real RLB.
Rank
Data
All
RT-Traj-
O-shot GPT-4o
3.40
3.63
3.47
RT-Traj.
CaP GPT-4o
3.57
3.36
3.41
HAMSTER
VILA
Our Exc. Sim RLB.
1.78
2.39
2.13
HAMSTER
VILA
Our
1.59
1.28
1.40
Table Ranking-hasse based human evaluation different VL.Ms, averaged across various real-world evaluation tasks. Results indicate that HAMSTER including simulation data is most effective since it captures both spatial and semantic information across diverse tasks from RLBench. This significantly outperforms zero-shot VLM- based trajectory generation, described in Gu al. (2023)
RLBench dataset. In doing so, we analyze the ability the VL.M to predict intermediate paths to transfer across significantly varying domains (from RLBench to the real world).
The results suggest that: (I) zero-shot path generation, even from closed-source VLMs Gu et al. (2023) such as GPT-4o with additional help through Mode-as-Policcee (Liang et al., 2023), under- performs VLMs finetumed on cross-domain data as HAMSTER; (2) inclusion of significantly different training data such as low-fidelity simulation during finetuning improves the real-world per- formance of the VLM. This highlights the transferability displayed by HAMSTER across widely varying domains. These results emphasize that the hierarchical VLA approach described in HAM- STER can effectively utilize diverse sources of cheap prior data for 2D path predictions, despite considerable perceptual differences.
D.2 VLM REAL WORLD GENERALIZATEO STUDY
The full list of task descriptions for this study is below (see Appendix D.1 for the main experiment details). Duplicates indicate different images for the same task. We plot some additional comparison examples in Figure 13. Note that the path drawing convention images for this experiment differ from what is given to the lower-level policies as described in Section 4.2 as this multi-colored line easier for human evaluators to see.
1. screw in the light bulb on the lamp
2. screw in the light bulb on the lamp
3. screw in the light bulb on the lamp
4. screw out the light bulb and place it the holder
5. screw out the light bulb and place it on the holder
6. screw in the light bulb
7. screw in the light bulb on the lamp
8. move the blue block on Taylor Swift
9. pick up the left block and put it on Jensen Huang
10. move the block on the right to Taylor Swift
11. place the yellow block on Kobe
12. pick up the blue block and place on Jensen Huang
13. move the red block to Kobe
14. press the button on the wall
15. press the button open the left door
16. press the button open the right door
17. open the middle drawer
18. open the bottom drawer
19. open the top drawer
25

---

Published as a conference paper at ICLR 2025
20. open the middle drawer
21. open the bottom drawer
22. press the button
23. press the button
24. press the orange button
25. press the orange button with black base
26. press the button
27. pick up the SPAM and put it into the drawer
28. pick up the orange juice and put it behind the red box
29. pick up the tomato soup and put it into the drawer
30. pick up the peach and put it into the drawer
31. move the mayo to the drawer
32. move the dessert to the drawer
33. pick up the object on the left and place on the left
34. pick up the fruit on the left and put the plate
35. pick up the milk and put it the plate
36. press the button with the color of cucumber, then press the button with color of fire
37. press the button with color of banana
38. press the button with color leaf
39. press the button with color leaf, then press the one with color of banana
40. press left button
41. pick up the left block the bottom and stack it on the middle block top
42. make on top of C
43. put number 2 over number 5
44. stack block with lion over block with earth
45. pick up the left block on the bottom and stack it on the middle block on top
46. stack the leftest block on the rightest block
47. stack the block 25 over block
48. put the left block on first stair
D.3 HUMAN RANKING
pick up the block and place on Jensen Huong
An example results for human ranking. The trajectory is from blue to red with blue circle and red circle denotes gripper close point and open point respectively. The grader is asked to provide a rank to these trajectory about which trajectory has highest chance to succeed.
Figure 14:
Due to the variety of possible trajectories that accomplish the same task, we use human rankings compare how likely produced trajectories are solve the task instead of quantitative metrics
26

---

Published as a conference paper at ICLR 2025
such MSE. To do that, generate trajectories for 48 image-question pairs with HAMSTER w/o RLBench, HAMSTER, Code-as- Policcy (Liang et al., 2023), and GPT4o (Achiam et al., 2023). See Figure 14 for example.
We recruit 5 human evaluators, who are robot learning researchers that have not seen the path outputs of HAMSTER, to grade these 4 VLMs based the instruction: "Provide rank for each method (1 for best and 4 for worst). In your opinion, which robot trajectory is most likely to succeed. Traj goes from blue red, blue circle means close gripper, red circle means open gripper." The evaluators are allowed to give multiple trajectories the same score if they believe those trajectories are tied. As they are robot learning researchers, they are familiar with the types of trajectories that are more likely to succeed. Therefore, these rankings act as meaningful trajectory quality metric.
E FAILURE ANALYSIS
HAMSTER+IVTZ
HAMSTER+3DDA
Success
VLM Follure
Fail to Follow Trajectory
Action Failure
Figure 15: Performance Distribution of RVe2+Sketch and 3DDA+Sketch
This section outlines the failure modes observed during our experiments and provides detailed breakdown of the causes. Failures can be attributed to issues in trajectory prediction, trajectory adherence, and action execution.
E. DIFFERENT FAILURE MODES
Trajectory Prediction Failures The Vision-Language Model (VLM) may fail to predict the cor- rect trajectory due to several factors:
Failure to understand the language goal: Although the VLM demonstrates strong capabilities in handling diverse task descriptions, struggles when the training set lacks similar tasks. This can cause the model to misunderstand the goal and make inaccurate predictions.
Incorrect trajectory prediction: In some cases, the VLM predicts an incorrect trajectory, either by interacting with the wrong objects misinterpreting the direction of the affordance.
Dynamic changes in the environment: Since trajectories are generated at the beginning of a task, significant environmental changes during execution can lead to failure. The model lacks the ability to dynamically adjust the trajectory or reidentify the object initially referenced.
Trajectory Adherence Failures Failures in adhering to the predicted trajectory arise primarily due to:
3D ambiguity: The use of 2D trajectory predictions introduces ambiguities, such as determining whether point is positioned above behind object, leading execution emrrs.
27

---

Published as a conference paper at ICLR 2025
Incorrect object interaction: The low-level action model is not explicitly constrained to strictly follow the predicted trajectory. As a result, may deviate, interacting with the wrong object and causing task failures.
Action Execution Failures Even when the trajectory is correctly predicted and adhered to, action execution may still fail due to:
Execution-speciffc issues: Despite training diverse set of actions, the model may fail during exccution. For example, in grasping tasks, an incorrect grasp angle can cause the object to slip, resulting in failed grasp.
E.2 FAILURE ANALYSIS
Our analysis in Figure 15 reveals distinct failure tendencies across methods.
For RVT, 72% failures stemmed from the low-level model failing to follow the trajectory, while 28% were due to execution failures. In contrast, for 3DDA, only 10% of failures were related to trajectory adherence, with 90% attributed to execution failures.
We hypothesize that this discrepancy arises because RVT incorporates re-projection step, compli- cating trajectory adherence. In contrast, 3DDA leverages a vision tower that processes the original 2D image, simplifying trajectory interpretation.
SIMULATION EXPERIMENT DETAILS
Our simulation experiments are performed on Colos- seum (Pumacay et al., 2024), simulator built upon RL- Bench (James et al., 2020) containing a large number of visual and task variations formance of robot manipulation policies (see Figure 16 for a visualization of subset of the variations). We use the front_camera and remove all tasks which the camera does not provide clear view of the objects in the task, resulting in 14 out of 20 colosseum tasks (we remove get-ice.fron. www.ffrrrigeeeee.. move hanger, open.drawer, turn oven ove..nn). on).
test the generalization per-
Ligle-ooe
b1 Table-Coie
Tab Testin
Colosseum contains 100 training episodes for each task, with- out any visual variations, and evaluates on 25 evaluation episodes for each variation. We follow the same procedure instead multiple
Figure 16: Colosseum benchmark variations. Figure from Pumacay et al. (2024), taken with permis- sion.
other than using just the front_came cameras. We report results in Table 3 after removing variations with no visual variations (e.g., object friction).
Task
RVT2
3DDA
VLe
HAMSTER+RVT2
HAMSTER+3DDA
pick and place
0.28
0.19
0.46
0.79
0.78
press button
0.13
0.16
0.25
0.50
0.63
knock down
0.17
0.03
0.41
0.47
0.66
Table 6: Real world average success rates grouped by task type.
G DIFFERENT WAYS OF REPRESENTING 2D PATHS
To investigate the effect the number of points on the 2D path, we train the VLM to predict 1. paths simplified using RDP algorithm, which simplify paths in short horizon tasks to 3-5 points and
28

---

Published as a conference paper at ICLR 2025
GT
38-RDP
3B-20p
13B-RDP
Code SOAee
13B-20p
The task is to pick up the lid and close it the jar with correct color. Task description located on the top-left corner of each image. The trajectory goes from blue to red where blue circles denotes where the gripper should close and red circles denotes where the gripper should open. GT denotes ground truth, 3B and 13B denotes VILAI.5-3B and VILAI.5-13B, RDP denotes paths simplified using Ramer-Douglas, -Peckkrr algorithm while 20p denotes paths reprensented using 20 points.
Figure 17:
is what used in the paper. We denote these paths RDP in the following: 2. Paths represented with 20 points sampled on the path with same step size, denoted as 20p in the following. We keep points where the gripper is executing operation of open close in both methods.
We train the network on RLBench 80 tasks with 1000 episodes for each task and test on 25 episodes the task of close jar. We tried both VILA1.5-31 (denoted 3B) and VILAI.5- 13BBBB (denoted as 13B) as our backbone. Thus we have total 4 combinations over 2 backbones and designs of path representations. We visualize the result in this Figure 17.
From this result we can see that when using smaller models, like VILAI.5-3B, paths represented using points extracted using RDP algorithm outperforms paths represented with a fixed number of 20 points significantly. When the network becomes larger to the level of 13B, the VLM is able handle the representation using 20 points and both two path representations work perfectly. We believe that is because when points are simplified using the RDP algorithm, we usually need less points to represent the path and helps the model to pay more attention predict the accurate position for the gripper open/close points.
29


Reviews:
[{"review_id": "n28kN7xYAd", "reviewer": "Reviewer_95o8", "summary": "This paper introduces a hierarchical model for robotic manipulation using Vision-Language Models (VLMs) to generate intermediate 2D waypoints, which then guide a low-level control policy. The idea is to leverage high-level VLM-based planning to produce simple 2D paths, using less specific and more accessible data sources like videos or simulations. This approach aims to make manipulation models more generalizable across different environments without requiring extensive, robot-specific training data.", "questions": "- How would the model perform on tasks requiring intricate 3D manipulation, like pushing or rotating objects? Given its current reliance on 2D waypoints, would extending this approach to more complex, spatially-aware tasks be feasible?\n- Why were primarily pick-and-place tasks chosen? It would be insightful to see how the model handles a more varied set of challenges to get a clearer sense of its adaptability.\n- Since VLM-based waypoint generation has been explored in past work, could the authors clarify how this approach differs from or improves upon those existing methods?\n- Are there future plans to incorporate 3D input or multi-view data into the model? This could help address some of the limitations around spatial understanding and potentially broaden the approach’s applicability.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The hierarchical approach has a nice structure, separating high-level VLM predictions from low-level control, which could allow for integration of varied data sources in future applications.\n- The focus on using general, “cheap” data sources is interesting and relevant, as scaling data collection in robotics is always a challenge. This could be an efficient way to sidestep the need for large, domain-specific datasets.\n- The generalization goal aligns well with current interests in making robotic systems more adaptable to diverse environments and tasks, and this modular setup has potential.", "weaknesses": "- The paper doesn’t seem to address some core challenges in using VLMs for manipulation, such as understanding 3D spatial relationships. VLMs trained on language and images aren’t inherently equipped for the depth of 3D reasoning required for more complex manipulation.\n- The tasks used are relatively simplistic and limited mostly to pick-and-place scenarios, which doesn’t fully showcase the potential of this approach. Although these tasks have merit, it’s unclear how well the model would handle more challenging or continuous tasks that require a stronger 3D understanding.\n- In terms of novelty, the approach doesn’t seem to introduce significant improvements over existing methods for VLM-based waypoint generation. While the structure is clear and modular, there’s limited demonstration that it extends much beyond what current methods already accomplish.\n- A primary limitation is that the model relies only on 2D waypoints generated from single-view images, which restricts the range of tasks it can realistically handle. For tasks that require nuanced 3D spatial awareness, like pushing or rotating objects, the 2D-only input may limit its applicability.\n- While the system’s hierarchical design has promise, the 2D waypoint restriction means it may not leverage the full potential of VLMs in manipulation tasks, and this makes it less flexible for more sophisticated, real-world interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ar8uTSRiL", "reviewer": "Reviewer_Wx1Z", "summary": "The proposed method, as shown in Fig.1 and 2, leverages the advantages of imitation learning and VLA models to increase generalization ability. It distills knowledge of the task by visual path representation with VLA and uses dataset/task-specific models to deal with the lower-level controls.", "questions": "1. What is the use of the intermediate representations $l_i$ of the hierarchical VLM?\n2. In the fine-tuning phase, are the outputs of VLM normalized across datasets? For instance, the pixel locations are normalized to $[0, 1]$ or not?\n3. What is the length of the output of the Ramer-Douglas-Peucker algorithm?\n4. Which part of the system has the 3D knowledge of the world? Does VLA project 3D paths in the real world to images or does the policy deal with it?\n5. Please unify the notations in the problem formulation, e.g., proprioception vs. path-labeler and proprioception vs. states. Formally define and explain why they share the same notations if you have to use them repeatedly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The baseline comparisons are fair and detailed. More convincing with human evaluation.\n2. As stated in the summary, HAMSTER leverages the advantages of imitation learning and VLA models to increase generalization ability. It distills knowledge of the task by visual path representation with VLA and uses dataset/task-specific models to deal with the lower-level controls.", "weaknesses": "1. Notations of proprioception are used differently (s and p, line 336 and 234). It confuses and breaks the whole problem statement.\n2. There are no (a), (b), and (c) Fig. 5.\n3. The system still requires human-engineered prompts for the VLM and other efforts such as \"Conditioning on Paths\" (page 7) and running the Ramer-Douglas-Peucker algorithm.\n\nMinor:\n1. missing period in line 215.\n2. i.e -> i.e., (line 231).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKvsnsAMke", "reviewer": "Reviewer_MHuh", "summary": "The authors in this work propose a hierarchical action model consisting of a high-level vision-language model (VLM) and a low-level imitation learning policy. These two components communicate via a 2D path representation on the image that is predicted by the VLM and used to guide low-level imitation learning. The major advantage of such a design is to enhance the generalization of manipulation tasks by utilizing abundant \"off-domain\" data for VLM fine-tuning and reducing the requirement of massive amount of \"in-domain\" on-robot data required by the monolithic action models. Both simulated and real-robot experiments have been conducted to verify the claimed improved generalization performance against the other baselines.", "questions": "- Can this intermediate representation limit the versatility of the low-level motions? How do we express the versatility of the motions with the 2D path? On the other hand, the key-point representation can mitigate this constraint. For example, the authors can discuss or empirically evaluate specific types of motions or tasks that might be challenging to represent with 2D paths. \n\n- As mentioned in the experiment part, the number of points of the 2D path can be controlled. How would performance change if we vary the number of points?\n\n- The idea enhances the generalization by using a large VLM for 2D path guidance. Does this increase the computation requirement? Is it worthwhile?\n\n---------------------------------- Post Review ---------------------------------------\n\nI appreciate the efforts of the authors to address my concerns and am pleased to increase my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The target problem is important in enhancing the scalability and generalization of robot action models. The proposed idea of using 2D path intermediate representation to bridge the VLMs and low-level action models is neat and sensible;\n\n- The presentation is in general good to follow and clear to understand. \n\n- The idea of using \"off-domain\" data to fine-tune VLM and consequentially benefit the low-level policy learning is inspiring.\n\n- The experiments include results from both the simulation and the real robot, indicating the workload and applicability of the proposed method.", "weaknesses": "- The foremost concern from my perspective is the mismatch between the technical contribution and the way to sell the claimed idea of hierarchical action models. Technically, this work fine-tunes an open-source VLM with easy-to-collect and \"off-domain\" data for 2-D path/trajectory prediction, which is used to train a low-level imitation learning policy for task execution. The purpose of the authors to enhance the impact of the work is understandable. The current technical and experimental investigation seems insufficient for this purpose. I would suggest the authors either refine the selling points (e.g., \"fine-tuning cross-domain VLM for 2D path-guided imitation learning\") and switch to more concrete and matched contents for presentation or include more experiments with a special focus on the idea of the hierarchy mechanism itself (detailed below).\n  \n- For a more focused investigation of the hierarchy mechanism, if I understand correctly, the first key factor of this idea is the intermediate representation. This key factor has a large impact on the claimed advantage, namely the possibility of utilizing the abundant off-domain training data for fine-tuning. The authors propose to use 2D-path and discuss other alternatives such as 3D-path and key-point representation. It's fine to leave the 3D version for future work. However, the key-point representation can be easily obtained by extracting the start and ending points of the 2-D path. To enhance the focus on the hierarchy mechanism, it would be informative to conduct experiments on this. On the other hand, other related work [1] demonstrates accurate key-point prediction based on open-sourced foundation models without fine-tuning.\n\n- The second key factor in the hierarchy idea is the low-level action models. There are two points on this aspect. The first one is the way to condition the predicted intermediate representation in getting the low-level actions. Though the authors argue that overlapping the 2-D path on the existing images is a good and convenient way, it can help the community to get more insights into this \"augmentation\" if other ways, like concatenation, can be examined and analyzed both experimentally and empirically. The second one is about the choices of the low-level action model, [1] has demonstrated strong performance without using an imitation learning policy. Why do we need such a policy in this mechanism, which still has to be trained based on a few \"in-domain\" data? If yes, what's the difference between using a 2D and 3D policy? The investigation done in this work is insufficient.     \n\n- There is no explicit empirical evidence for this claim of \"drastically reducing the number of demonstrations needed for learning downstream tasks.\" In real-robot experiments, the better performance after being trained with the same amount of path-augmented images couldn't verify this claim. Having a quantitative experiment on how much data has been saved with the 2D-path guidance would be much more informative. For example, comparing learning curves with different amounts of training data to check if the augmented path can facilitate training efficiency.\n\n\n[1] Huang, Wenlong, et al. \"Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.\" arXiv preprint arXiv:2409.01652 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0RWUa4KmrN", "reviewer": "Reviewer_jRfz", "summary": "This paper proposes a two-level hierarchical approach to solving robotic manipulation tasks.  Given an image and a language prompt, a vision-language model (VLM) predicts a 2D gripper trajectory in image space.  Then, a low-level policy generates end-effector motions conditioned on the 2D gripper trajectory.  The authors show that the VLM can be trained using large-scale, offline datasets without expensive, on-robot data.  The low-level policy is trained via imitation learning with small on-robot datasets.  The proposed method, called HAMSTER, is empirically evaluated on a range of real-world and simulated robot manipulation tasks.  HAMSTER outperforms baselines when generalizing to different language prompts, spatial rearrangements, or objects.  The 2D gripper trajectory predictions of HAMSTER's VLM is evaluated based on human evaluations of their accuracy, and it is shown to produce better predictions than the baselines.", "questions": "- It's not clear how 'time' is encoded in the 2D trajectory image.  It is written \"we use a color gradient to indicate progression through time.\"  Is this based on an absolute or relative scale?  Would two trajectories of different lengths end at the same color?  Can this approach encode the speed that the gripper should be moving?\n- How was it decided that the VLM should generate 2D trajectories in image space **and** gripper apertures?  Given that the low-level policy has to predict 6DoF end effector motions, it does not seem unreasonable that it could also predict gripper motion? Alternatively, why not train the VLM to predict the rotation of the gripper (this information would be available in the Robot Simulation Data and the Real Robot Data)?  \n- Why do you only use the `front_cam` view in RLBench, as stated in Appendix A?  Given that the objective is to train a VLM that is robust to changes in view, it seems like the model would benefit from training on all camera views in RLBench.  \n- Most of the images of the experiments in the main paper show the a similar camera view (camera pointing toward robot base).  Did you evaluate the model on different camera views?  Do you have evidence or ideas on how HAMSTER would perform on different views?\n- Why do you think 3D-DA performed worse when the language prompts were removed?\n\nMinor typo: Missing period at bottom of page 4 \"is similar An illustrative\"", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper demonstrates that a VLM can be trained on offline, cheap datasets to produce 2D gripper trajectories to solve a range of robotic manipulation problems not seen during training.  A human evaluation study shows that the proposed training scheme produces trajectory predictions that are better ranked than baselines.\n- The proposed method, HAMSTER, combines this pre-trained VLM with a low-level policy to solve language-conditioned, long horizon manipulation tasks.  This hierarchical approach is shown to significantly outperform non-hierarchical baselines on real-world tasks, especially on variations not seen during training.\n- HAMSTER leverages a large VLM in a way that is still efficient at inference time.  The VLM is queried once at the start, and then only the lighter-weight, low-level policy is used.", "weaknesses": "- The authors do not discuss common failure modes of the proposed approach.  It is written, \"We refer readers to\nthe supplementary website for additional details on the failure modes and evaluation conditions.\"  This discussion is important for understanding the limitations of the approach and should be included in the main paper.  For instance, how often does the VLM produce infeasible paths? Does the low-level policy ever mess up a grasp but still follow the 2D trajectory?\n- The results in the simulation evaluation are not compelling.  On average, HAMSTER outperforms 3D-DA by only 8%, even though HAMSTER's VLM was trained on RLBench (what Colleseum was built upon).   The discussion of this experiment is very brief so it is difficult to gain any insights.  For instance, why does HAMSTER have lower performance and much higher variance on the \"no variation\" task?  Is there a reason why one would expect the VLM to be especially helpful on tasks like \"man obj size\" or \"tab col\"?  Why did the authors switch from RVT-2 to 3D-DA for this setting?\n- It is not clear why human ranking was used as an evaluation metric for the VLM's 2D trajectories.  No information is provided in the Appendix about how the participants were instructed to rank the trajectories, whether these participants were familiar with robot arms, nor how many participants were included.  Is there a reason that the 2D trajectories could not be evaluated based on mean squared error on a set of expert robot trajectories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)