Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
Authors: Lucy Xiaoyang Shi
brian ichter
Michael Robert Equi
Liyiming Ke
Karl Pertsch
Quan Vuong
James Tanner
Anna Walling
Haohuan Wang
Niccolo Fusai
Adrian Li-Bell
Danny Driess
Lachy Groom
Sergey Levine
Chelsea Finn
Venue: ICML 2025 poster (2025)
Keywords: Machine Learning
Robotics
Language
Vision-Language Models
Primary area: applications->robotics
TL;DR: Hi Robot enables robots to follow open-ended, complex instructions, adapt to feedback, and interact with humans.

Abstract: Generalist robots that can perform a range of different tasks in open-world settings must be able to not only reason about the steps needed to accomplish their goals, but also process complex instructions, prompts, and even feedback during task execution. Intricate instructions (e.g., "Could you make me a vegetarian sandwich?" or "I don't like that one") require not just the ability to physically perform the individual steps, but the ability to situate complex commands and feedback in the physical world. In this work, we describe a system that uses vision-language models in a hierarchical structure, first reasoning over complex prompts and user feedback to deduce the most appropriate next step to fulfill the task, and then performing that step with low-level actions. In contrast to direct instruction following methods that can fulfill simple commands ("pick up the cup"), our system can reason through complex prompts and incorporate situated feedback during task execution ("that's not trash"). We evaluate our system across three robotic platforms, including single-arm, dual-arm, and dual-arm mobile robots, demonstrating its ability to handle tasks such as cleaning messy tables, making sandwiches, and grocery shopping.
Videos are available at https://www.pi.website/research/hirobot

Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
Lucy Xiaoyang Shi 12 Brian Ichter' Michael Equi' Liyiming Ke' Karl Pertsch 123 Quan Vuong'
James Tamner
Anna Walling'
Adrian Li-Bell Danny Driess Lachy Groom Sergey Levine Chelsea Finn
Haohuan Wang' Niccolo Fusai
https://wwas
Abstract
inputs, corrections, and feedback. Achieving this kind of flexibility is essential for robots in open-ended, human- centric environments. For instance, consider robot tasked with tidying up table after meal: instead of rigidly fol- lowing single predefined set of steps, the robot would need to interpret dynamic prompts like "only take away someone's ' dishes they are done cating," respond to cor- rections like "leave it alone," and adapt when faced with un- familiar challenges, such as a delicate object that requires special handling. This paper aims advance robotic intel- ligence by enabling robots to interpret and act diverse natural language commands, feedback, and corrections step towards creating agents that reason through tasks, inte- grate human feedhack seamlessly, and operate with human- like adaptability. we can enable a robot to process and engage with complex natural language interaction, we can unlock not only better instruction following, but also the ability for users guide robot through new tasks and correct the robot in real time.
Generalist robots that can perform a range of dif- ferent tasks in open-world settings must be able to not only reason about the steps needed to ac- complish their goals, but also process complex instructions, prompts, and even feedback dur- ing task execution. Intricate instructions (e.g.. "Could you make me a vegetarian sandwich?" or "I don't like that one") require not just the ability to physically perform the individual steps, but the ability to situate complex commands and feedback in the physical world. In this work, we describe a system that uses vision- languae mod- els a hierarchical structure, first reasoning over complex prompts and user feedback to deduce the most appropriate next step fulfill the task, and then performing that step with low-level ac- tions. In contrast to direct instruction following methods that can fulfill simple commands ("pick the cup"), our system can reason through complex prompts and incorporate situated feed- back during task execution ("that's 's not trash"). We evaluate our system across three robotic plat- forms, including single-arm, dual-arm, and dual- mobile robots, demonstrating its ability to handle tasks such cleaning messy tables, mak- ing sandwiches, and grocery shopping.
Achieving this level of flexibility and steerability in robotic systems is challenging. While standard language- conditioned imitation learning can follow simple, atomic instructions such as "pick up the coke can" (Brohan al., 2022), real-world tasks are rarely so straightforward. Imag- ine more realistic prompt, such as: "Could you make me vegetarian sandwich? I'd prefer it without tomatoes. Also, if you have ham or roast beef, could you make sep- arate sandwich with one of those for my friend?" This re- quires not only understanding the language, but also the ability to situate commands within the current context and compose existing skills (e.g., picking up the roast beef) to solve tions and feedback ("thhtt's 's not how you do it, you have to get lower, otherwise you'll keep missing"), these must also be integrated dynamically into task execution. This challenge resembles the distinction between Kahneman's "System 1" and "System 2" cognitive processes (Kahne- man, 2011). The "automatic" System 1 corresponds a policy capable of executing straightforward commands by triggering pre-learned skills, while the more deliberative
1. Introduction
A defining feature of intelligence is its flexibility: people not only excel complex tasks but also adapt to new situa- tions, modify behaviors in real time, and respond to diverse
task. If the robot further receives correc-
PPhysical Intelligence "Stanford University 'University of Califomia, Berkeley, Correspondence to: Physical Intelligence www.earrhhhhhsscllattteee
company>.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Visio--Lagguage-Aciinn Models
PRONPTS
Pnottietions
reast beaf, and lettuce sandvicc??
put broad board
v, sick up scoas put breed lettune
sut lettice beef
Unseen Taske
"Cen cleas up only the trash, but not diskes?"
1. plactic cantairey
2. pice up rell tray
2. put contxiner in tresh
up teg chips
Aituated
"lhtt's net Lowss
higher
chiss
Constraints
"T's allargic to picklas."
Mill
pucklees
2. put bceat chapping bosed
1. pick bread
tomato
Deen ented
Fcompts
"Ie's mavte rightl Can you get come chips, Dreed, and arinks?"
2- put chis bag in basket
1. pick up crarge chip bag
5. rick viter bottle
bastet
Figure I: Open-ended instruction following. Hi Robot enables robots to follow multi-stage instructions, adapt to real-time corrections and constraints, complete unseen long-horizon tasks, and respond verbally when needed.
System involves higher-level reasoning to parse complex long-horizon tasks, interpret feedback, and decide on an ap- propriate course of action. Prior work in robotic instruction following has largely focused on atomic instructions (Step- puttis et al., 2020; Jang et al., 2022; Brohan et al., 2022), addressing only System 1-level behaviors.
ing grounding in the robot's 's capabilities.
The main contribution of paper is hierarchical interactive robot learning system (Hi Robot), a novel framework that uses VLMs for both high-level reasoning and low-level task execution. We show that our framework enables a robot to process much more complex prompts than prior end-to-end instruction following systems and in- corporate feedback during task execution (Figure 1). While some of the individual components of this system, such as the low-level VLA policy, have been studied in prior work, the combination of these components along with our syn- thetic data generation scheme are novel and enable novel capabilities. We evaluate Hi Robot on diverse robots, in- cluding single-arm, dual-arm, and mobile platforms. Our evaluation requires the robots to perform a variety of tasks, including new combinations of skills seen during training. in the context of scenarios that span cleaning of messy ta- bles, making sandwiches, and grocery shopping. Our ex- periments show that Hi Robot surpasses multiple prior ap- proaches, including using API-based VLMS and flat VLA policies, in both alignment with human intent and task suc- cess. By grounding high-level reasoning in both verbal and physical interaction, Hi Robot paves the way for more in- tuitive and steerable human-robot symbiosis, advancing the potential for flexible intelligence in well-woldd applications.
In this paper, we address the more intricate reasoning needed for complex prompts and feedback by introducing a hierarchical reasoning system for robotic control based on lision-language models (VLMs). In our system, the robot incorporates complex prompts and language feedback us- ing VLM, which is tasked with interpreting the current observations and user utterances, and generating suitable verbal responses and atomic commands (c.g., "grasp the cup")) to pass into the low-level policy for execution. This low-level policy is itself a vision-language model finetuned for producing robotic actions, also known as a vision- language-action (VLA) model (Black et al., 2024; Brohan et al., 2023a; Kim et al., 2024; Wen et al., 2024). We expect that robot demonstrations annotated with atomic commands will not be sufficient for training the high-level model to follow complex, open-ended prompts, and we therefore need representative examples of complex prompt following. To acquire this data, we propose to synthetically label datasets consisting of robot observations and actions with hypothetical prompts and human interjections that might have been plausible for that situation. To this end, we provide a sfate-of-the-arr vision-language model with a robot observation and target atomic command, and ask it to come up with a prompt or human interaction that may have preceded that observation and command, i.e. gener- ating high- policy prompts for different outcomes. By but situated ex-
2. Related Work
Our work relates to research on VLMs for robotic control, which we can categorize into two groups: directly training VLMs for robotic control and using VLMs out-of-the-bon with pre-defined robot skills. In the former category, meth- ods fine-tune VLMs to output robotic controls based on in- put images and language commands (Brohan et al., 2023a; Wen et al., 2024; Kim et al., 2024; Black et al., 2024; Liu
incorporating these synthetically-geeenai amples into high-level policy training, our approach gener- alizes to diverse prompts and interjections while maintain-
2

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Action Models
et al., 2024e; Li et al.. 2024; O'Neill al., 2024; Zawal- ski et al., 2024; Zheng al., 2025; Pertsch al., 2025). While such methods have demonstrated impressive gener- alization and instruction-followee
to the corrections seen in the human-written data; our ap- proach leverages VLMS and new data generation scheme to enable diverse prompts and open-ended corrections. Fi- nally, RACER (Dai et al., 2024) can also incorporate situ- ated corrections, but relies on a physics simulator to con- struct recovery behaviors; our approach only uses real robot demonstrations without intentional perturbations or correc- tions and applicable to open-ended prompts.
they are trained for rel- atively simple commands ("put the cup on the plate"). In contrast, we demonstrate tasks with intricate prompts and human interactions that require situated reasoning.
In the latter category, number of methods use LLMs and VLMs to reason over robot observations and commands, and break up multi-stage tasks into simpler steps that can be performed by low-level controllers. Earlier methods of this sort used language models in combination with var- ious learned or hand-designed skills (Huang et al., 2022:
3. Preliminaries and Problem Statement
A learned policy controls robot by processing observation inputs, which we denote oG, and producing one or more ac- tions A, = [a,, A(+1: -1+ where we use A, to denote an action chunk consisting of the next H actions to execute (Zhao al., 2023). Our system takes as input the images from multiple cameras I...., I,.,, the robot's con- figuration (i.e., joint and gripper positions) qt, and lan- guage prompt 4. Thus, we have o, = [T...., I 4,, q:l, and the policy represents the distribution p(A,)o,), Prior works have proposed various methods for representing and training such policies (Zhao et al., 2023; Chi et al., 2023; Octo Model Team et al., 2024; Pertsch al., 2025).
Brohan et al., 2023b; Liang et al., 2023; Shah et al., 2024; Singh et al., 2023; Wang et al., 2024; Li al., 2025a), but such systems have limited ability to incorporate complex context, such as image observations, into the reasoning pro- cess. More recently, multiple works have use VLMs to out- put parameters for pre-defined robotic skills (Huang et al., 2023; Liu et al., 2024a; Nasiriany et al., 2024; Chen et al., 2024; Liu et al., 2024b; Stone et al., 2023; Qiu et al., 2024; Zhi et al., 2024). Such methods can process more complex commands and situate them in the context of visual ob- servations, but these approaches have shown limited phys- ical dexterity and limited ability to incorporate real-time language interaction with humans (with some exceptions discussed below). In contrast, our system utilizes VLMs for both high-level reasoning and low-level control, with a flexible language interface between the two. These de- sign choices, along with new synthetic data generation scheme, allow our system to achieve both significant phys- ical dexterity and detailed promptability that prior works lack.
Since our focus will be specifically on complex, multi-stage tasks that require parsing intricate prompts and even dy- namic user feedback, we need our policies to be able to interpret complex language and ground it via observations of the environment. A particularly powerful approach for handling such complex semantics is provided by vision- language-action (VL.A) models (Black et al., 2024; Brohan et al., 2023a; Kim al., 2024; Wen et al., 2024), which use vision-language model (VLM) pre-training to initial- ize the policy p(A,)o.). A VLM is a language model that has also been trained to process image inputs, and represents a distribution p((" - the probability of a language suffix E' (e.g.. an answer to question) in re- sponse to an image-language prefix consisting of an im- age I and prompt (e.g.. visual question). The most commonly used VLMs represent p(E" II, E) via an autore- gressive decoder-only Transformer model, factorizing the distribution into a product of autoregressive token proba- bilities p(x++1 X1, X2, I), where x, denotes the tth token (not to be confused with a physical time step), and we have i = [x,,... ..., x,,] and E = +I,++; xt,,,,,, with tp the length of the prefix and L, the length of the suffix (Beyer et al., 2024). We also use such Transformer-baae based VLMs, but since we do not modify their architecture and their au- toregressive structure is therefore not relevant to our dis- cussion, we will use the more concise p( p(E' [I, E) notation to represent a standard VLM.
Many works aim to enable robotic language interaction with users, including model-based systems that parse lan- guage instructions and feedback and ground them via a symbolic representation of the scene (Swadzba et al., 2009; Matuszek et al., 2013; Namasivayam et al., 2023; Patki et al., 2019), and more recent learning-based methods that process feedback directly, typically with a hicrarchical ar- chitecture (Liu et al., 2023; Xiao et al., 2024; Shi et al., 2024; Belkhale et al., 2024; Singh et al., 2024; McCallum et al.; Driess et al., 2023; Dai et al., 2024; Hu et al., 2023; Li et al., 2025b). Our work builds on the latter class of methods, where user feedback is incorporated via high- level policy that provides atomic commands to a learned low-level policy. Unlike OLAF (Liu et al., 2023), which uses an LLM to modify robot trajectories, our approach can incorporate situated corrections based on the robot's obser- vations, respond to those corrections in real time, and fol- low complex prompts describing dexterous manipulation tasks. While YAY Robot (Shi et al., 2024) can handle sit- uated real-time corrections, it is limited to one prompt and
A standard VLA is produced by fine-tuning the VLM p(''I, E) such that the actions A, are represented by tokens in the suffix é', typically by tokenizing the actions via dis-
3

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Action Models
of observvation as the standard VLA described in Section 3, except that the language command & is replaced by the output from the high-level policy é, Thus, following the System 1/System 2 analogy, the job of the high-level pol- icy is to take in the overall task prompt & and accompa- nying context, in the form of images and user interactions, and translate it into a suitable task for the robot to do at this moment, represented by l. that the low-level policy is likely to understand. Of course, for simple and familiar tasks, this is not necessary if if we simply want the robot to perform a task that the low-level policy was directly trained for, we could simply set &e = le and proceed as in prior work (Brohan et al., 2022). The benefit of this hierarchical inference process is in situations where either the prompt & is too complex for the low-level policy to parse, too unfa- miliar in the context of the robot data, or involves intricate interactions with the user.
User Prompt Interjection
High-Level Policy (VLM)
Robot Verbal Respanse
Low-Level
Language Command
Low-Level Policy (VLA)
Actions
Joints
Figure 2: Overview of hierarchical VLA. The policy consists of a high-level and a low-level policy. The high-level policy pro- cesses open- open-ended instructions and images from base and wrist- mounted cameras to generate low-level language commands. The low-level policy uses these commands, images, and rohot states to produce actions and optionally verbal responses.
The high-level policy is represented by a VLM that uses the images and & as the prefix, and produces & as the suffix. We describe how this model is trained in Section 4.3.
cretization. We build on the To VLA (Black et al., 2024), which additionally handles multiple images and continuous state observations qt, and modifies the VLM to output con- tinuous action chunk distributions via flow-man Now-matching, but the high-level principles are similar. While such VLA mod- els can follow a wide variety of language prompts (Brohan et al.. 2023a), by themselves they are typically limited to simple and atomic commands, and do not handle the com- plex prompts and feedback that we study in this paper.
Since high-level inference is slower but also less sensitive to quick changes in the environment, we can comfortably run it at lower frequency. A variety of strategies could be used to instantiate this, including intelligent strategies where the system detects when the command é has been completed before inferring the next suitable command. In our implementation. we found a very simple strategy to work well: we rerun high-level inference and recompute & either when one second has elapsed, or when a new in- teraction with the user takes place. This provides reactive behavior when the user provides feedback or corrections, while maintaining simplicity.
4. Hi Robot
We provide an overview of our method in Figure 2. Our approach decomposes the policy (o.) into low-level and high-level inference process. where the low-level pol- icy consists of VLA that produces the action chunk A, in response to simpler, low-level language command, and the high-level policy consists of VI.M that processes the open-ended task prompt, and outputs these low-level lan- guage commands for the low-level inference process. The two processes run at different rates: the lowelevel process produces action chunks at a high frequency, while the high- level process is invoked less often, either after a set time or upon receiving new language feedback. Thus, the high- level process essentially "talks" to the low-level process, breaking down complex prompts and interactions into bite- sized commands that can be converted into actions.
4.2. Incorporating User Interaction
The user can intervene at any point during policy execution and provide additional information and feedback, or even change the task entirely. In our prototype, these interven- tions take the form of text commands or spoken language (which is then transcribed into text). When the system re- ceives user intervention, the high-level inference is trig- gered immediately to recompute &, The high-level policy has the option to include verbal utterance Ue in the com- mand f,, which can be confirmations or clarifications from the robot. When dn is included, we use a text to speech system to play the utterance to the user, and remove it from & before passing it into the low-level policy.
When an interjection ("leave it alone") has been fulfilled, the user can signal to the robot that it may switch back to the previous command and continue the task execution. Notably, the responses of the high-level policy are contex- tual, because it observes not only the prompt &, but also the current image observations. Therefore, it can correctly
4.1. Hierarchical Inference with VLAS
Formally, the high-level policy n (& ..... 17, Ex) takes in the image observations and an open-ended prompt l, and produces an intermediate language command &. The low- level policy ( ...,, Ig le, q:) takes in the same type
4

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Visio--Lagguaee-Aciion Models
generation of the synethetic dataset D Degn in Appendix A.
ROBOT DATA COLLECTION
We train the high-level policy pr ( .... ...... 1,, le) on D Dayn U Diabeled using the cross-entropy loss for next- token prediction.
<SKILL> pick up Kitkat
To train the low-level policy ..., I la. qi). we use Diabeled U Deemo using [I.,
flow fow-mmtching objective, following Black et al. (2024).
2
SYNTHETIC DATA
GENERATION
HIOH-LEVEL POLICY TRAINING
4.4. Model Architecture and Implementation
<INGS>
<USERD>
In our implementation, the low-level and high-level poli- cies use the same base VLM as starting point, namely the PaliGemma-3B VLM (Beyer et al., 2024). The low- level policy is the To VLA (Black et al., 2024), which is trained by finetuning PaliGemma-3B with an additional flow matching "action expert" to produce continuous ac- tions, while the high-level policy is fine-tuned on the image-language tuples described in Section 4.3 to predict commands. While we employ To for our experiments, our framework is inherently modular, allowing for the integra- tion of alternative language-conditioned policies as needed.
Bata-Generane VLM
High-Level Palicy
<USER> can you get me something sweet?
<ROBOT SPEECH> suEe, I can grab you kitkat!
<ROBOT SPEECH>
<SKILL>
Figure 3: Data collection and generation for training the high- level policy. We first collect leleoperated robot demonstrations and segment them into short skills (e.g.. pick up KitKat). Using this labeled data, we prompt vision-language model (VLM) to generate synthetic user instructions (e.g., "Can you get me some- thing "weet?") and robot responses. The resulting dataset is used to train the high-level policy, which maps image observations and user commands to verbal responses and skill labels.
5. Experiments
In our experimental evaluation, we study a range of prob- lems that combine challenging physical interactions with complex user interaction, including multi-stage instruc- tions, live user feedback in the middle of the task, and prompts that describe novel task variations. We compare our full method to prior approaches and to alternative de- signs that use other high-level policy training methods. The aims of our experiments are:
ground feedback like "that's not trash," which is not possi- ble with language-only systems.
4.3. Data Collection and Training Hi Robot
To train Hi Robot in scalable manner, we employ both human-labeled and synthetically generated interaction data, as illustrated in Figure 3. First, we collect robot demonstration data Ddemo via telcoperation. This yields trajectories with coarse language annotations of the over- all goal (e.g., make a sandwich). We then segment these full demonstration episodes into short skills, l,, such as pick up one piece of lettuce, which generally last between one and three seconds. We also heuristically extract ba- sic movement primitives (c.g., small corrective motions) such as move the right arm to the left from the raw robot actions. The resulting dataset Diabeled contains a set of Ie..,,..,II tuples that describe robot skills.
1. Evaluate the ability of our method to follow a variety of complex textual prompts and live user feedback.
2. Compare our full method to prior approaches that train a flat essruction-followinng VLA policy or that use foun- dation models out-of-the the-box for high-level reasoning.
3. Evaluate the importance of synthetic data and hierarchy for task performance and language following.
5.1. Tasks and Baseline Methods
We use three complex problem domains in our experi- ments, as shown in Figure 4.
Table bussing involves cleaning up a table, placing dishes and utensils into a bussing bin and trash items into the trash. The training data consists of full table cleaning episodes. This task is physically challenging because some items re- quire nuanced grasping strategies (e.g.. grasping plate by the edge), the robot must pick up and singulate different objects, and in some cases might even manipulate some objects using others (e.g., picking up plate with trash on it and tilting the plate to dump the trash into the trash bin). In our evaluation, the robot receives prompts that substan-
Next, we use a large vision-language model (VLM) paen to produce synthetic user prompts and interjections l, and corresponding robot utterance 14- Given Drabeled, we prompt pae with both the visual context 13, ..., I7 and the skill label l, (e.g., pick up the letfuce). ygen then imag- ines an appropriate interaction that might have led to l in real user interaction: it generates possible user prompts & (e.g.. "Can you add some lettuce for me?") along with the robot's verbal responses and clarifications vt. We detail the
5

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Actton Models
303E5
*hi robot, you clean only trash, but not dishes?"
"that's trach
TABLE BUSSING
BESOT
up plastic
thren ansy
up pleatic
open grisper
pack
put
RDRET
EESPONDD
*Bue thing"
rphaggs Sarry"
CORMAAD
leave it alone*
CORNAAD
theow plastic
lottle
choestick
home
EDSEUITION
CORMAN
robot, can you
zoast
ROROT COPMAAD
SANDMICH NAKINO
pick up slice of bread
put bread on chopping board
pick up slice of cheese
put chease on bread
piet slice of roast beer
REBST RESPONGO
VRIDE
add
lestuce?*
*ok want"
REOET
put up piece
put lettuce roast
pick up alice
put bread
put bread
return home
REBET RESPONSC
TDC CORE4""
AERUST Bale"
CDPNAND
*hi robot, just cid hike... can you get saze drinks and Twix?"
"I also want Kitkat*
COPNAND
shelf
drink
hand off Twix
EXSCUTION
CORNAND
*elright, let's go*
CORMAND
BUT Kitkat In basket
Nove arms home
to table
grab basket handles
put basket table
adjust basket handles
hove
RESPONGS
Figure 4: Task domains used our evaluation. Across three domains, we evaluate complex instructions, intermediate feedback, and user interruptions. For example, in Table Bussing, when the user says, "that's not trash." the robot correctly puts the bowl back down instead of putting it away. All images are from policy rollouts.
tively alter the goal of the task, such as "can you clean up only the trash, not dishes?", "can you clean up only the dishes. but not trash?", and "bus all the yellowish things". This requires the high-level model reason about the task and each object (e.g.. recognizing that reusable plastic cups are dishes, while paper cups are trash), then modify the
robot's "default" behavior of always putting away all items. This includes understanding what to do and also what not to do (c.g., avoid touching dishes when asked collect only trash). The robot might also receive contextual feed- back during the task, such as "this is not trash", "leave the rest", or "leave it alone," which require it to understand the
6

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Action Models
Tablo Bussing
Sandwich Making
Shopping
Average
INSTRACTION ACCURACY
TASK PROORESS
DISTRUCTION ACCURNCY
TASK PROORESS
INSTRACTEON RECORACY
TASK PROOROSS
INSTRACTION ACCURADY
TASK PROURESS
Flat VLA
SPT-40 High-Level
Hi Robot ... Expert Ruran High-Level (Dracle)
Figure 5: Comparisons to Prior Methods. Hi Robot outperforms GPT-4o and flat VLA on Table Bussing, Sandwich Making, and Grocery Shopping. Hi Robot averages over 40% higher instruction accuracy than GPT-4o, showing stronger alignment with user prompts and real-time observations, and approaches expert human guidance by leveraging its high-level policy.
interjection and respond accordingly.
finetuned with our real and synthetic datasets. This com- parison is similar to an advanced version of SayCan (Bro- han et al., 2023b), which uses an out-of-the-box LLM as high-level policy, while this baseline uses a VLM. To align GPT-4o with the robot's affordances, we carefully en- gineer the prompt to include task-relevant instructions that the low-level policy can follow, determined by ranking the most common skill labels in the -anaaat-d dataset, and ask GPT-4o to choose among them.
Sandwich making requires the robot to make a sandwich, using up to six ingredients as well as bread. This task is physically difficult, because the robot has to manipulate de- formable and delicate ingredients that have be grasped carefully and placed precisely. The data contains examples of different types of sandwiches, with segment labels (e.g.. "pick up one slice of bread"). We use this task to evalu- ate complex prompts, such as "hi robot, can you make me a sandwich with cheese, roast beef, and lettuce?" or "can you make me vegetarian sandwich? les". and live corrections, like "that's all, no more". Grocery shopping entails picking up combination of re- quested items from grocery shelf, placing them into bas- ket, and placing the basket on a nearby table. This task requires controlling bimanual mobile manipulator (see Figure 4) and interpreting nuanced semantics that involve variable numbers of objects. Examples of prompts include "hey robot, can you get me some chips? I'm preparing for movie night", "can you get me something sweet?", "can you grab something to drink?", "hey robot, you get me some Twix and Skittles?", as well as interjections such as "I also want some Kitkat".
Flat VLA: This comparison directly uses the same To low- level policy as in Hi Robot, but without any high level or synthetic data, representing state-of-the-art approach for instruction following (Black et al., 2024).
allergic to pick-
Flat VLA with synthetic data: This ablation uses the low-level policy by itself, without a high-level model, but includes the synthetic data in the training data for the low-level policy, such that it can still process the complex prompts used in our evaluation. This baseline allows us to evaluate the benefit of hierarchy independent from the ef- fect of synthetic data.
Hi Robot without synthetic data: sponds to our method without synthetic training data, eval- uating the importance of including diverse synthetically- generated prompts in training. This ablation can be seen as an advanced VLM-based version YAY Robot (Shi et al., 2024), prior system that uses a high-level model predict language commands for low-level model.
This ablation corre-
Comparisons and ablations. Our comparisons evaluate our full method and a number of alternative approaches, which either employ different type of high-level strategy, or do not utilize hierarchical structure. These include:
Expert human high level: This oracle baseline uses expert human in place of the high-level model, who manu- ally enters language commands for low-level behaviors that they believe are most likely to succeed at the task. This al- lows us to understand how much performance is limited by the low-level policy, with ideal high-level commands.
5.2. Metrics and Evaluation Protocol
We report two complementary metrics, measured by hu- man evaluator who is blind to the method being run. Each evaluation consists of 20 trials per task per method.
Instruction Accuracy (IA). This score measures how well the high-level policy's predicted instruction aligns with hu- man intent, requiring multi-modal understanding of the current environment and prompt. If the prediction from the high-level model consistent with both the user's com- mand and the current observation, the evaluator marks it
GPT-4o high-level model: This method uses the same high-level/low-leeel decomposition as Hi Robot, but queries the GPT-4o API-based model for the high level, while using the same low-level policy. GPT-4o is a sig- nificantly larger VLM than the one we use, but is not
7

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Actton Models
INPUTS
LOM-LEVEL CONNAND PREDICTIONS
USER PROMPT
JHAGZ OBSERVATION
ROBOT SYNTHETIC DATA
GPT-40 NIDH-LEVEL
ROSOT
you sandwich with cheese, lettuce?
slice
pick piece lettuce
pick slice of cheddar cheese
chediar
I'm proparing for a novie night. Can you get no some Oreo, TwIx, and chaps?
put Orea into the basket
pick up Twix
put Drea into the basket
Done! All trash has cleared. Let help with anything else!
trash, but not Can you clean dishes?
the
put the the bin
bowl
not that
respord: Sorry! open gripper
chopstick
chupstick
Figure 6: Qualitative Command Comparisons. GPT-4o often (a) misidentifies objects, (b) skips subtasks, or (c) ignores user intent. Hi Robot consistently produces commands aligned with the robot's ongoing actions and user requests. Without synthetic data, the high- level policy aligns well with image observations but ignores user constraints.
as a correct prediction; otherwise, it is labeled as incorrect. The Instruction Accuracy for a trial is then computed as the proportion of correct predictions out of the total number of predictions. For flat baselines, which lack interpretable language predictions, scoring is based on the evaluator's interpretation of the intent of the policy behavior.
often fails to maintain a coherent internal state, leading to commands like picking up new objects when the gripper is still occupied or prematurely switching tasks. The flat base- line, on the other hand, does not react to real-time feedback.
(3) Hi Robot is effective across diverse tasks, robots, and user constraints. On single-arm, dual-arm, and mo- bile bimanual platforms, Hi Robot is able to handle dis- tinct objects (from fragile cheese slices to tall bottles) while respecting dynamic constraints (e.g.. "bus only yellowish items," " "don't 't add tomatoes")). By contrast, the flat baseline and GPT-4o often revert to default behaviors (e.g., picking up every object in sight, or including almost all ingredients in a sandwich) when the prompt changes mid-episode
Task Progress (TP). Since all tasks we evaluate are com- plex and long-horizon, we record task progress to provide a granular view of task completion. Task progress quanti- fies how closely the robot matches the intended goal and is computed by the proportion of objects that are successfully placed in their correct locations or configurations.
5.3. Core Results
(4) Expert human guidance reveals the low-level pol- icy's strengths but underscores the need for high-level reasoning. With human high-level instructions, the low- level policy executes nearly flawlessly, showing that fail- ures stem more from reasoning than actuation. However, solely relying on human input is not scalable. Hi Robot bridges this gap via high-level VLM that aligns with user prompts and real-time observations, whereas GPT- 4o's lack of physical grounding and the flat baseline's lack of high-level reasoning hinder performance.
We present results for our system and two key baselines: GPT-4o policy and a flat VLA method. Quantitative and qualitative results are in Figure 5 and Figure 6, and we sum- marize our findings below.
(1) Hi Robot excels at open-ended instruction following. Across all tasks, Hi Robot exhibits substantially higher In- struction Accuracy and Task Progress, compared to GPT- 4o and the flat baseline. properly identifies, picks up, and places the correct items even when prompted to handle only certain objects or omit ingredients (e.g.. "I'm aller- gic to pickles").. In contrast, GPT-4o frequently loses con- text once physical interaction begins, issuing nonsensical commands (c.g., "pick up bermuda triangle") or sometimes labeling everything as "plate" or "spoon," " which disrupts long-horizon planning.
5.4. Ablation Studies
We conduct two key ablations to isolate the contributions of (1) synthetic data for high-level reasoning, and (2) hier- archical decomposition vs. a single "flat" policy.
(A) Synthetic data is critical for open-ended instruction following. Comparing Hi Robot (trained on human-labeled + synthetic data) to a variant trained solely on human- labeled data shows that synthetic interactions significantly boost language flexibility (Figure 7). Without them, the ab-
(2) Hi Robot shows strong situated reasoning and adap- fation to feedback. When users modify requests mid-task (e.g., "leave the rest," "I also want a KitKat"), ), Hi Robot up- dates low-level commands accordingly. GPT-4o, however,
8

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Visio--Language-Actinn Models
6. Discussion and Future Work
Table Bussing
Sandrrich Making
Grocery Shapping
Average
We presented Hi Robot, a system that uses vision-language models (VLMs) in a hierarchical structure, first reasoning over complex prompts, user feedback, and language inter- action to deduce the most appropriate next step to fulfill the task, and then performing that step by directly outputting low-level action commands. Our system can be thought of as a VLM-based instantiation of the "System I" and "Sys- tem 2" architecture (Kahneman, 2011) 1). The deliberative "System 2" layer takes the form of a high-level VIM pol- iey, which leverages semantic and visual knowledge from web-scale pre-training to reason through complex prompts and user interactions. The physical, reactive "System I" layer also takes the form of VLM, trained to directly out- put robot actions in response to simpler commands that de- scribe atomic behaviors.
100
4.5% SAP
39% GRRP
80
68
48
28
8
IA
Robot w/o synthatic data
Robot
Figure 7: Ablation on synthetic data. Synthetic data essential for handling open-ended instructions, as the model trained with- out it struggle with user-driven deviations, failing to integrate clar- ifications and constraints, whereas Hi Robot adapts seamlessly by leveraging diverse, compositional language prompts. (IA In- struction Accuracy, TP Task Progress)
The two VLMs have nearly identical architectures, with the only difference being that the low-level policy uses flow matching to output the actions. Indeed, the separation of roles at the model level is not fundamental to this design: natural step for future work is to combine both systems into one model, and draw the "System 1 vs "System 2" dis- tinction purely at inference time. Future work could also interleave high-level and low-level processing more intri- cately while our system simply runs high-level inference at fixed but lower frequency, adaptive system might si- multaneously process inputs and language asynchronously at multiple different levels of abstraction, providing for more flexible multi-level reasoning procedure.
Table Bussing
Sandtrich Making
Grocery Shopping
108
88
68
48
20
0
Our system also has a number of limitations that could be studied in future work. While we show that our high-level policy can often break down complex commands into low- level steps that the robot can perform physically, the train- ing process for this high level model relies in some amount of prompt engincering to produce synthetic training exam- ples that induce this behavior. The training process decou- ples the high-level and low-level models, and they are not aware of one another's capabilities except through the train- ing examples. Coupling these two layers more directly, e.g. by allowing the high-level policy to be more aware of how successfully the low-level policy completes each com- mand, would be an exciting direction for future work. More generally, by instantiating both high-level and low-level reasoning via VLMs, we believe that this design opens the door for much more intricate integration of these compo- nents, such that future work might create robotic vision- language-action models that dynamically reason about in- puts, feedback, and even their capabilities to produce suitable situated response in complex open- open-world settings.
Flat VLA w/synthetic data
Hi Robot
Figure 8: Hierarchical policy vs. flat policy, The hierarchical approach outperforms the flat variant trained on the same data, as it effectively integrates user feedback and partial instructions, whereas the flat model struggles with mid-tast task clarifications and nuanced task variations. (IA = Instruction Accuracy, TP = Task Progress)
lated model ignores clarifications (e.g., "this is not trash") or includes forbidden items (e.g., pickles), while Hi Robot smoothly adapts to such feedback, due to the broader cov- erage of compositional language in synthetic data.
(B) Hierarchical structure outperforms a flat policy. We next compare Hi Robot to a flat policy trained on the same synthetic data but without separate reasoning step (Fig- ure 8). The flat model often reverts to clearing all items or fails to handle partial instructions ("bus only the yellowish things"), whereas Hi Robot re-checks the prompt at each high-level step and responds coherently to mid-task up- dates. This suggests separating high-level reasoning from low-level control is benficial for multi-step coherence and adapting to dynamic user inputs.
9

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Action Models
Acknowledgments
Do i can, say: Grounding language in robotic affordances. In Conference on robot learning, pp- 287- 318. PMLR, 2023b.
We thank Ury Zhilinsky and Kevin Black for their help in setting up the data and training infrastructure. We thank Karol Hausman for valuable feedback and discussions on video demonstration and Aanguage-following evaluation. We are also grateful to Noah Brown, Szymon Jakubczak, Adnan Esmail, Tim Jones, Mohith Mothukuri, and Devin LeBlanc for their support in robot maintenance. We appre- ciate Suraj Nair and Laura Smith for their insightful dis- cussions that helped with policy debugging. We also thank Claudio Guglieri for help in creating visualizations used in this paper and on the project website. Finally, extend deepest gratitude to the entire team of robot operators at Physical Intelligence for their immense contributions to data collection, annotation, and policy evaluations.
Chen, H., Yao, Y., Liu, R., Liu, C., and Ichnowski, J. Automating robot failure recovery using vision- language models with optimized prompts, arXiv preprint Sedi:2200.03366. 2024.
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burch- fiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics:
Science and Systems (RSS), 2023.
Dai, Y., Lee, J., Fazeli, N., and Chai, Racer: Rich language-guided failure recovery policies for imitation learning. arXiv preprint arXiv:2409.14674 2024.
Impact Statement
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint MrxXix:3333.337755 2023.
This paper presents work where the goal is advance the fields of machine learning and robotics. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Fu, Z., Zhao, T. Z., and Finn, C. Mobile aloha: Learning bi- manual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117. 2024.
References
Belkhale, S., Ding, T., Xiao, T., Sermanet, P., Vuong, Q., Tompson, J., Chebotar, Y., Dwibedi, D., and Sadigh, D. Ri-h: Action bierarchies using language. arXiv preprint arXiv:2403.0182. 2024.
Hu, Y., Lin, F, Zhang, T. Yi, L., and Gao, Y. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning, 2023. URL https: / //arxiv. org/ abs//33at. 17842.
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X. Salz, D., Neumann, M., Alabdulmohsin, I., Tschan- nen, M., Bugliarello, E., et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models zero-shot planners: Extracting action- able knowledge for embodied agents. In International conference machine learning, pp. 9119 9118-9147. P00-9117..PMA... PMLR, 2022.
Black, K., Brown, N., Driess, D., Esmail, A., Equi M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., al. To: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.2411 2024.
Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., and Fei- Fei, L. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arX7::2307.05977 2023.
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A.,
Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.068/ 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action fer web knowledge to robotic control. arXiv preprint vrXiv:2307.158/6 2023a.
Kahneman, D. Thinking, fast and slow. Farrar, Straus and Giroux, New York, 2011. ISBN 9780374275631 0374275637.
models trans-
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open- ooen-source vision- Laiiunn- action model. arXiv preprint Xiv::22400.000 092466 2024.
Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et
10

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Visoon-Language-Action Models
Li, B., Wu, P., Abbeel, P., and Malik, J. Interactive task planning with language models, 2025a. https: //arxiv. orgg
Namasivayam, K., Singh, H., Bindal, V., Tuli, A., Agrawal, V., Jain, R., Singla, P., and Paul, R. Learning neuro- symbolic programs for language guided robot manip- ulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7973-7980. IEEF, 2023.
Li, Q. Liang, Y., Wang, Z., Luo, L.., Chen, X., Liao, M., Wei, F., Deng, Y., Xu, S.. Zhang, Y., et al. Cogact: A foundational vision-aagguage-cttionn model for synergiz- ing cognition and action in robotic manipulation. arXiv preprint ar arXiv:24/ 1. 19650, 2024.
Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang. J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.0785 2024.
Li, Y. Deng, Y., Zhang, J., Jang, J., Memmel, M., Yu, R., Garrett, C. R.. Ramos, F., Fox, D., Li, A., Gupta, A., and Goyal, A. Hamster: Hierarchical action models for open-world robot manipulation, 2025b. URL https: //arxiv. org/ abs / 2202. 05485.
Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., Kreiman, T., Tan, Y., Chen, L. Y., Sanketi, P., Vuong, Q. Xiao, T., Sadigh, D., Finn, C., and Levine, S. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Lan- guage model programs for embodied control. In 2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 9493-9500. IEEE, 2023.
O'Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. Open x-embodiment: Robotic learn- ing datasets and rt-x models: Open X-embodiment col- laboration O. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892-6903, IEEE, 2024.
Liu, F., Fang, K., Abbeel, P., and Levine, S. Moka: Open- vocabulary robotic manipulation through mark- vi- sual prompting. In First Workshop Esion-Languuse Models Navigation and Manipulation at ICRA 2024, 2024a.
Patki, S., Daniele, A. F., Walter, M. R., and Howard, T. M. Inferring compact representations for efficient natural language understanding of robot instructions. 2019 International Conference on Robotics and Automation (ICRA), pp. 6926-6933. IEEE, 2019.
Liu, H., Chen, A., Zhu, Y., Swaminathan, A., Kolobov, A.,
and Cheng, C.-A. Interactive robot learning from verbal correction. arXiv preprint arXiv:2310.17555 2023.
Liu, P., Orru, Y., Vakil, J., Paxton, C.,, Shafiullah, N. M. M., and Pinto, L. Ok-robot: What really matters in integrat- ing spen-knowledge models for robotics. arXiv preprint arXiv:2401. 12202, 2024b.
Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q. Mees, O., Finn, C., and Levine, S. Fast: Efficient action tokenization for lasion-laggaag-acttin models. arXiv preprint arXiv:2501.09744 2025.
Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., and Zhu, Rdt- lb: a diffusion founda- tion model for bimanual manipulation. arXiv preprint arXiv:2410.0786/ 2024c.
Qiu, D., Ma. W., Pan, Z., Xiong, H., and Liang, J. Open- vocabulary mobile manipulation in unseen dynamic vironments with 3d semantic maps. arXiv preprint urXiv:2406.18/1 15, 2024.
Loshchilov, I. and Hutter, Decoupled weight decay reg- ularization, 2017.
Radford, A., Kim. J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International confer- ence on machine learning, pp. 18492-28511 PMLR, 2023.
Matuszek, C., Herbst, E., Zettlemoyer, L., and Fox, D. Learning to parse natural language commands to a robot control system. In Experimental Robotics: The 13th In- ternational Symposium on Experimental Robotics, vol- ume 88, pp. 403. Springer, 2013.
Shah, R., Yu. A., Zhu, Y., Zhu, Y., and Martin-Martin R. Bumble: Unifying reasoning and acting with vision- language models for building-wie wide mobile manipula- tion. arXiv preprint arXiv: 2210.06237 2024.
McCallum, S., Taylor-Davies M., Albrecht, S., and Suglia, A. Is feedback all you need? leveraging natural language feedback in coal-conditioned rl. In NeurIPS 2023 Work- shop on
Shi, L. X., Hu, Z. Zhao, T. Z., Sharma, A., Pertsch, K., Luo, J., Levine, S., and Finn, C. Yell at your robot:
Reinforcement Learning.

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Action Models
Improving on-the-fly from language corrections. arXiv preprint arXiv:2403.122 129100 2024.
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learning fine-graines bimanual manipulation with low-cost hard- ware. arXiv preprint 2304 2023.
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Prog- prompt:
Zheng, J., Li, J., Liu, D., Zheng, Y., Wang. Z., Ou, Z., Liu, Y., Liu, J., Zhang, Y-Q.. and Zhan, X. Universal tions for enhanced embodied foundation models. arXiv preprint arXiv:2501.10005 2025.
Generating situated robot task plans using large language models. In 2023 IEEE International Confer- ence Robotics and Automation (ICRA), pp. 11523- 11530. IEEE, 2023.
Zhi, P. Zhang, 7., Han, M., Zhang, 7., Li, Z., Jiao, Z Jia, B., and Huang, Closed-loop open-vocabulary mobile manipulation with gpt-4v. arXiv preprint arXiv:2404.10220 2024.
Singh, U., Bhattacharyya. P., and Namboodiri, V. P. Lgr2: Language guided reward relabeling for accelerat- ing hierarchical reinforcement learning. arXiv preprint arXiv:2406.0588/, 2024.
Stephan, M., Khazatsky, A., Mitchell, E., Chen, A. S., Hsu, S., Sharma, A., and Finn, C. Rlvf: Learning from ver- bal feedback without overgeneralization. arXiv preprint arXiv:2402.10899 2024.
Stepputtis, S., Campbell, J., Phielipp. M., Lee, S., Baral, C., and Ben Amor, H. sanguage-condition imitation learning for robot manipulation tasks. Advances in Neu- Information Processing Systems, 33:13139-13150, 2020.
Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.- H., Vuong, Q. Wohlhart, P., Kirmani, S., Zitkovich, B., Xia, F., et Open-worth world object manipulation us- ing pre-trained vision-language models. arXiv preprint arXiv:2303.00005. 2023.
Swadzba, A., Vorwerg, C., Wachsmuth, S., and Rickheit, G. A computational model for the alignment of hierar- chical scene representations in human-robot interaction. In Twenty-First International Joint Conference on Artifi- cial Intelligence. Citeseer, 2009.
Wang, S., Han, M., Jiao, Z., Zhang, Z., Wu, Y. N., Zhu, S.-C., and Liu, H. Llm" 3: Large language model-based task and motion planning with motion failure reasoning. arXiv preprint arXiv:2403.1 1552, 2024.
Wen, J., Zhu, Y., Li, J., Zhu, M., Wu, K., Xu, Z., Liu, N., Cheng. R., Shen, C., Peng. Y., et al. Tinyvla: Towards fast, data-efficient vision-language-actis robotic manipulation. arXiv preprint arXiv:2409.12514 2024.
models for
Xiao, A., Janaka, N., Hu, T., Gupta, A., Li, K., Yu, C., and Hsu, D. Robi butler: Remote multimodal inter- actions with household robot assistant. arXiv preprint arXiv:2409.20548, 2024.
Zawalski, M., Chen, W., Pertsch, K., Mees, O., Finn, C., and Levine, S. Robotic control via embodied chain- of thouhtt reasoning. arXiv preprint arXiv:2407.08699; 2024.
12

---

Hi Robot: Open-Ended Instruction Following with Hicrarchical Vision-Language-Actionn Models
A. Synthetic Data Generation
between task domains and more robust, open-ended robot behavior.
A.I. Scenario and Response Categorization
To ensure the quality and diversity of the synthetic data, we incorporate structured scenario classification and re- sponse categorization into the prompt design for p&m, fol- lowing (Stephan al., 2024). Specifically, we classify interactions into different scenario types, such as nega- tive task (where the user instructs the robot what not to do), situated correction (where the user adjusts an earlier command based on the evolving task state), and specific constraint (where the user specifies particular constraints, such as dietary preferences). In addition, we categorize the robot's responses into types such simple confirma- tions, clarifications, and error handling. These classifica- tions guide the generation process to ensure a broad range of user-robot interactions.
B. System and Robot Overview
Our system integrates speech-based interactions and real- time robotic control. Below, we detail the components of our system, including audio processing, GPU-based infer- ence, and the robot configurations.
B.1. Perception and Language Processing
For speech-basee based interaction, we use consumer gradd lavalier microphone for audio input. Speech-to-text tran- scription is handled locally using Whisper large-v2 (Rad- ford et al., 2023). For text-to-speech synthesis, we employ the Cartetia API to generate natural and expressive speech outputs.
A.2. Prompt Construction for Contextual Grounding
B.2. Inference Hardware
In prompt P, we include detailed description of the task (e.g., bussing& a table, making a sandwich, grocery shop- ping) and instruct the model to ground responses in visual observations and prior context. key advantage lever- aging large pretrained VLMs is their ability to incorporate world knowledge when generating interactions. For in- stance, the model can infer dietary constraints when gener- ating prompts for sandwich-making producing user com- mands such as "Can you make a sandwich for me? I'm lactose intolerant" and an appropriate robot response like "Sure, won't put cheese on it." Similarly, it can reason over ambiguous or implicit requests, such as inferring that "I want something sweet" in a grocery shopping scenario should lead to suggestions like chocolate candy.
To support real-time inference, we utilize one to two NVIDIA GeForce RTX 4090 consumer-grade GPUs.
B.3. Real-Time Inference Latency
We measured latency across components on consumer- grade RTX 4090.
Low-Level Policy Per-Step Inference Times
Component
Time (ms)
Image encoding
14
Observation processing
32
Action prediction (x10)
27
Total (on-board)
73
To maintain consistency in multi-step tasks, we condition pee on prior skill labels within an episode lo....,,- 4--1, allowing to generate coherent user commands that the robot has already placed lettuce and tomato on sandwich, the generated user prompt might request additional in- gredients that logically follow. This ensures that the synthetic interactions reflect realistic task progression rather than isolated commands. As such, we leverage (f,, ue [I I, &,,,,,,,,,,, P) to produce richer, more diverse synthetic dataset D Daym that provides mean- ingful supervision for training our high-level policy.
Total (off-board WiFi)
86
High-Level Policy (Single Decoding Step)
account for past actions. For instance,
RTX 4090: 47 ms (prefill) + 13.2 ms (decode)
H100: 17.3 ms (prefill) + 5.7 ms (decode)
These measurements confirm real-time feasibility ~ 10 Hz control rates. With action chunking (Zhao et al., 2023), we can use it to control robots at Hz.
B.4. Robot System Details
We employ three different robot configurations with vari- ous manipulation and mobility capabilities.
While in this work we generate separate Daym and train separate high-level policy for each task (e.g., sandwich making vs. table cleaning) for clarity and ease of bench- marking, the architecture is readily amenable to unified multi-task formulation. In principle, the same hierarchical approach could be used to train a single high-level policy across a multitude of tasks, facilitating knowledge transfer
This setup features 6-DoF robotic arm equipped with a parallel jaw gripper. It includes two cameras: a wrist-mounted camera and an over-the-shoulder camera. The system operates within a 7-dimensional configuration and action space.
URSe.
13

---

Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Actton Models
Bimanual ARX. This configuration consists of two 6- DoF ARX arms. The system is equipped with three cam- eras: two wrist-mounted cameras and one base camera. The combined system has a 14-dimensional configuration and action space, enabling dextrous bimanual manipulation tasks.
Temporarily ignoring instructions: e.g., grabbing cheese when the robot is close despite user's
lactose intolerance (due training bias toward proximal objects)
Error accumulation and out-of-distribution (OOD) recovery: c.g., dropped objects
Beyond the future directions discussed in the main text, several additional mitigations may help address observed limitations, including but not limited to:
Mobile ARX. Built on the Mobile ALOHA (Fu et al., 2024) platform, this system integrates two 6-DoF ARX robotic arms mounted nomic base introduces two additional action dimensions, resulting in 14-dimensional configuration space and a 16- dimensional action space. Similar to the bimanual setup, includes two wrist-mounted cameras and a base camera, providing robust visual feedback for navigation and manip- ulation.
mobile base. The nonholo-
Stronger mnstruction-foloowii model
Long-context model
Adversarial data generation for edge cases
Diverse data collection including failure recovery and annotation
C.S. System Prompt for GPT-40
C. Model and Experiment Details
In the system prompt for GPT-4o, we include a description of the task, robot setup, and common instructions to select from. Below is example for the Table Cleaning task.
C.1. Model Initialization
While our method can be trained from scratch or fine-tuned from any vision-language model (VLM) backbone, prac- tice we use PaliGemma (Beyer et al., 2024) as the base model. PaliGemma an open-source, paaramee VLM that offers a good balance between performance and computational efficiency. We unfreeze the full model for fine-tuning.
Listing 1: GPT-4o Baseline Prompt
You are an AI assistant guiding a single- arm robot to bus tables.
The robot can optionally place trash in the trash bin and utensils and dishes the plastic box.
Every 2 seconds, you can issue one Instruction from provided list. You will receive images from two cameras: one for a global view and one on the robot' wrist for detailed views.
C.2. Optimizer and Hy Hyperparameters
We the Adam W optimizer (Loshchilov & Hutter, 2017) with 31 0.9, 32 0.95, and no weight decay. Gradient norms are clipped to a maximum magnitude of 1. We main- tain an exponential moving average (EMA) of the network weights with a decay factor of 0.999. The learning rate is warmed up over the first 1,000 steps and then held constant 10- 5, We use batch size of 512.
Interpret the user's instruction into one from the provided list for the robot to execute. Adhere strictly to the user's instruction. If ambiguous, reason out the best action for the robot. Only provide the exact instruction from the list without explanation.
C.3. Training Duration and Resources
Training the high-level policy is highly efficient, requiring approximately 2 hours on 8 8xH100 GPUs. The low-level policy follows similar training pipeline, though training time may vary depending on the dataset size and the com- plexity of the target tasks for action prediction.
You will select your instruction from the following list: put food container in trash bin pick up chopstick drop wrapper in trash
pick up plastic plate pick up the cup pick up white bowl place bowl to box pick up spoon place trash drop box in trash place take out box to trash nove to the left pick up container drop plate in bin pick up the trash
C.4. Failure Cases
We observed the following failure modes:
trash bin
1. High-level:
Difficulty with instructions requiring long- context reasoning, since the current system lacks memory
2. Low-level:
14

---

Hi Robot: Open-Ended Instruction Following with Hierarchical Visonn-Language-Actinn Models
pick up plastic bowl go higher
place spoon to box pick up the paper container
drop fork in bin
pick up the bowl
pick up the plastic container go lower
pick up box move to the right
drop plastic 1id into recycling bin pick up wrapper pick up the plate put bowl in box pick up the container
put the plate in the bin pick up cup put cup into box throw in the trash pick up food container pick up blue cup
drop the bowl into the bin towards pick up napkin rotate counterclockvise put the cup in the bin throw trash away rotate clockwise
drop plastic bowl into box open gripper pick plastic cup pick up the plate close gripper move away from me go back to home position
15


Reviews:
[{"review_id": "IhiSOZSuXo", "reviewer": "Reviewer_sNP7", "summary": "In this work, the authors introduce Hi-Robot, a  System-1/System-2 approach that leverages a Vision-Language Model (VLM) to interpret complex prompts and generate a more suitable sequence of instructions for a Vision-Language-Action Model (VLA) to complete a given task. The system also integrates feedback during execution. The authors evaluate Hi-Robot across a diverse set of robotic platforms, in tasks that require novel combinations of learned skills in real-world scenarios.  The results show that Hi-Robot outperforms several prior approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and constructive suggestions. We address each point below and will incorporate these improvements in the revision.\n\n> Real-time inference timing analysis\n\nWe provide detailed latency measurements across components (tested on consumer-grade RTX 4090):\n\n*Low-Level Policy Per-Step Inference Times*\n| Component                  | Time (ms) |\n|----------------------------|----------|\n| Image encoding             | 14       |\n| Observation processing      | 32       |\n| Action prediction (x10)     | 27       |\n| Total (on-board)        | 73   |\n| Total (off-board + WiFi)| 86   |\n\nFor the high-level policy (single decoding step):\n- RTX 4090: 47ms (prefill) + 13.2ms (decode)\n- H100: 17.3ms (prefill) + 5.7ms (decode)  \n\nThese measurements confirm real-time feasibility at ~10Hz control rates. With action chunking [1], we can use it to control robots at 50Hz.\n\n> Impact of different model choices\n\nOur ablation (Fig 8) shows hierarchy improves VLA model performance even with identical data. Future work directions include:\n- **Architecture Studies**: E.g. video-based VLMs for temporal reasoning\n- **Scaling Laws**: How VLM/VLA size affects performance\n- **Transfer**: Which models better inherit internet pre-training knowledge\n\nWe will expand this discussion in §6 (Future Work).\n\n> Move synthetic data analysis to main text\n\nWe will:\n1. Relocate the synthetic data section from Appendix A to §4.5\n2. Add example prompts for data generation\n3. Include examples of bad samples and how to avoid them\n\n> Some videos not working\n\nWe've:\n1. Converted all videos to SDR format\n2. Added streaming-optimized versions\n3. Included **new demos** showing diverse instruction following\n\n> Plan to release model weights?\n\nWe will discuss this with collaborators before finalizing the camera-ready version of the paper. \n\nThank you again for your valuable feedback!\n\n[1] Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.", "claims_and_evidence": "The system demonstrates advanced reasoning capabilities, allowing it to process complex prompts, dynamically incorporate feedback, and execute instructions beyond its training data. It enables real-time corrections during open-ended tasks, enhancing adaptability. Its novel capabilities stem from the combination of a high-level LLM planner, a low-level VLA policy, and synthetic data generation. Furthermore, the framework is inherently modular, allowing for the integration of alternative language-conditioned policies as needed. Experimental evidence supports these claims. As shown in Sections 5.3.1 and 5.3.3, the system outperforms larger models like GPT-4o in instruction accuracy and task progress, particularly in handling complex prompts and adapting to mid-episode prompt changes across different platforms. Section 5.3.2 highlights the system’s ability to modify actions based on feedback, though the term \"real-time\" might be misleading, as it requires inference from two 3B models; a time analysis would be beneficial. Additionally, Section 5.4.1 provides quantitative evidence that synthetic data improves system performance. While the system’s modularity is acknowledged, further analysis is needed to evaluate how different model choices, when fine-tuned on the same data, impact overall performance.", "methods_and_evaluation_criteria": "Yes, but since the synthetic data are part of the contribution I would like to see a more detailed analysis on the creation of the said dataset on the main text and not in the appendix.", "theoretical_claims": "There are no theoretical claims in the paper, hence this section is not applicable.", "experimental_designs_or_analyses": "All the experiments are well designed and the analysis is sound.", "relation_to_broader_scientific_literature": "This work is highly related to the broader scientific literature and specifically $\\pi_0$ and applying the general concept of \"system 1\" \"system 2\" cognitive processes something really popular in the LLM-VLM and robotics.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "The paper is well-written and very easy to follow.", "other_comments_or_suggestions": "I applaud the authors for developing a system that can run on consumer-grade GPUs, making research in VLAs for robotics more accessible to a wider audience.", "questions_for_authors": "Do the authors plan to open-source the model weights?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qswHh1tbzd", "reviewer": "Reviewer_9d4b", "summary": "This paper presents Hi Robot, a hierarchical vision-language-action (VLA) model for open-ended instruction following. The system integrates a high-level vision-language model (VLM) that interprets complex prompts and user feedback with a low-level VLA policy that executes atomic actions. A synthetic data generation pipeline augments training by creating user interactions that improve generalization to diverse tasks. Hi Robot is evaluated on three real-world robotic applications, demonstrating superior performance over GPT-4o-based high-level policies and flat VLAs. The results highlight significantly improved instruction accuracy, task progress, and real-time adaptability to human corrections.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We address each concern below and will revise the paper accordingly.\n\n> The hierarchical structure resembles prior work on dual-process systems.\n\nWhile building on foundational ideas, Hi Robot introduces key innovations:\n1. **Interactive Open-Ended Instruction Following**: Unlike DriveVLM [1] (trajectory planning for low-level execution) or DP-VLA [2] (BC-Transformer policy for low-level execution), our VLM+VLA framework enables:\n   - Real-time language feedback incorporation\n   - Generalization to open-vocabulary tasks via synthetic data\n   - Physical dexterity (e.g., separating sticky cheese slices while making sandwiches)\n2. **Scalable Synthetic Data**: Manual annotation in [1] limits scalability; our pipeline automates diverse interaction generation.\n\n*Table: Capability Comparison*\n| Feature          | DriveVLM | DP-VLA | Hi Robot |\n|------------------|----------|--------|----------|\n| Open-ended instructions | ✗ | ✗ | ✓ |\n| Real-time feedback | ✗ | ✗ | ✓ |\n| Synthetic data scaling | ✗ | ✗ | ✓ |\n| VLA low-level policy | ✗ | ✗ | ✓ |\n\n> How does Hi Robot improve over LLMPlanner+OpenVLA?\n\nOur experiments revealed critical limitations of ungrounded planners:\n- **Physical Grounding**: GPT-4o (stronger than LLMPlanner's GPT-3) fails to recover from real-world errors (e.g., misgrasps) due to lack of embodied understanding (Fig 6).\n- **Scalability**: LLMPlanner uses only 8 predefined actions; Hi Robot supports thousands and more via language-conditioned skills.\n- **Performance**: Hi Robot outperforms GPT-4o by 40% in instruction accuracy (Fig 5).\n\n*Key Advantage*: Hi Robot’s high-level VLM is aware of low-level affordances, enabling physically-realizable plans and feedback integration.\n\n> How does Hi Robot handle mobile challenges?\n\nThe framework treats mobility as an augmentation of manipulation:\n- **Unified Control**: Base velocity commands are additional action dimensions, enabling whole-body coordination (e.g., reaching high shelves by moving forward while raising arms).\n- **Results**: \n  - 85% task success in grocery shopping (vs. 71.7% for GPT-4o).\n  - Failure recovery: Autonomous adaptation from teleop data (e.g., freeing stuck baskets by adjusting base/arm coordination).\n\n*Failure Analysis*: Primary issues involve unseen edge cases (e.g., dropped objects). Future work can expand teleop and synthetic data coverage to make the system more robust.\n\n> Why not compare with GPT-4o-1 (o1)?\n\nWhile o1 excels in mathematical reasoning:\n- **Speed**: >10s/inference makes it impractical for real-time control.\n- **Relevance**: Coding/math strengths don’t directly translate to embodied reasoning.\n\nGPT-4o represents the fairest *practical* baseline for real-world deployment.\n\n> Is this a unified or separate models?\n\nHi Robot uses **separate but co-trained models**:\n1. **High-level VLM**: Specialized for instruction decomposition and feedback integration.\n2. **Low-level VLA**: Focused on action generation.  \nThe interface is learned through synthetic examples that map high-level commands to executable skills and annotated teleop data that map skill commands to low-level actions, preserving reasoning capability while enabling precise control.\n\nThank you again for your insightful questions—they’ve helped us better articulate Hi Robot’s contributions. We’ll incorporate these clarifications in the revision.", "claims_and_evidence": "I summary my review as below. See Other Strengths And Weaknesses.", "methods_and_evaluation_criteria": "See Other Strengths And Weaknesses.", "theoretical_claims": "See Other Strengths And Weaknesses.", "experimental_designs_or_analyses": "See Other Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "See Other Strengths And Weaknesses.", "essential_references_not_discussed": "See Other Strengths And Weaknesses.", "other_strengths_and_weaknesses": "Strength：\n\n1.\tThe VLM+VLA architecture separates high-level task decomposition from low-level execution. The high-level VLM dynamically adjusts commands using real-time visual context, while the low-level VLA handles physical nuances, outperforming flat VLAs by 40% in instruction accuracy.\n\n2.\tSynthetic data generation expands the model's ability to generalize beyond training data, enhancing real-world usability.\n\nWeakness:\n\n1.\tLack of Novelty\n\nThe proposed hierarchical robot closely resembles prior work on dual-process reasoning and control, as explored in [1]. While a high-level vision-language model (VLM) for reasoning and a low-level vision-language-action (VLA) model for execution is effective, similar paradigms have been previously established [2]. Beyond the overlap in datasets, it is important to clarify the contribution of Hi Robot’s hierarchical structure.\n\n2.\tUnclear Differentiation from Planner + VLA\n\nHi Robot leverages a VLM for high-level policy generation and π0 as the low-level control policy, which shares similarities with LLMPlanner [3] + OpenVLA [4]. A more explicit comparison is needed to highlight the architectural and results between Hi Robot and LLMPlanner+Openvla. Specifically, how does Hi Robot’s hierarchical decomposition improve over LLMPlanner’s approach to task abstraction and skill execution. A deeper discussion on these aspects would strengthen the paper’s contribution.\n\n3.\tFor mobile manipulation\n\nWhile Hi Robot is evaluated across single-arm, dual-arm, and mobile bimanual robots, the results do not clearly differentiate its effectiveness in mobile manipulation scenarios. Given the additional challenges posed by spatial reasoning, and bimanual coordination, how does the hierarchical policy structure adapt to these factors? Does the high-level VLM account for mobile constraints when generating commands, and how does it compare to prior mobile manipulation frameworks that integrate LLMs or VLMs for motion planning and task execution? More details on task success rates, failure cases, and adaptation strategies in dynamic mobile environments would provide a clearer assessment of Hi Robot’s scalability in real-world settings.\n\n4.\tUnfair of comparison\n\nThe paper uses GPT-4o as the primary LLM-based high-level policy baseline, but it is unclear why GPT-4o was chosen over GPT-4o-1 (o1), which has stronger reasoning capabilities. Since Hi Robot's high-level policy relies heavily on structured reasoning for hierarchical task decomposition, a fair comparison should include a model with comparable reasoning ability.\n\n5.\tConfusion of model architecture\n\nThe paper does not clearly specify whether the high-level and low-level policies are implemented as a single unified model or two separate models. If Hi Robot employs a single model for both high-level reasoning and low-level action generation, how does it maintain the ability to output action tokens while retaining reasoning? As discussed in the OpenVLA framework, VLA models are typically limited in their language reasoning abilities after being finetuned for action generation. If the high-level reasoning and low-level action generation are handled by separate models, then the system appears very similar to a Planner + VLA architecture. Clarification on the architecture would help assess the novelty and contribution of the proposed system.\n\n[1] Tian, Xiaoyu, et al. \"Drivevlm: The convergence of autonomous driving and large vision-language models.\"\n\n[2] Han, ByungOk, Jaehong Kim, and Jinhyeok Jang. \"A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLM.\"\n\n[3] Song, Chan Hee, et al. \"Llm-planner: Few-shot grounded planning for embodied agents with large language models.\".\n\n[4] Kim, Moo Jin, et al. \"Openvla: An open-source vision-language-action model.\".", "other_comments_or_suggestions": "Based on the author's rebuttal, I will consistently adjust my rating.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TAh561TgPW", "reviewer": "Reviewer_7g4f", "summary": "This paper, inspired by \"System 1\" and \"System 2\" cognitive processes, proposes a hierarchical VLM-based system to interpret high-level instructions and convert them into commands for a low-level VLA model. To train the model, the authors employ both human-labeled and synthetically generated interaction data. Some real-world experiments are conducted to demonstrate the model's ability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We address your comments in detail below and will update our paper accordingly.\n\n> For the comparison with GPT-4o high-level instruction decomposing experiments, what is the user prompt?\n\nThe user prompts for evaluation (e.g., \"Hi robot, can you make me a sandwich with cheese, roast beef, and lettuce?\") are the same across baselines and are included in the paper. If you are asking about the *system prompt* for GPT-4o, we provide it below (example for the Table Cleaning task) and will include it in the camera-ready version:\n\n>```\nYou are an AI assistant guiding a single-arm robot to bus tables.\nThe robot can optionally place trash in the trash bin and utensils and dishes in the plastic box.\nEvery 3 seconds, you can issue one instruction from a provided list.\nYou will receive images from two cameras: one for a global view and one on the robot's wrist for detailed views.\nInterpret the user's instruction into one from the provided list for the robot to execute. Adhere strictly to the user's instruction. If ambiguous, reason out the best action for the robot. Only provide the exact instruction from the list without explanation.\nYou will select your instruction from the following list:\nput food container in trash bin\npick up chopstick\ndrop wrapper in trash\npick up plastic plate\npick up the cup\npick up white bowl\nplace bowl to box\npick up spoon\nplace trash to trash bin\ndrop box in trash\nplace take out box to trash\nmove to the left\npick up container\ndrop plate in bin\npick up the trash\npick up plastic bowl\ngo higher\nplace spoon to box\npick up the paper container\ndrop fork in bin\npick up the bowl\npick up the plastic container\ngo lower\npick up box\nmove to the right\ndrop plastic lid into recycling bin\npick up wrapper\nput bowl in box\npick up the container\nput the plate in the bin\npick up cup\nput cup into box\nthrow it in the trash\npick up food container\npick up blue cup\ndrop the bowl into the bin\nmove towards me\npick up napkin\nrotate counterclockwise\nput the cup in the bin\nthrow trash away\nrotate clockwise\ndrop plastic bowl into box\nopen gripper\npick up plastic cup\npick up the plate\nclose gripper\nmove away from me\ngo back to home position\n<truncated due to character limit>\n\n> What if using in-context learning, chain-of-thought, o1 or deepseek r1 model for better reasoning?\n\n- **In-context learning**: We explored this with GPT-4o but found it did not generalize beyond in-context examples and significantly slowed inference due to long visual context. The VLM struggled to reason across many images for closed-loop tasks (e.g., sandwich making). Future work could improve learning from long-context inputs or summarize visual observations into text.  \n- **Chain-of-thought**: A promising direction for future work.  \n- **o1**: While strong in mathematical reasoning, its slow inference (>10s per step) makes it impractical for real-time robotic control.  \n- **Deepseek R1**: Does not support visual reasoning, limiting its applicability to our task.\n\n> some other papers should be discussed, such as [A-C].\n\nWe will cite these concurrent works in the camera-ready version. Key differences:  \n\n- HAMSTER [A]: Focuses on high-level VLMs generating 2D end-effector trajectories. Our work outputs language commands, enabling finer dextrous behaviors (e.g., separating sticky cheese slices during sandwich making) and on-the-fly verbal corrections. The approaches are complementary; trajectory prediction could be integrated as part of chain-of-thought reasoning in future work.  \n- RoboDual [B]: Separates generalist (latent representations) and specialist (actions), emphasizing training efficiency. We focus on open-ended instruction following.  \n- Code-as-Monitor [C]: Uses VLM-generated code for failure detection. Our work uses VLMs as high-level policies to guide low-level VLAs and interact with humans.  \n\n> The proposed method merely involves understanding high-level instructions, which are cascaded structures rather than a second system.\n\nOur framework explicitly separates high-level reasoning (VLM) from low-level execution (VLA). The VLM acts as a \"second system\" by decomposing abstract instructions into actionable commands, unlike monolithic VLAs that conflate reasoning and execution. We will clarify this distinction in the paper.\n\n> Different VLAs may prefer different command styles. How to make the method adaptable to other action models?\n\nHi Robot is architecture-agnostic and can integrate any language-conditioned policy. Future work could:  \n1. **Fine-tune the high-level policy** using successful rollouts (e.g., via SFT) to adapt to a low-level policy’s \"affordance\" (i.e., its language-following capabilities).  \n2. Use **policy performance as feedback** (e.g., via RLHF) to align the high-level policy’s outputs with the low-level policy’s strengths.  \n\nThank you again for your thoughtful suggestions—they have strengthened our paper. We will incorporate these changes in the revision.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "The experiments are only conducted on real-world robots. The demos are convincing, but the comparison with other methods has significant shortcomings in experimental settings. For example, for the comparison with GPT-4o high-level instruction decomposing experiments, what is the user prompt? What if using in-context learning, chain-of-thought, o1 or deepseek r1 model for better reasoning?", "relation_to_broader_scientific_literature": "This paper provides a dual system for robotics. This work is one of the early efforts in understanding, translating, and decomposing high-level human instructions, and it combines VLA models to design an integrated model from human instructions to actions.", "essential_references_not_discussed": "Yes. The concept of “System 1” and “System 2” is not the first proposed in this paper, some other paper should be discussed, such as [A-C].\n\n\n[A] Li et al. HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation. ICLR 2025.\n\n[B] Bu et al. Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation. arXiv:2410.08001.\n\n[C] Zhou et al. Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection. arXiv:2412.04455.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The hierarchical understanding and reasoning for high-level human instructions are necessary for robotic VLA models.\n\nWeaknesses:\n- The motivation and method are somewhat disconnected. The proposed method merely involves understanding high-level instructions, which are cascaded structures rather than a second system.\n- This work only experiments with the pi0 VLA model as the action model. Different VLAs may have specific preferences for different low-level language command styles. How to address this issue to make the proposed method adaptable to different action models?", "other_comments_or_suggestions": "NA.", "questions_for_authors": "Please refer to the motivation and experiment concerns mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HbOzXoiRik", "reviewer": "Reviewer_yW3d", "summary": "This paper proposes a hierarchical framework for tackling the complex instruction following challenge in vision-language-action-based robotic control. The paper highlights challenges in existing methods that struggle with following intricate instructions. The proposed method, Hi Robot, addresses these issues by decomposing tasks into a high-level VLM policy, which interprets complex prompts and user feedback to generate low-level commands, and a low-level vision-language-action (VLA) policy. Evaluations on various tasks demonstrate that Hi Robot outperforms baselines.\n\n## update after rebuttal\nI acknowledge the effort put into this work and appreciate that the authors have partially addressed my concerns. However, I still have reservations regarding the novelty of the work, how the two-layer framework is aligned, and its generalizability. Therefore, I am maintaining my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We address each point below and will incorporate these improvements in the revision.\n\n> Limited contribution: hierarchical VLMs with synthetic data generation.\n\nWhile individual components build on prior work, Hi Robot's novel synthesis enables critical real-world capabilities:\n- **Open-ended instruction following** (e.g., \"Can you make me a vegetarian sandwich? I don’t like pickles though.\")\n- **Real-time feedback integration** (e.g., \"That’s all I want\")\n- **Unseen task generalization** via synthetic data (demonstrated in §5.3 and video supplementary at https://hi-robot-vla.github.io/)\n\n> No validation of synthetic data quality\n\nWe evaluate data quality **end-to-end** through policy performance, as offline metrics for embodied data remain an open challenge. Future work could explore:\n- Automated fidelity checks for physical plausibility\n- Language-grounding consistency metrics\n- Interaction diversity\n\n> Statistical significance reporting\n\nWe conducted **20 trials per task per method** (more than typical real-world robotic experiments). Error bars will be added to all plots in the camera-ready version.\n\n> Lack of tests in unseen domains\n\nWe evaluated generalization through:\n1. **Instruction perturbations**: \n   - \"I want something sweet\" (requiring object categorization and physical grounding)\n   - \"I’m allergic to pickles\" (requiring semantic knowledge and physical grounding))\n2. **Partial task execution**: The model is trained only on full-table cleaning, but we request the robot to “clean up only the trash, but leave the dishes”\n\n*Future Work*: Cross-environment transfer (e.g., kitchen at home → kitchen in restaurant).\n\n> Discussion of failure modes\n\nCommon Failure Cases:\n1. *High-level*:  \n   - Temporarily ignore instruction: E.g., grabbing cheese when the robot is close to it despite user’s lactose intolerance (due to training bias toward proximal objects)\n2. *Low-level*:  \n   - OOD recovery: Dropped objects (recovery behavior is absent from training data)\n   \nMitigations (Future Work):\n- Stronger instruction-following model\n- Adversarial data generation for edge cases\n- Diverse data collection including failure recovery\n\n> Add more technical details\n\nWe will expand:\n- Appendix Table: Full hyperparameters (learning rates, architecture specs)\n- Data Generation: Prompt templates and filtering examples\n- Failure Logs: Representative error cases\n\n> Can Hi Robot resume interrupted tasks?\n\nCurrent Implementation:  \n- Can revert to previous objectives with explicit user permission  \n- Future: Auto-resume via success detection (e.g. via value function learning)\n\n> Difference from RT-H?\n\n| Feature                | RT-H                     | Hi Robot                  |\n|------------------------|--------------------------|---------------------------|\n| **High-Level Action Space**       | Primitive movements (e.g., \"Move arm forward\")      | Semantic commands (e.g., \"Place a slice of bread on the chopping board\") |\n| **Synthetic Data**     | ✗                        | ✓ (enables open-vocab feedback) |\n| **Instruction Scope**  | Seen tasks         | Open-ended                |\n\n*Key Advantage*: Hi Robot's rich language-action space supports real-world ambiguity and feedback (e.g., handling \"This isn't trash\" corrections).\n\nThank you for your suggestions—they have strengthened our paper. We will address all points in the revision.", "claims_and_evidence": "The main claims are well-supported within the evaluated domains.", "methods_and_evaluation_criteria": "The methods and evaluations are well-justified for the problem.", "theoretical_claims": "No proofs.", "experimental_designs_or_analyses": "The experiments robustly validate Hi Robot’s performance within the tested domains.", "relation_to_broader_scientific_literature": "Vision-Language-Action Models, Robot Control", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The related work is well investigated.\n2. The idea is very intuitive.\n3. Tested across diverse real-world robotic platforms.\n\nWeaknesses: \n1. Limited contribution: hierarchical VLMs with synthetic data generation.\n2. Lack of tests in unseen domains.\n3. No validation of the quality of the generated data.\n4. Only the average values of the metrics are counted, ignoring uncertainty and statistical significance.", "other_comments_or_suggestions": "1. Include a discussion of failure modes.\n2. Add more technical details to improve reproducibility.\n3. Including statistical tests would strengthen claims.\n4. include cross-domain tasks.", "questions_for_authors": "1. Have you tested Hi Robot on unseen domains?\n2. During task execution, if an instruction interrupts the task, can the system restore its previous state to resume the original objective?\n3. What is the main difference between Hi Robot and RT-H?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes to use hierarchical VLMs with synthetic data generation. The paper receives unanimously accept side opinion due to practicality of the proposed approach. Even though the accept side opinions, there are a number of weaknesses raised by the reviewers as follows; (1) lack of novelty, (2) unfair comparisons, (3) motivation and method are not well connected and (4) no validation of quality of generated data. Given all comments by the reviewers, the AC recommends to accept the submission to ICML 2025.
Decision: Accept (poster)