Robotic Control via Embodied Chain-of-Thought Reasoning
Authors: Michał Zawalski
William Chen
Karl Pertsch
Oier Mees
Chelsea Finn
Sergey Levine
Venue: CoRL 2024 (2024)
Keywords: Vision-Language-Action Models
Embodied Chain-of-Thought Reasoning
TL;DR: We train vision-language-action models to perform *embodied* chain-of-thought reasoning, substantially improving their success rate without any additional robot data.

Abstract: A key limitation of learned robot control policies is their inability to generalize outside their training data. 
Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models as the backbone of learned robot policies can substantially improve their robustness and generalization ability. Yet, one of the most exciting capabilities of large vision-language models in other domains is their ability to reason iteratively through complex problems. Can that same capability be brought into robotics to allow policies to improve performance by reasoning about a given task before acting? Naive use of "chain-of-thought" (CoT) style prompting is significantly less effective with standard VLAs because of the relatively simple training examples that are available to them. Additionally, purely semantic reasoning about sub-tasks, as is common in regular CoT, is insufficient for robot policies that need to ground their reasoning in sensory observations and the robot state. To this end, we introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features like object bounding boxes and end effector positions, before predicting the robot action. We design a scalable pipeline for generating synthetic training data for ECoT on large robot datasets. We demonstrate, that ECoT increases the absolute success rate of OpenVLA, the current strongest open-source VLA policy, by 28\% across challenging generalization tasks, without any additional robot training data. Additionally, ECoT makes it easier for humans to interpret a policy's failures and correct its behavior using natural language.

Robotic Control via Embodied Chain-of-Though Reasoning
Michal Zawalski William Chen"!, Karl Pertsch 1.3
Oier Mees', Chelsea Finn", Sergey Levine' UC Berkeley, University of Warsaw, SStanford University
Abstract: A key limitation of learned robot control policies is their inability to generalize outside their training data. Recent works on vision- saooo-llnguage-cttin mod- els (VLAs) have shown that the use of large, internet pre-trained -aigunla models as the backbone of learned robot policies can substantially improve their robustness and generalization ability. Yet, one of the most exciting capabilities of large vision-language models in other domains is their ability to reason iteratively through complex problems. Can that same capability be brought into robotics to allow policies to improve performance by reasoning about a given task before acting? Naive use of "chain-of-thoughh (CoT) style prompting is significantly less effective with standard VLAs because of the relatively simple training examples that are available to them. Additionally, the tasks common to regular CoT is insufficient for robot policies that need to ground their reasoning sensory observations and the robot state. To this end, intro- duce Embodied Chain-of-
semantic reasoning about sub-
Reasoning (ECoT) for VLAs, in which train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features like object bounding boxes and end effector positions, before predicting the robot action. We design a scalable pipeline for generating synthetic training data for ECoT on large robot datasets. We demonstrate that ECoT increases the absolute success rate of Open VI.A, the current strongest open-source VLA policy, by 28% across challenging generalization tasks without any additional robot training data. Additionally, ECoT makes it easier for humans interpret policy's ' failures and correct its behavior interactively using natural language. Finally, we show that our model learns to transfer ECoT reasonings to unseen embodiments and tasks.
Keywords:
Vision Vision-Langugge-Acciin Models, Embodied Mlain-of-Though Rea-
soning
Introduction
Robotic policies trained end-to-end for outputting raw low-level actions in response to observations provide a powerful and appealing learning- based approach to robotic control, obviating the need for complex sensing and control stacks [1, 2, 3J. However, this kind of "reactive" control is not necessarily the best choice in settings that demand broad generalization, such as novel scenes or interactions with unfamiliar objects. In such situations, person might need to think more carefully: if they were asked put fruit one plate and vegetables another, they might first try figure out which objects are fruits and which are vegetables, rather than simply performing a learned skill from "muscle memory." Likewise, we would like our robotic policies to both perform well- cell-practiced end-to-end control and to "reason through" novel situations before grounding their commands into actions. Such reasoning might include identifying and locating task-relevant objects, producing a plan to accomplish task, and translating sub-tasks and observations into movements.
Vision-language-actoo
models (VLAs) pre-trained vision-language models (VLMs) fine-tuned to produce robot actions have gained popularity as an approach for leveraging the diversity of Internet data captured within large foundation models [4] in simple and scalable policy learning
Equal contribution. Website at https://embodieedoo....itt github..o Correspondence: m. www.lskiuuw...mmmmmm.... sdu,p,, verityw@berkeley.cek edu, pertsch@berkeley.edd
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

recipe. Despite achieving state-of ofte-of-thhe-r art performance across wide range of tasks and robot embodiments [5, 6, 71. VLAs typically learn a direct mapping from observations to actions without any intermediate reasoning. However, there have been many recent works exploring how language models (which serve as the backbone of VLAs) can be prompted to textually "think step-by-step" about a given task. Such chain-of- of-thought reasoning (CoT) [8] significantly improves their performance on complex reasoning tasks and is now de-facto standard practice in language modeling [9].
We thus hypothesize that we can similarly boost
Regular VLA Policy
VLA w/ Embodied Chain-of-Thoughe
VLA performance by training them to rextually reason about their plan, environment, and mo- tions, thereby allowing them to produce more accurate and robust robot actions. However, simply applying the CoT techniques from lan- guage to the robotics domain has several chal- lenges. For one, current VLAs build on rel- atively small, open-source VLMs that cannot match closed models' ability to perform mean- ingful reasoning when simply prompted to think step-by-step [8]. Additionally, the most com- mon LLM CoT reasoning task, breaking tasks into sub-tasks, is helpful but insufficient for rea- soning about robotic tasks. The VLA policy needs to ground its plans and reasoning in its observations of the environment and robot state. Only then can the reasoning direct the agent's attention toward fine-grained spatial or semantic perceptual features that are key to robot manip- ulation tasks. Put simply, need VLAs to not only "think carefully", but also "look carefully." To this end, we introduce Embodied Chain-of- Thought Reasoning (ECoT) for VLA policies. In contrast to prior VLAs, embodied chain-of- thought policies perform multiple steps of tex- tual reasoning before predicting the next robot action (see Fig. 1, right). In contrast to existing CoT reasoning approaches for LLMs, they interleave semantic-level reasoning about sub-tasks with "embodied"" reasoning tasks that require the policy to pay attention to its multi-modal inputs, from predicting bounding boxes of objects in the scene to reasoning about low-level movement primitives that need be executed based on the current robot state. To enable the relatively weak LLM back- bones of open-source VLAs perform such reasoning effectively, synthetically generating embodied CoT training data for large robot datasets. Concretely, we use powerful pre-trained foundation models to generate the reasoning supervision for our policies.
User: Cleen the sink.
User: Clean the sink
OpenVLA:
[ax, a0, AGrial ....
CoT OpenVLA:
Ooject bounding boxes: Gripper position:
CoT OpenVLA:
Task Plan: ....
Next Task: pick up carrol
Next Move: move right
CoT OpenVLA: [ax, Ail, AGrip]
We propose embodied chain-of of-in-of-thought rea- soning for vision-language-atne VLAS directly predict the next robot action given the task (left), we instead train VLA policies "think step-by- step" (right). Crucially, reasoning through low-level em- bodied features like object bounding boxes and gripper positions in addition to purely textual CoT elements like sub-task task plans, forces the policy "think carefully" and "look carefully" before acting. This approach increases the absolute success rate of state-of the-art Open VLe VLA policies [7] by 28% in challenging generalization tasks.
Figure I:
models (VLAs)): prior
design a scalable pipeline for
Our experiments show that by training state-of-the-art VLAS to reason before predicting actions, we can substantially boost their ability to perform challenging generalization tasks. Our approach increases the absolute success rate of Open VLA [7], the current best-performing open-source VLA, by 28% across suite of robot manipulation tasks that involve generalization to objects, scenes, viewpoints, and instructions without any extra robot training data. Beyond raw performance improve- ments, we find that embodied CoT makes policy failures more interpretable and allows humans to easily correct its behavior by modifying faulty reasoning chains via natural language feedback.
2 Related Work
Scaling robot learning. long- oongsstaddng goal of robot learning is to train policies that can gen- eralize wide range of unstructured real-world environments. Towards this goal, recent works have explored training "generalist robot policies" [10, 11, 12, 13, 14, 15, 16, 17] on diverse robot datasets [18, 19, 10, 20, 21, 22, 23, 24, 13, 14, 25, 15, 26, 6]. As a result of their diverse robot

---

training datasets, many of these policies can be prompted in natural language to solve various manip- ulation tasks, and some generalist policies can even control multiple robot embodiments [16, 27, 6]. Importantly, these works demonstrate that training robot policies on large and diverse datasets is a promising approach towards improving policy robustness and generalization ability.
Vision-language models for robot generalization. In push towards generalization far beyond what is observed in robot datasets, the recent development of strong, open-source vision-language models that learn Aisuo-ingguistc representations [28, 29], generate images from text [30], or generate text in response to images and prompts [31, 32, 33, 34, 35] have resulted in a large number of works that explore the integration of such models into robot learning pipelines, c.g., to generate goals [36], to provide reward signals [37, 38, 39], or to learn visual state representations [40, 41, 42]. Since collection of the afforeentioned large-scale robot datasets is challenging, using models pre-trained on Internet-scale data is an appealing alternate path towards robust robot policies that can act in variety of unstructured real-world environments. Most relevant to our work are recent approaches for integrating pre-trained vision-language models into learned robot policies. While some works use strong structural priors in their policies to enable this integration [43, 44, 45], vision-language- action models (VLAs) have recently been proposed as a simple yet scalable alternative [5. 6, 7], achieving state-of-the- aht performance for generalist robot policies [7] and showing impressive levels of generalization to new objects and scenes. However, existing VLAs do not sufficiently leverage some of the most appealing properties of the underlying language and vision-language models, specifically their ability to reason through the steps required to solve given task.
Reasoning for language and control. Such step-by-step reasoning is a key ingredient for the ability of large language models (LLMs) to solve a wide range of complex tasks. Prompting LLMs (directly [46] or with in-context examples [8]) "think step-by-step! about the problem before formulating an answer can significantly improve their performance, with such chain-of-thought reasoning techniques becoming standard practice in language modeling and wrision)Jlanguee training [9, 47]. A number of works have explored similar techniques in the context of high-level task planning for robotics [48, 49, 50, 51, 52, 53, 54, 55, 56]. These approaches use pre- trained or fine-tuned LM to decompose tasks into high-level sub-tasks, but rely on pre-trained low-level policies to execute them. However, we argue that (1) careful reasoning can be beneficial for both high- level sub-task reasoning and during low-level control and (2) all such levels of reasoning should be strongly grounded in visual observations of the scene and the agent's 's state. Thus, in contrast to these prior works and language-only COT,, our approach trains VLA policy autoregressively generate CoT' (for high- and low-level reasoning) and actions given input instructions and observations, ensuring that both are firmly grounded in the agent's 's environment. We empirically confirm that such a formulation is critical to effectively leveraging (V)LM reasoning capabilities for control.
model
3 Preliminaries: Vision-Language-Actinn Models
Our work leverages VLAs as the backbone for our embodied of-in-of-though, policies. VLAs use a simple policy learning recipe: starting from pre-trained vision-language model, they directly finetune the model to autoregressively predict the next robot action a given the current image observation I and task instruction T. To enable this, the continuous robot actions are typ- ically converted to discrete action tokens To in the vocabulary of the vision-language model via per-dimension action discretization scheme that assigns each continuous value to one of 256 bins [5, 7].
OpenVLA
Actionn Le- Tolver Loue
AK MODO
Liama 2 7B
"Dut agaaiann In bow"
Tokoriber
Figure 2:
The VL. model. Reproduced with permission from Kim et al. [7].
In this work, we use the recently released Open VLA model [7] (see Fig. 2), since it achieves state- of-the-art performance and is fully open-source. The model builds on the Prismatic VLM [35] and consists of fused visual encoder that combines pre-trained SigL.P [57] and/or Dino V2 [58] features and Llama 2 7B [59] LLM backbone. During training, input images are encoded into visual token embeddings using the pre- pre-trained vision encoders, the task instruction is mapped to task tokens using Llama 2's text tokenizer, and the model is trained to map these inputs the target action tokens with
3

---

Input
Generated Embodied Chain-of- Thought
Robot Action
TASK:
SUBTASK REASONING:
MOVE REASONING:
GRIPPER N66.
ACTION: [ax, 6Grg) p)
Place the watermaion on towel
The watermilo fiss object rooct needs interaet with. The mobtl is not yel close lo the weter malonn, so the robot needs to move ploser
The watermslon is behind the robet, reeck to mine barkeward
PLAN: Move watermaion
VISIBLE E OBJJCTS:
Walamadon
2. Fimily green
MOVE:: Move baskwarrd
Towel (P3, 0o, 213, 1981
USER: "Put the wanermalon Bhe lowel
4. Place watermelon on towel
Mose towel
SUBTASK:
Miooe lo the madarnteii oio
Figure 3: Steps of our embodied chain-of thought reasoning. We interleave several intermediate reasoning steps into the mapping from inputs to robot actions. Green: "standard" linguistic chain-of of-thought steps that break a given instruction into the required sub-tasks. Purple: Embodied -hain-of-thought steps that require grounding the policy's 's reasoning in the scene and robot state. Our experiments show that these grounded reasoning steps are key to improving policy performance with chain-of-thought reasoning.
the standard autoregressive next token prediction objective. We likewise use this objective, but also insert reasoning text tokens in the training sequences such that the policy learns to produce the ECoT reasoning followed by the action in response to the observation and instruction. Next, we will discuss how we can improve upon this conventional VLA training recipe by enabling the VLA reason through the task at hand before deciding which action to take.
Embodied Chain-of-Though Reasoning for Visuomotor Policies
We now discuss our approach for training VLAs to perform embodied chain-of-though reasoning about plans, sub-tasks, motions, and visual features before predicting the next robot action (see Fig. 1). Unlike many proprietary large language models, the relatively small LLM backbones used in current VLAs struggle to perform involved reasoning when simply prompted to think step-by-step [8]. Instead, we propose to explicitly train VLA models to perform embodied CoT reasoning. Concretely, we label data from existing robot datasets post-hoc with reasoning chains filled with features extracted from various pre-trained models and use the resulting dataset of observation Observation-rcasoningg rewaoning-aateeieeeeeeeo for training. In practice, we ensure that all elements of the generated reasoning data can be represented as strings, such that we can use the Llama 2 text tokenizer to translate them into reasoning tokens. Then, train the VIA to autoregressively predict these tokens, directly followed by action tokens.
tuples
While this approach is conceptually simple, its implementation requires answering multiple key questions: (1) Which reasoning steps are suitable for guiding policies in solving embodied robot manipulation tasks (Fig. 3)? (2) How can we generate training data for these reasoning steps at scale on existing robot datasets (Section 4.2)? Another practical consideration arises after training, while using ECoT policies for robot control: carefully reasoning through each action can significantly slow down policy inference. We discuss solutions to these problems in the following sections.
4.1 Designing Embodied Chain-of-Thought Reasoning Steps
Our goals when designing the steps of our embodied thain-of-thougeee reasoning chains are twofold: encourage the model to (A) reason through the required high-level steps of the task at hand and determine which step needs to be executed next, and (B) increasingly ground this reasoning in lower-level features of the scene and robot state before predicting the robot action.
We visualize the ECoT reasoning steps that we train the VLA to perform for an example task in Fig. 3. From left to right, the model is trained to first rephrase the task instruction (TASK) and predict a high-level plan of steps for achieving the instructed task (PLAN). Next, it reasons through which of the sub-tasks should be executed at the present step (SUBTASK). ) a task which requires understanding the current state of the scene and robot. Then, the model predicts an even lower-level language command like "move left" or "move up" (MOVE) that is closely related to the low-level actions the robot needs to execute. Finally, we ask the model to predict precise, spatially grounded features that describe the scene and thus force the model to pay close attention to all elements of the input image- specifically, the pixel position of the robot end effector (GRIPPER) and the names and bounding box pixel coordinates of all objects in the scene (OBJECTS).
While we believe that our choice of reasoning tasks and their order is well-aligned with our intuition for sensible step-by-step solution to the task, we by means exhaustively explored all possible

---

Robot Trajectory from Dataset
Describe scene
Compute motion primitives
Generate plans subtasks
Task Instruction
Prismatic-VLMM
Proprio > Primitives
Gemini LLM
Put the watermatin on the towel
Please describe
Explste the plan, subtask, and movement for each stap, givent
Proprio 1: [ix, A9, AGrp!
State Info
This watermelon, spcon, mustroom
Task
Desi:
Nouss
Right down, down, ...]
Proprio 1: [ax, 28, dGrip)
Plat: Go to matorolon, groap, go mare miusse
Observations
2. Extract bounding boxes
Compute gripper position
nobot greap the wateretion first
Subdasio:
Grounding DINO
OWL SAM
The now graup it
Movwe:
robodo
The seeterattos de eiger of the robot more wiights
watormoton bolw modent
Figure 4: Our pipeline for generating ECoT data at scale for given robot dataset. We use Prismatic VLM [35] create a scene description (1), and Grounding Dino [29] detect bounding boxes for all objects (2). We
then compute templated motion primitives from low-level robot states (3) and the robot gripper position using OWLv2 [60] and SAM [61] (4), Finally, we use Gemini [62] to create synthetic reasoning chains (5).
reasoning tasks. Testing alternative tasks and task orderings, and finding ways automatically determine sensible reasoning chains are important directions for future work.
4.2 Generating Embodied Chain-of-Thought Data at Scale
While human annotations are the gold standard, is impractical to get them for large robot learning datasets [6], which consist of millions of transitions. Thus, we instead propose to leverage pre-traines vision and/or language foundation models automatically generate ECoT training data, akin to synthetic data generation in NLP [63]. We give an overview of our data generation pipeline in Fig. 4.
For given image-instruction pair, we first prompt Prismatic-7B VLM [35] to describe the scene. We then concatenate the instruction and this description and input them into grounding DINO [29], an open- open-vocabulary object detector, detect all relevant objects and their bounding boxes with the corresponding language snippet labels from the input text. See App. A for examples and details.
Next, we generate the per-step motion primitives in MOVE (e.g.. "move left"). Following Belkhale et al. [64], we use robot proprioception to determine the movement for the next 4 time steps (assuming a fixed camera), and translate this into one of 729 templated movement primitives (see App. B). We use OWLv2 [60] and SAM [61] to detect 2D end effector positions in the training images (GRIPPER) paired with 3D positions extracted from the robot state to fit robust estimate of the projection matrix using RANSAC [65]. We then use the 2D projections of the robot end-effector position for our training. This is repeated for each trajectory, eliminating the need to assume fixed camera parameters.
To generate the final reasoning chain, we feed each episode's 's task instruction, scene description, and per-step movement primitives into Gemini 1.0 [62] and prompt it to produce both a high-level plan of sub-tasks in accordance with the task instruction and observed movement primitives and the current sub-task for each step. We also ask it to briefly explain the primitive movement and chosen sub-task in each step, which include in the ECoT training data. We run our data generation pipeline on the complete Bridge v2 dataset [13], with more than 2.5M transitions, over the course of 7 days.
4.3 Efficient Chain-of- Chain-of-Thought Inference for Robot Policies
Inference speed is key challenge for ECoT policies. They needs to predict around 350 reasoning tokens per timestep, compared to Vpen'LA's 7 action tokens. We explore simple solution for speeding up inference: as our policy learns next token prediction, it can continue arbitrary input reasoning prefixes; thus can keep parts of the reasoning chain like the high-level plan or the current sub-task fixed for multiple steps. Encoding previously predicted tokens is much faster for Transformer-based policies than generating them. We compare two such strategies: (1) synchronous execution, where we predict the high-level reasoning every N steps, and (2) asynchronous execution, in which one ECoT policy instance continually updates the high-level reasoning chains, while a second policy instance uses the most recent reasoning chain to predict low-level reasoning steps and actions. We report the performance and inference speed for all approaches in Section 5.5. Note that these speed-ups are orthogonal to widely used approaches for improving throughput of LLMs. We explore one such approach- compiling with TensorRT-LLM [66] in App. F.
5

---

Table 1: Comparison success rates for OpenVLA, RT-2-X, and ECoT across two scenes (one with in-distribution camera view and one with out-of-distriuttinn) Mean 1 one StdErr. On aggregate, our ECoT policy achieves the highest success rate, improving absolute success rate by 45% 22% 19%, and 18% over Octo, OpenVLA, RT-2-X, and naive CoT respectively in the in in-distrbution view setting and 48%, 34%, 16%, and 16% the out-of -it--fffditiuttiion view setting.
Algaitton
Algurtthon
Orta
OvarVLA
RT:3X
Natro CxT
BCxT
Octo
Opea
RT:3-X
Noive CuT
BCaT
Pat srnikreees in
29%
55%
1405
75%
1005
2075
55%
75%
76%
65%
Pai spen
007
037
109%
4055
906
100%
19%
Wape (plaz] paa) with andd
13%
30%
35%
15%
50%
0%
50%
075
175
65%
Spattal Rdaliiss
Pai muddeee
(ihh right resikfe
225
175
22%
33%
09
175
1276
555
67R
Pat pappl
Pai Ingai lell obgei unddk-otjet otjet
13%
066
50%
63%
04
25%g
15%
50%
631
OOD 0bjert
More
meaving tap deeepnn soammtio)
foreeiiiion
35%
209
507
50%
309
305
204
9016
50%
50R
Pat easskmeen in cill cup
47S
10%
70%
375
1975
206
2141
78%
000
Pvb up vas obgect thet
sposac/ towell]
59%
33%
58%
50R
42%
176
17%
671
25%
67%
Par whhle dejec booel
75%
13%
88%
09
176
0985
Pai shacc caed (satig/
dockknng ou londl
25%
38%
389
384
75%
136
069
156
3866
75%
Aggrigate
215 1-355
485 - 3.95
67.1 4375
06.1 1175
66% 0. 330
185
3.9
33%
1.3855
08-
1.05
(85
1-105
$4.
5 Experiments
In this section, we investigate the effectiveness of ECoT for robot control across range of challenging manipulation tasks. We answer the following questions: (1) Does embodied chain-of-thought reasoning improve the performance of VLA policies (Section 5.2y? (2) Does embodied chain-of- thought reasoning make it easier to interpret and correct policy failures (Sections 5.3 and 5.4)? (3) How can we optimize the runtime efficiency of policies with embodied CoT reasoning (Section 5.5)?
5.1 Experimental Setup
Robot setup and training data. We evaluate on a 6-DoF Widow robot arm, a commonly-used setup for evaluating learned robot policies [16, 7]. Given a single 3rd person camera and natural language instruction, the policy predicts end-effector velocity actions to control the robot. Bridge v2 [13] provides a diverse dataset of 60k teleoperated demonstrations. We apply our pipeline for synthetic generation of tfain-of-thought data (Section 4.2) on this dataset to obtain our training dataset.
Evaluation tasks. We design a suite of challenging evaluation tasks that focus on testing the policies' generalization ability along multiple axes: processing spatial relations, interacting with unseen objects, and performing unseen instructions. All policies are evaluated on the same real-world setups to control for camera angle, lighting, and background. We perform 314 total trials per approach.
Comparisons. We compare policy (ECoT) to state-of-ther -het-rtt--rtt VLA policies, namely Open- VLA [7], the same model our approach is built upon, but trained without chain-of-thougle reasoning. and RT-2-X [6], a 55B parameter closed VLA policy. To ensure fair comparison, we train the Open VLA policy on the same dataset we use for training our approach (the Bridge V2 data [13]), denoted as Open VLA (Bridge). For RT-2-X we cannot control the data distribution in the same way since the model is closed, but it was trained on Bridge V2 data and additional datasets from the Open X-Embodiment dataset [6]. Thus, it was trained on more data than our approach. We also compare against Octo [16], which is also trained on that dataset, but not fine-tuned from VLM (i.c., it is not a VLA). Finally, we compare to Naïve CoT, version of our model that only uses non-embodied CoT reasoning about sub-tasks akin to conventional CoT reasoning in language modeling (see Fig. 3). This comparison will test the importance of using embodied reasoning for VLA policies.
5.2 Embodied Chain-of- Chain-of-Thought Reasoning Improves Policy Generalization
We report performance of all approaches our evaluation set in Table 1. We that while VLe (Bridge) achieves high performance on in-distribution tasks, it struggles on the hard generalization cases we test. RT-2-X performs better than vanilla Open VLA (Bridge), potentially due to the larger robot pre-training dataset (note again that Open VLA and our approach are only trained on the Bridge dataset) and the fact that it co-trains the policy with Internet-scale vision- languaee data and robot data, while all other approaches only use robot data during fine-tuning.
Importantly, we find that our ECoT policy substantially outperforms Open VLA (Bridge) across all generalization evaluations but one. This is notable, since both policies have the exact same VLM base

---

"Move the mushroom to the detergent"
"Put the loftnost object on the middle object
"Pick up the screwdriver
stuffed animal
a white mushroom
blue plastic bottle
blue towel
8 red screwdriver
detergent
TASK the mushroom
TASK: Put pink stuffed animal the bluse townl
TASK, sorewdrtver to the container
PLAN: Move the mustroom, grasp it, move to the detergent, release the mushroom
PLAN: Move the plnk stutted animal, Pick up the
pirs stuffed animall Mowe to the blue fowel. Put the towel.
PLANC Move to screwtitver, pick it up, and move t0 the container
SUETARK The mushroom robot, meeda be greaped firat Move the mushroom
plikk stuffed animal
SUSTAAS The pink stufted animal hold the
SUB TASK: The screwediver the tops -eff-elf currar, the robut needs move forward and down reach it > Move to the acrewdriver
rebotis amm and the blue towel the miodle object >
MOVE mushroom right robot, so right Move right
Put the pink stuffed animal on the bluee towrl!
souffed needs
MOVE;
Stop
Move
Figure 5: Qualitative ECoT predictions from our model for two successful trajectories (left, middle) and one failure (right). Irrelevant bounding boxes are greyed for readability. Left: high-level reasoning and low-level object segmentations are correct, leading to a successful rollout. Middle: the command is correctly rephrased to refer to specific objects (i.e., "the leftmost object" is identified as the pink toy). Right: the hammer is incorrectly identified as a screwdriver, causing the robot to take inappropriate actions.
model and use the same robot data for fine-tuning, The only difference in the use of CoT reasoning by approach. Curiously, our ECoT model even surpasses the performance of RT-2-X in the tested tasks, even though RT-2-X is trained on 10 additional robot datasets and uses neural network that is 7x larger (55B vs. 7B). Finally, the results in Table 1 show that including embodied reasoning about visual inputs and the low-level robot state significantly boosts performance over the "Naive CoT ablation of our approach, which only reasons about high-level linguistic features like sub-task plans.
We visualize qualitative examples of our model's 's reasoning in Fig. 5. The left two examples show that the model successfully breaks down the task into a sequence of sub-tasks and then crucially grounds those sub-tasks in the scene by predicting the relevant bounding boxes and gripper position of the robot, before deciding on the next move and concrete low-level robot action. We visualize more thain-of-thouseee examples from our evaluation tasks in Fig. 8.
5.3 Diagnosing Policy Failures Through Inspecting Reasoning Chains
In addition to improving performance, such embodied chain-of-thought reasoning provides tool for operators to better understand the decisions the policy takes. By inspecting and visualizing the model's 'ss reasoning steps, we can discover potential mistakes in the reasoning chain that led to policy failure downstream. For instance, in Fig. 5 (right), the ECoT policy failed to solve the task pick up the screwdriver. Inspecting the reasoning chain, we can see that the hammer is incorrectly identified as a screwdriver, causing the robot to reach for that instead. Note that inspecting reasoning chains is not "bullet- "bullee-proof" for interpreting end-to-end policy failures: the model could predict a particular plan and then still deviate from it when choosing the final action. However, in practice we find that reasoning chains often correlate strongly with the executed actions. We provide more examples for diagnosing policy failures via its reasoning chains in Fig. 8.
5.4 Chain-of-TT Thought Reasoning Enables Interactive Policy Correction
Training a policy reason through task step-by-step provides powerful mechanism for humans to interact with the policy and correct its behavior: instead of needing teleoperation equipment to provide direct robot action feedback like in DAgger [67]. users can simply modify its reasoning chains via natural language feedback to correct the policy's 's behavior. Prior work used carefully designed architectures explicitly trained to support such correction via language [68, 69], but we test whether similar capabilities emerge naturally by training VLA policies to perform ECoT reasoning.
We rerun evaluations of our ECoT policy on some of the most challenging tasks from Table (put mushroom in cup. pick up out-of-distribttion object, and pick up non-yellow object), in which policy without human intervention achieved an average success rate of only 32%. We allow human operator to interrupt policy execution once over the course of the episode and provide natural
7

---

User: Pick any objest that is not yellow.
ChatGPT: Correct reasoning...
No Intervantion one human intervention
CoT OpenVLA:
Nest Task: neach the purple object Next Move: move forward
-12%
60
+4856
CoT OpenVLA: Noxt Task: find
+20%
32
Noxt Move: move left
Warl of the
purple object
OpenVLA (Bridge)
RT-2-X
ECoT (Ours)
Figure 6: Embodied chain-of of thought training enables interactive human policy correction in natural language. Left: given a human intervention natural language, we use ChatGPT to correct our model's reasoning chains. Right: our embodied chain-of- of-thhuttttt policy can benefit from a human language intervention most, increasing success rate by 48% on our most challenging evaluation tasks. language feedback (e.g. "no, the screwdriver is in the back right comner, "release the mushroom now!", or "the cup is tall")) (see Fig. 6). Then, we use ChatGPT to adapt our model's reasoning chain based on the language feedback, prompting it to produce a corrected reasoning chain (see Fig. 11 for the exact prompt used). Finally. we feed this corrected chain back into our policy and continue execution, holding the corrected reasoning chain fixed for 5 steps.
The results in Fig. 6 (right) show that our ECoT policy can make effective use of the human language VLA policy and RT-2-X in the same way, providing each with single human language correction per rollout, but find that neither of them can benefit from the human intervention to the same degree (for both we also use ChatGPT to incorporate the intervention into the original task instruction to allow for fair comparison). This is because standard VLAs that directly map from images and instructions to actions do not expose any part of their internal decision-making process. Not only does this make "debugging" incorrect behaviors difficult, it also only permits changing the prompt for interventions. Inference
intervention, increasing its success rate by 48%. In contrast, we evaluate the vanilla OLe
5.5 Efficient
We compare approaches for accelerating ECoT policy inference (see Section 4.3) to naively running the full generation every step of execution in Table 2. We also report the speed-up up that is achieved by both pro- posed approaches from Section 4.3 vs. naïve execution. Both approaches achieve inference speed improvements while at least matching perfor- mance, with asynchronous execution achieving the largest speed-up at the cost of doubling the compute required at inference time (since two policy instances running in parallel). We use the 5-step freeze ap- proach for the main results presented in Table 1, since it provides the best performance-speed tradeoff. We used a small task subset (put mushroom in pot, move mushroom to detergen//measurigg tape, and put the left/right object on the middle).
Table 2: Performance of ac- celerated CoT inference ap- proaches and their relative speed-ups over the naive approach, averaged across 3 tasks (25 trials total).
Success
Speed Up
Naive 5-Step Asyne
63%
72%
24%
65%
40%%
6 Discussion and Limitations
In this work, we demonstrate that training VLA policies to perform chain-of of-thought reasoning can substantially increase their performance without the need to collect additional robot training data. Instead of simply applying the CoT recipe from language modeling, our experiments underline the importance of adding reasoning steps that are strongly grounded in the task, scene, and robot state. While our results are encouraging, our approach has several limitations. First, our model does not adapt the structure of its reasoning chains to the task at hand; it always performs all steps of reasoning in the fixed order we chose. A more effective strategy may involve executing only a subset of reasoning steps based the robot and scene state, and future work can explore directly optimizing the model to pick the best reasoning steps. Additionally, scaling the ECoT training to a larger subset of the OXE dataset [6] will improve transfer of ECoT capabilities to more robots.
8

---

Acknowledgments
We would like to thank Sidd Karamcheti for his help with using the Prismatic VLM and VLe codebases. We also thank Google DeepMind for providing Alpha API access the RT-2-X model. This research was partly supported by ONR under X00014-20-1-2388 NSF IIS-2150826 and IIS- 2246811 and the AI Institute. Michal Zawalski was funded by the Polish National Agency for Academic Exchange (PPXISTA200///100079///000
References
[1] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision, 2022.
[2] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023.
[3] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.022 2024.
[4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, I., Fei-Fei, C. Finn, T. Gale, I.. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, Hewitt, D. E. Ho, J. Hong, K. Hsu, Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna. R. Kuditipudi, A. Kumar, Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Wu, S. M. Xie, M. Yasunaga, You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Zhang, Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models, 2022.
J. Huang, T. Icard, Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling,
[5] A. Brohan, N. Brown, J. Carbajal, Chebotar, X. Chen, K. Choromanski, Ding, D. Driess, A. Dubey, Finn, P. Florence, C. Fu, M. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee. S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023.
[6] Embodiment Collaboration, A. O'Neill, A. Rehman, A. Maddukuri. A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie,
A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess- Limerick, Kim, Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp. G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Wang, H. Su, H.-S Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovie, Leal, J. Liang. J. Abou-Chakra, J. Kim, J. Drake, Peters, J. Schneider, J. Hsu, J. Bohg, Bingham, Wu, Gao, J. Hu, Wu, Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang. J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang, J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go.

---

K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. J. Fan, L. Ou, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip. M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart in-Mart in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang. R. Mendonca, R. Shah, R. Hoque, R. Julian, Bustamante, Kirmani, S. Levine, Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang. Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Matsuo, Z. Ma, Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
[7] M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, P. Sanketi, Q Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-actoo model. 2024.
[8] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[9] H. W. Chung, L.. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, F. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling cestructiionfinetunne language models, 2022.
[10] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation, 2018.
[11] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski,, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale, 2021.
[12] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets, 2021.
[13] H. Walke, K. Black, A. Lee, M. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL., 2023.
[14] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023.
10

---

[15] H. Bharadhwaj. Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023.
J. Vakil, M. Sharma, A. Gupta, S. Tulsiani, and V. Kumar. Roboagent:
[16] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics; Science and Systems, Delft, Netherlands, 2024.
[17] D. Shah, A. Sridhar, N. Dashora, Stachowicz, K. Black, N. Hirose, and Levine, Vint: A foundation model for visual navigation, 2023.
[18] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours, 2015.
[19] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung. J. Gao, J. Emmons, A. Gupta,
E. Orbay, S. Savarese, and L.. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation, 2018.
[20] A. Gupta, A. Murali, Gandhi, and Pinto. Robot learning in homes: Improving generaliza- tion and reducing dataset bias, 2018.
[21] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning, 2020.
[22] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference Robot Learning (CoRL), Auckland, New Zealand, 2022.
[23] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, O. Sushkov, D. Barker, J. Scholz, M. Denil, de Freitas, and Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning, 2020.
[24] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002, PMLR, 2022.
[25] H.-S. Fang, H. Fang, Z. Tang, J. Liu, J. Wang. H. Zhu, and C. Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
[26] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, I. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, Chung, J. Drake, E. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le, Y. Li, K. Lin, Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martin, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip. Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.
[27] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. Lee, M. Bauza, T. Davchev, Y. Zhou, A. Gupta, A. Raju, A. Laurens, C. Fantacci. V. Dalibard, M. Zambelli, M. Martins, R. Pevce- viciute, M. Blokzijl, M. Denil, N. Batchelor, T. Lampe, E. Parisotto, K. Zoina, S. Reed, S. G.
11

---

Colmenarejo, J. Scholz, A. Abdolmaleki, O. Groth, J.-B. Regli, O. Sushkov, T. Rothörl, J. E. Chen, Y. Aytar, D. Barker, J. Ortiz, M. Riedmiller, J. T. Springenberg, R. Hadsell, F. Nori, and
N. Heess. Robocat: A self-improving generalist agent for robotic manipulation, 2023.
(28) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[29] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023.
[30] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and Ommer. High-resolution image synthesis with latent diffusion models, 2022.
[31] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction tuning. 2023.
[32] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-languag understanding and generation, 2022.
[33] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
[34] W. Dai, J. Li, Li, A. M. H. Tiong, Zhao, W. Wang, B. Li, P. Fung, and Hoi. Instructblip: Towards general- purepseep vision-language models with instruction tuning, 2023.
[35] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models, 2024.
[36] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023.
[37] Y. Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi. Vision-language models success detectors, 2023.
[38] S. A. Sontakke, J. Zhang, S. M. R. Arnold, Pertsch, E. Bıyık, Sadigh, C. Finn, and L. Itti. Roboclip: One demonstration enough to learn robot policies, 2023.
[39] J. Ma, W. Liang, V. Som, Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language- image representations and rewards for robotic control, 2023.
[40] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation, 2022.
[41] S. Karameheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language-driver representation learning for robotics, 2023.
[42] W. Chen, O. Mees, A. Kumar, and S. Levine. Vision languaeee models provide promptable representations for reinforcement learning, 2024.
[43] A. Stone. T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, S. Kirmani, B. Zitkovich, F. Xia, C. Finn, and Hausman. Open-world object manipulation using pre- trained vasion-language models. In arXiv preprint, 2023.
[44] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[45] F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting, 2024.
[46] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners, 2023.
12

---

[47] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. Wu, S.-C. Zhu, and J. Gao. Chameleon: Plug-and-play compositional reasoning with large language models, 2023.
[48] W. Huang, F. Xia, T. Xiao, H. Chan, Liang, P. Florence, A. Zeng. J. Tompson, Mordatch, Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, Levine, K. Hausman, and Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022.
[49] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control, 2023.
[50] A. Zeng, M. Attarian, B. Ichter, Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit,
M. Ryoo, V. Sindhwani, Lee, Vanhoucke, and Florence. Socratic models: Composing zero-shot multimodal reasoning with language. 2022.
[51] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[52] Sharma, A. Torralba, and J. Andreas. Skill induction and planning with latent language, 2022.
[53] C. Huang, O. Mees, A. Zeng, and W. Burgard. Audio visual language maps for robot navigation. In Proceedings of the International Symposium on Experimental Robotics (ISER), Chiang Mai, Thailand, 2023.
[54] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[55] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[56] H. Ha, P. Florence, and S. Song. Scaling up and distilling down: Language-guided robot skill acquisition, 2023.
[57] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training, 2023.
[58] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[59] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, Lavril, J. Lee, D. Liskovich, Lu, Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, Reizenstein, R. Rungta, Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[60] M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 36, 2024.
[61] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026 2023.
13

---

[62] Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
[63] Mukherjee, Mitra, G. Jawahar, Agarwal, H. Palangi, and A. Awadallah. Orca: Progres- sive learning from complex explanation traces of gpt-4, 2023.
[64] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and Sadigh. Rt-h: Action hierarchies using language, 2024.
[65] M. A. Fischler and R. C. Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):
381-395, 1981.
[66] NVIDIA. Tensorrt-llm. readme-ore -oee--oo-fleeeee 2024.
[67] M. Kelly, C. Sidrane, K.
and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts, 2019.
[68] P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and
D. Fox. Correcting robot plans with natural language feedback, 2022.
[69] I. X. Shi, Z. Hu, T. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn. Yell at your robot: Improving on-the-fly fll from language corrections, 2024.
[70] Micikevicius, Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, Ha, A. Hei- necke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu. Fp8 formats for deep learning, 2022. URL https::// http://axxvvvv.re org/ www.o/////////////////..//..e/// aaa//2ba2222 05433.
[71] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding, 2023.
[72] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kir- mani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405. 05941, 2024.
[73] D. Zhang, C. Li, R. Zhang, S. Xie, W. Xue, X. Xie, and S. Zhang. Fm-ov3d: Foundation model-based cross-modal knowledge blending for open-vocabulary 3d detection, 2023. URL https://arxix: org/ abs 2212. 14465.
[74] L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alab- dulmohsin, M. Tschannen, F. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong. J. Eisenschlos, R. Kabra, M. Bauer, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Pa- palampidi, Henaff, X. Xiong, Soricut, Harmsen, and X. Zhai. Paligemma: versatile 3b vim for transfer, 2024. URL http:://:: /arxivv org/ 2407 .07226.
[75] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. SpatialvIm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455-14465. June 2024.
[76] J. Yang, M. S. Mark, B. Vu, A. Sharma, J. Bohg, and C. Finn. Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning, 2023.
14

---

A Grounding DINO Detections and Prismatic Descriptions
We provide example scene descriptions provided by Prismatic VLM and bounding boxes provided by Grounding DINO in Fig. 7. We filter the predictions based on the provided confidence score, only keeping detections with a box- and text-confidence larger than 0.3 and 0.2 respectively to use for the OBJECT features.
6.37
A robot is holding a carrot above green platts with sllver red poppar, and a yallow bow! on the counter
A wooden table holds blue bottle, grey stuited animal, white stu stufeed animal, and bike towel, with bax and black arm behlind them
green pickllo the countar, silver pot on the stove burner, and white mushroam in white bowl. The stove burner ow the and pickle the counter
EICE veiDe
while plate stove, and white mushroom the plaza. Tha sink is to the loft the stowe, and the pot is the sink
The acens
roboo is putting canot cutting board, which is on the countor. There is green cutting beard. a red popper, a yallow peppor, and a black pot nearby. The counter white with ailver knobs and sink.
yallow cloth on the tablo, and black deviloe. The clott is to be moved from the left side of the table to the center.
Figure Examples captions of observations from the Bridge dataset as generated by our Prismatic VLM, as well as associated bounding boxes generated by Grounding DINO.
15

---

purple object middle container"
"Move the watermalon
"Pick any object aponga"
TASK:
Move the purple object to the silver container
TASK: The watermalon needs to be placed on the towel
TASK. Pick up any object that not a sporge, PLAN: Approach the yellow duss, pick up the
PLAN:
Move the purple object, greap the
PLAN: Move to the watermaion, Pick up the watermelon. 3. Move the sowel. 4, Place the watorrmeoon on towel.
purple object, move to the silver container,
ynloww duick move the yellow duck away from the
mieense the purple object
spange:
SUBTASK: have gresped the purple object, so
SLITAAKK:
oatermaion has been picked up,
SUBTASK: The yollow duck
abject sponge and needs be picked up.
we move to
contalinor
and the now close the towel, Move towel.
Move the aliver container.
Approash yellow
The silver container robol,
the
MOVE:
watermelon,
The yellow duck
amm
needs be approached.
Move left.
*Put the mushroom the white cup"
"Move the muehroom to the meesuring tape"
"Put the mushroom in the white cup"
mushroom
cup,
TASK:
mushroom
mushroom
PLANE Move the mushroom, pisk the mushroom while picked willle neel object to interact with. +
the mushroom, the mushroom mushrcom,
PLANe Move to mushroom, grasp the
SUSTASK:
mushroom, meesuring tapa, place the mushroom on the messuring tapa.
SUBTASK:
The mushroom is picked and the
Move the white
directly below the Move the cup. MOVE The cup is above the arm, the arm newds move to reach Move
MOVE: The white cup to the left and sighttly
SUBTASK: The mushroom should be Iited so that It can be movedt > Lift the mushroom.
move to the
above the amm, the arm needs to move that
direction. > Move left
MOVE: Stop. + Stop.
detergentt
up
object middle object
Save
TASK: Move the mushroam to the detergent, FLANE Grosp the mushroom, move the mushroom
TASKK Pick up the watermaton.
PLAN Move to the watermolon, pick up the
TASKC Put the yollow duck the blue towel PLAN: Move to the yellow ducck, Graup the
To The detergent, release the mustroom..
watormelon, move to the target location, drop the walermelon.
yullow duclk, Move to the blue towel. Releasse the
Twedd be moved the detergent. Grasp the mushroom.
SUBTASK: The musitroom is the object that
SUBTASK: The watermeion is on the table, and the robot needs move to pick it up. + Move to the watermelon.
and the robot ourrently not at the yellow duck.
yellow ducck.
ASTASKK The yellow duick the lettmoet object
The gripper fully alosed yot. close
MCVVE: wntermeton to left the robot,
Move the yellow ducck. The yellow duck now right below robot. Stop.
gripper
the rahot needs to move left get to
Figure 8: More qualitative examples of successful and failed chain-of-thought reasonings.
16

---

townt wion the josto par)
mw- the wotootskddergggnootteeeeeeee
Pud paple obact (xtrestnsddingttaodd contangr
Pathe hww.rcsihtttnaaaeeeeoo. otrect the
Move mwwttoom to jeasuring
Pet Rethmmakeen
Pick up coonet ENTIS Jaboats meel
Put sEject usen (treking) antingloo
PVS BN oddie obant BY bow!
Figure 9: Example starting scenes and associated prompt for all task types.
17

---

B List of Movement Primitives
To classify movement, we take the difference between the current state of the robot and its position four steps ahead. Based the where the difference exceeds threshold of 0.03, we assign it label of the following form:
move [Forward/backkard [left/right] [up/down], tilt [up/down], rotate [clockvise/counne /flockviss//outteeoooooliilll [close/open] gripper
Whenever the movement in certain axis is below the threshold, we omit its block for simplicity. For instance, if the robot just moving left, the label move left. If no movement is detected, the label stop.
While technically it results in 3 729 possible labels, only 54 are used in more than 0.1% of cases:
stop (26.9%)
close gripper (10.8%)
open gräpper (7.2%)
move down (6.8%)
move laft (6.6%)
move right (6.1%)
7. move up (5.7%)
move forward (3.0%)
move backward (2.4%)
10. up, open gripper (2.1%)
11. forward right
12. up, close gripper (1.0%)
13.
left (1.0%)
14. forward left (0.9%)
(0.8%)
16. move dovn, close gripper (0.8%)
17. move right down (0.8%)
18. move laft up (0.8%)
19. move right up (0.8%)
20. move right, rotate clockvise (0.8%)
21. move left, rotate counterclockuise (0.8%)
22. move backward right (0.8%)
23. rotate counterclockiise (0.7%)
24. doun, open gripper (U.7%)
25. clockuise (0.7%)
26. forvard (0.7%)
27. up,
(0.5%)
up, counterclockrrse (0.5%)
baclward up (0.5%)
30. move left, rotate clockvise (0.3%)
31. move backward down (U.3%)
32. move right, open gripper (0.3%)
33. move forvard up (0.3%)
34. move left, open gripper (0. (0.3%)
35. move right, rotate counterclockoise (0.3%)
36. move backward, open gripper (0.2%)
37. move down, rotate clockvise (0.2%)
39. forvard, counterclockvisee (9.2%)
38.
countorclockrisee (0.2%)
40. forvard,
(0.2%)
forvard, open gripper (0.2%)
42. right,
gripper (0.2%)
43. move backward, rotate clockuise (0.2%)
44. move backward, rotate counterelockuiee (0.2%)
45. move left, close gripper (0.2%)
46. move backward right, rotate clockrise (0.1%)
47. move backvard left, rotate counterclockviee (0.1%)
48. move right open gripper (0.1%)
49. move right close gripper (U.1%)
50. move backward, close gripper (0.1%)
51. rotate clockvise, close gripper (0.1%)
52. counterclockrise, close gripper (0.1%)
53. laft up, open gripper (0.1%)
54. forvard right, clackvise (0.1%)
18

---

C Prompts
We now provide all the prompts used for data generation and policy language conditioning.
For using generating scene descriptions with Prismatic (step in Fig. 4) we use the prompt: "Briefly describe the things in this scene and their spatial relations to each other." We prepend "The robot task is: [TASK]." if the given demonstration trajectory contains corresponding task instruction (where we ensure that said instruction contains at least one space character to remove noisy instructions).
We provide the prompt for Gemini data labeling (step 5 in Fig. 4) in Fig. 10. We re-run generation if Gemini fails to produce reasonings of the correct format.
The prompts used for our language-condttioned policies are provided in Fig. 9, along with example starting scenes for the associated tasks. For the OpenVLA VLA-based policies, said prompts are inserted into the template provided by the original authors [7]: "A chat between curious intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: What action should the robot take to [PROMPT]? ASSISTANT"", The agent then generates reasoning text (if trained to do so) and an action.
and artificial
We provide the prompt used for human interventions with ChatGPT in Fig. 11.
D Clutter Evaluations
We provide additional evaluations of how our policy compares against OpenVLA in settings with significant clutter (9 or more objects per scene). We evaluate two rollouts for eight tasks in the four categories used in Table 1: in-distribution (put mushroom on cloth, put corn pot), spatial relations (put mushroom [leftmost/ rightmost] container, put mushroom in [leftmost rightmost container]), outto--iistribution objects (move mushroom to measuring tape, place watermelon on pan), and some tasks are not from the original tasks in Table 1, which we chose to make them more amenable to evaluation in cluttered scenes. We show example starting arrangements for each task in Fig. 12.
instructions (put edible object on towel, put animal in pot). Note that
We find that our ECoT policy achieves 75% success rate while OpenVLA only achieves 25%, indicating that reasoning may aid robustifying against clutter. Qualitatively, we note that many the presence of clutter, while our ECoT policy still
failure cases are exacerbated for Open enjoys the same benefits noted in the default task suite. Namely, OpenVLA often gets distracted by commonly-repreeeented clutter objects (like the mushroom) while failing to reason about object semantics, which our approach deals with well (e.g., successfully identifying toy foods and stuffed animals when queried to pick the edible objects and animals respectively, even though such wordings are rare or non-existent in our training data).
E
Task Objects
We provide visualizations of example representative starting setups for all tasks in Fig. 9. The tasks involve the following objects:
1. Put mushroom in metal por: mushroom, shallow metal pot, light blue towel, spoon
2. Put carrot on plate: carrot, eggplant, shallow metal pot, green plate
3. Put spoon on towel: spoon, light blue towel, mushroom, shallow metal pot
4. Wipe [plate / pan]: plate, pan, dark blue towel, yellow towel
5. Put the (purple object/ / mushroom] in the [leftmost / middle / rightmost] container: purple beet, mushroom, tan plastic bowl, metal bowl, white plastic bowl
6. Put (rightmost/ / leftmost] on middle: mushroom, spoon, pink bear, green frog, light blue towel
7. Move mushroom to [measuring tape / detergent]: measuring tape, detergent, mushroom, light blue towel
8. Put the watermelon the towel: watermelon, mushroom, spoon, light blue towel
9. Pick up the [hammer / watermelon / measuring tape / detergent screwdriver]: hammer, watermelon, measuring tape, detergent, screwdriver
19

---

10. Put mushroom in white cup: tall white cup, mushroom
11. Pick up any object that is not [yellow / a duck a sponge / a towel]: yellow duck, yellow towel, sponge, purple beet
12. Put the object for [drinking/ / eaaingg on the towel: dark blue towel, spoon, cup, mushroom
13. Put the edible object in the bowl: tomato, pink bear, carrot, yellow duck, banana, green frog, corn, monkey
We frequently use the mushroom and light blue towel as distractors, as they are well-represented in our training data, and so non-reasoning policies often erroneously pick them up or place things on them when instructed otherwise, especially when dealing with the less common or disttiibittiitto objects (measuring tape, detergent, watermelon, hammer, screwdriver, white cup).
F Additional Analysis and Experiments
Can we improve inference speeds by compiling the language model backbone? As our model shares an architecture with Open VLA [7], which likewise is fine-tuned from a Prismatic VLM [35], the policy itself has vision and language model backbone. The latter where much of the inference time is spent: it is fine-tuned Llama [59] model that has generate 300 chain-of-thought reasoning language and action tokens in response to 256 soft prompt image embedding tokens. However, since it does use this popular language model architecture, our policy can benefit from the numerous recent developments in improving LLM inference speeds.
We show this by compiling our ECoT VLA's language model backbone with TensorRT-LLM [66]. Using FP8 quantization [70], we were able to lower the inference time of generating a full reasoning chain followed by actions from seconds). We note that further optimizations (both with TenSorRT-LLM or with other techniques, like speculative decoding [71]) are possible and that the policy can be made smoother with techniques like action chunking [2], but we leave these approaches to future works.
seconds to ~ 1 second (or 0.8 seconds with freezing, from
Can we improve speed and interpretability of the ECoT reasoning? We test two modifications to the structure of our reasoning chains. First, we move the bounding box generations earlier in the chain, right after the plan. This way, we can keep the bounding boxes fixed in our N-step inference (see Section 4.3, Since the bounding box generation represents a significant fraction of the predicted tokens, this change can speed up ECoT inference by 30- 50% in our experiments. Secondly, train the model to autoregressively predict the next four future gripper positions, in addition to the current one. Not only does this gives operators a rough visualization of what the ECoT policy expects its motion to be in the future, but it serves as an (albeit imperfect) proxy indicating how our policy would behave. This will be important for the following experiments on other robot embodiments, for which we do not have the ability to run real world rollouts.
We evaluate this policy (and all subsequent ones) on large subset of tasks on the -ntoof-distribution view station, totaling 106 trials per policy. We find that, while this frozen bounding box policy does perform worse than our base ECoT model, it nonetheless outperforms all baselines (Octo, OpenVLA (Bridge), and RT-2-X), as shown in Table 3. Thus, due to its relative higher speed and ability to visualize rollouts, we adopt this structure for all subsequent experiments.
Does co-training with vision-language data help? During VLA fine-tuning, both BCoT and Open VLA lose the base VLM's ability to respond conversationally to natural language questions. This can be remedied by co-training the VLAs with languaee training data in addition to robot action data. Prior work found that such co-training can improve VLA capabilities [5]. We test performance of an ECoT model co-trained with robot data and the vision-language training dataset of the base Prismatic VLM [35] at 3:1 ratio. Qualitatively, we find that the co-trained model indeed retains its ability to answer questions in chat format in addition to robot control. We compare the performance of this model with our base ECoT model across a large subset of the robot control tasks in Table 1 (in-distribution view), totaling 106 trials per model. The results in Table 3 suggest that co-training does not lead to measurable performance improvements on our evaluation tasks. Anecdotally, we find that the co-trained model more reliably recognize celebrities and improve performance tasks like "bring coke can to Taylor Swift" (4/4 successes vs. 0/4 for our base ECoT).
20

---

Table Success rate ECoT trained with various design choices, as evaluated on large subset trials on the harder out-of-distrbutionn view setting. While our base policy performs the best on aggregate (69%), the other approaches achieve higher performance on certain tasks. All policies in this table outperform Open (Bridge), RT-2-X, X, and Octo's performances the same trial subset (29%, 46%, and 14% aggregate success rates respectively).
Base ECoT
Frozen Bbox
Co-trained ECoT
Fine-tuned ECoT
Task
ECoT
Put mushroom in pot Put spoon on towel
57%
86%
86%
86%
80%
60%
40%
80%
Put carrot plate
83%
67%
100%
100%
Wipe [plate / pan] with towel
75%
50%
25%
25%
Put mushroom in
89%
55%
1 11%
44%
Ileft / right/ middle] container
Put purple object in
44%
67%
44%
67%
[left / right/ middle] container
Put [right/ left] object on middle object
63%
75%
75%
75%
Pick up [screwdriver hammer/ measuring tape/ detergent watermelon]
50%
60%
80%
70%
Move mushroom to Imeasuring tape detergent] Put mushroom in tall cup
90%
90%
100%
60%
20%
20%
20%
40%
Place watermelon on towel
60%
0%
20%
40%
Pick up any object that is not [yellow duck/ a sponge / a towel] Put the edible object in the bowl Put the object used for [eating/
67%
58%
42%
17%
100%
75%
50%
38%
75%
38%
50%
25%
drinking] on towel
Aggregate
69%
60%
56%
54%
Does ECoT capability transfer to other robots? We test whether fine-tuning a generalist VLA policy with ECoT data can transfer ECoT reasoning between robot embodiments. Concretely, we use the official checkpoint of the OpenVLA-P2 model [7], which was trained on a mix of robot datasets. We continue training the released checkpoint on this mix, but replace the original BridgeData V2 dataset with generated ECoT dataset. As result, approximately 13% of the training data is ECoT data. We make two key findings. First, fine-tuning pre-trained VLA perform ECoT reasoning is substantially faster than training an ECoT VLA from the base VLM. We observe that within 20k training steps the fine-tuned model nearly matches the performance our original ECoT model trained for 80k steps (Table 3). Qualitatively, we even observe comparable performance after only 2500 steps. This represents a 4x and 30x reduction in required compute respectively.
Secondly, we find that the fine-tuned model can perform ECoT reasoning on other robot embodiments than has been trained for, simply by prompting it with the beginning a ECoT sequence ("TASK:") (Fig. 13). recognizes robot grippers, objects and their positions, and predicting future gripper movements, despite the large differences in robot appearance, scene layout and camera setup. This result is surprising, since only provided ECoT training data for single robot embodiment: the WidowX robot in the BridgeData V2 dataset. We hypothesize that the VLM pre-training enables the model to generalize the concepts of robot end-effector position and movement, and object idendity and positions between robots and scenes. We also tried rolling out the fine-tuned ECoT model in the SIMPLER real-to-sim enviromments of [72] on the Google Robot tasks, while prompting for FCoT prediction as described above. However, we found that the ECoT model suffered from the real-to-sim domain gap of the SIMPLER environments, producing more faulty reasoning chains than real Google robot images, and thus not improving overall performance compared to OpenVLA baseline without ECoT.
21

---

G Extended Qualitative Analysis of Reasoning Behaviors
Along with Fig. 5, we provide further qualitative examples of successful and failed BCoT reasonings in Fig. 8. We find that most failures are due to some kind of semantic scene misunderstandinng the most common form being incorrect bounding box labels (as shown in the rightmost example in Fig. 5). However, even when bounding boxes are somewhat correct (see top middle example in Fig. 8 where the watermelon is labeled believes the watermelon has been grasped, when it actually grasped the spoon. We also note that our model can often handle noisy labels: the bottom right example in Fig. 8 labels the green and yellow frog as a "yellow duck," however the reasoning referring to it still aligns with the specified task. Finally, we note that failures from incorrect plans or subtasks (but correct scene understanding) are typically uncommon, especially because the plans are re-generated after freezing and are often corrected in the process.
red object and the spoon has label), the model still erroneously
H Extended Discussions of Design Choices and Alternatives
We now discuss some the advantages and disadvantages of various alternative design choices and optimizations.
H.1 Inference Optimizations
Performing ECoT inference is significantly slower than "standard" VLA counterparts. Specifically, VLe and RT-2-X achieve control frequencies of 3 and 2 Hz respectively, while our freezing approach achieves 0.5. However, we note that this slower inference speed is nonetheless empirically sufficient for outperforming these standard VLAs (see Table 1). Additionally, there are standard inference speed optimizations that can still be implemented in conjunction with the ones we tried. Most prominently, as mentioned in Section 4.3, TenSorRT-LLM [66] claims to enable inference speeds of over 1000 tokens per second for Llama 2, which acts as the language model backbone of our ECoT policy. Given that our ECoT reasoning chains are typically around 250 tokens, this should enable control frequencies on par with non-reasoning VLAs. However, as primary contribution is to show how reasoning aids in robot control policies, we leave such optimizations to future works.
H.2 Alternative Reasoning Steps and Orders
As shown in App. F, our approach can be applied for other reasoning step features or orderings while staying more performant than our baselines (see Table 3). Such alternative reasoning steps can give benefits like better interpretability for casier debugging, as demonstrated with our future gripper position feature in Fig. 13. Some other features include open vocabulary 3D bounding boxes, which could be extracted by systems like FM-OV3D [73] (albeit requiring depth estimates, which our training data is not annotated with, but can be extracted using other foundation models). Likewise, PaliGemma [74] can be used for open-vocabulary semantic segmentation (instead of our current 2D bounding boxes), which also has the benefit of providing such segmentation maps in tokenized textual formats. As more foundation models are released for different semantic tasks, we expect that the type of reasoning steps our model can Iearn via distillation to only increase.
H.3 External Modules
We choose to train our model to perform all reasoning steps end-to-end. However, in principle, some of these reasoning steps can be performed by external modules, which may improve generalization. However, this would require additional compute load the additional models and introduce more system-building complexity. Likewise, we find that current open-source VI.Ms are unable to generate the bulk of the reasoning. While proprictary LLMs like ChatGPT may be able to do so, they are naturally not trained to automatically generate reasonings of the expected format, and thus would require in-context examples or long prompt instructions on formatting (like with Fig. 10), slowing inference speeds. Furthermore, API calls would need to be made for each new image, incurring costs.
Another possibility would be to external modules to give feedback on generated reasoning chains. In practice, this is difficult to do zero-shot with off-the-shelf pre-trained models; most works using VLMs as feedback for learning or inference typically need to fine-tune them for their specific use case [75, 76].
22

---

Annotata the training trajactory with reasching
Specification of the experinetaa setup
You're expert reinfereement learning recearcher, You've trained optinal policy for controlling robot succestully completed
The
epecissed by the instruction: "unfold fron top right to lete". For that purpose, the reboric are exocuted a sequence of actions. Consecutive states that vere visited can be characterized by the folloving features:
1'1 "pythhon
traj trajectory_ feattry_
''stop"
1: "xsop"
"noveu forvardd forvardulwfte
"eovoutorvard.dom
"novey "noveyforens foreard
36: "stop"
Bach entry in that dictionary corresponda to single step on the trajectory and describes the zove that in about to be erecuted.
Scme deacription
The robot operating following environrent. black red stove yellow banana silver pot, blue gray aink.
toy
purpla
countar,
by
tt Your objective
I you
given trajectory with ressoning. sbould be chosen, but importantly what reasoning justittee that action choice. I vant you to be descriptive and include all the ralavant. inforsation available. The reasningg should include the task to complate, the rwriinnn high-lavel staps, the level novenente that chould executed and they required, the prenises that allow inferring the direction move, including locations relevant objects, possible obstacles difticulstes avoid, any other just ficatio.
each step, need know only which action
ARD Begin by describing
Start by giving an everview of the tack. Make it nore corprehencive than the cimple instraction. Include the activity, the objects the robotic ars interacte with, and their relative locations in the environeet.. Then, describe the high- novemnta that vere noxt likely asecuted, based the task that complated and the prinitive rovementa that executed, Then, high-level nowweent vrite a justifteation for why it should be executed. Write an anmer for this part using narkdown and natural language, Be descriptive and highlight all the relerant details, but ensure that your description is consistent the trajectory that executed, specified the features listed
each high-lovol zovenent write
interval stops
novement consiete
the trajectory.teataree dictionary.
ttt List the roasoninge for each stop
Pinally, for each step describe the remsening that allows to deternine the correct action. For each step describe the remsining part of objective, the current progress, objects are still relevant deternining plan, and the plan for the cest stepe, based on the available features. Start the reasoning from high level and gradually add finer features. tast and the executed trajectery. Write the the initial trajectory there should exactly one separate stea anspers. The final distionary should have exactly the same set of intoger keys as the dictionary of fearures provided in the 'trafectory, features' dictionary abore. The reasoning chould be single string that describes the reasoning in natural language and includes the required features.
need you be descriptive and very precise. Easure that the reaseming is consistent with the this part
Python executable dictionary. ketep tressonno....... Do group
every
Bach reaconing string chould have the tolloving form:
Describe the full task that remaing to be cospleted (but only describe what zensins), and place it inside tag <tasko. Describe the complete high level plan coxpleting the remsining task (the 11st of rensining high level steps), place inside tag <plas>.
Describe the high-level step that should be executed now (chosen from the 1ist of high-lovel stops), and place it inside tag <rubtasio.
Describe the chosen high-leve) step should be executed decision, and it should be
which reatures of current enviromeet influence that Flace within tag coubcaskk reason>
Describe the current primitive novenent of the am that needs to be exocuted, and place it inside a tag <nore>. Describe vay the chosen novereat should be executed now and which featurea of the curzent anvirenment influeace that decision. Place it inside tag reove_reason.
#2 Teak aurmary
breakdown of zhat needs dose:
Describe the task.
Deacribe the high-lewel aosements that verw executed, based the completed task and the listed featurea.
Describe the for the solution that allowed the robot cosplete the task succmaafully.
connect action. The reasonngg should be descriptive and precise. You should provids szactly one ressoning string for each step on the trajectory specified by "rrajectory_foatt features'.
each stop the trafectory, describe zeasoning leads to deternining
At the very end the response, write single label FIBISHED indicate that
complete.
Figure 10: Prompt used for Gemini to generate plans, subtasks, and movement labels.
23

---

* Objective
You're espert reinfercemen: conputes the correct action the plan for complating that task, and the subiask that cursently needa to be dooe. I vant you to propare such cossesing, based feedback from user of that robot.
researchez.
policy for controlling stich includes cask that renaino coupleted,
policy
reasoning that
reasming muat have folloving elarmnta::
TASK: the task that remaing to be done.
PLAN: List of high-lwowl
that need to be erecuted.
SUBTASK REASONING: reasmnnng that deternines
REASOMING: reasoning that deternines
current aubtask.
SURTAAK:: the purrent subtaak that, should be axecuted.
HOVE REASOUEN:: ressoingg that deternines the current
Nzite the ansver as python atring. It will be used as on additional isput for the policy, ao keep the foreat exectly as descri bed.
Exssplua
Given the task "Put the tanato inside the pot the lwft burnwe" and feedback "you are too low, nove up", the resson pot left store, position it in the pot. SURTASK RRASONING: The torato grasped. The tonato in already near the pot, but below ths wige. SUBTASK; Posiition the tomata in the pat. HOVE REASONING: The pot. is above current position. Novw the arm up- NOVE: Hovo
the torato, groep traneport
Glien task "place wilver
the ailowr pot on upper right the table" faudlack reasoning sbould be "DASK: The lid neede be placed on silver pot on the upper right part of the scene, PLAM: Firet the ltd, then grip then to pot.
the pat "
the then place the lid the BURITAK REASONTMG: The lid grip,ed, pot."
Given the task "rove the fork to the botton laft aide of the commtrt and faedback "sove down to graap the fork", the PLat:: 1. Move the fork. doanoard
ebould be "DASK: Pick
left the
up
the first object that nedds from the current position of the arm, the arn continuee to move that direction. MOVE: nove down"
reached. SURTASK; 1. Moew the fark. REASCUTM:: fork
cuba" feedback "close", reasoning cbould be "TASK: Renove the cylinder fron the grees cabe and place it on top of the red cube. PLAB: Approach the green cube, close the gripper aroun the cylinder, nove the cylinder tovarde the red cube, open the gripper to place the cylinder, it should close gripper
eylinder
green
place
top
SUBTASK REASDNING: gripper REASUNIDG: cylinder has already been
gripper grab SUITAAK:
closing. HOVE: Close gripper'
Given the
up tovel" and feedback right', Grasp touel,
chould "TASK: Pick tovel. PLAM: SUBTASK:
topol. NOVE REASUKING: The topel is to the right, so the are should nove right. NOVE; nove right"
The task
The policy generated the tollowing reasoning: "TASK: Pat the mushroon in the white cup. PLAM: Hove to the muchroom, pick zove to the vith. SUBTASE: Nove the NOVE REMSONING; 176,
the aushroon in the cup. SUBTAsK REASOKISQ: The sustroox is picked up, and the cup is the next belou
MovE: Nove [13, 261,
cup
table [10, 21, 209, 260]*
Given the "puz mushreon the chite cup' feedback "no, cup actually frout of gripper", reasoning?
Figure 11: Prompt used for ChatGPT during human intervention experiments
[letmostiightmost] container
measuring tape
pot
Put corn the pot
the purple object in the llettmssrighmmosti container
tutthe watermelon the
Put the edible object on the towe!!
Figure 12: Example starting scenes for each of the cluttered evaluation tasks.

---

TABK Put the block bowl
TASK Mown the hananx. Move barans, gropp the banana, move the plot.
PLAN Mown track boot sink. Drop black bowl The datk.
SUBITAK
The block bowl is the fint oeject interct
neads more left a Move
sydo baxen
The bend to
MOVE: The robot noads to mows
TASIC
spange
PLAME
red bowl Reliass curos.
towel put down sporgss.
grappet canot and needs Move
SUBTARS, The
mosil to
table, and raee reached
sponge.
MONS: The
MOUE: sponge balcer
nedda to move right roach Move right
Figure 13: Our OXE fine-tuned ECoT model can generate reasonings for non- Non-WidowX robots too, despite
never having seen reasoning annotations for said embodiment.
25


Reviews:
[{"review_id": "zOJxGbbtVP", "reviewer": "Reviewer_DLrg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work builds upon OpenVLA and incorporates chain-of-thought reasoning while grounding it in sensory observations and the robot state. Since OpenVLA builds on relatively small open-source VLMs, its chain-of-thought reasoning capability does not match that of large closed LLM models. Furthermore, this step-by-step reasoning should be grounded in the robotics task. Thus, the authors propose training a VLA policy (using the OpenVLA architecture) to output CoTs and actions given input instructions and observations, ensuring that  both are firmly grounded in the agent’s environment.", "summary_of_recommendation": "Interesting approach for incorporating embodied chain of thought reasoning to OpenVLA. It is a useful contribution towards improving the capabilities of VLA models.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "More details on the training objective and prediction pipeline including some of the clarification questions mentioned above could improve the clarity of the paper.\n\nTo evaluate if the accelerated CoT inference approaches mentioned in the paper are sufficient, the authors should consider providing comparisons of inference times against the baselines considered.\n\nIt would be useful to include the set of objects considered for in-distribution and OOD tasks in the appendix.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Gc9KaSvKx", "reviewer": "Reviewer_LC5L", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper looked into a pretty straightforward (yet not investigated) idea, which is on applying the chain of thought (CoT) approach to vision-language-action models (VLAs). It has been evident that the CoT technique is useful in large language models (LLMs) for different types of reasoning tasks. Recent advances on VLAs, such as the series of RT research, shows that pretrained foundation models are able to directly  perform robot control. The proposed approach is called ECoT (embodied chain of thought reasoning) for computing VLA policies for robots. OpenVLA and RT-2-X were used as baselines for comparisons. Results showed that ECoT performed better than the two baselines on real robots and human feedback could further improve the performance.", "summary_of_recommendation": "Nice paper that described a novel idea that can potentially produce good impact to the community. The experiments were well performed and the results were convincing.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Was the collected dataset used for fine tuning the VLAs? What if we use the training data as examples for few-shot inference? \n\nAre there robot tasks that the current pipeline is inapplicable to? \n\nHow is the performance sensitive to the current pipeline? For instance, what if we flip the order of the first two steps? What if the 2D bounding boxes are replaced with 3D boxes?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yOnWpbdJRN", "reviewer": "Reviewer_Nt3h", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel robotic control pipeline by including chain-of-thought in the end2end VLA backbone. The CoT involves language subtask, boudning box and other combination, and the experimental results is promising.", "summary_of_recommendation": "The paper proposes a novel robotic control pipeline by including chain-of-thought in the end2end VLA backbone. The CoT involves language subtask, boudning box and other combination, and the experimental results is promising.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See weakness.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ### Strengths

- Re: Novelty: All reviewers agree that the paper should be considered a significant advance, as it introduces a novel robotic control pipeline that integrates chain-of-thought (CoT) reasoning into the end-to-end vision-language-action (VLA) model. R-Nt3h mentions that the CoT framework includes diverse components such as natural language subtasks, motion planning, and object localization, which they further suggest are appropriate and comprehensive for the task.
- Re: Evaluation: R-DLrg highlights that the approach is compared against strong and relevant baselines, and includes important ablations (e.g., Naïve CoT) that highlight the benefits of embodied reasoning. R-DLrg and R-LC5L note that the proposed ECoT approach outperforms existing models (OpenVLA and RT-2-X) in most generalization tasks and real robot evaluations. R-Nt3h suggests that the training dataset used is of high quality and is well-collected.
- Re: Clarity: R-LC5L and R-Nt3h suggest that the paper is generally well-organized, clear, and the experiments are well-performed, making the results convincing.

### Weaknesses

- Re: Method: R-DLrg cites dataset augmentation issues, wherein the augmentation of the Bridge v2 dataset is seen as tedious and unreliable, with high dependence on pre-trained models and no measures for quality assurance. R-LC5L sees the interactive correction mechanism as somewhat disconnected from the main focus of the paper, which they assert is the CoT strategy. R-DLrg expressed concern over the fact that the current policy is mostly open-loop, with no replanning framework for handling failures or incorrect sub-tasks — limiting the policy’s ability to adapt and recover from mistakes. R-DLrg further suggests that the learned policy is very sensitive to camera viewpoints, which may affect its generalizability, and that the speed of execution remains a limiting factor, despite efforts to improve inference time. 
- Re: Evaluation: R-Nt3h points out that the absence of experiments in a simulator for fair comparison with existing works is noted as a limitation. R-DLrg expressed concern over a lack of challenging tasks, asserting that the experimental results are limited to a small set of tasks and that the paper does not address performance in more complex scenarios such as cluttered environments. R-Nt3h calls for more ablation studies on key components of the CoT and their combinations to better understand their impact.
- Re: Clarity: R-LC5L expressed concern over the lack of clarity on how sensitive the performance is to different pipeline components, such as the impact of replacing 2D bounding boxes with 3D boxes.

### Post-rebuttal Meta Review Statement

I applaud the reviewers and authors for engaging in a productive discussion that resulted in several improvements to the paper, especially as it relates to the addition of new experimental evaluations (in response to R-Nt3h and R-DLrg, enabling the latter to raise their score) and sensitivity analysis of the policy to alternative reasoning steps and orderings (in response to R-LC5L).

I recommend for Accept as Oral, given the initial reviews and the extent to which the reviewer feedback was satisfied. 

Still, I encourage the authors to follow through with all the promised changes and to incorporate all additional reviewer requests, to improve the paper even further.
Decision: Accept