Robotic Control via Embodied Chain-of-Thought Reasoning

Authors: michał  zawalski
william  chen
karl  pertsch
oier  mees
chelsea  finn
sergey  levine
u c berkeley
δ  δθ
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:41:44

Abstract: A key limitation of learned robot control policies is their inability to generalize outside their training data. 
Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models as the backbone of learned robot policies can substantially improve their robustness and generalization ability. Yet, one of the most exciting capabilities of large vision-language models in other domains is their ability to reason iteratively through complex problems. Can that same capability be brought into robotics to allow policies to improve performance by reasoning about a given task before acting? Naive use of "chain-of-thought" (CoT) style prompting is significantly less effective with standard VLAs because of the relatively simple training examples that are available to them. Additionally, purely semantic reasoning about sub-tasks, as is common in regular CoT, is insufficient for robot policies that need to ground their reasoning in sensory observations and the robot state. To this end, we introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features like object bounding boxes and end effector positions, before predicting the robot action. We design a scalable pipeline for generating synthetic training data for ECoT on large robot datasets. We demonstrate, that ECoT increases the absolute success rate of OpenVLA, the current strongest open-source VLA policy, by 28\% across challenging generalization tasks, without any additional robot training data. Additionally, ECoT makes it easier for humans to interpret a policy's failures and correct its behavior using natural language.

Main paper: 
 introduction: 
 Robotic policies trained end-to-end for outputting raw low-level actions in response to observations provide a powerful and appealing learning-based approach to robotic control, obviating the need for complex sensing and control stacks [1, 2, 3] . However, this kind of "reactive" control is not necessarily the best choice in settings that demand broad generalization, such as novel scenes or interactions with unfamiliar objects. In such situations, a person might need to think more carefully: if they were asked to put fruit on one plate and vegetables on another, they might first try to figure out which objects are fruits and which are vegetables, rather than simply performing a learned skill from "muscle memory." Likewise, we would like our robotic policies to both perform well-practiced end-to-end control and to "reason through" novel situations before grounding their commands into actions. Such reasoning might include identifying and locating task-relevant objects, producing a plan to accomplish a task, and translating sub-tasks and observations into movements. Vision-language-action models (VLAs) -pre-trained vision-language models (VLMs) fine-tuned to produce robot actions -have gained popularity as an approach for leveraging the diversity of Internet data captured within large foundation models [4] in a simple and scalable policy learning Figure 1 : We propose embodied chain-of-thought reasoning for vision-language-action models (VLAs): prior VLAs directly predict the next robot action given the task (left), we instead train VLA policies to "think step-bystep" (right). Crucially, reasoning through low-level embodied features like object bounding boxes and gripper positions in addition to purely textual CoT elements like sub-task plans, forces the policy to "think carefully" and "look carefully" before acting. This approach increases the absolute success rate of state-of-the-art OpenVLA policies [7] by 28% in challenging generalization tasks.We thus hypothesize that we can similarly boost VLA performance by training them to textually reason about their plan, environment, and motions, thereby allowing them to produce more accurate and robust robot actions. However, simply applying the CoT techniques from language to the robotics domain has several challenges. For one, current VLAs build on relatively small, open-source VLMs that cannot match closed models' ability to perform meaningful reasoning when simply prompted to think step-by-step [8] . Additionally, the most common LLM CoT reasoning task, breaking tasks into sub-tasks, is helpful but insufficient for reasoning about robotic tasks. The VLA policy needs to ground its plans and reasoning in its observations of the environment and robot state. Only then can the reasoning direct the agent's attention toward fine-grained spatial or semantic perceptual features that are key to robot manipulation tasks. Put simply, we need VLAs to not only "think carefully", but also "look carefully." To this end, we introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLA policies. In contrast to prior VLAs, embodied chain-ofthought policies perform multiple steps of textual reasoning before predicting the next robot action (see Fig. 1 , right). In contrast to existing CoT reasoning approaches for LLMs, they interleave semantic-level reasoning about sub-tasks with "embodied" reasoning tasks that require the policy to pay attention to its multi-modal inputs, from predicting bounding boxes of objects in the scene to reasoning about low-level movement primitives that need to be executed based on the current robot state. To enable the relatively weak LLM backbones of open-source VLAs to perform such reasoning effectively, we design a scalable pipeline for synthetically generating embodied CoT training data for large robot datasets. Concretely, we use powerful pre-trained foundation models to generate the reasoning supervision for our policies. Our experiments show that by training state-of-the-art VLAs to reason before predicting actions, we can substantially boost their ability to perform challenging generalization tasks. Our approach increases the absolute success rate of OpenVLA [7] , the current best-performing open-source VLA, by 28% across a suite of robot manipulation tasks that involve generalization to new objects, scenes, viewpoints, and instructions without any extra robot training data. Beyond raw performance improvements, we find that embodied CoT makes policy failures more interpretable and allows humans to easily correct its behavior by modifying faulty reasoning chains via natural language feedback. 
related work: 
 Scaling robot learning. A long-standing goal of robot learning is to train policies that can generalize to a wide range of unstructured real-world environments. Towards this goal, recent works have explored training "generalist robot policies" [10, 11, 12, 13, 14, 15, 16, 17] on diverse robot datasets [18, 19, 10, 20, 21, 22, 23, 24, 13, 14, 25, 15, 26, 6] . As a result of their diverse robot training datasets, many of these policies can be prompted in natural language to solve various manipulation tasks, and some generalist policies can even control multiple robot embodiments [16, 27, 6] . Importantly, these works demonstrate that training robot policies on large and diverse datasets is a promising approach towards improving policy robustness and generalization ability. Vision-language models for robot generalization. In a push towards generalization far beyond what is observed in robot datasets, the recent development of strong, open-source vision-language models that learn visuo-linguistic representations [28, 29] , generate images from text [30] , or generate text in response to images and prompts [31, 32, 33, 34, 35] have resulted in a large number of works that explore the integration of such models into robot learning pipelines, e.g., to generate goals [36] , to provide reward signals [37, 38, 39] , or to learn visual state representations [40, 41, 42] . Since collection of the aforementioned large-scale robot datasets is challenging, using models pre-trained on Internet-scale data is an appealing alternate path towards robust robot policies that can act in a variety of unstructured real-world environments. Most relevant to our work are recent approaches for integrating pre-trained vision-language models into learned robot policies. While some works use strong structural priors in their policies to enable this integration [43, 44, 45] , vision-languageaction models (VLAs) have recently been proposed as a simple yet scalable alternative [5, 6, 7] , achieving state-of-the-art performance for generalist robot policies [7] and showing impressive levels of generalization to new objects and scenes. However, existing VLAs do not sufficiently leverage some of the most appealing properties of the underlying language and vision-language models, specifically their ability to reason through the steps required to solve a given task. Reasoning for language and control. Such step-by-step reasoning is a key ingredient for the ability of large language models (LLMs) to solve a wide range of complex tasks. Prompting LLMs (directly [46] or with in-context examples [8] ) to "think step-by-step" about the problem before formulating an answer can significantly improve their performance, with such chain-of-thought reasoning techniques becoming standard practice in language modeling and (vision-)language model training [9, 47] . A number of works have explored similar techniques in the context of high-level task planning for robotics [48, 49, 50, 51, 52, 53, 54, 55, 56] . These approaches use pre-trained or fine-tuned LLMs to decompose tasks into high-level sub-tasks, but rely on pre-trained low-level policies to execute them. However, we argue that (1) careful reasoning can be beneficial for both highlevel sub-task reasoning and during low-level control and (2) all such levels of reasoning should be strongly grounded in visual observations of the scene and the agent's state. Thus, in contrast to these prior works and language-only CoT, our approach trains a VLA policy to autoregressively generate CoTs (for high-and low-level reasoning) and actions given input instructions and observations, ensuring that both are firmly grounded in the agent's environment. We empirically confirm that such a formulation is critical to effectively leveraging (V)LM reasoning capabilities for control. Our work leverages VLAs as the backbone for our embodied chain-of-thought policies. VLAs use a simple policy learning recipe: starting from a pre-trained vision-language model, they directly finetune the model to autoregressively predict the next robot action a given the current image observation I and task instruction T . To enable this, the continuous robot actions are typically converted to discrete action tokens T a in the vocabulary of the vision-language model via a per-dimension action discretization scheme that assigns each continuous value to one of 256 bins [5, 7] . In this work, we use the recently released OpenVLA model [7] (see Fig. 2 ), since it achieves stateof-the-art performance and is fully open-source. The model builds on the Prismatic VLM [35] and consists of a fused visual encoder that combines pre-trained SigLIP [57] and/or DinoV2 [58] features and a Llama 2 7B [59] LLM backbone. During training, input images are encoded into visual token embeddings using the pre-trained vision encoders, the task instruction is mapped to task tokens using Llama 2's text tokenizer, and the model is trained to map these inputs to the target action tokens with The watermelon is behind the robot, so it needs to move backwardWatermelon [126, 146, 141, 125] , Towel [20, 59, 218, 198] , Spoon [114, 93, 141, 125] … into the mapping from inputs to robot actions. Green: "standard" linguistic chain-of-thought steps that break a given instruction into the required sub-tasks. Purple: Embodied chain-of-thought steps that require grounding the policy's reasoning in the scene and robot state. Our experiments show that these grounded reasoning steps are key to improving policy performance with chain-of-thought reasoning.the standard autoregressive next token prediction objective. We likewise use this objective, but also insert reasoning text tokens in the training sequences such that the policy learns to produce the ECoT reasoning followed by the action in response to the observation and instruction. Next, we will discuss how we can improve upon this conventional VLA training recipe by enabling the VLA to reason through the task at hand before deciding which action to take.4 Embodied Chain-of-Thought Reasoning for Visuomotor PoliciesWe now discuss our approach for training VLAs to perform embodied chain-of-thought reasoning about plans, sub-tasks, motions, and visual features before predicting the next robot action (see Fig. 1 ). Unlike many proprietary large language models, the relatively small LLM backbones used in current VLAs struggle to perform involved reasoning when simply prompted to think step-by-step [8] . Instead, we propose to explicitly train VLA models to perform embodied CoT reasoning. Concretely, we label data from existing robot datasets post-hoc with reasoning chains filled with features extracted from various pre-trained models and use the resulting dataset of observation-reasoning-action tuples for training. In practice, we ensure that all elements of the generated reasoning data can be represented as strings, such that we can use the Llama 2 text tokenizer to translate them into reasoning tokens. Then, we train the VLA to autoregressively predict these tokens, directly followed by action tokens.While this approach is conceptually simple, its implementation requires answering multiple key questions: (1) Which reasoning steps are suitable for guiding policies in solving embodied robot manipulation tasks (Fig. 3 )? (2) How can we generate training data for these reasoning steps at scale on existing robot datasets (Section 4.2)? Another practical consideration arises after training, while using ECoT policies for robot control: carefully reasoning through each action can significantly slow down policy inference. We discuss solutions to these problems in the following sections. 
designing embodied chain-of-thought reasoning steps: 
 Our goals when designing the steps of our embodied chain-of-thought reasoning chains are twofold: encourage the model to (A) reason through the required high-level steps of the task at hand and determine which step needs to be executed next, and (B) increasingly ground this reasoning in lower-level features of the scene and robot state before predicting the robot action.We visualize the ECoT reasoning steps that we train the VLA to perform for an example task in Fig. 3 . From left to right, the model is trained to first rephrase the task instruction (TASK) and predict a high-level plan of steps for achieving the instructed task (PLAN). Next, it reasons through which of the sub-tasks should be executed at the present step (SUBTASK), a task which requires understanding the current state of the scene and robot. Then, the model predicts an even lower-level language command like "move left" or "move up" (MOVE) that is closely related to the low-level actions the robot needs to execute. Finally, we ask the model to predict precise, spatially grounded features that describe the scene and thus force the model to pay close attention to all elements of the input image-specifically, the pixel position of the robot end effector (GRIPPER) and the names and bounding box pixel coordinates of all objects in the scene (OBJECTS).While we believe that our choice of reasoning tasks and their order is well-aligned with our intuition for a sensible step-by-step solution to the task, we by no means exhaustively explored all possible This scene has a watermelon, spoon, towel, and mushroom. We use a Prismatic VLM [35] to create a scene description (1), and Grounding Dino [29] to detect bounding boxes for all objects (2). We then compute templated motion primitives from low-level robot states (3) and the robot gripper position using OWLv2 [60] and SAM [61] (4). Finally, we use Gemini [62] to create synthetic reasoning chains (5).reasoning tasks. Testing alternative tasks and task orderings, and finding ways to automatically determine sensible reasoning chains are important directions for future work.While human annotations are the gold standard, it is impractical to get them for large robot learning datasets [6] , which consist of millions of transitions. Thus, we instead propose to leverage pre-trained vision and/or language foundation models to automatically generate ECoT training data, akin to synthetic data generation in NLP [63] . We give an overview of our data generation pipeline in Fig. 4 . For a given image-instruction pair, we first prompt a Prismatic-7B VLM [35] to describe the scene. We then concatenate the instruction and this description and input them into grounding DINO [29] , an open-vocabulary object detector, to detect all relevant objects and their bounding boxes with the corresponding language snippet labels from the input text. See App. A for examples and details. Next, we generate the per-step motion primitives in MOVE (e.g., "move left"). Following Belkhale et al. [64] , we use robot proprioception to determine the movement for the next 4 time steps (assuming a fixed camera), and translate this into one of 729 templated movement primitives (see App. B). We use OWLv2 [60] and SAM [61] to detect 2D end effector positions in the training images (GRIPPER) paired with 3D positions extracted from the robot state to fit a robust estimate of the projection matrix using RANSAC [65] . We then use the 2D projections of the robot end-effector position for our training. This is repeated for each trajectory, eliminating the need to assume fixed camera parameters. To generate the final reasoning chain, we feed each episode's task instruction, scene description, and per-step movement primitives into Gemini 1.0 [62] and prompt it to produce both a high-level plan of sub-tasks in accordance with the task instruction and observed movement primitives and the current sub-task for each step. We also ask it to briefly explain the primitive movement and chosen sub-task in each step, which we include in the ECoT training data. We run our data generation pipeline on the complete Bridge v2 dataset [13] , with more than 2.5M transitions, over the course of 7 days.Inference speed is a key challenge for ECoT policies. They needs to predict around 350 reasoning tokens per timestep, compared to OpenVLA's 7 action tokens. We explore a simple solution for speeding up inference: as our policy learns next token prediction, it can continue arbitrary input reasoning prefixes; we thus can keep parts of the reasoning chain like the high-level plan or the current sub-task fixed for multiple steps. Encoding previously predicted tokens is much faster for Transformer-based policies than generating them. We compare two such strategies: (1) synchronous execution, where we predict the high-level reasoning every N steps, and (2) asynchronous execution, in which one ECoT policy instance continually updates the high-level reasoning chains, while a second policy instance uses the most recent reasoning chain to predict low-level reasoning steps and actions. We report the performance and inference speed for all approaches in Section 5.5. Note that these speed-ups are orthogonal to widely used approaches for improving throughput of LLMs. We explore one such approach -compiling with TensorRT-LLM [66] -in App. F.  
experiments: 
 In this section, we investigate the effectiveness of ECoT for robot control across a range of challenging manipulation tasks. We answer the following questions: (1) Does embodied chain-of-thought reasoning improve the performance of VLA policies (Section 5.2)? ( 2) Does embodied chain-ofthought reasoning make it easier to interpret and correct policy failures (Sections 5.3 and 5.4)? ( 3) How can we optimize the runtime efficiency of policies with embodied CoT reasoning (Section 5.5)?Robot setup and training data. We evaluate on a 6-DoF WidowX robot arm, a commonly-used setup for evaluating learned robot policies [16, 7] . Given a single 3 rd person camera and natural language instruction, the policy predicts end-effector velocity actions to control the robot. Bridge v2 [13] provides a diverse dataset of 60k teleoperated demonstrations. We apply our pipeline for synthetic generation of chain-of-thought data (Section 4.2) on this dataset to obtain our training dataset. Evaluation tasks. We design a suite of challenging evaluation tasks that focus on testing the policies' generalization ability along multiple axes: processing spatial relations, interacting with unseen objects, and performing unseen instructions. All policies are evaluated on the same real-world setups to control for camera angle, lighting, and background. We perform 314 total trials per approach.Comparisons. We compare our policy (ECoT) to state-of-the-art VLA policies, namely Open-VLA [7] , the same model our approach is built upon, but trained without chain-of-thought reasoning, and RT-2-X [6], a 55B parameter closed VLA policy. To ensure fair comparison, we train the OpenVLA policy on the same dataset we use for training our approach (the Bridge V2 data [13] ), denoted as OpenVLA (Bridge). For RT-2-X we cannot control the data distribution in the same way since the model is closed, but it was trained on Bridge V2 data and additional datasets from the Open X-Embodiment dataset [6] . Thus, it was trained on more data than our approach. We also compare against Octo [16] , which is also trained on that dataset, but was not fine-tuned from a VLM (i.e., it is not a VLA). Finally, we compare to Naïve CoT, a version of our model that only uses non-embodied CoT reasoning about sub-tasks akin to conventional CoT reasoning in language modeling (see Fig. 3 ). This comparison will test the importance of using embodied reasoning for VLA policies.We report performance of all approaches on our evaluation set in Table 1 . We see that while OpenVLA (Bridge) achieves high performance on in-distribution tasks, it struggles on the hard generalization cases we test. RT-2-X performs better than vanilla OpenVLA (Bridge), potentially due to the larger robot pre-training dataset (note again that OpenVLA and our approach are only trained on the Bridge dataset) and the fact that it co-trains the policy with Internet-scale vision-language data and robot data, while all other approaches only use robot data during fine-tuning. Importantly, we find that our ECoT policy substantially outperforms OpenVLA (Bridge) across all generalization evaluations but one. This is notable, since both policies have the exact same VLM base Left: high-level reasoning and low-level object segmentations are correct, leading to a successful rollout. Middle: the command is correctly rephrased to refer to specific objects (i.e., "the leftmost object" is identified as the pink toy). Right: the hammer is incorrectly identified as a screwdriver, causing the robot to take inappropriate actions. model and use the same robot data for fine-tuning. The only difference is in the use of CoT reasoning by our approach. Curiously, our ECoT model even surpasses the performance of RT-2-X in the tested tasks, even though RT-2-X is trained on 10 additional robot datasets and uses a neural network that is 7x larger (55B vs. 7B). Finally, the results in Table 1 show that including embodied reasoning about visual inputs and the low-level robot state significantly boosts performance over the "Naïve CoT" ablation of our approach, which only reasons about high-level linguistic features like sub-task plans.We visualize qualitative examples of our model's reasoning in Fig. 5 . The left two examples show that the model successfully breaks down the task into a sequence of sub-tasks and then crucially grounds those sub-tasks in the scene by predicting the relevant bounding boxes and gripper position of the robot, before deciding on the next move and concrete low-level robot action. We visualize more chain-of-thought examples from our evaluation tasks in Fig. 8 .In addition to improving performance, such embodied chain-of-thought reasoning provides a tool for operators to better understand the decisions the policy takes. By inspecting and visualizing the model's reasoning steps, we can discover potential mistakes in the reasoning chain that led to policy failure downstream. For instance, in Fig. 5 (right), the ECoT policy failed to solve the task pick up the screwdriver. Inspecting the reasoning chain, we can see that the hammer is incorrectly identified as a screwdriver, causing the robot to reach for that instead. Note that inspecting reasoning chains is not "bullet-proof" for interpreting end-to-end policy failures: the model could predict a particular plan and then still deviate from it when choosing the final action. However, in practice we find that reasoning chains often correlate strongly with the executed actions. We provide more examples for diagnosing policy failures via its reasoning chains in Fig. 8 .Training a policy to reason through a task step-by-step provides a powerful mechanism for humans to interact with the policy and correct its behavior: instead of needing teleoperation equipment to provide direct robot action feedback like in DAgger [67] , users can simply modify its reasoning chains via natural language feedback to correct the policy's behavior. Prior work used carefully designed architectures explicitly trained to support such correction via language [68, 69] , but we test whether similar capabilities emerge naturally by training VLA policies to perform ECoT reasoning.We rerun evaluations of our ECoT policy on some of the most challenging tasks from Table 1 (put mushroom in cup, pick up out-of-distribution object, and pick up non-yellow object), in which our policy without human intervention achieved an average success rate of only 32%. We allow a human operator to interrupt policy execution once over the course of the episode and provide natural Left: given a human intervention in natural language, we use ChatGPT to correct our model's reasoning chains.Right: our embodied chain-of-thought policy can benefit from a human language intervention most, increasing success rate by 48% on our most challenging evaluation tasks.language feedback (e.g. "no, the screwdriver is in the back right corner", "release the mushroom now!", or "the cup is tall") (see Fig. 6 ). Then, we use ChatGPT to adapt our model's reasoning chain based on the language feedback, prompting it to produce a corrected reasoning chain (see Fig. 11 for the exact prompt used). Finally, we feed this corrected chain back into our policy and continue execution, holding the corrected reasoning chain fixed for 5 steps.The results in Fig. 6 (right) show that our ECoT policy can make effective use of the human language intervention, increasing its success rate by 48%. In contrast, we evaluate the vanilla OpenVLA policy and RT-2-X in the same way, providing each with a single human language correction per rollout, but find that neither of them can benefit from the human intervention to the same degree (for both we also use ChatGPT to incorporate the intervention into the original task instruction to allow for fair comparison). This is because standard VLAs that directly map from images and instructions to actions do not expose any part of their internal decision-making process. Not only does this make "debugging" incorrect behaviors difficult, it also only permits changing the prompt for interventions. We compare approaches for accelerating ECoT policy inference (see Section 4.3) to naïvely running the full generation every step of execution in Table 2 . We also report the speed-up that is achieved by both proposed approaches from Section 4.3 vs. naïve execution. Both approaches achieve inference speed improvements while at least matching performance, with asynchronous execution achieving the largest speed-up at the cost of doubling the compute required at inference time (since two policy instances are running in parallel). We use the 5-step freeze approach for the main results presented in Table 1 , since it provides the best performance-speed tradeoff. We used a small task subset (put mushroom in pot, move mushroom to detergent/measuring tape, and put the left/right object on the middle). 
discussion and limitations: 
 In this work, we demonstrate that training VLA policies to perform chain-of-thought reasoning can substantially increase their performance without the need to collect additional robot training data. Instead of simply applying the CoT recipe from language modeling, our experiments underline the importance of adding reasoning steps that are strongly grounded in the task, scene, and robot state. While our results are encouraging, our approach has several limitations. First, our model does not adapt the structure of its reasoning chains to the task at hand; it always performs all steps of reasoning in the fixed order we chose. A more effective strategy may involve executing only a subset of reasoning steps based on the robot and scene state, and future work can explore directly optimizing the model to pick the best reasoning steps. Additionally, scaling the ECoT training to a larger subset of the OXE dataset [6] will improve transfer of ECoT capabilities to more robots.We provide example scene descriptions provided by Prismatic VLM and bounding boxes provided by Grounding DINO in Fig. 7 . We filter the predictions based on the provided confidence score, only keeping detections with a box-and text-confidence larger than 0.3 and 0.2 respectively to use for the OBJECT features.A robot is holding a carrot above a green plate, with a silver pot, red pepper, and a yellow bowl on the counter A wooden table holds a blue bottle, a grey stuffed animal, a white stuffed animal, and a blue towel, with a box and a black arm behind them A green pickle is on the counter, a silver pot is on the stove burner, and a white mushroom is in a white bowl. The stove burner is below the pot, and the pickle is near the counter A silver pot is in the sink, a white plate is on the stove, and a white mushroom is on the plate. The sink is to the left of the stove, and the pot is in the sinkThe scene has a table with a blue chair, a yellow cloth on the table, and a black device.The cloth is to be moved from the left side of the table to the center.A robot is putting a carrot on a cutting board, which is on the counter. There is a green cutting board, a red pepper, a yellow pepper, and a black pot nearby. The counter is white with silver knobs and a sink. We now provide all the prompts used for data generation and policy language conditioning. For using generating scene descriptions with Prismatic (step 1 in Fig. 4 ), we use the prompt: "Briefly describe the things in this scene and their spatial relations to each other." We prepend "The robot task is: [TASK] ." if the given demonstration trajectory contains a corresponding task instruction (where we ensure that said instruction contains at least one space character to remove noisy instructions). We provide the prompt for Gemini data labeling (step 5 in Fig. 4 ) in Fig. 10 . We re-run generation if Gemini fails to produce reasonings of the correct format.The prompts used for our language-conditioned policies are provided in Fig. 9 , along with example starting scenes for the associated tasks. For the OpenVLA-based policies, said prompts are inserted into the template provided by the original authors [7] : "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: What action should the robot take to [PROMPT]? ASSISTANT:". The agent then generates reasoning text (if trained to do so) and an action. We provide the prompt used for human interventions with ChatGPT in Fig. 11 .We provide additional evaluations of how our policy compares against OpenVLA in settings with significant clutter (9 or more objects per scene). We evaluate two rollouts for eight tasks in the four categories used in Table 1 : in-distribution (put mushroom on cloth, put corn in pot), spatial relations (put mushroom in [leftmost / rightmost] container, put mushroom in [leftmost / rightmost container]), out-of-distribution objects (move mushroom to measuring tape, place watermelon on pan), and out-of-distribution instructions (put edible object on towel, put animal in pot). Note that some tasks are not from the original tasks in Table 1 , which we chose to make them more amenable to evaluation in cluttered scenes. We show example starting arrangements for each task in Fig. 12 . We find that our ECoT policy achieves 75% success rate while OpenVLA only achieves 25%, indicating that reasoning may aid in robustifying against clutter. Qualitatively, we note that many failure cases are exacerbated for OpenVLA in the presence of clutter, while our ECoT policy still enjoys the same benefits noted in the default task suite. Namely, OpenVLA often gets distracted by commonly-represented clutter objects (like the mushroom) while failing to reason about object semantics, which our approach deals with well (e.g., successfully identifying toy foods and stuffed animals when queried to pick the edible objects and animals respectively, even though such wordings are rare or non-existent in our training data).We provide visualizations of example representative starting setups for all tasks in Fig. 9 . The tasks involve the following objects: We frequently use the mushroom and light blue towel as distractors, as they are well-represented in our training data, and so non-reasoning policies often erroneously pick them up or place things on them when instructed otherwise, especially when dealing with the less common or out-of-distribution objects (measuring tape, detergent, watermelon, hammer, screwdriver, white cup).Can we improve inference speeds by compiling the language model backbone? As our model shares an architecture with OpenVLA [7] , which likewise is fine-tuned from a Prismatic VLM [35] , the policy itself has a vision and language model backbone. The latter is where much of the inference time is spent: it is a fine-tuned Llama 2 [59] model that has to generate ∼ 300 chain-of-thought reasoning language and action tokens in response to 256 soft prompt image embedding tokens. However, since it does use this popular language model architecture, our policy can benefit from the numerous recent developments in improving LLM inference speeds.We show this by compiling our ECoT VLA's language model backbone with TensorRT-LLM [66] . Using FP8 quantization [70] , we were able to lower the inference time of generating a full reasoning chain followed by actions from ∼ 5 seconds to ∼ 1 second (or ∼ 0.8 seconds with freezing, from ∼ 2 seconds). We note that further optimizations (both with TensorRT-LLM or with other techniques, like speculative decoding [71] ) are possible and that the policy can be made smoother with techniques like action chunking [2] , but we leave these approaches to future works. Can we improve speed and interpretability of the ECoT reasoning? We test two modifications to the structure of our reasoning chains. First, we move the bounding box generations earlier in the chain, right after the plan. This way, we can keep the bounding boxes fixed in our N-step inference (see Section 4.3. Since the bounding box generation represents a significant fraction of the predicted tokens, this change can speed up ECoT inference by 30 -50% in our experiments. Secondly, we train the model to autoregressively predict the next four future gripper positions, in addition to the current one. Not only does this gives operators a rough visualization of what the ECoT policy expects its motion to be in the future, but it serves as an (albeit imperfect) proxy indicating how our policy would behave. This will be important for the following experiments on other robot embodiments, for which we do not have the ability to run real world rollouts. We evaluate this policy (and all subsequent ones) on a large subset of tasks on the out-of-distribution view station, totaling 106 trials per policy. We find that, while this frozen bounding box policy does perform worse than our base ECoT model, it nonetheless outperforms all baselines (Octo, OpenVLA (Bridge), and RT-2-X), as shown in Table 3 . Thus, due to its relative higher speed and ability to visualize rollouts, we adopt this structure for all subsequent experiments. Does co-training with vision-language data help? During VLA fine-tuning, both ECoT and OpenVLA lose the base VLM's ability to respond conversationally to natural language questions. This can be remedied by co-training the VLAs with vision-language training data in addition to robot action data. Prior work found that such co-training can improve VLA capabilities [5] . We test performance of an ECoT model co-trained with robot data and the vision-language training dataset of the base Prismatic VLM [35] at a 3:1 ratio. Qualitatively, we find that the co-trained model indeed retains its ability to answer questions in chat format in addition to robot control. We compare the performance of this model with our base ECoT model across a large subset of the robot control tasks in Table 1 (in-distribution view), totaling 106 trials per model. The results in Table 3 suggest that co-training does not lead to measurable performance improvements on our evaluation tasks. Anecdotally, we find that the co-trained model can more reliably recognize celebrities and improve performance on tasks like "bring coke can to Taylor Swift" (4/4 successes vs. 0/4 for our base ECoT). [7] , which was trained on a mix of 27 robot datasets. We continue training the released checkpoint on this mix, but replace the original BridgeData V2 dataset with our generated ECoT dataset. As a result, approximately 13% of the training data is ECoT data. We make two key findings. First, fine-tuning a pre-trained VLA to perform ECoT reasoning is substantially faster than training an ECoT VLA from the base VLM. We observe that within 20k training steps the fine-tuned model nearly matches the performance of our original ECoT model trained for 80k steps (Table 3 ). Qualitatively, we even observe comparable performance after only 2500 steps. This represents a 4x and 30x reduction in required compute respectively.Secondly, we find that the fine-tuned model can perform ECoT reasoning on other robot embodiments than it has been trained for, simply by prompting it with the beginning of a ECoT sequence ("TASK:") (Fig. 13 ). It recognizes robot grippers, objects and their positions, and predicting future gripper movements, despite the large differences in robot appearance, scene layout and camera setup. This result is surprising, since we only provided ECoT training data for a single robot embodiment: the WidowX robot in the BridgeData V2 dataset. We hypothesize that the VLM pre-training enables the model to generalize the concepts of robot end-effector position and movement, and object idendity and positions between robots and scenes. We also tried rolling out the fine-tuned ECoT model in the SIMPLER real-to-sim environments of [72] on the Google Robot tasks, while prompting for ECoT prediction as described above. However, we found that the ECoT model suffered from the real-to-sim domain gap of the SIMPLER environments, producing more faulty reasoning chains than on real Google robot images, and thus not improving overall performance compared to an OpenVLA baseline without ECoT.Along with Fig. 5 , we provide further qualitative examples of successful and failed ECoT reasonings in Fig. 8 . We find that most failures are due to some kind of semantic scene misunderstanding, the most common form being incorrect bounding box labels (as shown in the rightmost example in Fig. 5 ). However, even when bounding boxes are somewhat correct (see top middle example in Fig. 8 where the watermelon is labeled as a red object and the spoon has no label), the model still erroneously believes the watermelon has been grasped, when it actually grasped the spoon. We also note that our model can often handle noisy labels: the bottom right example in Fig. 8 labels the green and yellow frog as a "yellow duck," however the reasoning referring to it still aligns with the specified task. Finally, we note that failures from incorrect plans or subtasks (but correct scene understanding) are typically uncommon, especially because the plans are re-generated after freezing and are often corrected in the process.We now discuss some the advantages and disadvantages of various alternative design choices and optimizations.Performing ECoT inference is significantly slower than "standard" VLA counterparts. Specifically, OpenVLA and RT-2-X achieve control frequencies of 3 and 2 Hz respectively, while our freezing approach achieves 0.5. However, we note that this slower inference speed is nonetheless empirically sufficient for outperforming these standard VLAs (see Table 1 ). Additionally, there are standard inference speed optimizations that can still be implemented in conjunction with the ones we tried. Most prominently, as mentioned in Section 4.3, TensorRT-LLM [66] claims to enable inference speeds of over 1000 tokens per second for Llama 2, which acts as the language model backbone of our ECoT policy. Given that our ECoT reasoning chains are typically around 250 tokens, this should enable control frequencies on par with non-reasoning VLAs. However, as our primary contribution is to show how reasoning aids in robot control policies, we leave such optimizations to future works.As shown in App. F, our approach can be applied for other reasoning step features or orderings while staying more performant than our baselines (see Table 3 ). Such alternative reasoning steps can give benefits like better interpretability for easier debugging, as demonstrated with our future gripper position feature in Fig. 13 . Some other features include open vocabulary 3D bounding boxes, which could be extracted by systems like FM-OV3D [73] (albeit requiring depth estimates, which our training data is not annotated with, but can be extracted using other foundation models). Likewise, PaliGemma [74] can be used for open-vocabulary semantic segmentation (instead of our current 2D bounding boxes), which also has the benefit of providing such segmentation maps in tokenized textual formats. As more foundation models are released for different semantic tasks, we expect that the type of reasoning steps our model can learn via distillation to only increase.We choose to train our model to perform all reasoning steps end-to-end. However, in principle, some of these reasoning steps can be performed by external modules, which may improve generalization. However, this would require additional compute to load the additional models and introduce more system-building complexity. Likewise, we find that current open-source VLMs are unable to generate the bulk of the reasoning. While proprietary LLMs like ChatGPT may be able to do so, they are naturally not trained to automatically generate reasonings of the expected format, and thus would require in-context examples or long prompt instructions on formatting (like with Fig. 10 ), slowing inference speeds. Furthermore, API calls would need to be made for each new image, incurring costs.Another possibility would be to use external modules to give feedback on generated reasoning chains. In practice, this is difficult to do zero-shot with off-the-shelf pre-trained models; most works using VLMs as feedback for learning or inference typically need to fine-tune them for their specific use case [75, 76] .Annotate the training trajectory with reasoning ## Specification of the experimental setup You're an expert reinforcement learning researcher. You've trained an optimal policy for controlling a robotic arm. The robot successfully completed a task specified by the instruction: "unfold the cloth from top right to bottom left". For that purpose, the robotic arm executed a sequence of actions. Consecutive states that were visited can be characterized by the following features:''' python t r a j e c t o ry _ f e a t u r e s = { 0: " stop " 1: " stop " 2: " move ␣ forward ␣ left " 3: " move ␣ forward ␣ down " 4: " move ␣ forward ␣ down " ... 36: " stop " } ''' Each entry in that dictionary corresponds to a single step on the trajectory and describes the move that is about to be executed.The robot is operating in the following environment. A black and red toy stove with a yellow banana in a silver pot, a blue toy brush, and a purple towel on the counter, surrounded by white tiled walls and a grey sink. ## Your objective I want you to annotate the given trajectory with reasoning. That is, for each step, I need to know not only which action should be chosen, but importantly what reasoning justifies that action choice. I want you to be descriptive and include all the relevant information available. The reasoning should include the task to complete, the remaining high-level steps, the high-level movements that should be executed and why they are required, the premises that allow inferring the direction of each move, including the locations of relevant objects, possible obstacles or difficulties to avoid, and any other relevant justification.Start by giving an overview of the task. Make it more comprehensive than the simple instruction. Include the activity, the objects the robotic arm interacts with, and their relative locations in the environment. Then, describe the high-level movements that were most likely executed, based on the task that was completed and the primitive movements that were executed. Then, for each high-level movement write the interval of steps that movement consists of. Also, for each high-level movement write a justification for why it should be executed. Write an answer for this part using markdown and natural language. Be descriptive and highlight all the relevant details, but ensure that your description is consistent with the trajectory that was executed, specified by the features listed above in the 'trajectory_features' dictionary. ### List the reasonings for each step Finally, for each step describe the reasoning that allows to determine the correct action. For each step describe the remaining part of the objective, the current progress, the objects that are still relevant for determining the plan, and the plan for the next steps, based on the available features. Start the reasoning from a high level and gradually add finer features. I need you to be descriptive and very precise. Ensure that the reasoning is consistent with the task and the executed trajectory. Write the answer for this part as a Python-executable dictionary. For every step in the initial trajectory there should be exactly one separate item of the form <step id>:<reasoning>. Do not group the answers. The final dictionary should have exactly the same set of integer keys as the dictionary of features provided in the 'trajectory_features' dictionary above. The reasoning should be a single string that describes the reasoning in natural language and includes all the required features.Each reasoning string should have the following form: -Describe the full task that remains to be completed (but only describe what remains), and place it inside a tag <task>.-Describe the complete high-level plan for completing the remaining task (the list of remaining high-level steps), and place it inside a tag <plan>.-Describe the high-level step that should be executed now (chosen from the list of high-level steps), and place it inside a tag <subtask>.-Describe why the chosen high-level step should be executed now, which features of the current environment influence that decision, and how it should be done. Place it within a tag <subtask_reason>.-Describe the current primitive movement of the arm that needs to be executed, and place it inside a tag <move>.-Describe why the chosen movement should be executed now and which features of the current environment influence that decision. Place it inside a tag <move_reason>.Here is a breakdown of what needs to be done:-Describe the task.-Describe the high-level movements that were executed, based on the completed task and the listed features.-Describe the plan for the solution that allowed the robot to complete the task successfully.-For each step on the trajectory, describe the reasoning that leads to determining the correct action. The reasoning should be descriptive and precise. You should provide exactly one reasoning string for each step on the trajectory specified by 'trajectory_features'.-At the very end of the response, write a single label FINISHED to indicate that the answer is complete.  
 
 Appendix:  


Decision: ### Strengths

- Re: Novelty: All reviewers agree that the paper should be considered a significant advance, as it introduces a novel robotic control pipeline that integrates chain-of-thought (CoT) reasoning into the end-to-end vision-language-action (VLA) model. R-Nt3h mentions that the CoT framework includes diverse components such as natural language subtasks, motion planning, and object localization, which they further suggest are appropriate and comprehensive for the task.
- Re: Evaluation: R-DLrg highlights that the approach is compared against strong and relevant baselines, and includes important ablations (e.g., Naïve CoT) that highlight the benefits of embodied reasoning. R-DLrg and R-LC5L note that the proposed ECoT approach outperforms existing models (OpenVLA and RT-2-X) in most generalization tasks and real robot evaluations. R-Nt3h suggests that the training dataset used is of high quality and is well-collected.
- Re: Clarity: R-LC5L and R-Nt3h suggest that the paper is generally well-organized, clear, and the experiments are well-performed, making the results convincing.

### Weaknesses

- Re: Method: R-DLrg cites dataset augmentation issues, wherein the augmentation of the Bridge v2 dataset is seen as tedious and unreliable, with high dependence on pre-trained models and no measures for quality assurance. R-LC5L sees the interactive correction mechanism as somewhat disconnected from the main focus of the paper, which they assert is the CoT strategy. R-DLrg expressed concern over the fact that the current policy is mostly open-loop, with no replanning framework for handling failures or incorrect sub-tasks — limiting the policy’s ability to adapt and recover from mistakes. R-DLrg further suggests that the learned policy is very sensitive to camera viewpoints, which may affect its generalizability, and that the speed of execution remains a limiting factor, despite efforts to improve inference time. 
- Re: Evaluation: R-Nt3h points out that the absence of experiments in a simulator for fair comparison with existing works is noted as a limitation. R-DLrg expressed concern over a lack of challenging tasks, asserting that the experimental results are limited to a small set of tasks and that the paper does not address performance in more complex scenarios such as cluttered environments. R-Nt3h calls for more ablation studies on key components of the CoT and their combinations to better understand their impact.
- Re: Clarity: R-LC5L expressed concern over the lack of clarity on how sensitive the performance is to different pipeline components, such as the impact of replacing 2D bounding boxes with 3D boxes.

### Post-rebuttal Meta Review Statement

I applaud the reviewers and authors for engaging in a productive discussion that resulted in several improvements to the paper, especially as it relates to the addition of new experimental evaluations (in response to R-Nt3h and R-DLrg, enabling the latter to raise their score) and sensitivity analysis of the policy to alternative reasoning steps and orderings (in response to R-LC5L).

I recommend for Accept as Oral, given the initial reviews and the extent to which the reviewer feedback was satisfied. 

Still, I encourage the authors to follow through with all the promised changes and to incorporate all additional reviewer requests, to improve the paper even further.

Reviews: review: title: title: Interesting approach for incorporating embodied chain of thought reasoning to OpenVLA. The work compared to strong baselines and demonstrates improved OOD performance on a selection of tasks. More details on the training and inference pipeline needed and further clarification questions on results, additional baselines, incorporating feedback and inference speed are desired.
paper_summary: paper_summary: This work builds upon OpenVLA and incorporates chain-of-thought reasoning while grounding it in sensory observations and the robot state. Since OpenVLA builds on relatively small open-source VLMs, its chain-of-thought reasoning capability does not match that of large closed LLM models. Furthermore, this step-by-step reasoning should be grounded in the robotics task. Thus, the authors propose training a VLA policy (using the OpenVLA architecture) to output CoTs and actions given input instructions and observations, ensuring that  both are firmly grounded in the agent’s environment.
main_review: main_review: Strengths:

Important extension to prior work – training VLAs to output step-by-step (CoT) grounded reasoning steps

A pipeline for generating the grounded CoT training data at scale

The proposed approach is compared to a strong set of baselines (OpenVLA and RT-2-X) as well as an important ablation (Naïve CoT) is considered that highlights the need for embodied reasoning in robotics tasks

ECoT policy outperforms OpenVLA and RT-2-X  on most generalization tasks for in-distribution viewpoints. 

Weaknesses:

Augmentation to the Bridge v2 dataset seems tedious and unreliable. High reliance on pre-trained models and no measures are taken for ensuring quality/correctness of the extracted features.

More details on the training objective and prediction pipeline could improve the clarity of the paper.

Results are demonstrated on a very small set of  tasks. How does the policy perform in clutter when the object detection modules will likely find it very challenging to identify objects. It would be useful to show results (even in simulation) on a set of more challenging tasks especially in clutter

If the human is not in the loop, the policy is mostly open-loop. No replanning framework considered for when object-detection fails, sub-tasks are incorrect or policy fails? One of the benefits of CoT can be to reason about past mistakes and replan accordingly. However, since the training data does not contain any failure and recovery scenarios, the current CoT framework cannot do that.  

Learned policy is very sensitive to camera viewpoints

Even though steps are taken to improve inference time, speed of execution is still limiting

Clarification questions:

What are the exact inputs and outputs of the policy? What is the training objective? Is the model predicting everything – task, plan, subtask, bounding boxes, gripper position and move primitives? Are the predictions sequential? If so, is the prediction pipeline similar to the data generation pipeline, i.e. first a task description is generated, then a scene description, then the bounding boxes (given the task and scene), then the gripper position and finally the plans, subtasks and move primitives (given the task, scene, bounding boxes and gripper position).

Instead of learning to predict the bounding boxes, why not, during inference, utilize Prismatic VLM and Grounding Dino to get the bounding boxes and gripper position, while using the rest of the prediction pipeline to get the embodied CoT? This could improve generalization to OOD objects.

RT-2-X seems to have a 0% success rate for the “Put mushroom in tall cup” while OpenVLA significantly outperforms ECoT. This is very surprising since RT-2-X is trained on much more data. Very curious to know the failure modes for this task for RT-2-X and ECoT.

The OOD Instructions seem quite challenging since most object detection modules will really suffer on ‘not <object>’ predictions. It is intriguing to see that both RT-2-X and ECoT perform better with Out-of-Distribution Viewpoint than with In-Distribution Viewpoint. It will be useful for the authors to share the reasoning behind that.

In section 5.4, it is unclear why ECoT benefits significantly more than the baselines. In the example shown, it seems like the human intervenes and corrects the object detection. This should benefit all the baselines equally.

For efficient Chain-of-Thought Inference, did the authors consider updating the high level plan based on feedback from the environment for instance based on a success/failure detection module (once a subtask is completed)?

In section 5.4, ChatGPT is used for correcting the reasoning chain. It is evident that including task information in the prompt allows ChatGPT to generate grounded CoT reasoning steps. Curious to know the author's thoughts on another baseline where OpenVLA is trained to perform language-conditioned  subtasks and during inference the subtasks are generated from ChatGPT using grounded CoT reasoning as is done in section 5.4.

The failure cases mentioned in the paper and in the appendix seem like object detection or scene understanding failures. It would be interesting to see if failure modes where the object detections or scene understanding are correct but the method fails to reason about them using ECoT. 

As an important implementation detail, it would be useful to the reader to know more about how, in section 5.5, part of the model prediction is kept fixed while the rest is updated? In section 5.4, it is mentioned that “we feed this corrected chain back into our policy and continue execution”, how is this done exactly? Does the policy take as input an initial reasoning chain? What are the exact inputs and outputs of the policy?
questions: questions: More details on the training objective and prediction pipeline including some of the clarification questions mentioned above could improve the clarity of the paper.

To evaluate if the accelerated CoT inference approaches mentioned in the paper are sufficient, the authors should consider providing comparisons of inference times against the baselines considered.

It would be useful to include the set of objects considered for in-distribution and OOD tasks in the appendix.
limitations: limitations: 2
review_summary: review_summary: Interesting approach for incorporating embodied chain of thought reasoning to OpenVLA. It is a useful contribution towards improving the capabilities of VLA models.
title: title: CoT being used for vision-language-action models
paper_summary: paper_summary: This paper looked into a pretty straightforward (yet not investigated) idea, which is on applying the chain of thought (CoT) approach to vision-language-action models (VLAs). It has been evident that the CoT technique is useful in large language models (LLMs) for different types of reasoning tasks. Recent advances on VLAs, such as the series of RT research, shows that pretrained foundation models are able to directly  perform robot control. The proposed approach is called ECoT (embodied chain of thought reasoning) for computing VLA policies for robots. OpenVLA and RT-2-X were used as baselines for comparisons. Results showed that ECoT performed better than the two baselines on real robots and human feedback could further improve the performance.
main_review: main_review: The key idea of this paper, i.e., applying the chain of thought technique to vision-language-action models, is solid and well presented. The reviewer is not aware of existing references studying this idea (the other reviewers can correct me if I am wrong). The baselines were well selected and represent the state of the art, which is good too. 

The "chain" of thought in the proposed approach was manually created and predefined, as demonstrated in Figure 4. While it worked for the tasks in the dataset used for evaluations in this research, it's a limitation and dynamically figuring out the reasoning chain can potentially be an improvement. It's unclear how sensitive the performance is to the current pipeline. For instance, if the bounding boxes are replaced with 3D poses, it's unclear how the results would be different. 

This is a minor comment. The interactive correction part doesn't connect to this work well. It's pointing to a different direction (HRI) while the focus of this paper is a novel prompting strategy for VLAs.
questions: questions: Was the collected dataset used for fine tuning the VLAs? What if we use the training data as examples for few-shot inference? 

Are there robot tasks that the current pipeline is inapplicable to? 

How is the performance sensitive to the current pipeline? For instance, what if we flip the order of the first two steps? What if the 2D bounding boxes are replaced with 3D boxes?
limitations: limitations: 3
review_summary: review_summary: Nice paper that described a novel idea that can potentially produce good impact to the community. The experiments were well performed and the results were convincing.
title: title: The review for Embodied CoT
paper_summary: paper_summary: The paper proposes a novel robotic control pipeline by including chain-of-thought in the end2end VLA backbone. The CoT involves language subtask, boudning box and other combination, and the experimental results is promising.
main_review: main_review: **Summary**:
The paper proposes a novel robotic control pipeline by including chain-of-thought in the end2end VLA backbone. 
The chain-of-thought involves natrual language subtask, next step motion planning, subtask reasoning, visible object localization, boudning box prediction and other combination. The model is build upon OpenVLA, and the experiments over the self-collected dataset demonstrates its efficiency.

**Strength**:
1. The novel framework for predicting mid-level results like the language subtasks and the bounding box are reasonable. Adding these as pre-output of the VLA model is an approprate pattern.

2. The chain-of-thought is very diverse, which includes natrual language subtask, next step motion planning, subtask reasoning, visible object localization, boudning box prediction. These mainly contains all the essential components i could image.

3. The collection of the training dataset is very reasonable and with high quality.

**Weakness**:
1. Some experiments on simulator could be helpful for fair comparison with existing works.

2.More ablation on key components of the CoT and different combination should be added.
questions: questions: See weakness.
limitations: limitations: 2
review_summary: review_summary: The paper proposes a novel robotic control pipeline by including chain-of-thought in the end2end VLA backbone. The CoT involves language subtask, boudning box and other combination, and the experimental results is promising.