LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
Authors: Xiang Li
Cristina Mata
Jongwoo Park
Kumara Kahatapitiya
Yoo Sung Jang
Jinghuan Shang
Kanchana Ranasinghe
Ryan D Burgert
Mu Cai
Yong Jae Lee
Michael S Ryoo
Venue: ICLR 2025 Poster (2025)
Keywords: Robot Learning
VLM
Behavior Cloning
Instruction Tuning
Self-supervised Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: Effortlessly and efficiently transform a pretrained VLM into a robot policy and enhance its performance using auxiliary data generated in a self-supervised manner.

Abstract: Vision Language Models (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language-Action (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by a limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets, aligning robotic actions with image pixel coordinates. Further, we enhance this dataset in a self-supervised manner by defining six auxiliary tasks, without requiring any additional action annotations. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across multiple simulated and real-world tasks, we demonstrate that LLaRA achieves state-of-the-art performance while preserving the generalization capabilities of large language models. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA.

Published as a conference paper at ICLR 2025
LLARA: SUPERCHARGING ROBOT LEARNING DATA FOR VISION-LANGUAGE POLICY
Xiang Li
Cristina Mata'
Jongwoo Park'
Kumara Kahatapitiya'
Yoo Sung Jang'
Jinghuan Shang'
Kanchana Ranasinghe'
Ryan Burgert
Mu Cai
Yong Jae Lee²
Michael Ryoo'
Stony Brook University
University of Wisconsin-Madison
xi siangli8@cn 1iccs. stonybrook.ede
ABSTRACT
Vision Language Models (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language--ctiinn Actioon (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, framework that formulates robot action policy visuo-textual conversations and enables effi- cient transfer of pretrained VLM into powerful VLA, motivated by the success visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate ronversation-style instruction tuning data for robots from ex- isting behavior cloning datasets, aligning robotic actions with image pixel coor- dinates. Further, we enhance this dataset in a self-supervised manner by defin- ing six auxiliary tasks, without requiring any additional action annotations. We show that VLM finetuned with limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across mul- tiple simulated and real-world tasks, demonstrate that LLaRA achieves state- of-the-art performance while preserving the generalization capabilities of large language models. The code, datasets, and pretrained models available https//gitubb.com//osxxee//////
D-aBC
Funtrs Preduction
Detection
<task>Place all other
objects <oolarge sovl</p>.</task> List the actions required
Temponal Ratutoonll
Action
on this scene
up the <p>earrot</p>
<b> (0.22, 0.660)</b> rotate </14-)))</ degrees, and drop it at <b> 50.54, 0.500) </b>.
LIaRA
Env
Agent
Figure 1: A real-world demonstration of LLaRA solving unseen task. In this setting, LLaRA converts only eight thousand simulated expert trajectories from VIMA (Jiang et al. (2023) into strong instruction tuning data and generates auxiliary datasets a sele-supervised manner. With such and about one thousand additional in-domain images, a pretrained Vision Language Model (VLM) is finetuned into powerful robot policy that can solve many unseen real-world tasks. At inference, the VLM is queried to generate actions in natural language where the robot action is aligned to image pixel coordinates.
INTRODUCTION
Large Language Models (LLMs) such as GPT-4 (OpenAl 2023). Llama (Touvron al.] [023] AI 2024). and Gemini (Anil et al. 2023) exhibit unprecedented abilities across diverse language tasks. Such L.LMs often trained together with visual data by taking advantage of pretrained

---

Published as a conference paper at ICLR 2025
vision encoders, forming powerful Vision Language Models (VLMs). Effective and efficient training of such VLMs significantly depends on the style and format of the Assio-language data used for training. leading to the exploration of various instruction tuning (Taori et al., 2023; Chiang et al.
2023
Liu et al. 2023a 2024b) strategies. The resulting VL.Ms exhibit strong Asionnlanguuge skills, but without limitations such as spatial awareness (Chen et al., 2023 Ranasinghe al. 22024b)
or niche-domain understanding. The latter has led to domain-specific visual instruction tuning
al.
2023 Kuckreja et al. 2024 Hong et al. 2023).
Several robot learning approaches also leverage pretrained LLMs VLMs (Zng et al.
2022 Brohan
al.
2023c; Driess et al. 2023] Vemprala et al. 2024 Kim et al., 2024 Niu al. 2024 Zheng
et al.
2024) with recent work focusing on low-level robot actions (Broban et al. 2023b a Padalkar
al.
2023p. The strong performance of such approaches can be attributed to the extensive world
knowledge (e.g., understanding of physics, human common- sense)) (Yu et al.) 2023] iZhao et al.)
2024) and powerful reasoning abilities (Creswell & Shanahan 2022 Liu et al. (203b)) underlying
language models. However, studies curating and using conversation-style data for instruction tuning in robotics have been rather limited (Zhen et al., 2024).
Motivated by the promising attributes of VLMs, we explore process, called Visuomotor Instruc- tion Tuning. of adapting a VLM to a robot action policy that can handle diverse visuomotor control challenges. More specifically, sational dataset, Instruct-BC (inBC) for VLM finetuning. Given a state described in visual-textual modalities, our VLM is trained generate suitable actions in text, aligned to image pixel coor- dinates. Such a formulation based on conversation-style instruction-ressonse data enables us to convert VLM into a robot action policy effortlessly. However, we also note that the effectiveness of an instruction-tuned VLM depends heavily on the quality of data formulation (Liu et al. 2023a Ranasinghe al.) [024b] Kuckreja al., 2024). which is non con-trivial automate (or, scale) new domains (Li et al. 2023] Kuckreja et al., 2024 Hong et al., 20233 Chen et al.) [023] Ranasinghe al. 2024b). Therefore, to strengthen (or, supercharge) such data, we construct auxiliary datasets
transform typical behavior cloning (BC) dataset into conver-
based on the same BC dataset, in self-supervised fashion without any extra action label.
Our overall framework termed LLaRA (Large Language and Robotics Assistant) generates visuo- motor instruction data that can efficiently and effectively finetune a VLM into a robot action policy. is motivated by LLaVA (Liu et al., 2023a), which is designed primarily for vision tasks. Similarly, LLaRA offers the entire framework of data generation, model formulation, and training pipeline for VLMs, now specialized for robot learning. Our key contributions are:
1. Formulating robot manipulation tasks into iestruction-responee pairs described in natural language, which enables successful instruction tuning of VLM as a policy.
2. Aligning robot actions to image pixel coordinates, makes the transfer of a pretrained VLM to the robotics domain more efficient, particularly when the training data is limited.
3. Identifying and generating auxiliary data that further enhances robot policy learning in a self-supervised manner.
We conduct extensive experiments the proposed framework to establish the effectiveness of both our automated data generation pipeline and Instruction-tunar VLM in solving robotics tasks.
2 RELATED WORK
Instruction data generation. Finetuning LLMs VLMs with instruction data has shown lot of potential for vision tasks (Taori et al., 2023 Chiang et al. 2023: Liu et al., 2023a 2024b). While effective construction of such datasets with high quality and quantity beyond trivial (Liu et al. 2023a), especially for specialized visual domains (L.i et al. 2023 Bazi et al. 2024 Kuckreja et al.,
2024 Thawwar et al. (023) such as roboties. Yet these modified VLMS suffer the robotics do-
mains Ranasinghe et al. 2024a).
Spatial reasoning in VLMs. Several recent works investigate how VLMs can be modified for
spatial awareness within images (Zhang al. 2023 Zhao al.) 2023 Zang et al. 2023 Peng al.)
2023
Chen et al., 2023 Ranasinghe et al. [024bb You et 2003 Cai ai et al. 2024 Shtedritski et al,
2023
which is a critical ability of visuomotor policy. One line of work explores prompting using
textual or specialized tokens to encode locations, followed by instruction tuning on localization-
specific datasets (Peng et al. 2023 Chen et al. 2023) Ranasinghe al.
2024b You et al. 2023). In
2

---

Published as a conference paper at ICLR 2025
contrast, framework extends beyond localization to directly predict robot actions. An alternate line of work explores visual prompting but without any focus on robot learning (Shtedritski et al.)
2023
Cai et al. 2024). An extension of such ideas to robotics is explored in PIVOT (Nasiriany et al. 2024). These ideas are complementary to LLaRA, where additional visuomotor instruction tuning directly generates actions optimal for robotics tasks.
Robot learning via sequence models, LLMs, and VLMs. The use of sequence models in robot
learning has longstanding tradition (Zheng al., 2022 Shang al. 2022a b). Recent foundation
LLMs / VLMs provide a source of common-sense data for robot policy training (Qian et al. 2024]
Ingelhag et al. 2024 Wu et al. 2023b Yoneda et al. 2023 More recent work such as Robotics
Transformer (Brohan et al.) 2023b]a a, Padalkar et al. 2023), Gato (Reed et al. [0222, GR-1 (Wu
ct al.
2023a) and Octo (Octo Model Team al.] 2024) learn generalist robot policies with multi- modal sequence models, taking advantage of the flexibility of encoding and decoding images/actions
via token representations similar to VLMs.
PALM-E (Driess et al. 2023) utilizes pretrained VLM for embodied reasoning and planning. How- ever, assumes access to additional policies capable performing low-level skills from limited vocabulary, with the VLM generating natural language that conditions these low-level commands. SpatialVLM (Chen et al. 2024) focuses on enhancing the spatial reasoning ability of the VLM, re- sulting in a VLM that serves spatial relationship predictor rather than robot policy. To embed SpatialVL.M VLM into a policy, oh-huuh spatial reasoning is required.
There are concurrent works (Kim al., 2024 Yuan al. 2024 Niu al. 2024 Zheng et al., 2024) which also utilize a LLaVA style VLM as a robot policy. All such approaches employ a VLM robot action policy, processing both visual observations and textual instructions inputs. OLe (Kim et al. 2024) employs specialized tokens to represent quantized actions, a method akin to RT-2 (Brohan al. [023a).. Open VLA also incorporates DINOv2 (Oquab et al. 2023) as an additional visual encoder along with SigLIP (Zhai et al. 2023), compared to single CLIP (Radford) al. (02T) vision encoder used in LLaVA. RoboPoint Yuan al. 2024) introduces novel point- based action space and scalable data pipeline tailored for spatial affordance prediction. Actions are delineated via points on an RGB image, which are subsequently translated to 3D space using depth data. This approach eliminates the reliance predefined action primitives (Liang al. al.) 2023] Singh et al., 2023), external object detectors (Huang et al.] 20236 Liu al.) 2024a), and iterative visual prompting (Nasiriany al. 2024). LLARVA (Niu et al. 2024), differs by generating both 2D visual traces in image coordinates and corresponding textual actions as outputs, with the former functioning an auxiliary task.
In contrast, our method describes robot action using image coordinates in natural language, which creates a more intuitive link between vision and action without modifying the existing VLM model architecture. This enables a much more efficient transfer of the pretrained VLM to our robot VLA model. Moreover, all the aforementioned studies lack of comprehensive investigation into the meth- ods for generating self-supervised auxiliary datasets from existing robot data, as well as the impli- cations of integrating such datasets for the training. Our work LLaRA distinctively addresses this gap, marking a critical divergence from the concurrent studies.
Self-supervised learning in robotics. Self-supervised learning (SSL) has long been explored in robotics (Pinto & Gupta 2016; SSermanet et al. 2018 Yarats et al., 2021 Laskin et al. 2020 Schwarzer et al. (021), with recent work showing a flavor of visual representation learning (Li al. 2022; 2024b| Wang et al., 2023). We introduce auxiliary SSL objectives (i.e., pretext tasks)
based on instruction prompts, motivated by similar ideas in computer vision (Doersch et al., 2015
Yun et al. 2022 Ranasinghe et al. 2022 Walker et al. (20TT). More specifically, we create set of domain-specific auxiliary datasets (e.g., action or future prediction), that enable LLaRA to under- stand information that is useful for the downstream roboties tasks, when used for finetuning.
3 PRELIMINARY: VISUAL INSTRUCTION TUNING
Visual Instruction Tuning is framework to finetune a large Vision Language Model (VLM) using task-specific instructions. The objective is to develop a general-purpon multimodal model capable of solving diverse vision tasks described by language (Huang al. 2023a). A typical example of
3

---

Published as a conference paper at ICLR 2025
Visual Instruction Tuning is LLaVA (Liu et al., 2023a). whose model consists of three components: a large language model BLLM, a visual encoder by and an adapter layer 9MLP (see Fig. 2
Consider conversation data sample that contains single image Ze and user instruction text 21 as inputs. During inference, the visual encoder By and adapter layer OMLP process by successively to pro-
Images (c.)
Text Input (i)
Viauad Encoder (42)
Text Tokenizer
duce visual tokens that can be directly concatenated with the textual tokens of zy in language embedding space of OLLM. Next, OLLM au- toregressively generates new tokens conditioned on both the visual tokens from Ty and text tokens from z₁. Finally, generated tokens are decoded into natural language as the output of the model.
Actepter Layer (AMLP)
Concrenaton
(fLMM)
LLaVA overview. A Large Language Model (LLM) connected to the visual domain with suitable encoder and adap- tor neural networks.
Figure 2:
A two-phase training process is executed in Liu al. (202a)) with next-token prediction objective over the 0LLM output in each phase. First, only 0MP is trained using image-caption pairs
(similar to datasets for CLIP (Radford et al., [202]) training).
Then, both OMLP and 0LIM are finetuned using a specialized instruction tuning dataset termed contains human-style instruction- response conversations, automatically created by a powerful LLM (OpenAI, (023) from a generic object detection dataset. This instruction tuning process is what focus this paper. We start from a pretrained LLaVA model and only perform a single-stage finetune, updating iLLM and fMLP based on the instruction tuning data formulation we introduce in this paper.
4 VISUOMOTOR INSTRUCTION TUNING
We leverage large Vision Language Models (VLMs) as a generalist to address diverse visuomotor control challenges. Specifically, we transform typical behavior cloning dataset into an instruction tuning dataset and subsequently finetune a pretrained VLMM on this tailored dataset, a process we call Visuomotor Instruction Tuning. The resulting LLaRA framework benefits from the broad, inherent knowledge embedded within the VLM and the alignment between robot action and image pixel coordinates, enabling more efficient visuomotor policy learning.
In this section, we present our LLaRA framework to a) convert a set of robot manipulation expert trajectories into a visuomotor instruction tuning dataset, and b) finetune a VLM on such dataset as a robot policy. In the next section, we show that c) such VI.Ms can be further improved for robot control using auxiliary instruction tuning data created a self-supervised manner.
Problem formulation. We consider a behavioral cloning (BC) setting over a Markov Decision Process (MDP), described by the tuple AS P). Here, s, E S denotes the state at timestamp I, de E A represents the action at I, and P encapsulates the transition dynamics, expressed as P((4+1 (8e,,r)) ur). Our goal is to finetune a VLM as a robot policy ı that best recovers an unknown policy n²" using demonstration dataset containing multiple tasks D {(s', ,a)} collected by 'τ" The policy и is designed to be conditioned on the task description, and our empirical findings suggest adding historical actions is beneficial condition. Consequently, the policy be articulated as r (ajls), ), where h- = 1,2. t - 1, indicating that æ takes into account all preceding actions.
Instruction tuning data from trajectories. Next, turn set of expert trajectories D into a visuo- motor instruction tuning dataset. For each state transition, we convert the state action pair (st, de) into a single-round conversation. The current visual observation and textual task description, form- ing 8;, can be directly assimilated by a VLM as the user instruction. However, the numerical action aj needs conversion into textual format to be generated by VLM. In contrast to the approach em- ployed by GATO (Reed et al.) 2022), RT-2 (Brohan et al. (023a). and OpenVLA (Kim et al.) 2024). which utilizes special tokens to encode quantized action values (binning tokenization). we adopt 2/) Image
to [0. - represent the positions in actions. Additionally, any rotational component of the action, denoted as a, (such as the angle of a joint), is expressed textually as "rotate <r>a,</I> degrees". To further improve the performance, we incorporate the history of executed actions into the query and structure the response to predict multiple future action steps until the episode concludes. More details are covered in Appendix A. Our approach creates an intuitive connection between visual input and robotic actions, enabling effortless transfer across various robotic embodiments. By implementing these strategies, effectively convert trajectory
4

---

Published as a conference paper at ICLR 2025
LLaRA inBC
LLaRA 2-inBC
Current Observation
Textual Task Description ceaaks/irat <pozaiabov letter ze/p> into <prwcoden houtc/pp then pat, the akje proviouely Kpowoodan bowls/pp.. basas
Current Observation
Multi-modal Task Description
Reference Imagee
desoribed in toxt
Exerm
Exerm, 1
Exame
objecte
[Deeo drame_ [Deac. Scame 2)</baak..
of frame_13
<aceoa>>
ryored letter vergo as (o, (0.694, 0.12554/bb
Visual Encoder Adapter (0y, MLP)
Visual Encoder & Adapter (0y, (MPP))
Text Takenizer
Text Tokenizer
spoeatsbow letter velgo
Cacener
LLM (OLLM)
LLM (0LLM)
Step 1: Pick up the spoeninbew latter T</p> at <b> 500, o. 617)/4/b>, rotate <0>10]<//>> degrees, and deop it at <b> (0.746, 6/617)//b>> Stop 2:
Step 1: Pick up the opbgeeen paisley letter Ve/p> at <h> 466, 0.617)<//>,, rotate <>>(0]</<>> degrees, and drop it at <bo (0. 711, 0.617)//b>>
Stap 2:
Figure 3: Example data visuomotor instruction tuning, (left) inBC is our instruction luning data created from BC, only taking textual task description. (right) D-inBC is designed for multi- modal task description that considers reference images by taking their corresponding textual description. Both inBC and D-inBC belong the LLaRA family.
into a format comprising both image and text, which readily processed by the VL.. We name the converted dataset Instruct-BCC (inBC) and present examples in Fig. 3 Tab. annd Tab.
Aggregate multiple images for a single-image VLM. The VLM policy we use here closely fol- lows the design of LLa (Liu et al., 2024b). The model consumes a single image and language instruction and generates language output. In scenarios where a single observation s comprises an observation image and multiple reference images, systems such LLaVA suffer from low perfor- mance because it was never trained on multiple images (see Tab. [5). Instead, we convert each additional reference image into a language description with the help of object detection the addi- tional images (e-g., images in the task description, rather than the main visual observation). The main image is still directly used as the visual input by the VLM without any changes. Compared to inBC, we name this method that takes additional object detection results as Description- Instiinn-nnttuut-CCC (D- inBC), and our model trained on this data referred to with the same name as earlier. Examples are also available in Fig. 3 Tab. g and Tab. 9 In Appendix A.1.3 we also discuss another variant of D-inBC that uses natural language instead of structured text to describe the reference images: DDiiBC (L).
Inference pipeline. During inference, the prompt for the VLM prepared using the same template as either inBC or D-inBC. As illustrated in Fig. 3 for model trained inBC, each conversation turn encapsulates the current visual observation, the task description, and the previous actions de- scribed in the text. For model trained on D-inBC, the process starts with object detection to convert any extra images from the task description into text. The task description, now including the detec- tion outcomes, is then combined with the current visual observation and the action history to get the complete query. Then LLaRA generates the text output that contains numerical values (e.g.. 2D image coordinates) for robot action. The extracted rotation angle can be directly mapped to the rotation of the end effector. The 2D image coordinates will be further converted to robot action space via predefined mapping, which can be estimated using visual calibration in both simulated and real-world environments. After that, the robot executes the action, and a new observation can be captured from the environment. This initiates another round of interaction, setting the stage for the subsequent action.
SUPERCHARGING VISUOMOTOR INSTRUCTION TUNING DATASET
Motivated by the success of ILaVA LLu et al. 2023a instruction tuning data, and subsequent domain-specific instruction tuning datasets (c.g.. ., for reasoning, grounding, or referring) (Chen et al., 2023 Ranasinghe et al. 2024b Peng et al. 2023), LLaRA creates auxiliary robotics instruction tuning datasets that enhance a VLM in sulf-supervised manner.
More specifically, given an expert trajectory, we generate conversations that reveal useful infor- mation for learning policy implicitly via scripted templates. In addition to visual observations, we make use of automatically generated object labels, locations, and geometries (e.g., rotation) to- gether with task descriptions expert (or oracle) information. We primarily introduce six auxiliary
5

---

Published as a conference paper at ICLR 2025
Localization
Detection
sjoobjs//p
Provide description for each object in the image,
in the
<ppobi,</p>
<b> (RL </bb.
Output: cocened [Loca Leation of algill
(Localiaation of abjsi </aoeee>
Action Prediction
Input: <leage?>
Future Prediction
Could you desoribe the stepe regutced to change the scens ahown in the image of sacend scens? The second soens in dapleted [Dasc, <lasgeui))
<image:>
Observation kimage?>
Displayed in the image in acese conteining aut ciple objecta. How you pactocn LAction at auly what will the scece look like?
Astion
[Action at sc]
Output: ziss up Iecsiisation abdil, zotabe crolau] degress, drop obj. cirageuss))
Output
(Desc. of <inageuuP)
Tamporal Relationship
Spatial Relationship
the image shous
at the initial
timeatarp. and the second image described as [Deac. of <imageo can describe how relative posiitions batwesn op0obg/</30 sppobjss/pp has changed??
positioo of apoobgrk/p conpazed to <pacbdi</p> in this inage?
Output: <problic/p> right fxon <posbjis/pp
Observation at [p] <inaggeuu?
Kuol idean
4,14/
Outputt
qpoeetys/pp noves far Exos qpoebtus/pp. /po. (spatial rel. changes]
<o>[s]</ep,
Figure 4: Auxiliary datasets for visuomotor instruction tuning, Given input trajectory. we make use of expert information (e.g-, object detections) to formulate conversations related to auxiliary semantics. Please refer Tab. TT and Tab. 12 for the actual examples of each datasct.
datasets, augmenting existing inBC or D-inBC to better finetune VLMs. In each dataset, sider a standard instruction, which is further rephrased multiple times using GPT-4 (OpenAI 2023) infuse reasonable diversity. The answers follow simple rule-based templates that format expert information as expected responses for each conversation. The following paragraphs briefly describe each auxiliary dataset. We kindly refer the readers Fig, 4 and Appendix A.1.4 for formatting details and qualitative samples.
2023)
Localization and detection. single-ture conversations for both localization of individual objects and higher-level detection of entire scenes. For localization, each conversation involves pinpointing an object within an ob- servation, where the query specifies the object's location and the response is given in the format "<p>OBJ</p> at <b> (x, y) , [w, h) </>/>>.. Here, (x, y) indicates the midpoint of the object's s bounding box, while {w, h} denotes its width and height, with coordinates normal- ized to [0, 1] as detailed in Sec. 4 For detection, the setup extends to multiple objects, describing the scene via list of bounding boxes.
To enhance the visual grounding ability of VLM, we first generate
Action and future prediction. These datasets are designed to improve the VLM's capability to un- derstand dynamics. In the action prediction dataset, given initial observation, the query includes a text description of the subsequent observation, formatted similarly to the detection dataset. The VLM identifies the specific action that connects these states. For future prediction, the roles of query and answer are reversed: the query now presents a single-step action and requests a text description of the next observation.
Spatial and temporal relationships. We further enrich the dataset with the relationships between objects. For spatial relationships, given objects in image, format conversation to ad- dress their 2D configuration (e.g., left, right, above, below) including Euclidean and axial distances, with an exemplar provided for better instruction comprehension by the VLM. For temporal rela- tionships, the dataset examines changes in these spatial relationships over time (i.e., between two timesteps). This involves two observations the initial in image form and the subsequent described in text-focusing on queries about movements (e.g., getting closer or further away), with responses detailing changes in Euclidean and axial distances.
By introducing these auxiliary datasets, we explicitly drive the VLM to strengthen the ability of spatial and temporal understanding, which benefits robot learning. Note that all these datasets can be automatically generated from existing trajectories, without introducing external task/action pervision. In Sec. we empirically explore the best practices to apply these auxiliary datasets.
EXPERIMENTS
We conduct experiments in both simulated environments and the real world. For the simulated tasks, we tune a VLM on diverse tasks as a generalist. Then, we conduct real-world robot experiments using three protocols: zero-shot generalization, finetuning, and joint training.

---

Published as a conference paper at ICLR 2025
6.1 SIMULATION EXPERIMENTS
Settings. We employ MMM--Bench (Jaag et al. 2023). a simulated table-top robot manipulation environment to evaluate VLMS trained by our instruction tuning dataset. The environment contains 17 tasks and each task is associated with a multi-modal instruction, including text instructions and reference images that refer to objects of interest or a particular scene arrangement. The robot action space is two 2D coordinates for pick and place positions and two quaternions for rotations. We uniformly subsample the VIMA dataset (Jiang et al.) 2023) to form three subsets with different sizes: VIMA-0.8k, VIMA-8K, and VIMA-80K where the number indicates the number of expert trajectories in the dataset.
Methods. We compare variants of our method with baselines that follow the recipe of RT-2 (Brohan et al. 2023a), RT-2 Style, and D-R7-2 Style. As introduced in Sec. inBC is the dataset converted from the expert trajectories. Prefix DD means the additional reference images in the task description are described by object detection results in text and the object detection will be performed on the reference images first during inference. Suffix Aux means the auxiliary datasets introduced Sec. D are included in the training set, and the letter after it (e.g.. (DJ) indicates the detailed configuration of the auxiliary dataset (see Tab. 2}. Oracle means that the groundtruth bounding box of objects is used as the object detection results only in the reference images. RT-2 Style is similar to inBC but the output of the VLM (i.e., robot actions) quantized actions. Specifically, in VIMA, the action space is two positions in robol coordinates and one rotation. D-RT-2 Style (I) is similar to D-RT-2 Style but the output tokens present quantized two positions in image coordinates and one rotation angle, which is the quantized version of what LLaRA generates in natural language, presented by the same set of special tokens.
set of special tokens that can be directly mapped to
We train all methods these three datasets and evaluate them with 3 levels of difficulties following the test protocol (LI to L3). Due to missing data in the original VIMA dataset, we were not able to evaluate our approach L4 properly. For each task, evaluate all the methods with 20 random seeds and report the average success rates of each level and the average of three levels.
INESS-Igit Crack
HaaA (ABC (71)
Liak (DAIS Dradit
Hakk (HA0C Avc (II)
ILAA (HLDC Auc (8) Drado)
Nau of oopert oplecee
Nex ogger. spledee
Nex of ogett spledt
Nrxx of ogent spledd
Figure 5: Success rates the models trained on VIMA subsets. The log-scale x-axis shows the number of expert episodes used in the training set. See Appendix the underlying numerical data.
Effectiveness of LLaRA framework. Fig. 5 iillustrates the performance of selected methods across various dataset scales. All methods generally benefit from increased training data. Key observations include:
inBC consistently surpasses the RT-2 Style baseline, and similarly, D-inBC outperforms D-RT-2 Style. This supports the effectiveness of our approach, which utilizes instruction- tuning style data aligned to image pixel coordinates. It is casier to benefit from the existing VLMs pretrained in a conversation style. Methods based on RT-2 Style improve when more robot supervision data is available; however, they significantly underperform compared to our methods when data limited. Even with 100% data, RT-2 Style still underperforms our method trained on only 12% data (Appendix B.2).
D-inBC generally excels over inBC. owing to the additional information from reference images. This trend is similarly observed between D-RT-2 Style and RT-2 Style.
Auxiliary datasets prove to be highly beneficial, particularly when the size of the train- ing set is limited. However, in scenarios like VIMA-SOK where the dataset is substantial, sometimes D-inBC performs better and the gain from the auxiliary datasets is marginal.
We hypothesize that when training data is adequate, the benefits of the auxiliary dataset diminish, and these data may even distract the model from effectively performing the behavior cloning tasks. It is advisable to carefully regulate the amount of auxiliary data when there is abundance of expert
7

---

Published as a conference paper at ICLR 2025
episodes to avoid overwhelming the primary learning objectives. By default, randomly sample from each auxiliary dataset so that the size of each auxiliary dataset is identical to inBC or D-inBC.
Comparison to VIMA. VIMA takes both front and top view images from
Table I: Comparison to VIMA (Jang al. 2023). Our best model not only achieves better performance but also requires less input and is trained on only 12% of the expert trajectories used in VIMA.
the environment while ours only
takes the front view. We test the most capable model released by VIMA, which is trained on 660k expert tra- jectories, and the results are shown in Tab. Compared to VIMA, our best model not only achieves better performance but also requires less input and is trained on only 12% of the expert trajectories used in VIMA. The full comparison in Tab. 13
Method
Config
Data
(1)
(.(%)
(%)
VIMA-200M Oracle
LLaRA (Ouns)
D-anCC Aux (B) Orncle
12%
90.0
$8.1
79.2
Effectiveness of auxiliary datasets. Tab. shows the different combinations of the auxiliary datasets we studied. For each setting, we randomly sample the same amount of examples from each auxiliary dataset and combine them with the converted behavior cloning dataset. The *** after a ''' means the reference images that appeared only in the task description are not used to generate this dataset, which makes the dataset less diverse. We train the model on both inBC and D-inBC with different auxiliary dataset settings. On VIMA-0.8K, we control the total number of samples from the auxiliary dataset relative to the samples from the converted BC datasets and train all mod- els for epochs. Results in Fig. 6 show that in general, on VIMA-0.8K the model performs better with more auxiliary data. Tab. shows that on VIMA-8k, auxiliary datasets can significantly benefit RT-2 Style models as well.
Naming of diffcrent auxiliary dataset configurations. We always randomly sample the same amount of examples from each dataset. Det.: detection;
Table 2:
35.
32.
[ 38.0-
25.
A*
Loc.: localization; Act.: action prediction; Fut.: future predic- tion; Spa.: spatial relationship; temporal relationship.
172
ducaet idutti 3B0
Thtal
Total matlee LBBC
Temp.:
33.-
Ast. Fut. Spa. Tamp.
35.0-
23.5-
33.0-
55
Table RT-2 Style benefits from auxiliary datasets, all mod- els are trained on VIMA-8K.
53.
Thad
Trtal
inBC (top) and D-inBC (bottom) with different auxiliary dataset settings. Each model is trained on VIMA-0.8k for epochs. In general, the model performs better with more auxiliary data. See Appendix for the un- derlying numerical data.
Figure 6:
RED Sade AT2 Sele Ass WBC
18
3.1
1.7
36.9
35.8
12.1
A comprehensive ablation of other design choices
is available in Appendix
6.2 REAL-WORLD ROBOT EXPERIMENTS
We further conduct zero-shot generalization, finetuning, and joint training experiments in novel real-world environment, in which there is robot arm with gripper and fixed RGB camera positioned above the arm to collect observations (see Fig-Z) 7). The action space is the same as the one VIMA. As all the objects are placed on a plain surface and the camera is fixed, similar to VIMA, linear mapping between the image coordinates and the robot action space can be estimated by calibration. For the meth- ods that rely on object detection Le.. D-inBC, we employed one-shot detection using OWLv2 (Minderer et al. 2024). We benchmark each policy three tasks:
Fixed webeam
xAma? w/ Gripper
T1: "Move the {object) into the large bowl."
T2: "Rotate the (object) by (angle) degrees."
Table
T3: "Move the (object) top of the tray."
Figure 7: Our world robot setting.
8

---

Published as a conference paper at ICLR 2025
where the {object] is chosen from set of 10 plastic toys and the (angle) in T2 is chosen from the following set: {30, 90, 120, 180}. In all tasks, all objects are placed randomly on the table before running episode and they never appear in the training set. A success in T1 and T3 is if the object placed more than 50% inside the bowl or on the tray. A success in T2 if the object appears to be clockwise rotated by the angle by visual inspection. We run 20 randomly initialized episodes for each setting and report the average success rate. A visual reference for typical task start states and success ends provided in Fig.
TI w/ Distractors T2 w/ Distractor Figure 8: Visual reference for the initial image (top row) and successful end positions (hottom row) three real-world tasks. also show some examples that distracting objects in scene.
TI
T2
Real-world robot experiment results. All models are trained on VIMA-8k unless otherwise stated. "Aux' means trained on auxiliary datascts and 'OD' means using an object detector (OWLv2-base base-11) to de- scribe the observation before the VLM query. In Protocol, 'ZS' stands for zero-shot evaluation, 'FT" means the model first trained on VIMA-8K for epochs and then finetuned Add. Data for epoch. 'JT" means model trained
Table 4:
combination VIMA-8k and Add. Data for 2 epochs.
Protocol
Method
Add. Data
T1 (%)
T2 (%)
T3 (%)
Avg. (%)
4o ver. 2024-05-13
20
45
30
31.6
RT-2 Style
VIMA-8k
0
RT-2 Style
VIMA-660k
0
inBC
40
50
20
36.6
LLaRA
inBC Aux (D)
10
30
10
16.6
(Ours)
D-inBC OD
30
40
25
31.6
D-inBC Aux (C) OD
35
40
5
26.6
RT-2 Style
VIMA-8k
XAm--Dt
0
0
RT-2 Style
VIMA-660K
XAm-Det
0
0
inBC
xArm-Det
0
inBC
xAmm-Action
30
45
5
26.6
FT
inBC Aux (D)
xArm-Det
100
95
70
88.3
LLaRA (Ours)
inBC Aux (D)
xAmm-Actio
70
75
80
75
D-inBC OD
XAmm-Det
0
D-inBC + OD
xAmm-Action
80
55
60
D-inBC + Aux (C) OD
xArm-Det
70
90
70
76.6
D-inBC + Aux (C) OD
xAm-AActio
90
100
85
91.6
inBC
XAm--Det
80
75
70
75
inBC
XAm--Acton
60
75
55
63.3
inBC Aux (D)
xAm-Det
75
85
70
76.6
LLaRA (Ours)
inBC Aux (D)
xAmm-Action
50
90
50
63.3
D-inBC OD
xAm--Det
65
95
45
68.3
D-inBC OD
XAm-AActon
65
55
25
48.3
D-inBC Aux (C) OD
XAm-Det
70
95
85
83.3
D-inBC Aux (C)+ OD
XAm-Actiin
45
70
20
53.3
Zero-shot generalization. For zero-shot generalization and finetuning, we use the pretrained mod- els from Sec. 6. only trained on simulated VIMA data. We also make the setting very challenging by selecting the pretrained model only using 1.2% of the VIMA training data (VIMA-8k). We benchmark the models without any further training, along with new baseline, GPT-4o (OpenAl, 2023). The results are presented in the upper part of Tab. In this setting, inBC achieves the best overall performance. We hypothesize that the auxiliary datasets used in inBC + Aux and D-inBC Aux drive the model to focus 4o achieves commendable performance, largely attributable to its extensive dataset and substantial parameterization.
the VIMA domain, leading to overfitting. Meanwhile, GPT-

---

Published as a conference paper at ICLR 2025
Finetuning real-world data. In the finetuning setting. first collect two in-domain real- world datasets xArm-Det and xArm-Action (see Appendix CC. Then the models trained on VIMA- 8k introduced in Sec. 6.1 are further tuned on the new real-world datasets for epoch and the evaluation results are presented in the middle of Tab. 4 In general, finetuning outperforms other settings because it allows the model to focus more on real-world data distribution, rather than be distracted by simulated VIMA data. Auxiliary data contributes to large boost in performance during finetuning, showing that explicitly uncovering information from the existing in-domain data can benefit the model. It seems inBC is not trained to take advantage of such new in in-domain robotics data, performing worse than zero.shot. Similar to our findings in simulation experiments, LLaRA in general benefits from pretrained VLM since LLaRA is trained on dataset that has been organized in a conversation style that is similar to what is used to pretrain the VLM. contrast, RT-2 Style requires large amount of data so it suffers more from limited data in our real-world experiments.
Joint training on both simulated and real-world data. In the joint training setting, we combine both VIMA-8K with XArm-Det LAm--Action and tune a VLM jointly on both simulated and real- world datasets. The full results are presented at the lower part of Tab. In the joint training setting, xArm-Dea is equally or more beneficial than XAmm-Action highlighting the importance of detection and localization information from the real-world setting. Auxiliary data proves most beneficial when jointly trained with xArm-Det, achieving the highest performance across all joint training settings. This observation suggests that careful exploration of auxiliary dataset usage is crucial to maximize their benefits when joint training is performed.
More generalization results. We further validate the generalization ability of LLaRA on tasks with dis- tractors. There are three test set- tings and none of them is included in the simulated training set or the dataset for finetuning: T1 w/ distrac- tor: where we randomly put another random toy on the table as a distrac- tor; T1 w/ complex rephrasing: where we rephrase the task description in 10 different unseen ways using GPT-4; Unseen task T4: the task is "Weight the object using the scale then place it back on the table." " which is never included in VIMA or xAm--Action The results are presented in Tab. [5) Our model demonstrates robust performance on generalized tasks, achieving a 75% success rate in TI with distractors. This suggests that our VLM retains its generalization ability even after being trained on our instruction dataset. With a small amount of in-domain vision data, it can achieve strong performance on previously unseen tasks.
Table 5: More generalization results of D-inBC Aux (C). The model is tested on three unseen tasks using two protocols and it shows strong performance with minimal finetuning on in-domain vision data.
Protocol
ZS
FT on xArm-Action
T1 w/ distractor (%)
10
75
T1 w/ complex rephrasing ( (%)
0
40
T4 (%)
0
20
Summary. Our findings highlight the strong generalization ability of LLaRA. When pretrained on simulated VIMA data, LLaRA requires significantly less effort and data to adapt to real-world tasks. By leveraging image coordinates, LLaRA establishes a clear, straightforward mapping between vi- sual inputs and robot actions. In contrast, RT-2 Style struggles under limited data conditions, largely because of the extensive data needed to learn the semanties of newly introduced tokens.
We discuss all implementation details and qualitative results in Appendix
7 CONCLUSION
We present LLaRA. a framework that turns a pretrained vision language model (VLM) into a robot policy using curated instruction tuning datasets. Firstly, we create a conversation-style instruction- tuning data using existing behavior cloning data and align the robot action to image pixel coor- dinates. Instruction tuning the VLM on such data confirms the effectiveness of our framework, particularly when the training data is limited. Then we construct auxiliary datasets using the same robot trajectory data through self-supervision, without any additional action labels. Experiments across several synthetic environments and robot manipulation tasks in the real world validate the effectiveness of our proposed LLaRA framework. We believe the new observations Assion-Language Action Model (VLA) representation transfer we share with this paper are valuable not only to the robot learning community but also the computer vision community.
the VLM to
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government foundation [24ZB120] Research of Human-centered tonomous intelligence system original technology]. The authors would like to thank Bohong Chen for verifying our code implementation and reproducing experiment results, well Xiaodong Liu and Hanyi Yu for their valuable input.
REPRODUCIBILITY STATEMENT
The code, datasets, and pretrained models are available at httgs//gtthubbcom///ss/KeeLLLLa
REFERENCES
Meta AL. Llama 3, 2024. URL https: /at.///laaaaaaaammaam monaaammtacomm iimeta.com/1. 11mma3/ 24.
Accessed: 2024-06-
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptistt Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar 2308 /2966, 2023.
Yakoub Bazi, Laila Bashmal, Mohamad Mahmoud Al Rahhal, Riccardo Ricci, and Farid Melgani. Rs-llava: A large vision-language model for joint captioning and question answering in remote sensing imagery. Remote Sensing, 16(9):147)) 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding. Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- lartoo-lnnuugg--cce models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 20n.:32/77///8 158/8, 2023a.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo. Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Roboties transformer for real-world control at scale. Robotics science and systems (RSS), 2023b.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as can, not as say: Grounding language in robotic affordances. In Conference Robot Learning (CoRL), pp. 287-318. PMLR, 2023c.
Mu Cai, Haotian Liu, Dennis Park, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, and Yong Jae Lee. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh. Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing languaee models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14455- 14465, 2024.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Imm's referential dialogue magic. arXiv preprint arXiv:2306. /5/95, 2023.
11

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
htthtts::
//1 hamsss.org///......
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language mod-
els. ArXiv, (ttp/2208.1221/ 2022. URL http://///pp..smaantiioc.ooomm CorpusID:25192931
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the International Conference on Computer Vision (ICCV), pp. 1422-1430, 2015.
Danny Driess, Fei Xia, Mehdi SM Saijadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.0333
2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the International Conference on Computer Vision (ICCV), pp. 2961-2969, 2017.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2023.
Jiaxing Huang, Jingyi Zhang, Kai Jiang, Han Qiu, and Shijian Lu. Visual instruction tuning towards general-purpose multimodal model: A survey, arXiv preprint 1.XXi:2312216602 2023a.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023b.
Nils Ingelhag, Jesper Munkeby, van Jonne Haastregt, Anastasia Varava, Michael C. Welle, and Dan- ica Kragic. robotic skill learning system built upon diffusion policies and foundation models. arXiv preprini arXiv:2403. 16730, 2024.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with mul- timodal prompts. In Proceedings of the International Conference Machine Leurning (ICML), 2023.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.099 09246, 2024.
Kartik Kuckreja, Muhammad S. Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fa- had S. Khan. Geochat: Grounded large vision-language model for remote sensing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), pp. 5639-5650. PMLR, 2020.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-ande vision assis- tant for biomedicine in one day. arXiv preprint arXiv:2306.00880 2023.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next: Tackling multi-image, video, and 3d in large multimodal models, June 2024a. URL htttps://llaaa- wit..i.....t.h...... io/blog// 2024-06-18 16- next- interneerree leaeel
12

---

Published as a conference paper at ICLR 2025
Xiang Li, Jinghuan Shang, Srijan Das, and Michael S. Ryoo. Does self sulf-super learning really improve reinforcement learning from pixels? In Advances in Neural Information Processing 2022.
Systems (NeurIPS), volume 35, pp.
Xiang Li, Varun Belagali, Jinghuan Shang. and Michael S. Ryoo. Crossway diffusion: Improving diffusion-based visuomotor policy via self-supervvsed learning. In IEEE International Conference on Robotics and Automation (ICRA), pp. 16841-16849. IEEE, 2024b.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500, IEEE, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 740-755. Springer, 2014.
Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03174 2024a
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition 2024b.
(CVPR), pp.
Xiao Liu, Da Yin, Chen Zhang, Yansong Feng, and Dongyan Zhao. The magic of if: Investigating causal reasoning abilities large language models of code. In Annual Meeting of the Associa- tion for Computational Linguistics, 2023b. httpst/::: /api. semant cittchcsshhss. org/
CorpusID:25896811
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling ppcabulcyy object detection.
Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Edward 'sang-Weii Lee, Kuang- Huei Lee Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07877 2024.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406.. 11855, 2024.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Robotics science and systems (RSS), Delft, Netherlands, 2024.
OpenAI. GPT-4 technical httpps:/arrxiv..orr////2223333
2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arxiv:2304.077 193, 2023.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain,
Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:23 10.008600 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
13

---

Published as a conference paper at ICLR 2025
Lerrel Pinto and Abhinav Gupta. Supersizing self sule-vuuervision: Learning to grasp from 50k tries and 700 robot hours. In IEEE International Conference on Robotics and Automation (ICRA), pp- 3406-3413. IEEE, 2016.
Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Erran Li. Affordancellm: Grounding affordance from vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference Machine Learning (ICML), pp. 8748-8763. PMLR, 2021.
Kanchana Ranasinghe, Muzammal Naseer, Salman Hameed Khan, Fahad Shahbaz Khan, and Michael S. Ryoo. Self-supervised video transformer. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2864-2874, 2022.
Kanehana Ranasinghe, Xiang Li, Kumara Kahatapitiya, and Michael Ryoo. Understanding long videos in one multimodal language model pass. arXiv preprint arXiv:2403. 16998, 2024a.
Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaced, Michael S. Ryoo, and Tsung-Yo Yu Lin. Learning to localize objects improves spatial reasoning in visual-Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo. Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Freitas de Nando. generalist agent. In Trans. on Machine Learning Research, 2022.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with momentum predictive representations. In Pro- ceedings of the International Conference on Learning Representations (ICLR), 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1134 1141. IEEE, 2018.
Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, and Michael S. Ryoo. Starformer: Transformer with state-action-reward representations for visual reinforcement learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 462 479. Springer, 2022a.
Jinghuan Shang, Xiang Li, Kumara Kahatapitiya, Yu-Cheol Lee, and Michael S. Ryoo. Starformer: Transformer with state-action-reward representations for robot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-16, 2022b.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle: Visual prompt engineering for vlms. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay. Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11530 IEEE, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followinn llama model. https::/ge thub. hoom/aatsus labi habpstante stanford_a lpaca 2023.
Omkar Thawkar, AbdeIrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muham- mad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radio- graphs summarization using medical vision-language models. arXiv: 2306.07971, 2023.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09228 2023.
Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. IEEE Access, 2024.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. ArXiv, abs/2103.01950, 2021. URL https://api
232092596
Jianren Wang, Sudeep Dasari, Mohan Kumar Srirama, Shubham Tulsiani, and Abhinav Gupta. Ma- nipulate by seeing: Creating manipulation controllers from pre-trained representations. In Pro- ceedings of the International Conference on Computer Vision (ICCV), pp. 3859-3868, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's ' perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint 1wx.i::332..31333 2023a.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. In Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 10674-10681, 2021.
Takuma Yoneda, Jiading Fang, Peng Li, Huanyu Zhang, Tianchong Jiang, Shengjie Lin, Ben Picker, David Yunis, Hongyuan Mei, and Matthew R Walter. Statler: State-maintaining language models for embodied reasoning. arXiv preprint arXiv:2306. 17840, 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any gran- ularity. ArXiv, abs/2310.07704, 2023. URL https: semantisceantttccholoo... org/
CorpusID:263834720
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, al. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296, 2023.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: jision-languee model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 10721, 2024.
Sukmin Yun, Jachyung Kim, Dongyoon Han, Hwanjun Song, Jung-Wor Ha, and Jinwoo Shin. Time is matter: Temporal self-supervision for video transformers. arXiv preprint arXiv:2207.0900 2022.
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXhv:2305.18279 2023.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Puro- hit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint S00i:22044005999 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- traiirain In Proceedings of the International Conference on Computer Vision (ICCV), pp. 11975-1 1986, 2023.
15

---

Published as a conference paper at ICLR 2025
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of regtees-f-iiterest.. interest. arXiv preprint arXiv:2307.03601, 2023.
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal 1lms. arXiv preprint arXiv:2307.0858/ 2023.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d vision-languagg-actiinn generative world model. arXiv preprint 10Xiv22003.06611, 2024.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint wrxiv:2412.10344 2024.
APPENDIX
The appendix covers all the implementation details, expanded experiment results in both simulation and real-world, further discussions, raw results, and all the prompt templates we used in the paper.
A IMPLEMENTATION DETAILS
In this section, we provide more implementation details regarding dataset preparation, model train- ing, inference, and RT-2 style baselines.
A.1 DATASET PREPARATION
In this subsection, we give more details and examples of the datasets used in this paper.
A.1.1 BUILD inBC DATASET FROM EXPERT TRAJECTORIES
We formulate the dataset in single-image single-turn conversation setting to emulate policy where the user queries the Vision Language Model (VLM) with the current observation, and the VLM generates response that can be directly translated into concrete action, as introduced in Sec. The image and text from the current observation can be directly processed by the VLM using a fixed template, while the conversion of numerical actions into text is facilitated through the use of normalized 2D image coordinates. Tab. g contains an example using a VIMA- VIMA-Bench sample.
Given the current limitations of LLaVA (Lu al. 2023a). to optimize performance, propose two techniques:
Action history in query. Each query to the VLM includes a history of actions previously executed in the episode. This approach enhances the VLM's understanding of task context within the constraints of a single-turn conversation. (See the orange text in Tab. X)
Plan 'twice' act once multi-step planning. In action generation, the dataset is prepared such that the VLM is designed to generate all successive actions in the response, literally performing multi-step action planning. However, only the first action is executed. For the next state, we query the VLM again with a new observation and a question and only take the first action generated by the VLM.
16

---

Published as a conference paper at ICLR 2025
These designs proved beneficial in our ablation study, detailed in Tab. 14
The rest of this section details the process on VIMA-Bence (Jiang et al. 2023). ). VIMA-Bende introduces 17 simulated robot manipulation tasks using multimodal prompts that combine text and reference images. The full accompanying dataset includes 660k expert trajectories, covering 13 of the 17 tasks. All tasks occur in environment where robot arm, equipped with either spatula or suction cup, is positioned alongside a flat table. Multiple objects with diverse textures are randomly initialized on the table according to the specific task setting.
Each episode in the dataset features a multimodal task description that clarifies the episode's goal, incorporating images referred to as 'reference images'. Two third-person view cameras capture the scene, providing top-down view and a front view looking down at the table. The dataset includes RGB, instance segmentation masks and meta information (e.g., textures and shapes) of the objects in images for the reference images and the observations captured by these cameras. We first extract the bounding boxes of each object from the instance-level segmentation mask, as an object detection oracle. Additionally, the dataset contains the expert action sequence required to complete the episode. The action space consists of two poses: for the robot equipped with a spatula, these poses indicate the start and end points of push; for the robot with suction cup, they specify where the robot picks up and places an object.
Due to the constraints of LLaVA (Liu et al. 2023a) and our findings presented in Tab. 15 in each conversation, only the current image from the front view camera is retained as the visual input to the VLM. Other reference images are substituted with short captions that describe the texture (color) and shape of the referred object, which are details extracted from the meta information (See inBC in Tab. D. However, in principle, there is nothing stopping us from extending the current framework to multi-image setting, taking advantage of a VLM that can handle multiple interleaving images in a conversation.
A.1.2 BUILD D-inBC FROM inBC
While inBC in Tab. 8 has shown its great power in many aspects when the reference image contains scene that has multiple objects instead of one, the inBC will fail to deliver any useful information (e.g., Tab. Dp. To address this issue, we utilize an object detector to parse an image of a scene into a list of objects with its corresponding bounding box. The new dataset is named D-inBC because reference image will be 'described' as a list of objects now. Tab. @shows an example where D-inBC delivers more critical information than inBC.
A.1.3 D-inBC (L): DESCRIBING THE REFERENCE IMAGES IN NATURAL LANGUAGE
Following the suggestion from the reviewer, we trained a variant of D-inBC, which completely re- moves all the coordinate numbers in the prompt even from the reference images. Instead, we use a pretrained VLM. Bwwen--LL-B--nntreee
(Wang et al. 2024), to generate natural language descrip- tions of all reference images, omitting any use of image coordinates. These textual descriptions replaced the original lists of image coordinates in the input prompt for the D-inBC setting, resulting in variant we refer to D-inBC (L), where (L) stands for language. Then we train on VIMA-Sk and the conversations describing the reference image. During inference, LLaRA first generates tex- tual descriptions of all reference images using its capabilities and subsequently leverages the generated text to perform the task.
The following prompt is used to generate image description: "Write short descriptions of each object in the image, specifying their color, texture, shape, and relative positioning.
Tab. presents an example of generated image description of the [frame_0) 0) in Tab. 9 We present the performance of D-inBC (L) at Tab. Tab. (VIMA) and Tab. 7 worl...
In general, D-inBC (L) achieves comparable but slightly worse performance than D-inBC, which uses structured text to describe the reference images.
17

---

Published as a conference paper at ICLR 2025
Table 6: D-inBC (L) performance when trained on VIMA-8K
Method
L1 (%)
1,2 (%)
L3 ( (%)
inBC Aux (D)
59.2
58.8
52.1
D-inBC + Aux (D)
62.3
57.7
52.1
D-inBC (I.) Aux (D)
61.9
54.2
55,4
Table 7: D-inBC (L) real-world experiment results.
Method
Add. Data
T1 (%)
T2 ( (%)
T3 (%)
Avg. (%)
inBC Aux (D)
XArm-Det
100
95
70
88.3
inBC Aux (D)
XArm-Action
70
D-inBC (L) + Aux (D)
xArm-Det
65
90
70
75
D-inBC (L) + Aux (D)
xArm-Action
75
90
75
A.1.4 AUXILIARY DATASETS
The auxiliary datasets created using the template outlined in Fig. During dataset generation,
for each sample, one template is randomly selected from pool of 15 templates. These templates
were initially rephrased from single sentence using GPT-4 (OpenAl 2023). The full list of the
pools are listed in Tab. 30 Tab. 31 Tab. 29 Tab. 32 Tab. 33
Tab. 34 Tab. 35 Tab.
36
Tab. 37
Tab. 38 and Tab. 39
The qualitative examples available Tab. and Tab.
A.2 TRAINING
We initiate training using a pretrained LLaVA-1.5-7B (Liu et al. 2024b) model and finetune all parameters, including the language model and the projection layer, with the exception the vision encoder. The training settings closely align with those of the original LLaVA stage 2. Specifically, we utilize single-cycle cosine annealing scheduling with 0.03 warm-up ratio and a maximum learning rate of 2 x 10 However, for VIMA-0.8k and VIMA-8K, employ batch size of 32, whereas for VIMA-800 we restore the batch size to 128.
A.3 INFERENCE
During inference, for the models trained inBC, new conversation is initiated each step. The model is queried with single current image observation from the front-view camera and an in- struction that has been prepared the same format as the dataset. Additionally, prompt randomly selected from Tab. 28 is added to the instruction during inference only, although it appears to have limited impact in retrospect (see Appendix [B7]).
Model sampling is disabled during training, ensuring that the model consistently outputs the token with the highest probability. This approach designed to maximize the consistency and reliability of the model's responses.
For the models trained on D-inBC, before we query the VLM, we would first run object detection on all reference images and take the detection results to fill the instruction template of D-inBC. Besides this, the settings are identical to the models introduced ahead.
In this paper, we explore three approaches to object detection. The first method involves using single query from Tab. 30 and Tab. and employing the same VLM as the policy to perform object detection.
The second approach utilizes a et al., 2016) backbone pretrained on the COCO dataset (Lin al. 2014) and subsequently finetuned using the VIMA dataset. A suffix 'OD' will be added to the model name if the model uses this method.
(He et al. 2017) model, which features a ResNet50 (He
Finally, we test the versions that use the groundtruth detection results. suffix 'Oracle' will be added to the model name if the model uses this information.
18

---

Published as a conference paper at ICLR 2025
Table 8: Comparison between our converted instruction tuning datasets inBC and D-inBC for a VIMA-Bench sample. The task description of the episode is in green. The description of an object in the reference image (oracle detection results) is in cerulean. The action is in orange and the action history is in magenta.
{dragged_oby _obj]
{base_obj]
Task
First put (dragged_ obj} into {base_obji then put the object that was pre- viously at its south into the same (base_ohe obj).
Input
front camera view.)
<task>Firrt put <p>rainbow letter T</p> into <powooden bowl</p>> then put the object that previously south into the same <p>wooden howl</pa </tas>>>//>>>>>>>>
Every action you take must include two locations in the format of <b>(x. y></b> and one clockwise rotation angle in the format of <>>//<<>> The first location is the image coordinate where you use suction cup to pick up the object, and the second location is where you place the object. The image conrdinate ranges from O to I. The rotation angle indicates how many degrees you rotate object clockwise, and ranges from -359 to 359.
inBC
You have finished: Step Pick up the object <b>(0.480, 0,367)</b>, <r> degrees, and drop at <h>(0.727, 0.547)</ho.
Output
Step 2: Pick up the <p>rainbow letter V</p> <b>(0.500,, 0.617)</b>, rotate <r>[0]</>>> degrees, and drop at <h. 0.617)</h>.
Input
(current front camera view.)
<task>-First put <porainhow letter T</p> at <bs-(0.500, 0.594), 10.102. 0.188)</b> into <p>wooden bowl</p> <bo<00.457. 0.531), (0.195. 0.328|<</b then put the object was previously at south into the same <pswooden bowl</pp> <bo<(0.457, 0.531), (0.195, 0.328) </b>///ka Every action you take y></h> and one clockwise rotation angle in the format of <>>//<<>>> The first location is the image coordinate where you up the object, and the second location is where you place the object. The image coordinate ranges from 0 to 1. The rotation angle indicates how many degrees you rotate the object clockwise, and it ranges from -359 to 359.
D-inBC
include locations in the format of <b>(x,
suction cup to pick
You have finished: Step 1: Pick up the object <h><004800, 0.367)</b>, rotate <r>(0) (0]/</>/>>>/// degrees, and drop at <b>(0.727, 0.547)</b>.
Output
[Same inBC
19

---

Published as a conference paper at ICLR 2025
Table Another comparison between two converted instruction tuning datasets. example, the reference images in the task description depict reference images with multiple objects. The task description of the episode is in green. The description of an object in the reference image (oracle detection results) is in cerulean and the action is in orange.
(frame 0}
[frame_1]
Original BC
[frame_2] 2}
Task
Stack objects in this order (frame 0) {frame [frame_2) 2}
Input
(current front camera view.) <tasko-Stack objects
order frame_ 0 frame frame 2.</takk> include locations the format of <h>(x, y)</b> and clockwise rotation angle in the format of <r>[r]<r>> The first location is the image coordinate where you up the object, and the second location is where you place the object. The image coordinate ranges from 0 to 1. 'The rotation angle indicates how many degrees you rotate the object clockwise, and it ranges from -359 to 359.
Every action you take
suction cup pick
inBC
Output
Step Pick up the epogreen paisley letter Vetps at <ho-(0.65, 0.617)</b>, rotate <r>10]</>>> degrees. and drop at <b>-(0.711. 0.617)</b>.
Step Pick up the epored letter Velpo at <bo(00281,, 0.633)</hs, <r>(0]</r>> degrees, and drop it <b>(0.770, 0.633)</bb.
Input
(current front camera view.) <taskoSStack objects in this order
<seene><p>red letter V<lp> at <bo<00.254, 0,570), [0.094, 0.156)</b>.
<pegreen paisley letter Velp> <b><00.000, 0.570), (0.094. 0/156)</b>, <p>rainbow letter V<lp> at <b>(0.746, 0.570), (0.094. 0. bb15///bscccceee>>
<scene><p>red letter V</p> <b>(0.254, 0.570), [0.094, 0.156)</b>. <p>rainhow letter V</p> at <h>(0.746, 0.578), (0.094, //133///b>.
<p>green paisley letter V<lp> <b><0.746, 0.516), (0.094.
0. <bo./cebessscceee>>>
D-inBC
<scenc>><p>rainbow letter V</p> at <b>(0.746. 0.602). (0.094.
0.086]=/b>. <pagreen paisley letter V<lp> at <bs-(0 0.531), (0.094, </12 <pored letter V</p> <b>(0.746, 0.469), (0.094. (ool)///sc//ceess
Every action you take include locations the format of <h>(x, yk//>> and clockwise rotation angle in the format of <r>|r]<r>> The first location is the image coordinate where you up the object, and the second location is where you place the object. The image coordinate ranges from 0 to 1. The rotation angle indicates how many degrees you rotate the object clockwise, and it ranges from -359 to 359.
suction cup pick
Output
[Same as inBC
20

---

Published as a conference paper at ICLR 2025
Table 10: Image description used in D-inBC (L), an example of (frame 0f in Tab.
The image displays three distinct objects, each with unique colors, shapes, textures, and spatial relationships.
1. Object on the Left:
Coloo:: Red
Texture: Matte
Shupe: Triungular
SSatial Relationship: Positioned to the far left of the image.
2. Object Middle:
-Color: Green and white
Texture: Speckled
Shape: Arrowhead
Spatial Relationship: Situated to the right the first object.
3. Object the Right:
-Color: Multicolored (red, blue, green, yellow)
Texture: Glossy
Shupe: Trapezoidul
Spatial Relationship: Located to the far right of the image.
These objects are arranged in lincar sequence, with each subsequent object placed the right of the previous one, creating horizontal alignment. The colors and textures vary, adding visual interest to the composition.
A.4 RT-2-STYLE BASELINES
For the RT-2-Style baseline, we follow the original RT-2's design (Brohan et 2023a) and adapt special tokens to present the numerical action in VIMA action space. The original action space of VIMA contains two points for pick and place. Each point contains a 2D coordinate in robor coordinates for position and a 3D orientation of the gripper. Since we only have yaw rotation in the dataset, we simplify two 3D orientations into one yaw angle. So the action space is now Degrees of Freedom (DoF): 4 numerical values present the 2D pick and place locations in robot coordinates, and I value presents yaw rotation. our implementation, the procedure begins by normalizing each element of the action to range from 0 to 1. We then quantize these values into 256 bins and map the resulting discrete values to tokens indexed from 31000 to 31255 the tokenizer. The dataset employed is similar to inBC, but it omits the prompt for the output format, and the action is represented by the mapped special tokens. Additionally, we have prepared a version similar to D- inBC, named D-RT-2 Style, which takes advantage of object detection reference images. D-RT-2 Style (1) is similar to DDRTT-2 Style. However, four special tokens present pick and place positions in two normalized 2D image coordinates for each action, and one token presents the rotation, instead of an action in VIMABench action space. It is the same action presentation generated by our LLaRA but our method responds natural language while D-RT-2 Style (I) responds in special tokens.
B EXPANDED SIMULATION EXPERIMENTS
In this section, we include more experiment results. All the numerical results reported in this paper, especially in the figures, collected in Tab. 21 and Tab. [22] (VIMA-0.8k). Tab. 23 and
Tab. 24 (VIMA-8K), Tab. and and Tab. 26 (VIMA-80K), and Tab. 27 (full VIMA dataset).
B. CLARIFICATION ON L4 RESULTS
We adopted the testing protocol from VIMA-Bench (Tiang et al., 2023), conducting evaluations across four levels of difficulty, from L1 to L4. However, during our analysis, we discovered an inconsistency in the training set: the rotation information for the robot end effector was recorded as zero for all sweeping tasks where the end effector is spatula. Given the importance of the spatula orientation a sweeping task and the fact that sweeping tasks constitute 25% of the evaluations at the L4 difficulty level, concluded that our ability to accurately evaluate compromised. Considering that the original VIMA model released by the authors appears to include this rotation information, have chosen not report the results for L4 in our study.
method at L4 was
21

---

Published as a conference paper at ICLR 2025
Table Qualitative examples of auxiliary datasets. In this table, the examples all the datasets take the same input image as shown in the row. The description an object or scene (oracle detection results) in blue. The action is in orange.
Image Input
(current front camera view.)
Input
Could you tell me the locution of <p>red letter V-c/p> in the image? Use the format <b>(x. y). [w, hj</b>, where and y denote center of the bounding and and the width and heighr. Coordinates start at the left corner and should be normalized between and 1.
Object Localization
Output
<p>red letter V<lpp> at <b>(0.254, 0,570), [0.094, 0.156)
Input
Enumerate and describe every object found in the image. For each object, utilize the format <h>(x, y), [w, hJ<//>, where x, y are bounding box coordinates and bounding hox. The coordinates begin the top left corner and normal- ized between 0 and 1.
the dimensions (width and height) of the
Object Detection
Output
<scene>
<pered letter Vel/p> <bs-(0 0.570), [0.094, 0.156) </b>) </b>. <p>green paisley letter V<dpp> <b><(0.500 500, 0.570). (0.094, 0.156]</b>> <p>rainbow letter Ve/po> at <b.(00 0,570), (0.094, 0.156 </bo> <p>green paisley block</p> <bo-(0.668, 0.414), (0.094, 0.227)</b>. </scene>
Input
Can you explain what
done to adjust scene shown the im- age resemble the second scene? The second <scene><p>red letter
V</p> at <b>(0.255 2544, 0,594). 0.094, 0.156</b>. <p>green paisley letter
V</p> at <b>(0.742 0.547), 0.098, 0,156c/b>. <p>rainbow letter V</p>>
at <bo<00.746, 0.609), 0.094, 0. 0.125e/bo. <pegreen paisley blocke/ps at <h>-00.668, 0.414), 0.094, 0.227</hs.c/cennn consists of object bounding boxes provided in the format <bo(x, y). [w. h//b>, Here, and represent the center courdinates, and and h are the width and height. The courdi- nates should start from the top left comer and be normalized to scale of Every action you must include two locations the <b>(x, y)//b>> und one clockwise rotation angle the format of <>>/r)</r>> The first location the image coordinate where
Action Prediction
use suction cup to pick up the object, and the second location is where you place the object. The image coordinate ranges from 1. The rotation angle indicates how many degrees you the object clockwise, 359.
ranges from -359
Output Pick up the <p>green paisley letter V<dp>> at <bo<00.465, 0.617)</b>, rotate <r>[0]</>>> degrees. drop it <b>(0.711, 0.617)</bo.
Input
An image depicts a scene containing multiple objects. Now you pick up the <pagreen paisley letter V<lps (6.60.665, 0.617)c/s, rotate <r>(0)</>> degrees, and drop at (0.710 0.617)</bo, what will the seene look like? Write the list of object hounding hoxes. The bounding boxes should formatted as <h>(x, y), [w, h|</hs, where x and denote the center courdinates, and w
width height. The courdinates start from the top left corner and normalized scale of 0 1.
Future Prediction
Output
epered letter Velpo at cho-(0.254, 0.594), (0.094, 0.156) <seenc>
<pegreen paisley letter V-dpp <h><(0 742, 0.547), (0.098, 0. 0/156) <p>rainbow letter V<s/p> <b540. 0.609), (0.094, 0.125) <p>green paisley block</p> <b>(0.668,, 0.414), (0.094, 0.227)</b>. </scenc>
22

---

Published as a conference paper at ICLR 2025
Table 12: (Continued) Qualitative examples of the auxiliary datasets. In this table, the examples of all the datasets take the same imput image shown in the first row. The description of an object or scene (oracle detection results) is in blue. The examplar is in purple.
Image Input
(ourrent front camera view.)
Input
Can you explain the relative spatial positions of <pored letter V</p> com- pared to <pogreen paisley letter V<ip> in this image? Use terms such as left, right, above, below, etc. Also, determine the center distance and the Euclidean center distance between them. Your output should match this format: <p>green paisley hlock</pp is left and top from sporainhow letter V-dip> with 2d center distance (x.y) of <db(-0.078, -0. 16)x</ds> and euclidean center distance of <2s0. 175-dess The coordinates are image co- ordinates normalized scale of 0 to starting from the top left comer.
Spatial Relationship
Output
<p>red letter V</p> is left and bottom from <pegreen paisley letter V</p>> with 2d center distance (x.y) of <d>(-0.246, 0.000)c/db and euclidean cen- ter distance of <c>0.246-//c>
Input
At the initial timestamp, the image shows a scene, and the second image described as <scene><p>red letter Vedp> <b><00.254, 0.594), [0.094, 0.156] </bo,, <po-gree paisley letter V-dpa <h2-(0.742, 0.547), (0.098, 0.156] <p>ruinbow letter V<lp> at <ho(0.746, 0.609), [0.094, 0.125)</bo. <po-green paisley block<//p> at <h>(006668, 0.414), (0.094, 0,27)/</b_c/s/cno/ depicts the timestamp. Can you describe the change in the relative position of <p>red letter V</p> compured to <p>green paisley letter V-dp>> between these two timestamps? Use relative distance terms such as getting closer moving further away, etc. Also, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must follow this format: <porainbow letter V<lp> moves far away from <po-green paisley block-/po.. cen- ter distance (x.y) <psgreen puisley block</p> from <p>rainbow letter V-clp> changes by <db(0.000, -0.119))/d>> and Huclidean center distance between them 
0.036e/es. The coordinates are image coordinates nor- malized to a scale 0 starting from top left corner.
Temporal Relationship
Output
<p>green paisley letter V</p>> moves far away from <p>red letter V</p>>, 2d center distance (x.y) spored V<dpp> from <pogreen paisley letter V<c/p> changes <db(-00242, 0,047)</d> and Euchidean center distunce between them <050.244cllo>.
B.2 EXTENDED COMPARISON TO VIMA AND OTHER BASELINES
We would first clarify the difference between our method and VIMA (Jiang et al. 2023) in terms of the inputs. VIMA takes both front and top view images from the environment while ours only takes the front view. We test the most capable model released by VIMA, which is trained on 660k expert trajectories, as well as other RT-2-Style baselines trained on the same whole 660k dataset. The results are listed in Tab. [3] Compared to VIMA (Jiang et al. (023) and other baselines, our best model not only achieves better performance but also requires less input and is trained on only 12% the data used in VIMA.
B.3 WILL BASELINES BENEFIT FROM LONGER TRAINING?
The introduction of auxiliary datasets in the training set naturally increases the number of optimiza- tion steps per training epoch. In this subsection, we confirm that the performance improvement of LLaRA with auxiliary datasets is due to the data itself, rather than simply more optimization steps. To validate this, we trained several baselines without auxiliary data for longer epochs on three VIMA subsets. As shown in Fig. g Fig. [0] and Fig. [I] extending the training epochs does not enhance performance, indicating that the auxiliary data provides additional information that boosts model performance.
23

---

Published as a conference paper at ICLR 2025
Table 13: Comparison VIMA (Jiang et al. 2023 and other baselines. Our best model only achieves better performance but also requires less input and trained only 12% of data used VIMA.
Method
Config
Data
L1 (%)
L2 (%)
L3 [%)
VIMA (Jiang al. 2023
VIMA-200M Oracle
100%
80.7
81.9
77.9
12%
56.9
53.1
46.2
RT-2 Style
100%
65.0
59.6
42.5
D-RT-2 Style + Oracle
12%
74.2
70.4
69.2
D-RT-2 Style
D-RT-2 Style + Oracle
100%
86.2
88.1
75.8
D-RT-2 Style (7) + Oracle
12%
58.1
54.2
55.4
D-RT-2 Style (I)
D-RT-2 Style (I) + Oracle
100%
70.0
70.0
59.2
D-inBC Oracle
12%
87.3
85.4
82.1
LLaRA (Ours)
D-inBC + Aux (B) Oracle
12%
90.0
88.1
79.2
D-inBC + Aux (D) Oracle
12%
83.8
88.1
78.8
30
25
L3 Succiss fann H61
25
Rate
20
20
Succes
20-
15
Style
30
30
10
D RT Styfell
D-RT-2-Style Oracte
Oracle
LLSRA
LEARA (D-IBCC Orack)
Relative number
Relative number ttertions
Relstive number therattons
Figure 9: Model performances on VIMA-0.8K for longer epochs.
70-
60
60
60-
Sucoess hatu 1961
50
50-
Surcess
40
40
30-
30
RT-2-Style
20
20
20
D-II-2-Siyeelt Drocle
D-R1-2-Siyle -Grace
10
LEARA (iBC)
0-
LEARA (D-iNCC Drack))
Rebotve number itorstions
Ralative numbor lecations
Reistive rumber Iberationss
Figure 10: Model performances on VIMA-8k longer epochs.
L3 Success Pant (N)
Success
70
70
RT---6iyle
50
D 2- Grade
D D-RT-2-Syyelll Orsile
50
4BA (rBC)
LLSRA Orack))
Helative
Figure 11: Model performances on VIMA-80K for longer epochs.
B.4 ABLATION ON ACTION HISTORY AND MULTI-STER PLANNING
As described in Appendix A.I.I we enable action history and multi-step planning when generating inBC and D-CnBC. Tab. shows that these designs are helpful.
24

---

Published as a conference paper at ICLR 2025
Table 14: Ablation action history and multi-step planning. His. stands for enabling action history and Plan means enabling multi-step planning. models are trained VIMA-8K epochs.
Method
His.
Plan
L1 ( (%)
L2 (%)
L3 (%)
inBC
57.3
46.2
42.9
V
inBC
X
43.5
36.9
36.7
inBC
X
45.8
39.6
35.8
D-inBC + OD
V
63.5
51.5
50.0
D-inBC OD
V
58.5
41.5
47.1
D-inBC + OD X
53.5
39.2
37.5
B.5 ABLATION ON MULTIPLE IMAGE INPUTS
This ablation studies multiple image ipputs within single conversation, Each image is processed by the vision encoder and the projection layer to generate a series of tokens. These image tokens are then integrated into the textual conversation at the points where the corresponding images are referenced.
However, because LLaVA is trained with one image per conversation instead of an interleaving style. The performance drops significantly when applying multiple image inputs, listed Tab. [5
Table 15: Ablation on multiple image inputs (Mul.). models are trained on VIMA-Sk epochs.
Method
Mul.
L1 (%)
L2 (%)
L3 ( (%)
inBC inBC
57.3
46.2
42.9
/
44.6
27.7
34.2
D-inBC OD
X
63.5
51.5
50.0
D-inBC + OD
/
27.7
24.6
34.2
D-inBC + Aux (C)
X
64.6
58.8
49.6
D-inBC Aux (C)
/
31.2
28.5
27.1
B.6 ABLATION ON OBJECT DETECTOR
This ablation studies three types of object detectors in this paper: the VLM model itself, an external object detector separately trained the training set (the methods with suffix OD), and oracle from the groundtruth dataset (Oracle). Fig. 12 shows that reliable object detector is highly benefi- cial, enhancing the accuracy of image-based inputs and consequently improving model performance.
181
00.
01
80-
40
BRTT-Syy : 00
D-I---9 Syy Ondk
20-
DiBC Am (D)
DinC Auc
eplcode
Figure 12: Impact of different object detectors.
25

---

Published as a conference paper at ICLR 2025
B.7 ABLATION ON ACTION PROMPT DURING INFERENCE
This ablation studies the effect of the prompt sentence used in action generation. By default, during test time we formulate an instruction similar to Tab. to query the VL.M. In addition, we randomly sample sentence from Tab. 28 named action prompt, and add the instruction. We study the following scenarios to understand the effect of this action prompt selection:
random: this is the default behavior, random action prompt will be added to each instruc- tion;
no no_prompt: none instruction has the action prompt:
prompt Pe: where Pe E [0, 14]: all the instructions in this setting use the pd prompt (0-
index) in Tab. 28
The results are presented in Fig. where two variants of D-inBC are trained on VIMA-80K. In general, the selection of the action prompt does not result a significant difference in performance except that D-inBC Aux (D) seems perform better L1 and L3 without the action prompt.
2
D4NIC PAN 60) Orace
Figure 13: Effect of prompt sentence for action generation.
B.8 EXTENDED EXPERIMENTS ON RT-2 Style VARIANTS
As suggested by the reviewer, we evaluate two new variants of the RT-2 Style settings, RT-2 Style (2R) and RT-2 Style + Aux (initially reported in Tab. B. In the original RT-2 Style setting, single token is assigned to present the rotation in the action space. However, considering there are only 256 possible values of a token, causes an unfair comparison between our method and RT-2 Style in terms of action space of rotation and may compromise RT-2 Style's performance. Therefore, we propose RT-2 Style (2R), which uses two tokens to represent rotation, resulting 65,536 options to encode rotation. This new configuration introduces significantly larger action space compared to our method. At the same time, we evaluate another variant RT-2 Style + Aux (D) by simply mixing RT-2 Style training data with auxiliary datasets used in L.aRA. Evaluation results of these models on VIMA-8k are summarized Tab. [6
Table 16: Performance of RT-2 Style variants on VIMA-8k
Method
L1 (%)
L2 (%)
L3 (%)
RT-2 Style
3.8
3.1
1.7
RT-2 Style (2R)
1.9
2.3
1.7
RT-2 Style + Aux (D)
36,9
35.8
32.1
inBC
57.3
46.2
42.9
inBC + Aux (D)
59.2
58.8
52.1
As shown in Tab. 16 additional token for rotation in RT-2 Style (2R) does not significantly change the performance of the RT-2 Style baseline, however, the introduction of auxiliary data significantly improves the performance of RT-2 Style, demonstrating the effectiveness of our method.
26

---

Published as a conference paper at ICLR 2025
B.9 ROBUSTNESS OF THE MODEL IN FAILURE GRASP SCENARIOS
As suggested by the reviewer, show the robustness of our method, especially the ability to recover from failure, we run additional tests in a revised VIMA environment. The revised environment has 10% chance that ignores the action from the model simulate the action failure, which we believe much higher than the actual case.
Tab. 17 shows that without any changes, existing model is robust to such failure.
Table Robostness evaluation results D-inBC + Aux (B) Oracle (VIMA-8OK, epochs)
Prob. of Failure
L1 (%)
L2 (%)
L3 (%)
90.0
88.1
79.2
10
87.7
86.2
75.8
B.10 FEW-SHOT EXPERIMENTS USING LLM VVMM
We expanded evaluation to include the few-shot performance of existing Large Language Mod- els (LLMs) and Vision Language Models (VLMs). Consistent with the inference settings described in Appendix A.3 we provided three additional examples from either inBC or D-inBC for each test scenario. The results of these evaluations are detailed in Tab. 18 where we examine three variables:
Vision: When this option is enabled, the model will have access the current image observation, allowing it to integrate visual data into its response. If disabled, the model operates solely as a text- based system.
Oracle: This variable is checked when the examples are drawn from the D-inBC dataset. It indicates that the examples provided for few-shot learning include descriptive textual information about the reference images, potentially offering richer context. Same-task: When this is checked, the examples used for prompting are from the same task type as those in the evaluation environment.
From the observed low performance, we infer that the data utilized for robot control significantly deviates from the conversation contexts in which these models were originally trained. This indi- cates pressing need further finetune the models specifically for robot control tasks to achieve adequate performance. Additionally, findings hint that visual input considerably enhances the functionality of GPT-4o.
Table 18: Three-shot performance of LLMs/ / VLMs
Output
Vision
Oracle
Same-task
L1 (%)
L2 (%)
L3 (%)
L4 (%)
GPT-4o
X
X
/
1.4
1.4
0.0
N/A
GPT-4o
X
X
0.4
0.4
0.4
0.0
GPT-4o
X
/
1.4
1.8
0.5
N/A
GPT-4o
X
X
1.2
0.4
0.4
1.2
GPT-4o
V
X
/
5.9
4.5
4.0
N/A
GPT-4o
/
X
6.9
5.8
4.2
1.2
Llama 3 70B (4-bit)
X
/
0.0
0.0
0.0
N/A
Llama 3 70B (4-bit)
X
X
0.8
1.2
0.4
1.2
Llama 3 70B (4-bit)
X
/
0.0
1.4
0.5
N/A
Llama 3 70B (4-bit)
X
X
0.0
0.4
0.4
0.0
LlaVA-1.5-1 LIaVA-1.5-78
V
/
0.9
1.4
0.5
N/A
/
X
0.0
0.0
1.0
1.2
LlaVA-1 1.6-34B
/
/
1.4
1.4
0.5
N/A
LlavA-1.66 1.6--116-44B
/
x
1.2
0.8
2.1
0.0
27

---

Published as a conference paper at ICLR 2025
C DETAILS ON REAL-WORLD ROBOT EXPERIMENTS
In this section, we provide more details on real-world robot experiments first introduced in Sec.
C.I ENVIRONMENT SETTING
We utilize xArm7 robot arm equipped with gripper and Logitech C140 RGB webcam posi- tioned above the arm to gather observations. The camera is mounted in a fixed position above and slightly in front of the arm, providing third-person view of the scene. The captured images have raw resolution of 640 x 480, which are then center-cropped to 640 x 320 to match the aspect ratio of VIMA images. The robot's action space includes 4 DoF for pick-and-plase positioning and I DoF for rotation, consistent with the simplified VIMA setup.
The object detector for all real-world experiments is an OWLv2 (Minderer al.) 2024 (ViT-base, patch size 16) running in one-shot detection mode. The object detector takes prompt image from the same domain and detects the objects in the observation. The list of plastic toys we used as [object] (and detected) is: (duck, croissant, cupcake, blueberries, carrot, banana, grapes, donut, tomato, corn, pepper)
C.2 GPT-40 BASELINE
In the GPT-4o baseline, the model is fed the same prompt as in RT-2, with additional prompt describing the robot arm: "You are operating a robot arm at a table to complete task specified between <task></task>,
Given the current image, you need to generate a series of actions to control the robot arm accomplist the task.". The model's answer is parsed extract the pick and place points.
C.3 REAL-WORLD ROBOT DATASET
We further collect 1,256 real-world images from the same real-world robot setting as the in-domain data. Out of these images, 721 feature single object placed randomly and uniformly the table by robotic arm, while the remaining images display multiple objects scattered across the table. We employed one-shot detection using OWLv2 (Minderer al. 2024) to identify and generate bounding boxes for each object in the images.
These images, along with their bounding box data, have been utilized to create two specialized in- domain instruction tuning datasets, each containing 2,447 single-turn conversations of equivalent size: XArm-Det and xArm-Action. The XArm-Det dataset includes conversations that mirror the structure from our auxiliary object localization and detection dataset, with 1,191 examples dedicated object localization and 1,256 object detection. In contrast, the Aamm-cction dataset is designed similarly to data for D-inBC, featuring two distinct tasks: Rotation: rotating an object (akin to Task T2 with 1,191 examples) and on top: putting one object on top of another (reminiscent of Task T3 with 1,256 examples).
In the Rotation task, the agent instructed to rotate a specific object by a random angle ranging from -180 to 180 degrees, while ensuring the object's position remains unchanged post-rotation. The expert action is straightforwardly generated by directly replicating the specified rotation angle from the task description and the pick and place points are set the centroid of the object bounding box from object detection.
In the Put Top task, applicable to images containing more than two objects, we randomly select two objects. The task involves generating a command that directs placing one object atop The expert actions, specifically the pick and place points, determined using the centroids of the bounding boxes of the two selected objects from object detection.
the other.
Tab. 19 shows examples of the real-world datasets collected.
28

---

Published as a conference paper at ICLR 2025
Table 19: Qualitative examples of our weal-world datasets XArm-Det and XArm-Action. The description of an object in the image (onc-shot detection results) is in blue. The task description of the episode is in green and the action in orange.
Image Input
Input
Could you tell me the location of epobluchery</p> in the image? Use the format <b>(x, y). {w. h/</b>, where and y denote the center of the bounding and and are the width and height. Coordinates start top left corner should be normalized between and
XArm-Det Localization
Output
<pabluchenycclp <8-80.478, 0.725), {0.058, 0.125)</b>.
Input
Describe all the objects <h><s, y). [w, h) </<//>> The and values are the courdinates for the center of the bounding box, while w and h represent its width and height. The coordinates should be normalized from the top corner, within range to
in the image, and list them using the format
Output
<seenc>
<p>cupcake</p> at <b>(0.4440, 0.128), [0.063, 0. 0.123)<//b..
XArm-Det Detection
spocorn-/pp> <bo-(0.237, 0.826), (0.061, 0.246)-/hh>.
<p>curott<ps> at <b>(0.608, 0.135). (0.152, ///b,)</b>,,
<p>donut</ps> <bo(0.396, 0.355), (0.079, 0.145/</bs
<p>grapo</pp> <b>-(03318, 0.534). (0.077, 0.143)</be.
<p>duck</p>> at <h><00.95,, 0.280), (0.066, 0.128]</hs.
<p>green pepper</p> at <b>(0.688, 0.411), (0.062, 0.187)</b>.
<p-blucheny<</p> at <b><(0.478, 0.725), (0.058, 0.12/////b.
<pstomatne/po at <ho,00.543, 0.421), (0.078, 0.154) <hscene>
Input
<task>Rotate the <p>cupcuke</p> 0. 0.123)</b>> by 550 l/grses.</hask>>
<b><0.440, 0.128). (0.063,
Every action you take must include two locations in the format of <he<x, y)</b> and one clockwise rotation angle in the format of <>>|)</>> The first location is image coordinate where you use suction cup pick up object, and second location where you place object. The image coordinate ranges from many degrees rotate the object clockwise, and ranges from -359 to 359.
XArm-Action Rotation
The rotation angle indicates how
Output
Pick up the <p>eupcake<c/p> at <b>00000400 0/128)</b>, rotate <>>|550kkin degrees, and drop it at <bo-(0.440, 0.128)</b>.
Input
Moae the <pscupeakee/ps-
<h>x0..400, 0.128). (0.063,
0. 0.123)</bb the top of the <pocorec/p> <bo200.377 0.826), [0.061, 0.246) (.246)</bbo-ddd <//bo<//task>>>
Every action you take must include two locations the format of <h>(x, y) and one clnckwise rotation angle in the format <r>(r]</>> The first location is image coordinate where you use suction cup pick up the object, and the second location where you place the object. The image coordinate ranges from to 1. The rotation angle indicates how many degrees you rotale the object clockwise, and ranges from -359 to 359.
xArm-Action
Put on top
Output
Step 1: Pick up the <p>cupeake</p> at <b>(0.440, 0.128) rotate <r>(0]<<>>> degrees, and drop at <b>(0.237, 0/826)</b>.
29

---

Published as a conference paper at ICLR 2025
C.4 EXTENDED WOOLD EXPERIMENTS
In addition, following the suggestion from the reviewer, we have added experiments with a new set of tasks that require more language understanding and spatial reasoning. These tasks include:
T5: Put all yellow objects into the bowl. There is a bowl and objects on the table. Between 1 and 3 of the objects are yellow. A success state is one in which all yellow objects are in the bowl and no non-yellow objects are in the bowl.
T6: Transport the baked goods to the large bowl. There is bowl and 3 objects on the table. Either or 2 of the objects baked goods. A success state is one in which all baked goods are in the large bowl and none of the non-baked goods are in the large bowl.
T7: Feed the pet healthy food. There is an animal on the table and 2 objects. One of the objects is a fruit or vegetable and the other from the animal to start. A success state is one in which the healthy food is sitting next to the animal while the unhealthy food is further away from the animal.
baked good. The 2 objects are equidistant
Without a proper language understanding, these tasks cannot be easily solved. These tasks are particularly challenging none of these tasks were seen in the training data in either sim real. Some of the objects in the scene (e.g., baked goods and bowl) were never seen by the model either during the robot data training, and each episode takes multiple steps to finish.
We compare RT-2 Style and LLaRA (inBC + Aux (D)) both first trained on VIMA-8k and then finetuned on xArm-Det, which the model used in Tab. A without any changes. That is, the new tasks and the unseen tasks for the model. For each task, we run 10 randomly initialized episodes and report the success rate. Tab. 20 shows the results.
Table 20: Real-worke experiment results more unseen tasks that require challenging semantic understanding.
Method
Add. Data
T5 (%)
T6 (%)
T7 (%)
RT-2 Style
xAmm-Det
O
0
inBC + Aux (D)
xAmm-Det
60
55
50
As we are able to observe, LLaRA is able to handle tasks requiring spatial language understanding tasks to certain degree. It sometimes gets confused due its smaller LLM size (7B), but it shows the potential to effectively leverage the knowledge of LLaVA, enabling efficient transfer to robot control with minimal data. This capability is primarily facilitated by our key contributions: the alignment between action and image coordinates, and the integration of self-supervised auxiliary data.
RT-2 Style on the other hand, simply fails to utilize the limited amount of training data provided in our setting, despite using the same model architecture.
C.5 QUALITATIVE EXAMPLES ON UNSEEN REAL-WORLD TASKS
In Fig. 14 we qualitatively show that LLaRA can complete some unseen tasks that are not in the training dataset. In the first example, we ask LLaRA to 'place all other objects into the large bowl'. LLaRA is able to recognize the objects to move and not to move and out- puts the correct actions. In the second example, tomato and put. it back'. LLaRA can put the tomato on the scale and move the tomato back to the similar place the initial position, showing the understanding of "putting it back'. These examples show the potential generalization ability of LLaRA. Video demonstrations are also included in the supplementary material.
let LLaRA 'get the weight of the
D LIMITATIONS
While our model has demonstrated significant achievements and holds considerable potential, is important to acknowledge that there remains space for further optimization.
30

---

Published as a conference paper at ICLR 2025
inpurt Observations
Input Prompt
inpurt Observations
Input Prompt
Place al other objects into bow!. Robot Actions
weight tomato and put back
Robot Actions
Figure 14: Examples of using LLaRA for tasks. We show the input texts from the and input images each step of the task, and images about the robot's actions accomplish each step.
First, some concepts in the reference images still can not be easily and precisely described by lan- guage, while in many cases these features are critical for robot control. Due to the single image limit of the current VLM, we still rely on the object detector trained on limited number of classes to describe the additional images the task description, which limits the generalization ability this method. In principle, in the future, this can be addressed by VLM trained on interleave image text data like (Bai et al. 2023 Li et al. 2024a).
Second, the information extracted from the dataset be noisy, which may lead to model confusion. In Tab. [8] there are discrepancies such as the location of an object in reference image differing from its actual location in the current image observation. At the same time, in VIMA dataset, when referring to the shape of an object without a color, the object in the reference image will appear in gray, which is totally different color than the object with the same shape in the current observations.
Finally, our method relies on 2D to 2D mapping to convert image coordinates into actual actions, which may face challenges when the complex 3D movement of the robot is required.
We believe enhancements in the mentioned aspects could further improve performance and broader applicability complex, real-world applications.
E
RAW RESULTS AND PROMPT TEMPLATES
This section contains all the numerical results of VIMA experiments and all prompt templates used in the paper.
31

---

Published as a conference paper at ICLR 2025
Table 21: Results VIMA-0.8k dataset. Ep: number of epoch; U: normalized number of model iterations. The following sizes datasets are relative the size of inBC. Loc.: relative
localization dataset; Det.: relative size of the detection dataset; Act.: relative size of the action prediction dataset; Fut.: relative size of the future prediction dataset; Spa.: relative size of the spatial relationship dataset; Temp.: relative size of the temporal relationship dataset. The means the reference images that appeared only in the task description used generate this dataset.
Method
Ep
U
Loc.
Det.
Act.
Fut.
Spa.
Temp.
L1 (%)
L2 (%)
L3 (%)
RT-2 Style
1
1
0.0
0.4
0.4
RT-2 Style
2
0.4
RT-2 Style
4
4
1.5
1.2
2.5
RT-2 Style
6
6
2.3
1.5
0.8
RT-2 Style
8
8
2.3
3.5
2.1
inBC
1
1
23.1
16.5
16.7
inBC
2
2
30.0
23.8
19.6
iBBC
4
4
22.3
21.2
15.4
inBC
6
6
21.9
14.6
17.5
inBC
8
8
17.7
inBC
10
10
20.4
18.1
14.6
inBC Aux
I
1.2
0.1
0.1
13.8
13.1
9.2
inBC + Aux
1*
26.5
24.2
inBC + Aux
2
2.4
0.1*
0.1*
29.6
21.9
23.3
inBC + Aux
2
2.4
0.1
0.1
23,5
20.4
17.5
BBCC Aux
2
4
0.5
0.5
26.9
23.8
24.6
inBC Aux
2
4
0.2
0.2
0.2
0.2
35,0
24.2
26.7
inBC + Aux
0,2
0.2
33,5
26.2
inBC Aux
2
1*
31.9
26.5
inBC + Aux
2
6
1
30.4
27.3
27.1
iBCC Aux
2
7
0.5
0.5
0.5
0.5
0.5
36.2
29.6
29.2
inBC Aux
2
8
0,5*
0.5*
0.5
0.5
0.5*
0.5
33.5
29.2
33.3
iBBC Aux
8
0.5
0.5
0.5
0.5
0.5
0.5
31.9
29.6
26.7
inBC + Aux
2
10
37.7
30.8
28.7
inBC + Aux
2
12
1
I
38.8
34.6
29.6
inBC + Aux
2
14
1
1
1
37.7
29.6
27.9
inBC Aux
2
14
40.0
35.8
31.2
D-inBC + Aux
1.2
0.1
0.1*
18.5
17.7
13.8
D-inBC + Aux
3
1
1*
28.8
25,4
23.3
D-inBC Aux
2
2.4
0.1
0.1
23.5
16.2
22.5
D-inBC + Aux
2
2.4
0.1
21.5
21.2
23.3
D-inBC + Aux
2
4
0.5
0.5
28.1
25.8
21.7
D-inBC + Aux
2
4
0.2
0.2
0.2
0.2
32.3
29.6
22.1
D-inBC + Aux
2
4
0.2
0.2
0.2
0.2
0.2
30.0
26.5
25.4
D-inBC Aux
2
6
1
1*
33.8
30.8
29.2
D-inBC Aux
2
6
35.8
28.5
28.3
D-inBC + Aux
2
7
0.5
26.9
D-inBC + Aux
2
8
0.5*
0.5*
0.5
0.5
0.5*
0.5
37.3
26.5
34.6
D-inBC + Aux
2
8
0.5
0.5
0.5
0.5
0.5
0.5
34.2
29.6
30.0
D-inBC + Aux
2
10
34.6
33.1
28.3
D-inBC Aux
2
12
40.0
38.5
37.5
D-inBC + Aux
2
1
I
30.4
32.9
D-inBC + Aux
2
14
I
40.0
35.4
36,2
32

---

Published as a conference paper at ICLR 2025
Table Results on VIMA-0.8K dataset with object detector Oracle. Ep: number of epoch; U: normalized number of model iterations. The following sizes of datasets are relative to the size of inBC. Loc.: relative size of the localization dataset; Det.: relative size of the detection dataset; Act.: relative size of the action prediction dataset; Fut.: relative size of the future prediction dataset, Spa.: relative size of the spatial relationship dataset; Temp.: relative size the temporal relationship dataset.
Method
Ep
Loc.
Det.
Act.
Fut.
Spa.
Temp.
(%)
1,2 (%)
(%)
D-R7-2 Style OD
1
0.0
0.0
0.0
D-RT-2 Style + OD
2
2
0.4
1.5
2.5
D-R72 Style + OD
4
1.9
1.5
2.5
D-RT-2 Style + OD
1.9
0.8
D-R7-2 Style OD
1.9
0.0
D-inBC OD
17.3
11.9
12.5
D-inBC + OD
2
2
17.3
14.6
15.0
D-inBC + OD
32.7
24.2
24.2
D-inBC + OD
28.1
21.7
D-inBC +
26.2
22.9
D-inBC + Aux OD
2
36,9
25.8
30.4
D-inBC + Aux OD
2
41.2
31.9
32,9
D-inBC + Aux OD
44.6
37.9
D-inBC Aux
2
38.8
D-RT-2 Style Oracle
1
0.0
0.4
0.8
D-RT-2 Style
Oracle
2
0.8
0.8
1.7
D-RT-2 Style
Oracle
4.2
D-RT-2 Style Oracle
6
2.7
3.5
0.8
D-RT-2 Style Oracle
1.2
1.2
D-RT-2 Style (1) Oracle
1.2
0.0
0.8
D-KT-2 Style (1) Oracle
2
2
2.3
1.9
1.7
D-RT-2 Style (I)
Oracle
0.8
0.8
D-RT-2 Style (I) Oracle
1.5
0.4
D-RI-2 Style (I) Oracle
0.4
D-inBC + Oracle
16.9
13.5
10.4
D-inBC + Oracle
2
2
20.0
19.2
17.5
D-inBC + Oracle
37.7
29.2
D-inBC + Oracle
30.0
27.5
D-inBC Oracle
27.7
27.9
D-inBC + Aux Oracle
2
45.0
36.9
39,2
D-inBC + Aux Oracle
43.8
38.5
37.9
D-inBC + Aux Oracle
52.7
50.4
D-inBC Oracle
49.6
47.9
33

---

Published as a conference paper at ICLR 2025
Table 23: Results VIMA-8K datasct. Ep: number of epoch; U: normalized number of model iterations. The following sizes of datasets are relative to the size of inBC. Loc.: relative size of the localization dataset; Det.: relative size the detection dataset; Act.: relative size the action prediction datasct; Fut.; relative size of the future prediction dataset; Spa.: relative size of the spatial relationship dataset; Temp.: relative size of the temporal relationship dataset. The means the reference images that appeared only in the task description used generate this dataset.
Method
Ep
C
Loc.
Act.
Fut.
Spa.
Temp.
(%)
1.2 (%)
L,3 (%)
RT-2 Style
1
3.1
3.5
0.8
RT-2 Style
2
2
3.8
3.1
1.7
RT-2 Style (2R)
2
2
1.9
2.3
1.7
RT-2 Style
4
8.1
6.5
5.8
RT-2 Style
RT-2 Style
RT-2 Style + Aux
2
14
36.9
35.8
32.1
inBC
1
44.6
34.2
36.2
inBC
2
2
57.3
46.2
42,9
inBC
4
58.5
48.5
47.1
inBC
54.6
45.4
40.4
iBBC
inBC
iBC Aux
2
2.4
0.1*
0.1*
59.6
50.0
45.4
inBC Aux
2.4
0.1
0.1
57.3
52.3
45.4
inBC Aux
4
0.5
0.5
60.4
56.2
52.1
inBC Aux
4
0.2
0.2
0.2
0.2
59.6
59.2
51.2
inRC Aux
1+
59.2
52.3
48.3
inBC Aux
1
57.7
$3.8
inBC Aux
7
0.5
0.5
inBC AuX
2
10
I
60.4
55.8
54.2
inBC Aux
2
12
1
1
60.8
55.4
50.0
inBC Aux
2
14
1
19
1
1
1
63.1
58.8
53.8
inBC Aux
2
14
59.2
58.8
52.1
D-inBC
2.4
0.1
0.1*
$3.5
48.8
45.4
D-inBC
2.4
0.1
D-inBC
54.6
D-inBC + Aux
2
4
0.2
0.2
0.2
0.2
58.1
52.3
51.2
D-inBC + Aux
2
6
1*
1
63.1
57.7
55.4
D-inBC Aux
2
6
1
60.0
53.5
50.0
D-inBC Aux
2
7
0.5
0.S
0.5
0.5
0.5
62.3
55.8
48.8
D-inBC Aux
2
10
I
I
63.1
58.1
49.6
D-inBC AuX
2
12
64.6
58.8
49,6
D-inBC AuX
2
14
1*
1
1
62.3
54.6
51.2
D-inBC Aux
14
62.3
57.7
52.1
D-inBC (I.)
14
61.9
54.2
55.4
D-inBC
4.8
D-inBC + Aux
12
61.2
57.7
49.6
D-inBC + Aux
20
62.3
56.5
49.6
D-inBC Aux
4
24
63.5
56.2
52.5
34

---

Published as a conference paper at ICLR 2025
Table Results on VIMA-8K dataset with object detector Oracle. Ep: number of epoch; U: normalized number of model iterations. The following sizes of datasets are relative to the size of inBC. Loc.: relative size of the localization dataset; Det.; relative size of the detection dataset, Act.: relative size of the action prediction dataset; Fut.: relative size of the future prediction dataset, Spa.: relative size of the spatial relationship dataset: Temp.: relative size the temporal relationship dataset. The *** means the reference images that appeared only the task description are not used generate this dataset.
Method
Ep
Loc.
Det.
Act.
Fut.
Spa.
Temp.
L1 (%)
L2 (%)
L3 (%)
D-R7-2 Style OD
1
1.5
3.1
1.2
D-RT-2 Style + OD
2
10.4
8.8
7.1
D-K72 Style + OD
4
16.9
14.2
15.8
D-RT-2 Style + OD
6
14.2
11.5
12.9
D-R7-2 Style OD
8
10.4
6.7
D-inBC OD
1
55.4
45.8
45.0
D-inBC + OD
2
63.5
51.5
50.0
D-inBC + OD
4
63.8
46.9
55.8
D-inBC + Aux OD
2
74,2
58.5
56.7
D-inBC + Aux OD
2
10
75.4
66.2
57.9
D-inBC + Aux + +D
2
76.9
63.8
57.9
D-inBC + Aux +D
2
79.2
64.2
58.3
D-inBC + Aux +D
1
1
72.7
55.4
57.1
D-inBC + Aux OD
2
79.2
65.4
60.4
D-inBC + Aux
60.0
D-inBC Aux OD
0.I
65.4
50.0
55.0
D-inBC Aux OD
4
76.9
63.1
55.8
D-RT-2 Style Oracle
1.2
D-RT-2 Style
Oracle
2
7.5
D-RT-2 Style
Oracle
4
21.9
17.7
15.0
D-RT-2 Style Oracle
6
13.5
16.2
11.2
D-KT-2 Style Oracle
8
16.2
11.9
11.7
D-RT-2 Style (I)
Oracle
1.2
1.2
D-RF-2 Style (I) Oracle
2
21.9
21.5
20.8
D-RT-2 Style (I) Oracle
4
25,8
23.1
19.6
D-RT-2 Style (f) Oracle
6
17.3
18.8
15.8
D-KT-2 Style (I) Oracle
8
21.9
20.0
20.4
D-inBC Oracle
1
60.4
59.2
54.6
D-inBC Oracle
2
73.5
62.7
63.3
D-inBC Oracle
4
4
70.8
61.2
66.2
D-inBC + Oracle
6
71.9
61.5
59.6
D-inBC Oracle
8
70.4
63.8
60.0
D-inBC Aux Oracle
2
78.1
73.8
65.4
D-inBC + Aux Oracle
2
10
78.1
73.8
70.0
D-inBC + Aux Oracle
2
12
1
1
78.5
78.1
71.2
D-inBC + Aux + Oracle
2
1a
1
76.2
65.8
66.2
D-inBC + Aux + Oracte
2
14
1
1
82.3
78.1
69.6
D-inBC + Aux + Oracle
4
0.1
0.1
71.9
64.6
61.7
D-inBC + Aux Oracle
78.8
75.4
67.1
35

---

Published as a conference paper at ICLR 2025
Table 25: Results VIMA-80K dataset. Ep: number of epoch; U: normalized number of model iterations.
The following sizes datasets are relative the size of inBC. Loc.: relative
localization dataset;
Det.: relative of detection dataset; Act.: relative size of action prediction dataset; Fut.: relative size of the future prediction dataset; Spa.: relative size of the spatial relationship dataset; Temp.: relative size of the temporal relationship dataset.
Method
Ep
Loc.
Det.
Act.
Fut.
Spa.
Temp.
(%)
L2 (%)
(%)
RT-2 Style
1
1
53.1
46.9
42.1
RT-2 Style
2
2
56.9
53.1
46.2
RT-2 Style
4
43.8
39.6
RT-2 Style
6
6
51.2
45.4
38.3
RT-2 Style
8
8
46.2
43.5
35.8
inBC
1
68.5
61.5
58.8
inBC
2
2
66.2
62.3
56.2
inBC
4
4
67.3
58.8
54.2
inBC
6
6
65.4
60.0
53.8
inBC
65.8
56.5
54.2
inBC + Aux
3
62.7
60.4
50.8
inBC Aux
66.9
63.8
56.2
inBC + Aux
59.6
55.8
inBC + Aux
7
68.1
60.8
54.6
inBC + Aux
2
6
65.8
64,6
56.7
inBC Aux
10
69.2
66.9
55.8
inCC Aux
12
65.4
60.0
50.0
iBCC + Aux
14
68.1
65.0
51.7
inCC + Aux
12
65.8
62.7
49.6
inBC + Aux
20
65.8
63.1
52.5
inBC Aux
24
63.8
59.6
48,8
iBCC + Aux
28
68.1
65.0
52.9
iBCC + Aux
18
67.7
61.5
51.2
inBC + Aux
6
30
67.3
60.4
49.2
inBC + Aux
6
36
61.9
60.4
47.1
inBC Aux
42
65.4
63.8
52.9
inCC Aux
24
67.3
62.7
52.1
inBC + Aux
8
40
66.9
61.2
51.7
inBC + Aux
8
48
66.2
58.1
50.0
inBC + Aux
8
56
66.9
63.8
50.8
D-in + Aux
66.2
63.5
57.5
D-inCC + Aux
5
66.2
60.8
57.1
D-inBC + Aux
6
65.4
60.4
52.5
D-inBC + Aux
7
63.8
61.9
54.2
D-inBC + Aux
2
61.9
54.6
D-inBC + Aux
2
10
70.0
65.4
56.2
D-inBC + Aux
2
12
69.6
62.7
56.2
D-inBC + Aux
2
14
70.4
63.5
56.7
D-inBC + Aux
12
67.7
65.4
60.0
D-inBC + Aux
20
68.
66.5
57.5
D-inBC + Aux
24
66.5
61.2
52.9
D-inBC + Aux
4
28
69.6
58.5
52.9
D-inBC + AuX
18
69.2
63.8
57.9
D-inBC + Aux
30
68.1
58,8
D-inBC + Aux
6
36
66.5
62.3
47.5
D-inBC + Aux
6
42
70.0
61.9
54.2
D-inBC + Aux
8
24
67.7
61.5
50.8
D-inBC Aux
8
40
69.6
66.5
57.1
D-inBC + Aux
69,2
51.7
D-inBC + Aux
8
56
68.1
67.7
57.9
36

---

Published as a conference paper at ICLR 2025
Table Results on VIMA- 80K dataset with object detector or Oracle. Ep: number of epoch: U: normalized number of model iterations. The following sizes of datasets are relative to the size of inBC. Loc.: relative size of the localization dataset; Det.: relative size of the detection dataset, Act.: relative size of the action prediction dataset; Fut.: relative size of the future prediction dataset; Spa.: relative size of the spatial relationship dataset; Temp.: relative size of the temporal relationship dataset.
Method
Ep
U
Loc.
Det.
Act.
Ful.
Spa.
Temp.
L1 (%)
1.2 (%)
L33
D-RF-2 Style OD
1
49.2
43.1
36.2
D-RF2 Style OD
2
2
52.3
50.0
45.8
D-RT-2 Style +OD
52.3
46.5
37.9
D-RT-2 Style
47.3
47.3
36.2
D-RF-2 OD
8
44.2
42.3
36.7
D-inCC OD
1
81.9
75.0
67.5
D-inCC OD
2
2
78.8
75.0
69.2
D-bBCC + OD
81.2
75.4
D.inHC
81.9
75.8
D-inCC OD
8
83.8
73.8
69.2
D-inRC Aux OI)
3
85.0
79.2
65.8
D-inBC + Aux OD
85.0
77.7
68.3
D-bnCC Aux OD
79.6
73.5
D-inCC Aux
84.2
74.6
D-inBC Aux OD
2
81.5
75.8
64.2
D-bnBC Aux OD
2
10
86.2
83.1
69.6
D-inHC + Aux OD)
2
12
80.0
73.1
62.5
D-inBC
OD
2
77.3
D-inBC Aux OD
4
12
78.5
73.8
68.8
D-inBC + Aux OD
4
20
82.3
77.3
71.7
D-inBC + Aux OD
4
24
82.3
71.9
63.7
D-inHC
83.5
71.5
62.5
D-inBC Aux OD
6
18
80.0
75.0
67.1
D-inBC Aux OD
30
86.2
81.5
69.6
D-bnBC + Aux OD
36
79.6
74.6
64.6
D.inBC
77.3
D-inCC Aux OD
8
24
78.5
73.1
68.3
D-inCC Aux OD
8
40
86.5
82.7
70.8
D-inBC + Aux OD
8
48
82.3
78.1
65.4
D-inCC Aux OD
82.7
76.5
D-RT-2 Oracle
71.2
66.5
D-RT-2 Style Oracle
2
2
74.2
70.4
69.2
D D-RF-2 Style
Oracle
4
73.1
70.4
63.3
D-RT-2 Style
Oracle
6
67.7
64.2
60.4
D-RT-2 Style Oracle
69.2
61.5
55.4
D-RT-2 Style (f) Orucle
57.7
54.6
D-RT-2 Style (f) Oracle
2
2
55.8
54.6
55.4
D D-RF-2 Style (f) Oracle
4
58.1
54.2
55.4
D-RT-2 Style (f) Oracle
6
53.5
49.6
47.9
D-RT-2 Style Oracle
8
53.8
50.8
D-inBC
84.2
D-inBC Orucle
2
84.2
82.3
77.5
D-inKC + Oracle
4
85.4
83.8
80.0
D-inBC + Orucle
6
86.9
85.8
80.8
D-inCC Oracle
87.3
85.4
D-inBC
88.8
D-inBC Aux Oracle
1
86.2
84.6
79.2
D-inBC + Aux + Oracle
1
6
82.3
77.3
69.6
D-inBC + Aux + Oracle
1
86.2
83.5
77.5
D-inBC
Oracle
2
88.5
84.2
D-inBC Aux Oracle
2
10
91.5
88.1
80.4
D-inBC Aux Oracle
2
12
81.2
80.4
67.5
D-inBC + Aux + Oracle
2
88.5
84.6
78.3
D-inHC
Oracle
85.0
D-inBC Aux Oracle
4
20
88.8
84.6
79,6
D D-inHC + Aux + Oracle
4
24
81.5
77.3
65.8
D-inHC + Aux + Oracle
28
84.2
80.0
71.7
D-inBC Aux Oracle
84.6
85.4
76.7
D-inHC Aux Oracle
6
30
87.7
87.3
78.3
D-inBC Aux + Oracle
6
36
81.2
80.8
70.8
D-inBC Aux + Orucle
6
42
88.8
81.2
70.0
D-inHC
Oracle
86.9
D-inBC Aux Oracle
8
40
90.0
88.1
79.2
D-inBC Aux Oracle
8
48
82.3
83,1
70.4
D-inHC Aux + Oracle
8
56
83.8
88.1
78.8
37

---

Published as a conference paper at ICLR 2025
Table Results on full VIMA dataset Oracle. Ep: number of epoch; U: normalized number model iterations. The following sizes of datasets are relative to the size of inBC. Loc.: relative size of the localization dataset: Det.: relative size of the detection dataset: Act.: relative size of the action prediction dataset; Fut.:
relative size of the future prediction dataset; Spa.: relative size of the spatial relationship dataset; Temp.: relative size of the temporal relationship dataset.
Method
Ep
Loc.
Det.
Act.
Fut.
Spa.
Temp.
(%)
L2 (%)
(%)
RT-2 Style
61.9
55.8
47.1
KI-2 Style
2
65.0
59.6
42.5
RT-2 Style
3
63.1
57.7
44.6
RT-2 Style
4
64.2
60.0
44.2
D-KI-2 Style Oracle
1
82.3
80.8
74.6
D RT-R Style Oracle
2
2
86.2
88.1
75.8
D-RT-2 Style Oracle
3
3
87.7
83.5
76.2
D-RT-2 Style Oracle
4
87.3
85.4
74.6
D-KT-2 Style (I) + Oracle
68.8
68.1
59.6
D-RT-2 Style (I) + Oracle
2
68.5
66.2
58.3
D-RT-2 Style (I) + Oracle
3
70.0
70.0
59.2
D-RT-2 Style (7) + Oracle
4
69.6
69.2
58.3
Table 28: Prompt candidates for action generation during inference.
'Could you write down what needs be done to complete the task on this scene?'
'List out the actions needed to accomplish the task in this scene."
'What actions are necessary perform the task on this scene?"
'Can you describe what needs to be done on this scene to complete the task?'
'What steps are required to perform the task shown in this scene?"
'List the actions needed to perform the task given below.'
'On the following scene, could you what actions are required to perform the task?'
'Describe what actions needed on this scene to complete the task.'
'What do you need to do on this scene to accomplish the task?'
'List the actions required to perform the task given this scene.' 'Could you please describe the steps needed to perform the task on this scene?'
'Write down the actions required to perform the task this scene."
'Please write down the actions required to perform the task shown below.'
"Can you explain what needs be done to perform the task in this scene?'
Describe the actions required to complete the task on this scene.'
38

---

Published as a conference paper at ICLR 2025
Table 29: Prompt candidates for object localization. (object) will be replaced with the actual object name in the dataset.
"Where is (object) located in the image? Please use the format <b>(x, y).lw. ww. h|</b> where x and represent the center coordinates of the bounding box, and w and h are the width and height. The coordinates start from the top left corner and are normalized scale of to 1.'
"Can you provide the location of [object] in the image? Format <b>(x, y)./w, w. h|</b>, with x and y as the center coordinates of the bounding box and w and as the width and height. The coordinates should begin at the top left corner and be normalized from 0 to 1.' 'What are the coordinates of [object) in the image? ? Use the format <b>(x, y). h h|</b>, where and y the center of the bounding box, and and h represent the width and height. Coordinates should start at the top left corner and be normalized to range of 0 to 1.'
'Please specify the location of (object) in the image. List it in the format <b>(x, y). (w, </<//>, where and denote the bounding box center coordinates, and and h are the width and height. The coordinates begin from the top left corner and should be normalized to 0 to 1.'
'What is the position of (object] within the image? Use the format <b>(x, y). (w, h/</b> to describe it, with and the center coordinates of the bounding box, and and h as the width and height. The coordinates start at the top left corner and are normalized to scale of 0 to 1.'
'Describe the location of [object) the image using the format <b>(x, y), hj</b>, In this format, and y denote the center coordinates of the bounding box, while w and h represent its width and height. Coordinates should normalized from the top left corne,, ranging from to 1.'
'Can you detail the location of [object) in the image? Format it as <b>(x, y). h h/</b>, where and indicate the bounding box center, and and h represent the width and height. The coordinates should be normalized a scale of 0 to starting from the top left corner.'
'Provide the location of [object] in the image using the format <b>(x, y). (w. hj</b>, Here, and y are the center coordinates of the bounding box, and w and are the width and height. The coordinates begin at the top left corner and are normalized from 0 1.' 'Where is {object] positioned in the image? Use the format <b>(x, ().[w h) <//>,, where x and y denote the center coordinates of the bounding box, and and are the width and height. The coordinates should be normalized to range of 0 to starting from the top left corner."
'Specify the location of {object] } in the image in the format <h>(x, y).(w, [w. h <></b>. In this format, x and y represent the bounding box center, and and are the width and height. The coordinates should start from the top left corner and be normalized between O and 1.' 'What is the exact position (object) in the image? Format the coordinates as <b>(x, (), h/</b>, where and width and height. The coordinates start from the top left corner and are normalized to scale of 0 to 1.'
the center of the bounding box and and h denote its
'Describe where (object) is located in the image using the format <b>(x, () hj</b>, Here, and indicate the bounding box center coordinates, and and h specify its width and height. The coordinates should be normalized starting from the top left corner, within the range of 0 to 1.'
'Could you tell me the location of [object) the image? Use the format <b>(x, y). h) </</,, where and denote the center of the bounding box and w and h are the width and height. Coordinates start at the top left corner and should be normalized between and 1.'
'Provide the coordinates of (object) in the image in the format <b>(x, y). h)</b>, Here, x and y are the center of the bounding box, while and h represent its width and height. The coordinates should start from the top left corner and be normalized 'How is the (object) located in the image? List its coordinates using the format <b>(x, (),(w, h) </b>, where x and are the center coordinates of the bounding box, and w and indicate its width and height. The coordinates begin at the top left corner and are normal- ized a range of 1.¹
1.'
39

---

Published as a conference paper at ICLR 2025
Table 30: Prompt candidates for object detection.
"Identify and describe each object in the image. For each object, list it in the format <b>(x, (), h/</b>, where x and represent the coordinates of the bounding box center, and w and represent the width and height of the bounding box. The image coordinates should start from the top left corner and be normalized between O and 1.²
"Catalog all the objects present in the image. For every object, the format <b>(x, y) (w, h) <//>, with x and indicating the center of the object's 's bounding box coordinates, and and specifying the width and height. The coordinates corner, ranging from 0 to 1.'
normalized from the top left
'List each object in the image and deseribe it. Use the format <b>(x, y). h/</b> for each object, where x and denote the center coordinates of the bounding box, and w and h the width and height the bounding box. The coordinates should start from the top left corner and be normalized to a scale of 0 to 1.' 'Provide descriptions for all objects within the image. Each object should be listed using the format <b>(x, y).[ w, h h/<//>, where x and y are the coordinates the bounding box center, and and h the width and height. The coordinates should be normalized, starting from the top left corner, within a range of to 1.'
'Enumerate and describe every object found in the image. For each object, utilize the format <h>(x, y). w, h) </h>, where x, are the bounding box center coordinates and w, are the dimensions (width and height) of the bounding box. The coordinates begin the top left corner and are normalized between () and
'Detail all the objects within the image, listing each using the format <b>(x, y). h) </b>.> Here, x and y represent the coordinates of the bounding box center, while w and h indicate the width and height. The coordinates start from the top left corner and normalized to the range of 0 to 1.'
'Document each object present in the image. For each object, the format <b>(x, y). (w, h </b>, where x and y are the coordinates of the center of the bounding box, and w and the width and height. The coordinates should be normalized, starting from the top left corner, and range from O to 1.'
'For each object in the image, provide description using the format <h>(x, y).(w, </<//>> Here, x and y denote the coordinates of the bounding box center, and w and h represent the width and height of the bounding box. The coordinates normalized scale of 0 to 1, starting from the top left corner.' 'Describe all the objects h) <///b>> The x and y values are the coordinates for the center of the bounding box, while and h represent its width and height. The coordinates should be normalized from the top left corner, within a range of 0 to
in the image, and list them using the format <b>(x, y). (w,
'Identify and list each object found in the image. For each one, the format <b>(x, y).(w (w. h/</b>> In this format, x and y are the coordinates for the bounding box center, and be normalized starting from the
and are the width and height. The coordinates top left corner, ranging from 0 to 1.'
'List and describe each object in the image using the format <b>(x, y). h) <//>/>. Here, x and y correspond to the coordinates of the bounding box center, and w and h specify the width and height of the bounding box. The coordinates should start from the top left corner and be normalized to the range 'Provide description for each object in the image, formatted as <b>(x, y).(w, h}</b>. The x and y values indicate the center coordinates of the bounding box, while w and represent the width and height. The coordinates start from the top left comer and are normalized between and
to 1.'
40

---

Published as a conference paper at ICLR 2025
Table (Continued) Prompt candidates for object detection.
'Catalog each object within the image, using the format <b>(x, (),(w, h)</b> for each one. In this format, and are the coordinates for the center of the bounding box, and w and are the width and height. The coordinates should be normalized, beginning at the top left corner and ranging from to 1.'
'Enumerate all the objects in the image, providing descriptions for each using the format <b>(x, y), [w. h b///>>. The x and y values represent the center coordinates the bounding box, while and h indicate its width and height. The coordinates normalized starting from the top left corner, within a range to
DDescribe cach object in the image, listing them in the format <b>(x, y).lw, h/</b>. Here, x and y denote the center coordinates of the bounding box, and w and specify the width and height. The coordinates should be normalized from the top left corner, ranging from 0 to 1.'
Table 32: Prompt candidates for action prediction. (scene) will be replaced with list of objects in the scene.
'Could you detail the steps needed to transform the seene shown in the image into the second scene? The second scene is provided as collection of object bounding boxes {scene). The format for these bounding boxes is <b>(x, y), {w. h)</b>, where x and represent the center coordinates, and and h are the width and height. The coordinates should be normalized to scale of 0 to 1, starting from the top left corner.'
'Can you describe what actions are required to rearrange the scene shown in the image to match the second scene? The second scene is given {scene). These bounding boxes follow the format <b>(x, y), (w, h) <//>,>, where and y indicate the center coordinates, and w and h represent the width and height. The coordi- nates should start from the top left corner and be normalized to a scale of 1.'
set of object bounding boxes
"Could you list the steps necessary to modify the scene shown in the image to the second scene? The second scene is described as collection of object bounding boxes ( (scene). The bounding box format is <b>(x, y), [w, h)</b>, with x and y denoting the center coor- dinates, and w and representing the width and height. The coordinates are normalized to a scale of 0 to 1, starting from the top left corner.'
"Can you explain what needs be done to adjust the scene shown in the image to resemble the second scene? The second scene (scene) } consists of object bounding boxes provided in the format <b>(x, y), {w. h) </<>>>, Here, and y represent the center coordinates. and w and h are the width and height. The coordinates should start from the top left corner and be normalized a scale of 0 1.'
'Could you outline the necessary actions to arrange the scene shown in the image into the second scene? The second scene defined by collection of object bounding boxes {scene). These bounding boxes follow the format <b>(x, y), [w, h) h)</b>, where and denote the center coordinates. and and are the width and height. The coordinates start from the top left corner and should be normalized to a scale of 0 to 1.'
'Can you specify what needs be done convert the scene shown in the image into the second scene? The second scene is provided as a series of object bounding boxes [scene]. ). The format for these bounding boxes <b>(x, y). {w, h)</b>, with x and y representing the center coordinates, and w and h indicating the width and height. Coordinates should be normalized from the top left corner
scale of 0 to 1.'
'Could you describe the steps required to change the scene shown in the image to the sec- ond scene? The second scene is depicted
collection of object bounding boxes (scene). The bounding box format is <b>(x, y). [w, h <///b>, where and denote the center coor- dinates, and and h represent the width and height. The coordinates normalized scale of to 1 starting from the top left corner."
41

---

Published as a conference paper at ICLR 2025
Table 33: (Continued) Prompt candidates for action prediction. (scene) will be replaced with a list of objects in the scene.
'Can you list the actions necessary to transform the scene shown in the image into the second scene? The second scene is described using object bounding boxes (scene). The format of these bounding boxes is <b>(x, y), (w, h)</b>, where x and are the center coordinates, and and represent the width and height. Coordinates should be normalized to a scale of 0 to starting from the top left corner."
'Could you explain the process to arrange the scene shown in the image match the second scene? The second scene is provided as collection of object bounding boxes {scene). These bounding boxes are formatted as <b>(x, y), [w, h/</b>, where x and y represent the center coordinates, and and h are the width and height. The coordinates should start from the top left corner and be normalized "Can you detail what needs to be done to rearrange the scene shown in the image to the second scene? The second scene is given as series of object bounding boxes (scene), ). The bounding box format is <b>(x, y). [w, h)</b>, where and denote the center coor- dinates, and and represent the width and height. Coordinates should be normalized to a scale of 0 to starting from the top left corner."
scale of to 1.'
'Could you specify the steps needed modify the scene shown in the image resemble the second scene? The second scene is described as set of object bounding boxes [scene], ). These bounding boxes follow the format <b>(x, y), {w, h)</b>, where x and y represent the center coordinates, and w and indicate the width and height. The coordinates start from the top left corner and should be normalized
scale of 0 to 1.¹
'Can you outline the necessary actions to change the scene shown in the image into the second scene? The second scene [scene] consists of object bounding boxes provided in the format <b>(x, y), [w, h) h></b>, where x and denote the center coordinates, and w and h represent the width and height. Coordinates should be normalized to a scale of starting from the top left corner.'
'Could you describe the steps adjust the scene shown in the image the second scene? The second scene is given as a collection of object bounding boxes (scene). J. The format for these bounding boxes is <b>(x, y), {w, h/</b>, where and y represent the center coordinates, and w and are the width and height. The coordinates should start from the top left corner and be normalized
scale of 0 'Can you explain what needs to be done to transform the scene shown in the image into the second scene? The second scene is depicted using object bounding boxes (scene). The bounding box format is <b>(x, y), [w, h///b>, with x and y representing the center coordinates, and and indicating the width and height. The coordinates start from the top left corner and are normalized to a scale of 0 to 1.¹
'Could you detail the steps necessary convert the scene? The second scene is described as set of object bounding boxes (scene). These bounding boxes follow the format <b>(x, y). [w, h)</b>, where x and represent the center coordinates, and w and denote the width and height. The coordinates should be normalized scale of 0 to starting from the top left corner.'
shown in the image to the second
42

---

Published as a conference paper at ICLR 2025
Table 34; Prompt candidates for future prediction. {pick and place) will replaced with the action text.
'The image shows scene with multiple objects. Now you (pick and place), what will the scene look like? List the object bounding boxes. The bounding box format is <b>(x, y), [w, h)</b>, where x and represent the center coordinates of the bounding box, and w and are its width and height. The coordinates should start from the top left corner and be normalized to scale of to 1.
'An image depicts a scene containing multiple objects. Now you {pick and place), what will the scene look like? Write the list of object bounding boxes. The bounding boxes should be formatted as <b>(x, y), [w, <//>,>, where and y denote the center coordinates, and w and h are the width and height. The coordinates start from the top left corner and are normalized to a scale of 1.¹
"The image presents a scene with several objects. Now you [pick and place), what will the scene look like? List the object bounding boxes. The format for these bounding boxes is <b>(x, y), {w. h)</b>, where and y represent the center coordinates, and and h are the width and height. Coordinates should start from the top left corner and normalized to scale of 0 to 1.'
'Displayed in the image is scene containing multiple objects. Now you (pick and place), what will the scene look like? Write down the list of object bounding boxes. These bound- ing boxes follow the format <b>(x, y), [w, h/</>>, with x and as center coordinates, and w and h as the width and height. The coordinates should be normalized starting from the top left corner to scale of
"The image illustrates scene with multiple objects. Now you [pick and place), what will the scene look like? Write the list of object bounding boxes. The bounding boxes are formatted as <b>(x, y). [w, h|</>>, where x and y denote the center coordinates, and w and represent the width and height. Coordinates should start from the top left corner and be normalized to a scale of 0 to 1.'
"The image depicts a scene with several objects. Now you (pick and place), what will the scene look like? List the object bounding boxes. The bounding box format is <b>(x, y), [w, h/</,>, where and y represent the center coordinates, and w and height. The coordinates should be normalized to scale of 0 to starting from the top left corner."
denote the width
'In the image, there is scene with multiple objects. Now you [pick and place). what will the scene look like? Write the list object bounding boxes. The format of these bounding boxes is <h>(x, y). [w, h)</b>, where and indicate the center coordinates, and and h represent width and height. The coordinates start from the top left corner and are normalized to scale of 0 to 1.'
'An image shows a scene with various objects. Now you (pick and place), what will the scene look like? Write down the list of object bounding boxes. The bounding boxes follow the format <b>(x, y), [w, h <//b>>, where x and y denote the center coordinates, and w and h are the width and height. The coordinates should start from the top left corner and be normalized scale of 0 to
'The image presents scene containing several objects. Now you (pick and place), what will the scene look like? List the object bounding boxes. The bounding box format is <h>(x, y), [w, h)</b>, where x and represent the center coordinates, and w and h are the width and height. Coordinates should start from the top left corner and be normalized to scale of 0 to 1.'
"The image displays scene with multiple objects. Now you (pick and place). what will the scene look like? Write the list of object bounding boxes. The bounding boxes should be in the format <b>(x, y), [w, h)</b>, where and y denote the center coordinates, and w and h represent the width and height. The coordinates should start from the top left corner and be normalized
scale of to
'An image illustrates a scene with multiple objects. Now you (pick and place), what will the scene look like? Write down the list of object bounding boxes. These bounding boxes are formatted as <b>(x, y), [w, h/ h/</b>, where and represent the center coordinates, and and h denote the width and height. Coordinates should be normalized to a scale of 0 to starting from the top left corner."
43

---

Published as a conference paper at ICLR 2025
Table 35: (Continued) Prompt candidates for future prediction. {pick and place) will replaced with the action text.
"The image shows a scene with various objects. Now you {pick and place), what will the scene look like? List the object bounding boxes. The format for these bounding boxes is <b>(x, y), [w, h/</b>, with x and y representing center coordinates, and w and as the width and height. Coordinates should start from the top left corner and be normalized to a scale of 0 to
'Displayed in the image is scene containing multiple objects. Now you [pick and place), what will the scene look like? Write the list of object bounding boxes. The bounding box format is <b>(x, y). [w, h h///b>, where x and denote the center coordinates, and w and h are the width and height. The coordinates start from top left corner and are normalized to a scale of () to 1.¹
"The image illustrates scene with various objects. Now you (pick and place), what will the scene look like? List the object bounding boxes. The bounding boxes are formatted <b>(x, y). (w, h <///b>, where x and y indicate the center coordinates, and and represent the width and height. Coordinates should be normalized from the top left corner to a scale of ( to
'An image depicts scene with multiple objects. Now you [pick and place), what will the scene look like? Write the list of object bounding boxes. The bounding box format is <h>(x, y), [w, h)</b>, where x and represent the center coordinates, and w and h denote the width and height. The coordinates should start from the top left corner and be normalized to scale of to 1.'
Table 36: Prompt candidates for spatial relationship. {ego_obi] and [ref_obj} will be replaced with object names and [example] will be replaced with random spatial relationship from the same image.
"Can you describe the relative spatial locations of [ego_obj} compared to [re[_obj) in this image? Use relative location words like left, right, above, below, etc. Also, find the 2D center distance and the Euclidean center distance between them. Your output must follow this format: (example). The coordinates image coordinates normalized to scale of O to starting from the top left corner."
'Could you describe the relative spatial positions of (ego_obj) in comparison to [re__ojj] in this image? Use terms like left, right, above, below, etc. Also, calculate the 2D center distance and the Euclidean center distance between them. Your output should be formatted as follows: (example). }. The coordinates are image coordinates normalized to a scale 1 starting from the top left corner.'
'Please describe the relative spatial locations of [ego_ob) compared to [ref_obj] in this image. Use words like left, right, above, below, etc. Additionally, find the 2D center distance and the Euclidean center distance between them. Your output must be in this format: (example). The coordinates are image coordinates normalized to scale of 0 to starting from the top left corner.'
'Can you explain the relative spatial positions of (ego_obj} compared (ref_on [ee_ in this image? Use terms such as left, right, above, below, etc. Also, determine the 21) center distance and the Fuclidean center distance between them. Your output should match this format: (example). The coordinates are image coordinates normalized to a scale of 0 to starting from the top left corner"'
'Describe the relative spatial locations (ego_obi _bjooo]] compared to (ref (ref_obj] ) in this image using words like left, right, above, below, etc. Also, calculate the 2D center distance and the Euclidean center distance between them. Your output must follow this format: (example). J. The coordinates image coordinates normalized to scale of to starting from the top left corner."
44

---

Published as a conference paper at ICLR 2025
Table 37: (Continued) Prompt candidates for spatial relationship. (ego obj] and (ref obj] will be replaced with object names and [example) will be replaced with random spatial relationship from the same image.
"Could you describe the spatial relationship between (ego _bjo and (ref _ojj] in this im- age using relative location words like left, right, above, below, etc.? Also, find the 2D center distance and the Euclidean center distance between them. Your output should be formatted as follows: (example). The coordinates are image coordinates normalized to scale of to starting from the top left corner'
'Can you detail the relative spatial positions {ego_obs compared to (ref [bjo in this image? Use words like left, right, above, below, etc. Also, determine the 2D center dis- tance and the Euclidean center distance between them. Your output must be this format: (example). ). The coordinates from the top left corner."
image coordinates normalized to scale of to starting
"Could you explain the spatial relationship between (ego_ob) and (ref obj) in this image using terms such as left, right, above, below, etc.? Also, calculate the 2D center distance and the Euclidean center distance between them. Your output should match this format: /example). The coordinates are image coordinates normalized to scale to starting from the top left corner.'
'Describe the relative spatial positions of (ego [ego_objj _bjj compared to {ref obj) in this image. Use relative location words like left, right, above, below, etc. Also, find the 2D center distance and the Euclidean center distance between them. Your output must follow this format: [example). The coordinates are image coordinates normalized to scale of 0 to starting from the top left corner''
'Can you describe how [ego_obj) is positioned relative to [ref_ in this image using words such as left, right, above, below, etc.? Also, find the 2D center distance and the Euclidean center distance between them. Your output should be in this format: (example). The coordinates are image coordinates normalized to scale to starting from top left corner.'
'Could you detail the relative positions of (ego obj) compared to (ref tre__bjji } in this image using terms like left, right, above. below, etc.? Also, calculate the 2D center distance and the Euclidean center distance between them. Your output must be formatted as follows: (example). The coordinates are image coordinates normalized to scale of to starting from the top left corner.
'Please describe the spatial relationship of {ego_objj in comparison to {ref_ rej_ in this image using relative location terms such as left, right, above, below, etc. Additionally, find the 2D center distance and the Euclidean center distance between them. Your output should match this format: (example). The coordinates are image coordinates normalized to scale of 0 to starting from the top left corner."
'Can you describe the relative spatial locations of (ego_obj) compared [ref_obj) this image? Use relative location words like left, right, above, below, etc. Also, calculate the 2D center distance and the Fuclidean center distance between them. Your output should follow this format: (example). The coordinates are image coordinates normalized to scale to starting from the top left corner.'
'Could you describe the spatial locations of (cgo_obj) relative to (ref_obj) obj) in this image using words such as left, right, above, below, etc.? Additionally, find the 2D center distance and the Euclidean center distance between them. Your output must be in this format: (example). The coordinates are image coordinates normalized to scale to starting from the top left corner."
45

---

Published as a conference paper at ICLR 2025
Table 38: Prompt candidates for temporal relationship (scene) will replaced with list of objects, (ego_obj) and [ref_obj] will be replaced with object names and {cxample] will be replaced with random temporal relationship from the same image.
"The image shows a scene at the first timestamp, while the second image described as {scene) shows the next timestamp. Can you describe the change in the relative location of [ego_obj] compared to [re[_obj) between these two timestamps? Use relative distance words like getting closer further away, etc. Also, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must follow this format: (example). The coordinates are image coordinates normalized scale of to starting from the top left corner."
'In the first timestamp, the image shows scene, and the second image described (scene) depicts the next timestamp. Can you describe the change in the relative loca- tion of (ego_ob) compared (ref vef_obj] between these two timestamps? Use terms like getting closer or moving further away, etc. Additionally, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must fol- low this format: (example). The coordinates are image coordinates normalized to a scale of 0 to I starting from the top left corner.'
"The scene in the first image is at the initial timestamp, and the second image described (scene) shows the subsequent timestamp. Can you explain the change in the relative location of (ego_obj) compared to (ref_obj) between these two timestamps? Use words like getting closer moving further apart, etc. Also, calculate the change in the 2D center distance and the Euclidean center distance between the two images. Your output should be formatted follows: (example). The coordinates image coordinates normalized scale of to starting from the top left corner.'
'At the first timestamp, the image shows scene, and the second image described [scene) represents the next timestamp. Can you detail the change in the relative location of (ego obj) compared to (ref _bb]] between these two timestamps? Use relative distance words like moving closer or getting further away, etc. Additionally, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must follow this format: /example). The coordinates are image coordinates normalized to scale of 0 to starting from the top left corner.'
"The first image shows a scene at an initial timestamp, and the second image described as (scene) depicts the next timestamp. Can you describe the change in the relative position (ego_obj compared to [ref_obj) between these two timestamps? Use terms such as getting closer moving further apart, etc. Also, determine the change in the 2D center distance and the Euclidean center distance between the two images. Your output should follow this format: (example). The coordinates scale of to starting from the top left corner.'
image coordinates normalized
"The initial timestamp shows scene in the first image, and the second image described as (scene) } represents the next timestamp. Can you describe how the relative location of dego_obji compared to (ref obj) changes between these two timestamps? Use relative distance words like getting closer or moving further away, etc. Also, find the change in the 2D center distance and the Euclidean center distance between the images. Your output must be in this format: [example). ). The coordinates are image coordinates normalized to scale of 0 to starting from the top left corner.'
'The image shows scene at the first timestamp, and the second image described as (scene) shows the subsequent timestamp. Can you detail the change in the relative lo- cation of (ego_obj) compared to (ref_obj) between these two timestamps? Use words like getting closer moving further apart, etc. Also, calculate the change in the 2D center distance and the Euclidean center distance between the two images. Your output should be formatted follows: (example). The coordinates are image coordinates normalized to scale of to 1 starting from the top left corner.'
46

---

Published as a conference paper at ICLR 2025
Table 39: (Continued) Prompt candidates for temporal relationship [scene) will be replaced with list of objects, (ego_obj) and {ref_obj] will be replaced with object names and (example) will be replaced with random temporal relationship from the same image.
'At the initial timestamp, the image shows a scene, and the second image described as [scene) depicts the next timestamp. Can you describe the change in the relative position of [ego_obj] compared to [re[_obj] between these two timestamps? Use relative distance terms such as getting closer or moving further away, etc. Also, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must follow this format: (example). The coordinates are image coordinates normalized to a scale of O to starting from the top left corner."
"The scene in the first image is the initial timestamp, and the second image described as (scene) shows the following timestamp. Can you describe the change in the relative location of (ego_obj) compared (ref [ee__bjj between these two timestamps? Use words like getting closer or moving further apart, etc. Additionally, calculate the change in the 2D center distance and the Euclidean center distance between the images. Your output should follow this format: (example). The coordinates are image coordinates normalized scale of 0 to starting from the top left corner."
"The first image shows scene at an initial timestamp, and the second image described (scene) depicts the next timestamp. Can you explain how the relative location of [ego_obj} compared to [re[_obj] changes between these two timestamps? Use relative distance words like moving closer getting further away, Also, determine the change in the 2D center distance and the Euclidean center distance between the two images. Your output must follow this format: (example). The coordinates image coordinates malized to a scale of 0 to starting from the top left corner."
"The image shows scene the initial timestamp, and the second image described (scene) shows the next timestamp. Can you describe the change in the relative position of fego_obj] __jj]] compared to rre ___jj] between these timestamps? Use words like getting closer or moving further apart, etc. Also, calculate the change in the 2D center distance and the Euclidean center distance between the two images. Your output should follow this format: [example). The coordinates are image coordinates normalized to a scale of 0 to starting from the top left corner.'
'At the first timestamp, the image shows scene, and the second image described as (scene) depicts the next timestamp. Can you detail how the relative location of rego_obj) compared to [ref_obj] changes between these two timestamps? Use relative distance terms such as moving closer getting further away, etc. Also, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must fol- low this format: (example). The coordinates image coordinates normalized scale to 1 starting from the top left corner.' "The initial timestamp shows
in the first image, and the second image described as {scene] represents the next timestamp. Can you describe the change in the relative location of [ compared to (ref obj) between these two timestamps? Use words like getting closer or moving further apart, etc. Also, determine the change in the 2D center distance and the Euclidean center distance between the two images. Your output should follow this format: (example). The coordinates are image coordinates normalized scale of 0 to starting from the top left corner''
'The first image shows scene at the initial timestamp, and the second image described (scene) shows the following timestamp. Can you describe how the relative position (ego_obj) compared to (ref [ref_obj] changes between these two timestamps? Use terms like moving closer getting further away, etc. Additionally, find the change in the 2D center distance and the Euclidean center distance between the two images. Your output must follow this format: (example). The coordinates are image coordinates normalized to a scale of 0 starting from the top left corner''
47


Reviews:
[{"review_id": "wiPhjytQtX", "reviewer": "Reviewer_aFtC", "summary": "The manuscript proposes a framework for performing auxiliary instruction-tuning of LLaVA, using a robotics behavior cloning dataset that has been converted into a conversation-style instruction-tuning dataset, with the goal of enabling the model to better understand spatiotemporal context.", "questions": "No additional questions — please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The pursuit of robotics-oriented pretext tasks for training large-capacity open source models remains quite compelling.", "weaknesses": "Section 2, Section 6: The manuscript introduces relevant contemporary approaches — some with strong inductive biases — but the experimental comparisons with the method proposed by the manuscript remain shallow. I would like to see some direct comparisons with existing VLAs in the experiments section, in addition to ablations on, e.g., different action space representations (e.g., versus RoboPoint).\n\nSection 2 (L146-148): The manuscript states, \"Moreover, all the aforementioned studies lack of comprehensive investigation into the methods for generating auxiliary datasets from existing robot data, as well as the implications of integrating such datasets.\" OpenVLA uses mixtures of exisisting OXE component datasets, for a next action token-prediction pretext.\n\nSection 5: The dependency on high-quality behavior cloning data may limit LLaRA’s performance in environments significantly different from the training data. Other methods cited by the manuscript do not share this limitation.\n\nSection 5: The pretext task proposed by this manuscript, which relies on object locations (in a tabletop coordinate system) and joint rotations will be less general than, e.g., the next-token prediction pretext and action tokenization proposed by OpenVLA. The manuscript should better motivate the reduced scope, whereas many of the models mentioned in Section 2 pursue general training and execution paradigms: across domains, embodiment, and task families.\n\n&nbsp;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fVzafWi89J", "reviewer": "Reviewer_wHkn", "summary": "The paper aims to transform behavior cloning data into a form that is more accessible for VLM training, enabling the VLM to leverage its pretraining for downstream tasks. The approach involves converting the BC dataset into image-text data and having the model output 2D coordinates rather than the 3D outputs. The study introduces curated datasets to assist in 2D grounding, action prediction, next observation prediction, and understanding spatial relationships between objects. The model is evaluated against RT-2 and VIMA, showing performance gains, particularly in low-data regimes.", "questions": "- RT-2 also co-trains on auxillary tasks and internet data. How does this compare to the proposed approach?\n- For the comparison with VIMA, can additional insights be provided on which specific modeling choices (beyond single-view versus dual-view inputs) contribute to LLaRA’s superior performance?\n- Can the comparisons with RT-2 and VIMA be unified to allow for a clearer, combined comparison?\n- More analysis on Figure 6 is needed to understand the dataset contribution. Which datasets contribute most? A visualization or table that clearly shows the impact of each dataset on different aspects of model performance would be helpful.\n- Is the primary difference between LLaRA and the RT-2 baseline solely the output token space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper reformulates BC data into an image-text format compatible with VLMs. This is not a new idea (RT-2), but they do introduce a new output space and auxiliary objectives. \n- Restricting output to 2D coordinates sets the model apart from prior 3D-focused models like RT-2, aligning well with tasks where 3D positioning is unnecessary. Various curated datasets support 2D grounding and action prediction, aiding spatial and relational understanding. \n- The model outperforms versions of VIMA and RT-2, particularly in low-data settings.\n- They show gains on simulation and real tasks.", "weaknesses": "- It was not directly clear to me which components contribute to performance in the RT-2 and VIMA comparison. RT-2 also uses auxiliary tasks and internet data; it's unclear how the proposed auxiliary tasks compares. An ablation of the auxiliary tasks used in RT-2 versus LLaRA would help isolate the contribution of each component.\n- The contributions of each dataset are unclear; a more interpretable format is needed to highlight key influences on performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrymxMYRYl", "reviewer": "Reviewer_HVHL", "summary": "The authors introduce a mechanism for (1) leveraging VLMs for robot task completion and (2) augmentation of trajectories with object/task auxiliary data.", "questions": "1. See weakness discussion\n2. There's work on the limitations of numerical representations in transformers.  Presumably this means we should see failure cases due to misunderstanding in this work, can the authors provide those examples or justify why they wouldn't occur in this domain?\n3. Are the augmentations validated for correctness?\n4. Can Fig 5 be augmented with another plot that show how many training samples (not just trajectories) each approach has?\n5. Why was 12% chosen (e.g. Table 1), is that simply the first time the model outperforms VIMA? Do the results plateau afterwards? \n6. Fig 6, small confirmation that the X-axis means \"2x, 4x, ...\" ?\n7. L434: What's the coverage on 0-180 [sort of related to question #2]\n8. JT is worse than FT, why? -- similar confusion or missing justifications for Tables 18/19", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "They present these results on both a simulated and physical benchmark. The paper also includes a substantial set of results in the appendix.", "weaknesses": "I'm concerned about the generality of the work, based in large part on the inconclusive trends presented.  I'll provide a series of questions below but my primary concern is that it's unclear on training.  It appears that training either has no effect or hurts performance in most cases.  There's some improvement when two epochs are run (in some conditions, but not all).  Similarly, how much data and when/where/why it helps are unclear.  This doesn't detract from the fact that a nice system was deployed but it does make it difficult to determine where this research leads in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kepvqzggRQ", "reviewer": "Reviewer_3YdQ", "summary": "The authors propose to fine-tune an LLM into a robot policy by formatting robot decision-making as textual “conversations.” They provide an automated pipeline for converting existing demonstration data into their expected conversational text format. Additionally, they augment that dataset with additional helpful features, such as object bounding boxes extracted from some other vision model / simulation ground truth.", "questions": "**[All questions should NOT be ignored, they still apply.]** \n- Why does sim LLaRA fail at the real-world tasks, if they seem to be so easy? Why doesn’t fine-tuned LLaRA achieve near 100% success, except in the last row of the FT section of Table 3?\n- Does LLaRA ever experience e.g., failure to grasp, object slipping from gripper, or other similar failure cases in sim, and if so, is it able to recover?\n- Similarly, it looks like in Table 5, LLaRA is trained on text traces that say “you have completed __.” Does it have to predict this itself, or is it just assumed that, after executing an earlier step, that step was successful, and so it can go in the action history?\n  - If that’s the case, that seems to be another weakness compared to ECoT, as the latter approach has a reasoning step that determines which step of the plan it generated it should try to execute given the current observation. Thus, if it observes a failure case, it can return to and older step to try again\n- A big point of ECoT (and even non-reasoning works that train a VLM into a robot policy, like OpenVLA) is that the VLM provides good visuo-semantic representations that could be useful for robots, especially for generalization. Is this observed by LLaRA, and if so, how / to what extent?\n  - \"In general, LLaRA demonstrates a robust capacity for generalization. Our observations confirm that it takes much less effort and data to transfer LLaRA models pretrained on simulated VIMA data to a real-world domain\"\n- Did formatting things as conversations actually help in some way, rather than just formatting in general (potentially templated) language? It seems like the conversational responses are nonetheless very structured and templated, so it’s unclear to me how much the VLM’s conversational abilities are retained (or even matter for pretraining).\n- More broadly, did formatting things in natural language make a difference? What happens if the policy's response was always just in action tokens, akin to the RT-2-style?\n  - Related: Did RT-2-style have the same expressivity as your approach? Since you're outputting <x, y> coordinates as string representations of floating points, it seems like your approach needs more tokens than what RT-2-style policies are even allowed to output. Since you output the rotation in degrees (from -359 to 359), it seems unfair to give RT-2 only a single rotation token with 256 bins. That seems to artificially make RT-2-style policies less expressive than yours.\n- Similarly, what were the failure cases of RT-2 style? How did it achieve literally 0 across all real-world tests? Likewise, what were the empirical qualitative failures that made it work less well in simulation as well? As it stands, it seems to me that one such reason would be that, by expressing object positions and motions in the same coordinate system, it becomes artificially easier, since the VLM backbone may have a strong bias to being able to just learn to copy the object coordinates as the action parameters.\n- Related: it seems like having prompts like \"Place the object at <coordinates> in the bowl at <coordinates>\" makes things a lot easier (when you've got a good object detector), especially when your actions are specifically parameterized in that coordinate space. What happens when you remove the coordinates in the prompt? This is important to test, since 2D pick-and-place of this form is one of the few robotics settings where that mapping from object coordinates to action coordinates is extremely trivial; most realistic robot learning applications will not be like that.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- LLaRA seems well-motivated, given recent interest in fine-tuning (V)LMs for control\n- It seems to make intuitive sense that, if you’re starting from an instruction-following “conversational” policy, you’d want to fine-tune it to similarly respond to “conversations.” \n  - That said, the extent to which this is the case is unclear: see questions.\n- The authors communicate the format of their data and tasks extremely well, providing extensive examples.", "weaknesses": "**PLEASE READ MY FOLLOW-UP COMMENT.**\n\n**[IGNORE] This paper is extremely similar to [Embodied Chain-of-Thought Reasoning (ECoT)](https://embodied-cot.github.io/)**, which precedes this work (being accepted at CoRL 2024) and goes uncited in the paper. For reference:\n- **[IGNORE]** ECoT also proposes to format useful features for robot control into a string, then fine-tune a vision-language model into a robot policy that predicts said strings.\n  - The paper does not explicitly state they formulate robot control as “chat,” but as reasoning instead.\n  - However, this seems to be a somewhat superficial distinction; the only difference is the extent to which the “reasoning” is dressed up as a conversation. It is unclear if there’s a reason that one format would be better than another.\n- **[IGNORE]** As with this paper, the ECoT authors propose to extract e.g., bounding boxes, which serve as an intermediate feature which the model predicts.\n  - A minor difference is that ECoT always predicts the bounding boxes as part of generating its reasoning and action, whereas LLaRA requires a separate call of the model (if I am understanding Appendix A.3 correctly), which may introduce additional system complexity. In either case, both ECoT and LLaRA can accept detections from other models as well.\n- **[IGNORE]** Both papers use pretrained models to \"supercharge\" robot demo datasets with additional automatically-generated synthetic annotations. \n  - **[IGNORE]** While the reasoning steps employed by ECoT are not identical to the auxiliary datasets in LLaRA, a lot of the ones in the latter seem either simpler than the ones in ECoT (e.g., the future prediction step does not require reasoning by an external model, just formatting of existing data into a template / rephrased variations thereof) or are intractable or not robustly detectable in the real world with careful human annotation (action prediction, temporal prediction both are only easy if you have simulator state information).\n  - In contrast, all experiments in ECoT were done in the real world, and all features and reasoning steps were extracted via pretrained foundation models. While noisy, the authors of that paper nonetheless showed empirical improvement in having the policy predict those features.\n  - Additionally, ECoT demonstrates their pipeline on the Bridge dataset, which is much more representative of existing real-world robot demo datasets (with other datasets in collaborative efforts like Open-X-Embodiment having similar structure). As the BC datasets considered by LLaRA only do 2D pick-and-place high-level actions, it seems unclear if this will be practical for the more complex kinds of actions present in e.g., OXE again.\n\n**Likewise, most of the limitations mentioned in the appendix seem to be addressed by the ECoT paper:**\n> First, some concepts in the reference images still can not be easily and precisely described by language, while in many cases these features are critical for robot control. Due to the single image limit of the current VLM, we still rely on the object detector trained on a limited number of classes to describe the additional images in the task description, which limits the generalization ability of this method.\n\n- **[IGNORE the first part, but the second part is still applicable]** ECoT specifies that they use a VLM scene captioner and open-vocabulary bounding box generator to circumvent this. This does not completely remove the issue that some scenes are just naturally hard to describe in language, but at least removes the limitation of using object detectors trained on a limited number of classes.\n\n> Second, the information extracted from the dataset can be noisy, which may lead to model confusion. In Tab. 5, there are discrepancies such as the location of an object in a reference image differing from its actual location in the current image observation. \n\n- **[IGNORE]** ECoT seems like it would be much less susceptible to noisy labels, since it’s just conditioning on them, rather than relying on them to exactly determine the action (parametrized by pixel coordinates that seem to be precisely dependent on the detected position of objects).\n- **[IGNORE]** All main experiments in ECoT were conducted in the real world, where one must expect there to always be more noise than in simulation. It thus seems like said approach seems robust to the noise.\n- Likewise, the real-world experiments for LLaRA seem very limited. In particular, from the examples shown in Table 13, it seems like all real world tasks can be completed by simply copying the values from the detector:\n  - “Rotate the <obj>.at <x, y> by <r> degrees” is correctly solved by the command “Pick <obj> at <x, y>, rotate <r>”\n  - “Put <obj1> at <x1, y1> on top of <obj2> at <x2, y2>” is correctly solved by the command “Pick <obj1> at <x1, y1>, rotate 0, place at <x2, y2>”\n  - Given that no complex spatial reasoning is needed here, this seems like an insufficient demo of LLaRA’s performance.\n  - It is likewise unclear why LLaRA pretrained in sim doesn’t succeed at this, or why LLaRA trained on real-world data doesn’t get 100% (intuitively the task seems exceedingly easy, since it just involves copying values?)\n\n> Finally, our method relies on 2D to 2D mapping to convert image coordinates into actual actions, which may face challenges when the complex 3D movement of the robot is required. We believe enhancements in the mentioned aspects could further improve performance and broader applicability in complex, real-world applications.\n\n- **[IGNORE]** ECoT states that they adopt the same tokenization scheme as OpenVLA / RT-2, which is significantly more expressive than the pick-and-place actions parametrized by start and end 2D location and rotations.\n- Some of the tasks they test on seem to be very hard to express in the simple 2D-to-2D action parameterization used in LLaRA (e.g., wiping); it thus seems clearly important to demonstrate approaches that work well with such action representations (the “RT-2 style,” as named in the paper), but there is limited evidence that LLaRA could do so, especially since so much of it seems inherently tied to the simplified 2D-to-2D case (e.g., expressing 3D rotations in language is much more unwieldy than the 2D rotation case presented in this paper).\n- Additionally, unless I am misunderstanding, LLaRA seems to only consider cases where tasks can be solved by a handful of 2D pick-and-place actions (with the real world experiments seemingly only to require one per episode). In the execution of that one action, the robot seems to be open-loop (that is, it cannot detect failure over the course of a single pick-and-place action, only before and after -- please correct me if I am misunderstanding). **[IGNORE]** In contrast, since ECoT uses \"lower-level\" actions, it can detect, reason about, and correct mistakes earlier.\n\n**Given all this, it is unclear if LLaRA makes significant contributions over ECoT.** As discussed above, there are certainly subtle differences between the two approaches, but many of those differences seem to either be design choices or in ECoT's favor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)