LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning
Authors: Dantong Niu
Yuvan Sharma
Giscard Biamby
Jerome Quenum
Yutong Bai
Baifeng Shi
Trevor Darrell
Roei Herzig
Venue: CoRL 2024 (2024)
Keywords: LMMs
Vision Action Instruction Tuning
Robot Learning
TL;DR: We propose LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic configurations and introduces the concept of visual traces to align the vision and action spaces further.

Abstract: In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question answering; yet leveraging these models remains an open question for robotics. Prior LMMs for robotics applications have been extensively trained on language and action data, but their ability to generalize in different settings has often been less than desired. To address this, we introduce LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic learning tasks, scenarios, and environments. Additionally, we show that predicting intermediate 2-D representations, which we refer to as *visual traces*, can help further align vision and action spaces for robot learning. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model, and we evaluate on 12 different tasks in the RLBench simulator as well as a physical Franka Emika Panda 7-DoF robot. Our experiments yield strong performance, demonstrating that LLARVA — using 2-D and language representations — performs well compared to several contemporary baselines, and can generalize across various robot environments and configurations.

LLARVA: Vision-Action Instruction Tuning
Enhances Robot Learning
Dantong Niu
Yuvan Sharma"
Giscard Biamby Trevor Darrell
Jerome Quenum Roei Herzig
Yutong Bai
Baifeng Shi
Berkeley AI Research, UC Berkeley
Project Webpage: httpss://i http:://11arva4 wtw.aaaaaa.....u.....aaa. github. 10/
Abstract:
In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question an- swering; yet leveraging these models remains an open question for robotics. Prior LMMs for robotics applications have been extensively trained on language and action data, but their ability to generalize different settings has often been less than desired. To address this, we introduce LLARVA, model trained with novel instruction tuning method that leverages structured prompts to unify range of robotic learning tasks, scenarios, and environments. Additionally, we show that predicting intermediate 2-D representations, which we refer to as visual traces, can help further align vision and action spaces for robot learning. We generate 8.5M image-visuda visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model, and we evaluate on 18 different tasks in the RLBench sim- ulator as well as a physical Franka Emika Panda 7-DoF robot. Our experiments yield strong performance, demonstrating that LLARVA- MAARRA-uiing- 2-D and language prrresentatiossperffoeen well compared to several contemporary baselines, and can generalize across various robot environments and configurations.
Keywords: LMMs, Vision Action Instruction Tuning, Robot Learning
1 Introduction
Recently, instruction-tuned Large Multimodal Models (LMMs), such as InstructBLIP [1], Instruct- GPT [2], LLaVA [3, 4], PALM [5] and others have demonstrated state-of-the-art performance variety of rision-and-language tasks. However, existing LMMs for robotics [6. 7, 8. 9] do not always demonstrate the same success and consistency across various embodied settings. This may result from the unique challenges encountered in robotics, such as the variability of real-world environ- ments, the differences between robots, and the need to control actions reliably. Since LMMs have been proven to be successful in part due multimodal instruction tuning, it is natural leverage this technique in robotics setting well. Here, propose vision-action instruction tuning method that can bridge the gap between a language model's 's fundamental pre- training objective- next-word prediction-and the goal of enabling the model to handle various robotics settings.
In this work, we introduce our Large LAnguage model for Robotic Vision and Action (LLARVA), an open-source instruction-tuned LMM for robotic applications that can generalize efficiently across various environments and robotic configurations. Our key idea is the formulation of novel instruc- tion prompt that encapsulates robot type, task, scene configuration, and control regime in natural language prefix amenable to contemporary LMMs. We present an instruction tuning procedure tailored to the robotic domain: when given an instruction that describes the robot model, control mode, robot task, and proprioceptive information, the model needs to predict future actions given
"Equal contribution. tEqual advising.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Visualization
Robotic Action
The 2-D visual [[3 ... [85,44]]
The next action is: [1.2,-0.3.,, -0.24]
LLARVA
You are Google robot using end-effector control. The task is "put can aside", and the previous five (including current) steps are ([-0,1060, 0.1280, 0060777,0 ...].. Can you predict the 2D visual trace of the end effector and the action of the next step?
URS EE Control Slide cloth"
Google Robot EE Control
Franka Panda Joint Control Stack red cube"
PPt can aside"
Figure 1: Overview of LLARVA. We introduce novel instruction tuning method that leverages structured prompts to unify range of robotic learning tasks, scenarios, and environments and 2-D visual traces further align vision and action spaces. The model works via language instruction that contains robot model, control mode, robot task, proprioceptive informati on, and number of predi cted steps, and outputs text with the robot action (s) and the visual trace for the remainder of the episode.
the natural language prompt. This architecture allows us to leverage structured language prompts as a "lingua franca" for robotic perception and control (See Figure 1).
However, aligning the vision and action modalities to produce meaningful robotic outputs is still not trivial task. Though recent robotic models have used 3-D representations such as voxels and point clouds to overcome this, these representations are difficult to incorporate into most existing open-source LMMs because they typically accept a single image plus language as input. For these reasons, we utilize 2-D images, which are easy to scale and integrate with existing LMMs.
We find that predicting intermediate 2-D representation, which refer to visual traces, help align the vision and action spaces across different robot and task configurations. In particular, we generate the 2-D visual trace (projection) of an end-effector and force the model to predict this trace alongside the next robot action(s). This waypoint prediction helps align each robotic action to the end-effector's location, allowing the model to focus on fine- ane-grained localization and resulting in more accurate prediction of robot actions. To achieve this, we construct instructions with such visual traces using the Open X-Embodiment dataset (OXE) [10], carefully categorizing the action space, the robot type, and the control type.
Through empirical study, we show that our vision-action instruction tuning approach using struc- tured prompts leads to generalization across various robot environments and configurations. Addi- tionally, we show that predicting visual traces can help further align vision and action spaces. We evaluate LLARVA on 18 different tasks in RLBench's simulated environment and on picking, stack- ing, and destacking tasks with real 7-DoF Franka Emika Panda robot. Finally, evaluate model's generalization across robots in RLBench on four tasks. We show that LLARVA using 2-D and language pnnresentations-perriee well compared to several contemporary baselines.
2 Vision-Action Instruction Tuning
2.1 Preliminaries
LMMss are designed handle multiple data modalities simultaneously. such as images and their corresponding text descriptions. Each modality encoded into a shared embedding space, which is then utilized for reasoning by language model f parameterized by 6. Specifically, an image is encoded using pre-trained visual encoder, denoted parameterized by 6. A corresponding text

---

description is tokenized and encoded using a fixed language encoder parameterized by 7. Given an input image o and a language task description 1, the language model generates a text response R as follows: R = fo(vo(o),
In this paper, we use LMM within the context of robotic episodes, which are characterized by temporal sequences of visual observations o1:N and proprioceptive states $1:N- Here, NN denotes the length of an episode. Notably, in the realm of LMMs for robotics applications, the output R typically encompasses one or more predicted actions for an episode. Next, we describe our LLARVA model.
2.2 LLARVA Model
2-D Visual Traces. Visual traces play a key aspect our vision-action instruction methodology. The choice 2-D traces is made to match the high availability of image-based large robotics datasets such OXE, but our method can also be implemented with 3-D data. To achieve alignment between visual inputs and robotic actions, predict visual traces as auxiliary task, as we find that this helps to gain better fine fine-grained localization, resulting in a more accurate prediction of robot actions.
We define 2-D Visual Traces as a sequence of coordinates (z, y) in a two-dimensional space, which aligned with the input image o at time step t. These coordinates represent the trajectory of the gripper (or end-effector, hand, etc.) throughout the episode. The visual trace timestep | is:
PV.N (II, vi) i t, +
(I)
Here, (Ia, Vi) denotes the i-th coordinate in the entire visual trace for the episode, and N represents the number of time steps in the episode. We note that language model decoders are crucial in converting multimodal inputs into actionable outputs in robotics. By leveraging the shared vision- action embedding space, our decoder produces responses that the robotic system can use.
LLARVA architecture comprises two components. First, we have the visual observation oz, an image capturing the state of the environment at timestep t. Second, we have the language instruction input 4, which prompts the model to forecast specified number of subsequent steps, integrating embodied information such as the robot, control mode, and previous propriocep- tive states as well as the task directives. Specifically, we formulate an instruction template featuring the robot type R (e.g., Franka, UR5, xArm), control mode M (e.g., joint or end-effector control, absolute delta control), task instruction I (e.g., "open the drawer"), proprioceptive information S (c.g.. positions velocities), and query indicating the number of future actions to predict, denoted as The complete instruction is formulated as follows:
Input. The input
= "You are [R] robot using [M] control. The task is [2], and the previous [h] steps are [S]. Can you predict the trajectory of the end-effector and the action of the next [n]] steps?"
To develop a versatile and adaptive framework capable of accommodating training for tasks with varying time horizons, we add flexibility to the proprioceptive information input. Specifically, this information structured as S 8t-Axt, representing sequence of past joint and/or gripper states. Here, is the number of previous time steps the model is conditioned on, and is decided based on the task. This approach ensures robustness and adaptability across spectrum of task durations, enabling effective training for both short-term and long-term objectives.
Architecture. Our objective is to develop a model capable of predicting robotic actions that exhibit generalization across diversity of robotic tasks, scenarios, and environments. The model archi- tecture is illustrated in Figure 2. Our instruction-tunee model æ designed leverage both the current visual observation o and the accompanying language instruction 4 input. Subsequently, it predicts the action sequence for the next steps Autn-1 and the future 2-D Visual Traces of the end-effector Pr.N, spanning from the current step to the final step within the episode:
4) + Am PEN
(2)
where 4 is constructed as defined above.
3

---

"The 2-D Visual Trace: P.N The action:
Autin-i-"
In our proposed pipeline, the input im- age undergoes processing by the frozen vi- sion encoder (-).), which extracts visual features and projects into a latent space an MLP layer H. This aligns the vi- sual features with the dimensionality of the language tokens. Simultaneously, the language input undergoes tokenization us- ing a language encoder. The visual tokens and word tokens are then concatenated and fed into the auto-regressive transformers of the LMM fo. which are trained for next- token prediction.
Language Decoder
fo
Auto-regressive lrrnnsormmers fo
Projection Layer H
Language Encoder
Cry
Visual Encoder f
Language Input 4
Image Input Oş
Tunable weights
Frozen weights
Figure 2: Architecture of LLARVA.
2.3 Training
While keeping the vision encoder and language encoder frozen, we use instruction tuning to train the auto-regressive transformers using standard LoRA adapters [11] for both the pre-training and fine-tuning stages. Each image o, for an episode is accompanied by a language instruction 4, and the predictions consist of robotic actions Am+4n- and visual traces P..v Next, given on and 4, we predict the next actions and 2-D visual trace. Specifically, for a response R. we compute the probability of the target actions and target visual traces by the following equation:
R
p(Auttn-1, PLN ot, (6)
II (o(ss oz,
where 0 represents the trainable parameters, H; is the current prediction token, and n N. To calculate loss, we the standard cross-entropy function with these probabilities. Next, we describe our two-step training process, the large-scale pre-training and the fine-tuning for downstream task.
Step 1: Vision-Action Instruction Pre-training. We begin with an LMM that has been pre- trained on vision-language (VL) tasks. In order to generalize across robotic tasks, scenarios, and environments, the model pre-trained on our large-scale vision-action instruction dataset. Due to the diversity of this dataset, model is trained simultaneously for multiple configurations of prompt variables such as robot type R. control mode M or task instruction I 1 Using language as input allows us to bridge fundamental gaps between subsets brought by these different configura- tions. This extensive and varied training process can establish a powerful LMM framework that can be further fine-tuned and adapted to handle various robotic settings. We note that this pre-training stage is different from standard LMM pre-training. As opposed to aligning the two modalities using a projector in VL, here we align the two modalities for generalizing robotic configurations.
Fine-tuning for Downstream Tasks. Unlike other fields, robotic model must be fine- tuned on downstream task before it can be evaluated due to the practical considerations of real- world physical properties. Therefore, we fine-tune the pre-trained model using a small dataset with a fixed configuration for the factors defined in Section 2.2 (e.g.. the instruction has the same robot type R. control mode M, etc.). Having seen diverse data samples makes it easy for the model to adapt specific downstream settings resembling what has already encountered in pre-training. Given new tasks, environments, robot types, LLARVA can be adapted by fine-tuning demonstrations. In addition, data from new modalities such as 3-D, depth or tactile information can potentially be incorporated and utilized in the fine-tuning by modifying the instruction template.
Step 2:
example
When LLARVA to be used for a new setup or environment, this fine-tuning must first be carried using example demonstrations.
Details of the configuration for each subset are available in Section B of the Supplementary Material.

---

Task
Method
open off
put
push sweep slide close serew place reach and stack
drawer
grill
tap
money buttons dustpan block jar
bulb
wine
drag
blocks
3-D methods
C2FARM-BC
20
20
68
12
72
0
16
24
8
18
24
PERACT
80
84
80
44
48
56
72
60
24
12
68
36
2-D methods
Image-BC (CNN)
4
0
8
4
0
0
0
8
0
0
Image-BC (ViT)
16
0
0
0
0
16
0
LLARVA
60
80
56
44
56
84
100
28
8
12
52
Table 1: Success rate (%) on RLBench Multi-Tast setting, We fine-tuned (with visual trace prediction) the pre-trate trained model on 12 tasks and evaluate with 25 episodes per task. Each evaluation episode is scored either 0 for failure or 100 for success. We gray out methods with 3-D information.
2.4 Vision-Action Instruction Dataset
In order to pre-train LLARVA, we generate 8.5M image-visual trace pairs from the Open X- Embodiment (OXE) dataset [10]. As shown in Figure in Supplementary, our dataset consists of images from diverse collection of 37 OXE subsets with 13 different robots, including wide assortment of tasks, environments, cameras (and thus images), and end-effectors, among other fac- tors. For each image in an episode, we calculate the 2-D visual trace of the end-effector Pe.N. For this purpose, we use a bounding box detector [12] that is trained specifically on each of the different end-effectors OXE, The center points bounding boxes are used for a simpler representation, and the visual trace for step t then the ordered list of all center points from image t image N. More details of the dataset shown in Section B in Supplementary.
3 Experiments and Results
We evaluate LLARVA on 18 tasks in RLBench and compare to both existing 2-D and 3-D models. In addition, we also test on real 7-DoF Franka Emika Panda robot.
3.1 Implementation Details
LLARVA is implemented using PyTorch [13] with the official LLaVA 1.5 [4] implementation. The base LMM uses a Llama 2 7B -arpaaameter LLM, the default image projection layer, and the CLIP ViT- L/14 vision encoder. We use 8 NVIDIA A6000 GPUs for training, and A6000 GPU for evaluation. Additional information, such as training and fine-tuning recipes, are in Supplementary Section C.
3.2 RLBench Evaluation
Experimental Setup. We evaluate using the same 18 RLBench tasks in [7]. In the fine-tuning stage. the front view is chosen Oz, conditioned the previous 5 joint positions (i.e. h 5). LLARVA predicts the visual trace and the next action step (i.e. n 1). which is an 8-dimensional vector consisting of 7 joint velocities and a binary gripper state. For evaluation, we take 25 episodes per task the validation set and score each episode either for failure or 100 for success. We use seeds, which are averaged to get the final success rate.
Baselines. We compare LLARVA to several baselines using 2-D and 3-D information. Image- BC (CNN) and Image-BC (ViT) [14] are 2-D language-conditiones models that use CNN and ViT vision encoders, respectively, reported in PerAct [7]. The PerAct [7] model uses voxels as 3-D input to calculate actions, as does C2FARM-BC [15]. In contrast to the 3-D line of work, our input uses only one camera view without any 3-D information.
Results. We show results for 12 of the tasks in Table 1; additional long-horizon task results (such those with multiple steps or subtasks) presented in appendix A. Our model completely out-
5

---

Prod 2-D Visual Trace
Ground Truth
Obs
Pred
dottak
dctack
Eclads
cvankr
pic.cu
zedrerange dannz
Obs
Pred
light
topushell
pash geven
Figure 3: We visualize the ground truth (yellow line), and predicted 2-D visual trace (green dash line) after downstream tasks fine-tuning. Our predicted 2-D visual trace gives reasonable route planning to achieve the goal, even sometimes diverges from the ground truth.
Task
Method
2-D Visual Trace
pick
stack
destack
one button
buttons
three buttons
RPT [16]
87.50
31.25
93.75
Octo [8]
56.25
12.5
37.5
53.75
8.75
I.ARVA
x
81.25
50.00
87.50
96.25
68.75
32.5
LLARVA
93.75
56.25
100
97.5
83.75
58.75
Table Success rate (%) LLARVA on a real robot. We compare LLARVA with RPT and Octo by taking each pre-trained model and fine-tuning them the same of demonstrations, LLARVA outperforms the others on all the tasks.
performs other 2-D based methods: achieve average success rate of 43.3%, while Image-BC (CNN) and Image-BC BC (VIT) both achieve 1.3%. In addition, LLARVA competes with and even beats 3-D based methods, like C2FARM-BC, which averages 22.7% %, and PerAct, which achieves 55.3%. We note that 2-D methods here also include those that use multiple input images from dif- ferent camera views, while LLARVA uses only one camera view and displays stronger performance. Moreover, we observe that despite some cases of occlusion that arise from using just one camera view, LLARVA still succeed in such situations, showing the adaptability of our model.
3.3 Real Robot Evaluation
Experimental Setup. Following the setting in [16], we use 7-DoF Franka Emika Panda robot with the default 1-DoF parallel jaw gripper. The input image comes from the right side RGB camera. We evaluate ILARVA on three pick tasks: "pick cube", "destack cube". and "stack cubes", and another task "push buttons".. During fine-tuning, we use the same number episodes (1920) as [16] for pick tasks. The "push buttons" task is more complicated real world task than the pick tasks as it requires the model to push up to three buttons from four color variations (red, blue, green, yellow) in specified sequence. The setup is shown in Figure 4 in Supplementary. In our experiment, we use our pre-trained weights on OXE to fine-tune the model with total of 450 episodes (150 episodes each for the one, two, and three button tasks).
During training, we condition on joint positions of the previous steps, and predict 7-dimensional delta joint positions and 1 I-dimensional gripper status for the following 16 steps. To calculate the final success rate, we evaluate LLARVA over 16 episodes and take the average across 5 runs.

---

Instruction
2-D
Task
Task
Pre-
Visual Trace
reach and
place meat off
slide
Training
drag
wine
grill
block
Robot
sweep
put
push
meat off
x
40
8
36
48
dustpan
money
buttons
grill
x
v
48
12
40
76
Franka
84
44
56
80
x
44
4
56
80
52
12
80
100
Sawyer
80
48
48
72
Table 3: Left: The effect of the instruction pre-training and 2-D visual trace. We ablate the in- struction pre-training (step in Section 2.3) and the visual traces for four random tasks in RLBench. Right: Evaluation across different robots. We show the success rate (%) of four randomly chosen tasks in RLBench. Using the same training recipes, we fine-tune our pre-trained model with visual trace but different robots. For evaluation, we take 25 episodes per task, and each of them is scored for failure 100 for success.
Baselines. We compare LLARVA with RPT [16] and Octo [8], both of which claim the benefit that pre-training brings the downstream tasks. RPT uses BERT-like [17] formulation and must additionally pre-train in-domain data, while Octo is pre-trained on the mixed dataset OXE and maps the various configurations to the same action space. In contrast, LLARVA is pre-trained in more diverse manner, without imposing a single action space, via a unified language template.
Results. We show the results in Table LLARVA achieves the highest success rate across all pick tasks. Additionally, compared to RPT, LLARVA only needs one unified pre-trained model that is first instruction-tunce
dataset and then fine-tuned on each downstream task. In contrast, RPT needs to be separately pre-trained on each individual task before being adapted to them, which brings additional time and computation cost. In addition, Octo has difficulties adapting to a downstream task using different control mode as the model has only seen end-effector control in the pre-training stage. LLARVA shows better quality both in terms efficiency for adapting to downstream tasks and compaaibllityygenrralzzaooo with different control modes.
Robot Planning with 2-D Visual Traces. To align the vision and action spaces, we predict the end- effector's visual traces, which forces the model to develop more comprehensive understanding. In Figure 3, we show through visual traces that our model can plan alternative yet correct paths. For example, in the top left image, the gripper takes a different path (go left) than the ground truth (go right), yet it still succeeds in destacking the cube. Figure also shows qualitatively that the visual trace can help with long-horizon tasks by acting number of previous robotic states the model can handle. For instance, in the bottom right image in Figure 3, the task is "push maroon button, then push green button." It can be seen that the visual trace helps the model reach the "green button" after finishing the first subtask.
memory buffer that compensates for the limited
3.4 Ablations
The Effect of Instruction Pre-training. We ablate the instruction pre-training in Section 2.3 to evaluate its effect. As shown in Table 3, our model can achieve an average improvement of 17.5% across the four tasks regardless of visual traces. We believe this improvement due to our diverse, large-scale pre-training, which leverages language instructions to create a strong robotics L.M backbone that can successfully adapt to downstream robotic settings.
The Effect of 2-D Visual Traces. To test the importance of visual traces, we fine-tuned the model with and without the visual trace prediction task. Table 3 shows that the traces provide 15% average improvement across the four tasks. Complete visual trace ablation results with additional analysis can be found in Table 5 and appendix A.1. In particular, an improvement can be seen in tasks with frequent occlusion objects such as "meat off grill" and "slide block." We note that using both pre-training and 2-D visual traces can help the model complete the task even when the target object is not visible. The "put money on top shelf" example in Figure 3 demonstrates this: the shelf is not visible, and the gripper becomes occluded once it moves behind the counter. While the
7

---

results show that visual traces increase performance across all tasks tested, we find that the greatest benefit of visual traces occurs in "meat off grill" and "slide block", both tasks with distractor objects.
Cross-Robot Generalization. To evaluate LLARVA's generalization across different robots, we randomly selected four tasks in RLBench, and finetuned and evaluated on Sawyer instead of Franka Emika Panda. As shown in Table 3, LLARVA achieves similar results across both robots, which further proves the generalization brought by our large-scale vision-action instruction pre-training.
4 Related Work
Instruction Tuning. LLMs [18. 19, 20] are typically pre-trained on a large corpus of text with an unsupervised training objective of next-word prediction. Instruction tuning (IT) helps to bridge the gap between the language model's fundamental pre-training objective of next-word prediction and the user's goal of having the model perform specific tasks using input-output pairs whose inputs include text phrased as instructions. Flamingo [19]. GPT-4 [21], and BLIP [22. 23] were pioneering early LMMs, and LLAVA [24] prompted GPT-4 with image-caption pairs to generate multimodal instruction tuning data. Many recent VL models [1, 25, 26, 27, 28, 29] have followed this approach, as IT has been shown improve generalization in zero-shot and few-shot tasks [2, 30, 31]. Unlike these works, here we construct instructions from custom template that fill in with relevant information about the robotic episode.
Language-condiitione Robot Agents. Recent language-conditioneed models for robotics have tried different approaches, such as generative video pre-training (GR-1 [32]), tokenized voxels for incor- porating 3-D information (PerAct [7]), co-training on internet-scale VL data (RT-2 [9]), and large- scale diverse pre-training with transformer architecture (Octo [8]). However, each approach has distinct drawbacks that we attempt to address in model. For instance, GR-1 requires archi- tectural changes to account for different action spaces when fine-tuning, while LLARVA is more flexible and does not need such changes. Moreover, PerAct relies on 3-D information unavailable on a large scale across environments, which is why we chose to use 2-D images when implementing our model. RT-2 uses co-training on a separate web-based dataset, while we more efficiently use instruction-tuning on a smaller, diverse vision-action instruction dataset to create a strong backbone. Lastly, Octo uses a specialized head to predict actions, while the language head in our model is trained predict both robotic actions and visual traces in "natural" language.
Trajectory Prediction using Object Representations. Trajectory modeling has been a funda- mental aspect of many machine vision applications. Many previous works have investigated the use of object representations for trajectory modeling, ranging from classical image understanding tasks [33, 34, 35] to an object-centric approach for video understanding [36, 37, 38, 39, 40] with object tracking and interactions to even scene graphs [41, 42, 43, 44]. More recently, there has been an increased interest in trajectory modeling effectiveness for vision and language, such the refer- ring expression localization task [45, 46, 47] and semantic segmentation using text prompts [48, 49]. In particular, the localized narrative dataset [50] enabled new task that aims to align a long and detailed image caption to a human trajectory. Our work takes a different approach as we use LMMs to exploit the concept of a 2-D visual trace for implicit end-effector object representation.
5 Conclusion
Our proposed model, LLARVA, represents significant advancement in the application of instruction-tuned LMMs for robotics. By leveraging structured prompts to unify a range of robotic configurations and introducing the concept of visual traces, we have demonstrated a generalization method that better aligns vision and action modalities. Our extensive training on 8.5M image-visude visuul trace pairs derived from the Open X-Embodiment dataset and evaluation in both simulated and real- world settings highlight the model's superior performance and generalization capabilities compared to existing approaches. This work marks meaningful step forward in the integration of LMMs with robotics, promising enhanced adaptability and efficiency in various robotic applications.
8

---

Acknowledgments
We would like to thank Max Fu for his assistance with data collection, Ritwik Gupta for his compute support, well as Xiaolong Wang and Lija Radosavovic for their helpful feedback and discussions. We also thank Nicole Walters for creating our lovely logo.
References
[1] W. Dai, J. Li, Di, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. Instructblip: Towards ceneral-purposs vision-language models with instruction tuning, 2023.
[2] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[3] H. Liu, C. Li, Q. Wu, and Lee. Visual instruction tuning. In NeurlPS, 2023.
[4] H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning, 2023,
[5] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. C. Hutchinson, R. Pope,
J. Bradbury, Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, Dev, H. Michalewski, X. Garcfa, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Díaz, O. Firat, M. Catasta, J. Wei, K. S. Meier- D. Eck, J. Dean, S. Petrov, and N. Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240:11 13, 2022. URL https: //api.app.p/ www.:iieeeaaaaihhhh...oooo org/ Corpus ID: :ID::199799...
[6] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tomp- son, Q. H. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng. I. Mordatch, and P. R. Florence. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, 2023. URL https: / //ppp/pp.../// semanticscholar org/ Peg/CorrpssID... 257364842.
[7] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[8] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy, arXiv preprint arXiv:2405.12211 2024.
[9] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, K. Choromanski, T. Ding, D. Driess, K. A. Dubey, C. Finn, P. R. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, I. Leal, S. Levine, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. S. Ryoo, G. Salazar, P. R. Sanketi, P. Sermanct, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q H. Vuong, A. Wahid, S. Welker, P. Wohlhart, T. Xiao, T. Yu, and B. Zitkovich. Ri-2: Vision- language-ace
models transfer web knowledge to robotic control. ArXiv, abs/2307.15818. org/ :Dtp://orpsII:::::: 2602933142
2023. URL https: / http://ppp.aaaaaiiiceee www.steiaacnheeeeeee
[10] O. X.-E. Collaboration, A. O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee,
A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna. A. Wahid, B. Burgess- Limeriik, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, Xu, C. Le, C. Finn,

---

C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Mor- ton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Wang. H. Su, H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen. H. Furuta, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang,
Kim, J. Drake, J. Pe- ters. J. Schneider, J. Hsu, J. Bohg. J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang. J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. J. Fan, L. Ou, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, J. J. Kim, N. Kanazawa, N. Hansen, N. Heess, J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart Mart'inn in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Busta- mante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Song, Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davehev, T. Z. Zhao, T. Arm- strong, T. Darrell, T. Chung, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang. X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. https: / aarxiv.vvvv.....//aa.a/aaaaa/aa org/ ab3/ 088644 2023.
[11] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0968: 2021.
[12] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick. Detectron2. facebokkresarcccceeetttozz 2019.
com/
[13] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[14] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learn- ing, pages 991-1003 PMLR, 2022.
[15] James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13748, 2022.
[16] I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and J. Malik. Robot learning with sensorimotor pre-training. Conference on Robot Learning, pages 683-693. PMLR, 2023.
[17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 04805, 2018
[18] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan,
P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin,
10

---

S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https::/ppooeeeddgss neurips. cc oat.://:..aapaaaa00/////////0660 /TTT1//0000TTT0000000000000000.0
pdf.
[19] J.-B. Alayrac, Donahue, Luc, A. Miech, Barr, Hasson, K. Lenc, A. Mensch, K. Millliaan, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Saman- gooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a vi- sual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022. URL https:
semaaaicccnn www.oooggg........ htt.a//aaaaaaataoooooooaoaaaaaaa 248476411.
[20] H. Touvron, Lavril, G. Izacard, Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, Hambro, Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023, URL https:
//api.semante www.sseeaaannoooo.e.eeem.. .rrg Werg/CorpuSID... : 257219404..
[21] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. semanticscholar org/CorpusID :255532281 257532815.
URL https://api.
[22] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified
vision-language understanding and generation. arXiv preprint arXiv:2201. 0a:::2001.20000 12086, 2022.
[23] J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[24] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[25] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4; Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[26] H. Zhang. X. Li, and L. Bing. Video-llama: An instruction- tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. URL https://arxi::
brg/abs/2300 02858.
[27] Q Ye, H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q. Qian, J. Zhang, F. Huang, and J. Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. ArXiv, abs/231 1.04277, 2023. URL https: / //ppi. semanti wtapatticcccolaoooo//rrrrrrlpee org/CorpusID: 265050943.
[28] Zhang, X. Wang, Cao, C. Xu, L. Ouyang, Zhao, S. Ding, Zhang, H. Duan, H. Yan, X. Zhang, W. Li, J. Li, K. Chen, C. He, X. Zhang, Y. Qiao, D. Lin, and J. Wang- Intemln-xxcoposser A vision-language large model for advanced text-image comprehension and composition. ArXiv, abs/2309.15112, 2023. URL https: //tpp/app..... semanticscholar. org/Corpus /Bg//orrpusDD::222445877 262824937.
[29] B. Li, Y. Zhang, L. Chen, J. Wang. F. Pu, J. Yang, C. Li, and Z. Liu. Mimic-it: Multi-modal in-context instruction uning. arXiv preprint arXiv:2306.05425 2023.
[30] H. W. Chung, Hou, Longpre, Zoph, Y. Tay, W. Fedus, Li, X. Wang, M. Dehghani, Brahma, al. Scaling -inettinn-- language models. Journal of Machine Learning Research, 2577))1-- 2024.
[31] J. Wei. M. Bosma. V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models zero-shot learners. arXiv preprint 2011. 2021.

---

[32] H. Wu, Y. Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong, Unleash- ing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:23/2.13133 2023.
[33] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 28. Cur- ran Associates, Inc., 2015. httptttpp:p//pppepiiiiiee pes.urdddnng.euurrppp....... wttp:/ccppppppe///teeee.ee saaper.dft.
1 14bf 441226bb118754444222222222228833
[34] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In 2017 IEEE International Conference Computer Vision (ICCV), pages 2980-2988, 2017.
[35] W. Battaglia, J. B. Hamrick, Bapst, A. Sanchez-Gonzalez, Zambaldi, M. Malinowski,
A. Tacchetti, D. Raposo, A. Santoro, Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 180.000611... 2018.
[36] R. Herzig, E. Ben-Avraham, K. Mangalam, A. Bar, G. Chechik, A. Rohrbach, T. Darrell, and A. Globerson. Object-region video transformers. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[37] E. B. Avraham, R. Herzig, K. Mangalam, A. Bar, A. Rohrbach, L. Karlinsky, T. Darrell, and A. Globerson. Bringing image scene structure to video via frame-clip consistency of object tokens. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
[38] X. Wang and Gupta. Videos as space-time region graphs. In ECCV, 2018.
[39] F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori. Object level visual reasoning in videos. In ECCV, pages 105-121, 2018.
[40] D. Shan, J. Geng, M. Shu, and D. Fouhey. Understanding human hands in contact internet scale. In CVPR, 2020.
[41] Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668-3678. 2015.
[42] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene Graph Generation by Iterative Message Passing. In CVPR, pages 3097-3106, 2017.
[43] R. Herzig, A. Bar, H. Xu, G. Chechik, T. Darrell, and A. Globerson. Learning canonical representations for scene graph image generation. In European Conference Computer Vision, 2020.
[44] A. Bar, R. Herzig, X. Wang, A. Rohrbach, G. Chechik, T. Darrell, and A. Globerson. Compo- sitional video synthesis with action graphs. In ICML, 2021.
[45] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Mdetr mod- ulated detection for end-to-end multi-modal understanding. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1760-1770, 2021. URL https: semanticscholar. Mwrg/CorpusID:23333 233399992.
[46] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 11-20, 2016.
[47] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring ex- pressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part II 14, pages 69-85. Springer, 2016.
12

---

[48] X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia.
Lisa: Reasoning seg- URL. httpss//appl
mentation via large language model. ArXiv, abs/2308.00692 2023.
semanticscholar. www.llarrrrrroooee arg/CorpusID:: 260351228.
[49] T.-H. Wu, G. Biamby, D. Chan, L. Dunlap, R. Gupta, X. Wang, J. E. Gonzalez, and T. Darrell. See and segment: Teaching Imms to overcome false premises. In Proceedings of the 1EEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 13459- 13469, June 2024.
[50] J. Pont-Tuset, J. R. R. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari. Connecting vision and language with localized narratives. In European Conference on Computer Vision, 2019.
URL https: / /app/.pp.../// www.atiicccolleeee.............e org/CorpusID 208957532.
[51] H. Liu, W. Yan, M. Zaharia, and P. Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint 2402:2240,.08268, 2024.
[52] R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440, 2015.
13

---

Supplementary Material for LLARVA
Here, we provide additional information about our experiments, our model's emergent properties, our constructed dataset, and implementation details. Specifically, Section A provides additional ex- periment results, Section provides details about our constructed vision-action instruction dataset, and Section provides additional implementation details.
A
Additional Experiment Results
A.1 Additional Experiments
Task
Method
open drawer
meat off
turn
put
push
sweep dustpan
slide block
close
grill
tap
money
buttons
jar
bulb
3-D methods
C2FARM-BC
20
20
68
12
72
0
16
24
PERACT
80
84
80
44
48
56
72
60
24
2-D methods
Image-BC (CNN)
4
0
8
4
0
0
4
0
Image-BC (VIT)
0
16
0
0
0
LLARVA
60
80
56
44
56
84
100
28
Task
Method
place wine
reach and
stack
put in
sort shape
insert
stack cups
put in
place
drag
blocks
drawer
peg
cupboard
cups
3-D methods
C2FARM-BC
8
4
PERACT
12
68
36
68
20
0
0
16
2-1> methods
Image-BC (CNN)
0
0
0
8
0
0
0
0
Image-BC (ViT)
0
0
0
0
0
0 0
0
LLARVA
12
52
0
0
0
0
0
0
Table 4: Success rate (%) RLBench Multi-Task setting. We fine-tuned (with visual trace prediction) the pre-trained model on 18 tasks and evaluate with 25 episodes per task. Each evaluation episode is scored either 0 for failure 100 for success. We gray methods with 3-D information.
Complete Simulation Results on RLBench. Following [7, 15], we evaluate LLARVA on 18 tasks in RL.Bench, with the comprehensive results presented in Table LLARVA demonstrates signifi- cant improvements over 2-D methods and shows comparable performance to 3-D methods in most tasks. However, for "long horizon" tasks, which extended durations. LLARVA exhibits similar limitations to other methods. Specifically, for the "place cups" task, where success is defined by placing a specified number of cups rack, our experiments reveal that the model often successfully places the first cup but then becomes confused and wanders randomly. As discussed the main paper, while the introduction of 2-D visual traces provides the model with a rudimentary sense of memory (as seen the "push buttons" case), the length of this memory remains limited along the temporal axis. This issue can be partially addressed by incorporating additional information from previous steps; however, it will place greater demands on the maximum context length that vision-language models (VLMs) can handle. Fortunately, sev- eral recent works, such as [51], have demonstrated promising solutions and performance for VLMs with long context length.
complex and involve multiple sub-steps
Real Tasks: Push Buttons. We evaluate LLARVA on "push buttons"", a more complicated real world task than the pick and place tasks involving cubes. This task requires the model to push up to three buttons from four color variations (red, blue, green, yellow) in specified sequence. The setup is shown in Figure 4. The variation in the number, order and color of buttons makes the task
14

---

as well as the required motion of the end-effector more complicated. In our experiment, we use our pre-trained weights on OXE to fine-tune the model with total of 450 episodes (150 episodes each for the one, two, and three button tasks). In our evaluation, we have three tasks: push one button given a specific color, push two buttons given specific colors and order, and push three buttons given specific colors and order. Results are presented in Table 2.
Figure 4: LLARVA completing the task "push red button, then blue button, then green button" on the Franka 7 DoF setup. The predicted 2-D visual trace is also visualized, with purple representing the current timestep and yellow representing the predicted end of episode.
Visual Trace Ablations for 12 RLBench Tasks. For a more comprehensive ablation study on the 2-D visual traces, we extended our ablation study described in Section 3.4 to 12 tasks. We present these results Table 5. Overall, we find an average improvement of 13.7% with the visual traces, which further demonstrates their utility. We further note the improvement in tasks with distractor objects/variatios such as meat off grill, open drawer, and slide block, and long horizon tasks such push buttons.
open meat off turn put
slide close screw place reach and stack money buttons dustpan block jar
push
sweep
2-D Visual Traces
drawer
grill
tap
bulb
wine
drag
blacks
x
36
80
72
56
52
24
44
0
24
4
24
0
60
100
84
80
56
28
52
8
44
12
56
Table 5: Results for the visual trace ablation study across 12 tasks in the RLBench setting.
Vision and Language Modality Ablations. We performed an ablation study on the vision and language modalities to test the importance of each. For the vision-only ablation study, when fine- tuning, we remove the language instruction mentioned the main paper and only feed in the visual input (images) to the model: the ground truth output stays unchanged. We evaluated on the simulated 12 RLBench tasks, and this model has a 0% success rate. The main reasons for this are: (1) the model does not receive any language instruction, and therefore cannot determine what task to perform, and (2) the model is no longer guided by the proprioceptive information acquired during the previous steps. We also evaluated the model on the real Franka Panda DoF setup for the three tasks, and performed poorly in this case as well. Results are presented in Table 6. can be seen that tasks that have several variations (such as several cubes) are very hard to complete.
For the language-onle ablation study, do not input any images to the model: only feed the language instructions. This model again has 0% success rate across both RLBench and real tasks.
15

---

We note that this result is expected since the model is unable to see the environment, and therefore can never know where exactly the robot is with respect to the target objects.
Model
pick
stack
destack
LLARVA (language-only)
0
LLARVA (vision-only)
5
0
3.75
LLARVA (vision & language)
93.75
56.25
100
Table 6: Results on vision and language ablation studies conducted on our real world Franka setup.
Additional Explorations for Behavior-speciffc Tasks. In order to provide a more comprehensive evaluation of LLARVA, we explored additional tasks in RLBench with specific behavior patterns. The results for 5 additional tasks are presented in Table 7. using the same fine-tuning and evaluation settings as in the main paper. These 5 tasks are categorized into two types: "Bending Tasks" and "Bending Tasks," " the robot arm required to grab the target object and move it down to certain height. LLARVA demonstrates excellent performance on these tasks. In "Placement Tasks," the robot arm must grab the target object and move it pre-specified area. LLARVA performs well overall, except in the "grab" or "place" stages. For example, in the "put knife" task, most failures occur because the gripper misses the thin and delicate handle of the knife. Conversely, in the "put umbrella" task, most failures occur during the "place" stage, as the umbrella stand has a very small hole requiring precise positioning of the gripper during insertion. These issues are primarily due to the lack of detailed information from the visual observation, given that LLARVA uses only single view image with a 128x 128 resolution. Our future work will try enable our VLM process multiple view images or to adapt it with more informative vision encoder that can better capture task-related features.
"Placement Tasks".
requiring delicate operations during either
Bending Task
Placement Task
2-D visual Trace
toilct seat
close
put
put
move
down
laptop lid
knife
umbrella
hanger
X
88
56
36
88
Table 7: Evaluation results on more tasks in RLBench. We explore additional tasks in RLBench, which can be further categorized into "Bending Task" (i.e. the robot arm is supposed to grab the target object then bend and move the target down) and "Placement Task" (i.e. the robot arm is supposed grab the target object, hold and move it to a specified area).
A.2 Emergent Properties
Multiple Attempts after Failure. The top of Figure shows example of the model failing to pick up object, and then retrying soon as the end-effector comes back into view without the object in its grasp. Specifically. the third image represents the moment where the end-effector be- comes visible again, and LLARVA then attempts to complete the task again. This behavior emerges from the fact that the instruction prompt fed into LLARVA at this moment is similar to the prompt at the start of the first attempt, with the main difference being the previous actions/positions included in the prompt. We highlight this as an interesting emergent property since the training data does not include any examples with such behavior.
Handling Obstructed Views. In both pre-training and fine-tuning stages of LLARVA, we use only a single camera view to provide visual inputs. Using only 2-D inputs creates a challenge since robotics tasks require very accurate action predictions in three dimensions.
The model should be able the exact location of the target and also have sense of depth, which other works typically achieve by using either multiple camera views or 3-D representations.
16

---

The task in this episode is "take the steak off the grill". As we see in the third image, LLARVA fails to pick up the steak in the first attempt. However, it tries again and succeeds the second time, showing capability of attempting a task multiple times. Bottom: The single view which is used as the model input shows the top and back of a safe. The task for the robot to move the money stack from the top the safe one of the shelves inside the safe. LLARVA can still move the money to the correct shelf in the safe despite the camera view not showing the shelves.
Figure 5: Emergent Properties. Top:
We note that our model uses single camera view due to input limitations of current open-source LMMs. These limitations can certainly be overcome and are left for future work.
Using single camera view presents further challenges when objects in the scene occlude each other. However, find that LLARVA can often complete tasks even in these occluded situations. For example, as shown in the second row of Figure 5, the task is "put the money away in the safe on the top sheif". The camera view only shows the top and back side of the safe, which is enough information to pick up the money from the top of the safe. However, the top shelf of the safe is not visible, and LLARVA can still predict the correct actions to place the stack of money there. This example shows LLARVA can, some cases, work despite visual obstructions, which we believe in part attributable the introduction of the visual traces. We hypothesize that understanding and predicting the visual trace provides the model with information about a successful end-effector trajectory in the presence of such occlusions.
A.3
Efficiency Analysis
Step Vis. Trace Singl Prompt Time Norm.No.of Prompts/Episodes Norm. Total Inference Time Suce. Rate
4
x
0.16
4
0.64
87.5
0.50
4
2.01
97.5
0.29
0.86
0.63
3
1.89
90
12
x
0.53
2
1.06
82.5
0.83
2
1.67
92.5
16
0.63
0.63
81.25
93.75
Table 8: Efficiency analysis results for LLARVA with varying number of predicted steps. (*Norm. denotes Normalized, No. denotes the number of.)
In order explore the inference efficiency with respect to the number of predicted steps, we conduct experiments our real Franka setup with the pick cube task. The results are shown in Table 8. We vary the number of predicted steps with 4, 8, 12, 16, training both with/without visual traces for each value. We then find the total inference time for each case as the product of the time for
17

---

single prompt and the number of prompts needed per episode. We normalize all three values (total inference time, inference time for single prompt, and number of prompts) relative to the n= case, which what we used in the main paper. can be seen that the two factors of single prompt inference time and number of prompts needed should balanced appropriately. For instance, visual traces model has the highest aceuracy, but takes double the time that the n=16 visual traces model does due to higher frequency of prompting. Thus, conclude that the n=16 steps model is both reasonably efficient and accurate.
B Additional Dataset details
Here, we provide more information about our constructed dataset.
utaustin_man muuexx
0.9%
kuka
mask vit
12.3%
0.7%
stanford hydra dataset con
conve 0.8%
fumiture bench. dataset co
9.2%
fraclel20208817 dala
0.8%
8.9%
1000 net
5.8%
8.5M IMAEES
language Iable
maniskil dataset converted
10.4%
2-D VISUAL TRACES
10.7%
fusion 0.6%
bridge
2.3%
be z 14.1%
barkeloy converted
18.0%
Figure 6: Data distributions. We do vision-action instruction pre-training for LLARVA on a dataset built upon Open X-Embodiment [10], including 8.5M image-2-D visual trace pairs.
Data Distribution. As mentioned in the paper, we construct the vision-action tuning dataset from a subset of Open X-Embodiment (OXE) [10]. We excluded OXE subsets with poor image quality, smaller image resolution, ambiguous action spaces, or those with widely different robot morpholo- gies, such as Autonomous Mobile Robots (AMRs, which involve locomotion), resulting in 8.5M image-text pairs, whose distribution is shown in Figure 6 and Table 9. Overall, ensured that the resulting dataset contains subsets of [10] that use end-effector control and joint control, in addition to including both absolute and delta control modes.
The Generation of 2-D Visual Traces. The 2-D visual traces can be seen as a trace of the end- effector location in the image plane across time. To generate these traces, we trained an object detector to locate the end-effictor from input 2-D images. A bounding box detector was used as opposed to single point detector because there exist very stable implementations of such models with pre-trained weights publicly available. Specifically, we use the Detectron2 [12] implementation of Faster R-CNN [52] to obtain bounding boxes enclosing the end-effector, and then use the center point of the bounding boxes as the end-effector keypoint. For training, we first loaded the ResNet- 101 weights pre-trained on ImageNet. We then randomly collected 200 images from each OXE subset, split the collected 6400 images into a training and validation set with a ratio of 9:1, and trained the detector for 40 epochs using batch size of 16 and learning rate of 2-3 Some examples of the resulting detector training set are shown in Figure 7, where the 2-D visual traces are shown in yellow. Note that the traces are sequence of 2-D coordinates, and Figure is visualization of
18

---

OXE Subset
Number of Image 2-D visual trace pairs
kuka
1044466
austin sailor cesttaset.covvveteet externaly ro.rdds
70758
fractal20220817_dhnt
753647
909568
cmu.play_fusionn
47115
bc.z
1198963
oerredyyyy....ovrrrroememmmmmeer bridge
1533451
195745
language.table
885876
stanford Stanford.kuka.multtnni
30128
robo_net
496454
65527
furniture
786692
72160
13545
6512
57012
2276
berkeley-fanuc_mee www.aaanwmmmaauatiiiccc
11854
utaustin mutex
72461
taco_play
47780
berkeley_autooe auttllab.ur ur5
19621
custin..sirius dataset converted externally to.r
56101
columbia cairlab pusht real
5486
22894
roboturk
37120
berkeley cable routing
7797
nyu franka play dataset converted externally to
9118
jaco.play
15515
viola
15146
tokyo Ismo converted externally to rlds
2398
audsin httplaaeeee.oneeeeeeeeeetteeeeee -oorrds dlr sara wawrrappuu.coooooo convertedd externally ro.rlds
6771
2695
utokyo xarm pick and place converted exterally rlds
1381
utokyo_pin pr2 tabletop manipulation converted externally to rlds
6545
dlr wwr.edan_ssharee caaprred__oottoo__oooo www.eeooooeeeeeeeeeeeeee exterrnallyyyooolllee dir sara grid clamp converted externaly to rids
1543
Table 9: The vision-action instruction tuning dataset.
19

---

2D Visual Trace
Subset
Visual Obscrvations
Robot
Control Mode
fractal20220817
Franka
End effector control
data
berkeley_ _pp nverted external ly ___o_rlds
Franka
Joint control
toto
Franka
End effector control
berkeley autola u
URS
End effector control
Figure 7: few samples from our constructed vision-action tuning dataset. We visualize some samples of the instruction tuning dataset used the pre-training stage of LLARVA, with the corre- sponding robot type and control mode.
these sequences. During training the sequences predicted in language token space and compared to ground truth.
We further note that the end-effector detector fairly robust as it has been trained on images that cover a wide range of lighting conditions, cameras, and environments. In addition, it can easily be fine-tuned with very small number of sample images (200) for new setup, making this approach adaptable.
Metric
AP
ABso
ARio
Value
75.2
96.3
87.2
Table 10: Evaluation results for the end-effector detector.
Additional Implementation Details
C.1 RLBench Experiments
LLARVA is evaluated on 18 tasks from RLBench. All RLBench tasks include two or more vari- ations of language instruction describing the goal. For example, there might be three variations of the instruction for the same task: "open the top drawer". "grip the top handle and pull the top drawer open" and "slide the top drawer open". For simplicity, for training. Below, we describe the RLBench tasks we use for simulator evaluation, along with any modifications we made to the tasks. The intention behind the modifications is to increase the variations the tasks, such as adding distractor objects with different colors. This exercises the model's s language grounding abilities. All tasks are unmodified unless otherwise noted.
use the first instruction variant
Training Setup. We start with LLARVA model that has undergone vision-actio action instruction pre- training on OXE as described in Section 2.3, and perform step 2 (Section 2.3) instruction fine-tuning for four epochs on task-specific downstream data (c.g., picking, stacking, destacking) using eight A6000 GPUs. Step instruction tuning is done using 800 demonstrations for each RLBench task. The domain gap between step and step is large as we change from almost entirely real data to simulation while at the same time changing robots and tasks. We note that while other works train on smaller amount of data, they use roughly the same order of magnitude of data LLARVA, and exploit the power of 3-D representations. For example, PerAct [7] uses 100 examples per task but exploits boxel-based 3-D representations, which are and difficult to obtain. Our approach has
20

---

the advantage of being able to leverage 2-D representations, which may require additional data but with roughly the same order of magnitude as methods that utilize 3-D.
Open Drawer. The task to open one of three drawers. The success metric is full extension of the prismatic joint of the target drawer.
Meat off Grill. The task is to take either a piece of chicken or steak off the grill and put it on the side. The success metric is the placement of the specified meat on the side, away from the grill.
Turn Tap. The task is turn either the left or right handle of the tap. Left and right are defined according the orientation of the faucet. The success metric is the joint of the specified handle being at least 900 away from the starting position.
Put Money. The task is to pick up the stack of money and place it on the specified shelf of a safe. The safe has three shelves: top, middle, and bottom. The success metric is the placement of the stack of money on the specified shelf in the safe.
The task is to push the colored buttons in the specified sequence. There are always three buttons present in the scene, whose colors are sampled from 20 options, and the number of buttons to press is between one and three. The success metric is all specified buttons being pressed in the right order.
Push Buttons.
Sweep Dustpan. The task to sweep the dirt particles into the specified dustpan. There are two dustpans, one short and one tall, and both are always present in the scene. The success metric is all five dirt particles being inside the specified dustpan. We modified this task by adding variation with a different-sizer dustpan.
Slide Block. In this task there is a block and four colored squares in the scene (green, blue, pink, and yellow). The task is slide the block onto either the green or pink squares. The success metric used is some part of the block being on the specified target square. The original task only had one target square, and we modified it by adding three additional colored squares distractors.
target and two
Close Jar. The task is to screw in the lid on the jar with the specified color. There are always two colored jars in the scene, one target jar and one distractor jar. The success metric used is the lid being on top of the specified jar and the robot gripper not grasping any object. We modified this task so that the target jar color is drawn from list of two possible colors (blue or teal). The color for the distractor jar was still chosen out of 20 options.
Screw Bulb. There are two bulb holders of different colors, and the task is to pick up a light bulb from the stand specified by color and screw it into the bulb stand. The color of the target holder is sampled from two colors, while the color of the distractor holder is sampled from the original 20 color options. The success metric used is the bulb from the specified holder being inside the bulb stand. We modified this task to use two colors for the target holder (yellow and purple) rather than 20 as in the original task specification.
Place Wine. The task to pick up the wine bottle and place it at the specified location in a wooden rack. The rack has three locations: left, middle, and right. The success metric the placement of the bottle on the specified location in the rack.
Reach and Drag. The environment has a cube, stick, and four possible colored target squares. The task is to pick up the stick and use it to drag the cube to the target square of a specified color. The other three squares are considered distractors. The success metric used is some part of the block being inside the target's area. We modified this task to sample the target color from a list of three colors (maroon, magenta, teal). The colors for distractor squares are still sampled from 20 options.
Stack Blocks. The scene starts with 8 blocks and green platform. Four of the blocks are of target color, and the other four have a distractor color. The task is to stack N blocks of the target color on the green platform. The success metric is N blocks being inside the area of the green platform.
21

---

Put Item in Drawer. There is block kept on top of chest of closed drawers. The task is to place the block into the specified drawer among three possible options: top, middle, or bottom. The success metric is the placement of the block inside the specified drawer.
Sort Shape. The scene has four distractor shapes and one correct shape. The task is to pick up the shape specified in the language instruction and place it in the correct hole in the sorter. The success metric is the correct shape being inside the corresponding hole.
Insert Onto Square Peg. The scene has a platform with three differently colored pegs, and one square shaped object with hole the middle. The three colors are sampled from 20 color instances. The task is pick up the square and put it on the peg specified in the language instruction, with the success metric being the placement of the square fully the peg-
Stack Cups. The scene has three cups with colors sampled from 20 options. The task is to stack all cups inside the cup specified in the language instruction. The success metric for this task is all other cups being inside the specified cup.
Put Groceries in Cupboard. The scene always has nine grocery items and cupboard. The task is to place the item specified in the language instruction inside the cupboard. The success metric used is the placement of the item inside the cupboard.
Place Cups. The scene always has one cup holder with three spokes and three cups with handles. The task is place N of the cups on the cup holder (N E 21)2. The success metric used the alignment of each cup's handle with spoke on the cup task.
Toilet Seat Down. The scene consists of a toilet which initially has its seat up. The task is to put the toilet seat down. The success metric used is the joint of the toilet seat being at an angle consistent with the seat being fully down.
Close Laptop Lid. The scene consists of laptop which initially has its lid open. The task is to close the laptop. The success metric used is the joint of the laptop lid being at an angle such that the screen is fully down.
Put Knife on Chopping Board. The scene consists a knife inside a knife holder, and a chopping board. The task to pick up the knife from the holder, and place on the chopping board. The success metric used is the knife being on the surface of the chopping board, and the robot gripper not grasping anything.
Put Umbrella in Umbrella Stand. The scene consists of an umbrella and an umbrella holder. The task is to pick up the umbrella and put into the stand. The success metric used is the umbrella being inside the stand, and the robot gripper not grasping anything.
Move Hanger. The scene consists of clothes hanger and two racks. The task is move the hanger from its current rack to the other rack. The success metric used is the hanger being placed on the other rack.
C.2 Real Robots Experiments
Hardware Setup. We use a Franka Emika Panda robot with Franka gripper for real robot data collection and evaluations. A Logitech BRIO 4K camera positioned to the right of the Franka robot provides single-view RGB (without depth data) vision input to model, as shown in Figure 8. Camera autofocus is disabled, and the data is captured at 640x480 resolution. The model inference is done on 48GB NVIDIA A6000.
Data Collection. We use the data collection code and process from https://gitub..co//M Fu/franka- seriptee collect data for picking, stacking, and destacking tasks. The script generates data for an arbitrary number of episodes. For each episode, the process generates x-y positions on the table plane using uniform random distribution for each axis. The script directs the robot to place the cube at each location and then collects the camera and joint information as the robot is
22

---

Right Camera used for Visual Input
Figure 8: The real robot setup with Franka Emika Panda used for evaluating LLARVA.
directed pick, stack, or destack the cubes. Vision not used during this process the cube locations are all generated and therefore known.
Training and Execution. For the Franka Emika Panda robot experiments, we start with our LLARVA model that has undergone vision-action instruction pre-training on OXE as described Section 2.3, and perform step (Section 2.3) instruction fine-tuning for four epochs on 1920 episodes of task-specific downstream data (e.g.. picking. stacking, destacking) using A.00 GPUs. This is similar other baselines, such as RPT [16], that uses equal amount of in-domain episodes (1920) for pre-training, with additional 120-240 episodes used for fine-tuning depending on the task. Additionally, [16] uses three camera views for each episode, while LLARVA uses only one. Nevertheless, it can be observed that LLARVA demonstrates superior performance on all three tasks tested despite using comparable or even fewer episodes. Finally, each real robot evaluation consists of 16 repeated pick, stack, or destack operations at random x-y location on the table plane for each repetition. We report the success rate of the 16 operations.
D Discussion, Limitations and Future Work
While LLARVA offers substantial benefits for enhancing robot learning across various environ- ments, it is important recognize certain limitations that accompany our approach. First, even though LLARVA already shows generalization capabilities using instruction language format, we think future instructions should include 3-D information about the real world. Secondly, it is necessary for LMMs to be able to process multiple views leveraging depth or voxels, which may re- quire the use of interleaved tokens as is done in a few existing LMMs. We therefore believe that the next robotic LMMs should utilize this technology, and we leave this for future work. This presents a great opportunity for future work on next-generation instruction-tun -unee LMMs for robotics.
E Licenses and Privacy
The license, PII, and consent details of each dataset are in the respective papers. In addition, we wish emphasize that the datasets we do not contain any harmful offensive content, as many other papers in the field also them. Thus, do not anticipate specific negative impact, but, as with any machine learning method, we recommend exercising caution.
23


Reviews:
[{"review_id": "rHFXAjMNPk", "reviewer": "Reviewer_s7WX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper transfers the robot action prediction task into natural language generation task through instruction tuning. The paper generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset as pretraining dataset.  They show that pretraining improves performance over non-pretraining. They propose predicting 2D visual traces before the next step action, which helps improve performance as well. They finetuned a LMM based on LLaVA by LoRA.", "summary_of_recommendation": "In total, this work is promising and provides new angle of solving robot action prediction problem by transfer it into natural language generation problem. The proposed method is technically sound. However, this work necessiates more experiments to validate the model’s ability to generalize in different settings. What’s more, tasks in real scenarios are less and simple.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Experimental setup in section 3.3, the evaluate strategy is not clear, please explain why the numbers of evaluation episodes are different for each method.\n2. It will be beneficial to provide more experiments on inference efficiency(especially the number of predicted steps) and more different robot settings mentioned in weaknesses.\n3. It would be helpful to provide more experiments on the ablation of the 2-D visual trace.\n4. In line 152, it said there is a detector trained specifically on OXE and l am curious about how to get original training data and data quantity.\n5. The OXE dataset includes various types of robotic arms with different degrees of freedom and shapes. I would like to ask if the robotic arms selected from the dataset all have the same degrees of freedom？", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I53qajScgp", "reviewer": "Reviewer_Uugi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this study, the authors present LLARVA, an open-source instruction-tuned LMM to predict robotic actions for different tasks, environments, controls, and robots using given visual and language representations. They pre-trained an LMM using 8.5M data and evaluated it on 12 RLBench tasks and real-world, showing it out-performs similar benchmarks.", "summary_of_recommendation": "I am convinced this is a really powerful framework with clear novelties and improvements, but I still think the current evaluations and, most importantly, interpretations of the results are not sufficient to demonstrate it.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Why do the ablation study accuracies for the visual traces differ this much according to the task? Is it related to task difficulty, robot, controller type, etc.?\n\nWhat happens when you use a new robot, task, controller, or environment? Please answer in the light of pre-training and fine-tuning, with examples from experiments if possible.\n\nWould using depth or RGBD images eliminate the problem of 2D to 3D robot action mapping? Does the dataset have these types of data?\n\nSince it is shown as an upper bound in the experiments, what are the advantages of using this method over other accurate methods shown in the paper using 3D representations? Please further explain by comparing possible aspects.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNtI5FLUHk", "reviewer": "Reviewer_WWDV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "LLARVA (Large LAnguage model for Robotic Vision and Action) is a Large Multimodal Model (LMM) instruction-tuned on structured language prompts unifying diverse robotic configurations. LLARVA also uses 2D visual representation visualizing the trace of the robot's end effector. to align vision and action spaces effectively. Pre-trained on 8.5 million image-visual trace pairs from the Open X-Embodiment dataset, LLARVA is evaluated on 12 tasks in the RLBench simulator and a physical Franka Emika Panda robot in the real-world. The results indicate that LLARVA outperforms several baselines, demonstrating strong performance and generalization across robots.", "summary_of_recommendation": "The paper integrates visual trace and structured language prompts to make a robust and generalizable LMM for robots. However, the above comments reduce the clarity and the overall assessment of the work. I would like to see the authors' response on the weaknesses and questions mentioned above.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. L98: It seems the task horizon is fixed for each training. How does the proposed approach become robust and adaptable across a spectrum of task durations?\n2. How is the bounding box end effector detector trained? What is the reason the detector directly outputs a bounding box, instead of a single point? What is the loss function? Were there any concerns using the trained detector, since it is only trained on 200 samples per robot?\n3. Extending the paper's setup into a dynamic camera viewpoint, how can the proposed method be adopted to the new setup?\n4. How were the RLBench data collected and labeled? Based on Section 3.2, I assume that there were both successful and failed episodes. Please share the data generation and success labeling procedure more in detail.\n5. Can the model be easily adapted to incorporate 3-D visual traces or other forms of 3-D data?\n6. Why do Image-BC baselines perform so poorly, scoring 1.3% success rate? What is the performance of the proposed model without visual traces in RLBench?\n7. What is the performance of the pre-trained model on Sawyer on the right table of Table 3? Could the authors clarify if the 'Franka' row is the performance of the pre-trained model, or the model fine-tuned on a small Franka/Sawyer robot dataset?\n8. In the limitations section, why do the authors think that future instructions should necessarily include 3D information and multiple views? This seems to contradict the results, since the results in the paper show that using the proposed visual representation outperforms using 3D representation or multiview observations.\n\n[1]  J. Gu, S. Kirmani, P. Wohlhart, Y. Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrishnan, Z. Xu, et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023.\n\n[2] H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint arXiv:2405.01527, 2024.\n\n[3] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024.\n\n[4] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint arXiv:2308.15975, 2023.\n\n[5] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the  presented approach of modeling the action prediction task as a language generation task. The idea is well-motivated and clearly communicated, and the approach was evaluated on both simulated and real-world experiments. However, the reviewers have identified several shortcomings that should be addressed before publication:

- All three real-world tasks are basic pick-and-place tasks that don’t require complicated trajectories/traces to succeed, leaving doubt about the real-world capability of the approach.
- The paper claims generalization in different settings, but the validation is only on two robots in RLBench.
- Vision-only and language-only ablations would strengthen the paper further.
- The paper should elaborate on the fine-tuning procedure for a new task/environment/platform.
- The experiment/results should be discussed in more detail.
- Additional clarification on why an exocentric fixed camera perspective does not lead to a mismatch between the end-effector trajectory and visual trace would be beneficial.

## Post Rebuttal
The rebuttal addressed all major concerns of the reviewers and was well received. The reviewers unanimously agree that this work should be presented at CoRL 2024. However, some clarifications and details should still be incorporated into the main manuscript as agreed upon by the authors.
Decision: Accept