LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

Authors: dantong  niu
yuvan  sharma
giscard  biamby
jerome  quenum
yutong  bai
baifeng  shi
trevor  darrell
roei  herzig
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-19 06:19:35

Abstract: In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question answering; yet leveraging these models remains an open question for robotics. Prior LMMs for robotics applications have been extensively trained on language and action data, but their ability to generalize in different settings has often been less than desired. To address this, we introduce LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic learning tasks, scenarios, and environments. Additionally, we show that predicting intermediate 2-D representations, which we refer to as *visual traces*, can help further align vision and action spaces for robot learning. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model, and we evaluate on 12 different tasks in the RLBench simulator as well as a physical Franka Emika Panda 7-DoF robot. Our experiments yield strong performance, demonstrating that LLARVA — using 2-D and language representations — performs well compared to several contemporary baselines, and can generalize across various robot environments and configurations.

Main paper: 
 preliminaries: 
 LMMs are designed to handle multiple data modalities simultaneously, such as images and their corresponding text descriptions. Each modality is encoded into a shared embedding space, which is then utilized for reasoning by a language model f parameterized by θ. Specifically, an image is encoded using a pre-trained visual encoder, denoted as v parameterized by ϕ. A corresponding text description is tokenized and encoded using a fixed language encoder e parameterized by γ. Given an input image o and a language task description l, the language model generates a text response R as follows: R = f θ (v ϕ (o), e γ (l)).In this paper, we use an LMM within the context of robotic episodes, which are characterized by temporal sequences of visual observations o 1:N and proprioceptive states s 1:N . Here, N denotes the length of an episode. Notably, in the realm of LMMs for robotics applications, the output R typically encompasses one or more predicted actions for an episode. Next, we describe our LLARVA model.2-D Visual Traces. Visual traces play a key aspect in our vision-action instruction methodology. The choice of 2-D traces is made to match the high availability of image-based large robotics datasets such as OXE, but our method can also be implemented with 3-D data. To achieve alignment between visual inputs and robotic actions, we predict visual traces as an auxiliary task, as we find that this helps to gain better fine-grained localization, resulting in a more accurate prediction of robot actions.We define 2-D Visual Traces as a sequence of coordinates (x, y) in a two-dimensional space, which is aligned with the input image o t at time step t. These coordinates represent the trajectory of the gripper (or end-effector, hand, etc.) throughout the episode. The visual trace at timestep t is:EQUATIONHere, (x i , y i ) denotes the i-th coordinate in the entire visual trace for the episode, and N represents the number of time steps in the episode. We note that language model decoders are crucial in converting multimodal inputs into actionable outputs in robotics. By leveraging the shared visionaction embedding space, our decoder produces responses that the robotic system can use.Input. The input to our LLARVA architecture comprises two components. First, we have the visual observation o t , an image capturing the state of the environment at timestep t. Second, we have the language instruction input l t , which prompts the model to forecast a specified number of subsequent steps, integrating embodied information such as the robot, control mode, and previous proprioceptive states as well as the task directives. Specifically, we formulate an instruction template featuring the robot type R (e.g., Franka, UR5, xArm), control mode M (e.g., joint or end-effector control, absolute or delta control), task instruction I (e.g., "open the drawer"), proprioceptive information S (e.g., positions or velocities), and a query indicating the number of future actions to predict, denoted as n. The complete instruction is formulated as follows: To develop a versatile and adaptive framework capable of accommodating training for tasks with varying time horizons, we add flexibility to the proprioceptive information input. Specifically, this information is structured as S = s t-h:t , representing a sequence of past joint and/or gripper states.lt = "You are a [R] robot using [M] control.Here, h is the number of previous time steps the model is conditioned on, and is decided based on the task. This approach ensures robustness and adaptability across a spectrum of task durations, enabling effective training for both short-term and long-term objectives.Architecture. Our objective is to develop a model capable of predicting robotic actions that exhibit generalization across a diversity of robotic tasks, scenarios, and environments. The model architecture is illustrated in Figure 2 . Our instruction-tuned model π is designed to leverage both the current visual observation o t and the accompanying language instruction l t as input. Subsequently, it predicts the action sequence for the next n steps A t:t+n-1 and the future 2-D Visual Traces of the end-effector P t:N , spanning from the current step to the final step within the episode:EQUATIONwhere l t is constructed as defined above.Projection LayerLanguage Input"The 2-D Visual Trace: . The action:."Tunable weights Frozen weights In our proposed pipeline, the input image undergoes processing by the frozen vision encoder v ϕ (•), which extracts visual features and projects into a latent space via an MLP layer H. This aligns the visual features with the dimensionality of the language tokens. Simultaneously, the language input undergoes tokenization using a language encoder. The visual tokens and word tokens are then concatenated and fed into the auto-regressive transformers of the LMM f θ , which are trained for nexttoken prediction.While keeping the vision encoder and language encoder frozen, we use instruction tuning to train the auto-regressive transformers using standard LoRA adapters [11] for both the pre-training and fine-tuning stages. Each image o t for an episode is accompanied by a language instruction l t , and the predictions consist of robotic actions A t:t+n-1 and visual traces P t:N . Next, given o t and l t , we predict the next actions and 2-D visual trace. Specifically, for a response R, we compute the probability of the target actions and target visual traces by the following equation:EQUATIONwhere θ represents the trainable parameters, x i is the current prediction token, and n ≤ N . To calculate loss, we use the standard cross-entropy function with these probabilities. Next, we describe our two-step training process, the large-scale pre-training and the fine-tuning for a downstream task.Step 1: Vision-Action Instruction Pre-training. We begin with an LMM that has been pretrained on vision-language (VL) tasks. In order to generalize across robotic tasks, scenarios, and environments, the model is pre-trained on our large-scale vision-action instruction dataset. Due to the diversity of this dataset, our model is trained simultaneously for multiple configurations of prompt variables such as robot type R, control mode M or task instruction I1 . Using language as input allows us to bridge fundamental gaps between subsets brought by these different configurations. This extensive and varied training process can establish a powerful LMM framework that can be further fine-tuned and adapted to handle various robotic settings. We note that this pre-training stage is different from standard LMM pre-training. As opposed to aligning the two modalities using a projector in VL, here we align the two modalities for generalizing robotic configurations.Step 2: Fine-tuning for Downstream Tasks. Unlike other fields, a robotic model must be finetuned on a downstream task before it can be evaluated due to the practical considerations of realworld physical properties. Therefore, we fine-tune the pre-trained model using a small dataset with a fixed configuration for the factors defined in Section 2.2 (e.g., the instruction has the same robot type R, control mode M, etc.). Having seen diverse data samples makes it easy for the model to adapt to specific downstream settings resembling what it has already encountered in pre-training. Given new tasks, environments, or robot types, LLARVA can be adapted by fine-tuning on some example demonstrations. In addition, data from new modalities such as 3-D, depth or tactile information can potentially be incorporated and utilized in the fine-tuning by modifying the instruction template.When LLARVA is to be used for a new setup or environment, this fine-tuning must first be carried out using example demonstrations. Table 1 : Success rate (%) on RLBench Multi-Task setting. We fine-tuned (with visual trace prediction) the pre-trained model on 12 tasks and evaluate with 25 episodes per task. Each evaluation episode is scored either 0 for failure or 100 for success. We gray out methods with 3-D information.In order to pre-train LLARVA, we generate 8.5M image-visual trace pairs from the Open X-Embodiment (OXE) dataset [10] . As shown in Figure 6 in Supplementary, our dataset consists of images from a diverse collection of 37 OXE subsets with 13 different robots, including a wide assortment of tasks, environments, cameras (and thus images), and end-effectors, among other factors. For each image in an episode, we calculate the 2-D visual trace of the end-effector P t:N . For this purpose, we use a bounding box detector [12] that is trained specifically on each of the different end-effectors in OXE. The center points of bounding boxes are used for a simpler representation, and the visual trace for step t is then the ordered list of all center points from image t to image N . More details of the dataset are shown in Section B in Supplementary. 
experiments and results: 
 We evaluate LLARVA on 18 tasks in RLBench and compare to both existing 2-D and 3-D models.In addition, we also test on a real 7-DoF Franka Emika Panda robot.LLARVA is implemented using PyTorch [13] with the official LLaVA 1.5 [4] implementation. The base LMM uses a Llama 2 7B-parameter LLM, the default image projection layer, and the CLIP ViT-L/14 vision encoder. We use 8 NVIDIA A6000 GPUs for training, and 1 A6000 GPU for evaluation.Additional information, such as training and fine-tuning recipes, are in Supplementary Section C.Experimental Setup. We evaluate using the same 18 RLBench tasks as in [7] . In the fine-tuning stage, the front view is chosen as o t , conditioned on the previous 5 joint positions (i.e. h = 5).LLARVA predicts the visual trace and the next action step (i.e. n = 1), which is an 8-dimensional vector consisting of 7 joint velocities and a binary gripper state. For evaluation, we take 25 episodes per task in the validation set and score each episode either 0 for failure or 100 for success. We use 5 seeds, which are averaged to get the final success rate.Baselines. We compare LLARVA to several baselines using 2-D and 3-D information. Image-BC (CNN) and Image-BC (ViT) [14] are 2-D language-conditioned models that use CNN and ViT vision encoders, respectively, reported in PerAct [7] . The PerAct [7] model uses voxels as 3-D input to calculate actions, as does C2FARM-BC [15] . In contrast to the 3-D line of work, our input uses only one camera view without any 3-D information.Results. We show results for 12 of the tasks in Table 2 : Success rate (%) of LLARVA on a real robot. We compare LLARVA with RPT and Octo by taking each pre-trained model and fine-tuning them on the same set of demonstrations. LLARVA outperforms the others on all the tasks. performs other 2-D based methods: we achieve an average success rate of 43.3%, while Image-BC (CNN) and Image-BC (ViT) both achieve 1.3%. In addition, LLARVA competes with and even beats 3-D based methods, like C2FARM-BC, which averages 22.7%, and PerAct, which achieves 55.3%. We note that 2-D methods here also include those that use multiple input images from different camera views, while LLARVA uses only one camera view and displays stronger performance. Moreover, we observe that despite some cases of occlusion that arise from using just one camera view, LLARVA can still succeed in such situations, showing the adaptability of our model.Experimental Setup. Following the setting in [16] , we use a 7-DoF Franka Emika Panda robot with the default 1-DoF parallel jaw gripper. The input image o t comes from the right side RGB camera. We evaluate LLARVA on three pick tasks: "pick cube", "destack cube", and "stack cubes", and another task "push buttons". During fine-tuning, we use the same number of episodes (1920) as [16] for pick tasks. The "push buttons" task is a more complicated real world task than the pick tasks as it requires the model to push up to three buttons from four color variations (red, blue, green, yellow) in a specified sequence. The setup is shown in Figure 4 in Supplementary. In our experiment, we use our pre-trained weights on OXE to fine-tune the model with a total of 450 episodes (150 episodes each for the one, two, and three button tasks).During training, we condition on joint positions of the previous 16 steps, and predict 7-dimensional delta joint positions and 1-dimensional gripper status for the following 16 steps. To calculate the final success rate, we evaluate LLARVA over 16 episodes and take the average across 5 runs. Right: Evaluation across different robots. We show the success rate (%) of four randomly chosen tasks in RLBench. Using the same training recipes, we fine-tune our pre-trained model with visual trace but different robots. For evaluation, we take 25 episodes per task, and each of them is scored as 0 for failure or 100 for success.Baselines. We compare LLARVA with RPT [16] and Octo [8] , both of which claim the benefit that pre-training brings to the downstream tasks. RPT uses a BERT-like [17] formulation and must additionally pre-train on in-domain data, while Octo is pre-trained on the mixed dataset OXE and maps the various configurations to the same action space. In contrast, LLARVA is pre-trained in a more diverse manner, without imposing a single action space, via a unified language template.Results. We show the results in Table 2 . LLARVA achieves the highest success rate across all pick tasks. Additionally, compared to RPT, LLARVA only needs one unified pre-trained model that is first instruction-tuned on our dataset and then fine-tuned on each downstream task. In contrast, RPT needs to be separately pre-trained on each individual task before being adapted to them, which brings additional time and computation cost. In addition, Octo has difficulties adapting to a downstream task using a different control mode as the model has only seen end-effector control in the pre-training stage. LLARVA shows better quality both in terms of efficiency for adapting to downstream tasks and compatibility/generalization with different control modes.Robot Planning with 2-D Visual Traces. To align the vision and action spaces, we predict the endeffector's visual traces, which forces the model to develop a more comprehensive understanding.In Figure 3 , we show through visual traces that our model can plan alternative yet correct paths. For example, in the top left image, the gripper takes a different path (go left) than the ground truth (go right), yet it still succeeds in destacking the cube. Figure 3 also shows qualitatively that the visual trace can help with long-horizon tasks by acting as a memory buffer that compensates for the limited number of previous robotic states the model can handle. For instance, in the bottom right image in Figure 3 , the task is "push maroon button, then push green button." It can be seen that the visual trace helps the model reach the "green button" after finishing the first subtask.The Effect of Instruction Pre-training. We ablate the instruction pre-training in Section 2.3 to evaluate its effect. As shown in Table 3 , our model can achieve an average improvement of 17.5% across the four tasks regardless of visual traces. We believe this improvement is due to our diverse, large-scale pre-training, which leverages language instructions to create a strong robotics LMM backbone that can successfully adapt to downstream robotic settings.The Effect of 2-D Visual Traces. To test the importance of visual traces, we fine-tuned the model with and without the visual trace prediction task. Table 3 shows that the traces provide a 15% average improvement across the four tasks. Complete visual trace ablation results with additional analysis can be found in Table 5 and appendix A.1. In particular, an improvement can be seen in tasks with frequent occlusion of objects such as "meat off grill" and "slide block." We note that using both pre-training and 2-D visual traces can help the model complete the task even when the target object is not visible. The "put money on top shelf" example in Figure 3 demonstrates this: the shelf is not visible, and the gripper becomes occluded once it moves behind the counter. While the results show that visual traces increase performance across all tasks tested, we find that the greatest benefit of visual traces occurs in "meat off grill" and "slide block", both tasks with distractor objects.Cross-Robot Generalization. To evaluate LLARVA's generalization across different robots, we randomly selected four tasks in RLBench, and finetuned and evaluated on Sawyer instead of Franka Emika Panda. As shown in Table 3 , LLARVA achieves similar results across both robots, which further proves the generalization brought by our large-scale vision-action instruction pre-training. 
related work: 
 Instruction Tuning. LLMs [18, 19, 20] are typically pre-trained on a large corpus of text with an unsupervised training objective of next-word prediction. Instruction tuning (IT) helps to bridge the gap between the language model's fundamental pre-training objective of next-word prediction and the user's goal of having the model perform specific tasks using input-output pairs whose inputs include text phrased as instructions. Flamingo [19] , GPT-4 [21] , and BLIP [22, 23] were pioneering early LMMs, and LLAVA [24] prompted GPT-4 with image-caption pairs to generate multimodal instruction tuning data. Many recent VL models [1, 25, 26, 27, 28, 29] have followed this approach, as IT has been shown to improve generalization in zero-shot and few-shot tasks [2, 30, 31] . Unlike these works, here we construct instructions from a custom template that we fill in with relevant information about the robotic episode.Language-conditioned Robot Agents. Recent language-conditioned models for robotics have tried different approaches, such as generative video pre-training (GR-1 [32] ), tokenized voxels for incorporating 3-D information (PerAct [7] ), co-training on internet-scale VL data (RT-2 [9] ), and largescale diverse pre-training with a transformer architecture (Octo [8] ). However, each approach has distinct drawbacks that we attempt to address in our model. For instance, GR-1 requires architectural changes to account for different action spaces when fine-tuning, while LLARVA is more flexible and does not need such changes. Moreover, PerAct relies on 3-D information unavailable on a large scale across environments, which is why we chose to use 2-D images when implementing our model. RT-2 uses co-training on a separate web-based dataset, while we more efficiently use instruction-tuning on a smaller, diverse vision-action instruction dataset to create a strong backbone. Lastly, Octo uses a specialized head to predict actions, while the language head in our model is trained to predict both robotic actions and visual traces in "natural" language.Trajectory Prediction using Object Representations. Trajectory modeling has been a fundamental aspect of many machine vision applications. Many previous works have investigated the use of object representations for trajectory modeling, ranging from classical image understanding tasks [33, 34, 35] to an object-centric approach for video understanding [36, 37, 38, 39, 40] with object tracking and interactions to even scene graphs [41, 42, 43, 44] . More recently, there has been an increased interest in trajectory modeling effectiveness for vision and language, such as the referring expression localization task [45, 46, 47] and semantic segmentation using text prompts [48, 49] .In particular, the localized narrative dataset [50] enabled a new task that aims to align a long and detailed image caption to a human trajectory. Our work takes a different approach as we use LMMs to exploit the concept of a 2-D visual trace for implicit end-effector object representation. 
conclusion: 
 Our proposed model, LLARVA, represents a significant advancement in the application of instruction-tuned LMMs for robotics. By leveraging structured prompts to unify a range of robotic configurations and introducing the concept of visual traces, we have demonstrated a generalization method that better aligns vision and action modalities. Our extensive training on 8.5M image-visual trace pairs derived from the Open X-Embodiment dataset and evaluation in both simulated and realworld settings highlight the model's superior performance and generalization capabilities compared to existing approaches. This work marks a meaningful step forward in the integration of LMMs with robotics, promising enhanced adaptability and efficiency in various robotic applications. 
: 
 We introduce a novel instruction tuning method that leverages structured prompts to unify a range of robotic learning tasks, scenarios, and environments and 2-D visual traces to further align vision and action spaces. The model works via a language instruction that contains robot model, control mode, robot task, proprioceptive information, and number of predicted steps, and outputs text with the next robot action(s) and the visual trace for the remainder of the episode.the natural language prompt. This architecture allows us to leverage structured language prompts as a "lingua franca" for robotic perception and control (See Figure 1 ). However, aligning the vision and action modalities to produce meaningful robotic outputs is still not a trivial task. Though recent robotic models have used 3-D representations such as voxels and point clouds to overcome this, these representations are difficult to incorporate into most existing open-source LMMs because they typically accept a single image plus language as input. For these reasons, we utilize 2-D images, which are easy to scale and integrate with existing LMMs.We find that predicting an intermediate 2-D representation, which we refer to as visual traces, can help align the vision and action spaces across different robot and task configurations. In particular, we generate the 2-D visual trace (projection) of an end-effector and force the model to predict this trace alongside the next robot action(s). This waypoint prediction helps align each robotic action to the end-effector's location, allowing the model to focus on fine-grained localization and resulting in a more accurate prediction of robot actions. To achieve this, we construct instructions with such visual traces using the Open X-Embodiment dataset (OXE) [10] , carefully categorizing the action space, the robot type, and the control type.Through empirical study, we show that our vision-action instruction tuning approach using structured prompts leads to generalization across various robot environments and configurations. Additionally, we show that predicting visual traces can help further align vision and action spaces. We evaluate LLARVA on 18 different tasks in RLBench's simulated environment and on picking, stacking, and destacking tasks with a real 7-DoF Franka Emika Panda robot. Finally, we evaluate our model's generalization across two robots in RLBench on four tasks. We show that LLARVA-using 2-D and language representations-performs well compared to several contemporary baselines.2 Vision-Action Instruction Tuning 
 
 Appendix: Here, we provide additional information about our experiments, our model's emergent properties, our constructed dataset, and implementation details. Specifically, Section A provides additional experiment results, Section B provides details about our constructed vision-action instruction dataset, and Section C provides additional implementation details.A We fine-tuned (with visual trace prediction) the pre-trained model on 18 tasks and evaluate with 25 episodes per task. Each evaluation episode is scored either 0 for failure or 100 for success. We gray out methods with 3-D information.Complete Simulation Results on RLBench. Following [7, 15] , we evaluate LLARVA on 18 tasks in RLBench, with the comprehensive results presented in Table 4 . LLARVA demonstrates significant improvements over 2-D methods and shows comparable performance to 3-D methods in most tasks. However, for "long horizon" tasks, which are more complex and involve multiple sub-steps or extended durations, LLARVA exhibits similar limitations to other methods. Specifically, for the "place cups" task, where success is defined by placing a specified number of cups on a rack, our experiments reveal that the model often successfully places the first cup but then becomes confused and wanders randomly. As discussed in the main paper, while the introduction of 2-D visual traces provides the model with a rudimentary sense of memory (as seen in the "push buttons" case), the length of this memory remains limited along the temporal axis. This issue can be partially addressed by incorporating additional information from previous steps; however, it will place greater demands on the maximum context length that vision-language models (VLMs) can handle. Fortunately, several recent works, such as [51] , have demonstrated promising solutions and performance for VLMs with long context length.Real Tasks: Push Buttons. We evaluate LLARVA on "push buttons", a more complicated real world task than the pick and place tasks involving cubes. This task requires the model to push up to three buttons from four color variations (red, blue, green, yellow) in a specified sequence. The setup is shown in Figure 4 . The variation in the number, order and color of buttons makes the task as well as the required motion of the end-effector more complicated. In our experiment, we use our pre-trained weights on OXE to fine-tune the model with a total of 450 episodes (150 episodes each for the one, two, and three button tasks). In our evaluation, we have three tasks: push one button given a specific color, push two buttons given specific colors and order, and push three buttons given specific colors and order. Results are presented in Table 2 .Figure 4 : LLARVA completing the task "push red button, then blue button, then green button" on the Franka 7 DoF setup. The predicted 2-D visual trace is also visualized, with purple representing the current timestep and yellow representing the predicted end of episode.Visual Trace Ablations for 12 RLBench Tasks. For a more comprehensive ablation study on the 2-D visual traces, we extended our ablation study described in Section 3.4 to 12 tasks. We present these results in Table 5 . Overall, we find an average improvement of 13.7% with the visual traces, which further demonstrates their utility. We further note the improvement in tasks with distractor objects/variations such as meat off grill, open drawer, and slide block, and long horizon tasks such as push buttons. Table 6 : Results on vision and language ablation studies conducted on our real world Franka setup.Additional Explorations for Behavior-specific Tasks. In order to provide a more comprehensive evaluation of LLARVA, we explored additional tasks in RLBench with specific behavior patterns. The results for 5 additional tasks are presented in Table 7 , using the same fine-tuning and evaluation settings as in the main paper. These 5 tasks are categorized into two types: "Bending Tasks" and "Placement Tasks". In "Bending Tasks," the robot arm is required to grab the target object and move it down to a certain height. LLARVA demonstrates excellent performance on these tasks.In "Placement Tasks," the robot arm must grab the target object and move it to a pre-specified area. LLARVA performs well overall, except in cases requiring delicate operations during either the "grab" or "place" stages. For example, in the "put knife" task, most failures occur because the gripper misses the thin and delicate handle of the knife. Conversely, in the "put umbrella" task, most failures occur during the "place" stage, as the umbrella stand has a very small hole requiring precise positioning of the gripper during insertion. These issues are primarily due to the lack of detailed information from the visual observation, given that LLARVA uses only a single view image with a 128x128 resolution. Our future work will try to enable our VLM to process multiple view images or to adapt it with a more informative vision encoder that can better capture task-related features. Table 7 : Evaluation results on more tasks in RLBench. We explore additional tasks in RLBench, which can be further categorized into "Bending Task" (i.e. the robot arm is supposed to grab the target object then bend and move the target down) and "Placement Task" (i.e. the robot arm is supposed to grab the target object, hold and move it to a specified area).Multiple Attempts after Failure. The top row of Figure 5 shows an example of the model failing to pick up an object, and then retrying as soon as the end-effector comes back into view without the object in its grasp. Specifically, the third image represents the moment where the end-effector becomes visible again, and LLARVA then attempts to complete the task again. This behavior emerges from the fact that the instruction prompt fed into LLARVA at this moment is similar to the prompt at the start of the first attempt, with the main difference being the previous actions/positions included in the prompt. We highlight this as an interesting emergent property since the training data does not include any examples with such behavior.Handling Obstructed Views. In both pre-training and fine-tuning stages of LLARVA, we use only a single camera view to provide visual inputs. Using only 2-D inputs creates a challenge since robotics tasks require very accurate action predictions in three dimensions.The model should be able to see the exact location of the target and also have a sense of depth, which other works typically achieve by using either multiple camera views or 3-D representations. We note that our model uses a single camera view due to input limitations of current open-source LMMs. These limitations can certainly be overcome and are left for future work.Using a single camera view presents further challenges when objects in the scene occlude each other. However, we find that LLARVA can often complete tasks even in these occluded situations. For example, as shown in the second row of Figure 5 , the task is "put the money away in the safe on the top shelf". The camera view only shows the top and back side of the safe, which is enough information to pick up the money from the top of the safe. However, the top shelf of the safe is not visible, and LLARVA can still predict the correct actions to place the stack of money there. This example shows LLARVA can, in some cases, work despite visual obstructions, which we believe is in part attributable to the introduction of the visual traces. We hypothesize that understanding and predicting the visual trace provides the model with information about a successful end-effector trajectory in the presence of such occlusions.Step In order to explore the inference efficiency with respect to the number of predicted steps, we conduct experiments on our real Franka setup with the pick cube task. The results are shown in Table 8 . We vary the number of predicted steps with n = 4, 8, 12, 16, training both with/without visual traces for each value. We then find the total inference time for each case as the product of the time for a single prompt and the number of prompts needed per episode. We normalize all three values (total inference time, inference time for single prompt, and number of prompts) relative to the n=16 case, which is what we used in the main paper. It can be seen that the two factors of single prompt inference time and number of prompts needed should be balanced appropriately. For instance, the n = 4 visual traces model has the highest accuracy, but it takes double the time that the n=16 visual traces model does due to higher frequency of prompting. Thus, we conclude that the n=16 steps model is both reasonably efficient and accurate.Here, we provide more information about our constructed dataset. Data Distribution. As mentioned in the paper, we construct the vision-action tuning dataset from a subset of Open X-Embodiment (OXE) [10] . We excluded OXE subsets with poor image quality, smaller image resolution, ambiguous action spaces, or those with widely different robot morphologies, such as Autonomous Mobile Robots (AMRs, which involve locomotion), resulting in 8.5M image-text pairs, whose distribution is shown in Figure 6 and Table 9 . Overall, we ensured that the resulting dataset contains subsets of [10] that use end-effector control and joint control, in addition to including both absolute and delta control modes.The Generation of 2-D Visual Traces. The 2-D visual traces can be seen as a trace of the endeffector location in the image plane across time. To generate these traces, we trained an object detector to locate the end-effector from input 2-D images. A bounding box detector was used as opposed to a single point detector because there exist very stable implementations of such models with pre-trained weights publicly available. Specifically, we use the Detectron2 [12] implementation of Faster R-CNN [52] to obtain bounding boxes enclosing the end-effector, and then use the center point of the bounding boxes as the end-effector keypoint. For training, we first loaded the ResNet-101 weights pre-trained on ImageNet. We then randomly collected 200 images from each OXE subset, split the collected 6400 images into a training and validation set with a ratio of 9:1, and trained the detector for 40 epochs using a batch size of 16 and learning rate of 2 -3 . Some examples of the resulting detector training set are shown in Figure 7 , where the 2-D visual traces are shown in yellow. Note that the traces are a sequence of 2-D coordinates, and Figure 7 We further note that the end-effector detector is fairly robust as it has been trained on images that cover a wide range of lighting conditions, cameras, and environments. In addition, it can easily be fine-tuned with a very small number of sample images ( 200) for a new setup, making this approach adaptable. LLARVA is evaluated on 18 tasks from RLBench. All RLBench tasks include two or more variations of a language instruction describing the goal. For example, there might be three variations of the instruction for the same task: "open the top drawer", "grip the top handle and pull the top drawer open" and "slide the top drawer open". For simplicity, we use the first instruction variant for training. Below, we describe the RLBench tasks we use for simulator evaluation, along with any modifications we made to the tasks. The intention behind the modifications is to increase the variations of the tasks, such as adding distractor objects with different colors. This exercises the model's language grounding abilities. All tasks are unmodified unless otherwise noted.Training Setup. We start with a LLARVA model that has undergone vision-action instruction pretraining on OXE as described in Section 2.3, and perform step 2 (Section 2.3) instruction fine-tuning for four epochs on task-specific downstream data (e.g., picking, stacking, destacking) using eight A6000 GPUs.Step 2 instruction tuning is done using 800 demonstrations for each RLBench task. The domain gap between step 1 and step 2 is large as we change from almost entirely real data to simulation while at the same time changing robots and tasks. We note that while other works train on a smaller amount of data, they use roughly the same order of magnitude of data as LLARVA, and exploit the power of 3-D representations. For example, PerAct [7] uses 100 examples per task but exploits voxel-based 3-D representations, which are rare and difficult to obtain. Our approach has the advantage of being able to leverage 2-D representations, which may require additional data but with roughly the same order of magnitude as methods that utilize 3-D.Open Drawer. The task is to open one of three drawers. The success metric is a full extension of the prismatic joint of the target drawer.Meat off Grill. The task is to take either a piece of chicken or steak off the grill and put it on the side. The success metric is the placement of the specified meat on the side, away from the grill.Turn Tap. The task is to turn either the left or right handle of the tap. Left and right are defined according to the orientation of the faucet. The success metric is the joint of the specified handle being at least 90 • away from the starting position.Put Money. The task is to pick up the stack of money and place it on the specified shelf of a safe. The safe has three shelves: top, middle, and bottom. The success metric is the placement of the stack of money on the specified shelf in the safe.Push Buttons. The task is to push the colored buttons in the specified sequence. There are always three buttons present in the scene, whose colors are sampled from 20 options, and the number of buttons to press is between one and three. The success metric is all specified buttons being pressed in the right order.Sweep Dustpan. The task is to sweep the dirt particles into the specified dustpan. There are two dustpans, one short and one tall, and both are always present in the scene. The success metric is all five dirt particles being inside the specified dustpan. We modified this task by adding a variation with a different-sized dustpan.Slide Block. In this task there is a block and four colored squares in the scene (green, blue, pink, and yellow). The task is to slide the block onto either the green or pink squares. The success metric used is some part of the block being on the specified target square. The original task only had one target square, and we modified it by adding three additional colored squares -one target and two distractors.Close Jar. The task is to screw in the lid on the jar with the specified color. There are always two colored jars in the scene, one target jar and one distractor jar. The success metric used is the lid being on top of the specified jar and the robot gripper not grasping any object. We modified this task so that the target jar color is drawn from a list of two possible colors (blue or teal). The color for the distractor jar was still chosen out of 20 options.Screw Bulb. There are two bulb holders of different colors, and the task is to pick up a light bulb from the stand specified by color and screw it into the bulb stand. The color of the target holder is sampled from two colors, while the color of the distractor holder is sampled from the original 20 color options. The success metric used is the bulb from the specified holder being inside the bulb stand. We modified this task to use two colors for the target holder (yellow and purple) rather than 20 as in the original task specification.Place Wine. The task is to pick up the wine bottle and place it at the specified location in a wooden rack. The rack has three locations: left, middle, and right. The success metric is the placement of the bottle on the specified location in the rack.Reach and Drag. The environment has a cube, a stick, and four possible colored target squares. The task is to pick up the stick and use it to drag the cube to the target square of a specified color. The other three squares are considered distractors. The success metric used is some part of the block being inside the target's area. We modified this task to sample the target color from a list of three colors (maroon, magenta, teal). The colors for distractor squares are still sampled from 20 options.Stack Blocks . The scene starts with 8 blocks and a green platform. Four of the blocks are of a target color, and the other four have a distractor color. The task is to stack N blocks of the target color on the green platform. The success metric is N blocks being inside the area of the green platform. directed to pick, stack, or destack the cubes. Vision is not used during this process as the cube locations are all generated and therefore known.Training and Execution. For the Franka Emika Panda robot experiments, we start with our LLARVA model that has undergone vision-action instruction pre-training on OXE as described in Section 2.3, and perform step 2 (Section 2.3) instruction fine-tuning for four epochs on 1920 episodes of task-specific downstream data (e.g., picking, stacking, destacking) using 8 A100 GPUs. This is similar to other baselines, such as RPT [16] , that uses an equal amount of in-domain episodes (1920) for pre-training, with an additional 120-240 episodes used for fine-tuning depending on the task. Additionally, [16] uses three camera views for each episode, while LLARVA uses only one. Nevertheless, it can be observed that LLARVA demonstrates superior performance on all three tasks tested despite using comparable or even fewer episodes. Finally, each real robot evaluation consists of 16 repeated pick, stack, or destack operations at a random x-y location on the table plane for each repetition. We report the success rate of the 16 operations.While LLARVA offers substantial benefits for enhancing robot learning across various environments, it is important to recognize certain limitations that accompany our approach. First, even though LLARVA already shows generalization capabilities using our instruction language format, we think future instructions should include 3-D information about the real world. Secondly, it is necessary for LMMs to be able to process multiple views leveraging depth or voxels, which may require the use of interleaved tokens as is done in a few existing LMMs. We therefore believe that the next robotic LMMs should utilize this technology, and we leave this for future work. This presents a great opportunity for future work on next-generation instruction-tuned LMMs for robotics.The license, PII, and consent details of each dataset are in the respective papers. In addition, we wish to emphasize that the datasets we use do not contain any harmful or offensive content, as many other papers in the field also use them. Thus, we do not anticipate a specific negative impact, but, as with any machine learning method, we recommend exercising caution.Details of the configuration for each subset are available in Section B of the Supplementary Material. 


Decision: The reviewers appreciate the  presented approach of modeling the action prediction task as a language generation task. The idea is well-motivated and clearly communicated, and the approach was evaluated on both simulated and real-world experiments. However, the reviewers have identified several shortcomings that should be addressed before publication:

- All three real-world tasks are basic pick-and-place tasks that don’t require complicated trajectories/traces to succeed, leaving doubt about the real-world capability of the approach.
- The paper claims generalization in different settings, but the validation is only on two robots in RLBench.
- Vision-only and language-only ablations would strengthen the paper further.
- The paper should elaborate on the fine-tuning procedure for a new task/environment/platform.
- The experiment/results should be discussed in more detail.
- Additional clarification on why an exocentric fixed camera perspective does not lead to a mismatch between the end-effector trajectory and visual trace would be beneficial.

## Post Rebuttal
The rebuttal addressed all major concerns of the reviewers and was well received. The reviewers unanimously agree that this work should be presented at CoRL 2024. However, some clarifications and details should still be incorporated into the main manuscript as agreed upon by the authors.

Reviews: review: title: title: Reviewer for LLARAVA
paper_summary: paper_summary: The paper transfers the robot action prediction task into natural language generation task through instruction tuning. The paper generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset as pretraining dataset.  They show that pretraining improves performance over non-pretraining. They propose predicting 2D visual traces before the next step action, which helps improve performance as well. They finetuned a LMM based on LLaVA by LoRA.
main_review: main_review: **Strengths:**

1. The paper models the action prediction task as natural language generation task by formating the robot’s input in language form and predicting the actions token by token which propose a new way to employ off-the-shelf LMM into robotics.
2. The paper finetuned a LMM based on LLaVA which using LoRA, and they use ViT as vision backbone while Llama-7B as language decoder. LLaVA is totally open-sourced making it easy to follow.
3. The authors generates 8.5M image-text visual trace pairs from the Open-X Embodiedment dataset to pretrain the LMM which has 13 different types of robots. Then finetuning the model with downstream tasks. They showed that pretraining helps model adapt to specific downstream settings.
4. The authors report the results both in simulated and real scenarios.
5. The authors propose to predict 2D positions of the end effector first instead of directly predicting the action(like joint position) which improves the performance. 

**Weaknesses:**

1. All 3 tasks in real scenarios are pick&place type which are too simple to complete. Pick cube only needs lift up the cube which only consists of the variation of z-axis. And only a change occurred in the y-axis for the visual traces. Stack and destack cube are mostly identical as well as the visual traces except the height of pick and place. It’s hard to tell whether the model can predict more complicated traces in reality.
2. In abstract, the authors said they introduce LLARVA to address the generalization in different settings. But the paper only validate this generalization in RLBench with four tasks across two kinds of robotic arm. Both sawyer and franka are 7 Dof robotic arms which means the difference is tiny. And the model is separately finetuned with the data corresponding to the setting. As a result, it’s not sufficient to demonstrate the model’s generalizability.
3. In section 3.1, the authors claimed that they utilized the same language encoder, ViT and projection layer in LLaVA-1.5. And LLaVA-1.5 has two model sizes: 7B and 13B. Both sizes of model are computational heavy during real-time robot control.  As seen in figure 1, the model need to predict all the remaining visual traces which largely increase the numbers of tokens. Though the authors mentioned that the model can predict next n actions, they didn’t report a comprehensive  comparison of task success rates when increasing n. 
4. In the video, the ‘sweep to dustpan’ part, the variation of predicted visual traces is significant and deviates greatly from the ground truth. And l concern if visual traces is wrong, does this really help improve the accuracy of predicted actions? And the table 3 left only involves 4 tasks which l think not enough to validate the effectiveness of predicting 2-D visual trace.
questions: questions: 1. Experimental setup in section 3.3, the evaluate strategy is not clear, please explain why the numbers of evaluation episodes are different for each method.
2. It will be beneficial to provide more experiments on inference efficiency(especially the number of predicted steps) and more different robot settings mentioned in weaknesses.
3. It would be helpful to provide more experiments on the ablation of the 2-D visual trace.
4. In line 152, it said there is a detector trained specifically on OXE and l am curious about how to get original training data and data quantity.
5. The OXE dataset includes various types of robotic arms with different degrees of freedom and shapes. I would like to ask if the robotic arms selected from the dataset all have the same degrees of freedom？
limitations: limitations: 2
review_summary: review_summary: In total, this work is promising and provides new angle of solving robot action prediction problem by transfer it into natural language generation problem. The proposed method is technically sound. However, this work necessiates more experiments to validate the model’s ability to generalize in different settings. What’s more, tasks in real scenarios are less and simple.
title: title: LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning
paper_summary: paper_summary: In this study, the authors present LLARVA, an open-source instruction-tuned LMM to predict robotic actions for different tasks, environments, controls, and robots using given visual and language representations. They pre-trained an LMM using 8.5M data and evaluated it on 12 RLBench tasks and real-world, showing it out-performs similar benchmarks.
main_review: main_review: In this study, the authors present LLARVA, an open-source instruction-tuned LMM to predict robotic actions for different tasks, environments, controls, and robots using given visual and language representations.

The idea and the motivation are very clear. The paper is well-written and easy to understand. You can find my comments below:

It would be beneficial to see an ablation study for only-vision and only-language modalities, as both are presented as crucial for a detailed understanding of the environment, task, and robotic actions.

Fine-tuning and its limitations are not clear in the paper. The authors should explain what needs to be done for a new task/environment/robot if one wants to use this method off the shelf. I understand that fine-tuning improves accuracy as it introduces the new configuration to the model as training data with ground truths. However, it is only presented as correcting real-world properties. Isn't OXE already a robotic dataset based on physical real-world properties? This should be discussed further.

The authors present their method as capable of recovering from failures. I wonder what would happen when multiple failures occur such that t approaches an average episode length for the given task. For example, let's assume a stacking action usually takes 60 steps; what happens when t becomes 120 and the robot has still not grasped the object due to prolonged failures? Does it prioritize reaching the final position, or can it still prioritize grasping the object? It is also important to state whether h is ever reset to zero after a long time or failure.

It would also be really powerful if there was an experiment showing that the method is accurate for a new robot that was not in the training set. It would also support the fact that fine-tuning is an important aspect of this method since these are new test-time additions and modifications.

My biggest concern is that, despite being convinced this is a really powerful framework, I still think the current evaluations and, most importantly, interpretations of the results are not sufficient to demonstrate it. There is a significant lack of interpretation of the results and why they are good, bad, or important. These should be further explained after stating that the method outperforms the benchmarks. Without the authors' insights, any aspect of the paper, whether it is really this amazing or not, is just a wild guess. For example, the accuracy difference in the ablation study for using visual traces drastically changes depending on the task. Why is that happening? Is it because the visual traces only track the end-effector but not the joint space so that the large improvements are for end-effector control tasks and the small improvements are for joint control tasks? This is a wild guess for a reader that the authors should have insight into.

Minor edits:

Line 181: change "strong" to "stronger."

Line 188: specify exactly how many demonstrations when mentioning "a comparable number of demonstrations."

Equation 3 states A and P are ground truths but uses the ^ notation, confusing the reader.
questions: questions: Why do the ablation study accuracies for the visual traces differ this much according to the task? Is it related to task difficulty, robot, controller type, etc.?

What happens when you use a new robot, task, controller, or environment? Please answer in the light of pre-training and fine-tuning, with examples from experiments if possible.

Would using depth or RGBD images eliminate the problem of 2D to 3D robot action mapping? Does the dataset have these types of data?

Since it is shown as an upper bound in the experiments, what are the advantages of using this method over other accurate methods shown in the paper using 3D representations? Please further explain by comparing possible aspects.
limitations: limitations: 2
review_summary: review_summary: I am convinced this is a really powerful framework with clear novelties and improvements, but I still think the current evaluations and, most importantly, interpretations of the results are not sufficient to demonstrate it.
title: title: The paper introduces LLARVA, a Large Multimodal Model (LMM) model instruction-tuned on structured language prompts unifying diverse robotic configurations and visual representations showing the trace of the robot's end effector.
paper_summary: paper_summary: LLARVA (Large LAnguage model for Robotic Vision and Action) is a Large Multimodal Model (LMM) instruction-tuned on structured language prompts unifying diverse robotic configurations. LLARVA also uses 2D visual representation visualizing the trace of the robot's end effector. to align vision and action spaces effectively. Pre-trained on 8.5 million image-visual trace pairs from the Open X-Embodiment dataset, LLARVA is evaluated on 12 tasks in the RLBench simulator and a physical Franka Emika Panda robot in the real-world. The results indicate that LLARVA outperforms several baselines, demonstrating strong performance and generalization across robots.
main_review: main_review: The paper presents LLARVA, an instruction-tuned Large Multimodal Model (LMM) for robotics. The proposed method of using structured prompts and visual traces is well-conceived and effectively addresses the challenges of diverse robotic configurations and aligning vision and action spaces, respectively.

Strengths:

1. The use of 2D visual traces to align vision and action spaces is simple and effective, and addresses an important problem in existing robotic LMMs.
2. The model is evaluated on a wide range of tasks both in simulated and real-world environments, demonstrating its robustness and versatility.
3. LLARVA outperforms all baselines with 2D visual representations without visual traces on most tasks in RLBench, particularly in generalization across different robot configurations and environments.

Weaknesses:

1. While the use of 2D visual traces in LMMs is underexplored, visualizing the robot's trajectory has been studied in numerous previous work [1, 2, 3, 4, 5] and limits the novelty of the proposed approach.
2. Is this end-effector detector robust on camera/lighting/environment change? Is this method scalable, or is it only applicable in in-distribution data? If not, how could these issues be resolved in the future?
3. Especially in case of fixed camera viewpoint, you can easily know camera intrinsics and extrinsics, and get (perfect in simulation environment) the projection mapping from the joint positions to the exact 2D visual trace. What is the advantage of using a trained detector in fixed camera viewpoint setting, despite the discrepancy between ground truth and predicted visual traces as shown in Figure 3?
4. Why does LLARVA predict both visual trace and next action? It seems the paper assumes an exocentric fixed camera viewpoint (please correct if wrong), there exists an exact projection function from a 3D end-effector trajectory to its corresponding 2D visual trace in the camera observation. Since LLARVA is predicting both visual trace and next action, they might not match since there is no hard constraint while training to ensure that the projection is correct.
5. L257: Object-centric Grounding for Vision Tasks in the related work section seems less relevant to the paper. I recommend having a section for visual motion representations, especially on visualizing the end effector trajectory.
questions: questions: 1. L98: It seems the task horizon is fixed for each training. How does the proposed approach become robust and adaptable across a spectrum of task durations?
2. How is the bounding box end effector detector trained? What is the reason the detector directly outputs a bounding box, instead of a single point? What is the loss function? Were there any concerns using the trained detector, since it is only trained on 200 samples per robot?
3. Extending the paper's setup into a dynamic camera viewpoint, how can the proposed method be adopted to the new setup?
4. How were the RLBench data collected and labeled? Based on Section 3.2, I assume that there were both successful and failed episodes. Please share the data generation and success labeling procedure more in detail.
5. Can the model be easily adapted to incorporate 3-D visual traces or other forms of 3-D data?
6. Why do Image-BC baselines perform so poorly, scoring 1.3% success rate? What is the performance of the proposed model without visual traces in RLBench?
7. What is the performance of the pre-trained model on Sawyer on the right table of Table 3? Could the authors clarify if the 'Franka' row is the performance of the pre-trained model, or the model fine-tuned on a small Franka/Sawyer robot dataset?
8. In the limitations section, why do the authors think that future instructions should necessarily include 3D information and multiple views? This seems to contradict the results, since the results in the paper show that using the proposed visual representation outperforms using 3D representation or multiview observations.

[1]  J. Gu, S. Kirmani, P. Wohlhart, Y. Lu, M. G. Arenas, K. Rao, W. Yu, C. Fu, K. Gopalakrishnan, Z. Xu, et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023.

[2] H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation. arXiv preprint arXiv:2405.01527, 2024.

[3] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024.

[4] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint arXiv:2308.15975, 2023.

[5] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023.
limitations: limitations: 1
review_summary: review_summary: The paper integrates visual trace and structured language prompts to make a robust and generalizable LMM for robots. However, the above comments reduce the clarity and the overall assessment of the work. I would like to see the authors' response on the weaknesses and questions mentioned above.