TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies
Authors: Ruijie Zheng
Yongyuan Liang
Shuaiyi Huang
Jianfeng Gao
Hal Daumé III
Andrey Kolobov
Furong Huang
Jianwei Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Language Model
Robot Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: Visual trace prompting enhances VLA models' spatial-temporal understanding, boosting robotic manipulation performance.

Abstract: Although large vision-language-action (VLA) models pretrained on extensive robot datasets offer promising generalist policies for robotic learning, they still struggle with spatial-temporal dynamics in interactive robotics, making them less effective in handling complex tasks, such as manipulation. In this work, we introduce visual trace prompting, a simple yet effective approach to facilitate VLA models’ spatial-temporal awareness for action prediction by encoding state-action trajectories visually. We develop a new TraceVLA model by finetuning
OpenVLA on our own collected dataset of 150K robot manipulation trajectories using visual trace prompting. Evaluations of TraceVLA across 137 configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and 3.5x on real-robot tasks and exhibiting robust generalization across diverse embodiments and scenarios. To further validate the effectiveness and generality of our method, we present a compact VLA model based on 4B Phi-3-Vision, pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B OpenVLA baseline while significantly improving inference efficiency.

Published as a conference paper at ICLR 2025
TRACEVLA: VISUAL TRACE PROMPTING ENHANCES SPATIAL-TEMPORE AWARENESS FOR GENERALIST ROBOTIC POLICIES
Ruijie Zheng 1+f
Yongyuan Liang'
Shuaiyi Huang'
Jianfeng Gao
Hal Daumé III'
Andrey Kolobov
Furong Huang
Jianwei Yang?
1 University of Maryland, College Park
2 Microsoft Research
ABSTRACT
Although large vision-language-actinn (VLA) models pretrained extensive robot datasets offer promising generalist policies for robotic learning, they still struggle with spatial-temporal dynamics in interactive robotics, making them less effective in handling complex tasks, such as manipulation. In this work, we introduce visual trace prompting, simple yet effective approach facilitate VLA models' spatial- temporal awareness for action prediction by encoding state-action trajectories visually. We develop a new TraceVLA model by finetuning Open VLA on our own collected dataset of 150K robot manipulation trajectories using visual trace prompting. Evaluations of VLA across 137 configurations in SimplerEnv and 4 tasks on physical WidowX robot demonstrate state-of-the-arr performance, outperforming OpenVLA by 10% on SimplerEny and 3.5x on real-robot tasks and exhibiting robust generalization across diverse embodiments and scenarios. To further validate the effectiveness and generality our method, we present compact VLA model based on 4B Phi-3 Vision, pretrained the Open- X- Embodiment and finetuned on our dataset, rivals the 7B OpenVLA baseline while significantly improving inference efficiency.
INTRODUCTION
Robotic manipulation policies, typically trained on specific task demonstrations, often struggle to generalize beyond their training data, particularly when faced with novel objects, environments, instructions, and embodiments. In contrast, foundation models for vision and language- -uchh as CLIP (Radford et al., 2021), LLaVA (Liu et al., 2024b), Phi-3-Vision (Abdin et al., 2024b), and GPT-4V (Achiam et al., 2023)-have demonstrated impressive generalization across diverse vision- language tasks. However, these models are not equipped to handle the challenges unique to robot manipulation, such understanding kinematics, adapting to different embodiment configurations, and executing reliable physical actions. Vision- Vision-Language.. Language-Action models (VLAs) (Brohan et al., 2022; Kim al., 2024) seek address this gap by fine-tuning vision-languse models to generate robot control actions using large-scale robotic datasets (e.g.. Collaboration et al., 2023b), combining the generalization power of foundation models with task-specific robotic expertise. The approach has yielded promising results in developing generalist robot policies capable of adapting to wide range of manipulation tasks.
However, ILA-powered robots often struggle to maintain awareness of their past movements, leading to decisions that are more reactive to current inputs rather than informed by spatial history. We posit that this limitation arises because simply mapping image inputs current states to control actions is insufficient. To address this, we propose explicitly computing multi moiti--point temporal trajectories and overlaying them directly onto the image inputs for VLA models. We hypothesize that this will effectively provide spatial and temporal relations necessary for improving manipulation tasks (Wen al., 2023; Yuan et al. 2024).
"Equal contribution.
Work done during internship Microsoft Research.

---

Published as a conference paper at ICLR 2025
Action Takens
TrAceVLA
Visual Trace Embodiment Data
doOd
FOOO
Separator Token (Leamable)
Image Tokanizer
Text Tokenizer
Prompting with Language Instructions
150k Trajectories
Original Image
Image with visual trace
images: original robot olaservation,
marked
effector moving objects, separated by special
separator action robot move sponge near blue plastic bottle?
Downstream Manipulation
SimplerEnv: Googla Roboot
WidowX
Figure 1: An illustration of our method. The first image shows the original robot's observation, while the second contains the image with overlaid visual traces. separator token is then inserted between the visual tokens these two images, then concatenating with text tokens and feeding into the underlying vision language model backbone to output action tokens.
Our approach introduces an additional multi-point visual input during VLA training that tracks the robot's past movement trajectory (Karaev al., 2023). We refer these multi multi-point trajectories as visual traces, and show that even with only 2D images as inputs (which allows for better scalability and integration with existing VLM models), VLA models enhanced with visual traces exhibit improved spatial-temporal awareness. This visual trace prompting technique enables better adaptation to variations in manipulation tasks and improves overall generalization.
We introduce VLA 7B ---armmeter VLA model fine-tuned from Open VLA using our novel visual trace prompting dataset, which includes 150K robot manipulation trajectories as shown in Figure In additional, we finetuned more compact VLA model, TraceVLA-Phi3 using the 4B-parameter Phi-3-Vision as a backbone on the Open X-Embodiments dataset, which comprises 970K trajectories across diverse robot embodiments, tasks, and scenes. TraceVLA Phi3 VLA-Phi offers improved inference efficiency and reduced computational requirements while maintaining robust performance.
To assess the generalization capabilities of our models, we conducted evaluations across 131 diverse environment settings on Google Robots in the SimplerEnv simulator, which closely mimics real-robot scenarios. Additionally, we design diverse manipulation experiments on a physical WidowX robot to evaluate performance real-world settings. Notably, our models consistently outperform existing VLA models across all embodiments and environments, demonstrating exceptional generalization under environmental variations.
Our key contributions are summarized as follows.
Method. We introduce visual trace prompting, novel technique that significantly enhances VLA models' spatial-temporal reasoning in manipulation tasks.
Dataset & models. We provide visual trace prompting dataset and present state-of-the-art 7B and 4B VLA models fine-tuned using our proposed visual trace prompting, offering an efficient method to boost VLA model performance.
Validation. Our approach is rigorously validated through extensive evaluations in both simulated and real-world robot tasks across diverse embodiments, demonstrating superior generalization capabilities by leveraging spatial-temporal information.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
Visual-Language-Action Models. Vision-language-actoon models (VLAs) extend vision-language models (VLMs) to predict discretized robot actions. Their architecture comprises: (1) visual encoder (7hai et al., 2023; Oquab et al., 2023) that converts input images into patch embeddings, (2) a projector that maps these embeddings to the language model's imput space, and (3) large language model backbone (Touvron et al., 2023). Action discretization, crucial feature of VLAs, involves mapping continuous robot actions to discrete tokens. This process typically divides each action dimension into 256 bins based on data quantiles. These discrete actions are then incorporated into the language model's vocabulary, often replacing the least frequently used tokens. VLA training builds upon the foundation established during VLM training. During VLM training, the model is trained end-to-end with next text token prediction objective on paired or interleaved vision and language data curated from various Internet sources. VLA training then extends this approach, encompassing fine-tuning of the pretrained VLM.
Generalist Control Policies. Robotic policy learning typically relies on task task-specific demonstrations
D {TI, T2,...... ). where each {(or, 8t, ax)) ) /= represents expert-level trajectory. The learning architecture comprises a visual encoder Fø. mapping image observations o, to features Fo,o.) and policy network wa outputting action distributions à nT(-12, Training minimizes the error between predicted à and optimal a actions. To overcome task-specificity limitations, generalist policies are being developed, aiming handle diverse sensors, action spaces, and robotic platforms various scenarios. Vision-Language- Lan--aaaggggg-ttionn (VLA) models, leveraging VLM's visual understanding and multimodal reasoning capabilities, show promise in creating more adaptable generalist policies. These models offer improved generalization across tasks, enhanced semantic understanding of environments, and the ability follow natural language instructions, paving the way for more flexible and intuitive robotic control in broader applications.
3 TRACEVLA
Our ultimate goal is to equip the Vision- Lanoon-Lnggage-tttton (VLA) model with the necessary context to better understand both temporal and spatial dynamics. In this section, we describe the details of our method, TLAc in Section 3.1. Next, we explain the model architecture of Trace VLA in Section 3.2. Finally, we provide the implementation details in Section 3.3.
achieve this objective. First, introduce visual trace prompting
3.1 VISUAL TRACE PROMPTING
Initial State
Final State
Original Image
Visual Trace Prompting
CoTracker
Visual Trace Prompting
User: [Prompting visual inputs] |Language instruction]
Multi-Point Tracking
TraceVLA: [Ax, 40, AGripl
Actions
f
Visual Trace Generation
Closed-loop Robot Control
Figure 2: An illustration of visual trace generation. Given a sequence of historical image observations, we first use Co-tracker to extract dense point trajectories and keep active point trajectories with significant movement. We then overlay active point trajectories the robot's initial observation frame visual trace prompting. We feed both the image overlaid with visual traces the original image into VLA as model input.
straightforward way to have VLMs understand temporal history information is to concatenate historical frames and feed them into the VLA model. However, this approach often distract the
3

---

Published as a conference paper at ICLR 2025
model, the frames are typically highly visually similar and redundant, making it difficult for the model to focus on the control-relevant information.
In this work, we introduce visual trace prompting to address this challenge. Instead of naively concatenating history frames, we employ an off-the-shelf point tracking algorithm to generate traces of key points. These visual traces then visually overlaid the robot's original observations, serving as visual prompts that provide the model with a spatial memory of its historical actions.
At any given timestep and time window budget N, we first extract set of dense point trajectories, P, from a sequence of historical image observations, h, (Ot-N,...., 0t), using a dense point tracking model. Specifically, point trajectory represents the trace of a moving point over time. Thus, a set of dense point trajectories captures the traces of multiple critical moving points. The specific dense point tracking model we employ is Co-Tracker (Karaev et al.. 2023), chosen for its efficiency and simplicity. Co- Tracker partitions the starting frame o, into a K x K grid and tracks each grid cell across N frames to construct point trajectories. Consequently, we generate total of K K dense point trajectories, with each trajectory representing the location of single point from timestep t t- N timestep L.
While Co-Tracker provides a K K grid of dense point trajectories, it does not inherently identify "active points." To address this, we identify active point trajectories in P by analyzing changes in pixel locations, focusing those with significant movement and distinguishing them from static background points. For each point trajectory p E P, we first compute the absolute movement Apr between two adjacent frames at timestep t' and t' as Apr |Pr+1 1= We then identify active point trajectories P by computing the total movement over N timesteps, keeping those whose movement exceeds a threshold k. In other words, P p E P 1 Apr > x). From P.
we randomly sample M active point trajectories, denoted as P. for use in visual prompting.
Finally, we generate the visual trace by overlaying the sampled active point trajectories P onto the robot's original observation frame o,, shown in Figure 2. This overlaid frame serves as a visual prompt, providing the model with spatial information about its historical states and actions.
3.2 MODEL ARCHITECTURE DESIGN
In Figure 1, we outline the design of Trace using visual trace prompting. While the overlaid visual trace provides valuable spatial-temporal information about the robot's historical movements, it may obstruct the robot's 's end-effector or key objects, potentially hindering the model's ability to generate the correct action. To address this, we also include the original image observation in the model input, inserting special separator token between the two images. As shown in Figure 1, adjust the text prompt to inform the VLA model of this additional visual input before requesting the appropriate action output.
Additionally, since visual traces may not be available in all test-time scenarios-such as when the Cr-Tracker model fails under pool lightning conditions-we implement dropout mechanism during training. For each training example, with probability a, we replace the visual trace prompt image with the original image and remove the corresponding hint from the text prompt. This dropout strategy introduces variability into the inputs, encouraging the model to effectively utilize both the original image and the visual trace. As result, at test time, even if the Co- To-rkeceer model unable to extract the visual trace, our model can still function correctly.
3.3 IMPLEMENTATION DETAILS
In this section, we describe the implementation details of the dataset and models used in this work.
For the visual trace generation pipeline in training, we use grid size of K=40, sample M=5 active point trajectories, and employ a time window N=6. To reduce computational overhead, run dense point tracking every N step for the future 2N frames, rather than at every timestep. We divide demo trajectories into overlapping 2N-sized segments (e.g., [0, 2N), IN, 3N), etc.) and run Co- Tracker once per segment. This approach ensures that for each timestep t > N, at least N steps of historical context available, while significantly reducing computational costs.
4

---

Published as a conference paper at ICLR 2025
To create our dataset with visual trace annotations, applied our visual trace generation pipeline to the following training datasets: BridgeData-v2 (Walke et al., 2023), Google RT1 Robot datasets (Bro- han et at.,, 2022), as well to 120 demonstrations collected from 4 manipulation tasks our WidowX- 250 Robot setup for downstream evaluation. In total, we gathered approximately 150,000 robot trajectories annotated with visual trace prompting. forming our fine-tuning dataset for Trace
To obtain real-time visual traces during inference, we reduce the computational overhead of densely querying the Co-tracker at every timestep by tracking M active points and sparsely querying Co- tracker. Specifically, at timestep t=0, we perform dense K x point tracking to identify these active points (see Sec. 3.1 for details). For every timestep t>00, we query Co-tracker only for these active points to generate M corresponding traces, and update the tracked active points from the traces. For further details and pseudocode of our VL. inference pipeline, please refer to Appendix E.
For the VLA models, we started with Open VLA (Kim et al., 2024), 7B VLA model based on the Prismatic Languan model (Karamcheti et al., 2024), trained on the Open X-Embodiment dataset (Collaboration et al., 2023a). Additionally, as its backbone VLM (Abdin et al., 2024a), on the Open X-Embodiment dataset using a batch size of 4096 for 30 epochs with 32 H100 GPUs, following the same recipe Open This lightweight 4B model allows us to test the flexibility of our visual trace prompting across different VLM model architectures. Additionally, the 4B Phi3e-based VLA model will also provide the community with a more compact VLA model for finetuning compared to the larger 7B Prismatic model, while the reduced memory cost allows fine-tuning to be performed smaller GPUs such RTX4090 RTX A5000's. For both TraceVLA and TraceVLA-Phin we finetune the base VLA model for an additional five epochs.
pretrained 4B VLA model with Phi3-Vision
4 EXPERIMENT
To comprehensively evaluate our model's performance, conducted experiments across wide range of environmental setups, including tasks with 137 different configurations simulation and 4 tasks real robots.
Baseline. We benchmark our approach against the following generalist policies, including state-of- the-art open-sourced models:
Open VLA (Kim et al., 2024): A 7B parameter VLA trained on the Open-O-pebonneeee X-Embodiment (Collabora- tion al., 2023a) Dataset, representing large-scale generalist policies.
VLenVLA- A 4.5B parameter VLA pretrained model using Phi-3- as backbone.
Octo-Base (Team al., 2024): A 93M parameter transformer- policy trained 800k trajecto- ries from the Open-X X E-Ebbommment Dataset.
RT1-X (Collaboration al., 2023a): A compact 35M parameter model trained on the same dataset as Deto-Base, exemplifying efficient architectures.
Trace VLA and Trace VLA-PPh. Finetuned from Open VLA and Open with visual trace prompting.
4.1 SIMULATION EVALUATION
SimplerEnv. Our simulation evaluation utilizes SimplerEny, which incorporates two distinct settings: visual matching and variant aggregation. The visual matching setting aims to minimize the visual appearance gap between real environments and raw simulation, significantly enhancing the correlation between policy performance in simulation and real-world scenarios. Complementing this, the variant aggregation setting covers a wide range of environmental variations as shown in Figure 4, including backgrounds from different rooms, lighter and darker lighting conditions, varying numbers distractors, solid color and complex table textures, and different robot camera poses. This comprehensive set of variations allows to assess the robustness and adaptability of our approach in handling diverse manipulation scenarios, particularly evaluating the spatial and temporal awareness brought by visual trace prompting.
Overall Performance. As shown in Table 1, Trace consistently outperforms Open across various tasks and evaluation metrics in the SimplerEnv Google robot tasks. The improvements
5

---

Published as a conference paper at ICLR 2025
Visual Matching Pick Coke Can
Variant Aggregation
Models
Overall Performance
Move Near
Open/Close Drawer
Move Near
Pick Coke Can
Open/Close Drawer
VLA
16.1%
46.7%
22.5%
51.9%
49.7%
22.2%
39.9%
TraceVLA-Phi3
(1 4.3%)
50.4%
52.2%
5.5%)
(1 8.5%)
3.1%)
(1 2.7%)
(1 1.0%)
23.2%
(1 (1.4.1%)
44,09%
OpenVLA
47.1%
15.3%
49.5%
54.0%
52.8%
22.5%
40.25
TraceVLA
(1 6.6%)
(1 12.77%
(1 7.5%)
(1 (124%)
(1 7.2%)
(1 8,5%)
(1 7.5%)
Sxto-Base
3.0%
1.3%
1.0%
4.2%
17.0%
22.0%
8.2%
RTI-X
55.0%
Table 1: Performance results three SimplerEnv Google robot tasks under evaluation metries: visual matching and variant aggregation. Overall performance calculated as average over all the results.
evident both the full-scale 7B models (TraceVLA vs OpenVLA) and their 4B versions (TraceVLA Phi3 vs Open VLA-Phi3, Trace VLA shows significant performance gains, with improvements ranging from 2.4% 12.7% in all three tasks baselines like Octo-Base and RTI-X, both TrAceVLA and Trace VLA-Phi3 generally perform better, with few exceptions where RTI-X. shows competitive performance in specific tasks. These results suggest that the visual trace prompting technique employed in Trace VLA enhances the model's s ability generalize across different robotic manipulation tasks and environmental conditions, leading to improved performance in both visual matching and variant aggregation scenarios.
different metrics. When compared other
OpenVtAA Simpletrv
Phil Smplertny
OpenitA
Drowr
Drower
Oper/Close
Open/C
ta KX
Pick Colic
Pick Coli
Wove hear
Move hear
Succoda Rooe (6)
Success Rato (h5)
Figure 3: (Left): 7B TLA vs. VLen (Right): TraceVLA vs. VLA Numbers are averaged across the visual matching and variant aggregation metrics.
Success Rate (%)
Camera orientations
Lighting darker
Background change
Table texture
Distractor
OpenVLA
Camera
TracsVLA
Lighting
OpenVLA
Background
TraceVLA
OpenVLA
$4.3%
OpenVLLX
600%
TexcwViL
Aouns
Figure 4: Comparison of OpenVLA and TraceVLA performance across various environmental variations: camera orientations, lighting, background, distractors, and table texture.
Environmental Variant Aggregation. Figure demonstrates significant performance improvements of Trace VLA over VLe across various environmental conditions. Notably, Trace VLA shows substantial enhancements under camera orientation changes, distractor presence, and background alterations, with average improvement exceeding 20% in these categories. The of visual trace prompting proves particularly effective when camera angles shift, as the trace provides valuable spatial trajectory information. This helps the model maintain performance despite changes in perspective.

---

Published as a conference paper at ICLR 2025
Similarly, when faced with background changes, varying table textures, or lighting alterations, the visual trace allows the model to remain more stable and less influenced by environmental backgrounds. In scenarios with added distractors, the visual trace effectively reminds the policy of past trajectory interferences, evidenced by noticeable arm diversions. These varied environmental conditions collectively validate the spatial and temporal understanding provided by visual trace prompting.
4.2
REAL ROBOT EXPERIMENTS
3rd Persor
Widco 250
Camera
Robot Arm
Fold Cloth
Swippe Sink
Pickplace Pot
Pick Up Knifa
Grappy
Eggplant Plate
Lift AAA Battery
Right of Plate
Left Right
Figure Real robot setup. We design robot tasks with different manipulation skills and objects including 4 unseen tasks for generalization evaluation.
Real Robet Performance
Additional Real Robet Generalization
Ciet
Piccplace Sanata
Petplace Eppplent
OpenVLL TraceLA
OpenVLA
TracevLA
Picp ace
Busaryy
Pilp oce Com
hos Cluth
Succesefil
(a) Trace outperforms OpenVLA diverse real-robot manipulation tasks.
(b) TraceVLA showcases superior generalization unseen real robot experiments.
Figure 6: Performance comparison of VLA and OpenVLA on8 real-world WidowX-251 robot manipula- tion tasks.
We evaluate VLa physical WidowXX250 robot manipulation tasks using fixed- mounted third-person view camera capturing 256 x 256 RGB images. Despite sharing the same robot embod- iment BridgeDData-v2, differences in setup, lighting, and camera angles necessitated collecting 30 demonstration trajectories per task for finetuning. Detailed real-robot experimental setup and protocols are in Appendix A.
As shown in Figure 6a, TLA consistently outperforms the baseline across diverse tasks including soft object manipulation, pick-and-place operations, and object movement. Notably, in the pick- place corn task, which was not included in training data, TraceVLA achieved 8/10 successful trials compared to VLenVLA's 1/10, demonstrating strong generalization from similar training tasks, e.g.. picking and placing eggplant in pot.
To further evaluate generalization capabilities, we conducted additional experiments with four unseen tasks involving novel objects, goals, and language instructions. These trials included 2-3 random distracting objects in the scene except for pushing cloth. The results in Figure 6b demonstrate TraceVLA's superior generalization over Open In the pick-place banana task. TraceVLA's only failures occurred due to grasping issues, while VLA, even when successfully grasping the banana, failed follow the language instruction by placing it on the plate rather than to its right. This highlights TraceVLA's enhanced language grounding capability and resistance to spurious correlations.
7

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDIES
To analyze the performance gain from visual trace prompting, we further study the following questions.
Is performance gain of TraceVLA coming from further finetuning on a smaller subset of Open X-embodiment dataset? To answer this, we also tested the performance of the 7B Open VLA and 4B Open VL.- models finetuned on the exact same dataset as ours, but without using visual trace prompting. The results, as shown in Figure 7 (Left), indicate that the performance gain observed in VLA is unlikely due to finetuning the pretrained VLA model on a smaller subset. While finetuning the pretrained Open VLA model brings a 1.1% gain, finetuning ApennVAAPhhi model even degrades the performance by 0.3%. However, when visual trace prompting is incorporated, the success rate of VLe VLA model increases to 47.7%, highlighting the significant impact of visual traces on model performance.
TracoVLA vs. Directly Finctuned VLA
TraceVLA vs. OpenVLA finetured with History
finstuned
(Duns) VLA finsuuned W. sisual trace
OpenVLA
VLA History
VLA Visual Tacce
44.0%
4B VLA
39.5%
47. 74
39.9%
78 VLA
41.33 3%
10 2%
40 2%
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Success Rate (%)
Success Rate (%)
Figure 7: (Left): Comparison of average success rates between the base OLen and Open models and their finetuned versions, with and without visual trace prompting. (Right): Comparison of average success rates between the hase OLenVLAA,Trace VLA, and OpenVLA finetuned with sequence of 6 images.
Will appending historical image observations also give similar performance gain as TraceVLA? Instead of using visual trace prompting, the most straightforward way to integrate the model's historical movement knowledge is to provide it with multiple past image observations. To test this method, we input N frames into the VLA model, separated by a learnable separator token, similar to our approach in VLA. We also include sentence in the text prompt inform the model of this change in observation. Here, N = 6, which matches the length of the visual trace used in TraceVLA. In theory, the model should receive more information from these 6 complete frames compared to visual trace prompting. However, as shown in 7 (Right), finetuning Open with historical information not only fails to improve overall performance but also reduces it by 6%. This performance drop is likely due to redundant information between visual tokens at different timesteps, which may distract the model from focusing on the most relevant information for deciding next actions. In contrast, visual trace prompting offers useful hints that enhance the context for the vinion-language model.
Is visual trace prompting better than text trace prompting for grounding VLA with temporal- spatial understanding? addition to guiding the model with point tracking by overlaying the visual trace on the original image, we can also describe the movement of the points using 2D coordinates in text. To assess whether visual prompting is the most effective method for improving the VLA model's opatial-temporal understanding, we implemented an alternative approach that describes the points' movement verbally, treating the trace as text tokens, as shown in Figure 8 (Right).
Interestingly, using this text-based trace yields 2.4% average performance gain over the baseline VLA model, suggesting that point tracking information is indeed useful for the robot model. However, compared to our approach (Trace the additional 6.4% gain indicates that leveraging the vision encoder's ability to process visual prompts is much more effective than describing the points' location and movement in text. While text descriptions can precisely convey the location and movement of each point, they also increase token count (by 150 tokens) compared with visual trace prompting, leading to substantially higher GPU costs. Moreover, relying solely on text fails to fully leverage the multimodal grounding capabilities of current vasion-language models. Our visual trace prompting approach strikes an optimal balance between efficiency and efficacy, demonstrating superior performance gains (an additional 6.4% improvement) over text-based alternatives while maintaining computational efficiency.
8

---

Published as a conference paper at ICLR 2025
OpenVLA
Text Trace
Trace
You are glven an Lrage of robot's observation. Addtesonally, you
are given inverent informttion points the Tmage, Oven
the frores, the points noved as follons:
Point noved through positions: [244.88], [208, 115]. [177, 31]
Point noved through positions: [215, 96]
[289,122]
[185180]
Point noved through posistons: [237.115] (151,
[288,42],
[161,79]
Point noved through postaiios: [225.49] 491.
[206. 127].
[171.72].
Point noved through posi tions: [251, [251,95], [208, 103].
[18,991) 10
40.2%
20% Success Rate 196)
20%
40%
Non what robot should do
coke con?
Figure 8: (Left): Comparing visual trace prompting and text trace prompting. (Right) Text trace prompts example.
How does VLA perform under different lengths of vi- sual trace? An important hyperparameter of VLA is the length of visual trace prompting N, which determines the length of the historical information provided in the visual prompt. While longer N includes more past observations, it can clutter the visual context and potentially obscure key objects or the robot end-effector, while shorter N has less risk of covering the key information but contains less historical information. As shown in Figure 9, using smaller number of steps (N 3) results in 3.2% performance improvement.
52%
47.7%
Average Success Rate
435%
35%
Steps of M Mssal Trace
Figure 9: TraceVLA under differ- ent length of visual traces.
However, since visual trace of only three steps is often too brief, the performance gain is not as substantial compared to larger values of N. Conversely, setting N too large. such as N 12, may lead to less informative trace that obscures key scene elements and overlaps partially with previous motion traces of robots in the overlaid images, potentially distracting the VLM model, thereby slightly reducing overall performance. In practice, tuning N should be straightforward, and might be dataset- specific. involves sampling a few episodes and visually inspecting the generated trace to ensure that the selected N provides appropriate balance between historical context and the clarity of the scene.
LIMITATION ANALYSIS: TRAINING MEMORY COST AND INFERENCE SPEED
Since TLA introduces an additional image input into the model and uses CoTracker to obtain the visual trace during testing, we examine both the training memory cost and inference speed of TraceVLA. For evaluating memory cost, we launch a single-node multi-gpu training job with 8 H100 graphics cards under varying batch sizes, and we measure the maximum GPU memory usage across H100s. All tests are conducted with flash attention and torch.blloat.e as the datatype of model weights and inputs. As shown in Figure 10 (left), when the batch size is 32, the memory difference between VLA and models without visual trace prompting (for both Phi3 and Open VL.)) is less than 10GB. Notably, this difference becomes even smaller with a reduced batch size, indicating that while VLA incurs some extra GPU memory cost, this additional GPU memory cost remains manageable and not significantly impactful.
For testing inference speed, we evaluate the time cost using a single H100 GPU. In addition, we also analyze the time cost of each additional component introduced in TraceVLA compared the original Open VLA model. During inference time, the extra computation introduced by TraceVLA consists of approximately 300 additional image and text tokens for the transformer model each timestep, 5-point CoTracker point tracking for every timestep, and K x K (where K = 40) dense point tracking every 20 steps. For the 40 40 dense point tracking, as requires recalculation only every 20 steps, the average time cost per timestep is computed by evenly distributing the total cost over each timestep. As shown in Figure 10 (right), we observe that the additional text and image tokens result in negligible inference cost (around 0.002 seconds), likely due to GPU optimization for attention. Using M points (M = 5) for CoTracker per timestep adds an extra 0.03 seconds per step, while the dense 40 x 40 point tracking has amortized cost of 0.004 seconds per step.

---

Published as a conference paper at ICLR 2025
GPU Memory Cost Across Different Models
Inference Time Cost Across Different Models
TraceVLA
0.160s
Batch Size
16
OpenVLA Additional Text
Tokens 5 Points Cotracker
0.150s
OpenVLA Additional Text Image Tokens
0.1266
OpenVLA
(Gurs)
OpenVLA-fhi3
OpenVLA
TracOVLA PhiB (Ours)
20
40
60
80
0.00
0.02
0.04
0.06
0.08
0.10
0.32
0.14
0.16
0.18
GPU Memory Cost (GB)
Inference Time cost per timestep (seconds)
Figure Compaaionn of GPU memory cost of TraceVLA, OpenVLA and TraceVLA AhPPh,, OpenVLA-PPii (Right): Comparison of inference time across different models.
RELATED WORK
Vision Language Action Models. Several studies have investigated the application of vision- language models (VLMs) in robotics (Karamcheti et al., 2023; Gadre et al., 2022; Driess et al., 2023; Du al., 2023). Among them, Robopoint (Yuan al., 2024) and RepKep (Huang et al., 2024b) leverages VLM for explicit key point coordinates prediction, which is then converted to low-level actions through an off-the-shelf motion planner. Meanwhile, many recent works have explored fine-tuning large pretrained VLMs to directly predict robot actions as VLA models, treating these actions as tokens within the language model vocabulary (Brohan et al., 2023; Niu et al., 2024; Zhu et al., 2024; Li et al., 2024; Kim et al., 2024). Among them, RT-2 (Brohan et al., 2023) fine-tuned VLMs on both robotic trajectory data and Internet-scale vision-language data. LLARVA (Niu et al., 2024) generated both 2D visual traces in image coordinates and corresponding textual actions as outputs, with the former functioning as an auxiliary task. LLaRA (Li et al., 2024) generates multiple auxiliary datasets with complementary training objectives to provide additional supervision. RT- 2-X (Collaboration et al., 2023a) trains a 55B-parameter VLA policy on the Open X-Embodiment dataset. Open VLA (Kim et al., 2024) combines open VLM backhone with richer robot pertaining dataset. We build on top of Open VLA, but distinctively address the challenge of maintaining awareness of past spatial trajectories in VLA.
Visual Trace for Robotics. Visual traces of moving objects are vital for improving robotic action prediction, as they convey essential information about object dynamics. Various approaches have been developed to utilize visual traces in robotics, including using hand-drawn sketches for goal specifications (Gu al., 2023), predicting future traces and learning a trace-guided policy (Wen et al., 2023; Bharadhwaj et al.), identifying active points for motion planning (Vecerik et al., 2024), and localizing active regions of robot observations for video generation (Huang et al., 2024a). More recently, the vision language action model LLARVA (Niu al., 2024) predicts future 2D traces in text format as intermediate outputs alongside action tokens. In contrast, our approach integrates historical visual traces directly into VLA models as visual prompts. This novel method enhances VLMs' contextual understanding of the spatial and temporal dynamics, addressing an aspect previously underexplored in VL.A models.
7 CONCLUSION AND DISCUSSION
Our work advances assionlaaguaag--cttion (VLA) models for robotic manipulation by introducing a novel visual trace prompting technique and providing a dataset enriched with spatial-temporal information across diverse embodiments. With state-of-the-art 7B and 4B VLA models, we push the boundaries of VLA performance, demonstrating their effectiveness in both extensive simulated environments and real-world robotic tasks. By bridging the gap between visual perception, temporal awareness, and physical embodiment, we significantly enhance the generalization and adaptability of VLA models.
Looking ahead, promising directions for future research include incorporating multi-point spatial trajectory prediction, allowing models not only to react but also to anticipate and plan actions with greater foresight. Additionally, leveraging 3D point cloud data for training could further enrich spatial representations. capturing fine-gre mme-griieed details in complex scenes and objects, thus improving manipulation accuracy and robustness across diverse and dynamic scenarios. These advancements will continue to enhance the generalization capabilities of VLA models, driving further progress in robotic manipulation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao. Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wemxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmaan, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li,, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilae, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen. Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang. Guanhua Wang. Lijuan Wang. Shuohang Wang, Xin Wang. Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang. Jianwei Yang. Yifan Yang. Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024a. URL https: //apiv/aaaivv///aa//eaeeeaeiiiaa org/ abss 2404.44442 14219.
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14211, 2024b.
1
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXhv:2303.0877 2023.
Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinay Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation. 10
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 222.v:22222.666) 2022. 1,5
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et Rt-2: Vision Vision-language-acttion models transfer web knowledge to robotic control. arXiv preprint 2rXiv:2307.1581/ 2023. 10
Open X-Embodiment Collaboration, Abby O' Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ccola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su. Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj. Homer Walke, Hongjie Fang. Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake,
11

---

Published as a conference paper at ICLR 2025
Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu. Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik. João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang. Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs. Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart in-Mart in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore. Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma. Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma. Zhuo Xu. Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. attps://arxxx org/ abs 2310. 08864, 2023a. 5, 10
Open-X Embodiment Collaboration, A Padalkar, A Poolcy, A Jain, A Bewley, A Herzog, A Irpan, A Khazatsky, A Rai, A Singh, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08866 2023b. 1
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03376 2023. 10
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas and Serkan Cabi. Vision-language models as success detectors. arXiv preprint 0730002303.07280, 2023. 10
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint arXiv:2203. /0421, 3(4):7, 2022. 10
Jiayuan Gu. Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:23/1.01977 2023. 10
Shuaiyi Huang, Mara Levy, Zhenyu Jiang, Anima Anandkumar, Yuke Zhu, Linxi Fan, De-An Huang, and Abhinav Shrivastava. Ardup: Active region video diffusion for universal policies. arXiv preprint arXiv:2406. 0134406633300 13301, 2024a. 10
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In 8th Annual Conference on Robot Learning, 2024b. URI. https: tetttts://operreeeeee http///ppee//////eeeeeee/ee/eeee net www.neeetttn/ieieeeeeeeoe (tt.:rummmii991133f1011 G3 10
12

---

Published as a conference paper at ICLR 2025
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better to track together. arXiv:2307.07635, 2023. 2,4
Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang, Language-driven representation learning for robotics. arXiv preprint arXiv:2302.12266, 2023. 10
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML), 2024. 5
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-actiin model. arXiv preprint arXiv:2406.092 2024. 5, 10
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, et al. Llara: Supercharging robot learning data for Dision-language policy. arXiv preprint arXiv:2406.20099, 2024. 10
Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprini arXiv:240..03/74 2024a 18
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872 2024. 18
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darelll, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arxiv:2406. 1815, 2024. 10
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint 24X.v:2405.12211 2024. 5
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. 3
Mel Vecerik, Carl Doersch, Yi Yang. Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5397-5403. IEEE, 2024. 10
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pp. 1723-1736. PMLR, 2023. 5
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy leaming. arXiv preprint arXi::2401.00225 2023.
10
13

---

Published as a conference paper at ICLR 2025
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong. Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07560 2023. 18
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310. 11444, 2023a. 18
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 17421, 9()):, 2023b. 18
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision- model for spatial affordance prediction in robotics. In 8th Annual Conference on Robot Learning, 2024. URL https: 1, 10
//openreview
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023. 3
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint MxXiv:2401.09111 2024. 10
14

---

Published as a conference paper at ICLR 2025
A REAL ROBOT TASKS SETUP
Here we provide the detailed language instruction for each task that we designed. For each trial, we randomize the intial location of the target object, and for each trial except for folding cloth, we have also added 2-3 random distracting objects into the scene, including toy pepper, eggplant, ketchup, carrot, and donut.
Tasks included in the finetuning dataset.
Task 1: fold cloth:
Fold the cloth from right to left. The trial is counted as a success only when the robot succeesfully grasp the right edge of the cloth and fold it to the left.
Task 2: swipe corn sink: Pick up the brush and then use the brush to sweep the corn into the sink, while avoiding collision with other objects. The trial is counted as success only when robot successfully swipe the corn into the sink without colliding into existing objects on the table.
Task 3: pickplace corn pot: Pick up the corn and then put it into the pot. The trial is counted as success only when the robot correctly picks up the corn and place into the pot. Note that the primary goal of the this task is assess the model's 's generalization capability, this task is not part of the training dataset. Instead, the training data includes a task that involves picking up an eggplant and placing in a pot.
Task 4: pickup knife: Pick up the knife first, and then place on the plate. The trial is counted as success only when the robot correctly picks up the knife and place into the target plate.
Unseen tasks for generalization:
Task 1: pickplace banana: Pick up the banana and place it to the right of the plate. The trial is counted as a success only when the robot correctly picks the banana and places it to the right of the plate. This task particularly challenging because the banana object is unseen in our real-robot finetuning dataset. Additionally, solving this task requires the model leverage language understanding capability to ground spatial knowledge, rather than relying spurious correlations, the instructions in our finetuning dataset only involve placing objects on the plate.
Task 2: pickplace eggplant: Pick up the eggplant and place on the plate. The trial is counted as success only when the robot correctly places the eggplant on the plate. This task tests the model's capability for handling unseen goals, as the finetuning dataset only includes placing the eggplant into pot. Additionally, the eggplant is difficult to grasp, as incorrect placement of the end-effector could casily cause the eggplant to rotate and miss the target.
Task 3: lift battery: Lift the AAA battery. The trial is counted as a success only when the robot correctly picks up the battery and lifts it without dropping or damaging it. This task tests the model's is capability to handle unseen objects, as the battery is not included in our finetuning dataset.
Task 4: push cloth: Push the cloth from the left the right of the table. The trial counted as success only when the robot successfully pushes the cloth to within 1 inch of the right edge of the table. This task evaluates the model's motion generalization capability, as the finetuning dataset only includes tasks involving folding cloth.
15

---

Published as a conference paper at ICLR 2025
B QUALITATIVE RESULTS ON REAL ROBOT ROLLOUTS
In this section, present real robot manipulation rollouts for both the OpenVLA and Tra models. As discussed earlier, our TraceVLA model demonstrates significantly better generalization ability across various real robot manipulation tasks, unseen objects, and unseen language instructions. In Figures 11. 12, and 13, we qualitatively illustrate how the two models handle three tasks: "Pickplace Banana, Folding Cloth, and Pickplace Eggplant." " For the TraceVLA model, we also visualize the visual trace prompt that the model uses during evaluation.
Due to the proposed visual trace prompting, our TraceVLA model not only accurately picks up the banana and eggplant, grasps the edge of the folding cloth, and completes these tasks smoothly, but also demonstrates superior spatial understanding and reasoning capability compared to Open' VLA. In contrast, the OL model shows limited generalization capability, often overfitting to the finetuning distribution. For example, it places the banana directly onto the plate instead of following the instruction to place it to the right of the plate. These results further highlight the benefits of our visual trace prompting technique.
Figure 11: Pickplace Banana task. (Above): OpenVLA rollout. (Below): VLA rollout with visual trace prompting.
Figure 12: Fold Cloth task. (Above): OpenVLA rollout. (Below): TraceVLA rollout with visual trace prompting.
Figure 13: Pickplace Egeplant task. (Above): Open rollout. (Below): TraceVLA rollout with visual trace prompting.
16

---

Published as a conference paper at ICLR 2025
C ADDITIONAL ABLATION STUDIES
C.I THICKNESS, TRANSPARENCY, AND COLOR OF VISUAL PROMPTING
To further investigate the impact of visualization parameters, we conducted additional ablation studies. Specifically, we fine-tuned the TLaceVLA model on datasets with variations in trace visualization settings, including different line thicknesses, transparency levels, and color schemes. Our findings indicate that performance variations across these parameters are minimal within a reasonable range. Below, we provide detailed experimental results for each parameter setting.
The effect of varying the line thickness of visual traces on the SimplerEnv Average Success Rate is shown in Table 2. We observe only minor differences in performance when adjusting this parameter.
Thickness:
Thickness
SimplerEny Average Success Rate
1 newidth=]
47. 47.2%
1inewidth=2 (TraceVLA)
47.7 %
linewidth=3
47.8%
Table 2: Impact of line thickness on performance.
Transparency: We varied the transparency of the visual traces by adjusting the a parameter. Lower values result in more transparent traces. Table 3 summarizes the findings, demonstrating the robustness of TLAcs performance to these adjustments.
Transparency (a) SimplerEnv Average Success Rate
a (TraceVLA)
47.7%
0.8
47.3%
Table 3: Impact of transparency performance.
The choice of color scheme was also tested. The default Trace color scheme uses RYPBG (Red, Yellow, Purple, Blue, Green), while alternative scheme POBGG (Pink, Orange, Blue, Grey, Green) was evaluated, Results are presented in Table 4, showing negligible differences in success rates.
Color:
Color Scheme
SimplerEnv Average Success Rate
RYPBG VLraceVLA) POBGG
47.7 %
47.3%
Table 4: Impact of color scheme performance.
Takeaway: Our experiments reveal that the choice of visualization parameters, including thickness, transparency, and color, has negligible impact TraceVLA's performance when chosen within reasonable ranges. These results suggest that such parameters do not require extensive hyperparameter tuning, simplifying their selection process. This robustness underscores Trace VLA's reliability across different visualization settings.
C.2 BASELINE WITH DIFFERENT STEPS OF HISTORICAL OBSERVATIONS
In Figure 7, we compared VLA with OpenVLA using 6 steps of observation history to ensure both models had access to the same amount of historical information. Here, in Figure 14, we further compare LLA with OpenVLA fine-tuned using and steps of observation history on SimplerEnv. While slight performance improvement is observed with 2-step history for Open VLA, TraceVLA consistently and significantly outperforms the baseline success rates, highlighting the effectiveness of visual trace prompting.
17

---

Published as a conference paper at ICLR 2025
50%
Average Success Rate
45%
40.2%
39.5%
Number of historical observation steps
Figure 14: Comparison of Vrace against Open VLA with different steps of observation history.
D ADDITIONAL RELATED WORK
In this section, discuss additional works that apply visual prompting technique of VLM and their applications in robotics. In particular, visual prompting methods have emerged as new paradigm for VLM, complementing textual prompting and enabling more fine-grained and pixel-level instructions multimodal input for VLMs (Yang et al., 2023a;b), and has been widely used in roboties (Yan al., 2023; Liu al., 2024a; Nasiriany al., 2024). MOKA (Liu et al., 2024a) annotates key points visual marks images, converting affordance reasoning into series of visual question-an answernng problems that are solvable by the VLM. PIVOT (Nasiriany et al., 2024) cast robotic control tasks visual question ceestioo-anssering problems and iteratively refined visual prompts and action selection. Unlike existing work, our approach introduces visual trace prompting during fine-tuning of VLMs, overlaying key point traces on robot observations. Our novel visual trace prompting directly incorporates temporal information into the visual input, enhancing VLA models' spatial-temporal awareness for more effective action prediction in robotic tasks.
18

---

Published as a conference paper at ICLR 2025
E MORE IMPLEMENTATION DETAILS
During inference, we aim to make our visual trace prompting as efficient as possible, adding minimal computation to the original VLA model. Extracting the visual trace by querying Co-Tracker for K x K grid at each timestep is not feasible due to efficiency constraints. Instead, if we know the active points from the previous timestep, we can query Co-Tracee Tracker for only M active points, which is faster and more cost-effective.
Ideally, similar KV caching LLM inference, we only rum Co-Tracker with the K K grid once at the start of the trajectory to find the M active points. After that, we query Co-Tracker only for these M active points throughout the trajectory. However, in practice, we observe that Co- Co-Tracker might lose track after some steps (around 30 to 40, depending on the actions' magnitude). To address this, VLA periodically re-queries Co-Tracker to recalibrate after long interval. This ensures that the need for dense K X K point tracking is infrequent within an episode. As result, the total number dense queries during a trajectory is minimized, while tracking a few active points incurs little additional cost, adding minimal computational overhead to the model.
We refer the readers to Algorithm for the pscudocode of TraceVLA model inference.
Algorithm Python-style pseudocode for Trace Inference.
Co-Tracker Grid Size (a.g.,
40)
# Number Points to Track (e.g.,
# N: Trace Length for Co-Tracker (e.g.. 6)
Maximum timesteps for inference (e.g., 500)
redraw_ee steps: Number of steps for Recomputing the K*K dense point tracking
Initialization
image env.reset()
historical observations quaue = Quaue (max length=N)
tracked_pointe None
for in range (0, T):
s=
Kxx dense point tracking at timestep or every redraw_steps for avoiding losing tracks if t or (t redraw steps =s and t 0):
dense CoTracker point tracking grid generate grid points (K, shape)
Recalculate
Get x K grid points trace = costaokerlistooiicaalaaaaae observations quaue, grid points)
sample
Samplpe visual
Update tracked points by using active points on the latest frame trintss tracel:, -1,
else:
Continue tracking
existing points
trace cotracker (histtrrcal observations S queue, tracked tracked_pointal points)
x X
Update tracked points by tracked points tracel:, -1, :]
active points the latest frame
Overlay trace overlay trace (image, trace) action = tracevIA ([image, [age__overaad] trace_ prompt)
and compute action using visual language model
image
Use the prompt without trace hint for initial timesteps action = traceVIA ([image, image], prompt) stepp(attiioon) historical observations_ queue. append
19

---

Published as a conference paper at ICLR 2025
F ADDITIONAL EXPERIMENTAL RESULTS ON LIBERO SIMULATION BENCHMARKS
(a) PIBERO-Spation
(b) LIBERO-OBii Object
(c) LIBERO-GoR Goal
(d) LIBERO-Long
Figure 15: Four test suites of LIBERO simulation benchmark.
Method
Spatto-Spatial
LIBERO-Object
LIBERO-Goal
LIBERO-Long
Average
VLA finetuned
84.6% 0.2%
85.2%
0.4%
75.1%
0.3%
54.1% 1.0%
74.8% + 0.4% 70.6%....4
VLA tinetuned
82.6% 0.4%
83.8% + 0.6%
70.4% 0.5%
45.7% - 0.6%
Table 5: Multitask success rates on LIBERO simulation benchmarks.
In addition to SimplerEny and WIDOWX-250 real robot experiments, in this section, we conduct an additional experiment on LIBERO simulation benchmarks. In particular, we take the four suites from LIBERO: LIBERO-Iong, LIBERO-Spatie LIBERO-Object, LIBERO-Goal in LIBERO, each with 10 tasks and 50 human teleoperated demonstrations per task. We evaluate the multitask performance of the pretrained VLA policy on these four suites. Specifically:
LIBERO-Spatial: Contains the same set of objects but in varying layouts, testing the model's ability to understand spatial relationships. Example language instruction: pick up the black bowl between the plate and the ramekin and place it on the plate.
LIBERO-Object: Features consistent scene layouts but introduces different objects, evalu- ating the model's understanding of object types. Example language instruction: pick up the alphabet soup and place it in the basket.
Maintains the same objects and layouts while varying task goals, assessing the model's knowledge of diverse task-ort oriented behaviors. Example language instruction: put both the alphabet soup and the tomato sauce in the basket.
LIBERO-Goal
LIBERO- (also referred to as LIBERO-10): Comprises long-horizon tasks involving diverse objects, layouts, and task goals, challenging the model's ability to handle extended planning and execution. Example language instruction: open the middle drawer of the cabinet.
Following VLe, we preprocess the data by filtering out non-successful trajectories and removing all steps with actions that have near-zero norms and do not change the gripper's 's status. For VLa we also annotate visual trace following the exact same procedure as what we described earlier, Bridge and Google Robot dataset. Then we finetune both the Ope model and the TraceVLA-7B model on combined dataset from these four suites and report their multitask success rates on each suite in table 5. (Note that compared to the numbers reported in the Open VLA paper, here we finetune single model on the mixture of all four suites altogether instead of finetuning on each suite separately and report the numbers.) For each task, we evaluate each method using three random seeds. For each seed, we perform 50 rollouts with random initial states and report the average performance across all 10 tasks per suite. As shown with table 5, compared with OpenVLA, the superior performance of Trace VLA across each benchmark suite further demonstrates benefits of our visual trace prompting.
20


Reviews:
[{"review_id": "fWtptavFpw", "reviewer": "Reviewer_xXSr", "summary": "This paper proposed a visual trace prompting method called TraceVLA to enhance VLA models in manipulation tasks. \nBy incorporating Co-Tracker to visually prompt keypoint trajectories into existing VLA frameworks, TraceVLA achieves better performance than baselines in simulated and real robot experiments. It also shows better generalziation across environmental variations than vanilla VLA model and text prompted ones, further proves the advantage of the proposed visual trace method.", "questions": "1. How does this method generalize under camera orientation changes? By nature it helps the model against illumination/background changes, but how can VLA remain effective when the camera moves, given only 2D tracking result? If the change is small, then to what degree do you vary? You mentioned in the real robot section that finetuning is needed due to domain shift, but not for the simulation. Is that also due to the domain gap being small in simulated experiments?\n2. Regarding the active point selection, did you choose the $\\kappa$ so that $M=5$ takes a reasonable ratio in the active points, and therefore similar traces are learned in the training set? If not, does it have the potential to generalize across visually different robots?\n3. Will the color, thickness, or even transparency of the visual prompt affect the performance? \n4. Is K somehow related to the size of the image patch size? \n5. Will the performance drop if the action is moving into/out of the camera frame, or rotating in/out of it?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear description of the method. Comprehensive ablation on prompt setting and trace length, as well as analysis of memory and speed. \n2. Regarding the drawback in VLA practice, this work creatively applies tracking as visual prompting for VLAs to learn the invariance across different scene settings in manipulation.", "weaknesses": "1. Sec 4.1 is a bit hard to read. Mixing figures, tables, and your main context together with reduced blanks is not a good idea to present your experiment results clearly. \n2. Lack of details in experiments. For example, how do you define success, especially in real robot experiments? When you add noise to the environment, do you have a limit over related parameters? Is there any case study that qualitatively shows the advantage of applying visual tracking prompts? Please add them to your revision, which is also good for reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UScXq44pHV", "reviewer": "Reviewer_Gs1B", "summary": "This work leverages visual trace prompting to enhance VLA models' spatial-temporal awareness for action prediction by visually encoding state-action trajectories. The authors curated a dataset of 150K robot manipulation trajectories using visual trace prompting and fine-tuned it on OpenVLA.", "questions": "Please refer to weakness section for my questions. I would consider raising my rating if the weaknesses mentioned in my questions can be addressed. Overall, I find this to be a strong paper that offers a valuable perspective on training VLA models beyond simply adding more data.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces visual trace prompting, a novel technique that enhances spatial-temporal reasoning in VLA models for manipulation tasks by representing spatial-temporal relationships in robotic contexts.\n\n- A curated visual trace prompting dataset was developed, with state-of-the-art 7B and 4B VLA models fine-tuned using this approach, providing an efficient method to improve VLA model performance.\n\n- Their approach was rigorously validated through extensive evaluations across diverse simulated and real-world robot tasks, demonstrating superior generalization by leveraging spatial-temporal information.", "weaknesses": "- While the author presented results highlighting the importance of historical information from the trace, there were no results demonstrating spatial reasoning capabilities as claimed in the \"spatial-temporal\" contribution. One potential approach could involve fine-tuning a Vision-Language Model (VLM) for line-drawing tasks, connecting it to low-level policy, and leveraging the VLM independently for spatial reasoning, similar to the RoboPoint framework.\n  \n- The author relied entirely on real-world data to generate the trace, which may have introduced variability from environmental factors such as lighting. Generating trace data in simulation could mitigate these issues, potentially minimizing the sim-to-real gap. Additionally, evaluating performance would be easier with ground truth data generated from a platform like RLBench across diverse tasks.\n\n- Generalization in the Vision-Language Alignment (VLA) could be a strong aspect of this approach, as the data captures a general representation of actions conditioned on language. It would be beneficial for the author to test generalization on simulation benchmarks, such as the Colosseum, to showcase its robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h642BKKYas", "reviewer": "Reviewer_4XPZ", "summary": "This paper introduces TraceVLA, a novel method which leverages visual traces to enhance the spatial-temporal awareness of VLA models. \nSpecifically, the VLA model takes as inputs two images: a plain image and an image overlaid with traces of a set of active points.\nThe traces are generated by CoTracker and the active points are selected from a grid of dense points based on the changes in pixel locations.\nThe state-action history can be effectively communicated to the model through the provided visual traces.\nExperiment results show that the proposed method outperforms comparing baseline methods in the SimplerEnv simulation environment and four real-robot tasks.\nAblation studies show that using visual traces is a more effective method for conveying historical information than prompting with text traces or appending historical image observation.", "questions": "1. The proposed method leverages CoTracker to provide visual traces. Were any failure cases due to inaccurate tracking observed during experiments? How robust is TraceVLA in handling these inaccuracies?\n\n2. As the active points are selected based on changes in pixel locations, will points on moving background objects be chosen as active points? And how does the proposed method perform when the active points are located on irrelevant background?\n\n3. How many iterations were used for training during fine-tuning on the data in real-robot experiments? Is the data from the pre-training stage also included in fine-tuning as well?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method, TraceVLA, is a neat and effective approach for providing large VLA models with historical state-action information.\nVisual traces help VLA models to maintain awareness of the robot movements and do not introduce redundant information compared to appending historical image observations.\nExperiments in SimplerEnv show that TraceVLA outperforms multiple comparing baseline methods in overall performance.\nThe paper also evaluates on real-robot tasks to validate the proposed method in the real world.\nAblation studies offer good insights on how leveraging visual trace prompts compares to other prompting strategies.", "weaknesses": "1. The experiments in the real world contains only one task for generalization assessment. It would be better to evaluate the proposed method in more generalization settings, including novel backgrounds and more novel tasks. This would provide a more comprehensive insight on the generalization capabilities of the proposed method.\n\n2. RT1-X, despite having a much small number of parameters, outperforms TraceVLA-Phi3 with 3B parameters in SimplerEnv. The advantage of TraceVLA with 7B parameters is also not substantial. Is it possible to provide more discussions on potential reasons? Additionally, it would be beneficial to include RT1-X in real-robot experiments for comparison. In particular, observing how these two methods perform in generalization settings would be informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rw8RFDfyWP", "reviewer": "Reviewer_qxt2", "summary": "This paper proposes overlaying the history trace on the RGB image as an indicator of spatial and temporal information to help large foundation models better manipulate objects. Experiments are conducted in both simulation and the real world.", "questions": "Does the dataset used for fine-tuning OpenVLA contain scenes similar to those in simplerEnv? That is, are all the testing scenes entirely unseen in the training dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Although trace has been implemented in various aspects, from my knowledge, this is the first time it serves as a history spatial-temporal information indicator. Comprehensive experiments and analyses are conducted to demonstrate its effectiveness, and the authors also consider real-time inference by optimizing the inference process.", "weaknesses": "1. The choice of active points depends on a threshold. Does this threshold need to be chosen differently for each robot or dataset?\n\n2. In the analysis of trace length, the authors mention that a longer trace length can obscure key information. However, in the method, they state that they mitigate this by inputting both the original and overlay images. In that case, a longer trace length should not lead to worse results. This explanation seems self-contradictory.\"\n\n3. For fine-tuning OpenVLA with history images, six history frames work best for TraceVLA, but this might be too redundant when using image. Would using three or two history frames be better when using history images as input?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)