TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for
  Generalist Robotic Policies
Authors: name: Ruijie Zheng, name: Yongyuan Liang, name: Shuaiyi Huang, name: Jianfeng Gao, name: Hal Daumé III, name: Andrey Kolobov, name: Furong Huang, name: Jianwei Yang

Although large vision-language-action (VLA) models pretrained on extensive
robot datasets offer promising generalist policies for robotic learning, they
still struggle with spatial-temporal dynamics in interactive robotics, making
them less effective in handling complex tasks, such as manipulation. In this
work, we introduce visual trace prompting, a simple yet effective approach to
facilitate VLA models' spatial-temporal awareness for action prediction by
encoding state-action trajectories visually. We develop a new TraceVLA model by
finetuning OpenVLA on our own collected dataset of 150K robot manipulation
trajectories using visual trace prompting. Evaluations of TraceVLA across 137
configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate
state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and
3.5x on real-robot tasks and exhibiting robust generalization across diverse
embodiments and scenarios. To further validate the effectiveness and generality
of our method, we present a compact VLA model based on 4B Phi-3-Vision,
pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B
OpenVLA baseline while significantly improving inference efficiency.