Streaming 4D Visual Geometry Transformer
Authors: name: Dong Zhuo, name: Wenzhao Zheng, name: Jiahe Guo, name: Yuqi Wu, name: Jie Zhou, name: Jiwen Lu

Perceiving and reconstructing 4D spatial-temporal geometry from videos is a
fundamental yet challenging computer vision task. To facilitate interactive and
real-time applications, we propose a streaming 4D visual geometry transformer
that shares a similar philosophy with autoregressive large language models. We
explore a simple and efficient design and employ a causal transformer
architecture to process the input sequence in an online manner. We use temporal
causal attention and cache the historical keys and values as implicit memory to
enable efficient streaming long-term 4D reconstruction. This design can handle
real-time 4D reconstruction by incrementally integrating historical information
while maintaining high-quality spatial consistency. For efficient training, we
propose to distill knowledge from the dense bidirectional visual geometry
grounded transformer (VGGT) to our causal model. For inference, our model
supports the migration of optimized efficient attention operator (e.g.,
FlashAttention) from the field of large language models. Extensive experiments
on various 4D geometry perception benchmarks demonstrate that our model
increases the inference speed in online scenarios while maintaining competitive
performance, paving the way for scalable and interactive 4D vision systems.
Code is available at: https://github.com/wzzheng/StreamVGGT.