STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
Authors: name: Yushi Lan, name: Yihang Luo, name: Fangzhou Hong, name: Shangchen Zhou, name: Honghua Chen, name: Zhaoyang Lyu, name: Shuai Yang, name: Bo Dai, name: Chen Change Loy, name: Xingang Pan

We present STream3R, a novel approach to 3D reconstruction that reformulates
pointmap prediction as a decoder-only Transformer problem. Existing
state-of-the-art methods for multi-view reconstruction either depend on
expensive global optimization or rely on simplistic memory mechanisms that
scale poorly with sequence length. In contrast, STream3R introduces an
streaming framework that processes image sequences efficiently using causal
attention, inspired by advances in modern language modeling. By learning
geometric priors from large-scale 3D datasets, STream3R generalizes well to
diverse and challenging scenarios, including dynamic scenes where traditional
methods often fail. Extensive experiments show that our method consistently
outperforms prior work across both static and dynamic scene benchmarks.
Moreover, STream3R is inherently compatible with LLM-style training
infrastructure, enabling efficient large-scale pretraining and fine-tuning for
various downstream 3D tasks. Our results underscore the potential of causal
Transformer models for online 3D perception, paving the way for real-time 3D
understanding in streaming environments. More details can be found in our
project page: https://nirvanalan.github.io/projects/stream3r.