Streaming Visual Geometry Transformer
Authors: Dong Zhuo
Wenzhao Zheng
Jiahe Guo
Yuqi Wu
Jie Zhou
Jiwen Lu
Venue: ICLR 2026 Poster (2026)
Keywords: 3D reconstruction
geometry transformer
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Perceiving and reconstructing 3D geometry from videos is a fundamental yet challenging computer vision task. To facilitate interactive and low-latency applications, we propose a streaming visual geometry transformer that shares a similar philosophy with autoregressive large language models. We explore a simple and efficient design and employ a causal transformer architecture to process the input sequence in an online manner. We use temporal causal attention and cache the historical keys and values as implicit memory to enable efficient streaming long-term 3D reconstruction. This design can handle low-latency 3D reconstruction by incrementally integrating historical information while maintaining high-quality spatial consistency. For efficient training, we propose to distill knowledge from the dense bidirectional visual geometry grounded transformer (VGGT) to our causal model. For inference, our model supports the migration of optimized efficient attention operators (e.g., FlashAttention) from large language models. Extensive experiments on various 3D geometry perception benchmarks demonstrate that our model enhances inference speed in online scenarios while maintaining competitive performance, thereby facilitating scalable and interactive 3D vision systems.

Preprint. Under review.
STREAMING VISUAL GEOMETRY TRANSFORMER
Dong Zhuo
Wenzhao Zheng"
Jiahe Guo
Yuqi Wu
Jie Zhou
Jiwen Lu
Tsinghua University
hetps://wzzhen net /tt reamVGGT/
On-the-Fly
Reconstruction
Cached Memory Token
StreamVGGT
Streaming
Image
Geometry Token
Online Output
Input
Token
Figure I: Overview. Unlike offline models that require reprocessing the entire sequence and re- constructing the entire scene upon receiving each new image, our SGrT employs temporal causal attention and leverages cached token memory to support efficient incremental on-the-fly re- construction, enabling interactive and low low-latency online applications.
ABSTRACT
Perceiving and reconstructing 3D geometry from videos is fundamental yet chal- lenging computer vision task. To facilitate interactive and low-latency applica- tions, we propose a streaming visual geometry transformer that shares similar philosophy with autoregressive large language models. We explore a simple and efficient design and employ causal transformer architecture to process the input sequence in an online manner. We use temporal causal attention and cache the his- torical keys and values as implicit memory enable efficient streaming long-term 3D reconstruction. This design can handle low-latency 3D reconstruction by in- crementally integrating historical information while maintaining high-quality spa- tial consistency. For efficient training, propose to distill knowledge from the dense bidirectional visual geometry grounded transformer (VGGT) to our causal model. For inference, attention operators (e.g.. Flash Attentote from large language models. Extensive experiments on various 3D geometry perception benchmarks demonstrate that our model enhances inference speed in online scenarios while maintaining competi- tive performance, thereby facilitating scalable and interactive 3D vision sy systems.
model supports the migration of optimized efficient
INTRODUCTION
3D geometry reconstruction has long been fundamental task in computer vision (Hartley & Zis- serman, 2000; Özyeşil et al., 2017; Oliensis, 2000), which aims to estimate 3D geometry from set of images. As a bridge between 2D images and the world, finds broad applications in diverse fields including autonomous driving (Huang et al., 2023; 2024), AR/VR (Zheng et al., 2024; Hong et al., 2024), and embodied robots (Wu al., 2024; Wang et al., 2024b). With the development of
"Equal contributions. Projecct leader. Corresponding Author.

---

Preprint. Under review.
embodied intelligence, on-the-fly 3D reconstruction from streaming inputs is increasingly demanded to enable online interactive visual systems, where latency and temporal consistency are crucial.
Conventional methods like Structure-
from-Motion (SIM) (Snavely et al., 2006;
Agarwal et al., 2011; Frahm et al., 2010;
VGGT Time (ms)
Wu, 2013; Schönberger & Frahm, 2016;
StremVGGT Time (ms)
VGGT
Liu al., 2025) and Multi-View Stereo (MVS) (Furukawa & Ponce, 2009; Gu
Stream Mem (GB)
al., 2020) rely on explicit geometric constraints (Furukawa et al., 2015; Gal- liani et al., 2015) or global optimiza- tion (Niemeyer et al., 2020; Fu et al., 2022;
1.50
GPU Memory
Wei al., 2021; Yariv al., 2020), limit- ing scalability and speed. Recent learning- based approaches have shifted toward end- to-end frameworks that directly predict 3D structure from multi-view images. While pair-wise methods (Wang et al., 2024a; Leroy et al., 2024) have shown promising results by learning dense correspondences between image pairs, they utilize time-
93)
2.90
AFrames
Figure 2: Inference time and memory comparison for the current frame of varying sequence lengths be- tween StreamVGGT and VGGT for the online setting, consuming posoppsoes steps for global alignment during multi-view reconstruction. Memory- augmented methods (Wang& & Agapito, 2024; Wang et al., 2025b; Wu et al., 2025) maintain mem- Additionally. their recursive memory update designs have the ability address 3D reconstruction from videos. Nevertheless, error accumulation caused by causal architectures remains to be tackled. Fast3R (Yang al., 2025) and VGGT (Wang et al., 2025a) circumvent iterative alignment by employing feed-forward architectures that enable global dense-view interactions. Despite achieving satisfactory performance, their dependence on global self-attention necessitates reprocessing the entire sequence every step for frame-by-frane input and output scenarios. This offline paradigm precludes incremental reconstruction and diverges from the causal nature of human perception, limiting its practicality in streaming applications.
ory pool to eliminate the need for post-processing
In this paper, we propose VGGT, a causal transformer architecture specifically designed for efficient, low-latency streaming 3D visual geometry reconstruction, as shown in Figure 1. Unlike conventional offline frameworks that necessitate reprocessing the entire sequence with the arrival of each new frame, we introduce a temporal causal attention mechanism combined with an implicit historical-tokee memory module. This allows incremental processing of video frames, enabling progressive scene updates in an online streaming manner. EGreamVGGT leverages the inherent sequential and causal nature of real-world video data and performs online streaming perception, aligning how humans observe a scene. Furthermore, to reduce training cost while retaining high ac- curacy, we adopt distillation-based training strategy in which the densely connected, bidirectional VGGT (Wang et al., 2025a) serves as the teacher. By transferring its global geometric understanding. causal student model achieves performance comparable to full- sequence inference with signif- icantly fewer training resources and time. Distillation also unifies multi-task supervision through teacher-generated pseudo-GT, eliminating per-dataset engineering. In addition, the soft targets and confidence estimates of teacher act as effective regularizers, improving robustness and generaliza- tion. Equipped with FlashAttention-2 (Dao, 2023), our model supports fast inference for the cur- rent frame compared to VGGT, as shown in Figure 2. Experimental results demonstrate that our Stream VGGT reduces inference overhead long-tem sequences with only a slight performance trade-off, representing a crucial step toward low-latency responsive 3D vision systems.
2 RELATED WORK
Conventional 3D Reconstruction. 3D reconstruction, a fundamental task in computer vision (Hart- ley & Zisserman, 2000; Özyeşil et al., 2017: Oliensis, 2000), aims to recover the geometric structure of scenes from images or video sequences. Structure-from- MrotMMooon (SIM) (Snavely et al., 2006; Agarwal et al., 2011: 1: Frahm et al., 2010; Wu, 2013; Schönberger & Frahm, 2016; Liu et al., 2025) reconstructs sparse 3D point clouds by matching image features across overlapping views and jointly refining camera poses and scene points through bundle adjustment (BA). A typical workflow com- prises keypoint detection and description, feature matching with geometric verification, multi-view
2

---

Preprint. Under review.
initialization and incremental camera registration, triangulation, and global bundle adjustment. Although highly accurate in static scenes, SfM is fragile in dynamic or texture-poor environments and, because of its computationally intensive offline optimization, cannot readily update in real time. Multi-View Stereo (MVS) (Furukawa et al., 2015; Galliani et al., 2015; Niemeyer et al., 2020; Fu al., 2022; Wei et al., 2021; Yariv et al., 2020) exploits accurate camera poses to enforce photomet- ric consistency across views, producing dense depth maps or point clouds that can be converted into high-resolution meshes or volumetric models. Neural Radiance Fields (NeRF) (Mildenhall et al., 2021) extend this idea by fitting the volume- rendering equation to learn a continuous radiance field, faithfully capturing fine-detail geometry and photometrically consistent appearance. Yet both MVS and NeRF depend on heavy offline optimisation, cannot update incrementally, and therefore provide limited low-latency capability. As a result, they are best employed as offline dense-reconstruction modules appended to an SfM pipeline rather than as online SLAM.
Learning-Basse 3D Reconstruction. Building on the foundations of conventional 3D reconstruc- tion, recent end-to-end, learning-based methods utilize neural networks to encode scene priors, markedly improving robustness and cross-dataset generalisation (Wang et al., 2024a; Leroy et al., 2024; Zhang et al., 2024; Smart al., 2024; Fei et al., 2024; Dong et al., 2025). DUSt3R (Wang et al., 2024a) directly regresses view-consistent 3D point maps from just two RGB images without camera calibration, while its successor MASt3R (Leroy et al., 2024) introduces wonfidence-weighhed losses to approximate metric scale. Recently, the feed-forward wansformer-based architectures have emerged enable dense-view interaction within single pass (Yang et al., 2025; Wang et al., 2025a), delivering statc-of-the-an accuracy in a few seconds. Fast3R (Yang et al., 2025) extends the pair- wise DUSt3 idea to an N-view transformer equipped with memory-efficient Flash-Attention and parallel view fusion, allowing over 1000 images to be processed during inference. VGGT (Wang al., 2025a) scales this philosophy to a 1.2B parameter visual geometry grounded transformer that jointly predicts camera intrinsics/extrinsiess. dense depth, point maps, and 2D tracking fea- tures. However, its quadratic token-pair complexity and offline inference regime force complete re-encoding of every frame whenever new image arrives. This heavy memory footprint and non- causal processing preclude streaming 3D reconstruction and undermine low-latency applications demanding instantaneous, frame-by-frame scene understanding.
Streaming 3D Reconstruction. low-latency streaming 3D reconstruction grows to be indispensable for autonomous driving, robotics, and AR/VR, where systems update scene geometry and camera pose on every frame with low latency (Wang& & Agapito, 2024; Wang et al., 2025b; Wu et al., 2025). Spann3R (Wang & Agapito, 2024) augments DUSc3R-style encoder with token-addressable spa- tial memory, sustaining online point-map fusion but suffering drift on long or dynamic sequences due its bounded memory. CUT3R (Wang et al., 2025b) introduces a recurrent transformer that jointly reads and writes a learnable scene state to output camera parameters, dense depth, and novel- view completions in real time. However, it degrades when extrapolating far from observed regions, and demands heavy computation for recursive training. Point3R (Wu et al., 2025) couples an ex- plicit geometry-aligned spatial pointer memory with 3D hierarchical RoPE and an adaptive fusion mechanism, delivering low-drift online pose, depth, and point-map updates in real time. Instead of designing the memory mechanism to store past information, we follow the philosophy of large language models and employ causal transformer to implicitly cache historical visual tokens.
3 PROPOSED APPROACH
3.1 STREAMING 3D GEOMETRY RECONSTRUCTION
Given a set of images {4)T=s where each frame 4 E R3xHxW the generic 3D reconstruction pipeline can be written
F = E1coder(4.), G. = Decoder(F.), (Pr, C.) = He.d
(I) multi-
where the encoder maps each input frame 4 to a sequence of image tokens F; E R NCC view decoder then fuses tross-frame information, producing geometry tokens G2 N and MLP head predicts a point map P. E 33xHxW together with a per-pixel confidence map C. e R f from these geometry tokens.
This formulation provides a unifying paradigm for 3D reconstruction, but ofate-ofthe-ee art systems differ in how the decoder aggregates multi-view information, which can be grouped into three cat-
3

---

Preprint. Under review.
Streaming Input
Online Output
Encoder
Image
GEOO
T-2
Spatial Attention
Temporal Causal
Attention
Image Encoder
Point Map & Pose
Frame
Image Encoder
Depth
Depth Map
Spatial-Temporal Decoder
Figure 3: Framework of VGGT. Our model consists of three main components: image encoder, a spatio-temporal decoder, and multi-task prediction heads. During training, we utilize full- sequence inputs to provide the model with complete contextual information. To enforce temporal causality, we apply causal attention so the model can only attend to past frames at any given time step. This design encourages realistic temporal modeling suitable for streaming inference.
egories: pairwise, aannory-uggmetted, and global interaction. Each framework introduced special- ized modifications to the decoder and we summarize these designs as follows.
For pair-wiss wise approaches like DUSt3R (Wang et al., 2024a) and MASt3R (Leroy et al., 2024), two- branch cross-attention module jointly reasons over reference/target pair, and no persistent state is kept beyond the current pair:
{G1, G2) = Decoder (orooder(CrossAAmm(F F2)).
For memory-augmentee approaches like Spann3R (Wang & Agapito, 2024) and CUT3R (Wang et al., 2025b), an external memory M, updated online enables global consistency without post- processing during long sequence inference:
G₁ M₁- = Decoder(CrossAttm) (F, M,-1))--
For global interaction approaches like Fast3R (Yang al., 2025) and VGGT (Wang et al., 2025a), all frames attend to each other through all-to-all self-attention which achieves the highest accuracy the cost of O(N2) memory:
[G Decoder(Globar SelfAttm( (F.
Although global interaction approaches have achieved impressive reconstruction accuracy, their re- liance global self-attention mechanisms inherently limits their ability process streaming inputs efficiently. To overcome this limitation, we utilize a causal transformer architecture to explicitly model the causal structure intrinsic for streaming data. Specifically, for our StreamVGGT:
(G Decoder(Temppral SelfAttn (F.)
(5)
The temporal causal attention restricts each frame to attend only to itself and its predecessors, re- taining rich context while reducing latency O(N), which enables low-latency 3D perception.
3.2 CAUSAL ARCHITECTURE WITH CACHED MEMORY TOKEN
Building on the success of VGGT (Wang et al., 2025a), we propose a token-cached causal archi- tecture consisting of three key components: image encoder, spatio-temporal decoder, and multi-task heads, illustrated in Figure 3. We assume that the input images I are provided in sequential der, and all output attributes are predicted frame by frame. This sequential input-output structure aligns with the causal perception logic observed in humans and is well-suited for low-latency 3D visual geometry reconstruction tasks, where spatial consistency and causality play a crucial role.
Image Encoder. We patchfy each input image I, into set of N image tokens F, R NCC through DIuuu et al., 2023). During the training stage, the image tokens of all frames are subsequently processed through our causal structure, alternating spatial and temporal attention layers.
4

---

Preprint. Under review.
Cached Memory
Online Output
Multi-Task Head
Spatial-Temporal Decoder Layer
./2
Cached
Temporal
Spatial-Temporal Decoder Layer
Load
Attention
Spatial
Attention
Image
Encoder
Frame 1
Frame T-2
Frame T-1
Frame
Figure 4: Efficient inference of our model. During streaming inference, cache the historical keys and values as implicit memory to store information from past frames. This memory allows the model to efficiently reuse previously computed representations, avoiding redundant computa- tion and enabling consistent contextual understanding across time. Our model then processes input incrementally and achieves performance that is comparable to full-sequence inference.
Spatio-Temporyy Decoder. We introduce the spatio-temporal decoder by replacing all the global self-attention layers with temporal attention layers. In the standard global self-attention mechanism, each image token F; attends to all other tokens in the sequence, which can result in high computa- tional costs when handling long sequences and is not well-suited for streaming 3D reconstruction tasks. In contrast, by using temporal attention, each token is restricted to attend only to the cur- rent and previous frames the sequence, thereby respecting the inherent causal structure of the streaming inputs. This modification enables the model to maintain temporal consistency while sig- nificantly reducing the computational burden associated with global self-attention. In the training phase, we input all frames simultaneously, and the decoder generates geometry tokens G2 based solely on the context from the historical and current frames. This causal self-attention mechanism lays the foundation for enabling streaming input inference with minimal latency.
Cached Memory Token. Unlike the training phase, where all frames are input simultancously and processed with temporal attention, streaming 3D reconstruction requires the model handle frame-by- br-mm input and perform incremental 3D reconstruction during inference. To address this, we introduce an implicit memory mechanism that caches historical token M Tx Nxx x from previously processed frames, as shown in Figure 4. During inference. Stream VGGT performs cross attention between the eached memory tokens and the image tokens derived from the current frame:
GT Decoder(CrossAAtn (Frr, (M.) My TokknCaachdMeeorr(GG1
(6)
This design enables the model to replicate the temporal causal attention behavior observed during training. Through experimental validation, we demonstrate that the model with cached memory token achieves performance comparable that of full-sequence input inference. This confirms the effectiveness of our approach in maintaining high performance during streaming 3D reconstruction, making it well-suited for low-latency visual geometry reconstruction tasks.
Multi-Task Heads. Following the VGGT architecture, we utilize three distinct task heads, each dedicated to predicting key 3D attributes X of the scene. These heads responsible for estimating the camera pose E R point maps Pt E 3 HxW depth maps D, E IR "xW and point tracks Yt 2 Mf from every single frame. Specifically, for each input image I,, the image tokens F, produced by the image encoder are subsequently fed into the decoder to generate the geometry tokens G,. The geometry tokens are then passed through specialized task heads that predict the desired 3D attributes. We also leverage Icarnable camera token to mark the first frame as the global reference, so all subsequent frames are incrementally aligned within this shared coordinate system, enabling consistent, streaming 3D reconstruction without post-procces pros-processing.
Camera Head. This head responsible for predicting the intrinsic and extrinsic camera parameters. The output consists of the translation vector, rotation quaternion, and field of view (FoV), which together describe the pose of the camera in 3D space. The Camera Head uses self-attention layers to refine the camera parameters for each input frame.
5

---

Preprint. Under review.
Geometry Head. This head generates both the point map and depth map for each frame. Addition- ally, it outputs confidence maps (Kendall & Cipolla, 2016; Novotny et al., 2018) for both the point and depth predictions, indicating the certainty of the model in various regions. This head also pro- duces dense tracking features, which are passed to the Track Head for point tracking across frames. Specifically, it leverages DPT layer (Ranftl et al., 2021) to convert geometry tokens into dense feature maps, which are then processed by convolutional layers to produce the final point and depth maps with their corresponding confidence maps.
3.3 DISTILLATION- -AAEED TRAINING
Knowledge Distillation. To efficiently train our causal streaming architecture under limited com- putational resources, we employ knowledge distillation (KD) strategy that transfers the geometric understanding of the bidirectional VGGT (Wang et al., 2025a) teacher into the causal student. Al- though causal transformers naturally suited for low-latency inference, scalable training datasets typically requires substantial annotation resources and engineering, which KD effectively avoids by enabling supervision without full ground-truth labels per-dataset processing.
The VGGT teacher provides dense pseudo-labels for camera parameters, depth, point maps, and tracks, thereby unifying supervision across tasks and datasets. Its soft predictions and confidence estimates further offer uncertainty-an ness, consistent with prior KD findings (Furlanello et al., 2018; Romero et al., 2014).
guidance that stabilizes optimization and improves robust-
A key advantage of this strategy is the improvement in training efficiency. By inheriting the geomet- ric priors and multi-view consistency of the teacher, the student achieves high-quality performance with far fewer GPU hours compared to training on full annotations. As shown in Section 4.6, under the same training time and computational budget, removing distillation leads to notable accu- racy degradation, whereas the distilled student attains teacher-level performance while retaining the causal and streaming properties required for low-latency 3D perception.
Training Loss. We follow the loss design from VGGT (Wang et al., 2025a) but introduce a key difference: we use the output of a teacher model as pseudo-ground truth to supervise our causal student model. The teacher's soft targets and confidence estimates act effective regularizers, improving robustness and generalization The loss function is:
Leamer
Laepth + Lymap
The camera loss Teaaera supervises the predicted camera parameters 9. by comparing them to the ground-truth camera parameters g:. This is done using the Huber loss function:
Leanera Σ 11 gelle
(8)
where || lle represents the Huber loss function, which is robust to outliers the data.
The depth loss Ldepth incorporates depth confidence, which weighs the discrepancy between the predicted depth Dj and the ground-truth depth D; with the predicted confidence map Š.P The gradient-based term is applied to further refine the depth estimation, which is commonly used in monocular depth estimation tasks. The final form of the depth loss is:
N
Liepth
Σ 1=1
(D, ISP c (VD, -VD-- a log S.D
(9)
where ② denotes the element-wise product, and V represents the gradient.
The point map loss Lenup is defined similarly to the depth loss but for 3D point map. takes into account the point-map confidence 2 and ensures that the predicted 3D points P. match the ground-truth points P.. The loss is formulated
N
T Pmap
O -
P.)I+ E (VP, VP.)I a log D..
(10)

---

Preprint. Under review.
Table 1: Quantitative 3D reconstruction results on 7-Scenes and NRGBD datasets.
7 scenes
NRGBD
Accl
Compl
NC1
Acc.
Compl
NC1
Method
Type
Mean
Med.
Mcan
Med.
Mean
Med.
Med.
Mean
Med.
Mean
Med.
DUS:3RGG (Wang al., 2024a)
Pair-wise
0.146
0.077
0.181
0.067
0.736
0.839
0.144
0.019
0.154
0.018
0.870
0,982
MASER-GA (Leroy al. 2024)
Piir-wiee
0.185
0.081
0.180
0.069
0.701
0.792
0.085
0.033
0,063
0.028
0.794
0.928
MonST3R GA (Zhang el al., 2024)
Pair-wise
0.248
0.185
0,266
0.167
0.672
0.759
0.272
0.114
0.287
0.110
0.758
0.843
VGGT (Wang et al,, 2025a)
Dense-view
0.088
0.039
0.091
0.039
0.787
0.890
0.073
0.018
0.077
0.021
0.910
0.990
Spann3 (Wang Agupitu, 2024)
Streuming
0.298
0.226
0.205
0.112
0.285
0.684
0.789
CUT3R (Wang et ol., 2025h)
Streaming
0.126
0.047
0.154
0.031
0.727
0.834
0.099
0.031
0.076
0.026
0.837
0.971
Stream
Streuming
0.129
0.056
0.115
0.041
0.751
0.865
0.084
0.044
0.074
0.041
0.861
0.986
Table 2: Quantitative 3D reconstruction results on ETH3D dataset.
Method
Type
Acc.L
Comp.
Overall.
DUSt3R (Wang al,, 2024a)
Pair-wisc
1.167
0,842
1.005
MASGR (Leroy et al., 2024)
Pair-wise
0.968
0.684
0.826
VGGT (Wang al., 2025a)
Dense-view
0.928
0.443
0.686
CUT3R (Wang et al., 22226)
Streaming
1.426
1.395
1.411
VGGT
Streaming
0.609
0,545
0,577
Table 3: Quantitative 4D reconstruction results on TUM-dynamies,
Method
Type
Ace Mean.
Acc Med.1
Comp Mean.,
Comp Med.j
NC Mcant
NC Med.
VGGT (Wang et al., 2025a)
Dense-view
0,050
0.008
0,055
0.017
0.622
0,695
CUTER (Wang et al., 2025b)
Streaming
0.105
0.012
0.060
0.007
0,582
0.624
StreamVGGT
Streaming
0.085
0.011
0.058
0.007
0.617
0.690
Our framework unifies batch training and frame-by brafee inference through a causal, memory- cached architecture. During training, the student model distills the teacher's 's soft targets, which serve as effective regularizers that improve robustness and generalization, while significantly reducing the overall training cost. During inference, the cached memory tokens allow the model process streaming inputs incrementally, replicating the causal behavior observed during training without sacrificing accuracy. This design ensures both high training efficiency and low-latency inference, making our method particularly well-suited for streaming 3D reconstruction.
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Our model architecture follows VGGT (Wang et al., 2025a) with L = 24 layers of temporal and spatial attention modules. We integrated FlashAttention-22 (Dao, 2023) to accelerate the inference. We initialized Stream VGGT by using pre-trained weights from VGGT and fine-tuned approximately 950 million parameters (excluding the frozen image backbone) for 10 epochs. We employed the AdamW optimizer and a hybrid schedule of linear warm-up (first 0.5 epochs) followed by cosine decay, reaching peak learning rate of le-6. For each training iteration, we randomly sampled batch of 10 frames from diverse training scenes. Following Point3R (Wu et al., 2025), we processed input images with variable aspect ratios while resizing the maximum edge length to 518 pixels. Training was conducted on 4 NVIDIA A800 GPUs for days.
4.2 TRAINING DATASETS
Our Stream StreamVGGT VGGT was fine-tuned on a curated multi-domain collection comprising 13 datasets: Co3Dv2 (Reizenstein et al., 2021), BlendMVS (Yao et al., 2020), ARKitScenes (Baruch et al., 2021), MegaDepth (Li & Snavely, 2018), WildRGB (Xia et al., 2024), ScanNet (Dai et al., 2017), HyperSim (Roberts al., 2021), OmniObject3D (Wu al., 2023), MVS-Synth (Huang et al., 2018), PointOdyssey (Zheng et al., 2023), Virtual KITTI (Cabon et al., 2020), Spring (Mehl et al., 2023), and Waymo (Sun et al., 2020). This collection covers diverse visual domains spanning in- door/outdoor environments and temporal scales and balances synthetic data and real-world captures.
4.3 3D RECONSTRUCTION
Comparisons on the 7-scemes and NRGBD datasets. Following CUT3R (Wang al., 2025b), evaulated the 3D reconstruction performance of Stream on 7-Scenes (Shotton et al., 2013) and NRGBD (Azinović et al., 2022). Accuracy (Acc), completeness (Comp), and normal-consistency
7

---

Preprint. Under review.
Table 4: Single-Frame Depth Evaluation.
Sintel
Bonn
KITTI
NYU-v2 (Static)
Method
Type
Abs Rel L 1.25.25 1
[Abs Rel + 6x,11.25 1
Abs Rel + 8.1.25
Abs Rel 1 6x25.25
DUSt3R (Wang et al. 2024a)
Pair-wise
0.424
58.7
0.141
82.5
0.112
86.3
0.080
90.7
MAS13 (leroy al, 2024)
Pair-wise
0.340
0.142
0.079
0.129
84.9
MonST3R (Zhang al., 2024)
Pair wise
0.358
$4.8
0.076
93.9
0,100
89.3
0.102
88.0
VGGT (Wang al., 2025a)
Dense-view
0.276
0.055
0.072
0.060
Spann3R (Wang Agapito, 2024)
Streaming
0.470
0.118
0.128
0.122
CUT3R (Wang al., 2025bb
Stremiin
0.428
55.4
0.063
96.2
0,092
91.3
0,086
90.9
Point3R (Wo at....225)
Streaming
0.395
0.061
0.087
0.079
StreamVGGT
Streaming
0.254
68.5
0.052
97.1
0.072
94.7
0.055
95.9
Table 5: Video Depth Evaluation.
Sintel
BONN
KITTI
Method
Type
Abs Rel 1 6.25
Abs Rel
6.25
Abs Rel J <1,25
DUS13R-GA (Wang 2024a)
Pair-wise
0.656
45.2
0.155
83.3
0.144
81.3
MAS:3R-GA (Loroy et al., 2024)
Pair-wise
0.641
43.9
0.252
70.1
0.183
74.5
MonST3R- (Zhang
2024)
Puir-wise
55.8
96.3
74.4
VGGT (Wang et al., 2025a)
Dense view
0.298
68.1
0.057
96.8
0.061
97.0
Spann3R (Wang & Agapito, 2024)
Steemiig
0.622
42.6
0.144
81.3
0.198
73.7
CUT3R (Wang et al., 2025b)
Streamiin
0.421
47.9
0.078
93.7
0.118
88.1
PointR (Wi et al., 2025)
Streaming
0.452
48.9
0.060
96.0
0.136
84.2
SGrC
Streaming
0.323
65.7
0.059
97.2
0.173
72.1
(NC) scores are reported using sparse inputs (3-5 frames per scene on 7-Scenes and 2-4 frames per scene NRGBD). Table show that StreamVGGT performs competitively against existing streaming methods and surpasses the state-of-the-art streaming model CUT3R.
Comparisons the ETH3D dataset. Following VGGT (Wang et al., 2025a), we further evalu- ated the accuracy of the predicted point clouds on the ETH3D (Schops et al., 2017) dataset. For each scene, we randomly sampled 10 frames and report the results after discarding invalid points using the valid masks. Specifically, we measure accuracy (Acc), completeness (Comp), and over- all quality (Chamfer distance) for 3D reconstruction. Table shows that Stream Stream surpasses DUS(3R (Wang et al., 2024a) and MASt3R (Leroy et al., 2024) and matches the performance of the state-of-the-ard VGGT, despite relying solely on information from the current and past frames. Note that our causal design enables streaming 3D reconstruction and is more efficient than VGGT.
Comparisons on the TUM-dynamics Dataset. To evaluate reconstruction performance our Stream VGGT under dynamic scenarios, we conduct experiments TUM-dynamies (Sturm et al., 2012) dataset, which contains dynamic object. Specifically, 50 frames are sampled from each se- quence for this assessment. Based on these results in Table 3, we believe our method demonstrates strong capability for streaming 4D reconstruction in dynamic scenarios.
4.4 SINGLE-FRAME AND VIDEO DEPTH ESTIMATION
Single-Frame Depth Estimation. Following MonST3R (Zhang et al., 2024). we evaluated single- frame depth estimation across four datasets: KITTI (Geiger et al., 2013), Sintel (Alnegheimish et al., 2022), Bonn (Palazzolo et al., 2019), and NYU-v2 (Silberman et al., 2012), encompass- ing both dynamic/static scenes and indoor/outdoor environments. To ensure unbiased evaluation of cross-domain generalization, these datasets were strictly excluded during training. Following DUS13R (Wang et al., 2024a), we employ two principal metries: Absolute Relative Error (Abs Rel) and the percentage of predictions within a 1.25 factor of ground truth depth (61.25). Table 4 shows that our method not only matches the overall best performers but also outperforms the current state- of-the-art streaming model on all datasets, showing its superiority for online depth estimation.
Video Depth Estimation. We conducted video depth estimation by assessing both the depth quality on per-frame basis and the consistency of depth across frames. This was done by aligning the predicted depth maps with the ground truth using a per-sequence scale. We show the comparison of these methods in Table 5. Under aligned settings, our Stream VGGT exceeds CUT3R on both the Sintel and Bonn benchmarks and attains performance comparable to the offline VGGT.
8

---

fraanes Preprint. Under review.
Table 6: Camera Pose Estimation Evaluation on ScanNet, Sintel, and TUM-dynamics datasets.
ScanNet
Sintel
TUM-dynamies
Method
Type
ATE 1 RPE tran
RPE rot 1 AIE
RRPE trans
RPE rot
|AATE J RPE trans
RPE rot J
Robust, (Kopf et al., 20211
Pair
0.227
0.064
7.374
0.360
0.154
0.153
0.026
3.528
AM (Zhang et al., 2022)
Pair-wise
0.158
0.034
1.618
0.141
0.035
0.615
0.071
0.010
1.712
DUSER (Wang et al., 2024a)
Pair
0
0.784
0.417
0.250
5.796
3.567
MAS GA (Leroy et al. 2024)
Pair-wise
0.078
0.020
0.475
0.185
0,060
1.496
0.038
0.012
0,448
MonST3R-GA (Zhang et al., 2024)
Pair-wise
0.077
0.018
0.529
0.111
0.044
0.869
0.098
0.019
0.935
VGGT (Wang et al., 2025a)
Dense-view 0.035
0.015
0.377
0.169
0.064
0.474
0.012
0.010
0.307
Spann1R (Wang & Agapito, 2024)
Streaming 0.096
0.023
0.661
0.329
0.1
4.471
0.006
0.021
0,591
CUT3R (Wang
2025b)
Streatinng
0.600
0.213
0.066
0.473
Point3R (Wi et al., 2025)
Streaming 0.106
0.035
1.946
0.351
0.128
1.822
0.075
0.029
0,642
Stream VGGT
Streaming 0.048
0.019
0.557
0.219
0.103
1.041
0.026
0.012
0.316
Table 7: The comparison of inference-time and memory consumption for the online setting.
Method
frame /mss// CBB]
Acci ms]. [
10 fraiee
mms. [CB]
20
frames
Accl/ /mms] GB.
frames
SpanniK (Wang Apapioo, 20124)
5,3
0,038/ 87/
/ 82/5.5
120/
0.8K8
CUT3R (Wling 2025h)
0.035/ /3.4
0.03// 102/3.5
0.039 /9/
0031 338
BACT /40
0.023/ 102 /4.2 0.021 211 6.6
StreamYGGT
0.028/ 63.22.1
0,02A/ 881
0.02// 120/3.2
0021 /152/4.3
0.021 / 1.4/5.5
CUT3R
Ours
GT
Figure 5: Qualitative comparisons with CUT3R.
4.5 CAMERA POSE ESTIMATION
Table 6 shows the camera-pose estimation results on ScanNet (Dai et al., 2017), Sintel (Al- negheimish et al., 2022) and TUM dynamics (Sturm et al., 2012). Both Sintel and rUn-dynamics contain dynamic objects, which pose substantial challenges for conventional Structure-fromd Motion (SfM) and SLAM methods. We evaluate performance by reporting the Absolute Translation Error (ATE), Relative Translation Error (RPE trans), and Relative Rotation Error (RPE rot), all computed after applying Sim(3) alignment with the ground truth trajectories. Stream delivers perfor- mance on par with the best existing methods while uniquely supporting camera-pose prediction from streaming inputs, highlighting its practicality for low-latency applications.
4.6 EXPERIMENTAL ANALYSIS
Inference Time and Memory Consumption Analysis. To evaluate the efficiency of our design in streaming tasks, we compare the inference latency for the final frame of sequences containing 1, 5. 10, 20, 30, and 40 frames among Stream and VGGT (Wang et al., 2025a). All experi- ments were conducted on a single NVIDIA A800 GPU, For StrC and VGGT, we employ FlashAttention-2 (Dao, 2023) with an image resolution of 518 x 392. The results of inference time and memory consumption are summarized in Figure 2. Additionally, we provide comparison on inference consumption with CUT3R (Wang et al., 2025b) and Spann3R (Wang & Agapito, 2024). For fair comparison, the input resolution for both CUT3R and Spann3R is set to 512 384. As shown in Table 7, our method exhibits a moderate increase in inference time and memory usage when processing longer sequences. This is mainly because the number of cached memory tokens grows with the sequence length. However, we reckon that there is a trade-off between spatial mem- ory storage and performance, and VGGT shows good one for not very long sequences.

---

Preprint. Under review.
Table 8: Comparison of pruning strategies for online inference on a 200-frame sequence.
Method
Acc| (Mean/Med))
Compi (Mean/Mad))
NC1 (Mean/Med.)
Inference (Frame 200)
Peak Memory
Sparn3 (Wang & Agapiio, 2024)
0.225 1.6
0
0.535/0.557
221 054 ms
6.7 GB
CUT3R (Wang al., 2025b)
0.165 / 107
0.912
0.551/ /0.577
102.533
7.6 Gill
Ours No Prunning
0.058 0.035
0.026/0.010
0.632/0
733.083
Ours Window-bused (50 frames)
0.054/ / 0.030
0.023/ 0.009
0.636 /0.712
219.454 ms
8.2 GB
Oun. Window- based (100 frames)
0.062/ / 0.033
0.032/0.015
0.635/0.710
392.634 634 ms
14.0 GB
Ours K sccaaacst-- frames (K=50)
0.157/ 0.
0,101/0.011
0.605 / 006670
220. 109
8.1 GB
Ours K-neurest- framme 100)
0.054/ 0.032
0.019
0.626/0.702
392.634
Table 9: Effects of the distillation training strategy.
7 seemes
NRGBD
AccL
Comp4
NCT
Acc
Comp4
NCT
Method
Distillation Attention Mcan Med.
Mcan
Med.
Mcan
Med.
Mcan
Med.
Mean
Med.
Mean
Mod.
VGGT (Wang al., 2025a) StrT (wl KD) StreamVOGT (w/ KD)
Global 0.039
0.039
0.787 0.890
0.073
0.018
0.077
0.021
0.990
Causal
0.202
0,102
0.168
0.064
0.718 0.825
0.189 0,088
0.206 0.096
0,816 0.945
Causal
0.129 0.056 0.115 0.041 0.751 0.865 0.084 0,044 0.074 0.041 0.861 0.986
Table 10: Effects of the cached memory token mechanism for the online setting,
Method
Inference Time (at Frame 5)
Peak Memory
HashAAtt & Cache
1135. ms
GB
w/ FlashAttn
850. 699 88.193 ms
2.3 GB
w/ FlashAttn & Cache
2.7 GB
Still, StreamVGG1 faces scalability challenge for very long sequences. Therefore, we introduce two strategies: (i) windowed streaming, which partitions the sequence into fixed-length chunks and then aligns the point clouds produced for each chunk using the predicted camera extrinsics, so that peak memory stays within a predefined budget; and (ii) K-nearest-frames caching, where each frame attends only to tokens from the most recent K frames. As demonstrated in Table 8, both approaches effectively bound memory usage and latency during inference on extended videos, while maintaining high accuracy. This offers practical trade-off that effectively resolves computational concerns for sequeeeeeece handling without significant degradation in results.
Distillation Training Strategy. To assess the effectiveness of our sewwlede-diistllationn strategy, we evaluated three variants on the 7-Scenes (Shotton et al., 2013), NRGBD (Azinović al., 2022), and ETH3D (Schops et al., 2017) datasets: (i) the global self-attention teacher VGGT (Wang et al., 2025a), (ii) Stream without KD, and (iii) Stream VGGT with KD. Table shows that under the same training budget, the model trained without distillation exhibits higher reconstruction er- rors, indicating its limited ability to learn effective representations with constrained resources. In contrast, the distilled Stream VGGT achieves performance close VGGT across all 3D reconstruc- tion metrics while still operating within a low-latency, streaming architecture.
Cached Memory Token Mechanism. To evaluate the effect of our cached memory token mecha- nism and FlashAttention-22 (Dao, 2023) on inference speedup and memory reduction, we provide a ablation experiment in Table 10. The results clearly demonstrate that cached memory token mecha- nism reduces inference time and the FlashAttention-2 reduces the memory overhead.
Visualizations. VGGT delivers photorealistic scene reconstructions with superior geometric fidelity, fewer outliers, and accuracy in complex environments, as shown in Figure 5.
5 CONCLUSION
In this paper, we have presented Stream VGGT, a causal transformer architecture for low-latency streaming 3D visual geometry reconstruction. By replacing global self-attention with causal tempo- ral attention and introducing cached token memory mechanism, Stream achieves incremen- tal scene updates while preserving long-term spatial consistency. Extensive experiments demon- strate that Stream VGGT achieves comparable accuracy to the state-of-the-art offline model VGGT with small performance degradation, while surpassing current online state-of-the-art models across multiple tasks, including 3D reconstruction, single-frame depth estimation, and depth estimation. These advancements mark a critical step toward scalable, low-latency, streaming 3D vision systems for applications such as autonomous navigation and immersive AR/VR. Future work will explore dynamic scene adaptation and lightweight memory compression strategies to further bridge the gap between offline precision and mobile deployment requirements.
10

---

Preprint. Under review.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62336004, Grant 62125603, Grant 62321005, and Grant 62576188 188,, and in part by the Beijing Nat- ural Science Foundation under Grant No. L247009.

---

Preprint. Under review.
REFERENCES
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, lan Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome day. Communications of the ACM, 54(10):105-12, 2011.
Sarah Alnegheimish, Dongyu Liu, Carles Sala, Laure Berti-Equille, and Kalyan Veeramachaneni. Sintel: A machine learning framework to extract insights from signals. In SIGMOD, pp. 1855- 1865, 2022.
Dejan Azinović, Ricardo Martin Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In CVPR, pp. 6290-6301, 2022.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:21
2021.
Yohann Cabon, Naila Murray, and Martin Humenberger. arXiv:2001.10773 2020.
Virtual kitti 2. arXiv preprint
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pp. 5828- 2017.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
Siyan Dong, Shuzhe Wang, Shaohui Liu, Lulu Cai, Qingnan Fan, Juho Kannala, and Yanchao Yang. Reloc3r: Large-scale training of relative camera pose regression for generalizable, fast, and accu- rate visual localization. In CVPR, pp. 16739-16757 16752, 2025.
Xin Fei, Wenzhao Zheng, Yueqi Duan, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Ji- wen Lu. Driv3r: Learning dense 4d reconstruction for autonomous driving. arXiv preprint arXiv:2412.06677,, 2024.
Jan-Michael Frahm, Pierre GetegGe David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, al. Building rome on cloudless day. In ECCV, pp. 368-381. Springer, 2010.
Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. NeurlPS, 15:3403-3416 2022.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, pp. 1607-1616, 2018.
Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. TPAMI, 32 (8):1362-1376, 2009.
Yasutaka Furukawa, Carlos Hernández, al. Multi-view stereo: A tutorial. Foundations and Trends@ in Computer Graphics and Vision, 91-2):1-148, 2015.
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. ICCV, pp. 873-881, 2015.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 32(1 12:31 2013.
Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR, pp. 2495-2504, 2020.
Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2000.
Jin Gyu Hong, Seung Young Noh, Hee Kyung Lee, Won Sik Cheong, and Ju Yong Chang. 3d clothed human reconstruction from sparse multi-view images. In CVPR, pp. 677-687, 2024.
12

---

Preprint. Under review.
Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Sgaussian: Self-supervised street gaussians for autonomous driving. CoRR, 2024.
Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In CVPR, 2018.
Yuanhui Huang, Wenzhao Zheng, Yumpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. CVPR, pp. 9223-9232, 2023.
Alex Kendall and Roberto Cipolla. Modelling uncertainty in deep learning for camera relocalization. In ICRA, pp. 4762 4769. IEEE, 2016.
Johannes Kopf. Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In CVPR, pp. 161 1611-1621, 2021.
Yushi Lan, Yihang Luo, Fangzhou Hong, Shangchen Zhou, Honghua Chen, Zhaoyang Lyu, Shuai Yang, Bo Dai, Chen Change Loy, and Xingang Pan. Stream3r: Scalable sequential 3d reconstruc- tion with causal transformer. arXiv preprint arXiv:2508. 2025.
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2400609956. 2024.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. CVPR. pp. 2041-2050, 2018.
Shaohui Liu, Yidan Gao, Tianyi Zhang, Rémi Pautrat, Johannes Schönberger, Viktor Larsson, and Mare Pollefeys. Robust incremental fructure-frommmotion with hybrid features. In ECCV, pp- 249-269. Springer, 2025.
Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: A high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In CVPR, pp. 9981-4991, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, (5(1):99-106, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, pp- 3504-3515, 2020.
David Novotny, Diane Larlus, and Andrea Vedaldi. Capturing the geometry of object categories from video supervision. TPAMI, 42(2):261-275, 2018.
John Oliensis. A critique of structure- cenettrr-frommmoote from-motion algorithms. CVIU, 90(2):122-214, 2000.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from m Acta Numerica, 26:305-364, 2017.
Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss, Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In IROS, pp. 7855-7862. IEEE, 2019.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pp. 22179-12188, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cate- gory reconstruction. In ICCV, pp. 10901-10911, 2021.
13

---

Preprint. Under review.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, pp. 10912-10922, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arxiv 2014. arXiv preprint arXiv:1412.6550, 2014.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mottin revisited. In CVPR, 2016.
Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In CVPR, pp- 3260-3269, 2017.
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In CVPR, pp. 2930-2937, 2013.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. ECCV, pp. 746-760. Springer, 2012.
Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs. arXiv preprint arXiv:2408. 13912, 2024.
Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In SIGGRAPH, pp. 835-846. 2006.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A bench- mark for the evaluation of rgb-d slam systems. In IROS, pp. 573-580, IEEE, 2012.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for tonomous driving: Waymo open dataset. In CVPR, pp. 2446-2454, 2020.
Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint
arXiv:2408. /6061, 2024.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, pp. 5294 5294-5306, 2025a.
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Con- tinuous 3d perception model with persistent state, 2025b.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. CVPR, pp. 20697-20709, 2024a.
Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, al. Embodiedscan: A holistic towards embodied ai. In CVPR, pp. 19757-19767, 2024b.
3d perception suite
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou, Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, pp. 5610-5619,
October 2021.
Changchang Wu. Towards linear-time incremental structure from motion. In 3DV, pp. 127-134. IEEE, 2013.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Lorge-vocabulary 3d object dataset for realistic percep- tion, reconstruction and generation. In CVPR, pp. 803-814, 2023.
Yugi Wu, Wenzhao Zheng, Sicheng Zuo, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Embodiedoce: Embodied 3d occupancy prediction for basedn online scene understanding. arXiv preprint arXiv:2412. 04380, 2024.
14

---

Preprint. Under review.
Yuqi Wu, Wenzhao Zheng, Jie Zhou, and Jiwen Lu. Point3r: Streaming 3d reconstruction with explicit spatial pointer memory. In NeurIPS, 2025.
Hongchi Xia, Yang Fu, Sifei Liu, and Xiaolong Wang. Rgbd objects in the wild: scaling real-world 3d object learning from rgb-d videos. In CVPR, pp. 22378-22389, 2024.
Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fastår: Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint arXiv:2501.13998. 2025.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In CVPR, pp- 1790-1799, 2020.
Lior Yariv, Yoni Kasten, Dror Moran, Mcirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurlPS, 33:2492-2502, 2020.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, De- qing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024.
Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. ECCV, pp. 20-37. Springer, 2022.
Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps- gaassiaussnn Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In CVPR. pp. 19680-19690, 2024.
Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023.
15

---

Preprint. Under review.
Mean Ace Med. 1
Comp Mean
NC Mean NC Med.
Comp Med.
6:10-
Ace Valae
Valnes
0.6
0.04
0.1
612
0.2
810
10
20
30
73
10
40
#Prames
4'ramme e5
4Priamss
Figure 6: Per-frame results on 7-Scenes (>70 frames).
Table 11: Camera pose estimation on 7-Scenes under loop-closure scenarios.
Mothod
Type
Avg. Translational Error (m))
Avg. Rocational Error (*)
CUT3R (Wang et al., 2025h)
Streaming Streaming
0.0861
1.39
Stream
0.48
A APPENDIX
A.I
RECONSTRUCTION PERFORMANCE IN COMPLEX SCENARIOS
It is important to evaluate the reconstruction performance under complex scenarios such as long- sequence or dynamic scenes. We now provide comprehensive quantitative results of our method.
Per-frame Metrics on Long Sequence. To evaluate the temporal coherence of Stream on long-sequence. We conduct this experiment on the 7-Scenes (Shotton al., 2013) dataset, each sequence includes over 70 frames. We report the per-frame evaluation metrics every 10th frame as shown Figure 6. The results demonstrate that our approach maintains robust temporal consistency.
Loop Closure Analysis. Specifically, for each sequence in the 7-Scenes (Shotton et al., 2013) dataset, we select the first 50 consecutive frames and extend them into a lowward-nndreveed order to simulate loop closure scenarios. Then, evaluate the camera pose errors between the first and last frames (the same frame) for each sequence to assess the performance under loop closure conditions. The results in Table 11 demonstrate that model achieves lower drift and usable performance in loop closure scenarios.
Quantitative Reconstruction Performance when the Video Stream is Interrupted. It is im- portant for streaming methods to maintain usable performance when the input video stream is in- terrupted for period of time. As a result, we provide quantitative reconstruction results of our model on the 7-scenes (Shotton et al., 2013) dataset, where for each sequence we pick the first 50 frames and then remove the middle consecutive 25 frames to simulate the scenarios. The results in Table 12 demonstrate that our model can handle this challenging scenario better than the baseline CUT3R (Wang et al., 2025b).
Quantitative Reconstruction Performance to Noisy Frames. For noisy frames, we further cor- rupt inputs by randomly masking regions to simulate sensor noise. Table 13 shows reconstruction remains correct on the 7-scenes (Shotton et al., 2013) dataset, which proves the robustness of our StreamVGGT noisy frames.
A.2 MORE ANALYSIS
Cached Memory Token. To validate the effectiveness of the cached memory token, evaluated Stream VGGT on the ETH3D (Schops et al., 2017) dataset under two settings: full-sequence input and streaming input, shown in Table 14. "Time" denotes the inference latency of the final frame 40-frame stream. The results indicate that cached memory tokens not only accelerate streaming inference but also preserve temporal consistency and perceptual accuracy.
More Visualizations. Although some our reconstruction results are presented in the main paper, provide additional visualizations of the reconstruction outcomes of Stream and
16

---

Preprint. Under review.
Table 12: Results on 7-Scenes when the video stream is interrupted.
Method
Type
Acc Mean.
Acc Med.u
Comp Meanj
Comp Med.j.
NC Meant
NC Med.f
CUT3R (Wang 2025b)
Streaming
0.042
0.024
0.037
0.016 0.012
0.702 0.724
0.808
Stream
Streaming
0.036
0.020
0.033
0,833
Table 13: The robustness performance under noisy frames.
Method
Acc Mean]
Acc Med.
Comp Mean
Comp Med.
NC Mcant
NC Med.1
noise
0.034
0.007
0.666
0.760
10% noise
0.041
0.025
0.018
0.007
0.666
0.761
25% noise
0.054
0.031
0.023
0.006
0.655
0.747
Table 14: Effects of the cached memory token.
Method
Input Type
Acc.j
Comp-4
Overall
Time
VGGT VGGT (W/ Causal Attention) Stream VGGT (W Cached Token)
Full sequence
0.782 0.782
0.723
0.753
x
Streaming
0.723
0.753
0.07
GT
Ours
CUT3R
Figure 7: Visualization of the reconstruction results of Stream VGGT and CUT3R on 7-Scnnes.
Figure 8: Visualizations of failure cases.
CUT3R (Wang et al., 2025b) on the 7-Scenes (Shotton al., 2013) dataset. We compare our results of 3D reconstruction with CUT3R and the ground truth in Figure 7, demonstrating that our method achieves high-accuracy 3D reconstruction with fewer outliers and superior spatial consistency.
Failure Cases. We provide visualizations of some failure cases in Figure 8. These include instances of severe dynamic occlusion and aggressive ego-motion, which cause drift localization failure.
Visualizations of Pose Estimation. We provide visualizations of pose estimation on ScanNet (Dai al., 2017) in Figure and Figure 10. The visualizations demonstrate that our Stream can accurately estimate camera poses across diverse scenarios and performs better than CUT3R (Wang et al., 2025b) when estimating the trajectories of camera pose.
More Quantitative Results. We provide comprehensive comparison between StreamVGGT and concurrent work STream3R (Lan al., 2025) in Table 15, Table 16 and Table 17. Compared with
17

---

Preprint. Under review.
Stream V /GTT
CUT3R
2.3
2.3
Scene0724
2.2
2.2
2.1
2.1
2.0
2.0
1.8
0.8
1.0
1.2
1.4
1.8
0.8
1.0
1.4
1.5
1.8
y(m)
y(m)
7.8
7.8
7.6
7.5
7.4
Scene0708
7.2
7.2
7.0
7.0
6.8
2,0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
v(m)
v(m)
3.50
3.50
Prodooed
Scene0712
E
3.40
3,40
3.35
3.30
3.30
3.25
1.4
1.5
1.6
1.7
1.8
1.9
155
166
1.7
1.8
1.9
vim)
y(m)
Figure 9: Visualizations of the pose estimation of StreamVGGT and CUT3R Scannet.
Figure 10: Visualizations of point map and camera pose estimation of StreamVGGT
STream3R, which was trained on 29 datasets using 8 NVIDIA A100 GPUs over seven days, Stream achieves comparable results under constrained training resources by using only 13 training datasets and 4 A800 GPUs over the same seven-day period. This fully demonstrates the effectiveness of our knowledge distillation strategy, which not only accelerates the training process but also significantly reduces the demand for training resources.
18

---

Preprint. Under review.
Table 15: Quantitative 3D reconstruction results on 7-Scenes and NRGBD datasets.
scenes
NRGBD
Acc|
Compá
NC1
Acc|
Compl
NC1
Method
Type
Mcan
Mean
STream3R (Lan et al., 2025) Streaming 0.119
0.110
0.031
0.747
0.854
0.068
0.016
0.033
0.014
0.904
0.981
StreamVGGT
Streaming
0.129
0.056
0.115
0,038
0.751
0.865
0.084
0.044
0.074
0.041
0.861
0.986
Table 16: Single-Frame Depth Evaluation.
Sintel
Bonn
KITTI
NYU-12 (Static)
Method
Type
Abs Rel
1,25
Abs Rel
6x11.25
Rel 4. 6<1.25
Abs Rel 6<1.25
STream3R (I al., 2025) Streaming
0.229
70.7
0.061
96.7
0.064
95.5
0.057
95.7
VGGT
Streaming
0.254
68.5
0.052
97.1
0.072
94.7
0.055
95,9
Table 17: Quantitative 4D reconstruction results on TUM- TUM-dynamics.
Method
Type
Ace Mean..
Ace Med.j
Comp Mean..
Comp Med.
NC Meunf
NC Med.
STream3R
2025)
Streaming
0.012
0.007 0.007
0.665
StreamVGGT
Streaming
0.085
0,011
0.058
0.617
0.690
A.3 FURTHER DISCUSSIONS
Limitations. Although our cached token memory mechanism effectively retains historical frame information, this approach leads to substantial increase memory usage and computational over- head for long-term sequences shown in Figure 2. As more frames are processed, the memory footprint grows rapidly due to accumulation cached tokens. This scalability issue poses significant challenge for deploying the model on lightweight or mobile devices, where hardware resources are limited. Therefore, addressing memory efficiency while preserving accuracy remains critical area for future optimization.
Broader Impacts. The Stream VGGT model offers significantly broader impacts across multiple in- dustries, especially in the domain of low-latency 3D visual geometry reconstruction. By integrating temporal attention with a Cached Token Memory mechanism, Stream VGGT enables efficient incre- mental processing of multi-view images, ensuring low-latency scene updates with minimal latency while maintaining long-term spatial consistency. This innovative approach makes it crucial tech- nology for dynamic applications such as autonomous navigation, robotics, and immersive AR/VR experiences, where timely and accurate 3D scene understanding is essential.
19


Reviews:
[{"review_id": "XvXFPfuvQB", "reviewer": "Reviewer_KPdZ", "summary": "This paper introduces StreamVGGT, a transformer architecture for streaming 3D visual geometry reconstruction. The model draws inspiration from autoregressive language models and replaces global self-attention with temporal causal attention, enabling incremental 3D reconstruction with cached token memory. StreamVGGT supports online inference while maintaining spatial and temporal consistency. The approach achieves comparable reconstruction and depth estimation performance to offline methods (e.g., VGGT) while substantially improving inference speed on long sequences.", "questions": "* How does the model handle error accumulation or drift over long sequences (e.g., hundreds or thousands of frames)?\n* Does the cached token memory ever saturate or require pruning? If so, how is this managed without degrading accuracy?\n* Can the causal model generalize beyond static datasets (e.g., driving, dynamic human motion)?\n* How robust is StreamVGGT to missing or noisy frames, which are common in real streaming data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper addresses a practical problem: enabling low-latency 3D reconstruction suitable for real-time streaming systems.\n* The temporal causal attention and cached memory are conceptually simple, computationally efficient, and align well with online mapping needs.\n* The experimental evaluation is extensive, covering multiple datasets and comparing against both dense view and streaming baselines.\n* The writing is clear, structured, and easy to follow.\n* The paper demonstrates significant latency reduction (e.g., 10× faster inference for long sequences) while maintaining similar accuracy compared to VGGT.", "weaknesses": "* The issue of error accumulation over long sequences is not thoroughly analyzed. No explicit mechanisms (e.g., drift correction) are introduced beyond distillation, and the paper does not present accuracy trends across different sequence lengths compared to VGGT.\n* The scalability and memory-growth behavior of cached tokens for very long video sequences remains unclear.\n* The connection to causal or autoregressive modeling is somewhat superficial: there is no explicit probabilistic formulation or next-frame prediction, only causal masking.\n* The paper does not explore qualitative failure cases, such as dynamic occlusions or fast ego-motion, which are critical for practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgybeTjGBq", "reviewer": "Reviewer_XfbV", "summary": "This paper proposes StreamVGGT, a causal transformer architecture designed for real-time streaming 3D visual geometry reconstruction. Inspired by the success of large language models with causal attention, the authors integrate temporal attention and a cached token memory mechanism to enable efficient, incremental 3D scene updates without needing to reprocess full sequences. Extensive experiments on multiple 3D reconstruction benchmarks demonstrate that Stream3R achieves competitive performance.", "questions": "1. Since there are already many similar works (e.g., Stream3R, Lan et al.) that also adopt caching techniques, it would be beneficial if the authors could include a comparison with these approaches in the paper.\n2. Adding a pose estimation visualization and a comparison with CUT3R would make the paper more complete.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This demonstrates that using cache token technology can significantly improve VGGT's inference performance with long video inputs.\n2. The technical details are clear and easy to follow.", "weaknesses": "1. The authors’ demo video showcases the model’s capability to handle scenes with significant motion. It would be helpful if the paper included more analysis or discussion regarding these dynamic scenarios.\n2. I don't quite follow the motivation behind using knowledge distillation (KD). First, why not directly train the model on real-world data? Is it because the dataset size and quality are insufficient compared to VGGT, leading to suboptimal results? Second, since VGGT cannot be trained on long video sequences due to the computational cost of attention, wouldn't this limitation of the teacher model affect StreamVGGT’s performance on long sequences—especially for tasks like pose estimation?\n3. Figure 2 showcases the time efficiency advantage over the vanilla VGGT. However, since the authors emphasize the streaming nature of their approach, it would be more convincing if they could also compare the overall reconstruction time with other similar methods, such as CUT3R.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U8Cot1ATg0", "reviewer": "Reviewer_adCa", "summary": "This paper proposes StreamVGGT, which reconstructs 3D geometry from videos in an online manner. They do this by adopting a causal transformer architecture and caching the historical keys and values to use for each incremental new frame. They train StreamVGGT through knowledge distillation of VGGT, which is state-of-the-art, \"offline\", but slower. For 3D reconstruction, their approach increases inference speed relative to offline methods, but with a small performance reduction. They are competitive with (and sometimes slightly better than) other streaming methods.", "questions": "1. To what extent is the inference speedup and memory reduction (Figure 2) due to memory token caching versus FlashAttention-2?\n2. I think it would be helpful to discuss: what are the different methods (e.g., StreamVGGT, CUT3R) good at and where do they fail? How is this linked to their method (architecture, training strategy, training data, etc.)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper's approach of caching memory tokens for streaming reconstruction is important for efficiency, and it is useful for the field to see the results and comparisons they report. I think this makes the paper useful and worth publishing. \n2. The paper is written clearly and provides a very readable explanation of their method and other methods. \n3. StreamVGGT is competitive with other streaming methods, while adopting a significantly different approach. The additional experiments (inference speed, memory, distillation ablations) also serve as useful information.", "weaknesses": "1. It would be useful to explicitly compare the inference (and training, if possible, but understandably this may be harder) speed and memory consumption of their approach against other streaming reconstruction methods like CUT3R and Spann3R. I think this is crucial because the core argument of their paper is efficiency.\n2. \"StreamVGGT leverages the inherent sequential and causal nature of real-world video data, constraining the attention mechanism to past and current frames, thereby aligning with the causal structure observed in human perception.\" I agree that humans receive input frames in an online streaming manner. However, do we know if there is evidence that their representation of old views is not modified by (and so does not \"attend to\") new views, like in KV caching? One counterpoint is that our stored experiences and memories are modified by new information. It is possible that humans use some hybrid: KV-caching where stored keys/values are allowed to be modified. At a few points in this paper, the paper implicitly suggests that StreamVGGT is more human-like than full self-attention approaches. I think this requires more evidence, or otherwise hedge the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9andtlyBGo", "reviewer": "Reviewer_6AuX", "summary": "The paper proposes an online, low-latency 3D reconstruction pipeline. It achieves this by (1) temporal causal attention, cross-attending to cached KV-tokens from previously observed frames, and (2) utilizing knowledge distillation during training by setting VGGT (a full SA-based reconstruction model) as the teacher to provide the GT for the causal student model.", "questions": "1. Did the authors explore strategies for constraining/truncating the memory usage for very long scenes? Perhaps simply incorporating a recency bias i.e. cacheing and cross-attending to only the most recent K scenes' tokens instead. An ablation over different K can be helpful.\n2. Can the authors explain the results reported in Figure 6?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow. The problem is well motivated and explained.\n2. The proposed causal attention is simple but effective, and performs well empirically.", "weaknesses": "1. In Table 6, the impact of KD is rather drastic, which makes the contribution of the causal attention questionable. When reporting results for w/o KD, is the 'true' GT used as supervision? The authors claim that KD helps reduce error accumulation, but this claim needs a more involved analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)