FastVGGT: Fast Visual Geometry Transformer
Authors: You Shen
Zhipeng Zhang
Yansong Qu
Xiawu Zheng
Jiayi Ji
Shengchuan Zhang
Liujuan Cao
Venue: ICLR 2026 Poster (2026)
Keywords: 3D reconstruction
Primary area: applications to robotics, autonomy, planning
TL;DR: FastVGGT observes strong similarity in VGGT's attention maps and leverages training-free acceleration.

Abstract: Scaling visual geometry transformers for long image sequences poses a significant computational and memory challenge. In this work, we diagnose this issue in the state-of-the-art model VGGT, and trace the primary bottleneck to its Global Attention layer. Our analysis reveals a ``token collapse'' phenomenon, where many tokens attend to nearly identical regions, resulting in redundant computation and inefficiency. Motivated by this finding, we propose FastVGGT, a training-free framework that strategically prunes these redundant tokens. Instead of uniform merging, FastVGGT employs a tailored, three-part token partitioning strategy. It preserves initial-frame tokens as a stable global reference, retains salient tokens to maintain fine details, and utilizes region-based random sampling to ensure spatially balanced coverage. Extensive experiments on multiple 3D geometry benchmarks validate our approach's effectiveness. Notably, on sequences of 1000 images, FastVGGT achieves a 4$\times$ speedup over the original VGGT while simultaneously mitigating error accumulation, demonstrating its efficiency and robustness for long-sequence scenarios. For further details, please visit our project page: https://mystorm16.github.io/fastvggt/.

Published as a conference paper at ICLR 2026
FASTVGGT: FAST VISUAL GEOMETRY TRANSFORMER
You Shen', Zhipeng Zhang", Yansong Qu' Xiawu Zheng', Jiayi Ji', Shengchuan Zhang'. Liujuan Cao
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University
Schooo of Artificial Intelligence, Shanghai Jiao Tong University
Raw Taken
Sre Taken
Oot Token
Single forward with 1000 images (4 x speedup)
Fast-VGGT: O((N/r)3. d)
VOST
000
voe!" RasieSOT
A Single Pass
VOGT DOM
Merge
Self-Attn of
Unmerge
100
200
400
500
700
800
1000
#Framee
VS
VGGT: O(N d)
Fast-VOGT CD: 0.27
VGGT CD: 0.71
Self-Attn
We propose FastVGGT, training-free free framework that processes 1,000 images in a sin- gle inference, achieving faster while mitigating error accumulation. VGGT refers VRAM- Efficient of VGGT, enabling larger inputs.
Figure 1:
ABSTRACT
Scaling visual geometry transformers for long image sequences poses a significant computational and memory challenge. In this work, we diagnose this issue in the state-of-the-ant model VGGT, and trace the primary bottleneck its Global At- tention layer. Our analysis reveals "token collapse" phenomenon, where many tokens attend nearly identical regions, resulting in redundant computation and inefficiency. Motivated by this finding, we propose FastVGGT, training-fre framework that strategically prunes these redundant tokens. Instead of uniform merging, FastVGGT employs a tailored, three-part token partitioning strategy. It preserves initial-frame tokens as a stable global reference, retains salient tokens maintain fine details, and utilizes aegion-based random sampling to ensure spa- tially balanced coverage. Extensive experiments on multiple 3D geometry bench- marks validate our approach's effectiveness. Notably, sequences of 1000 im- ages, FastVGGT achieves a 4x speedup over the original VGGT while simultane- ously mitigating error accumulation, demonstrating efficiency and robustness for long-sequence scenarios. For further details, please visit our project page: https://mystorm/ wwt.:/..b.ioo/aaaatteeeeeeeeeeee
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Inferring the 3D geometric structure scene from visual inputs, is critical for enabling machines
to understand and interact with the physical world. Recent advances in deep learning have catalyzed
a paradigm shift in 3D geometric estimation (Wang et al. 2024b /hhng et al.] |025] Qu al. [023]
Wang ct al. 2025a: Qu et al. 2025 Lu et al. 2025 Ye al. 2024 Pan et al. [2024] Brachmann
ct al. 2024
Sun et al. 2021 Wang et al. 2025c] Lahoud et al., [022 Zou et al. 2024 Fu et al,
2024). enabling
from iterative, optimization-baser pipelines end-to-end neural networks that directly infer geometry from raw visual inputs. This transformation is exemplified by large-scale
architectures like DUS(3R (Wang al.] 2024b) and its follow-ups (Yang et al., 2025 Tang al.,
2025 Leroy et al. 2024 Chen et al. 2024 Wang & Agapito 2024). which showcase remarkable
capacity to reason about complex geometric relationships across image pairs.
Building upon this line of research, VGGT (Wang et al., 2025a) marks a significant advance. Its transformer-based, feed-forward architecture directly regresses key 3D attributes, including camera parameters, depth maps, and point tracks, to achieve highly stable and accurate reconstructions. While this establishes VGGT as a otate-of-the-art framework for 3D scene understanding, its scal- ability is impeded by two critical bottlenecks. First, the model's 's reliance on dense global token interactions across views or frames results in prohibitive computational costs. Although used tech- niques like Flash-Attention mitigate the memory complexity from (nn²) to O(nd), the underlying time complexity remains quadratic at dn) Second, the global attention mechanism, essential for capturing inter-frame relations, is susceptible to eπrоr accumulation. As the token space expands with each new frame, minor inaccuracies are amplified, leading to significant prediction drift. Col- lectively, these limitations restrict VGGT's applicability in large-scale scenarios and motivate the development of more efficient and scalable architectures.
To pinpoint the primary inference bottlenecks in VGGT, we first conducted detailed, component- wise performance analysis, as illustrated in Figure 2 The analysis reveals that while the compu- tational costs of "Frame Attention" (intra-frame interaction) and "Global Attention" (cross-frame interaction) are comparable for short sequences, the cost of Global Attention escalates rapidly with sequence length, eventually dominating the entire runtime profile. This finding motivates our cen- tral research question: Can the computational inefficiency of Global Attention be mitigated without compromising VGGT"s capability? To investigate this possibility, we visualized the attention maps in Figure. 3 which uncovered crucial insight that attention patterns across tokens exhibit a high degree of similarity, indicating substantial redundancy in the global computation.
Motivated by the observation of attention redundancy, adapt the training-free technique of token merging (Bolya et al., [2022] [Bolya Hoffman| 2023) Tran et al.] [0024] Lee al.] [024]] Cao al. 2023 Zeng et al. [222 Renggli et al. [0222] Li et al. [024]] Choi et al. 2024 Feng & Zhang 2023
to enhance VGGT's inference efficiency. Token merging consolidates redundant representations by partitioning tokens into source (sre) and destination (dst) sets and merging each src token into its
most similar dst counterpart. While effective in 2D vision tasks, its extension to structures designed
Frame Attention
Global Attention
Decoder
100
26152ms
Inference time (%)
2807mme
60
40
438ms
415ms
680ms
205ms
310ms
1138ms
450ms
2128ms
760ms
20 Frames
50 Frames
100 Frames
200 Frames
Figure 2: Component- analysis of VGGT inference time. As the number of input frames grows, the Global Attention module increasingly dominates the computational cost.
2

---

Published as a conference paper at ICLR 2026
for 3D geometry understanding remains underexplored. Unlike 2D settings that process single im- ages, VGGT relies on cross-image correspondences, making direct application of token merging highly challenging. To address this, introduce FastVGGT, novel, training-free framework that strategically applies token merging to mitigate the Global Attention bottleneck. Our approach be- gins by preserving the foundational coordinate system. Specifically, tokens from the initial frame, which serves as the global reference for the entire scene, are designated as high-priority destination (dst) tokens and are exempt from being merged ensure reconstruction stability, Furthermore, to maintain global consistency and preserve fine- grainerrinddd details, we identify and retain the most salient tokens across all frames, allowing them bypass the merging process entirely and participate di- rectly in the attention computation. Finally, inspired by ToMeSD (Bolya & Hoffman] (023), we implement region-basee random sampling within each subsequent frame. This ensures a spatially balanced selection of src and dst tokens, preventing critical information loss in localized regions during consolidation. is worth emphasizing that although 2D token merging also relies on simi- larity cues, token collapse in VGGT reveals much higher redundancy, allowing the model to safely adopt much more aggressive merging ratios and achieve greater speedups. For instance, ToMeSD, which is designed for 2D generative tasks, degrades noticeably once the merging ratio exceeds 0.3, whereas VGG maintains baseline-level performance even at a ratio of 0.9.
Our experiments demonstrate that this integrated approach allows FastVGGT to significantly reduce the computational overhead of Global Attention. For large-scale inputs of 1000 images, it achieves 4x inference speedup over the baseline VGGT while simultancously mitigating error accumulation in long-sequence reconstructions. Notably, the original VGGT suffers from prohibitive memory consumption, leading to Out-of-Memory (OOM) errors when processing sequences beyond 300 images. Through VRAM optimization, our modified VGGT successfully handles inputs of over 1000 images, demonstrating substantial improvement scalability.
In summary, our main contributions are as follows: 1) We identify and analyze the key bottleneck that limits VGGT's ability to process long sequences. 2) We uncover token collapse phenomenon in VGGT and introduce specialized merging strategy that exploits this redundancy for accelera- tion. 3) Extensive experiments demonstrate that our method significantly accelcrates VGGT on long sequences while preserving reconstruction quality. 4) For sequences exceeding 500 frames, our entirely training-free method surpasses the baseline by effectively mitigating error accumulation.
2 RELATED WORK
2.1
3D RECONSTRUCTION
Building on the foundations of traditional 3D reconstruction (Mur-Artal et al. 2015 Schonberger
& Frahm 2016 Mur-Artal Tardós (017), recent end-to-end learning- based methods (Wu et al.,
Wang et al.) 2024al 2023| Zhang et al. 2024a| Chen et al. 2025 ZZhng et al. 2024b Fan jet al. 2024, Lei al. 2025] [maart et al., 2024 Wang et al., 2025b Yu et al. 2024) leverage neural networks to encode scene priors, substantially enhancing robustness and cross-dataset gen- cralization. Early progress was marked by DUS13R (Wang et al. 2024b), which directly regresses view-consistent 3D point maps from only two RGB images without requiring camera calibration. The current oftte--f-hhe-rr,, VGGT (Wang et al.) 2025a) scales this philosophy to a 1.2B-paramen transformer that jointly predicts camera intrinsies, extrinsics, dense depth, point maps, and 2D tracks. However, as input sequences grow longer, the global attention mechanism must capture inter-frame relations within expanding token space. This not only increases computational over- head but also amplifies noise propagation, making long-sequence predictions more prone to drift. VGGT-Long (Deng et al. 2025) addresses the drifting issue by aligning sub-maps to suppress error accumulation, but at the cost of significantly reduced inference speed, undermining the efficiency of feed-forward 3D reconstruction. To overcome these challenges, we propose Fast which accelerates inference and mitigates error accumulation by reducing the number of tokens processed in Global Attention, thereby achieving balance between efficiency and accuracy.
2023
2.2 TOKEN MERGING
Visual token merging (Bolya al. 2022 Renggli et al., 2022 Zeng et al. 2022 Haurum et al. 2023)
was initially proposed freee technique to increase the throughput of Vision Transformers
3

---

Published as a conference paper at ICLR 2026
Taken 0
Taken 500
Taken 1041
Taken 1541
Taken 10410
Token 10910
Block 0
Block 11
Block
Block
Figure 3: Visualizations of the Global Attention maps in VGGT, using six representative tokens (including the camera token and several image tokens), show that at every stage the attention patterns of different tokens exhibit strong degree of similarity.
(ViT') (Cao et al. 2023 Choi et al. 2024 Shen al. 2024). These methods typically partition to- kens into two groups, match each token with its nearest neighbor, merge them via average pooling, and concatenate the resulting set. Related efforts reduce token redundancy through adaptive selec- tion, as in TokenLearnee (Ryoo al.) 2021), modular superpixel tokens in A Spitting Image (Aasan al., (024), or clustering-based compression such as Agglomerative Token Clustering (Haurum al. 2024h. PiToMe Wang al. (2025d) further proposes energy-score score-bebr- criterion for de- ciding whether a region should be merged. Complementary to token-level compression, another line of work accelerates attention itself through kernel-based approximations. Nyströmformer (Xiong jet al. 2021). inspired by the classical Nyström method (Williams & Seeger (000). approximates self-attention using landmark tokens, while Performer (Choromanski al. (0200) employs random feature mappings for linear-time attention. Although simple and effective, most applications remain limited to the image domain, while long-form video remains underexplored despite its spatiotem- poral tokens being both redundant and interdependent. Targeting feed-forward 3D reconstruction, we propose Fast which adapts token merging to VGGT and delivers significant acceleration without compromising reconstruction quality.
METHOD
3.1 VISUALIZATION ANALYSIS
We highlight an observation that motivates our optimization of the Global Attention module. As shown in Figure 3 we visualize VGGT's Global Attention maps on the ScanNet dataset. Each image represented by 1,041 tokens (one camera token, four register tokens, and 1,036 patch tokens from a 28 x 37 grid). The dense self-attention mechanism generates an attention map for every token, and visualizations across tokens and blocks reveal that many of these maps are highly similar.
The phenomenon of attention similarity, often referred to feature degradation, has been consis- tently reported in the DINO series (Oquab et al. 2023 Siméoni et al.) 20025) Caron et al. 2021 In DINO, while the CLS token becomes increasingly discriminative, patch-level features gradually lose local consistency as they converge toward the CLS token, which undermines performance on dense prediction tasks. A similar limitation appears in VGGT: its global self-attention layers ag- gregate information across all tokens via weighted averaging, and without explicit regularization task-specific constraints, the representational space is repeatedly compressed. This compression
4

---

Published as a conference paper at ICLR 2026
Global
Frame tokens
Camera
Tokanization
KOMDN
Merging
AMI
Head
-Atti
Depth
Unmerging
Head
Comora
Tokon
DODMO
Token
Sin Token
Decouple Sro from Oer for cence precicton
tokeea LMG, sarving the global neterencc.
Saklett takera for comeapondaace
rnd length in attentioonn
belanoed ention
Figure 4: Overview of the proposed token merging strategy. The pipeline begins with tokenization of input frames. To alleviate the Global Attention bottleneck, we design five-step procedure:
(1) fix initial-frame tokens destination (Dst) tokens, serving the global reference for spatial consistency, (2) retain the top-k salient tokens to strengthen correspondences, (3) apply region- based random sampling for balanced selection, (4) fuse source (Src) tokens into their nearest Dst tokens during attention, and (5) decouple tokens via unmerging for dense reconstruction. G-Attn and F-Attn denote Global Attention and Frame Attention, respectively.
drives tokens collapse along a dominant direction, eroding their distinctiveness. At fundamen- tal level, both DINO and VGGT lack explicit mechanisms to preserve token diversity, leading to progressive loss of local variation.
is important to note that the consequences of feature degradation differ markedly between DINO and VGGT. In DINO, the network must handle not only image classification but also dense pre- diction tasks such as segmentation, where local feature integrity critical. As training progresses, patch tokens collapse toward the CLS token, weakening dense feature discrimination and directly impairing downstream performance. This motivates corrective strategies such as Gram Anchor- ing (Siméoni al., 2025). By contrast, VGGT follows two-stage design: Global Attention and Frame Attention. Global Attention enforces global consistency by capturing holistic spa- tial-temporal relationships. Here, the apparent degradation, in which tokens collapse toward dominant subspace, can be interpreted not as flaw but as a deliberate distillation of global se- mantics. Frame Attention subsequently reintroduces local variability, achieving balance between global abstraction and local differentiation. At the same time, the strong feature similarity observed Global Attention reveals redundancy that can be leveraged reduce computational cost without compromising performance, thus offering natural solution to VGGT's speed bottleneck.
Motivated by the strong similarities ob- served in Global Attention maps, we pro- pose FastVGGT, which accelerates infer- ence via token merging. Conventional strategies typically partition tokens into destination (dst) and source (src) sets through random or fixed-stride sampling. To assess their suitability, tested both the random-sampling and fixed-stride vari- ants of ToMe Bolya al.] (2022). as well as the more advanced PiToMe Wang al. (2025d), which proposes an energy- score-based criterion for deciding whether a region should be merged. All meth- ods were evaluated under three different merging ratios. As shown in Table [ these approaches reduce inference time but severely degrade the Chamfer Dis- tance, thereby harming reconstruction ac-
Table 1: Results of applying existing merging strate- gies. ToMe-S denotes the fixed-stride version of ToMe, and ToMe-R denotes the random random-sampling version.
300 Frames
100 Frames
Method
Ratio
CD J
Time 1
CD CD.
Time
ToMe-R
0.451
7.%s
ToMo-S
0.3
0.633
98.4s
0.548
7.9s
PTToMe
0.454
112.9%
0.450
8.7s
ToMe-R
ToMe-S
0.6
0.563
49.3x
0.523
6.4s
PYToMe
0.489
61.6s
0.467
7.7s
ToMe-R
0,582
22.8s
0.494
5.0s
ToMe S
0.9
0.708
21.2s
0.628
4.8
PYToMe
0.567
37.5s
0.522
7.2s
VOGT
0.416
131.4s
0.423
9.1s
5

---

Published as a conference paper at ICLR 2026
euracy and failing to preserve VGGT's strong performance. This demonstrates that directly applying merging methods designed for 2D tasks to VGGT is non-trivial. To overcome this, we design three tailored strategies for feed-fore forward 3D reconstruction that preserves accuracy, accelerates inference, and mitigates error accumulation in long-sequence scenarios.
3.2 TOKEN PARTITIONING
In contrast to token merging methods primarily developed for 2D vision tasks, which typically op- erate individual images, feed-forward 3D reconstruction models process multi-view sequences with varying degrees of overlap across viewpoints. Such overlaps are critical for accurate 3D re- construction, as they provide the cross-view correspondences necessary for recovering consistent geometry. Therefore, to achieve acceleration through merging while preserving reconstruction fi- delity, it becomes essential to design an effective token partitioning strategy that operates across all frames in the sequence.
The design of our token partitioning strategy is govemned by three principles: 1) preserving cross- frame correspondence to ensure structural consistency of the overall scene; 2) ensuring uniform merging within each frame to prevent both over-compression and redundancy; and 3) merging the most redundant tokens into the most representative ones maximize acceleration efficiency. Ac- cordingly, we divide tokens into three categories: salient tokens, which capture the most distinctive features of each frame; destination (dst) tokens, which act representative anchors; and source (src) tokens, which correspond to redundant information to be merged. Based on these principles, design three token partitioning strategies detailed in the following.
Reference Token Selection. First, VGGT defines the first frame as the world coordinate system, with all tokens registered relative to this reference. This design makes the first frame key anchor for maintaining spatial consistency across the sequence. Supplementary visualization of attention maps further shows that tokens consistently exhibit stronger activations toward the first frame than any other, highlighting its central role in guiding scene-level representations. Consequently, we designate all tokens from the first frame as dst tokens due their strong representativeness.
Salient Token Selection. Second, since VGGT reconstructs scenes through cross-frame frame token in- teractions, subset of key tokens critical for establishing reliable correspondences across views. As illustrated in Figure these tokens resemble distinctive keypoints in traditional matching algo- rithms. To preserve them, we extend conventional token merging by introducing a third category, dividing tokens into salient, dst, and src groups. Salient tokens are excluded from merging opera- tions and instead participate directly in attention. For selecting salient tokens, we first apply a top-k strategy based on token norms to measure distinctiveness. However, its computational overhead grows with longer sequences. To improve efficiency, we adopt a fixed-stride sampling scheme that retains 10% of tokens per frame salient tokens. Experiments show that this achieves accuracy comparable to top-k selection while greatly reducing cost. We therefore use fixed-stride sampling the default strategy, balancing efficiency and effectiveness.
To further analyze the effects of the two strategies, we include additional experiments in Appendix E.2 that visualize the spatial distribution of salient tokens. We observe that top-K selection performs well in shallow blocks by accurately capturing semantically meaningful regions, but as the block depth increases, the selected tokens become increasingly concentrated. In contrast, although the fixed-stride strategy is less precise in locating key tokens, it consistently maintains a uniform spatial distribution across blocks.
Uniform Token Sampling, Finally, recognizing the dense prediction nature of 3D reconstruction, we ensure uniform intra-frame sampling to avoid local over-compression or redundancy. To this end, we assign dst and src tokens within each frame using region- based random sampling strategy inspired by ToMeSD (Bolya & Hoffman 2023) in diffusion models. Concretely, we first partition the input tokens by frame and arrange them into 2D grid of image patches. Within each grid cell, dst tokens are sampled according to a predefined merging ratio with stride K, while the remaining tokens. This region- based strategy ensures spatially balanced merging
tokens are designated and prevents artifacts such as the disappearance of large areas. Consequently, the merging process more coherent, and the reconstructed scene better preserves global structural stability.

---

Published as a conference paper at ICLR 2026
3.3 MERGING AND UNMERGING PROCEDURE
After token partitioning, sre tokens are merged into their most similar counterparts in dst. Formally. given a token representation x, we compute cosine similarity between each I, E sre and all za E dst as sim Fu) = I|,, E,-EA | Each source token I, is then assigned to its most similar destination token Fd, and updated by averaging 2 = 2 The updated z'j is retained while 2, is temporarily discarded, thereby reducing the number of tokens processed in attention computations.
Dense 3D reconstruction requires per-token outputs. To satisfy this requirement, we adopt an un- merging operation, inspired by ToMeSD (Bolya & Hoftman, (023), which restores the original token resolution and maintains full compatibility with the VGGT architecture. Specifically, suppose two tokens 1 22 æ2 Rs are merged into a single representation i -1_32 During unmerging. this representation is replicated recover the original sequence length x = æi,2 and 2/2 xi.2-
3.4 VRAM-EFFICIENT IMPLEMENTATION
In tests, the original VGGT encounters out-of-memory errors when processing sequences of around 300 frames. VGGT consists of 24 encoder blocks, but during inference only the outputs of layers 4, 11. 17, and 23 are required. Nevertheless, the original implementation stores intermediate results from all 24 blocks. To support longer input sequences, we introduce an simple optimized vari- ant, VGGT", which discards unused intermediate outputs during inference. This reduces memory consumption and enables processing of up to 1000 frames without affecting reconstruction quality. Figure 7 reports the GPU memory comparison between the original VGGT and VGGT* Unless otherwise specified, all experiments in this paper use VGGT* as the baseline.
We clarify that although 2D merggng methods reduce memory consumption by decreasing the number of attended tokens, this mechanism does not transfer effectively to VGGT. VGGT exhibits no meaningful token similarity within its frame-attention layers, and its alternating global-frame attention structure requires merged tokens to be fully restored before each frame-attention block when performing dense reconstruction, resulting in no reduction in the token count.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate FastVGGT on three benchmark datasets: ScanNet, NRGBD, and 7 Scenes, with the best results highlighted in bold and the second-best results underlined. For ScanNet, which con- tains 1,500 scenes, we uniformly sample 50 scenes to form reproducible benchmark, denoted as ScanNet-50. Our experiments focus two tasks: camera pose estimation and point map reconstruc- tion. Across both tasks, FastVGGT consistently achieves substantial speedups while maintaining accuracy. The overall architecture follows VGGT, comprising L=24 frame and global attention lay- ers, with Flash-Attentin22 (Dao) 2023) integrated to further accelerate inference. All input images follow VGGT's preprocessing, where the long side is resized to 518. To keep the method simple and efficient, we use a fixed merging ratio of 0.9 in Tables 7 which is sufficient to achieve substantial acceleration while maintaining performance comparable to the baseline. We further analyze the im- pact of different merging ratios in Table & and in the Appendix. All experiments are conducted on a workstation with an NVIDIA A800 GPU (80 GiB VRAM).
4.2 3D RECONSTRUCTION
Comparisons on the ScanNet-50 Dataset. We begin by evaluating FastVGGT on the ScanNet-50 dataset, reporting reconstruction quality using Chamfer Distance (CD). Experiments are conducted with input sequences of 1000, 500, and 100 images, enabling us to assess performance under vary- ing sequence lengths. The results in Table 2 show that although SOTA methods such as T' and Stream achieve strong performance on short sequences, they fail on long sequences due to memory constraints. Methods like Fast3R Yang et al. 2025) and CUT3R (Wang et al., 2025b) can process long sequences efficiently, but their reconstruction quality degrades severely. In contrast, FastVGGT delivers substantial acceleration over baseline VGGT across all settings while preserv-
7

---

Published as a conference paper at ICLR 2026
Table 2: Quantitative results of point cloud reconstruction the ScanNet-50 dataset with input sequences of 1000, 500, 300, and 100 images. 0OM denotes out-of-memory.
1000
500
300
100
Method
Time 4
CD 4
4
CD 4
Time
Time
Wang al.
2025d
0OM
OOM
0OM
00M
OOM
00M
00M
0OM
Stream
(Zuu et al.
2025
0OM
OOM
00M
0OM
OOM
00M
00M
0OM
Fast3R
Yang et al.
2025
0.684
397.8s
0.701
97.3s
0.711
34.9
0,723
4.8
CUT3R
Wang
2025b
0,786
34.8s
0.774
18.8s
0.775
11.1s
0.767
3,66
VGGT
Wang
2025a
0.471
724.6s
0.420
177.5s
0.416
131.4s
0.423
9.1x
FastVGGT
0.425
180.7s
0.411
55.2s
0.416
23.8s
0.426
5.48
Table 3: Quantitative results of point cloud reconstruction on the 7 Scenes dataset. Stride denotes keyframes sampled every or frames.
Secnes
Stride
Socnes
Stride
Method
Acc
Comp
NC
Acc
Comp-
NC
Time
Time
Mean Med. Mean Med. Meun Med.
Mean Med.
Mean Med. Mean Med.
Wang et
StreamVGGT
Zhuo al.
2025
0OM
0OM
OOM
OOM
OOM
0OM
0OM
0OM
0OM
OOM
0OM
OOM
0OM
0OM
3R
Yang et
2025
0.045
0.027
0.047
0.010 0.616
0.627
43.7s
0.040
0.021
0.056
0.013
0.639
0.657
5.5%
CUTER Wang
0.179
14.5s
0.651
4.2s
VGGT
Wang al.
2025a
0.019
0.008
0.027
0.010
0.611
0.628
76.7s
0.020
0.008
0.027
0.010
0.623
0.641
8.7s
FastVGGT
0.018 0.008 0.026 0.010
0.617 0.634
28.0s
0.018
0.008
0.027 0.010
0.628
0.648
5.1s
ing reconstruction accuracy. Notably, when processing very long sequences (e.g., 1000 images), Fat only maintains reconstruction fidelity but also significantly mitigates error accumu- lation, demonstrating robustness and scalability for large-scale 3D reconstruction.
Comparisons on 7 Scenes and NRGBD Datasets. Following the CUT3R protocol, we evaluate FastVGGT on the 7 Scenes and NRGBD datasets. We report accuracy (Acc), completeness (Comp), and normal consistency (NC) using long-sequence inputs, with keyframes sampled every or 10 frames. As shown in Table 3 and Table 4 FastVGGT maintains the robust performance demon- strated on ScanNet-50, further demonstrating its effectiveness for 3D reconstruction.
4.3 CAMERA POSE ESTIMATION
We evaluate FastVGGT the ScanNet- 50 dataset for camera pose estimation. Evaluation is conducted using four monly adopted metrics: Absolute Tra- jectory Error (ATE), reflecting trajectory- level accuracy; Absolute Rotation Er- ror (ARE). reflecting orientation accuracy: Relative Pose Error in rotation (RPE-rot). and Relative Pose Error in translation (RPE-trans), which jointly characterize lo- cal frame-to-frame consistency. As illus- trated in 5 FastVGGT matches the baseline VGGT on shorter sequences (100 and 300 frames), while on longer sequences (500 and 1000 frames) substantially reduces pose estimation error. Fig- Further visualizes pose trajectories, demonstrating FastVGGT's ability to suppress cumulative drift, thereby underscoring its effectiveness and practicality for real-world deployment. Additional qualitative visualizations are provided in Appendix Figure
FastVGGT
VGGT
Figure 5: Comparison of pose estimation performance
between FastVGGT and VGGT.
8

---

Published as a conference paper at ICLR 2026
Table 4: Quantitative results of point cloud reconstruction on the NRGBD dataset.
NRGBD Stride
NRGBD Stride
Method
Acc
Comp 4
NC
Aoc
Comp-4
NC
Time
Time
Mean
Med. Mean Med.
Mean Med.
Mean Med. Mean Med.
Mean Med.
Wang et
2025d
0OM
0OM
OOM
0OM
0OM
StreamVGGT
Zhuo al.
2025
0OM OOM
OOM
OOM
OOM
0OM
00M
00M 0OM OOM
0OM OOM
0OM
00M
FR
Yang et al.
2025
0.074
0.036
0.024
0.011
0.658
0.682
68.9%
0.061
0.028
0.031
0.013
0.669
0.712
7.4s
CUTER
Wang et
0.346
0.184
0.579
18.3s
0.132
0.056
0.669
5.7s
VGGT
Wang et al.
2025a
0.029
0.019
0.018
0.009
0.727
0.728
136.1s
0.016
0.010
0.017
0.009
0.735
0.738
13.9%
FastVGTT
0.029
0.021 0.019
0.010
0.730
0.738
53.18
0.018
0.011
0.018 0.009 0.736
0.741
7.3g
Table Quantitative results of camera pose estimation on the ScanNet-50 dataset.
Input
ATE +
ARF L
RPF
trans
Frames
Baseline
Ours
Baseline
Ours
Baseline
Ours
Baseline
Ours
1000
0.196
4.636
3.860
0.997
0.667
0.029
500
0.174
3.591
0.963
0.031
300
0.145
0.142
3.689
3.554
0.786
0.801
0.040
0.036
100
0.140
0.141
3.625
3.512
1.224
1.262
0.058
0.061
4.4
ABLATION STUDIES
Token Partitioning. We evaluate the effectiveness of different token partitioning strategies using 500-frame inputs from ScanNet-50. As shown in Table 7: (a) directly selecting dst and src tokens through random sampling yields poor performance; (b) adopting region-based intra-frame uniform sampling improves performance but remains suboptimal; (c) designating the global reference frame (first frame) as dst tokens brings substantial gains; and (d) further protecting salient tokens leads to the best performance.
Location and Intensity of Merging. To balance accuracy and efficiency, we evaluate the effect of varying the starting block for merging and the applied merging ratio. As shown in Table [8] increasing the merging ratio consistently reduces inference time, with only minor fluctuations in
Chamfer Distance. Consequently, we adopt an aggressive strategy that applies 90% merging ratio from block O across all subsequent layers, yielding a favorable balance of accuracy and efficiency.
Salient token selection strategies. Table 6 we provide clearer comparison between the topK-based and fixed-stride sampling strategies for selecting salient tokens. On ScanNet-50, we evaluate both performance and efficiency, and additionally analyze how different top percentages of tokens selected from the scene affect the results. The findings show that the fixed-stride strategy noticeably improves efficiency while maintaining performance comparable to the top-k approach. Considering both accuracy and efficiency, we ultimately adopt the fixed-stride strategy that samples 10% of the scene tokens.
4.5 QUANTITATIVE ANALYSIS OF TOKEN COLLAPSE
We conduct experiments Figure using average cosine similarity to analyze VGGT's token- collapse phenomenon. The experiments are performed on ScanNet-50, following the same setup as Figure 3 Specifically, we compute the similarity metric as the average pairwise cosine similar- ity across six tokens (0, 500, 1041, 1541, 10410, 10910) sampled from different frames. Cosine similarity ranges from -1 to 1, where indicates highly similar aatennttttniinn patterns, 0 indicates uncorrelated patterns, and -1 indicates opposite patterns. The results reveal substantial similarity across most global-attention blocks, with noticeable drops occurring around Blocks and 14.
Interestingly, even in layers where token similarity is relatively low, applying an aggressive merging ratio (0.9) does not noticeably degrade the final performance, as shown in Tables 5 To bet- ter understand this behavior, refer to conclusions drawn from our attention-map visualizations:

---

Published as a conference paper at ICLR 2026
Global Sievlarity
1931
nonr Caess spelato
Irpatt
Figure 6: Average cosine similarity of global at- tention over blocks.
Figure 7: Comparison of GPU memory between the original VGGT and VGGT*.
Table 6: Quantitative results of salient token selection strategies on the ScanNer-50 dataset.
500
300
100
Method
CD
Time 1
CD 4
Time
CD
Time L
TopK-15%
0.421
80.3s
0.410
43.8s
0.432
7.5s
TopK-106 10%
0.423
73.8s
0.423
35.3s
0.428
6.7s
TopK-5%
0.438
68.1s
0.412
31.4s
0.445
6.1s
Stride-15%
0.429
Stride- 10%
0.411
55.2s
0.416
23.8s
0.426
5.4s
Stride-5%
0.425
52.56
0.427
22.4s
0.438
4.9s
Table 7: Ablation for partitioning strategies.
Table 8: Ablation on ratio and blocks.
Methods
Uniform
Reference
Salient
CD&
ATE
0.3
0.6
0.9
Blocks
0.947
0.842
CD
Time |
CD
Time
CD
Time |
0.637
0.722
0.408
119.8
0.415
64.38
0.41
55.2s
(c)
v
0.431
0.149
10
0.418
146.2s
0.424
118.46
0.431
106.3s
(d)
0.411
0.141
20
0.423
172.9s
0.427
169.1s
0.411
157.36
VGGT's global attention is fundamentally driven by feature matching, with high activation values concentrated around keypoint-like regions, as shown in Figures 12 Therefore, even though to- ken similarity is lower in the middle blocks, we still apply an aggressive merging ratio, effectively retaining only a subset of the candidate matches. Our experiments show that this partial selection long sequences. There is conceptual parallel
already sufficient to maintain strong performance to classical SfM pipelines, where sparse yet reliable keypoints are often sufficient for high-quality pose estimation. Prior work (Zhao & Vela 2020) has shown that increasing the number of matched keypoints does not necessarily improve performance; in fact, can even degrade estimation quality by introducing additional outliers and noisy correspondences.
CONCLUSION
In this work, we propose FastVGGT, a training-free approach that accelerates VGGT inference through token merging while preserving reconstruction quality. Analysis identifies the global tention module as the main bottleneck for long-sequence inputs, and visualizations reveal token collapse in attention maps, where tokens display strong similarity and redundancy. To address this, propose three token partitioning strategies tailored to visual geometry tasks. Experiments show that FastVGGT achieves up to a 4x speedup on 1,000-image inputs with competitive accuracy in estimation and 3D reconstruction, while mitigating error accumulation in long sequences. Al- though extremely long sequences still have room for improvement before consistently outperform- ing shorter ones, Vast provides both the practical capability and the conceptual foundation needed to make such progress feasible. We hope this work will inspire and guide future research on advancing sequence processing.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by National Science and Technology Major Project (No. 2025YFE01 13500), the National Science Fund for Distinguished Young Scholars (No. 62525605), and the National Natural Science Foundation of China (No. U25B2066, No. U22B2051, and No. 62272401).
7 ETHICS STATEMENT
This work focuses on improving computational efficiency in 3D scene reconstruction by introducing a training-free strategy. All experiments are conducted on publicly available benchmark datasets, including ScanNet, 7Scenes, and NRGBD. No human subjects, sensitive personal data, privacy- related information are involved in this research. The proposed method does not present foresecable risks of harmful misuse, and the study has no conflicts of interest commercial sponsorship that could bias the results. We adhered to the ICLR Code of Ethics to ensure fairness, transparency, and academic integrity throughout the research process.
8 REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our results. The experimental setup, including datasets (ScanNet-50) 7Scenes, and NRGBD), input sequence lengths, evaluation metrics (e.g., Chamfer Distance, Absolute Trajectory Error, Relative Pose Error), and hardware environment (NVIDIA A800 GPU, 80GB VRAM), are clearly described in the main text. Comparative baselines (VGGT, Fast3R, CUT3R, etc.) and detailed ablation studies are reported to validate our claims. Supplementary materials will include code and inference scripts to enable researchers to replicate the experiments under the same conditions.
REFERENCES
Marius Aasan, Odd Kolbjørnsen, Anne Schistad Solberg. and Adín Ramirez Rivera. A spitting im- age: Modular superpixel tokenization in vision transformers. In European Conference on Com- puler Vision, pp. 124-142. Springer, 2024.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4599-4603, 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Aron Monszpart, Daniyar Tur- mukhambetov, and Victor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of relocalizer. In European Conference on Computer Vision, pp. 421-440, Springer, 2024.
Qingqing Cao, Bhargavi Paranjape. and Hannanch Hajishirzi. Pumer: Pruning and merging tokens for efficient vision language models. arXiv preprint arXiv:2305./755 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen. Easi3r: Estimating disen- tangled motion from dust3r without training. arXiv preprini arXiv:2503.2439), 2025.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat- Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference Computer Vision, pp. 370-386. Springer, 2024.

---

Published as a conference paper at ICLR 2026
Joonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi, and Hyunwoo J Kim. vid-tldr: Training free token merging for light-weight video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18771- 2024.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Aftoz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv: 147994 2020.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint (iX.v:22377086991 2023.
Kai Deng, Zexin Ti, Jiawei Xu, Jian Yang, and Jin Xie. Vggt-long: Chunk it, loop it, align it- pushing vggt's limits on kilometer-scale long rgb sequences. arXiv preprint arXiv:2507. 16443, 2025.
Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting 40 seconds. arXiv preprint arXiv:2403. 203090009, 2(3):4, 2024.
Zhanzhou Feng and Shiliang Zhang. Efficient vision transformer via token merger. IEEE Transac- tions on Image Processing, 12:4156-4160 2023.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In European Conference on Computer Vision, pp. 241-258. Springer, 2024.
Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which to- kens use? investigating token reduction in vision transformers. In Proceedings tfe IEEE/CVF International Conference on Computer Vision, pp. 773-783, 2023.
Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Agglom- erative token clustering. In European Conference Computer Vision, pp. 200-218. Springer, 2024.
Jean Lahoud, Jiale Cao, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer Salman Khan, and Ming-Hsuar Yang. 3d vision with transformers: A survey. arXiv preprint arXiv:2208. 04300, 2022.
Scon-Ho Lee, Jue Wang. Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long- form video understanding. arXiv preprint arXiv:2410.23782, 2024.
Jiahui Lei, Yijia Weng, Adam W Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6165-6177, 2025.
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pp. 71-91. Springer, 2024.
Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1486-7495, 2024.
Yuanxum Lu, Jingyang Zhang, Tian Fang, Jean-Daniz Nahmias, Yanghai Tsin, Long Quan, Xun Cao, Yao Yao, and Shiwei Li. Matrix3d: Large photogrammetry model all-in-one. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 11250-1 1263, 2025.
Raul Mur-Artal and Juan D Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics, 3355 2017.
Raul Mur-Artal, Jose Maria Martinez. Montiel, and Juan D Tardos. Orb-slam: A versatile and
accurate monocular slam system. IEEE transactions roboties, (555:1117711163, 2015.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX1v:2304.007/93, 2023.
12

---

Published as a conference paper at ICLR 2026
Linfei Pan, Dániel Baráth, Mare Pollefeys, and Johannes L Schönberger. Global structure-froo motion revisited. In European Conference on Computer Vision, pp. 58-77. Springer, 2024.
Yansong Qu, Yuze Wang, and Yue Qi. Sg-nerf: Semantic-guided point-based neural radiance fields. In 2023 IEEE International Conference Multimedia and Expo (ICME), pp- 570-575, IEEE, 2023.
Yansong Qu, Shaohui Dai, Xinyang Li, Yuze Wang, You Shen, Liujuan Cao, and Rongrong Ji. Deocc-l-to-3: 3d de-occlusion from single image via self-supervised multi-view diffusion. arXiv preprint arXiv:2506.21544, 2025.
Cedric Renggli, André Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learning to merge tokens vision transformers. arXiv preprint arXiv:2202. /2015, 2022.
Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. Advances neural information processing systems, 34:127441122222 12797, 2021.
Johannes L Schonberger and Jan-Michael Frahm. Stracture-frommmottion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1104-411,, 2016.
Leqi Shen, Tianxiang Hao, Tao He, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval. arXiv preprint arXiv:2409.00 156, 2024.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, al. Dinov3, arXiv preprint arXiv:2508. 10104, 2025.
Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs. arXiv preprint arXiv:2408. 13912, 2024.
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 15598-15607, 2021.
Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5283-5293, 2025.
Chau Tran, Duy MH Nguyen, Manh-Dut Nguyen, TrungTii Nguyen, Ngan Le, Pengtao Xie, Daniel Sonntag, James Y Zou, Binh Nguyen, and Mathias Niepert. Accelerating transformers with spectrum-presserving token merging. Advances in Neural Information Processing Systems, 37: 30772-30810, 2024.
Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408. 16061, 2024.
Jannyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9773 9783, 2023.
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pp. 21686-21697, 2024a.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294-5306, 2025a.
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Con- tinuous 3d perception model with persistent state. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 10510-10522, 2025b.
13

---

Published as a conference paper at ICLR 2026
Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for ppen-domain images with optimal training supervision. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5261-5271, 2025c.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709 2024b.
Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiang- miao Pang, Chunhua Shen, and Tong He. Scalable permutation-equivarian visual geometry learn- ing. arXiv preprint ar arXiv:2507. 13:.2337773.37,, 2025d.
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. Advances in neural information processing systems, 13, 2000.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Locge-vocabulary 3d object dataset for realistic percep- tion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 803-814, 2023.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14138-14148, 2021.
Jianing Yang, Alexander Sax, Kevin J Liang Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fastăr: Towards 3d reconstruction 1000+ images in one forward pass. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 21924-21935, 2025.
Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming- Hsuan Yang, and Songyou Peng- No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint arXiv:2410.24207 2024.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewerafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint SoKiv:2409.020444 2024.
Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, and Xiaogang Wang. Not all tokens are equal: Human- visual analysis via token clustering transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11101- 11111, 2022.
Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tul- siani. Cameras as rays: Pose estimation via ray diffusion. arXiv preprint 47X14.240..18817, 2024a.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, De- qing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024b.
Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, and Gordon Wetzstein. Flare: Feed-forward geometry, appearance and camera es- timation from uncalibrated sparse views. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 21936-219477 2025.
Yipu Zhao and Patricio A Vela. Good feature matching: Toward accurate, robust vo/vslam with low latency. IEEE Transactions Robolics, 36(3))657-675. 2020.
Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu. Streaming 4d visual geometry transformer. arXiv preprint 1 2025.
14

---

Published as a conference paper at ICLR 2026
Zi-Xin Zou, Zhipeng Yu, Yuan- Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Har Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transfommers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10324-10335, 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A LLM USAGE AND REPRODUCIBILITY STATEMENT
The authors employed large language models (LLMs), such as Google's Gemini, soiely to improve the language and readability of this manuscript. These models were not involved generating ideas or shaping the conceptual framework. All authors have thoroughly reviewed the final version and take full responsibility for content and claims. To ensure reproducibility, the source code of the core components has been submitted anonymous supplementary material. Upon acceptance, all code and related resources will be released a public GitHub repository.
B DISCUSSIONS OF TOKEN COLLAPSE PHENOMENON
B. THE CONNECTION BETWEEN TOKEN COLLAPSE DINOV3 AND VGGT
We observe token collapse in both DINOv3 and VGGT, though its implications differ. In DINOv3, the multi-crop self-distillation framework drives the CLS token to capture view-invariant semantics by aligning local-crop student embeddings with global-crop teacher embeddings. This improves global consistency but gradually reduces patch-toker diversity, patch features increasingly con- verge toward the CLS token. Such dense feature degradation reflects an overemphasis on global invariance at the expense of local discriminability. To address this, Gram Anchoring constrains the similarity structure of patch features by matching the Gram matrix of the student to that of an early-stage teacher checkpoint with stronger dense properties. Given P patches with d-dimensional Lg-normalized features X.s, XG E Rxxd from the student and Gram teacher, respectively, the loss defined as:
Gram = X|xXX X
By preserving patch-level similarity patterns while keeping the CLS representation robust, Gram Anchoring effectively alleviates token collapse in DINOv3.
In VGGT, collapse appears primarily in the global attention maps, where tokens exhibit strong simi- larity. Unlike DINOv3, this redundancy is less harmful, since global attention is designed to capture cross-view correspondences, echoing the classical role of "correspondeecce in 3D vision. However, the redundancy inflates computation. Token merging provides effective remedy by compressing redundant tokens while retaining reference and salient ones, thereby reducing inference cost and stabilizing long-sequence prediction.
Taken together, these observations indicate that token collapse is closely tied to architectural and training design choices-sude model structure, augmentation, and supervision-ane becomes more pronounced in large models trained for long schedules. It is therefore crucial factor when scaling both 2D representation learning and 3D geometry modeling.
C DISCUSSIONS OF SHORT SEQUENCES PERFORMANCE
Although long sequences can be effectively accelerated using an aggressive merging ratio while maintaining accuracy, we additionally conducted short-sequence experiments provide a more comprehensive analysis of how the merging ratio interacts with sequence length.
Table(yevaluate the impact of different merging ratios on ScanNet-50 with input lengths of 50, 30, 10, and 5 frames. The results show that aggressive ratios (e.g., 0.9) lead to noticeable performance degradation-ant even failure cases, as illustrated Figure whereas ratio of 0.3 achieves per- formance comparable the baseline. These findings further indicate that the optimal merging ratio highly dependent on the available temporal context, and that short sequences are substantially more sensitive to over-aggressive token reduction.
D DISCUSSION OF ROBUSTNESS TO THE FIRST FRAME
We include additional experiments evaluating robustness to first-frame blur and occlusion in Ta- ble 10, along with more extensive visualizations in Figure 9 On ScanNet-50, we apply a blur kernel
16

---

Published as a conference paper at ICLR 2026
Table Quantitative results of short sequence performance on the ScanNet-50 dataset.
Input
Ratio
ATE L
ARE
RPE-rot 4
RPE-trans 4
Time 4
0.9
0.143
3,474
1.820
0.087
2.230.
3.447
0.089
2.315s
30
3:466
Baseline
0.138
3.512
1,685
0.079
3,478
0.9
0.172
3,750
2.826
0.150
1,34%
0.6
0.166
3,679
2.722
0.146
1.373
30
0.3
0.158
3.693
2.610
0.130
1,567s
Baseline
0.154
3,705
2.543
0.125
1,977s
0.9
0.250
5.316
6.225
0,369
0.628
0.246
4,460
5.148
0.306
0.637:
10
0.201
4.326
5.003
0.255
0.651s
Baseline
4.683
0.247
0.712s
18.626
0.682
0.433
0.452
17,905
22.179
0.673
0,440s
5
0.3
0,330
10.568
12.687
0.434
0,443
Baseline
0.345
10.463
12.629
0.450
0.507s
Baseline
Ratio 0.3
Ratio 0.6
Ratio 0.9
Figure 8: Comparison of merging ratios with 5 input frames, where aggressive ratios (e.g., 0.9) lead to failure cases.
of size 201 (a = 30) and use a 40x40 mask that occludes either 50% or 80% of the first frame. Interestingly, FastVGGT achieves better performance than the baseline VGGT even under these challenging conditions. To further understand this behavior, we quantify the contribution of first- frame dst tokens: with 500 input frames, only 1.96% of merged dst tokens originate from the first frame, and with 100 input frames the proportion is 9.17%. These statistics indicate that when the first frame is degraded, the merging process naturally shifts reliance toward higher-confidence matches from later frames, thereby mitigating the negative impact of early-frame corruption.
In addition, this phenomenon can be analogized to traditional SLAM systems, where the first frame typically used for keypoint initialization. Corruption in this frame compromise both the quantity and reliability of the extracted keypoints. When these degraded keypoints are matched against highly redundant ones from subsequent frames, the likelihood of mismatches and noise is significantly increased. In contrast, FastVGGT's merging strategy mitigates such issues by reducing redundancy, thereby helping to suppress the propagation of noise introduced by the first-frames
E
CORE CODE EXPLANATION
E. PSEUDOCODE OF FASTVGGT
Algorithm outlines the inference procedure of FastVGGT. The method begins by tokenizing each input frame into patch tokens, which are grouped into three categories: (i) reference tokens from the first frame, (ii) fixed fraction of salient tokens that remain non-mergeable, and (ii) the remaining tokens that are assigned as mergeable sources or destinations via region-based based sampling.
At each transformer block, source tokens merged into their most similar destination tokens before global attention is applied, thereby reducing redundancy and computational cost. After the global attention step, merged tokens restored by replicating the representations of their destinations,
17

---

Published as a conference paper at ICLR 2026
Table 10: Quantitative results of first-frame occlusion blur on the ScanNet-50 dataset.
First Frame
Method
500
300
100
VGGT
0.501
0.482
0,475
Occlusion-50K3
FastVGTT
0.471
0.452
0.441
0.514
Occlusion- 80%
FasVVGTT
0.476
0.468
0.459
VGGT
0.509
0.495
0.470
FasTGGT
0.477
0.461
0.454
Fadr001
Faxe/oo
FaANOOT
Figure 9: Comparison FastVGGT and VGGT under first-frame occlusion or blurring.
ensuring compatibility with downstream decoding. Frame attention is then applied to all tokens without merging, preserving temporal consistency. Finally, the complete token set decoded into dense 3D outputs, including depth maps, camera poses, and point maps.
E.2 TOKEN PARTITIONING
We convert global index buffer into ordered source (alidz) and destination (b_idæ) token indices, where destinations predesignated (-1 in idx buffer.see If protection is enabled, ad- ditionally record protected ed.itedddd ixx to exclude high-importance tokens from merging. The split function materializes these index sets any tensor x, returning sre/dst subsets (and protected if applicable).
rand_idx = idx_buffer_see serfferrsserrrshapeeeeeeeee (1, -1, arggortt (dim=1)
num dun_dst_crig int (idx Beliddxbufffe__eee _sq -1) sum ())
Original src and dst indices a_idx_orig rand_idx[:, num num_dst_oriq:, :] b b_idx_orig rand_idx(;, :num_dst_orig, :] i = a_idx_orig b bii = b b_idx_orig
if enable_protection::
protected_idx idx protected_indicee unsqueeze (0) Lunsqueeze (-1)
18

---

Published as a conference paper at ICLR 2026
Algorithm FastVGGT Inference (Training-Free Token Merging)
Require: Image sequence (fr)f=1, merging ratio
Ensure: Dense 3D reconstruction outputs
1: Tokenize each frame y into tokens y;; let T + UfTr
2: Partition tokens:
(a) Set all tokens of frame 1 as destination D
(b) Select p fraction salient tokens Ssal (fixed stride, non-mergeable)
(c) Region-based sampling in each frame: assign remaining tokens to D or S
3: for each transformer block & do if is Global Attention then
4:
5:
Merge: for each I, S, find nearest zd E D by cosine similarity; update æa +
(ara xx,/2
6:
Run Global Attention on D U Saa
7:
Unmerge: replicate æa back its merged x,
8:
else
9:
Run Frame Attention on all tokens
10:
end if
end for
12:
Decode tokens to outputs (depth, pose, point maps, etc.)
num_protected = protected_ies shape [1]
else:
protected_idx None
num_protected = 0
num_sre = a a_idx. shape [1]
num_dst b_idx.sheaee [1]
Define an internal function separate tokens
def split (x): = x. shape
if enable enable_protection:
src gather (x, dim=1, index=a_idx. exppadddddxexxxxddiiioo (B, num_sre, C)
dst gather (x, dim=1, index=b_idx. bist-b_ixxeeee
ieaebb_dxxeexpanddcc expand (B, num dst, C)
protected gather(
x, dim=1, protected idx ixx.ex (B, num_protected, C)
return sre, dst, protected else:
sIc gather (x, dim=1, index=a_idx exppxaaddxxexxpaddd
expand (B, num_arc, C)
dst gather (x, dim=l, index=b _dx. exppdddddxxxxxdddddcccooco (B, pum_dst, C)
return sre, dst
E.3 MERGING
Given src + dst pairings, we preserve unmerged tokens (unm) and aggregate selected source tokens into their matched destination tokens via a linear reducer (mean by default). The same indices are applied any extra tensors (e.g., q_rope, rope v) to ensure consistent downsampling across modalities.
along the sequence dimension, using
Extract unmerged src tokens using actual unm_idx size unm_len = www..xxpppppp..eco.eooaaa [1]
unm = gather (src, dim=-2, index=unm_ idx. ieppaadddddxxedddddeeeaaaaa (n, unm_len, c)
sre len are_idx. shape [1]
src gather (src, dim=-2, index=are_ide sttpeerrr_ddxeeepaddddaeaaaaaadd expand (n, sre_len, c))
dst Bat.scatter_reduce (-2, dst__dx._e ixt_idxxexppand (n, arc_len, c), src)
19

---

Published as a conference paper at ICLR 2026
E.4 UNMERGING
To restore the pre-merge layout, we reconstruct full-length tensor out by scattering dsl, unm, and reconstructed src back to their original indices (b_ide for destinations; mapped a_idr entries for unmerged and merged sources). If protection was enabled, protected tokens are also scattered back to preserve their original positions.
с unm. shape
arc = gather (dst, dim=-2, index=dst_idx iexpepdd_ddxxeexaaddd (B, sre_len, c)) out torch. zeros (B, N, c, device device, dtype=x.dtype)
out scttttttttttieeee_etee index=b index (B, num_dst, c), src=dst)
out. sct.teateer_
dim=-2,
index=gather(
a_idx.exparee (B, LLidx.shapelf 1), dim=l, index=unn_idx
) expand (B, unm_len, c), unc,
out. sut.scatter_ dim=-2,
index-gather( a_idx.expand (B, shape 1), dim=1, index=sre_idx
) expaad (B, src_len, c), src,sic,
if enable_protection out scatter_(
dim=-2, index-protected_iax idx. expppeedddddxxxxxxdddeddddd num nam_protected, c), src=protected,
F MORE VISUAL RESULTS
F.1 ATTENTION MAP VISUALIZATION
Please refer to our project page http::/faattegtggitubb. for more visualizations on the ScanNet dataset. To demonstrate the generality of our observations on the attention maps, provide ad- ditional visualizations of token attention maps from the global attention layer of VGGT on the segment-10275/ scene of the Waymo outdoor dataset. Figures 11 and Figures 12 show the results for the reference frame and the tenth frame, respectively. These visualizations allow us to exam- ine the behavior of VGGT's global attention mechanism in greater depth, leading to the following conclusions:
Across both indoor and outdoor datasets, the attention maps corresponding to different tokens exhibit consistently high degree of similarity, suggesting that the model captures common structural patterns regardless of scene type.
2. As the number of blocks increases, the overall distribution of the attention maps follows a pattern of concentration, dispersion, and re-concentration. We hypothesize that this pattern reflects the emergence of a more sophisticated mechanism for keypoint matching.
3. The similarity among attention maps becomes increasingly pronounced with greater block depth, indicating that deeper layers reinforce the alignment across frames.
4. In the final block, high activation values predominantly concentrated on the reference frame and on frames that share a high degree co-visibility with it, while frames with low co-visibility generally exhibit low activation values.
20

---

Published as a conference paper at ICLR 2026
Took
Block
TopK
Blook 20
Figure 10: Visualization of different salient-token selection strategies on ScanNet-50.
F.2 SALIENT TOKENS VISUALIZATION
To further analyze the effects of the two strategies, we visualize the spatial distribution of salient tokens in Figure I0 The results show that top-K performs well in shallow blocks by accurately capturing semantically important regions of the scene, but as the block depth increases, the selected salient tokens become increasingly concentrated. contrast, although the fixed-stride strategy less precise in identifying key token locations, it consistently distributes salient tokens more uni- formly across different blocks.
G LIMITATIONS AND FUTURE WORKS
While FastVGGT demonstrates notable efficiency and robustness long-sequence 3D reconstruc- tion, several limitations remain:
Adaptability to sequence length. The current design is tailored for large-scale inputs. When applied to short sequences, aggressive merging ratios may remove valuable fine- grained details and degrade performance. Developing adaptive merging policies that adjust to input size and scene complexity is therefore promising direction.
Extension to other architectures. Recent follow-up works of VGGT, such as PI3, also exhibit token redundancy patterns according to our preliminary observations. Extending FastVGGT these architectures could further enhance their scalability and efficiency.
Long-horizon consistency. Despite the strong generalization capabilities of both VGGT and FGTT, cumulative drift persists very long sequences. Future research should fo- cus mechanisms that improve long-horizon consistency, potentially by combining token merging with temporal alignment or memory-based strategies.
Handling dynamic scenes. The current framework does not explicitly address dynamic objects, which can lead degraded reconstruction quality in highly dynamic environments. Future work may explore both data-driven solutions (e.g.. incorporating motion-aware su- pervision) and architectural extensions (e.g., integrating dynamic object filtering motion segmentation modules) to improve robustness in such scenarios.
21

---

Published as a conference paper at ICLR 2026
Token
Token 500
Token 925
Taken 4625
Token 9250
Block
Block 6
Block 12
Block 18
Block 23
Figure 11: Visualization of attention maps from the global attention layer of VGGT for the reference frame in the segment-102751 scene of the Waymo dataset.
Token 0
Token 500
Token 925
Token 4625
Token 9250
Block O
Block 6
Block 12
Block 18
Block 23
Figure 12: Visualization of attention maps from the global attention layer of VGGT for the tenth frame in the segment-1022 102751 scene of the Waymo dataset.
22

---

Published as a conference paper at ICLR 2026
FastVGGT
VGGT
FastVGGT
VGGT
arri afti
es con us 100
2114
-ce
Axe
MA 100
Figure 13: Additional visualizations of pose estimation results on the ScanNet dataset.
23


Reviews:
[{"review_id": "IJp6T7zgla", "reviewer": "Reviewer_SPNW", "summary": "This paper aims to solve the computational bottleneck of the SOTA model VGGT when processing long image sequences. The authors identify the bottleneck as computational redundancy in the Global Attention layer, a phenomenon they term \"token collapse.\" To address this, the paper proposes FastVGGT, a training-free token merging framework. Its core is a token partitioning strategy specifically tailored for 3D geometry tasks (preserving the reference frame, retaining salient tokens, and uniform sampling). Experiments show that this method achieves a 4x speedup when processing 1000-frame sequences while simultaneously improving reconstruction accuracy and reducing error accumulation.", "questions": "Q1 : You determined an aggressive 90% merge ratio starting from Block 0, based on 500-frame sequences (Table 7). Is this fixed hyperparameter combination optimal for all tested sequence lengths (100, 300, 1000 frames)? Could the slight performance drop on 100-frame sequences (Table 5 RPE) be resolved by using a more conservative (lower) merge ratio? Does this suggest that an adaptive merge strategy (e.g., merge ratio $r$ as a function of sequence length $N$) would be a better choice?\n\nQ2 : Could you please provide a specific quantitative comparison of accuracy (CD/ATE) and inference time between (a) the top-k based on token norms strategy and (b) the final fixed-stride sampling strategy for \"salient token\" selection? (You claimed they were \"comparably accurate\" in Section 3.2). This is crucial for justifying your final design choice.\n\nQ3 : Your method is highly dependent on using the first frame as a global reference. Did you test the method's robustness to the quality of this first frame? For example, how much does FastVGGT's performance degrade if the first frame is blurry or heavily occluded? Have you considered alternative reference selection strategies, such as choosing the sharpest frame in the sequence (via some quality metric) as the reference, rather than blindly using the first one?\n\nQ4 : For your \"token collapse\" motivation, did you perform any quantitative measurements beyond the qualitative visualizations in Figure 3 to confirm this phenomenon? For example, by calculating the average cosine similarity between different token feature vectors in the Global Attention layers and showing that it increases in deeper layers? This would provide stronger support for your motivation.\n\nQ5 : The most exciting result in the paper is the improvement in accuracy on long sequences, which you attribute to \"mitigating error accumulation.\" Could you please explain mechanistically, in more detail, why removing tokens (even redundant ones) helps prevent error propagation? My hypothesis is: by merging redundant information in early layers (starting from Block 0), the model is forced to form a more compact and robust global representation, which prevents \"over-thinking\" on noise and redundant features in later layers, ultimately reducing drift. Do you agree with this hypothesis, or do you have other insights?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The practical value of this paper is extremely high. It proposes a training-free, plug-and-play method that solves a critical performance bottleneck in the SOTA model VGGT. Achieving up to 4x inference speedup makes large-scale 3D reconstruction more feasible for practical applications.\n\nMost impressively, the method not only accelerates inference but also significantly improves accuracy on long sequences. By mitigating error accumulation, it outperforms the baseline VGGT on both pose estimation and 3D reconstruction tasks, which is a very strong and non-trivial contribution.\n\nThe method demonstrates good originality. Its three-part token partitioning strategy (preserving the reference frame, retaining salient tokens, uniform sampling) is specifically designed for the characteristics of 3D geometry tasks (like cross-frame correspondence and a global coordinate system), rather than just being a simple application of standard 2D token merging.", "weaknesses": "W1: Insufficient discussion of the performance trade-off on short sequences. The paper's strategy and hyperparameters (e.g., 90% merge ratio) are clearly optimized for long sequences (e.g., 500-1000 frames). However, on short sequences (100 frames), the performance is mixed, even performing slightly worse than the baseline on some key metrics (e.g., RPE-rot and RPE-trans in Table 5). The authors acknowledge this in Appendix E, but a deeper analysis is missing from the main experimental section.\n\nW2: Disconnect between motivation and implementation for \"salient token\" selection. The motivation for \"salient tokens\" is described as preserving unique, \"keypoint-like\" features, and the paper mentions that a \"top-k based on token norms\" strategy was considered. However, a simple \"fixed-stride sampling\" was ultimately used for efficiency. The claim that both are \"comparably accurate\" is unsubstantiated (no data in ablations), which creates a disconnect between the final design and its original motivation (to preserve the most unique features).\n\nW3: Robustness to \"reference frame\" quality is questionable. The entire method relies heavily on the first frame as the \"world coordinate system\" and global anchor, designating all its tokens as high-priority dst tokens. The paper does not discuss at all how the model would perform if this first frame is of poor quality (e.g., blurry, occluded, or unrepresentative of the scene). This reliance on a single, arbitrarily chosen frame could be a robustness liability.\n\nW4: Evidence for \"token collapse\" is primarily qualitative. The paper's core motivation—the \"token collapse\" phenomenon—is primarily argued via visualizations of attention maps in Figure 3. While these maps look similar, this is only a qualitative observation. The paper lacks a more rigorous quantitative analysis (e.g., calculating average cosine similarity or feature variance between tokens) to prove the degree of redundancy, which weakens the force of its core motivation.\n\nW5: Justification for the fixed-stride sampling strategy. Using \"fixed-stride sampling\" to select \"salient tokens\" seems like an arbitrary choice. \"Keypoints\" in a scene are unlikely to be uniformly distributed on an image grid. This sampling method runs counter to the motivation of preserving the \"most unique\" features and is potentially less effective than the (W2-mentioned) top-k strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXHBGcJ0jr", "reviewer": "Reviewer_aGnr", "summary": "FastVGGT accelerates VGGT's 3D reconstruction inference by applying training-free token merging to the Global Attention bottleneck. Through strategic token partitioning—preserving reference frames, retaining salient tokens, and uniform sampling—it achieves 4× speedup on 1000-image inputs while maintaining reconstruction quality and mitigating error accumulation in long sequences.", "questions": "See Weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The method achieves substantial 4× speedup on 1000-image inputs while maintaining or even improving reconstruction quality, and effectively mitigates error accumulation in long-sequence scenarios. This addresses a critical bottleneck in feed-forward 3D reconstruction models and demonstrates clear real-world applicability.\n\n2. The approach requires no model retraining or fine-tuning, making it immediately deployable to existing VGGT systems. The comprehensive experiments across multiple benchmarks (ScanNet, 7 Scenes, NRGBD) validate its effectiveness and generalizability across diverse 3D reconstruction tasks.", "weaknesses": "1. This paper suffers from insufficient experimental details and lacks fair comparisons. Critical hyperparameters are inadequately documented: while the ablation study (Table 7) examines merging ratios of 0.3, 0.6, and 0.9, the main experiments (Tables 2-5) do not explicitly state which configuration was used. \n\n2. More problematically, Table 1 presents baseline comparisons with naive merging strategies (random and fixed-stride sampling) but provides inconsistent experimental conditions. The random sampling uses merging ratios (r=0.5, 0.8) while fixed-stride sampling uses stride values (s=5, 8), these setting do not align with any setting of the proposed method. Furthermore, what is the testing data? The resolution? The number of input frames? The paper provides no explanation for why certain configurations were chosen or how they relate to the proposed method's settings. This makes it nearly impossible to assess whether the improvements stem from the proposed partitioning strategy or simply from different hyperparameter choices. A fair comparison would require evaluating all methods under equivalent compression rates with clearly documented settings.\n\n3. The technical contribution of this paper is limited. The core proposal consists of three token partitioning strategies: (1) designating first-frame tokens as destinations, (2) retaining high-norm tokens as salient, and (3) applying region-based random sampling. While these choices are reasonable for 3D reconstruction tasks, they represent relatively straightforward engineering adaptations rather than fundamental algorithmic innovations. The token merging mechanism itself is directly borrowed from existing 2D vision methods (ToMe, ToMeSD) without architectural modifications. The main contribution appears to be identifying where to apply existing techniques rather than developing novel methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50GfboVTXn", "reviewer": "Reviewer_f7WY", "summary": "This paper proposes FastVGGT, a method to speed up Visual Geometry Transformers (VGGT) when handling long image sequences.\nThe authors find that the Global Attention layer causes heavy computation. To solve this, they introduce a training-free token pruning framework that keeps only the most useful tokens.\nFastVGGT uses three steps: keeping the first-frame tokens for stability, preserving important tokens for details, and using region-based random sampling for balanced coverage.\n\nExperiments show that FastVGGT achieves up to 4× faster inference and reduces error accumulation, making it more efficient and robust for long-sequence 3D vision tasks.", "questions": "1. In Tables 2, 4, and 5, the meaning of the bold numbers is unclear. If bold font is intended to indicate the best performance, there are several cases where FastVGGT performs worse than VGGT* but is still highlighted. This inconsistency creates confusion and raises concerns about the fairness and clarity of result presentation.\n\n2. The claimed advantage in handling long sequences is not clearly demonstrated. For example, in Table 2, the reconstruction quality with 1000 frames is even worse than with 500 frames, which contradicts the paper’s stated goal of improving long-sequence performance. This raises doubts about whether FastVGGT truly provides benefits for very long input sequences.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an important and practical problem — improving the inference efficiency of VGGT on long image sequences.\n\n2. The proposed training-free token pruning strategy is simple yet effective. It significantly reduces computational cost while maintaining nearly the same accuracy.\n\n3. The identification of the “token collapse” phenomenon in VGGT’s Global Attention is insightful.\n\n4. The paper is clearly written and well-structured, with intuitive motivation and clear methodology.", "weaknesses": "1. The improvement is very limited. As shown in Tables 2, 3, and 4, FastVGGT shows almost no accuracy gain compared to VGGT*, and in some cases even performs slightly worse. This weak improvement reduces the overall significance of the contribution.\n\n2. A straightforward expectation is that FastVGGT should reduce memory usage during long-sequence inference by pruning redundant tokens. However, the paper does not provide any experimental evidence or quantitative analysis on memory consumption. As a result, it remains unclear whether FastVGGT actually achieves significant memory savings when performing global attention over long sequences.\n\n3. Comparisons are limited. Most baselines are just VGGT variants. In Table 4 the authors report Pi3 as OOM; it would be more informative to implement an optimized variant of Pi3 (analogous to VGGT*) so a direct, fair comparison is possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WdTDCUT033", "reviewer": "Reviewer_rRZP", "summary": "The paper concerns an efficient extension to VGGT, targeting the attention bottleneck in the original design. The authors perform a thorough analysis and find redundancy in the token representations. The authors call this \"token collapse\", where similarities for clusters of tokens end up becoming more or less indistinguishable, representing a clear redundancy. The authors exploit this by implementing structural merging via an explicit partitioning strategy. While the underlying ideas for reducing the complexity is not entirely new; the paper introduces specific mechanisms that seem well aligned with the objective. The authors highlight that their method achieves a $4\\times$ speedup on 1000 image sequences, all while maintaining reconstruction quality.", "questions": "1. The authors uses the term “geometry-aware” liberally, but the mechanism operates entirely in feature space. Anchoring to the first frame and preserving salient tokens is plausible, but not proven to maintain geometric invariants. This reviewer is left wondering what the authors mean when stating that the model is \"geometry-aware\", and whether the authors considers this worthwhile defining more clearer in the manuscript?\n2. The method’s efficiency stems from token merging, yet comparisons are made only to 3D reconstruction baselines, not to efficient-transformer methods like ToMe, Nyströmformer, or even Performer [8]. The question is then; why not compare against generic attention approximations? Of course, this may not be trivial, but a discussion around why such a comparison is not included would be prudent.\n3. The merging ratio is treated as a fixed hyperparameter with minimal analysis. Do the authors have any insights to share with practitioners who would like to tune this in more specific applications?\n4. The paper reports mean metrics but is missing a failure analysis. Are errors gradual or catastrophic? Do high-parallax / low-texture regions induce failure compared to the original? Does merging induce smoothing, or loss of information, as one would expect?\n\nWhile not as much a question, but this reviewer emphasises that the manuscript would be strengthened by extending the related work section to include previous works on partitioning for reducing token redundancy.\n\n---\n\n[8] [Rethinking Attention with Performers, Choromanski et al., 2020](https://arxiv.org/abs/2009.14794)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The paper solves an important computational challenge with VGGT training and inference, which is of significant interest. \n2. The methodology and experimental setup is well designed, clearly written, and comes across as technically competent engineering.\n3. The technical contribution is intuitive, and well aligned with the goals the authors establish. Redefining the merging cycle for multi-view consistency, anchoring to the first frame and protecting salient tokens comes across as thoughtful and insightful modelling choices.\n\nOn the whole, the contributions are well motivated and the paper comes across as a well presented, with clear applications for a prominent contemporary modelling framework.", "weaknesses": "1. While the work solves an important problem with VGGT, the work is somewhat incremental; though in a technically competent and practically valuable sense. This is a minor, high level weakness, but has some implications, detailed below.\n2. The proposed merging mechanism is presented as a novel architectural contribution, but has close parallels in existing literature. The authors mention the cosine-based assignment and replication in ToMeSD [1]; an extension of ToMe [2] for diffusion. However, other approaches such as TokenLearner [3], SPiT [4], and ATC [5] are other works that apply similar mechanisms for perceptual grouping and exploiting redundancy, explicitly setting out to capture partitions with constraints of connectivity in 2D images. More tangentially, the averaging and landmark selection procedure can be said to correspond to Nyström-like kernel compression and feature sketching [6; 7], and can also arguably related to general low-rank approximation. This reviewer emphasises that while the paper’s integration of these techniques into 3D feed-forward geometry transformers is valuable, the current phrasing makes the work come across as a somewhat *isolated discovery*. In this reviewers opinion, the contributions of the paper would be clearer if the authors explicitly positioned the contribution within the existing body of work on the topic.\n3. Relatedly, an analysis on how the constrains used in the work (e.g. first-frame anchoring, salient protection) differ / synergise with existing works would benefit the field of token merging. Discussing how / why \"geometry-aware\" settings align / demand this specific adaptation would also be an interesting addition, and strengthen the work; particularly if the work is contrasted with existing methods.\n4. The paper is a little sparse in discussing the inevitable failure cases and practical limitations, as well as intuition for hyperparameters (merging ratio). The effect of merging naturally induces a loss of information, and failure cases (if applicable) would not be unnatural, nor surprising given the improved efficiency of the approach.\n5. Certain notation could be clarified slightly. L314 has examples of \"src\" and \"dst\" that is written out in plain text, which makes the notation hard to parse when placed in the prose. This is a minor weakness, but addressing this would improve the readability of the manuscript.\n\n---\n\n[1] [Token Merging for Fast Stable Diffusion, Bolya & Hoffman, 2023](https://arxiv.org/abs/2303.17604)\n\n[2] [Token Merging: Your ViT But Faster, Bolya et al., 2022](https://arxiv.org/abs/2210.09461)\n\n[3] [TokenLearner: Adaptive Space-Time Tokenization for Videos, Ryoo et al., 2021](https://openreview.net/forum?id=z-l1kpDXs88)\n\n[4] [A Spitting Image: Modular Superpixel Tokenization in Vision Transformers, Aasan et al., 2024](https://arxiv.org/abs/2408.07680)\n\n[5] [Agglomerative Token Clustering, Haurum et al., 2024](https://arxiv.org/abs/2409.11923)\n\n[6] [Using the Nyström Method to Speed Up Kernel Machines, Williams & Seeger, 2001](https://papers.nips.cc/paper_files/paper/2000/hash/19de10adbaa1b2ee13f77f679fa1483a-Abstract.html)\n\n[7] [Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention, Xiong et al., 2021](https://arxiv.org/abs/2102.03902)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)