IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction
Authors: Hao Li
Zhengyu Zou
Fangfu Liu
Xuanyang Zhang
Fangzhou Hong
Yukang Cao
Yushi LAN
Manyuan Zhang
Gang YU
Dingwen Zhang
Ziwei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: 3D Scene Understanding; Multi-View Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions, enabling coherent and accurate understanding of complex scenes. However, most prior approaches prioritize training large geometry models for low-level 3D reconstruction and treat high-level spatial understanding in isolation, overlooking the crucial interplay between these two fundamental aspects of 3D-scene analysis, thereby limiting generalization and leading to poor performance in downstream 3D understanding tasks. Recent attempts have mitigated this issue by simply aligning 3D models with specific language models, thus restricting perception to the aligned model's capacity and limiting adaptability to downstream tasks. In this paper, we propose Instance-Grounded Geometry Transformer (IGGT), an end-to-end large unified transformer to unify the knowledge for both spatial reconstruction and instance-level contextual understanding. Specifically, we design a 3D-Consistent Contrastive Learning strategy that guides IGGT to encode a unified representation with geometric structures and instance-grounded clustering through only 2D visual inputs. This representation supports consistent lifting of 2D visual inputs into a coherent 3D scene with explicitly distinct object instances. To facilitate this task, we further construct InsScene-15K, a large-scale dataset with high-quality RGB images, poses, depth maps, and 3D-consistent instance-level mask annotations with a novel data curation pipeline. Unlike previous methods that bound with a specific language model, we introduce an Instance-Grounded Scene Understanding paradigm, where instance masks serve as the bridge connecting our unified representation with diverse Visual Language Models (VLMs) in a plug-and-play manner, substantially expanding downstream understanding capabilities. Extensive experiments on instance multi-view instance matching, open-vocabulary segmentation, and QA scene grounding demonstrate that IGGT outperforms state-of-the-art methods in both quality and consistency for semantic 3D reconstruction.

Published as a conference paper at ICLR 2026
IGGT: INSTANCE-GROUNDER GEOMETRY TRANS- FORMER FOR SEMANTIC 3D RECONSTRUCTION
Hao Li!.2,3 Zhengyu Zou', Fangfu Liu', Xuanyang Zhang Fangzhou Hong",
Yukang Cao², Yushi Lan". Manyuan Zhang*. Gang Yu". Dingwen Zhang 1s Ziwei Liu? NWPU Lg-Lab, NTU SStepFun, Inc. 'THU MMLab, CUHK
Reconstruction
Visustization
InsScene-15k 15K Dataset
Various Downstream Applications
Instance
Geametry
"salined"
Sellow
Sugmant
Grounded
Trensformer
15K Scenes
200M Imgs
High-Quality Synthests
High-Quality
Scalable
In total
Instance Spatial Tracking
Open- VecenVoocbb Segmontation
Scene Grounding
RG8D
Web- Captured
building upon our curated large-scale dataset InsScene- 15K, we propose a novel end-to-end framework that enables geometric reconstruction and contextual understanding in uni- fied representation. This paradigm facilitates a wide range of applications, including multi-view instance matching, 2D/ 3D open-vocabulary segmentation, and scene grounding.
Figure 1: IGGT:
ABSTRACT
Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions, enabling coherent and accurate understanding of complex scenes. However, most prior approaches prioritize training large ge- ometry models for low-level 3D reconstruction and treat high-level spatial un- derstanding in isolation, overlooking the crucial interplay between these two fundamental aspects of 3D-scene analysis, thereby limiting generalization and leading to poor performance in downstream 3D understanding tasks. Recent attempts have mitigated this issue by simply aligning 3D models with specific language models, thus restricting perception to the aligned model's 's capacity and limiting udaptability to downstream tasks. In this paper, we propose Instance- Grounded Geometry Transformer (IGGT), end-to-end large unified trans- former to unify the knowledge for both spatial reconstruction and instance-level contextual understanding. Specifically, design a ED--Conssttent Contrastive Learning strategy that guides IGGT to encode a unified representation with ge- ometric structures and instance-grounded clustering through only 2D visual in- puts. This representation supports consistent lifting of 2D visual inputs into coherent 3D scene with explicitly distinct object instances. To facilitate this task, we further construct InsScene-15K, large-scale dataset with high-quality RGB images, poses, depth maps, and 3D-consistent instance-level mask annota- tions with a novel data curation pipeline. Unlike previous methods that bound with specific language model, we introduce an derstanding paradigm, where instance masks serve as the bridge connecting
Scene Un-
PPooeect Leader. Corresponding Authors.

---

Published as a conference paper at ICLR 2026
unified representation with diverse Visual Language Models (VLMs) in plug- and-play manner, substantially expanding downstream understanding capabilities. multi-view instance matching, open-vocabulary seg-
Extensive experiments mentation, and QA scene grounding demonstrate that IGGT outperforms state-of- the-art methods in both quality and consistency for semantic 3D reconstruction. /tttp:://gtthb.. /gthhtt.t.u....m com/ Lff
https:
INTRODUCTION
foundational goal in the pursuit of spatial intelligence (Yngg et al. (2025) is to build representations that mirror human understanding- capturing both the precise geometric structure and rich semantic content of a scene from visual sensory inputs such as RGB images. Such representations are vital for enabling downstream tasks like robotic manipulation (Qu et al. 2025). AR / VR (Jiang al.)
2025), and planning (Zhang et al. 2024).
Previous methods (Zust al. 2025 Fan et al., 2024 Sun et al. 2025) tackle this challenge through fragmented paradigm, decoupling 3D geometric reconstruction and high-level semantic understand- ing into isolated tasks. Typically, they first leverage geometry-focusee focusee techniques (e.g.. Multi-View Stereo (MVS) methods (Schönberger et al.) 2016
Schönberger & Frahm 2016) or off-the-shelf
large Image-to-31D models (Wang et al. 2024 2025 to predict low-level 3D structures, followed by vision-language models (VLMs) (Bai et al. 2023 2025) or 2D segmentation models (Cheng et al.) 2022) to perform high-level semantic segmentation tasks. However, these disjointed approaches are inherently flawed, as they propagate errors between stages and fail to leverage the mutual context between shape and identity. preventing them from enhancing each other's 's capabilities and hindering their ability support model reconstruction.
Recently emerged methods (Fan et al. 2024, Sun et al. 2025) attempt to bridge this gap by align- ing spatial models with specific VLM (Li all 2022)
However, these approaches suffer from three critical limitations. First, since 3D geometry contains low-level, fine-grained structural sig- nals, foreing a strict alignment with high-level textual concepts can over-smooth the representation, degrading high-frequency geometric details and undermining multi-view consistency. Second, this tight coupling
specific VLM architecture inherently restricts the performance to the base model
(e.g., LSeg (Li et al. (022) ) and prevents the integration of newer, more powerful foundation models (e.g.. CLIP (Rafford al. 2021), I. SigLIP (Tschamnen et al. 2025)). Third, since these VLMs (Li) et al. 2022 Ghiasi et al. 2022) are mainly trained on 2D image-text pairs, their aligned features often fail to distinguish objects within the same semantic category, which significantly limits more downstream applications (e.g.. ,3 mnstance-consistee tracking under large viewpoint changes and spatial QA when interfaced with VLMs).
To address this, we propose Dostance-Grouncdd Geometry Transformer (IGGT), novel end-to-end framework that unifies the representation for spatial reconstruction and contextual understanding. Instead of simply aligning geometry with language features, our key idea is to couple both factors by joint training and encourage the model to autonomously learn the relationship between 3D instance- level semantics and their geometric structures, yielding mutual improvements in contextual under- standing and geometry reconstruction. Specifically. 1) we employ a large Unified Transformer to encode multi-view images into unified token representations of the 3D scene, which are decoded by a Geometry Head and an Instance Head into geometric point maps and an instance clustering fields, respectively. 2) we employ cross-modal fusion block with window-shifted attention mechanism. enabling the Instance Head to leverage fine ime-grained geometric features at pixel level to enhance its spatial awareness. 3) To further improve multi-view consistency of the instance fields, we design a 3DD-consstentt contrastive learning strategy that guides IGGT to learn both geometric structures and instance- grouuddd clustering features. As instance-level geometry-semanti aligned annotations remain scarce in the community, facilitate this task by presenting a large-scale dataset coined InsScene-15K, a meticulously constructed dataset comprising high-quality RGB images, poses, depth maps, and 3D-consistent instance masks.
One more thing, after training the full model (i.e., IGGT), we design an Instance-Grounded Scene Understanding strategy, where instance masks serve the bridge connecting IGGT with diverse VLMs. Such a paradigm not only enables the seamless, alug-and-pla integration of various vision- language models (VLMs) such as CLIP and IPP to lift downstream task performance, but also
2

---

Published as a conference paper at ICLR 2026
Yes
Video Framera
Froms & Initaa Massi Proposals
Propagate Fonward
Froms Musi Proposels
Hutti-fees Consletert Object Segmentation Hasko
Input
Uncovered
Threshold?
(8] Video Captured Mask Annotation Pipeline
Update Mosis
Mask Merge
No
Bi-directional Propagation
IesScene-1a
RGB
Iriiall Haak
Uncoweed
Refined Object Masks
InsSpero-150 (Scene)
Kinac
Auda Ban Drv.
with seee (b) RGBD Captured Mask Refinement Pipeline
Figure 2: Data Curation Pipeline. Our data is collected from various sources and then annotated by novel data engine driven by SAM2 (Ravi et al.] 2024). (a) For video captured scenes (i.e., RE10K (Zhou et al., 2018 we annotate them through customized SAM2 video dense prediction pipeline. (b) For RGBD-scan scenes (e.g., ScanNet++ (Yeshwanth et al. 2023)), we regenerate dense mask annotations for each image and align them with the projected coarse GT masks.
extends to Large Multimodal Models (LMMs) (Bai al. 2023 2025 unlocking more sophisticated scene understanding and broader spectrum of applications like scene grounding.
We validate our framework through extensive experiments on diverse downstream tasks (e.g., multi- view instance matching, open-vocabulary segmentation, and scene grounding), demonstrating its superiority over state-of-the-art methods both task performance and 3D scene coherence.
2 INSSCENE- 15K DATASET
We construct the InsScene-158 15K dataset (in Sec. 2). where each scene includes corresponding RGB images, depth maps, poses, and 3D-consistent instance segmentation masks. To maintain consis- tency, we ensure that each instance retains unique ID across all views.
Our data curation pipeline systematically integrates three distinet categories of data to ensure com- prehensiveness and diversity, as illustrated in Fig. 2 1) synthesis (Aria (Pan et al.) 2023), Infini- gen (Raistrick et al. 2024)): 2) Video captured (RE10K (Zhou et al. 2018)); 3) RGBD captured (ScanNet++ (Yeshwanth et al. ). For synthetic datasets (c.g., Aria and Infinigen), we simul- taneously generate the RGB image, depth map, camera pose, and object-level segmentation masks for each rendered view. Since the simulation environment provides perfectly accurate 2D ground- truth masks (in Fig. 3 (a)), we use them directly without any post-processing. Moreover, regarding real-world scenarios, propose novel data curation pipeline that includes multi-view mask
Our Refined
RGB IMORO
Gon, MEous
RGB Image
Vonill GT
Our Rofined
Synthesis Scene
(b) Weo-Captured Scenro
Figure 3: Visualization of mask annotations from three different sources. For the RGBD scan scene, we additionally compare the vanilla ground-truth masks from ScanNet++ (Yeshwanth et al. 2023) with our refined annotations, along with their corresponding matched IDs and mloU scores.
3

---

Published as a conference paper at ICLR 2026
notation and refinement stages, driven by SAM2 (Raii al. 2024). Specifically, for real-world video-captured scenes such as RE10K (Zhou et al. 2018) (Fig. 1) our method first employs SAM to generate dense mask proposals on the iiiiill frame. These proposals then used prompts for the SAM2 video object segmenter to propagate masks temporally throughout the sequence. To handle objects and mitigate drift, adopt iterative strategy that designates new keyframe whenever the unsegmented area increases, where SAM is reapplied to discover objects in the un- covered regions. After processing the entire video, a final bi-directional propagation pass ensures high temporal consistency across object tracks. This curation strategy provides scalable and diverse annotations that enhance the generalization ability of our model.
For challenging datasets with large-scale camera motion but coarse 3D annotations such as Scan- Net++ (Yeshwanth et al. 2023), we first project the 3D annotations into 2D to obtain initial image- level object masks. While this guarantees multi-view consistency of object IDs, the masks are often coarse and imprecise. To improve their quality, we use SAM2 to generate fine-grained initial mask proposals that are accurate in shape but lack identity information. These proposals are then aligned with the projected ground-truth masks to assign consistent object IDs (Fig. 3 (c)), and proposals belonging the same ID merged into complete masks. The process is iteratively refined until all image regions are covered. This pipeline (Fig. 2 (b)) achieves both multi-view ID consistency and shape-accurate annotations, substantially improving 2D mask quality for real-world scenarios.
METHODOLOGY
3.1 OVERVIEW
Our method consists of two main phases. Firstly, we propose IGGT (in Sec. 3.2). a unified foun- dation model that simultaneously predicts distrmiiddseeimiaatieee features at the spatial level and performs 3D reconstruction through 3D-consistent contrastive learning on large-scale datasets. Sec- ondly, we propose an mnstance-grounded scene understanding strategy (Sec. (.3). This strategy employs unsupervised clustering to partition the scene into instances by grouping the predicted fea- tures into masks with consistent instance IDs. These masks are then used to guide state-of-the- vasion-language models (VLMs, e.g., CLIP, OpenSeg) and large multimodal models (LMMs, e.g., GPT-4o, Qwen2.5-VL.) to perform open-vocabulary scene querying and grounding tasks.
3.2 ARCHITECTURE OF IGGT
aim to forge unified rep- resentation, enabling comprehensive 3D reconstruction and understanding mutually reinforcing manner. Specifically, we propose IGGT F. which predicts camera parameters t,, depth map D,, point map P., and 3D-consistent, instance-level feature maps S; in a feed-forward manner:
As illustrated in Fig. 4 given N input images (I' E RFxWx3
F: (1.) - DI, S
(I)
Our IGGT consists of three parts: 1) Large Unified Transformer to capture Unified Token Rep- resentation from multiple images; 2) two Downstream Heads with Cross-Mode Fusion Block to simultaneously predict geometric structures and corresponding instance features; 3) 3D consistent supervision empower the model to construct 3D-consistent feature fields of the scenes.
Large Unified Transformer. We follow VGGT to construct a 1B parameter large unified Trans- former, designed encode the multi-view images {4 into set of powerful unified token rep- resentations (T, E IR M 1i=1 where M denotes the numbers of the tokens for each image and D is the dimension of the token. Our large Unified Transformer first adopts pretrained DINOv2 (Oquab) al. 2023) extract patch-level image tokens. To support arbitrary multi-view inputs while main- taining permutation equivariance, learnable camera token is concatenated to each view's token sequences. Subsequently, 24 blocks of intra-view self-attention and global-view cross-attention are applied to transform the image tokens into unified tokens (T, { capturing both local and global context, which enables holistic and globally consistent understanding of the 3D scene.
Downstream Heads and Cross-Modal Fusion Block. We employ two downstream branches-Geomettr
Geometry Head and Instance Head-to decode the unified tokens T.} into geometric and instance features, respectively. The Geometry Head, inheriting its design from VGGT, is com- posed of three distinct modules: camera predictor, depth predictor, and point predictor. The
4

---

Published as a conference paper at ICLR 2026
DINO
Maptor
Geometry Head
Instance-Grounded Scene Understanding
Insturce Spatal
DINO
Lange Unitted Transformer
Cross-Modal Fusion
Maaka
Open VICEE SEmUTEE
Sog
Deo. Rocon.
Ins. Racon.
Sel
Festa
OTina Manka
Mappi
InsScene-
Instance Head
Cross-Modal Modal Fusion
Cross
Open Vocab Segmentation
QA Scene Grounding
Pieaie
Windforr
Shifted
Attention
Claster
Highighheelrs.
Figure 4: Overview of IGGT. Given input images, our method encodes them into a series of Unified Token Representations, which then processed by the Geometry Head and the Instance Head to produce high-quality geometric reconstructions and instance-grounde clusterings simultancously. In the end, we introduce Esstace-Grrundde Scene Understanding to perform multiple applications.
camera predictor tasked with regressing camera parameters, including extrinsics and intrinsics, from camera-specific tokens. For dense prediction, the depth and point predictors employ a DPT- like architecture (Ranfti et al., 2021). This architecture reconstructs hierarchical geometric feature F = (Fr
1,(0) 1==1 from the unified tokens through progressive upsampling and multi-scale fusion network & p-r()). Similar this dense prediction paradigm. our Instance Head (:ne(t) also adopts DPT-like architecture to perform dense instance features Fime (Fine 1
(PT") = (p.((T))), (F!"*)
iine ((T,)).
Moreover, to enhance the fine-grained spatial awareness of the instance head, we propose a cross- modal fusion block F-)n which utilizes a sliding window cross attention to embed spatial struc- ture into the instance representation, making them sensitive to object boundaries and spatial layouts while avoiding the quadratic complexity of global attention:
Pine
Fins + Fwin (Q F
Flayy V= FPt F...))
After that, we concatenate all refined instance features { (Ping) } and map them through a conventional
3 x 3 convolutional layer to 8 dimensional instance features Onss E N8x
3D-Consistent Contrastive Supervision. We enforce 3D consistency on the instance features
Oms R Nx8x HxW by applying a multi-view contrastive loss Lmves which is designed to pull features from the same 3D instance together across views while pushing features from different instances apart. Given set of sampled pixels P. the loss formulated as:
Lyve
Apult
Σ
d(Sr:: Apush
Σ PE-PYEP m(pl)mlpp))
max(0, M M-d00---(ff--.. Spy))
(4)
PLPPEE mmpp)=m(py)
Here, -) is the L.2 distance between normalized features, m(p:) ) is the instance ID of pixel pi- The coefficients Apull and Xpush balance the pulling and pushing terms, while M is margin hyper- parameter that controls the discriminative between different instances. This objective structures the instance representations according to the 3D scene geometry, improving generalization. Overall, we train the whole model in multi-task loss:
Loverall Lpune + Ldepth + Lymap + Livee
(5)
where geometry supervision terms pose Lposes depth Laeptho and point map Lpmap followed by the training paradigm of VGGT, which is used to supervise the outputs of the geometry head.
3.3 INSTANCE-GROUNDEA SCENE UNDERSTANDING
Unlike prior approaches that are tightly coupled with a specific language model (e.g., for Open- Vocabulary Segmentation) and thus limited single type of task, decouple framework from
5

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative Results on ScanNet (Dai et al. 2017). Here we showcase the capability overview and report the performance of multi-view instance matching (MV Ins. Mat.), reconstruc- tion, and 2D/ 3D open- vocabulary semantic segmentation. The bold denotes the best results.
Model
Capability
MV Ins. Mat.
Recon. Metric
Open- Vocab. Semantic Segment
Recon, Understand Mat.
T-mloU; T-SR
Abs. Rel.
2D mloU 2D mAcc: 3D mloU
LSeg
58.11
65.76
OpenSeg
42.33
68.06
NeRF-DFF
X
7.99
36.53
45.40
65.29
12.29
Feature-3DGS
/
X
6.48
41.63
57.69
63.26
23.42
LSM (2 Views)
7
/
X
4.22
58.66
53.07
53.86
LSM (ieww)
3.17
64.81
53.40
59.50
35.37
SpaTracker+SAM
X
X
26.43
38.57
SAM2*
X
53.74
71.25
VGGT
X
1.84
83.60
Ours
69.41
98.66
1.90
83.71
60.46
81.84
39.68
Table 2: Quantitative Results on ScanNet++ YYshhwanth et al. 2023).
Here we report the multi-
view instance matching quality, reconstruction accuracy, and 2D 3D open-vocabulary semantic segmentation accuracy.
Model
MV Ins. Mat.
Recon, Metric
Open-Vocab. Vocab. Semantic Segment
T-mloU;
T-SR
Abs. Rel,
2D mloU
mAce;
mloU
LSeg
22.61
34.42
OpenSeg
13.92
48.13
3DGS
41.64
22.47
33.14
10.59
LSM (2 Views)
4.22
74.02
17.76
26.95
LSM (Multi Views)
2.96
83.28
17.88
27.84
15.17
SAA
16.15
23.68
SAM2*
44.16
$7.89
VGGT
Ours
73.02
98.90)
2.61
85,66
31,31
70.78
20.14
specific language models and propose a novel Dostance-Grounder Scene Understanding strategy to support broad range of downstream tasks. As shown in Tab. our method is the only one that simultaneously enables multi-view instance matching, image-to-3D reconstruction, and scene understanding, while achieving state-of-the-art performance across all tasks.
Multi- View Instance Matching. Specifically, inspired by SAMPart3D (Liu al. 2025) we apply
the density-based clustering algorithm HDBSCAN (Melnnes al. 2017) that gathers multi-view 2D instance features {O:"*) into distinct clusters, where each cluster represents a unique object instance present in the scene. Then we re-project the assigned cluster labels to their corresponding pixel locations produces a set of 3D-consistent 2D instance masks (Min') KK 1 Such paradigm enables dense tracking and segmentation of specific instances across multi-view images by leverag- ing explicit 3D priors, in stark contrast to existing methods that either limited to discriminating category-level features or lose targets during significant camera motion.
Open- Vocabulary Semantic Segmentation. These 3D-consittent instance masks serve effective prompts for any off-the-shelf VLMs (Radford et al.] 2021 Ghiasi et al.] (002)), enabling them to perform robust open-vocabulary semantic segmentation by assigning semantic category to each mask-defined region. Here we take OpenSeg (Ghiasi et al. 2022) as an example. It first produces image-wise features {Elang Dx } which considers contextual information to enable accurate languall alignment the features. We then aggregate the features within each 2D instance mask [r( E IR J via average mask pooling, yielding a compact representation for each instance. This step not only integrates the mask priors into the visual-language space, but also sharpens object boundaries and captures fine-grained local category cues, making the subsequent semantic assignment more accurate and robust.
QA Scene Grounding. Unlike prior methods that directly align 3D features with language em- beddings, our approach offers greater flexibility by decoupling instance clusterings, which then interact with LMMs (Bai et al. 20255 TTam et al.] 2023) to support object-centric QA 3D scenes. highlight the image regions corresponding the same instance & with masks [M J.kien (rendered in red), and query the LMM with yes/no questions to verify object consistency across views. Finally, we aggregate all positive ("yes") responses and concatenate the corresponding masks form the final segmentation output.
Coneretely, as shown Fig. given N views,
IN

---

Published as a conference paper at ICLR 2026
Figure 5: Qualitative results on multi-view instance matching. We present two example scenes from ScanNet (Dai et al. 2017) and ScanNet++ (Yeshwanth al., [2023), and compare our method with SAM2* and Spa'l Tracker+SMM All instances are visualized with distinct IDs and colors for clarity.
Images
Ins. PCA
Masks
Sconnet ID: Scene0706 00
Images
Ins. PCA
Masks
50
Sconnot+- ID: 19869a81 07
2
Figure 6: We visualize our 3D-consistent PCA results with corresponding clustered masks derived from instance- instance-grounded features. Similar colors in PCA indicate higher feature similarity between instances. For clustered masks, the same object instance shares the same color across multi-views.
EXPERIMENTS
Evaluation Details. We conduct comprehensive experiments on the ScanNet (Dai et al. 2017) and ScanNet++ (Yeshwanth et al. 2023) datasets. From each dataset, we randomly select 10 scenes and sample 8-10 images per scene, with the selection strategy designed to maximize spatial coverage of the scene while preserving sufficient overlap to ensure cross-view consistency. (a) For Multi- View Instance Matching evaluation, we evaluate tracking performance using Temporal mloU (T- mloU) and Temporal Success Rate (T-SR). T-mIoU measures the segmentation accuracy of the same object across different views, while T-SR assesses whether the object is successfully tracked every view. (b) For Open- Vocabulary Segmentation evaluation, we follow LangSplat (Qin et al. 2024) and LangSurf (Li et al. (222bb). which adopt mloU and mAcc measure 2D segmentation accuracy. In addition, we evaluate the 3D mloU metric by aligning the reconstructed scene with the ground-truth point cloud. (c) For Reconstruction evaluation, we follow LSM (Fan al. 2024) and VGGT (Wang et al. 2025) that utilize Absolute Relative Error (Abs. Rel) and Inlier Ratio (r) with a threshold of 1.03 assess each scene. The details of these metries are shown in the appendix.
Evaluation of Multi View Instance Matching. To comprehensively evaluate the tracking qual- ity of proposed method and competing approaches, particularly under large viewpoint changes with multiple objects, we manually annotate a subset of objects across several scenes with pre- cise ground-trute labels (more visualization in the Appendix). For baseline methods, we modify SAM2 (Ravi et al. 2024) to support dense segmentation and tracking under multi-view inputs, de- noted as SAM2 In addition, we integrate SAM into SpaTrrccerV2 (Xiao al. (025). where tracking points are used as prompts to perform dense segmentation. Tab. [1 and Tab. present the quantitative results, demonstrating the significant superiority of our method. By leveraging implicit 3D reasoning, our approach successfully distinguishes object identities to achieve nearly 100% T- SR accuracy. In contrast, baseline methods fail at this crucial task, yielding a T-mloU below 30%, whereas approach surpasses 60%. This performance gap is visually demonstrated in Fig. [ where our method successfully tracks and segments the chair under large camera motions, while competing methods lose the track.
7

---

Published as a conference paper at ICLR 2026
SOGS
Figure Qualitative Results 2D Open- Vocabulary Segmentation on ScanNet and SeanNet++
Furthermore, we provide additional visualizations of our 3D-consistent instance features using Prin- cipal Component Analysis (PCA), along with their corresponding clustered masks, as shown in Fig. 6 As illustrated, IGGT produces 3D-consistent eettaccegrrunded features that remain dis- crimmative across multiple views: multiple instances with the same category exhibit similar yet distinguishable colors in the PCA space. This property serves as a crucial foundation for the Multi- View Instance Matching task, as it enables consistent tracking and segmentation of individual objects under large motions and in the presence of many similar instances.
We compare our method with other Image-to-3D feedforward method (Fan et al. 2024). per-scene optimized methods (Zhou et al. 2024 Kobayashi et al., (022). and 2D methods (Ghiasi et al.) 2022 Li et al. 2022 on both ScanNet and Scanent++ datasets. The results are reported Tab. [ and Tab. On ScanNet++, our method achieves leading performance, surpassing other ap- proaches by 8.34% in mloU for segmentation and 7.88% in MAce for object localization. This performance im- provement is attributed to method's superior multi-view consistency, which helps correct object recognition errors caused by incomplete views, as illustrated in Fig. [7] where the sink is difficult to identify due to limited viewpoint coverage. On the other hand, we also evaluate the accuracy of depth estimation on multi-view inputs. The results show that our method is on par with VGGT on ScanNet, and outperforms VGGT on ScanNet++ by 0.14 in Abs. Rel and 0.25 in т, benefiting from the mutual enhancement of semantics and geometry achieved through joint training. This performance improvement is further demonstrated in 3D segmentation (see Tab. [ and Tab. 2). where our method outperforms previous approaches by 4.31 % and 4.97% in terms of 3D mloU. As shown in Fig. & our method produces more consistent segmentations within the same regions and significantly fewer floaters in 3D, which indicates that our approach yields superior 3D semantic representations.
Evaluation of Voeab-oocyyllary Segmentation.
LSM-MV
(Ours)
Intken
tectpnct
Figure 8: Visualization of 3D Open-Vocabulary Segmentation.
Class-Agnostic Mask Segmentation. Following recent 3D instance segmentation works Yin al. 2024, Yang et al., 2024: Huang et al., 2024c). we provide a class-agnostic instance seg- mentation evaluation on the ScanNet dataset, comparing our method against VGGT + Graph Cut and VGGT SAI3D (Yin al.] (024). Our method outperforms VGGT+Graph Cut by 8.83 AP while avoiding its expensive mesh gen- eration, reducing runtime by about 8 minutes. Moreover, it approaches the performance of the per-scene optimization method VGGT+SAI3 with nearly a 5x speed-up (2.5 min v.s. 12.2 min), demonstrating the efficiency of our uni-
We report AP25, AP50, and AP in Tab.
Table 3: Comparison of class-agnostie segmenta- tion with different methods on AP metrics.
Method
AP
APsa
AP
Time
VGGT + Graph Cut
3.42
9.30
30.86
10.65min
VGGT SAI3D
14.94
31.06
50.07
12.22min
Ours
12.25
24.93
47.55
2.52min
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of runtime for different methods.
Method
SIM Time
Per-Scene Time
Infer. Time
Post Proc. Time
Final Time
NeRF-DFF
50.33s 50.338
3min 47min
3 3.84min
Feattur-3IDGS
47. 37.84min
LSM (Multi-View)
15,98s
13.72s
29.70 70s
Grahh+ Cut
0,4266
10.64min
10.65min
VGGT+SAI3
0.426s
12.21min
12.22min
Ours
0.545s
2.51min
2.52min
fied reconstruction and instance understanding framework. These improvements are also supported by the qualitative results in Fig. 9 More analysis and settings shown in the appendix.
Runtime Analysis. Here, we use 10 images from a single scene to evaluate the detailed runtime, as reported in Tab. Compared with NeRF/3D-GS or other per-scene optimization methods, our approach substantially reduces reconstruction time. In contrast, feed-forward methods such as LSM (Multi-View) achieve faster processing but suffer from limited reconstruction accuracy and are un- able to support tasks like instance matching or scene QA. More in the appendix.
(OOD) Results. We evaluate our model on various OOD scenarios: outdoor scenes (ETH3D (Schops et al. 2017)), autonomous driving scenes (Waymo Open Dataset (Sun et al. 2020 and egocentric- view data (robotics data and self-collected dataset). Qualitative results in Fig. 10 show that our method can reconstruct scenes and produce 3D-consistent instance clustering on these unseen domains, demonstrating strong generalization ability. More in the appendix.
RGB
VGGT Graph Cut
IGGT (Ours)
GT Instance
VGGT SAIDD
Figure 9: Visualization of the Class-Agnostic 3D Mask Segmentation Results.
Applications of QA Scene Grounding. We present the QA application results in Fig. 11 on the Teatime scene from the LERF-OVS (Kerr et al.] (202)) dataset. and compare our approach against the state-of-the-art Gemini 2.5 Pro (Comanici et al., (025). As shown, our instance- grounded querying fully leverages the reasoning capacity of LMMs, achieving accurate segmen- tation for complex prompts and superior multi-view consistency compared existing unified gen- eration-understandinn models, thereby enabling more complex QA tasks in 3D scenes.
Ablation Study. We show the training curve of IGGT in Fig. 1 Without the tross-modal fusion model, the instance head struggles to capture high-resolution geometric information, leading to more difficult convergence, as reflected in the sharpness of the chair edges the PCA visualization. We also conduct ablations integrating different VLMs into method (e.g., LSeg (Li et al. 2022). CLIP (Radford et al. 2021). OpenSeg (Ghiasi et al. 2022)). As shown in the table, LSeg and OpenSeg, with better global context representation, achieve higher accuracy on background classes (e.g.. cabinet), while CLIP, with superior text alignment, performs better on complex categories such as 'DALL-E' and 'Ottolegnghi' in Fig. [2] This further demonstrates the flexibility of our method in using different VLMs to achieve improved text query performance.
Figure 10: Visualization of OOD scenes (e.g., outdoor, autonomous driving, egocentric, robotics).

---

Published as a conference paper at ICLR 2026
CHLLL
No
Gemini 2.5 Pro
Ours Instance Grounded Scene Understanding QLen-VL2.5
Figure 11: Applications of QA Scene Understanding com- pared with vanilla Gemini 2.5 Pro Comanici et al. (2025).
Figure 12:
Visualization of our
method using different VLMs.
Loss
Ours wib Hulti Modal Fusion Ours vol Mocal Fusion
Table Integration with Different VLMs.
0.5
ScanNet
ScanNet++
0.4
Method
0.3
mloUp
mAce
mloUp
mAco
0.2
Ours w/ Lseg
60.46
81.84
22.72
63.56
0.1
Step
Ours w/ CLIP
49.36
62.68
21.52
61.36
Ours w/ OpenSeg
58.12
78.75
31.31
70.78
20k
40k
Figure 13: Ablation on
Fusion.
RELATED WORK
Spatial Foundation Models. Image-to-3D reconstruction has evolved from early SfM pipelines like COLMAP (Schonberger & Frahm 2016), which estimate camera poses and sparse point clouds, to more advanced methods like 3D Gaussian Splatting (3DGS) (Kerbl et al. 2023) for efficient novel view synthesis. Scene Representation Transformers (Sajjadi et al.) 2022 represent images as latent tokens, enabling view synthesis without accurate poses, but still struggle with explicit geometry and generalization. DUSt3R (Wang et al., (2024) improves upon this by directly regressing dense point maps from unposed image pairs, while VGGT (Wang et al. 2025) scales this approach to multiple images with competitive accuracy. However, these methods remain focused on geometric reconstruction, often neglecting higher-level scene understanding.
3D Scene Understanding. Integrating semantics into 3D reconstruction is vital for scene under- standing.
methods (Huang et al.) 2024b Yu et al., 2025) inject 3D priors into LMMs, whereas 3D-based methods (Peng et al. 2023 Li et al. 2025) align VLM features in 3D space for understanding, which relies on pre-reconstructed 3D scenes. Meanwhile, methods such as LangSplat (Qin et al.) (024) inject vision-language features into 3D Gaussian Splatting enable semantic reasoning, but they typically require dense multi-view inputs and per-scene optimization. Moreover, feed-forward approaches such as Panst3R (Zust et al.) 2025) and DUSt3R (Wang al.) 2024) attempt feed-forward scene understanding, but decouple geometry and semantics, limiting mutual benefits. Methods like LSM (Fan et al., (0024) and Uni3R (Sun et al.) 2025) align spatial models with vision-language models (e.g., LSeg (Li et al., 2022)), but face limitations in integrating stronger VLMs and struggle with fine-grained, instance-level queries in complex scenes.
CONCLUSION
In this paper, we introduce IGGT, a novel end-to-end framework that unifies the representation for both spatial reconstruction and contextual understanding in 3D scene. The key to our success is that we couple geometric and instance-level semantic features by joint training and unleash the potential of unified large transformer to achieve mutual improvements in contextual understanding and geometry reconstruction. To facilitate this task, we further present a large scale dataset called InsScene- 15K, including high-quality RGB images, poses, depth maps, and 3D-consistent instance masks. Moreover, our proposed pestancee-roundee scene understanding strategy enables IGGT with alug-and-play integration of various VLMs and LMMs, unlocking broader range of applications.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work focuses on improving spatial reconstruction and understanding. While our model is trained on self-annotated datasets based standard open-source images and tested in controlled settings, we acknowledge that any Al system may potentially exhibit biases or produce unexpected behaviors. Our research is intended for academic exploration only, and we emphasize that any such outcomes do not reflect the views of the authors. We support the development of Al technologies that are ethical, safe, and aligned with societal values.
REPRODUCIBILITY STATEMENT
All code and model checkpoints will be publicly released to ensure reproducibility.
REFERENCES
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question an- swering for spatial scene understanding. In proceedings the IEEE/CVF conference on computer vision and patlern recognition, pp. 19129-19139, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2tii:2500.1192... 2025.
Dampeng Chen, Hai Li. Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai Nan Wang. Haomin Liu, Hujun Bao, and Guofeng Zhang. Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction. IEEE Transactions Visualization and Computer Graphics, 2024.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference computer vision, pp. 202-221. Springer, 2020.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290- 1299, 2022.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5828-5839) 2017.
Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi. Zhangyang Wang, Danfei Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d. Advances in neural information processing systems, 777:40112-4022 2024.
Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. Inter- national journal of computer vision, (9(2):1771181, 2004.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European conference on computer vision, pp. 540-557. Springer, 2022.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pp- 1-11, 2024a.
11

---

Published as a conference paper at ICLR 2026
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37:113991- 114017, 2024b.
Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3d: Snap and lookup for 3d open- open-vocabulary instance segmentation. In European Conference on Computer Vision, pp. 169-185. Springer, 2024c.
Lihan Jiang, Yucheng Mao, Linning Xu, Tao Lu, Kerui Ren, Yichen Jin, Xudong Xu, Mulin Yu, Jiangmiao Pang, Feng Zhao, et al. Anysplat: Feed-forward 3d gaussian splatting from uncon- strained views. arXiv preprint arXiv:2505.23771 2025.
Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction, In Pro- ceedings of the fourth Eurographics symposium Geometry processing, volume 7, 2006.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields. Proceedings of the IEEE/CVF international conference on computer vision, pp. 19729-19779 2023.
Sosuke Kohayashi, Bicci Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in neural information processing systems, 15:23311-23330) 2022.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201 200.:22001003466,, 2022.
Hao Li, Yuanyuan Gao, Chenming Wu, Dingwen Zhang, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Junwei Han. Ggrt: Towards pose-free generalizable 3d gaus- sian splatting in real-time. In European Conference on Computer Vision, pp. 325-341. Springer, 2024a.
Hao Li, Roy Qin, Zhengyu Zou, Diqi He, Bohan Li, Bingquan Dai, Dingewn Zhang, and Junwei Han. Langsurf: aanguage-embedder surface gaussians for scene understanding. arXiv preprint arXiv:2412.17635 2024b.
Hao Li, Dingwen Zhang, Yalun Dai, Nian Liu, Lechao Cheng, Jingfeng Li, Jingdong Wang, and Junwei Han. Gp-nerf: Generalized perception nerf for context-aware 3d scene understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
21708-21715 2024c.
Yue Li, Qi Ma. Runyi Yang. Huapeng Li, Mengjiao Ma. Bin Ren, Nikola Popovic. Nicu Sebe, Ender Konukoglu, Theo Gevers, et al. Scenesplat: Gaussian splatting-base bbased scene understanding with vision-language pretraining. arXiv preprint arXiv:2503. 18052, 2025.
Minghua Liu, Mikaela Angelina Uy, Donglai Xiang, Hao Su, Sanja Fidler, Nicholas Sharp, and Jun Gao. Partfield: Learning 3d feature fields for part segmentation and beyond. arXiv preprint arXiv:2504. 1451, 2025.
Leland Melnnes, John Healy, Steve Astels, et al. hdbscan: Hierarchical density based clustering. J. Open Source Softw, 2(1 1):205, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07 300999000011....0.. 2023.
Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 20133-20143 2023.
12

---

Published as a conference paper at ICLR 2026
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 815-824, 2023.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d lan- guage gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20051-20060, 2024.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, al. Spatialvla: Exploring spatial representations for visual- language-action model. arXiv preprint arXiv:2501.155811 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma, and Jia Deng. In- finigen indoors: Photorealistic indoor scenes using procedural generation. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21783-21794, June 2024.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12179-12188, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint 2400:22008.0714 2024. URLttttss: //axxiv arwiv org/ abs/ / 0224088.01111
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6229-62334 2022.
Johannes I. Schonberger and Jan-Michael Frahm. Structueefroommmatiin revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structureffrommmotiio revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
Thomas Schops, Johannes L. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3260-3269, 2017.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3146-2454 June 2020.
13

---

Published as a conference paper at ICLR 2026
Xiangyu Sun, Liu Liu, Seungtae Nam, Gyeongjin Kang, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang, Eunbyung Park, et al. Uni3r: Unified 3d reconstruction and semantic under- standing via generalizable gaussian splatting from unposed multi-view images. arXiv preprint 2:Xiv:2508.0364 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23 / 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathyy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 2rX1v:2502.14786, 2025.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2294-5306, 2025.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovski,, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709, 2024.
Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: 3d point tracking made easy. arXiv preprint arXiv:2507. 2524. 2025.
Jihan Yang, Shusheng Yang. Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632-10643, 2025.
Yunhan Yang, Yukun Huang. Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y Lam, Yan-Pei Cao, and Xihui Liu. Sampart3d: Segment any part in 3d objects. arXiv preprint arXiv:2411.07184, 2024.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured mieti-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp. 767-783, 2018.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12 12-22, 2023.
Yingda Yin, Yuzheng Liu, Yang Xiao, Daniel Cohen-Or, Jingwei Huang, and Baoquan Chen. Sai3d: Segment any instance in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3292-3302, 2024.
Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14147-14157, 2025.
Dingwen Zhang, Hao Li, Diqi He, Nian Liu, Lechao Cheng, Jingdong Wang, and Junwei Han. Un- supervised pre-training with language-vision prompts for low-data instance segmentation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-and languaga navigation today and tomorrow: survey in the era of foundation models. arXiv preprint arXiv:2407.0703: 2024.
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Com- pufer Vision and Pattern Recognition, pp. 21676-21685, 2024.
14

---

Published as a conference paper at ICLR 2026
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.098//1 2018.
Lojze Zust, Yohann Cabon, Juliette Marrie, Leonid Antsfeld, Boris Chidlovskii, Jerome Revaud, and Gabriela Csurka Panst3r: Multi-view consistent panoptic segmentation. arXiv preprint arXiv:2506.2 21348, 2025.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A. ACKNOWLEDGMENT
This research is supported by cash and in-kind funding from NTU S-Lab and industry partner(s). This study is also supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 MMOE-22P220221-00 0000021-0011 1-0012, MOE-T2EP20223-0N tional Natural Science Foundation of China under Grant 62293543, Grant 62322605, and Grant 625B2148.
This work supported in part by the Na-
A.2 USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) used exclusively for minor grammar corrections and stylistic polishing of the manuscript. They are not involved in the design of the methodology, execution of experiments, analysis of results, or any other aspect of the scientific contribution.
A.3 RELATED WORK
Spatial Foundation Model Image-to-3D reconstruction a long-standing problem in computer vi- sion. Early pipelines such COLMAP (Schonberger & Frahm 2016) and related SfM methods estimate camera poses and sparse point clouds, often followed by multi-view stereo (MVS) to ob- tain dense geometry (Yao et al. 2018). Building on such SfM-based initialization, 3D Gaussian Splatting (3DGS) (Kerbl al.J 2023) introduced a highly efficient representation for photorealistic novel view synthesis, inspiring cronstruction-orientee extensions (Chen et al. 2024 Huang al, 2024a). To reduce the reliance on accurate calibration, Scene Representation Transformers (Sajjadi al. 2022 Li et al. 2024a) represent multiple images as latent scene tokens, enabling novel view synthesis under uncertain or missing poses, though they still struggle to produce explicit geometry and generalize reliably. DUSt3R (Wang al.) (2024) takes further step by directly regressing dense point maps from unposed image pairs, achieving pixel-aligned geometry without SIM initialization. In contrast, VGGT (Wang al., 2025) scales this paradigm to dozens hundreds of images in single feed-forware forwwrd pass. jointly predicting cameras, depth, point maps, and tracks with competitive accuracy to optimization based pipelines. Despite these advances, these methods remain focused on low-level geometric reconstruction while overlooking higher-level scene understanding.
3D Scene Understanding Integrating semantics into 3D reconstructions crucial for higher-level scene understanding tasks. Existing LLM- methods (Huang et al. 2024b Yu et al. 2025 in- ject 3D priors into LMMs and finetune them to perform 3D grounding.
However, these approaches
rely pre-reconstructed 3D scenes, Recent efforts (Zhang et al. 2025 Li et al. 2024c| Zhou al.)
2024 Li et al.] 2024b] Qin et al. 2024) like LangSplat (Qin et al. (024) inject vision-language features (e.g., CLIP (Radford et al. 2021)) into 3D Gaussian Splatting, enabling semantic reasoning over reconstructed scenes. However, these methods typically require dense multi-viee inputs and per-scene optimization, which hinders scalability. More generalizable approaches like Panst3R (Zust et al. 2025) build on DUSt3R (Wang et al. (024) to achieve feed-forward 3D scene understanding directly from posed unposed images. Yet. they often decouple reconstruction from understand- ing and freeze the geometry module, which restricts mutual benefits between the two and leads to suboptimal semantic grounding. Parallel attempts such as LSM (Fan et al., 2024) and Uni3R (Sun)
et al. 2025) seek to bridge geometry and semantics by aligning spatial models with specific vision- language models (e.g., LSeg (Li al.) (022)), but this tight coupling has two key drawbacks: (1) prevents seamless integration of stronger VI.Ms (Tschannen et al.) 2025) Siméoni et al.) (025)
as they emerge, thereby constraining text query performance; (2) the alignment is typically at the
category level rather than instance-level. these methods struggle with fine-graie grained, object-centric QA in scenes that contain multiple similar instances.
To address this problem, our proposed framework, IGGT, addresses these limitations by learning unified representation for both reconstruction and understanding. Instead of tightly coupling with a single VLM, we introduce instance grounded paradigm where instance masks serve bridge to connect with diverse VLMs and Large Multimodal Models (LMMs) plug-and and--andd-lyyyy manner, substantially expanding downstream capabilities.
16

---

Published as a conference paper at ICLR 2026
A.4 TRAINING DETAILS
Our model is initialized with weights from VGGT (Wang et al. 2025) and fine-tuned on the InsScene- 15K dataset, which contains 15,000 scenes. Training is performed GPUs for 2 days using the AdamW optimizer. The learning rate is set to 1 x 1005 for the large unified Transformer backbone and 10 5 for both the geometry and instance heads. For each training batch, we randomly sample 1-12 frames from randomly selected scene, yielding total of 24 images per batch. For hyper- parameer settings, we set Apull 2.0, Apult 1.0 and M 1.0.
8 NVIDIA A800
A.5 METRICS FOR DIFFERENT TASKS
Multi- View Instance Matching. For the Multi-View Instance Matching task, we evaluate tracking performance using Temporal mloU (T-mloU) and Temporal Success Rate (T-SR). Given an object and its predicted masks {Mp]7= It=1 across T views with corresponding ground -ruth masks {M, ] T-mloU defined as
T MF Mg U Mgl
M
T-mloU(o) -
==1
T-SR evaluates whether the object is successfully tracked across all views, and is defined as
(-) = I Vt E f1..... T7, [M,] > 00]
where [-] denotes the indicator function. The final scores are averaged over all objects in the dataset.
3D Semantic Segmentation mloU. To evaluate 3D semantic segmentation, we first obtain the RGB 3D points from per-image point maps and align them with the ground truth. Next, we assign seman- tic labels to the corresponding 3D points based the results of 2D open-vocabulary segmentation. These labeled 3D points are subsequently voxelized, and the 3D mloU is computed based on the voxel representation. Fig. 15 illustrates the overall pipeline. Additionally, Fig. [66 presents qualita- tive results of 3D open-vocabulary segmentation. For LSM, based on its two-view input, we apply the global alignment strategy of Dust3R to optimize the point maps across all views. For Feature- 3DGS, the ground-trute point maps are used as the initial input. However, due to the sparsity of input views, reconstruction quality remains limited.
A.6 ADDITION INFORMATION OF OUR INSSCENE- 15K DATASET
Fig. 17 presents the vanilla masks and the refined counterparts, together with the IDs that estab- lish the correspondence between them. The refined masks contain fewer unannotated regions and align more closely with the actual objects. Training with these high-quality instance-level masks facilitates more accurate instance-level segmentation and tracking. Meanwhile, we provide a dataset card to highlight the uniqueness of our dataset, as shown in Table 6 While web-captured datasets such as RE10K (Zhou et al.) 2018) offer large number of scenes (i.e.. high diversity). they lack depth and 3D-consistent instance masks. For RGB-D datasets like ScanNet (Dai et al. 2017) and ScanNet++ (Yeshwanth et al. 2023), although they provide accurate depth measurements, they only annotate coarse 2D masks (as shown in Fig. (7) and contain only around 1,000 scenes. For synthetic datasets like Infinigen (Raistrick et al., 2024), although they provide precise instance masks, most object assets are reused with relatively simple structures, which also leads to limited diversity. As for our InsScene-11 15K dataset, build upon
annotation pipeline re-annotate ScanNet++ and REIOK with instance masks, ensuring that our dataset provides high-quality poses, depth, and instance masks at scale (around 15,000 scenes).
A.7 ABLATION OF 3D-CONSISTENT CONTRASTIVE SUPERVISION
As shown in Tab. 7 we perform ablation study on ScanNet++ dataset regarding the weighting coefficient À of the contrastive loss Lave. with the weight of the geometric losses fixed to 1. Specifi- cally, we scale À by factors of x2 and x 10 to analyze its impact performance. Here, our method maintains comparable performance under the x2 setting (i.e., X = 0.5, ), demonstrating the ro- bustness of the overall training procedure. In contrast, under the À = 0.1 setting, the contrastive
17

---

Published as a conference paper at ICLR 2026
RGB Image
R
GT Labels
Our Results
RGB Image
GT Labels
Our Results
Figure 14: Visualization of our manually annotated tracking GT and our tracking results.
18

---

Published as a conference paper at ICLR 2026
Mci so Polors
Semande 30 Felon
Venultattee Semantic Points
Vanal Cempurionn for miel Evaleatinn
Ours
Figure 15: Visualization of the pipeline from RGB 3D points to semantic labeling, voxelization, and voxel comparison for 3D mloU.
Ours
LSMiMtuti
Fenture 30GS
RCG Prions
Somantic Paints
Figure 16: We visualize the RGB and semantic 3D points of the ground truth, (GrTs) LSM(Multi Views), and -DGS.
supervision fails to provide sufficiently discriminative instance features, leading to degraded mask quality in the post-processing clustering stage, resulting big drop in understanding performance (10.44% in 2D mloU). Meanwhile, under the À = 10 setting, the contrastive supervision becomes dominant and weakens the geometric supervision, leading degraded geometric reconstruction per- formance (a 6.28% drop in 7) as well as a decline in 3D segmentation accuracy (a 4.96% drop in 3D mloU).
Table 6: Comparison of Different Datasets. Here, we evaluate these datasets along five dimen- sions: RGB images, camera poses, depth, instance masks, and diversity. Datasets with good-quality annotations in a given dimension are marked with V", while those without such annotations, or with low-quality ones, marked with 'X"
Dataset
RGB
Pose
Depth
Instance Masks
Diversity
RE10K
X
ScanNet
X
X
ScanNet++
/
X
X
Infinigen
X
InsScene- 15K (Ours)
/
19

---

Published as a conference paper at ICLR 2026
RGB Images on Scannet++
Vanilla 2D GT Masks
InsScene- 15K Masks (Ours)
Figure 17: Comparison between vanilla ScanNet++ GT masks and our refined results.
Table 7: Ablation study with different values of À of contrastive supervision.
Metrics
0.1
0.5
10
2D mloU
20.63
31.07
32.60 20.14
31.31
31.86
3D mloU
13.89
19.79
20.01
15.18
85.35
85.43
85.66
85.19
79.38
A.8 CLASS-AGNOSTIC SEGMENTATION EXPERIMENTS
Following recent 3D instance segmentation works (Yin et al. 2024 Yang et al. 2024 Huang al., 2024c), we have also provided a class-agnostic instance segmentation evaluation on the ScanNet dataset, comparing our method against VGGT + Graph Cut and VGGT + SAI3D (Yin et al. 2024). For VGGT Graph Cut, we first use VGGT to reconstruct the entire scene point cloud from the input images, then compute normal vectors for the points and generate a mesh using Poisson reconstruction (Kazhdan al.] (2006). Finally, following SAI3D, apply normal-base graph cut algorithm (Felzenszwalb & Huttenlocheri (0044) to over-segment the mesh into segmentation results (superpoints). For VGGT + SAI3D, the pipeline directly built on VGGT Graph Cut, and the predicted depth, poses, and superpoints are then fed into SAI3D to obtain the final class-agnostic instance segmentation results.
We follow the ScanNet instance segmentation benchmark to evaluate class-agnostic instance seg- mentation. First, use the ground-truth depth and camera poses project the ground-truth 2D stance masks into point clouds, obtaining the ground-truth truth class -ans-agnostic 3D instance masks. Subse- quently, we align the point clouds predicted by IGGT and VGGT with the ground-trute -tuuh point clouds, and then apply nearest-neighbor matching to establish one-to-one correspondence between the pre- dicted and ground-truth class-agnostic 3D instance masks. Additionally, we followed the practice of SAI3D by ignoring instances representing 'wall' and 'floor'.
We report the average precision scores at loU thresholds 0.25 (AP25) and 0.50 (AP50), as well as averaged over IoU thresholds from 0.50 to 0.95 in increments of 0.05 (AP) in Tab, 3 Our method significantly outperforms graph- based grouping approaches such as VGGT+Graph Cut across all metrics, achieving an 8.83 improvement in AP. In Fig. [8] present the RGB point clouds, the ground-truth class-agnostic 3D instance segmentation, as well as the class-agnostic 3D instance segmentation results predicted by GrGh+Graph Cut, VGGT+SAI3D, and IGGT(Ours), allowing
20

---

Published as a conference paper at ICLR 2026
for direct visual comparison. Meanwhile, since this graph cut algorithm requires mesh input, a substantial amount of time is spent on mesh generation, whereas our approach can directly output multi-view consistent instance features for clustering, reducing the overall runtime by approximately 8 minutes. Moreover, our method even approaches the performance of the per-scene optimization method VGGT+SAI3D, while achieving nearly 5x reduction in runtime (2.5 min 12.2 min). This further showcases the effectiveness of our unified framework, which jointly learns geometric reconstruction and instance understanding.
21

---

Published as a conference paper at ICLR 2026
RGB
VGGT Graph Cut
IGGT (Ours)
Instance
VGGT SAI3D
RGB
VGGT + Graph Cut
IGGT (Ours)
GT Instance
VGGT SAI3D
Figure 18: Additional Visualization of the RGB point clouds, the ground-truth 3D instance segmen- tation (where black indicates ignored instances), and the predicted 3D instance segmentation from VGGT+Graph Cut, VGGT+SAI3,, and IGGT (Ours).
22

---

Published as a conference paper at ICLR 2026
A.9 RUNTIME ANALYSIS.
Here, we use 10 images from single scene to evaluate the detailed runtime, as reported in Tab.
nven-Vocabulary Semantic Segmentation. For NeRF and 3DGS methods, they require massive time consumption for per-scene optimization. Moreover, they also require camera poses input, which are typically estimated using COLMAP. For the LSM Mullii-Viwws method, building on its two-view inference,
further feed all images into Dust3R infer the camera parameters and point maps for each view, and then apply the global aligmment strategy of Dust3R to optimize the point maps across all views. Although takes only about 30 seconds, its open-vocabulary semantic segmentation performance is inferior to ours, and is unable to produce class-agnostic instance segmentation results.
Class- Class-Agnostic Agnostic Instance Segmentation. For the VGGT+Graph Cut method, we adopt the normal- based graph cut algorithm provided by the ScanNet official code. We compute normal vectors for the point clouds predicted by VGGT and generate a mesh using Poisson reconstruction. These additional steps are consuming and introduce significant overhead. For the VGGT+SAI3D method, which is built upon the GrGT+Graph Cut pipeline, substantial portion of the runtime is similarly spent on mesh processing. For our method, most of the runtime is spent on the HDBSCAN clustering step, which takes about two minutes. Nevertheless, our approach remains the second fastest among all compared methods.
A.10 ADDITIONAL VISUALIZATION ON GRANULARITIES
Since our method does not support prompt-basee based inputs like SAM or Cemantic-SAM, it is unable to directly predict instance features at different granularities. However, we can still obtain instance masks different scales by adjusting the parameters of HDBSCAN, As illustrated in Fig. [9| we showcase three different clustering granularities, where the red circle areas are the discriminative region.
RGB
PCA
Small
Medium
Large
Figure 19: Visualization on clustered masks with different granularities.
A. ADDITIONAL VISUALIZATION ON 3D VQA
As shown in Fig. 20 we showcase two tasks, object counting and spatial relation reasoning, derived from ScanRefer Chen et al. 2020) and ScanQA (Azuma et al. 2022). When provided with the masks generated by our method, Gemini 2.5 Pro successfully completes both tasks with high success rate. In contrast, the success rate drops significantly without our masks, highlighting the importance and effectiveness of explicit spatial parsing for general 3D scene understanding.
23

---

Published as a conference paper at ICLR 2026
Our method
Vanilla Gemini 2.5 Pro
Lock thans tragpe. Hew many Chains
imagee
numbe cauntin Po murmat granna ex "Coent whhrre & numbis..
"Cuanti X"
Count:
Buccess Rate 3/6
(a) Object Counting Task ScanRafer (Scene0011,
Our method
Vanilla Gemini 2.5 Pro
If the blue cash paject is blaok sound where located? Former
Where sound system located
Pormet your araver sa: "On the where Xi the uspect nunts.
X
On the table
Success Rate: 5/5
On the desk
Succesa Rate 4
(b) Spatial Relation Task ScanQA (Scene0010
Figure 20: Demonstration of 3D VQA tasks sampled from ScanRefer (Chen et al. 2020) and
ScanQA Azuma al. 2022
A.12 OUT-OF-DISTRIBUTION (OOD) RESULTS
To demonstrate the generalization capability of our model, we evaluated our model on: Outdoor scenes (ETH3D (Schops al. 2017)), Autonomous driving scenes (Waymo Open Dataset (Sun et al. ). Egocentric-view data (robotics data and a self-collected data), and low-light scenario. These visualization results are shown the appendix. As shown, our method is able correctly handle outdoor scenes (Fig. 21 Fig. 22 low-light scene (Fig. [23), and even capture short-term dynamic motion in both robotic manipulation and egocentric scenarios (Fig. (4(a)). In the robotic scenario, the motion of the robotic arm during the grasping sequence is correctly tracked (blue mask), while in the egocentric scenario, the cardboard box is also consistently tracked (blue mask). These diverse test cases demonstrate the robustness of our method, especially considering that it is trained only on static scenes. However, as shown Fig. (4(h). b), for the fisheye scenario in egocen- tric data, our method, similar to VGGT, encounters challenges in producing geometrically accurate reconstructions due to the more complex intrinsic parameters of fisheye cameras.
A.13 LIMITATION
Our method adopts an unsupervised clustering strategy on the proposed Instance-Grouue Grounded Clus- tering for post-processing. As result, the accuracy of object boundaries in the clustered masks cannot yet rival that of state-of-the-art segmentation models (e.g., SAM2 (Ravi et al. (024)). Future
work may integrate stronger DETR-basee bbsed (Cheng et al. 2022 instance heads and larger annotated datasets improve segmentation accuracy.
24

---

Published as a conference paper at ICLR 2026
RGB Images
PCA Visualization
Pred. Depth
Clustered Masks
RGB Images
PCA Visualization
Pred. Depth
Clustered Masks
Figure 21: Visualization of outdoor scenes from the ETH3D dataset (Schops et al. 2017
RGB Images
PCA Visualization
Pred. Depth
Clustered Masks
RGB images
PCA Visualization
Pred. Depth
Clustered Masks
Figure 22: Visualization autonomous driving scenes from the Waymo Open Dataset (un et al, 2020).
25

---

Published as a conference paper at ICLR 2026
RGB Images
PCA Visualization
Pred. Depth
Clustered Masks
Figure 23: Visualization of a low-light scenario. For comparison, the corresponding original scene under normal lighting conditions provided in Fig.
RGB Images
PCA Visualization
Pred. Depth
Clustered Masks
(a) Robotic Dynamic Manipulation
RGB Images
PCA Visualization
Prod. Depth
Clusterad Masks
(b) Egocentric Video (Fisheye Camera)
Figure 24: Visualization of dynamic data, including a robotic scenario and egocentric scenario (with fisheye camera).
26


Reviews:
[{"review_id": "1lE6bWIU39", "reviewer": "Reviewer_GKSC", "summary": "This work proposes a unified framework that jointly performs 3D reconstruction and class-agnostic instance segmentation from multi-view 2D images. The approach is enabled by introducing a new large-scale dataset that includes RGB images, depth maps, camera poses, and 3D-consistent instance masks, facilitating consistent supervision across views. \n\nTo achieve this, the authors design a unified transformer architecture that encodes multi-view RGB images into a shared latent space, from which two task-specific heads decode:\n\t1.\ta Geometry Head, predicting 3D point or depth maps, and\n\t2.\tan Instance Head, producing class-agnostic instance segmentation fields.\n\nTo guid the training 3D-Consistent Contrastive Learning strategy, which enforces multi-view consistency in both geometry and instance features by contrasting instance representations across different viewpoints of the same scene.\n\nIt also introduces InsScene-15K, a new large-scale dataset that combines both synthetic and real-world data, including RGB images, depth maps, camera poses, and 3D-consistent instance segmentation masks.\n\nExperiments on ScanNet validate the method’s improved performance in 3D reconstruction, instance tracking, and open-world segmentation tasks.", "questions": "- The quality of the instance labels obtained from the network is crucial part of the contribution. More information on this aspect would be necessary to confirm the reliability of the results.\n\n- The definition of the Spatial Tracking task is not entirely clear. Spatial trackers typically refer to temporal tracking in dynamic videos, where both scene motion and camera motion coexist. In static multi-view setups, pixel correspondences are largely determined by camera movement. From the visualizations, it appears to refer to instance matching across views? More clarity on this point would be helpful.\n\n- How is the granularity of the instance masks determined? Is this property controllable during inference or training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important and timely problem—jointly performing 3D reconstruction and instance-level scene understanding. This capability is highly relevant for downstream applications in robotics, AR/VR, and general 3D scene analysis, where both geometry and object-level understanding are required.\n\n- Producing instance masks alongside reconstruction is meaningful for many robotic and perception tasks, as most downstream reasoning and manipulation systems operate at the instance level rather than on raw pixels or voxels. The proposed framework aligns well with this need.\n- The authors contribute a well-curated, large-scale dataset (InsScene-15K) that includes RGB images, depth maps, poses, and 3D-consistent instance annotations. This dataset fills an important gap and could serve as a valuable benchmark for future research.\n- The proposed joint learning strategy—where instance-level understanding and geometric reconstruction reinforce each other—is conceptually sound and empirically validated. \n- The results showing improvements across multiple downstream tasks (spatial tracking, open-vocabulary segmentation, and scene grounding), demonstrating the model’s effectiveness and generality.", "weaknesses": "- An ablation study on the 3D-Consistent Contrastive Loss is missing. Since this loss is central to the paper’s claim of mutual learning between geometry and instance representation, its explicit contribution should be quantified through targeted ablations.\n- The paper lacks a standard class-agonistic instance segmentation evaluation. Common metrics such as AP, AP50, and AP25 on ground-truth instance labels—widely used in the 3D instance segmentation literature (e.g., SAI3D, SamPart3D, OpenIns3D)—would provide a more rigorous assessment of instance mask quality. Given that instance quality directly impacts downstream tasks such as scene grounding and spatial tracking, this omission weakens the quantitative validation.\n- 3D instances can often be extracted directly from point clouds using clustering or graph-based grouping methods (e.g., VGGT + standard 3D clustering (SuperPoint or Graph Cut). Therefore, evaluating the quality of class-agnostic instance masks against such baselines would strengthen the paper’s claim that joint learning between geometry and instance representation is beneficial.\n- A runtime analysis could be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tU7RolRayO", "reviewer": "Reviewer_iRBB", "summary": "This paper proposes the Instance-Grounded Geometry Transformer (IGGT) that unifies spatial reconstruction and instance-level contextual understanding in a unified transformer. Using only 2D inputs, IGGT can encode a unified representation with geometric structures and instance-grounded clustering. The authors also construct a dataset named InsScene-15K with instance-level mask annotations.", "questions": "Please open-source the code and dataset as soon as possible to enable community validation. \nI will also update my assessment after considering feedback and technical insights from other reviewers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I think paper is about the incremental improvements over VGGT by unifying spatial reconstruction and instance-level contextual understanding, which is interesting.\n2. The writing and presentation are good, and both quantitative and qualitative experiments have been conducted in sufficient detail.", "weaknesses": "1. The discussion of related work is somewhat insufficient. Several recent and more advanced 3D Multimodal LLMs—such as **Inst3D-LMM (CVPR 2025)** and **Chat-Scene (NeurIPS 2024)**—are not discussed. A more comprehensive review would strengthen the paper.\n2. Do the authors consider evaluating IGGT on more general 3D scene understanding tasks to further demonstrate its effectiveness—for example, 3D visual grounding on ScanRefer or Multi3DRefer, and 3D VQA on ScanQA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81UQVXj2HE", "reviewer": "Reviewer_Jjwe", "summary": "The paper presents IGGT, a unified transformer that consumes multi-view RGB and jointly predicts geometry (cameras, depth, point maps) and instance-grounded features suitable for open-vocabulary 2D/3D semantics and downstream multi-modal reasoning. A shared backbone drives two heads (Geometry / Instance). A window-shifted cross-modal fusion injects geometric context into the instance stream, and a 3D-consistent contrastive objective aligns instance identity across views. The authors also curate InsScene-15K, a collection with multi-view images, poses, depths, and multi-view-consistent instance masks assembled via a practical SAM/SAM2-based workflow.\n\nThe contribution can be summarized:\n1. A unified transformer with shared tokens and two heads (Geometry / Instance).\n2. A window-shifted cross-modal fusion to inject geometric cues into the instance stream.\n3. A 3D-consistent contrastive loss to keep instance identities aligned across views.\n4. InsScene-15K: practical multi-view data with pose/depth and multi-view-consistent instance masks (SAM/SAM2-based curation).\n5. Competitive results on ScanNet and ScanNet++, plus an easy interface to VLM/LMM tools via instance masks.", "questions": "1. Is your model finetuned from VGGT or trained from scratch? \n2. A dataset card to detail the strength and uniqueness of InsScene-15K will be good. \n3. Please fix the typos in the paper. \n4. I saw that all code and checkpoints will be publicly available, will the data also be public?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The dataset contribution in this 3D understanding community is pretty essential. InsScene-15K is a practical, geometry-aware, multi-view dataset with view-consistent instance IDs that scales via SAM-based curation, supports tracking/OVS/reconstruction in one place, and cleanly interfaces with VLM/LMM grounding, filling a real gap for unified 3D perception. \n\n2. Following VGGT in the geometry representation, IGGT uses multi-view images encoder to unifythe visual token represeentations. Additionally, it added an instance head to output the dense instance features. A Cross-Modal Fusion Block is used to fulfill this downstream head. \n\n3. The results are genuinely promising: on ScanNet/ScanNet++ it meets or exceeds strong baselines in instance spatial tracking (T-mIoU/T-SR), open-vocabulary 2D/3D segmentation, and geometric reconstruction, with clean qualitative masks and stable depth. The single-pass design (cameras, depth, point maps, instance features) is practical, and the instance masks plug neatly into VLM/LMM grounding. The reconstruction results could be comparable with VGGT and in some dataset is better, the semantic results are also good.  \n\n4. The idea is simple but efficient and perfroms good. \n\n5. Overall the paper is well-written and the figures are nice.", "weaknesses": "1. Some typos:\n(1) line 198: \" 3) a 3D consistent supervision to\" ? To what? Super curious about the following sentence. \n(2) In Fig2, I think you are refering to InsScene-15K. But the pie chart and the right column is ScenePart-15K \n(3) Scannet --> ScanNet ; Scannet++ --> ScanNet++\n2. There are some new baselines in such kind of unified model for semantic understanding. SceneSplat [1,2] trains a large model to take Gaussian Splatting in and open-vocabulary semantic out, and got a great performance on ScanNet and ScanNet++. \n\n[1] SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining\n[2] SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting\n\n3. No outdoor/egocentric/fisheye/low-light results. Can you add some results for the out of distribution data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)