SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment
Authors: Qi Xu
Dongxu Wei
Lingzhe Zhao
Wenpu Li
Zhangchi Huang
Shunping Ji
Peidong Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D Reconstruction
Scene Understanding
Gaussian Splatting
Feed-Forward
Multi-Modality
Primary area: applications
TL;DR: SIU3R: the first alignment-free framework for generalizable simultaneous understanding and 3D reconstruction from unposed images.

Abstract: Simultaneous understanding and 3D reconstruction plays an important role in developing end-to-end embodied intelligent systems.
To achieve this, recent approaches resort to 2D-to-3D feature alignment paradigm, which leads to limited 3D understanding capability and potential semantic information loss.
In light of this, we propose SIU3R, the first alignment-free framework for generalizable simultaneous understanding and 3D reconstruction from unposed images.
Specifically, SIU3R bridges reconstruction and understanding tasks via pixel-aligned 3D representation, and unifies multiple understanding tasks into a set of unified learnable queries, enabling native 3D understanding without the need of alignment with 2D models.
To encourage collaboration between the two tasks with shared representation, we further conduct in-depth analyses of their mutual benefits, and propose two lightweight modules to facilitate their interaction.
Extensive experiments demonstrate that our method achieves state-of-the-art performance not only on the individual tasks of 3D reconstruction and understanding, but also on the task of simultaneous understanding and 3D reconstruction, highlighting the advantages of our alignment-free framework and the effectiveness of the mutual benefit designs.

SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment
Qi Xul.2 Dongxu Wei Lingzhe Zhao?, Wenpu Li?, Zhangehi Huang?
Shunping Ji Peidong Liu
Wuhan University
Westlake University
Westlake Institute for Advanced Study *Zhejiang University
Project Website: https: ht//innsomiii/ aaac. github..hh......... ieriio/siiii s/urrr
Understanding
Sparse Views views
Understanding
Task:
or SDGS
Rasterize
only)
Rudiance Field
2D Features (predicted)
semaniic
& SIU3R Model
semantic
lextveffened
Uniffed Queries
testeclioned
DD-to33 Feature Aligament
invaanes
Dense Views
VLM
2D Features Compres
Fentures
punotic
SIUER Field
J panopiic
(e.g..-000 images)
(GT)
(compressed)
Speed:
Reconerration
Unnkerttnding
Speed:
Understanding Capabilitt
Optimication
Mutual Benefit Mechanism
seronds
(a) Feature Aligmment Paradigm Previous Methods
(b) Our Alignment- Parudigam
"Replace table with woodin
Semantie Say
the window"
Move tables to the right"
"Re-cllrriee the tuble of the right siide"
Panopric Seg.
Texi-Refenedd Seg.
Multi-Takk 3D Understanding
SIU3R
Versatile Editing
(c) Our SIU3R Applicatios
Figure Simultaneous Scene Understanding and 3D Reconstruction (SIU3R). (a) 2D-to-3D Feature alignment paradigm of previous methods. (b) Alignment-fre frre paradigm of our SIU3R method.
(c) Versatile 3D reconstruction, understanding (multi-tar task 3D segmentation) and editing applications of our SIU3R method.
Abstract
Simultaneous understanding and 3D reconstruction plays important role in developing end-to-end embodied intelligent systems. To achieve this, recent ap- proaches resort to 2D-to-3D feature alignment paradigm, which leads to limited 3D understanding capability and potential semantic information loss. In light of this, we propose SIU3R, the first alignment- freee framework for generalizable simultaneous understanding and 3D reconstruction from unposed images. Specifi- cally, STU3R bridges reconstruction and understanding tasks via pixel-aligned 3D representation, and unifies multiple understanding (segmentation) tasks into a set of unified learnable queries, enabling native 3D understanding without the need of
"Qi Xu and Dongxu Wei contributed equally: Corresponding author; This work was performed when Qi
Xu was an intern at Westlake University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

alignment with 2D models. To encourage collaboration between the two tasks with shared representation, we further conduct in-depth analyses of their mutual bene- fits, and propose two lightweight modules to facilitate their interaction. Extensive experiments demonstrate that our method achieves state-of-the performance not only on the individual tasks of 3D reconstruction and understanding, but also on the task of simultaneous understanding and 3D reconstruction, highlighting the advantages of our alignment-free framework and the effectiveness of the mutual benefit designs.
Introduction
In recent years, 3D scene reconstruction has achieved unprecedented quality and efficiency through differentiable rendering Sccniques([[12] while scene understanding has seen significant advance- ments via point cloud- ( or v approaches powered by large-scale learning. Despite their individual successes, a critical gap remains: current frameworks often treat recon- struction and understanding as separate tasks, hindering the development of end-to-end embodied intelligence systems. In light of this, recent work (2433 has endeavored to bridge these two tasks for simultaneous understanding and 3D reconstruction within unified framework.
Current approaches to simultaneous understanding and 3D reconstruction typically follow a 2D- to-3D feature alignment paradigm (Fig a). These methods [3 first extract 2D features from pre-trained 2D vision-language models (c.g., [L] LSeg (15)), then align and fuse them into 3D geometric representations (c.g., neural radiance [i or 3D Gaussians (2), through per-scene iterative optimization. The resulting 3D language field jointly encodes scene geometry and semantics, enabling language-guidn guuddddd 3D segmentation via similarity matching with textual features. However, the per-scene optimization leads to heavy expenses for these methods, where each new scene requires dense image captures and hours of training align 2D features with 3D structures through feature rasterization, severely hindering real-world deployment. The only exception is Large Spatial Model (LSM) [16), very recent method that employs large reconstruction model and achieves generalizable 2D-to-3D feature alignment by predicting 3D Gaussians and their features in a single forward pass.
However, the aforementioned approaches inherently have the following limitations due to the nature of 2D-to-3D feature alignment. 1) Limited instance-Level understanding: Existing feature alignment methods utilize off-the-shelf vision-language models that fall short in identifying instances, resulting in limited capability for instance-level understanding tasks such as instance and panoptic segmentation. 2) Information loss in feature compression: To efficiently embed 2D features into 3D representations and save the memory cost during feature rasterization, existing methods usually need to compress features to lower dimensions (e.g., from 512-dim 64-dim (16). Such compression discards fine-grained semantics, degrading performance on tasks requiring precise 3D understanding.
To address the challenges outlined above, we propose SIU3R, novel generalizable framework achieving SIMULTANEOUS UNDERSTANDING and 3D RECONSTRUCTION beyond feature alignment (Fig [b). At its core component, SIU3R introduces a Unified Query Decoder and Mutual Benefit Mechanism that bridge 3D reconstruction and scene understanding within a unified SIU3R field through pixel-aligned 3D representation, which enables native 3D understanding through explicit 2D- to-3D lifting rather than implicit 2D-to-3D feature aligmment. Specifically, we employ pixel-aligned representation for both reconstruction and understanding, ensuring the subsequently predicted 3D Gaussians and 2D masks can be naturally correlated with each other to enable 3D-level understanding. To ensure mask consistency across different views and understanding tasks, we propose Unified Query Decoder with a single set of learnable queries that shared across different views as well as multiple understanding tasks including semantic, instance, panoptic and text-referred 3D segmentation (Fig [] c). To further investigate the bidirectional interaction between reconstruction and understanding in the context of their shared representation, we introduce two lightweight modules to encourage mutual benefits between the two tasks, reaching the best of both worlds.
In summary, main contributions
follows:
We propose SIU3R, the first alignment-free framework for generalizable simultaneous under- standing and 3D reconstruction, which bridges reconstruction and understanding via pixel-aligned 2D-to-3D lifting, and unifies multiple 3D understanding tasks (i.e., semantic, instance, panoptic and text-referred segmentation) into set of unified learnable queries. This framework enables native

---

3D understanding without the need of alignment with 2D models, thereby avoiding limitations on 3D understanding imposed by 2D models and their feature compression.
To our knowledge, this is the first work that conducts in-depth exploration of inter-task mutual benefits in the realm of simultaneous understanding and 3D reconstruction. To encourage the bidirectional promotion between the two tasks, we incorporate two lightweight modules into our pipeline and achieve significant performance improvements in both tasks.
Extensive experiments on ScanNet[17] demonstrate that our method achieves state-of-the-art performance not only on the individual tasks of 3D reconstruction and understanding, but also on the integrated task of simultaneous understanding and 3D reconstruction, highlighting the advantages of our alignment-fret framework and mutual benefit designs.
2 Related Work
3D Reconstruction. Recent advancements utilizing techniques like Neural Radiance [ields[]] 20 and 3D Gaussian Splatting (2H233 have made remarkable progress in both reconstruction quality and rasterization speed. However, they typically require dense image captures as input and time-consuming per-scene optimization. To achieve reconstruction from sparse observations, large reconstruction models have emerged incorporate 3D representations such as neural radiance 3D Gaussians (3 and 3D point cloud 35 into neural networks, enabling generalizable reconstruction in a forwafo manner. Among these methods, DUS$3R 34 pioneers pose-free feed-forward 3D reconstruction given only two unposed images, avoiding the obstacles of acquiring camera poses in real world. By combining DUSt3R with additional 3D Gaussian head, [36] 37 can even surpass previous pose-required methods 29 30 in novel view synthesis. Inspired by this, our method also adopts a pose-free paradigm in 3D reconstruction to ensure generality.
Scene Understanding, Remarkable progress has been witnessed in both 2D and 3D understanding. For 2D understanding. LSeg[[5] achieves open open-vocabllary semantic segmentation through feature alignment with vision-language model (e.g., CLIP which lacks instance- understanding capabilities such as instance or panoptic segmentation. In parallel, MaskFormer 38 treats semantic segmentation as the task of identifying region proposals of different classes, where transformer model used to derive regions from a set of learned class queries. As extensions to MaskFormer subsequent works further achieve instance-leve understanding [39 40], as well as segmentation on videos
3 Unfortunately, the above 2D methods can only understand scenes from specific camera perspectives, lacking spatially consistent understanding the 3D level. For 3D understanding, researchers 3 41333 typically take 3D point cloud clustered into super points as input and employ proposal-bot based method, similar to MaskFormer, segment super points into 3D semantic regions or object instances. However, all of these 3D understanding methods rely on pre-scanned 3D point cloud and pre-processed super points, incurring significant costs in practical applications.
Simultaneous Understanding and 3D) Reconstruction. Despite the individual successes of the above methods, they all treat reconstruction and understanding as separate tasks, limiting the potential of building end-to-end embodied intelligence systems. Therefore, recent advances propose to embed language features into neural radiance fields 9 I0 44)46) or 3D Gaussians THE 47-51 to empower the reconstructed 3D scenes with understanding capabilities. Given dense image captures of scene, they typically process images into full-image [I 44 or object-wise [2 47
features using 2D foundation models (c.g., [4], LSeg 15), and then align 3D feature fields with 2D features by imposing losses on the feature rasterization process. Since these methods require time- consuming training for each scene, their efficiency is limited in practical applications. A very recent method Large Spatial Model (L. eliminates this drawback by performing feature alignment within a large reconstruction model, achieving generalizable 3D reconstruction and understanding for the first time. Nevertheless, all of these methods follow 2D-to-3ID feature alignment paradigm, which leads to limited understanding capability and semantic information loss.
Methodology
3.1 Problem Formulation and Pipeline
SIU3R processes sparse unposed multi-view images with corresponding camera intrinsics K"YY where V > 2 in our setting and denotes the number of input context views, and
3

---

Gaussiun Decader
100
DoO
I.K Image Eneoder
40
Unified Queries
3/ Geussion Points
Pizel-Aligned Representation
Unposed Images
STGER Field
Unified
Pixel-Aligned 2D-to-3D Lifting
"Char by wol" Reference
Text
Query
Encoder
Decoder
Cross-Piww Morks M,
(Optional)
(S2c.3.2)
Mutual Benefits (Sec. 3.3)
(Algorithm 1)
Figure 2: Pipeline. Our method consists of Image and Text Encoders for extracting multi-view and text features, Gaussian Decoder for decoding pixel-aligned 3D Gaussians, Unified Query Decoder for decoding pixel-aligned 2D cross-view masks, Mutual Benefit Mechanism for enabling bidirectional promotion between reconstruction and understanding tasks, Pixel-Aligned 2D-to-3D Lifting algorithm for obtaining SIU3R field that enables simultaneous understanding and 3D reconstruction.
learns feed-forward network Fe, parameterized by network weights O and N₂ learnable unified queries Q = {9 1 The network establishes two key outputs: 1) pixel-aligned multi-view 3D V.H Ju,ij=1
V.H.W for 3D reconstruction, where g {1,, az 8, cj is a single gaussian primitive and H, W specify the image resolution; 2) mask prediction logits M {m, } where m, E RVVxx and class prediction logits C = (cn) No
Gaussians {g
Jn-1 where Cn E No for scene understand- ing and NN. indicates total classes including background. Formally, the learning objective implements the mapping:
Fe (G.M,)
(1)
As illustrated in Fig our pipeline comprises Image Encoder, Text Encoder, Unified Query Decoder (S2c Gaussian Decoder, Mutual Benefit Mechanism (Sec and Pixel Aligned 2D-to-3ID Lifting Algorithm (Algo We design our Image Encoder following [52]* ss architecture as a Vision Transformer (ViT) enhanced with adapter module. This configuration enables simultaneous extraction geometry-focused features (f for reconstruction and semantic-focuse
features ( for understanding. The extracted semantic features {fi V along with object queries Q and optional text features frert extracted by CLIP text encoder [53) are fed into Unified Query Decoder to obtain pixel-aligned mask logits M and class logits shared across all views and all understanding tasks, enabling cross-view view and cross-task consistent mask predictions. At the same time, Gaussian Decoder consisting of a ViT decoder with a DPT he4] takes geometry features (fR) input and predicts pixel-aligned 3D Gaussians G, which are aligned with the 2D cross- view masks. To fully exploit the inherent correlation regarding the shared representation between semantic predictions (i.e., M. C) and geometric predictions (i.e.. 9), further propose Mutual Benefit Mechanism to encourage collaboration between reconstruction and understanding. In particular, promote understanding from reconstruction, module, which utilizes 3D geometric clues in G to aggregate semantic information from all views to improve cross-view consistency of M and C. Moreover, to improve reconstruction by understanding, we introduce Mask-Guided Geometry Refinement module that leverages 2D) masks to enforce intra- instance depth continuity for retining reconstructed 3D geometry. Finally, through Pixel-Aligned 2D-to-3D Lifting, we can obtain SIU3R field that supports simultancous understanding and 3D reconstruction.
propose Multi View Mask Aggregation
3.2 Unified Query Decoder
As shown in Fig (a), we employ set of learnable unified queries Q to jointly decode cross-view consistent masks for both instance and semantic segmentation tasks, where each query q.. E Q explicitly denotes a potential object instance semantic region that may appear in multiple views. To aggregate semantic logits from multi-view image features, we perform cross-attention between unified queries Q (query) and semantic-focusel multi-view features (fu }v= (key/value), followed by self-attention layer that enables inter-query correlations. This cross-/self-atteniis block is stacked L1 times to progressively consolidate semantic information across views, ultimately decoding them into multi-view mask logits M = {m 1 Ng.V.H. V.H.W and class logits C = (c.,). through linear projection. Benefiting from the shared representation between M and reconstructed multi-view 3D Gaussians G, we can correlate each m. with its 3D Gaussian counterpart g and lift it to 3D
n,0,A,j=1

---

Select (Bq.
k
Algoritian
Self Attention Layer
k
Cross Attention Layer
Cross Attention Layer xL;
Somantic-
Min :
Instunce Semandie Mosks
five :
Iia Refereed Masker
Focured
Features
Unijied Dlueries
(Clos-Vocabblary)
Text Features (optionaa)
(Open-Vocabulary)
(a) Cross-View Mask Decoding
(b) Cross-Modal Mutching
Figure 3: Unified Query Decoder. In (a). we employ multi-view semantic-focusee features fu- unified queries 2, and L, stacked cross-/self-attenioon layer blocks to decode cross-view instance and semantic masks. In (b), we employ L2 stacked cross-aftention layers to select the queries that best match the text features fteat: and further derive them into rext-referred masks.
space for multiple close-vocabulary 3D understanding tasks (i.e., instance, semantic and panoptic segmentation) according to Pixel-Aligned 2D-to-3D Lifting Algorithm derived in Algo
As shown Fig 3 (b), to enable text features fies = (figuli as input for open- vocabulayy 3D understanding, we further incorporate cross-modal matching module into Unified Query Decoder. Here we assume that our unified queries can perceive all potential object instances and semantic regions. What need do is then identify the queries that best match the text features. Specifically, we employ L2 cross-attention layers enable interaction between fie and Q. Then for each text feature ficu we can match it with the most correlated query Dleur and use its logit prediction to derive text-referred mask, which also be lifted to 3D for open-vocabulary 3D understanding following Algo same as vocablloyy tasks. The matching process derived as follows:
arg max (Attm(fieus Q) .q....
q,
where Attm(-.) denotes mross-modal attention between text feature fir: (query) and object queries Q (key/value) L2 times, followed by dot product operation with each query qn. E Q.
We also introduce the following loss in training enable matching supervision:
No.
Ciest
2) qm.,, 8%")
(3)
N2
where 6 denotes ground-truth one-hot label that indicates the best matched query. We obtain 6g by conducting Hungarian matching 33 55) between M and ground-truth text-referred masks.
3.3 Mutual Benefit Mechanism
Multi-View Mask Aggregation. Due to potential large viewpoint changes, occlusions, or lighting differences between images from different camera views, the semantic information may vary signifi- cantly. Thus, the masks directly predicted by the Unified Query Decoder may exhibit inconsistencies across different views. A typical case is shown in Fig (a), the Unified Query Decoder only considers cross-view semantic correlation of 2D pixels, without realizing that these pixels from different views probably neighbors in 3D space for the same instance. As result, the same instance is predicted
Pasli ID, GZ. ID,
Prod: ID; GT 1D,
Pred: ID; ID;
View
GT. ID,
0
Mack-Gaided
Multi- View Mask. Aggregation
Prof GT.
Prede
View 2.
S
3D Coussian
3D Goometry
3D Gloomecty (rctined)
o pisels
3D
(a) R-U: Reconstruction Helps Understanding
(b) U-R: Understanding Helps Reconstruction
Figure 4: Mutual Benefit Mechanism. (a), our Multi-View Mask Aggregation module utilizes reconstructed 3D Gaussians geometry clues to improve cross-view mask consistency in 3D understanding. (b), our Mask-Guided Geometry Refinement module employs segmentation masks semantic clues to refine geometry of reconstructed 3D Gaussians.
5

---

Algorithm 1 Pixel-aligned 2D-to-3D lifting,for for simultaneous understanding and 3D recontruction.
1" Model forward pass 1/
g <- Gaussian Decoder
b Pixel -aligned 3D Gaussians b. Last layer hidlee states of unified queries, aligned multi-view mask logits and class logits > Obiain textual Teatures of human instructions (Optional) reference
2. M.C t- Unified Query Decnder
Jou t- CLIP Text Encoder
Obtain predictions valid queries, and select query matches with
kept +- max(softmax(()) > To O arg max(soft [C) o 2
p Kept queries of high confidences and valid classes P Select the query that best matches the text reference following Eq. 2 Optional) Filter queries no interest Tum logits class confidence scores and multi-view query probability maps
eex id + Seleet( Q fwn)
e B'M''C' + 2 keppp M(Jhept_gs) C(kept_qs): N, f- len( (')
c ".M' softmax((") siggmoii sigmoid(MM")
2 + C'[None; ;, Noue, None]* M N, : b. Calculate class-wise query probubility maps 2 E V 4 x NN M x
Mulli- musk aggregation */
9" + 9 g = ax 8,
Replace original 3D Gaussian color attribute with semantic attribute by spatial-indoxing 2 b Fuse mulli semantics in 3D and propagale them back to the original views viu rasterization
2 + (asttrrize(g")
Derive cross view 2D masks
Ma I, + max 2, dim = 1)
> Derive class-wise masks and maximum-probabiliyy query indices, M,, I, E RV KNW Derive multi-view semantic masks by maximizing class confidences, Ma E RRXHXW
Mon + Ma, dim 1)
Mat + [Ma]
D- Derive multi-viow instance masks by taking M.r as class indices, Mar R V
2D-1o-3 mask lifting based
Gaussians multi-task understanding */
Iun +- (Goness sem__i [1, N.I): Samuj 4- 19 M sem__if
3D Gaussian-based semantic sogmentation
(9iu.s inr id [1, N, Sw..rr
[g MM iis_id)
D. 3D Gaussian hased instance segmentation
Jass 4- ( Gue)
b. 3D Gaussian-based panoptic segmentation
Iwn (9amst ins_id text_id3
Gaussian-based text-referred segmentation (Optional)
with different IDs across different views when semantic information alone is insufficient. Benefiting from the pixel-aligned representation shared between our predicted 2D masks and 3I) Gaussians, we can use 3D Gaussians as additional geometric clues to fuse multi-view 2D semantic information in 3D space and propagate it back to the original views to avoid inconsistency. We call this "Reconstruction Helps Understanding (R+uJ)
Specifically, we propose Multi-Viee Mask Aggregation module, which first lifts 2D semantic information (i.e., query logits M and C) from different views to the 3D Gaussians G for fusion, and then propagate them back to these views through alpha-blending mechanism inherent in the 3D Gaussian rasterization. As highlighted in [ this module only introduces minor extra computation to the vanilla 2D-to-3D lifting algorithm during the inference phase, without the need for additional training.
Notably, another design choice for the above multi-view aggregation is to apply the 3D Gaussian rasterization earlier to fu at the stage of feature encoding rather than the mask decoding, which is the core design of previous feature alignment- based methods L1 [2] [6]. However, due to extremely high dimension of 2D features, we have to perform feature compression before rasterization to avoid memory exhaustion, which inevitably corrupts the original semantic information. As demonstrated in See 6 such early aggregation not only requires additional training but also incurs significant memory overhead, yet its performance is far inferior to our training- Multi-View Mask Aggregation.
Mask-Guided Geometry Refinement. In general, adjacent 2D pixels within the same object instance or semantic region should correspond to continuous positions in 3D space. Leveraging this prior knowledge, we can geometries. We call this " "Understanding Helps Reconstruction (U-*)"' As shown 3D Gaussians corresponding to adjacent pixels within the same instance may be far apart without refinement, which can lead to unsatisfactory coarse geometry. Thus, to make 3D Gaussians within the same mask to be more clustered, we propose Mask-Guided Geometry Refinement module, which utilizes masks as guidance enforce intra-instance depth continuity based on the following loss:
mask predictions as semantic clues to refine the reconstructed 3D
Fig (b), the
K
Lovnt
(((P)
Σ D()) QENU
(4)
IN
k=IPEM;
where Mj denotes pixels belonging to the k-th mask, D(p) and D(q) represent the depths at pixel p and one of its neighbored pixel 9, and N₂, indicates the neighborhood of p within the same instance.
3.4 Training Objective
Through holistic integration of components, our framework enables end-to-end optimization across the complete learning pipeline. The overall training objective is derived as follows:
L = (g) - -ll + ALLPIS(((G)) + As + XaCcmm + As Last,
(5)
6

---

Table I: Quantitative comparisons. "f" "!" and "*" denote reconstruction-nnle understanding- only, and simultaneous scene understanding and 3D reconstruction methods, respectively. "." indicates that the corresponding method do not support the corresponding task. "R-U"" and "U-R" denote our Multi-View Mask Aggregation and Mask-Guided Geometry Refinement modules, respectively.
3D Reconstrucioo
Scene Understanding
Depti Estiaatio
Novel View Synthesis
Conlext Views (2Dy
Novel Views (3D aware)
AhcKel,
RMSEL
PSNK
SSIM [PIPS
mloU,
MAP
PQT m mlat.1
MAPT
PQT mloUi
pixelSplai 29
0.1812
0,4106
24.93
0.8065 0.2003
MVSplat300
0.1697
0.3923
23,80
0.7871
0.2284
NoPoSplat 3r)
0.0944
0.2434
25.91
0.8147
0.1878
2lsmer 39
0.5466
0,2486
0.6071
: (LSeg[13)
0.2601
0.2127
[6]
0.07468
0.2190
21.88
0.7336
0.M035
0.2745
0.1925
0.2707
0.19055
Ouurs w/o R-UU
0.07421
0.2081
25.96
0.8220
0.1841
0.5812
0.2529
0.6123
0.4572
-Ours w/o U-R
0.09619
0.2414
25.51
0.8168
0.1951
0.5893
0,2636
0.6569
0.5125
0.5875
0.2527
0.6456
0.5245
OOurs
0.07421
0.2081
25.96
0.8220
0.1841
0.5922
0.2817 0.6612
0.5273
0.5920
0,2714 0.6495
0.5270
where I and are rasterized and ground truth images, Laskk is derived from [39 55 Lmass Ace + Adice + Acts where Lee is binary cross-entropy loss, Ldice dice loss and Lats is classification loss. We follow (5555]] set Ace, Adice and Acts to 5.0, 5.0, 2.0. In our training, we
(55
leverage both photometric loss and segmentation loss to simultaneously supervise 3D reconstruction and understanding. We set Å1, 22. X3, 24, A to 1, 0.5, 0.05, 0.05, 1, respectively.
4 Experiments
4.1 Experimental Setup
Implementation Details. We utilize 17] for training and validation, the largest public dataset that concurrently provides multi-view images with dense semantic/instance segmentation labels and text-referred segmentation labels[56). We conduct training on 8 NVIDIA Geforce RTX 4090 GPUs, with our model trained for 100 epochs using a per-GPU batch size of (total batch size of 24) for about 2 hours. Adam optimizer [57 is employed with an initial learning rate of lc-4 followed by cosine decay scheduling.
Baselines. Our experiments encompass not only isolated evaluation on separate tasks of 3D recon- struction and understanding. but also integrated evaluation on simultaneous understanding and 3D reconstruction. Therefore, we evaluate our method against three types of baseline methods, all of which are ffte-off-the-rts their respective tasks: 1) Sparse-view 3D reconstruction: pixelSplat (2)). MVSplat (30, NoPoSplat [37]; 2) Scene understanding: Mask2Former 39] LSeg 3) Simulta- neous scene understanding and 3D reconstruction: LSM 16) All baseline methods are evaluated on ScanNet dataset under the same protocols as ours for fair comparison. To be more specific, for pixelSplat. MVSplat, NoPoSplat, Mask2Former, and LSM, we re-train their models following the training protocols of their official implementations using the processed ScanNet dataset same as ours. For reconstruction-only methods (i.c.. pixelSplat, MVSplat, NoPoSplat), we only use their original rendering losses for supervision. For understanding-ort only use its mask losses for supervision. For LSee, since it already possesses general language understanding capabilities, we directly adopt its pre- pre-trained weights for evaluation.
method (i.c.. Mask2Former), we
Metrics. For 3D reconstruction, we evaluate the performance from two aspects: depth estimation and novel view synthesis, using depth accuracy metrics (i.e., AbsRel and RMSE) and image quality metrics (i.e., PSNR, SSIM and LPIPS). respectively. For scene understanding, we employ distinet evaluation protocols for 2DD-baeed and 3D-based approaches. Specifically, without reconstructed 3D) structures, 2D-based methods can only perform segmentation on the input context views. Therefore. we conduct "2D-only" evaluation on context view segmentation for 2D-based methods. In contrast, 3D-based approaches to simultaneous understanding and 3D reconstruction can perform 3D-level segmentation on the reconstructed 3D structures (e.g., 3D Gaussians in LSM and our method). Thus, for these methods, we leverage their characteristic of 3D-to-2D rasterization to project 3D-level segmentation results onto 2D masks, and conduct "3D-aware" evaluation on novel views. Note that our adoption of such evaluation on novel views is due to the lack of ground-truth segmentation labels for reconstructed 3D structures, which vary across different methods even for the same scene. As for different understanding tasks, we employ mloU for semantic and text-referred segmentation, mAP for instance segmentation, and PQ for panoptic segmentation, where all metries are computed with
7

---

global IDs as ground truths to penalize inconsistencies across views. To differentiate the metrics for semantic and text-referred tasks, we denote them "mloU," and "mloU," " Table respectively.
4.2 Main Results
Quantitative Results. As shown in Table 1 approach outperforms all baselines across all tasks by clear margin. For 3D reconstruction, unlike MVSpl and PixelSplat that require camera poses as imput, LSM that relies ground-trutt toound-truth depth supervision, framework eliminates dependency on pose and depth priors while achieving superior geometric accuracy and novel view synthesis quality. For scene understanding, existing methods like LSeg and Mask2Former limited to 2D-only understanding of input context views and specific segmentation tasks. The only method
3D Reconstruction
Inpuls
GTs
Ours
NoPoSplat
LSM
MV
pixelSplat
(a) Novel View Synthesis (best viewed in enlarged resolution)
picture
toilet
bethtub
floor
hed
sofa
door
bookshelf
sink
otherfum
Close- Vocabulary Understanding
Mask2former
LSM
LSeg
(b) Semantic Segmentation (best viewed in enlarged resolution)
Open- Vocabulary Understanding
the small table. the small round table on the right by the couch.
there is leather couch, placed the side the wall.
GTs
L.SM
LSeg
Text-Referre Segmentation (best viewed in enlarged resolution)
Figure 5: Qualitative Results.
that can achieve 3D-aware understanding is LSM. However, its understanding capability is restricted by its source 2D model (LSeg) due to the nature of its feature alignment paradigm. Therefore. LSM can only support semantic and text-referred segmentation same as LSeg. Benefiting from our alignment-free paradigm and simultancous task modeling, our generalist model supports both 2D and 3D understanding on comprehensive tasks including semantic, instance, panoptic, and text-referred segmentation within unified framework, significantly exceeding other methods across all metrics. We also conduct experiments validate the generalizability of our method to more input views, unseen data domains and real-world scenarios. Please refer to appendices for more results.

---

Table 2: Comparisons between design choices Multi View Aggregation.
Reconstruction
Context Views/2D-Only)
Novel View (3D-aware)
Memory
PSNK SSIM
LPT'S.
mloU,1
mAPt
PQT
mlol,
mlol.
mAPT
PQT mlob,
VRAM
Farly Aggregate wio train.
00012
0.1015
0.1765 00016
Early Aggregate w/ train.
25.62
0.8153
0.196]
0.5442
0,2393
0.6011 0,4025
0.5434
0.2292
0,5849 0.3978
333G
Ours
25.96
0.8220
0.1841
0.5922
0.2817
0.6612 0,5273
0.5920
0.2714
0.6495 0.5270
23GB
Qualitative Results. The qualitative results also demonstrate the superiority of method. As illustrated in 5 (a), thanks to our Mask-Guided Geometry Refinement module that improves method exhibits better visual quality and fewer artifacts than others reconstructed 3D geometries, in novel view synthesis. As demonstrated in Fig (b), thanks to our simultancous task modeling and Multi-View Mask Aggregation mechanism, our method can effectively leverage geometric clues to improve understanding, and thus outperform 2D-only methods like Mask2Former with less erroneous inclusions in semantic masks. Besides, since our alignment-free framework decouples us from off-the-shelf 2D vision-language models, the quality of our segmentation masks is much better than methods like LSM and LSeg that rely on language-based querying for segmentation. Similar effects can also be observed in text-referred segmentation results (Fig (c)), where our methods significantly surpasses LSM and LSSeg with sharper boundaries and less fragments. Please refer to our appendices for more results (3D instance and panoptic segmentation, extension to versatile 3D editing, comparisons with more 3D-based baselines, real-world scenarios, etc).
4.3 Ablation Studies
Reconstruction Helps Understanding (R (R. We conduct ablations Multi- View Mask Aggregation module (denoted as "R-U"" Table We can see that this module can significantly
RGB
w/ R-U
w/o R-U
RGB
w/ R-U
w/o R-U
Figure 6: Ablation on Multi- Mask Aggregation (R (R~U)).
improve performance in both 2D-only and 3D-aware scene understanding, without sacrificing 3D reconstruction accuracy due to its training-fres nature. We attribute the improvement to our utilization of reconstructed 3D Gaussians geometric clues, which effectively enhance view mask consistency as shown in Fig 6 We also note that, without this module to predict novel-view masks via 3D Gaussian rasterization,
not enable 3D-av Daware tasks denoted by "_" in Table [
also conduct ablations on our Mask-Guide Geometry Refinement module (denoted as "U->R" in Table D.. With this module that employs mask guidance for geometry refinement, we can obtain much better 3D geometries ("depth estimation") and higher visual quality ("novel view synthesis") during reconstruction, which in turn enhances performance in "Scene Understanding" thanks to our simultancous modeling the two tasks.
Understanding Helps Reconstruction (U-RR).
we can also implement multi-view aggregation via feature rasterization the early feature encoding stage rather than mask decoding stage. However, as shown in Table such early aggregation leads to poor performance without re-training our model. To incorporate into training, we have to compress features to lower dimensions avoid memory exhaustion. Nevertheless, the performance is still far inferior to our original design, while incurs significant training-time memory overhead due to the memory-intensive feature rasterization. We attribute this to the low compatibility between 3D Gaussians and compressed
Design Choices of Multi- Aggregation. As discussed in Sec 3.3

---

2D features, which inevitably causes semantic information loss during feature rasterization, and in turn has negative impact on the intricate forward process of our Unified Query Decoder.
5 Conclusion
We have introduced SIU3R, the first alignment-free framework for generalizable simultaneous understanding and 3D reconstruction. SIU3R unifies multiple understanding tasks into set of unified learnable queries, which enables native 3D understanding without the need of alignment with 2D models. Two lightweight modules further bring significant mutual benefits between two tasks. Extensive experiments demonstrate the superiority of SIU3R to previous state-of-the-arr on both tasks, as well as its various applications including high fidelity 3D reconstruction, real-time native 3D understanding and versatile 3D editing.
Limitations. Currently, our SIU3R model is only trained on limited data compared to methods such as DUS(3R (34) and VGGT which hinders its generalizability to broader visual domains. Additionally, since all the cross-view mask annotations in ScanNet dataset are obtained by projecting noisy 3D point clouds onto 2D images, the quality of our ground-truth labels is relatively poor, which may have negative impact on our segmentation accuracy. We hope address this in the future by introducing higher-quality labels or employing self-supervised methods less dependent on labeled data.
Acknowledgements. This work was supported in part by NSFC under Grant 62202389, in part by grant from the Westlake Shiversity-Muyuns Joint Research Institute, in part by the Westlake Education Foundation, in part by the Hcadquarters Management Science and Technology Project of in part by the Tianjin Science and
State Grid Corporation of China (No. 52090025001-117-22 Technology Plan Project (No. 24YDPYSN00550, and in part by the Open Project Program of the State Key Laboratory of CAD&CG (Grant No. A2513), Zhejiang University.
References
[1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes Neural Radiance Fields for View Synthesis, August 2020.
[2] Bernhard Kerbl, Georgios Kopanas, Thomas I Leimkuchler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph. 42(4): 13:11-19::111144 July 2023.
[3] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216-8223, IEBE, 2023.
[4] Jiahao Lu, Jiacheng Deng, Chuxin Wang, Jianfeng He, and Tianzhu Zhang. Query Refinement Transformer for 3D Instance Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18516 18526, 2023.
[5] Wei Xu, Chunsheng Shi, Sifan Tu, Xin Zhou, Dingkang Liang. and Xiang Bai. A Unitied Framework 3D Scene Understanding. Advances in Neural Information Processing Systems, 37:59468 -.44909--999. December 2024.
[6] Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang. Yuan Zhang, and Pengfei Wan. DVIS: Decoupled Video Instance Segmentation Framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1282-1291, 2023.
[7] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007 00714, 2024.
[8] Tao Zhang, Xingye Tian, Yikang Zhou, Shunping Ji, Xuebo Wang, Xin Tao, Yuan Zhang, Pengfei Wan, Zhongyuan Wang, and Yu Wu. DVIS++: Improved Decoupled Framework for Universal Video Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1 12, 2025.
[9] Sosuke Kobayashi, Eicchi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in neural information processing systems, 35:23311 1--33111-330,,,.1111111031111113 2022.
10

---

[10] Justin Kerr, Chung Min Kim, Ken Goldberg. Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 19729-19739, 2023.
[11] Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari,, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 21676- 21685, 2024.
[12] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. LangSplat: 3D Language Gaussian Splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[13] Xingxing Zuo, Pouya Samangouci, Yunwen Zhou, Yan Di, and Mingyang Li. Fmgs: Foundation model embedded 3d gaussian splatting for holistic 3d scene understanding. International Journal of Computer Vision, 133(2)611-62) 2025.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transfcrable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 9748-8763. PMLR, July 2021.
[15] Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven Semantic Segmentation. In International Conference Learning Representations, Octoher 2021.
[16] Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijic Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, Boris Ivanovic, Marco Pavone, and Yue Wang. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024.
[17] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber. Thomas Funkhouser. and Matthias NieBner. ScanNet: Richly-Anmotated 3D Reconstructions Indoor Scenes. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2432-2443, July 2017.
[18] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Brualla, and Pratul P Srinivasan. Mep-nerf: A multiscale representation for anti- anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[19] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid- based neural radiance fields. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[20] Thomas Müller, Alex Evans. Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM transactions on graphics (TOG), 41(4):1-15, 2022.
[21] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatingg Alias-free gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19447-19456, 2024.
[22] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[23] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331-20344 2024.
[24] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.00
2023.
[25] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 4578 4587, 2021.
[26] Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualle Noah Snavely, and Thomas Funkhouser. Ibmet: Learning multi-view image- based rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699 2021.

---

[27] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18. Springer, 2025.
[28] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 140.224433.141,,, 2024.
[29] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. In Proceedings of the IFEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19457-19467, 2024.
[30] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky. Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024, pages 370-386, Cham, 2025. Springer Nature Switzerland.
[31] Tianqi Liu, Guangcong Wang. Shoukang Hu, Liao Shen, Xinyi Ye, Yuhang Zang, Zhiguo Cao, Wei Li, and Ziwei Liu. Fast generalizable gaussian splatting reconstruction from -iew stereo. arXiv preprint arXiv:2405. 12218, 2024.
[32] Stanislaw Szymanowicz, Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra- fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10208-10217, 2024.
[33] Dongxu Wei, Zhiqi Li, and Peidong Liu. Omni-scene: omni-gaussian representation for ego-centric sparse-view scene reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[34] Shuzhe Wang. Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUS(3R: Geometric 3D Vision Made Easy. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20697-20709, 2024.
[35] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding Image Matching in 3D with MASt3R June 2024.
[36] Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs, August 2024.
[37] Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Mare Pollefeys, Ming- Hsuan Yang, and Songyou Peng. No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images. In The Thirleenth International Conference on Learning Representations, October 2024.
[38] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-Pixel Classification is Not All You Need for Semantic Segmentation. In Advances in Neural Information Processing Systems, volume 34, pages 17864-17873 Curran Associates, Inc., 2021.
[39] Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- Attention Mask Transformer for Universal Image Segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[40] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yee Yen Lo, Piotr Dollar, and Ross Girshick. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[41] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, al. Openscene: 3d scene understanding with open vocahularies. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 815-824, 2023.
[42] Phuc Nguyen, Tuan Due Ngo, Evangelos Kalogerakis. Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4018 4028, 2024.
[43] Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20943-20953, 2024.
12

---

[44] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15887-1847, 2021.
[45] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recogmition, pages 9043-9052. 2023.
[46] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb EI Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised open vocabulary segmentation. Advances in Neural Information Processing Systems, 56:53433-53456, 2023.
[47] Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Jian Zhang. OpenGaussian: Towards Point Level Gaussian- based Open Vocabulary Understanding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024.
[48] Seokhun Choi, Hyeonseop Song, Jaechul Kim, Tachyeong Kim, and Hoseok Do. Click-Gaussian: Interac- tive Segmentation to Any 3D Gaussians. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Güi Varol, editors, Computer Vision ECCV 2024, pages 289-305, Cham, 2025. Springer Nature Switzerland.
[49] Guibiao Liao, Jiankun Li, Zhenyu Bao, Xiaoqing Yc, Jingdong Wang, Qing Li, and Kanglin Liu, Clip-gs: Clip-informed gaussian splatting for real-time and view-consistent 3d semantic understanding. arXiv preprint arXiv:2404. 14249, 2024.
[50] Wenbo Zhang, Lu Zhang, Ping Hu, Liqian Ma. Yunzhi Zhuge, and Huchuan Lu. Bootstraping clustering of gaussians for ciewsconnisttet 3d scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10166-10175, 2025.
[51] Yuzhou Ji, He Zhu, Junshu Tang, Wuyi Liu, Zhizhong Zhang, Xin Tan, and Yuan Xie. Fastlgs: Speeding up language embedded gaussians with feature grid mapping. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39. pages 3922-3930, 2025.
[52] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision Transformer Adapter for Dense Predictions. In The Eleventh International Conference Learning Representations, September 2022.
[53] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabricl Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings the the IEEE/CVF conference on computer vision and pattern
recognition, pages 2818 18-2229, 2023.
[54] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision Transformers for Dense Prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11179-1218, 2021.
[55] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020, pages 213-229, Cham, 2020. Springer International Publishing.
[56] Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 202-221, Cham, 2020. Springer International Publishing.
[57] Ilya Loshchilov and Frank Hutter. 17Xiv:711.0510// 2017.
Decoupled weight decay regularization.
arXiv preprint
[58] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[59] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[60] Tao Yu. Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint 030.2300.007700 2023.
13

---

[61] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[62] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov. Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robas robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2149-2159, 2022.
[63] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We have clearly stated the contributions and scope in the Abstract and Intro- duction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We describe the limitations in the Conclusion Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We establish our assumptions and proofs upon papers cited.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All relevant details are provided within the text of our paper or through the references we have cited.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We have included our code and its running instructions in our supplementary
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipps.cc/pulli///uui guwde//ddee/eee Coc//ooodeeesiseeeeeeeeee hwtpbbbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All relevant details are provided within the text of our paper or through the references we have cited.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted ablation experiments, qualitative experiments and quantitative experiments to demonstrate the advantages of the work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the type of resources used in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: have discusses the broader impact in Conclusion Section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The released model does not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the source code and datasets mentioned in the paper and adhered to the terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We committed to releasing our dataset and code as soon possible.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Appendix
In this appendix, we provide additional content to complement the main manuseript:
Appendix A
Additional Implementation Details
Appendix
Comparisons with Per-Scene Optimization Methods
Appendix
Extend Our Model to Multi- Inputs
Appendix
Comparison with Other Methods (e.g.. DUS13R, MASt3R, and VGGT)
Appendix
Additional Visualizations
A
Additional Implementation Details
A.1
Data Preprocessing
As described in 1 of our main manuscript, we utilize ScanNet 17 for training and validation. We adopt the official training and validation dataset splitting of ScanNet, and then resize and crop original images to centered images at 256 x 256 resolution. The camera's intrinsic parameters have also been adjusted accordingly. We followed (37's 's's camera conventions, where intrinsies normalized and extrinsic parameters are OpenCV-style camera-to-world matrices.
Our data samples obtained by randomly sampling context image pairs with certain overlaps. The overlap is determined by pair-wise Intersection over Union (IoU) metric as shown in Fig During training, constrain the IoU to [0.3, 0.8] to randomly select our training samples from scenes. Specifically, the loU metric can be calculated as follows:
IoU Matrix
8.8
Image Index
1000
160/ (00,088
0.4
1500
2000
Image Index
Figure I: IoU matrix of ScanNet scene0011 00
1. For a pair of images I, Iz, obtain their depths D,, D2, poses Pl, P2 and intrinsic K.
2. Unproject D2 into world coordinates and project them to D2's camera to obtain D'. Only depths satisfying D D2l 0.1 are considered valid.
3. Calculate the intersection over union ratio
IoU,, #valid projected depths
#total depths
4. Calculate IoU1-12 and 1oU2-11
5. Define the final loU as:
IoU LoU 1-2 IoUg 1
2

---

The same IoU-based sampling strategy is also adopted in our evaluation, where we select 1,860 context image pairs formulate the validation set. The curated evaluation benchmark and its processing scripts will be made publicly available for reproducing our results.
A.2 Network Architecture and Hyperparameters
In Table (a), the order from top to bottom are the network details of Image Encoder, Gaussian Decoder, Unified Query Decoder, respectively. In (b), we specify loss weights for Eq in our main manuscript, which is followed by parameters used in our training phase. To enable the Unified Query Decoder to leverage MASt3R features for scene understanding, we pre-trained the decoder on COCO dataset [59] while keeping the Image Encoder's 's weights frozen. The pre-trained weights will be publicly released to facilitate further research and development.
Network Architecture
architecture initialization
VIT encoder with Adapter S2
MAS13 35
depth VTT encoder
24
embed dim of VIT encoder
1024
heads VIT encoder positional embedding patchsize
RoPE
Image Encoder
interaction blucks of adapter attention adapter
[S. 11, 17, 23]
MSDeformAttn
attention heads of adapter
16
implanes of adapter spatial prior module
embed dim of adapter spatial prier module
1024
ref points
deform ratio
architecture
VIT decoder with DPT bead 54
initialization
MAS0R [35
depth VIT decoder
12
emhed dim of VTT decoder
768
Gaussian Decoder
heads VIT decoder
channels of DPT head
83
degree
min gaussian scale
0.5
max gaussian scale
architecture
mask deceder 38 39
queries
100
Unified Query Decoder
probubility threshold of queries Te
probability score threshold of pixels r layers for
0.3
sogmentation
(b) Hyperparameters
Loss Weights
>1, Ap, As.
1.0, 0.5, 0.05, 0.05,
learning rate scheduler epochs
Cosine
100
learning batch size on each device
le.4
Training Details
training devices
8 RTX 4090
optimizer
Adam [57
betal. betu2
0.9.
weight decay
0.05
warm up epachs gradient clip
1.0
Table I: Details of network architecture and hyperparameters. the table, "#" denotes numerical parameters. We present parameters that specify network architecture, and parameters used in our loss functions and training phase, in (a) and (b), respectively.
A.3 Implementation Details about Versatile 3D Editing
As shown in Fig of main manuscript, our simultaneous modeling scene understanding and 3D reconstruction enables diverse 3D scene manipulations through unified pixel-aligned representations, including instance removal, replacement, relocation, and recoloring.
Here we take instance removal as an example to derive the implementation of such 3D editing:
1. Conduct inference obtain SIU3R field with pixel-aligned 3D masks M and Gaussians g.

---

2. Remove Gaussians for specified instance (ID ins id);
g' = gg v..ij = ins_id]
3. The modified Gaussians g' are rendered into original context views to obtain images I', with off-the-shelf diffusion-based inpainting model [60] applied fill the removed regions while ensuring visual coherence.
4. Conduct inference once again and rebuild SIU3R field from I'.
For other 3D editing tasks (i.e. instance replacement, relocation and recoloring), adopt a similar approach powered by different diffusion models 661-63]
B Comparisons with Per-Scene Optimization Methods
We also compare our approaches to methods (i.e., Feature-3DGS[I1 and NeRF-DFF(9) that require dense view capturing and per-scene optimization. Both of the two per-scene optimization methods follow a feature alignment paradigm similar to the feed-forward method LSM[16]. where their 3D understanding capabilities are powered by off-the-shelf 2D vision language models that can only support language-guider segmentation. To enable the training of Feature-3DGS and NeRI-DFF; we uniformly select dense views (i.e., 100 images) as input and conduct per-scene optimization for each scene to align 3DGS or NeRF field with 2D features via rasterization. As shown in Table [ our method surpasses all of the feature dligmment-bsee approaches by large margin in the task of scene understanding, no matter they perform reconstruction in per-scene optimization (Feature- 3DGS and NeRF-DFF) or feed-forward (LSM) manner. Besides, benefiting from our align-free framework, our method can further enable instance-leve understanding tasks such as instance and panoptic segmentation. Furthermore, our method the fastest reconstruction speed, significantly surpassing Feature-3DGS and NeRF-DFF, and leading ahead of LSM. Considering that Feature- 3DGS and NeRF-DFF use much more training images than our method, our performance in novel view synthesis is acceptable while achieving the best depth accuracy owing to our mask-guided geometry refinement module, As shown in Fig M
we also make qualitative comparisons with these feature alignment-basse methods, where method achieves superior mask quality and semantic coherence.
Table II: Quantitative comparisons with per-scene optimization methods.
Depth Estimation
Novel View Synthesis
Scene Understanding
Efficiency
AbaRel 1
RMSE
PSNR
SSIM
LPPSS
mioU
mAP
PQ
Reconstruction Time
Reature-DDGS[1
0.1546
0.3585
28.69
0.8893
0.2171
0,3965
145_
NeRF DFF
0.1846
0.4151
20.12
0.6252
0.5136
0.3410
2.71min
[6
0.07468
0,2190
21888
0.7336
0.3035
0.2745
0.24s
Ours
0.07421
0,2081
25.96
0,8220
0.1841
0,5922
0.2817 0.6612
0.13s
C Extend to Multi-View Inputs
Building upon the insights of some works on feed-forward multi-view reconstruction (e.g.. 37 VGGT(58), make minor modifications to our model support more input views for simultaneous understanding and 3D reconstruction. Specifically, compared to the model with two views, where cross-attention is only performed between the tokens of the two views, for multiple views, we perform tross-attention between the tokens of each view and the concatenated tokens of all other views. This is then followed by our Gaussian decoder and Unified Query De- coder, which predict multi-view 3D Gaussians and masks, respectively. We re-trained four variant models with different number of input views (i.e., 2, 4, 6, views). The quantitative results, shown in the table III demonstrate that as the number of views increases, our model exhibits improved performance in not only reconstruction but also 3D-aware segmentation on novel views. We think this improvement is primarily due to the availability of additional view information, which reduces geometry uncertainty, and enhances the quality of 3D reconstruction. This, in turn, boosts 3D scene understanding performance, thanks to our nutuall-eneffi mechanism (i.e., R - U).
III

---

Table III: Quantitative results with different numbers input views.
#Views
InferTime
I
TraiVRAM
AbsRei
R
PSNR T
SSIM
LPIPS
mloU
mAP
PQ
0.067
24G
0.04
0.208
25.95
0.822
0.184
0.592
0.271
0.650
0.097s
5G
-8.85G
0.022
0.205
26.28
0.822
0.179
0.611
0.283
0.687
0.203
26.45
0.175
0. 185s
~ 7G
~ 77.355
0.071
0.201
26.65
0.838
0.170
0.620
0.289
0.697
D Comparison with Other Methods
Since DUS(3R, MAST3R, and VGGT are only designed to reconstruct scenes as 3D points rather than 3D Gaussians / masks, note that they cannot be evaluated for the tasks of novel view synthesis and multi-tase -ask understanding. The comparison results are shown in the table below.
As for reconstruction performance, although DUS:3R, MAST3R and VGGT used much more training data (DUSt3R: 9 datasets, MASt3R: 14 datasets, VGGT: 17 datasets) and ground-truth depths for explicit geometry supervision,
method is only slightly inferior to them in terms of depth accuracy. We think such performance gap is reasonable because our method learns reconstruction solely through the supervision of novel view synthesis (NVS) task and was trained only it is worth noting that our method exhibits the best depth accuracy among all the methods that support the novel view synthesis task (NVS), which focuses more the rendering quality rather than merely on the geometry quality.
single dataset. Besides,
As for computational efficiency, method exhibits faster inference speed compared to DUSt3 MAST3R, and VGGT. We attribute this to the adoption of complex point post-processing for DUSt3R and MASt3R, and the significantly larger model size for VGGT (ours: 650M, VGGT: 1.IB).
Table IV: Quantitative comparison on depth estimation task.
Method
Representation
Task
Runtime :
AbsRel
RMSE
DUSUR
Point
Depth
0.14s
0.058
0.178
MASER
Point
Depth
1.09s
0.057
0.175
VGGT 58
Point
Depth
0.17s
0,052
0.174
MVSp 30
3DGS
Depth & NVS
0.06s
0.170
0.392
NoPoSplat 37
3DGS
Depth NVS
0.06s
0.094
0.243
LSM
3DGS
Depth NVS & Segm.
0.19s
0.075
0.219
Ours
3DGS
Depth NVS & Segm.
0.07s
0.074
0,208
E Additional Visualizations
E.1
Real-World Captured Data
We collect some multi-view images in real-world scenarios using hand-held cellphones to further validate our generalization capability. As shown in Fig II we observe that our zero-shot performance in weal-world scenes is surprisingly good in both reconstruction and segmentation.
E.2 Instance, Panoptic and Text-Referred Segmentation
In our main manuscript, we have included qualitative comparisons with other methods and demon- strated our superiority in semantic segmentation. Here we provide additional qualitative results of instance and panoptic segmentation for further demonstration. As shown in Fig IV] compared 2D-based method Mask2Former(33) that leads to noisy mask boundaries, our method exhibits significantly higher mask quality. As shown in [V when performing panoptic segmentation, our method exhibits excellent mask consistency across different views and significantly outperforms other methods in mask quality. Similar effects can also be observed in text-referred segmentation results as shown Fig VT We attribute the superiority of our method to the simultaneous modeling of scene understanding and 3D reconstruction, which effectively leverages 3D geometric clues to aggregate semantic information from different views, and propagates them back to the original views to ensure cross-view consistency and improve segmentation accuracy.
IV

---

Input Views
Novel View RGB
Novel View Semantic Segmentation
Novel View
Instance Segmentation
Figure II: Qualitative result in real-world scenarios.
E.3 Depth Estimation
As illustrated Fig VII compared to other feed-forward reconstruction methods, our approach achieves significantly superior depth quality with less artifacts and better coherence. We attribute this to our mask-guided geometry refinement module, which ensures geometry consistency within the same object instances under semantic guidance of 2D masks, and thus reduces erroneous depth variations that typically observed in other approaches.
E.4 Versatile 3D Editing
As shown in Fig VIII we present comprehensive set of versatile 3D editing results, demonstrating SIU3R's potential for diverse 3D manipulation applications. Furthermore, this capability establishes an effective baseline that bridges geometric reconstruction, scene understanding and manipulation in 3D environments.

---

GTs
Ours
LSM
Feature-3DGS
NeRF-DFF
Figure III: Qualitative comparisons with per-scene optimization methods.
VI

---

RGB
GTs
Ours
Mask2former
Figure IV: Qualitative results of instance segmentation.
VII

---

RGB
GTA
Ours
MuskIoomeer
1.SM (sem
1.5rg (sem sog
Figure V: Qualitative Results of Panoptic Segmentation.
VIII

---

it is blue couch
this is a toilet. it is made of porcelain.
there rectangular black trash it in front of window.
this gray and green recycling bin. it is front cabinets.
swivel chair sits sondwhiched between
stationary
and fan. it's the closest chair the window.
Texts
RGB
GTs
Ours
LSM
LSeg
Figure VI: Qualitative results of text-referred segmentation.
RGB
GTs
Ours
NoPoSplat
L.SM
MVSplat
pixelSplar
Figure VII: Qualitative results of depth estimation.
IX

---

Original
"Re-colorize the white sofa into red"
"Move table little to the right"
"Remove Tables front of the white sofa"
"Replace Tables with wooden chairs"
Original
"Re-colorize the carpet blue"
"Move the trash bin little forward"
"Remove the toile!"
"Replace toilet with chain"
Original
"Re-colorize the table grey"
"Move the door to right"
"Remove the doar"
"Replace chairs behind the table with safa"
Figure VIII: Qualitative results of versatile 3D editing.
X


Reviews:
[{"review_id": "ulIQEtIilr", "reviewer": "Reviewer_bcN2", "summary": "This paper introduces SIU3R, a novel alignment-free framework for simultaneous scene understanding and 3D reconstruction from sparse unposed images. It tackles limitations of prior methods that rely on 2D-to-3D feature alignment, which are typically slow, require dense views, and suffer from semantic degradation due to feature compression. The proposed SIU3R replaces feature alignment with a pixel-aligned 2D-to-3D lifting mechanism, combining two key modules: a Unified Query Decoder to perform multiple understanding tasks (semantic, instance, panoptic, and text-referred segmentation), and a Mutual Benefit Mechanism that fosters collaboration between reconstruction and understanding. The framework achieves state-of-the-art results across several benchmarks, including depth estimation, novel view synthesis, and 3D-aware understanding, while being efficient and generalizable.", "questions": "1. Baseline implementation details: Are the baselines zero-shot results or fine-tuned results? If they are zero-shot, is the comparison fair? If fine-tuned, where are the implementation details?\n\n2. The baseline LSM is initialized from Dust3R, whereas SIU3R is initialized from MASt3R, with its unified query decoder pretrained on COCO (see Sup. L.32). This raises concerns about the fairness of the comparison.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Effectively unifies multiple 3D understanding tasks with a single set of learnable queries.\n\n2. Clear performance gains across all benchmarks and tasks compared to strong baselines like LSM, NoPoSplat, and Mask2Former.\n\n3. Lightweight yet effective modules (R→U and U→R) that improve results through cross-task collaboration.\n\n**Weaknesses**\n\n1. Generalization to other datasets: Baselines such as MVSplat, PixelSplat, and NoPoSplat also evaluate cross-dataset generalization. However, the proposed method only conducts in-domain evaluation on ScanNet.\n\n2. Quantitative evaluation for language grounding: Only qualitative results are provided; the actual performance remains unknown.", "was_revised": true, "final_justification": "I have thoroughly reviewed the comments from other reviewers and the authors' rebuttal.\n\nThe additional clarifications addressed some of my concerns (e.g., on training details and performance analysis). So **I will maintain my rating**. However, significant revisions are still needed, **particularly regarding further ablations on pretraining and the case study**. I recommend that the authors revise these areas if the paper is accepted. Alternatively, **rejecting the paper and reconsidering it through a new submission is also acceptable**.", "rebuttal": "We thank reviewer bcN2 for acknowledging our contribution in component designs (i.e., unified queries, R$\\rightarrow$U, U$\\rightarrow$R modules) and performance gains (i.e., surpass strong baselines across all tasks).\nOur detailed response to the feedback is as follows.\n\n**[W1] Generalization to other datasets besides ScanNet.**  \nThanks for raising this insightful concern.\n\nTo fully verify our performance on other datasets, we conducted two experiments, i.e. zero-shot generalization on datasets with different sensor characteristics (ScanNet++ & Replica), and train a general model with more datasets. The detailed results are shown in table below.\n\nIt can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.\n\nTo further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.\n\n||$\\qquad$ScanNet|$\\qquad$ScanNet++|$\\qquad$Replica|\n|:-|:-:|:-:|:-:|\n|**Method**| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑|\n|pixelSplat (train on ScanNet only)|24.93\\|0.181\\|N/A|17.91\\|0.274\\|N/A|18.12\\|0.286\\|N/A|\n|MVSplat (train on ScanNet only)|23.80\\|0.170\\|N/A|18.27\\|0.261\\|N/A|18.43\\|0.277\\|N/A|\n|NoPoSplat (train on ScanNet only)|25.91\\|0.094\\|N/A|19.76\\|0.107\\|N/A|19.88\\|0.110\\|N/A|\n|Ours (train on ScanNet only)|25.96\\|0.074\\|0.592|20.05\\|0.084\\|0.482|20.46\\|0.087\\|0.376|\n|Ours (train on ScanNet, ScanNet++, Replica)|**26.06**\\|**0.073**\\|**0.607**|**23.39**\\|**0.074**\\|**0.498**|**23.16**\\|**0.079**\\|**0.492**|\n\n**[W2] Quantitative evaluation for language grounding.**  \nWe note that we **have included such quantitative evaluation in Table 1 (measured by mIoU_t) of our initial manuscript**. The definition of mIoU_t can be found at P7-L241. For clarity, we will highlight in the caption that mIoU_t is the metric for language in our revision. Here we attach the results in the table below, where we can see that our SIU3R method outperforms strong baselines such as LSM and LSeg in both context-view (2D-only) and novel-view (3D-aware) language grounding tasks. Besides, the ablation study further demonstrates the effectiveness of our R$\\rightarrow$U and U$\\rightarrow$R modules in language grounding tasks.\n| Method | mIoU_t $\\uparrow$ (context-view) | mIoU_t $\\uparrow$ (target-view) |\n| :--- | :---: | :---: |\n| LSeg | 0.2127 | N/A |\n| LSM | 0.1925 | 0.1905 |\n| Ours w/o R$\\rightarrow$U | 0.4572 | N/A |\n| Ours w/o U$\\rightarrow$R | 0.5125 | 0.5245 |\n| Ours | **0.5273** | **0.5270** |\n\n**[Q1] Baseline implementation details.**  \nAs described at P7-L218, \"all baseline methods are evaluated on ScanNet dataset under the same protocols as ours for fair comparison\".\n\nTo be more specific, for pixelSplat, MVSplat, NoPoSplat, Mask2Former, and LSM, **we re-train their models following the training protocols of their official implementations using the processed ScanNet dataset same as ours**. For reconstruction-only methods (i.e., pixelSplat, MVSplat, NoPoSplat), we only use their original rendering losses for supervision. For understanding-only method (i.e., Mask2Former), we only use its mask losses for supervision. For LSeg, since it already possesses general vision-language understanding capabilities, we directly adopt its pre-trained weights for evaluation following previous feature-alignment-based 3D understanding methods such as LSM and LangSplat [1]. In our evaluation, all baseline methods are evaluated on the same validation set as ours. For clarity, we will add these details in our revision.\n\n[1] LangSplat: 3D Language Gaussian Splatting, CVPR 2024\n\n**[Q2] Ablations on initialization and pre-training for fair comparison with LSM.**  \nThanks for the constructive suggestion. We agree that different initialization and pre-training strategies may incur different performance outcomes.\nBy treating initialization and pre-training as two varying factors and combining them in different ways, **we obtained the following four variants of SIU3R to fully investigate the effects**: (1) SIU3R w/ DUSt3R initialization and w/o COCO pre-training, (2) SIU3R w/ MASt3R initialization and w/o COCO pre-training, (3) SIU3R w/ DUSt3R initialization and w/ COCO pre-training, (4) SIU3R w/ MASt3R initialization and w/ COCO pre-training (original version).\n\nThe comparison is shown in the table below. We can see that the metrics for reconstruction exhibit similar results in terms of novel view synthesis, indicating that the impact of different initializations (via either Mast3R or Dust3R) on performance is not particularly significant for NVS. Furthermore, SIU3R performs consistently better than LSM in terms of NVS for either initialization. \n\nSince neither MASt3R nor DUSt3R has been pre-trained on any understanding tasks, directly using their weights trained solely on the reconstruction task leads to a significant decline in segmentation performance. Despite this, SIU3R still performs competitively as LSM even not pretrained on COCO. \n\nThis aligns with the knowledge in the field of vision-based understanding. For 2D detection/segmentation methods (e.g., DETR [2], Mask2Former), it is common practice to use models pre-trained on COCO as initialization for tasks in vertical domains. Similarly, for 3D-level methods like LSM, their segmentation capabilities stem from LSeg, which is also pre-trained on datasets like ADE-20K [3], comparable in scale to COCO. By considering this, we humbly think the evaluations against LSM is fair in terms of COCO pre-training. \n\nIt also demonstrates the importance of pre-training on large-scale 2D perception datasets for improving downstream 3D understanding tasks, which could potentially provide good insight for future works.\n\n| Method | Recon. (Depth Estimation) | Recon. (Novel View Synthesis) | Understanding (Context-View) | Understanding (Novel-View) |\n| :--- | :---: | :---: | :---: | :---: |\n|  | AbsRel $\\downarrow$  RMSE $\\downarrow$ | PSNR $\\uparrow$ SSIM $\\uparrow$ LPIPS $\\downarrow$ | mIoU $\\uparrow$ mAP $\\uparrow$ PQ $\\uparrow$ | mIoU $\\uparrow$ mAP $\\uparrow$ PQ $\\uparrow$ |  \n| LSM (DUSt3R, w/ LSeg) | 0.075 \\| 0.219 | 21.88 \\| 0.734 \\| 0.304 | 0.275 \\| N/A \\| N/A | 0.271 \\| N/A \\| N/A |\n| Ours (DUSt3R, w/o COCO) | 0.092 \\| 0.257 | 25.31 \\| 0.802 \\| 0.190 | 0.251 \\| 0.150 \\| 0.252 | 0.250 \\| 0.160 \\| 0.241 |\n| Ours (MASt3R, w/o COCO) | 0.101 \\| 0.260 | 25.48 \\| 0.805 \\| 0.192 | 0.242 \\| 0.151 \\| 0.253 | 0.250 \\| 0.164 \\| 0.242 |\n| Ours (DUSt3R, w/ COCO) | **0.073** \\| **0.206** | 25.51 \\| 0.810 \\| 0.198 | **0.596** \\| 0.278 \\| 0.657 | **0.597** \\| 0.268 \\| 0.645 |\n| Ours (MASt3R, w/ COCO) | 0.074 \\| 0.208 | **25.96** \\| **0.822** \\| **0.184** | 0.592 \\| **0.282** \\| **0.661** | 0.592 \\| **0.271** \\| **0.650** |\n\n[2] End-to-End Object Detection with Transformers, ECCV 2020\n\n[3] Scene Parsing through ADE20K Dataset, CVPR 2017"}, {"review_id": "hF0v7didFH", "reviewer": "Reviewer_oN8Y", "summary": "This paper presents SIU3R, an alignment-free framework that jointly performs 3D reconstruction and 3D scene understanding from two-views, unposed images. Instead of rasterizing and aligning 2D foundation-model features in 3D, SIU3R utilizes a shared set of unified queries and lifts pixel-aligned 2D predictions into a 3D Gaussian field to drives both reconstruction and understanding. Two lightweight modules (1) the Multi-View Mask Aggregation (R→U) and (2) Mask-Guided Geometry Refinement (U→R) are proposed and enable bidirectional improvements between semantics and geometry  in a single forward pass. Experiments on ScanNet demonstrate superior  performance in 3D reconstruction and scene understanding, with fast inference speed.", "questions": "1. In Table 2 you compare “Early Aggregate w/o train” vs. “with train” vs. “Ours”. Could you clarify exactly which weights were or weren’t updated in the “with train” variant? Was the entire network fine-tuned, or only the aggregation layer? Why reconstruction quality was worse after training.\n\n2. In Fig 5(b) GT first row, what is the black part in the middle of the segemantic map, there is nothing in the RGB\n\n3. You set $ \\lambda_{1\\ldots5}=\\{1,0.5,0.05,0.05,1\\}$. How did you choose these weights? \n\n5. All visual examples (Fig 5) are from ScanNet. Could you provide some tests on in-the-wild image pairs?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strength\n\n1. Motivation: The authors highlight how prior feature‐alignment methods are both computationally intensive and prone to semantic inconsistencies, making a strong motivation for their alignment‐free framework that unifies 3D reconstruction and scene understanding.\n\n2. Novelty: The proposed unified query decoder, R→U (mask aggregation) and U→R (geometry refinement) components are novel and interesting. These components enable instance and semantic segmentation tasks with 3D reconstruction in feedforward pass and exploit geometric and semantic cues to improve each other’s quality.\n\n3. Multiple tasks: Across depth estimation, NVS, and a suite of 2D/3D segmentation tasks, SIU3R consistently outperforms the baselines in both reconstruction and scene understanding, show its effectiveness in ScanNet dataset.\n\n- Weakness\n\n1. Limited to Two Views. SIU3R uses exactly two views as input. This restriction prevents application to more general multi-view or video inputs and limits capture of global scene context.\n\n2. Single-Dataset Training/Evaluation: All experiments are conducted on ScanNet. Given ScanNet’s noisy 3D-to-2D projections and indoor bias, it is unclear how SIU3R performs on outdoor or higher-fidelity datasets (e.g. Matterport3D, Replica, ScanNet++). Its generative reconstruction ability and segmentation robustness is not clear beyond ScanNet.\n\n3.  If I understand correctly, the fixed set of unified queries limits SIU3R to a close-vocabulary regime, and cross-atten with text features gives it zero-shot capability. However, what happens when the number of instances in a scene is much larger than available query slots? How does the model handle more object masks than there are unified queries? Please correct me if I’ve misunderstood.\n\n4. The comparison with other baselines are unfair, the proposed model is overfit to one dataset, finetune other baselines on ScanNet could show more fair comparison", "was_revised": true, "final_justification": "I appreciate the authors providing a thorough response to my concerns. Most of my concerns have been addressed after the authors adding multi-view backbones and additional datasets for evaluation. I will raise my score to borderline accept.\n\nThat said, the paper still requires major revisions: particularly in motivation, backbone design, and experimental evaluation. I recommend that the authors revise it accordingly if it is accepted. However, rejecting the paper and reevaluating it through a new submission would also be acceptable", "rebuttal": "We thank reviewer oN8Y for recognizing our strong motivation, novelty, multi-task unification and evaluation.\nOur detailed response to the feedback is as follows.\n\n**[W1] Limited to two views.**  \nThanks for raising this concern.\n\nBuilding upon the insights of concurrent works on feed-forward multi-view reconstruction (e.g., VGGT), we **make minor modifications to our model to support more input views** for simultaneous understanding and 3D reconstruction.\nSpecifically, compared to the model with two views, where cross-attention is only performed between the tokens of the two views, for multiple views, we perform cross-attention between the tokens of each view and the concatenated tokens of all other views. This is then followed by our Gaussian decoder and Unified Query Decoder, which predict multi-view 3D Gaussians and masks, respectively.\nWe re-trained four variant models with different number of input views (i.e., 2, 4, 6, 8 views). The quantitative results, shown in the table below, demonstrate that **as the number of views increases, our model exhibits improved performance in not only reconstruction but also 3D-aware segmentation on novel views**.\nWe think this improvement is primarily due to the availability of additional view information, which reduces geometry uncertainty, and enhances the quality of 3D reconstruction. This, in turn, boosts 3D scene understanding performance, thanks to our mutual-benefit mechanism (i.e., R$\\rightarrow$U). We will include the quantitative results in our revision.\n\nBesides, we have also prepared some qualitative results to demonstrate our capability of handling 8 views and larger scene scales. However, due to the updated rebuttal policy, we cannot provide the results here. **We promise we will include the results in our paper and code repository in revision.**\n|No.Views|InferTime|InferVRAM|TrainVRAM|AbsRel$\\downarrow$|RMSE$\\downarrow$|PSNR$\\uparrow$|SSIM$\\uparrow$|LPIPS$\\downarrow$|mIoU$\\uparrow$|mAP$\\uparrow$|PQ$\\uparrow$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|2|0.067s|~4G|~24G|0.074|0.208|25.96|0.822|0.184|0.592|0.271|0.650|\n|4|0.097s|~5G|~38.5G|0.072|0.205|26.28|0.822|0.179|0.611|0.283|0.687|\n|6|0.140s|~6G|~56.5G|0.071|0.203|26.45|0.830|0.175|0.617|0.287|0.693|\n|8|0.185s|~7G|~77.5G|0.071|0.201|26.65|0.838|0.170|0.620|0.289|0.697|\n\nIt is also worth noting that most prior feed-forward reconstruction and understanding methods (e.g., pixelSplat, MVSplat, NoPoSplat, LSM) primarily utilized two-view images as input at the time of this paper's submission. **To maintain consistency, we adopted the same setting as these previous works in our initial submission**. Additionally, we acknowledge that recent feed-forward multi-view reconstruction methods (e.g., VGGT) can be regarded as our **concurrent works**.\n\n**[W2] Single-dataset training/evaluation.**  \nThanks for raising this insightful concern.\n\nTo fully verify our performance on more datasets, we conducted two experiments, i.e. zero-shot generalization on datasets with different sensor characteristics (ScanNet++ & Replica), and train a general model with more datasets. The detailed results are shown in table below.\n\nIt can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.\n\nTo further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.\n\n||$\\qquad$ScanNet|$\\qquad$ScanNet++|$\\qquad$Replica|\n|:-|:-:|:-:|:-:|\n|**Method**| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑|\n|pixelSplat (train on ScanNet only)|24.93\\|0.181\\|N/A|17.91\\|0.274\\|N/A|18.12\\|0.286\\|N/A|\n|MVSplat (train on ScanNet only)|23.80\\|0.170\\|N/A|18.27\\|0.261\\|N/A|18.43\\|0.277\\|N/A|\n|NoPoSplat (train on ScanNet only)|25.91\\|0.094\\|N/A|19.76\\|0.107\\|N/A|19.88\\|0.110\\|N/A|\n|Ours (train on ScanNet only)|25.96\\|0.074\\|0.592|20.05\\|0.084\\|0.482|20.46\\|0.087\\|0.376|\n|Ours (train on ScanNet, ScanNet++, Replica)|**26.06**\\|**0.073**\\|**0.607**|**23.39**\\|**0.074**\\|**0.498**|**23.16**\\|**0.079**\\|**0.492**|\n\n**[W3] How does the model handle the case that the number of instances in a scene is much larger than the number of queries?**  \nAs desribed in Table I of our appendix, we set number of queries to 100, which denotes that the maximum number of potential objects in a scene is up to 100.\nEmpirically, we observe that this setting is good enough to handle most cases in ScanNet, where **the average number of valid objects is less than 10 for two-view inputs**.\n\nHowever, as the scene scales increase, the number of objects actually present in the scene may exceed 100.\nIn consideration of this, we **increase the number of queries from 100 to 200 to accommodate scenes with more objects**, and present the comparison results in the table below.\nNote that we conduct this experiment just for verifying that our method can accommodate more objects by increasing the number of queries. Since the average number of objects (< 10) in scenes is far fewer than 200, the redundancy in queries would incur significant query-to-mask matching ambiguity during training, which leads to slight performance degradation for segmentation tasks.\n| Method | Recon. (Depth Estimation) | Recon. (Novel View Synthesis) | Understanding (Context-View) | Understanding (Novel-View) |\n| :--- | :---: | :---: | :---: | :---: |\n|  | AbsRel $\\downarrow$  RMSE $\\downarrow$ | PSNR $\\uparrow$ SSIM $\\uparrow$ LPIPS $\\downarrow$ | mIoU $\\uparrow$ mAP $\\uparrow$ PQ $\\uparrow$ | mIoU $\\uparrow$ mAP $\\uparrow$ PQ $\\uparrow$ |  \n| 200 queries |  0.073 \\| 0.206 | 25.87 \\| 0.820 \\| 0.191 | 0.569 \\| 0.244 \\| 0.658 | 0.569 \\| 0.237 \\| 0.647 |\n| 100 queries | 0.074 \\| 0.208 | 25.96 \\| 0.822 \\| 0.184| 0.592 \\| 0.282 \\| 0.661| 0.592 \\| 0.271 \\| 0.650 |\n\n**[W4] Finetune other baselines on ScanNet for fair comparison.**  \nAs described at P7-L218, \"all baseline methods are evaluated on ScanNet dataset under the same protocols as ours for fair comparison\".\nTo be more specific, **all of the baseline methods were re-trained on the same data as ours from scratch** following the training protocols of their official implementations.\nWe think such an experimental setup can ensure that our comparison is fair.\n\nFor clarity, we will highlight this in implementation details in our revision.\n\n**[Q1] Which weights were or weren't updated in \"Early Aggregate w/ train\"? Why reconstruction quality of \"Early Aggregate w/ train\" was worse than \"Early Aggregate w/o train\"?**  \n(1) **All the weights were learnable for \"Early Aggregate w/ train\"** variant during training. Specifically, for this variant, we train SIU3R model from scratch and perform feature rasterization to aggregate multi-view feature information before mask decoding. By doing this, the model could learn to adjust Gaussian parameters for better multi-view aggregation.\n\n(2) As described above, all the parameters of \"Early Aggregate w/ train\" variant could be updated after training, which inevitably causes its performance inconsistency with \"Early Aggregate w/o train\".\nAs for the minor performance degradation, the main reason is that **the predicted Gaussian parameters are used for not only reconstruction purpose but also feature aggregation purpose during training**, which can lead to suboptimal reconstruction quality. This also highlights the motivation of our training-free R$\\rightarrow$U module.\n\nWe will include the above details and discussions in our revision.\n\n**[Q2] What is the black part in the middle of the semantic map in Fig5(b)?**  \nSince the segmentation masks were originally annotated on 3D meshes rather than 2D images for ScanNet dataset, the rendered ground-truth 2D masks may occasionally have some unannotated pixels (e.g., the black part in Fig5) due to the errors of 3D mesh reconstruction.\n\n**[Q3] How did you choose the loss weights?**  \nWe set the loss weights this way because it keeps the values of each loss at approximately the same magnitude. Empirically, we find that this configuration is optimal in most cases.\n\n**[Q4] Can you provide some tests on in-the-wild image pairs?**  \nWe collected some in-the-wild images using hand-held cellphones to further evaluate our generalization capability. Remarkably, our model achieves surprisingly good zero-shot performance in real-world scenes for both reconstruction and segmentation, qualitatively matching its performance on ScanNet dataset. However, image/video results are not available during the rebuttal phase due to the updated rebuttal policy. **We promise we will include the results in our paper and code repository in revision.**"}, {"review_id": "iIgsGMrZ7p", "reviewer": "Reviewer_oS3w", "summary": "This paper introduces SIU3R, a framework for ​​simultaneous segmentation and 3D reconstruction​​ from two unposed images. SIU3R eliminates the need for 2D-to-3D feature alignment through an ​​alignment-free​​ approach leveraging ​​pixel-aligned 3D Gaussians. Specifically, there are two key components, including 1) a unified query decoder​​ that decodes multi-task (semantic, instance, panoptic, text-referred) masks from a single set of learnable queries shared across views; 2) a ​mutual benefit mechanism​​ comprising a) multi-view mask aggregation ​​ that uses reconstructed Gaussians to improve mask consistency across views without training, and b) ​​mask-guided geometry refinement that enforces intra-instance depth continuity to refine the geometry using predicted masks. Extensive experiments on ScanNet demonstrate advanced performance on individual tasks (3D reconstruction and segmentation) and validate the effectiveness of each component.", "questions": "1. In Equation 2, you use the queried text to search for the matching instance. If there are more than two instances of the same category appearing in the scene, will this solution also work?\n2. The comparison with LSM on training efficiency is notable. What are the primary factors contributing to this efficiency?\n3. What are the computational/memory bottlenecks when scaling beyond two views? Are there architectural modifications considered for efficient multi-view scaling?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1.  The proposed R→U (Multi-View Mask Aggregation) and U→R (Mask-Guided Geometry Refinement) modules are lightweight and intuitive and demonstrably improve both tasks bidirectionally. Table 1 clearly shows their positive impact.\n2. Training time (2h on 8x4090) is significantly lower than competitors (e.g., LSM's 3 days on 8xA100). The pose-free paradigm enhances practicality.\n\t\n**Weaknesses:**\n1. The experimental validation is ​​exclusively conducted on ScanNet. While ScanNet is a standard benchmark for indoor 3D scene understanding and reconstruction, this single-dataset focus raises questions about the ​​generalizability​​ of SIU3R's results. Performance on ​​outdoor scenes​​ (e.g., KIT) or datasets with ​​different sensor characteristics​​ (e.g., Matterport3D) remains unverified. \n2. The current framework is designed and evaluated exclusively on two-view inputs. The authors acknowledge this as a limitation for large-scale scene understanding. It is unclear how the architecture, particularly the unified query decoder, would scale computationally and effectively with a larger number of input views or video sequences.\n3. The ablation studies (Sec. 4.3) effectively demonstrate the individual contribution of the R→U and U→R modules. ​​However, some important experiments are missing from this paper, 1) multitask training without both R -> U  and U -> R, and 2) independent training for each task. These experiments will help you better demonstrate the effectiveness of your method.\n4. The paper ​​equates \"scene understanding\" solely with segmentation tasks​​ (semantic, instance, panoptic, text-referred segmentation). While segmentation is fundamental, ​​3D scene understanding​​ encompasses a broader spectrum, including ​​3D dense captioning​​, ​​3D visual grounding​​, and ​​3D question answering​​. SIU3R's capabilities and representations are not evaluated or discussed for these higher-level semantic tasks. This limits the claimed, \"versatile 3D reconstruction, understanding, and editing applications\" (Fig 1c, Abstract). Otherwise, I suggest the authors narrow down the claim and research scope made in this paper.", "was_revised": true, "final_justification": "This paper introduces SIU3R, a framework for simultaneous segmentation and 3D reconstruction using two unposed images. My initial concerns were primarily related to the limitations of the two-view setup, the evaluation on a single dataset, and the lack of an ablation study. The authors have effectively addressed most of these points in their rebuttal by providing additional experiments. I therefore recommend acceptance (borderline accept).", "rebuttal": "We thank reviewer oS3w for recognizing our lightweight and intuitive designs, as well as our significance in training efficiency compared to LSM.\nOur detailed response to the feedback is as follows.\n\n**[W1] Performance on datasets with ​​different sensor characteristics remains unverified**  \nTo fully verify our performance on other datasets, we conducted two experiments, i.e. zero-shot generalization on datasets with different sensor characteristics (ScanNet++ & Replica), and train a general model with more datasets. The detailed results are shown in table below.\n\nIt can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.\n\nTo further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.\n\n||$\\qquad$ScanNet|$\\qquad$ScanNet++|$\\qquad$Replica|\n|:-|:-:|:-:|:-:|\n|**Method**| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑|\n|pixelSplat (train on ScanNet only)|24.93\\|0.181\\|N/A|17.91\\|0.274\\|N/A|18.12\\|0.286\\|N/A|\n|MVSplat (train on ScanNet only)|23.80\\|0.170\\|N/A|18.27\\|0.261\\|N/A|18.43\\|0.277\\|N/A|\n|NoPoSplat (train on ScanNet only)|25.91\\|0.094\\|N/A|19.76\\|0.107\\|N/A|19.88\\|0.110\\|N/A|\n|Ours (train on ScanNet only)|25.96\\|0.074\\|0.592|20.05\\|0.084\\|0.482|20.46\\|0.087\\|0.376|\n|Ours (train on ScanNet, ScanNet++, Replica)|**26.06**\\|**0.073**\\|**0.607**|**23.39**\\|**0.074**\\|**0.498**|**23.16**\\|**0.079**\\|**0.492**|\n\n**[W2, Q3] How the architecture would scale computationally and effectively with more views. Modifications for efficient multi-view scaling?**  \nThanks for raising this concern.\n\nInspired by concurrent works on feed-forward multi-view reconstruction (e.g., VGGT), we **make minor modifications to our model and make it support more input views**.\nSpecifically, compared to the model with two views, where cross-attention is only performed between the tokens of the two views, for multiple views, we perform cross-attention between the tokens of each view and the concatenated tokens of all other views. This is followed by our Gaussian decoder and Unified Query Decoder to predict multi-view 3D Gaussians and masks, respectively.\nWe re-trained 4 variant models with different number of input views (i.e., 2, 4, 6, 8 views). The quantitative results are shown in the table below.\nWe can see that, **as No. Views increases, our model achieves better performance in not only reconstruction but also 3D-aware segmentation on novel views**. The main reason is that, with more view information available to reduce geometry uncertainty, the quality of 3D reconstruction improves, which, thanks to the design of our R$\\to$U module, in turn enhances the segmentation performance.\n\nWe have also prepared some qualitative results of handling 8 views and larger scene scales. However, due to the updated rebuttal policy, we cannot provide here. **We promise we will include the results in our paper and code repository in revision.**\n|No.Views|InferTime|InferVRAM|TrainVRAM|AbsRel↓|RMSE↓|PSNR↑|SSIM↑|LPIPS↓|mIoU↑|mAP↑|PQ↑|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|2|0.067s|~4G|~24G|0.074|0.208|25.96|0.822|0.184|0.592|0.271|0.650|\n|4|0.097s|~5G|~38.5G|0.072|0.205|26.28|0.822|0.179|0.611|0.283|0.687|\n|6|0.140s|~6G|~56.5G|0.071|0.203|26.45|0.830|0.175|0.617|0.287|0.693|\n|8|0.185s|~7G|~77.5G|0.071|0.201|26.65|0.838|0.170|0.620|0.289|0.697|\n\n**As for efficient multi-view scaling**, we believe this remains an open question within the field. However, from an engineering perspective, we can utilize FlashAttention and efficient optimizer tools (e.g., DeepSpeed ZeRO-3) to enable efficient training.\n\nWe also note that, at the time of our initial submission, most prior feed-forward reconstruction methods (e.g., pixelSplat, MVSplat, NoPoSplat, LSM) primarily utilized two-view images as input. We **adopted the same setting as these prior works to maintain consistency**. Additionally, we acknowledge that recent feed-forward multi-view reconstruction methods (e.g., VGGT) can be regarded as our **concurrent works**.\n\n**[W3] (1) Multi-task training without both R$\\to$U and U$\\to$R, and (2) independent training for each task.**  \nThanks for the constructive suggestion.\n\n(1) **We implement a variant model w/o R$\\to$U and U$\\to$R**, and show the comparison results in the table below. We can see that adding either module alone outperforms this variant model, demonstrating the effectiveness of the our R$\\to$U and U$\\to$R module designs.\n\nWe will include the results in our revision.\n\n|Method|DepthEstimation|NovelViewSynthesis|Understanding(context-view)|Understanding(novel-view)|\n|:-:|:-:|:-:|:-:|:-:|\n|R$\\to$U\\|U$\\to$R|AbsRel↓RMSE↓|PSNR↑SSIM↑LPIPS↓|mIoU↑mAP↑PQ↑|mIoU↑mAP↑PQ↑|\n|✘\\|✘|0.096\\|0.241|25.51\\|0.817\\|0.195|0.549\\|0.236\\|0.595|N/A\\|N/A\\|N/A|\n|✓\\|✘|0.096\\|0.241|25.51\\|0.817\\|0.195|0.589\\|0.264\\|0.657|0.588\\|0.253\\|0.646|\n|✘\\|✓|0.074\\|0.208|25.96\\|0.822\\|0.184|0.551\\|0.253\\|0.612|N/A\\|N/A\\|N/A|\n|✓\\|✓|**0.074**\\|**0.208**|**25.96**\\|**0.822**\\|**0.184**|**0.592**\\|**0.282**\\|**0.661**|**0.592**\\|**0.271**\\|**0.650**|\n\n(2) **We implement two variant models**, one with only photometric and depth continuity losses (**\"Recon. Only\"**), and another with only mask and text-matching losses (**\"Segm. Only\"**).\nFor the \"Recon. Only\" variant, we ablate our U$\\to$R module (i.e., mask-guided depth continuity loss) on it to validate its effectiveness on independent reconstruction task, where we use ground-truth masks as the depth guidance due to the absence of predicted masks. Note that we cannot add our R$\\to$U module to \"Segm. Only\" variant without the reconstructed 3D Gaussians, thus we only implement \"Segm. Only w/o R$\\to$U\" in the experiments. The results are shown in the table below.\n\nWe can see that, for reconstruction-only task, **our U$\\to$R module still significantly improves depth accuracy and novel view rendering quality**, demonstrating its effectiveness for independent reconstruction task. We can also see that, without reconstructed 3D Gaussians to help lifting multi-view masks to 3D, we cannot achieve 3D-aware segmentation on novel views. Moreover, without our R$\\to$U module that can leverage 3D Gaussians to improve multi-view mask consistency, the segmentation performance would also degrade significantly. These outcomes support the insight of this paper to unify understanding and reconstruction tasks into a single framework. We will include the results in our revision.\n\n|Method|DepthEstimation|NovelViewSynthesis|Understanding(context-view)|Understanding(novel-view)|\n|:-|:-:|:-:|:-:|:-:|\n||AbsRel↓RMSE↓|PSNR↑LPIPS↓|mIoU↑mAP↑PQ↑|mIoU↑mAP↑PQ↑|\n|Recon.Only w/o U$\\to$R|0.096\\|0.241|25.51\\|0.195|N/A\\|N/A\\|N/A|N/A\\|N/A\\|N/A|\n|Recon.Only w/ U$\\to$R|**0.073**\\|**0.207**|**26.05**\\|0.187|N/A\\|N/A\\|N/A|N/A\\|N/A\\|N/A|\n|Segm.Only w/o R$\\to$U|N/A\\|N/A|N/A\\|N/A|0.561\\|0.261\\|0.622|N/A\\|N/A\\|N/A|\n|Ours full|0.074\\|0.208|25.96\\|**0.184**|**0.592**\\|**0.282**\\|**0.661**|**0.592**\\|**0.271**\\|**0.650**|\n\n**[W4] Narrow down the claim and research scope.**  \nThanks for the constructive suggestion.\nIndeed, this paper primarily focuses on multi-task 3D segmentation, which falls under the narrow scope of fine-grained 3D understanding tasks and does not include broader tasks such as 3D captioning and 3D question answering.  \nTo address this, **we will narrow down our claim by replacing all occurrences of \"3D understanding\" with \"multi-task 3D segmentation\" in our revision**.\n\nNote that our initial claim on 3D understanding is inherited from previous works (e.g., OpenGaussian[1], FMGS[2]) that also focus on reconstruction-based segmentation, which is inappropriate given the scope of 3D understanding is broader than before.\n\n[1]  OpenGaussian: Towards Point-Level 3D Gaussian based Open Vocabulary Understanding, NeurIPS'24\n\n[2] FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding, IJCV'25\n\n**[Q1] What if more than two instances of the same category appear in the scene?**  \nYes. In our experiments, we observe that, if the text refers to a specific instance (e.g., \"Chair nearest to the door\"), no matter how many other instances of the same category appear in the scene, the segmentation result corresponding to the query with the highest matching similarity can effectively locate the instance. If the text refers to no specific instance but a category (e.g., \"Chair\"), the queries with topK matching similarities would find every single instance of this category, which is functionally similar to instance segmentation. We have prepared some qualitative results for verification. However, due to the updated rebuttal policy, we cannot provide here. **We promise we will include them in paper and code repository in revision.**\n\n**[Q2] Primary factors for training efficiency compared to LSM.**  \nThe main reason for our training efficiency compared to LSM is that we do not require high-dimensional feature rasterization to align 3D Gaussian features with 2D features. We find that such **feature-alignment paradigm requires much more training time to achieve convergence** than our alignment-free method as shown in the table below.\n|Method|AbsRel↓|PSNR↑|mIoU↑|\n|:-|:-:|:-:|:-:|\n|Ours vs. LSM(2hours train.)|0.074 vs. 0.107|25.96 vs. 19.56|0.592 vs. 0.174|\n|Ours vs. LSM(1day train.)|0.076 vs. 0.084|25.92 vs. 21.02|0.594 vs. 0.236|\n|Ours vs. LSM(3days train.)|0.076 vs. 0.075|25.89 vs. 21.88|0.595 vs. 0.271|"}, {"review_id": "MFOStuXZFa", "reviewer": "Reviewer_qLPk", "summary": "This paper introduces SIU3R, a novel, alignment-free framework for generalizable simultaneous 3D reconstruction and multi-task understanding. SIU3R bridges reconstruction and understanding tasks via pixel-aligned 3D representation, and unifies multiple understanding tasks into a set of unified learnable queries. Moreover, this paper further conducts in-depth analyses of the mutual benefits of reconstruction and understanding tasks and proposes two lightweight modules to facilitate their interaction.", "questions": "Based on the analysis above, there are two questions: (1) Given the limited evaluation on ScanNet, how does SIU3R perform on other diverse datasets and real-world scenarios to demonstrate its true generalization capabilities?   (2) How does SIU3R compare with state-of-the-art feed-forward methods like Dust3R, MAST3R, and VGGT in terms of reconstruction quality and computational efficiency?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis paper introduces a novel alignment-free framework for generalizable simultaneous understanding and 3D reconstruction, which bridges reconstruction and understanding via pixel-aligned 2D-to-3D lifting, and unifies multiple 3D understanding tasks (i.e., semantic, instance, panoptic and text-referred segmentation) into a set of unified learnable queries. Moreover, this work conducts in-depth exploration of inter-task mutual benefits in the realm of simultaneous understanding and 3D reconstruction. The framework is capable of handling multiple tasks, and its methodological design demonstrates innovation. The experimental results are good.\n\nWeaknesses:\n1.  Training and testing solely on a single dataset (ScanNet) is insufficient to demonstrate good generalization capability and algorithm effectiveness. Would it be possible to train a generalized network that, similar to Dust3R, possesses strong generalization abilities across different scenarios?\n2.  The paper compares performance on sparse-view 3D reconstruction tasks, but the baselines chosen are primarily methods that don't require extensive training periods, which may not constitute a fair comparison. Would it be more appropriate to compare with other feed-forward methods such as Dust3R, MAST3R, and VGGT?", "was_revised": true, "final_justification": "The responses and additional experiments have addressed most of my concerns. While the generalization capability of the method in outdoor scenarios remains to be thoroughly validated, I understand the time constraints during the rebuttal period make it challenging to conduct more experiments. Therefore, I keep my initial score of borderline accept.", "rebuttal": "We thank reviewer qLPk for recognizing our novel alignment-free framework, in-depth exploration of mutual benefits, and good experimental results.\nOur detailed response to the feedback is as follows.\n\n**For your information**, in addition to addressing the concerns mentioned in this review, we have provided more experimental results (e.g., extend SIU3R to support more input views as suggested by reviewer oN8Y, oS3w) that may be of your interest.\n\n**[W1] Would it be possible to train a generalized network that, similar to DUSt3R, possesses strong generalization abilities across different scenarios?**  \nDue to the limited time of rebuttal phase, we are unable to collect and process a large amount of datasets comparable in scale to DUSt3R (9 datasets) with both multi-view images and mask annotations.\nTherefore, we process two representative indoor multi-view datasets with different sensor characteristics (i.e., ScanNet++, Replica) as additional training / evaluation data.\nSpecifically, we conducted two experiments, i.e. zero-shot generalization on ScanNet++ & Replica, and train a general model by treating ScanNet++ & Replica as additional training data. The detailed results are shown in table below.\n\nIt can be seen that SIU3R has certain zero-shot generalization capability even it is trained solely with ScanNet. However, due to the size of training data (i.e. ScanNet), its generalization performance still has rooms to improve. Despite this, our method still exhibits better generalization performance compared to other baselines under the same training settings.\n\nTo further verify if SIU3R has performance gains with more training data, we train it with more datasets to get a general model. The results demonstrate that SIU3R achieved improved performance even the training data exhibits different characteristics. It demonstrates that SIU3R should achieve good performance as the dataset size further increases, as what VGGT (a concurrent work as ours, from CVPR'25) demonstrates.\n\n||$\\qquad$ScanNet|$\\qquad$ScanNet++|$\\qquad$Replica|\n|:-|:-:|:-:|:-:|\n|**Method**| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑| PNSR↑AbsRel↓mIoU↑|\n|pixelSplat (train on ScanNet only)|24.93\\|0.181\\|N/A|17.91\\|0.274\\|N/A|18.12\\|0.286\\|N/A|\n|MVSplat (train on ScanNet only)|23.80\\|0.170\\|N/A|18.27\\|0.261\\|N/A|18.43\\|0.277\\|N/A|\n|NoPoSplat (train on ScanNet only)|25.91\\|0.094\\|N/A|19.76\\|0.107\\|N/A|19.88\\|0.110\\|N/A|\n|Ours (train on ScanNet only)|25.96\\|0.074\\|0.592|20.05\\|0.084\\|0.482|20.46\\|0.087\\|0.376|\n|Ours (train on ScanNet, ScanNet++, Replica)|**26.06**\\|**0.073**\\|**0.607**|**23.39**\\|**0.074**\\|**0.498**|**23.16**\\|**0.079**\\|**0.492**|\n\nBesides, we collect some multi-view images in real-world scenarios using hand-held cellphones to further validate our generalization capability. We observe that our zero-shot performance in real-world scenes is surprisingly good in both reconstruction and segmentation, which is qualitatively on par with that on ScanNet dataset. Due to the updated rebuttal policy this year, we cannot provide the qualitative results here. **We promise we will include the results in our paper and code repository in revision.**\n\n**[W2] Compare with other feed-forward methods like DUSt3R, MAST3R, and VGGT.**  \nThanks for the constructive suggestion.\n\nSince DUSt3R, MAST3R, and VGGT are only designed to reconstruct scenes as 3D points rather than 3D Gaussians / masks, we note that they cannot be evaluated for the tasks of novel view synthesis and multi-task understanding.\nThe comparison results are shown in the table below.\n\n**As for reconstruction performance**, although DUSt3R, MAST3R and VGGT used much more training data (DUSt3R: 9 datasets, MASt3R: 14 datasets, VGGT: 17 datasets) and ground-truth depths for explicit geometry supervision, our method is **only slightly inferior to them in terms of depth accuracy**.\nWe think such performance gap is reasonable because our method learns reconstruction solely through the supervision of novel view synthesis (NVS) task and was trained only on a single dataset.\nBesides, it is worth noting that our method **exhibits the best depth accuracy among all the methods that support the novel view synthesis task (NVS)**, which focuses more on the rendering quality rather than merely on the geometry quality.\n\n**As for computational efficiency**, our method outperforms DUSt3R, MAST3R and VGGT by a clear margin.\nWe attribute this to the adoption of complex point post-processing for DUSt3R and MASt3R, and the significantly larger model size for VGGT (ours: 650M, VGGT: 1.1B).\n\nWe will include the results in our revision.\n\n|Method|Representation|$\\qquad$$\\quad$$\\text{ }$$\\text{ }$Task|Runtime $\\downarrow$|AbsRel $\\downarrow$|RMSE $\\downarrow$|\n| :---: | :---: |:---:| :---: | :---: | :---: |\n|DUSt3R|Point|Depth|0.14s|0.058|0.178\n|MASt3R|Point|Depth|1.09s|0.057|0.175\n|VGGT|Point|Depth|0.17s|0.052|0.174\n|MVSplat|3DGS|Depth & NVS|0.06s|0.170|0.392\n|NoPoSplat|3DGS|Depth & NVS|0.06s|0.094|0.243\n|LSM|3DGS|Depth & NVS & Segm.|0.19s|0.075|0.219\n|Ours|3DGS|Depth & NVS & Segm.|0.07s|0.074|0.208\n\n**[Q1] How does SIU3R perform on other diverse datasets and real-world scenarios to demonstrate its true generalization capabilities?**\nPlease refer to [W1] for details.\n\n**[Q2] How does SIU3R compare with DUSt3R, MAST3R, and VGGT in terms of reconstruction quality and computational efficiency?**\nPlease refer to [W2] for details."}, {"review_id": "7yE9JqkgCm", "reviewer": "Reviewer_Cqtb", "summary": "This paper introduces SIU3R,  framework for simultaneous 3D reconstruction and scene understanding from a sparse pair of unposed images. The method proposes an \"alignment-free\" approach that bridges the two tasks using a shared, pixel-aligned 3D Gaussian representation. The Unified Query Decoder uses a single set of learnable queries to perform multiple understanding tasks (semantic and text-referred segmentation). The authors also introduce a \"Mutual Benefit Mechanism,\" comprising two modules: one that uses 3D geometry to improve 2D mask consistency across views (Reconstruction -> Understanding) and another that uses 2D segmentation masks to refine the 3D geometry (Understanding -> Reconstruction). The authors demonstrate through experiments that their method achieves favorable results on both individual and combined tasks in the ScanNet dataset.", "questions": "- Could you provide a stronger motivation for the Unified Query Decoder? An ablation study comparing your approach to a simpler baseline—for instance, using an off-the-shelf 2D universal segmentation model (like Mask2Former) on each input view and then developing a method to lift these 2D masks to the 3D Gaussians—would be crucial to demonstrate the value of your proposed decoder.\n- The L_cont loss (Eq. 4) seems to heavily rely on a local surface smoothness assumption. How does this loss function affect the reconstruction of objects with non-flat or intricate surfaces? To isolate the effectiveness of this heuristic, could you perform a controlled study where this loss is applied to a vanilla 3D Gaussian Splatting framework (i.e., not feed-forward) using ground-truth semantic masks? This would provide a clearer picture of its benefits and limitations.\n- Could you please provide more detail on the implementation of this module? Specifically, Algorithm 1 suggests fusing multi-view semantics in 3D. How does the model aggregate class predictions and arrive at a single semantic label for each 3D Gaussian, particularly when faced with occlusions or conflicting predictions from different viewpoints?\n- For the paper to be self-contained, could you please explicitly define the L_mask loss function in the main text or appendix, rather than only citing external work [38, 54]? A brief description of the objective would be sufficient.", "limitations": "The authors have included a limitations section discussing the model's current constraint to two-view inputs and the impact of noisy ground-truth labels from the ScanNet dataset. This is a good start. However, the limitations I have raised in my review, such as the surface smoothness assumption in the L_cont loss and the lack of a clear mechanism for resolving multi-view conflicts in mask aggregation, are not addressed. Acknowledging these model-specific limitations would further improve the manuscript.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The work addresses a challenging and highly relevant problem at the intersection of 3D vision and scene understanding. A generalizable, pose-free model that can produce both high-quality geometry and dense semantic labels has wide-ranging applications in robotics and augmented reality.\n- The proposed \"alignment-free\" framework is  departure from recent methods that rely on distilling features from pre-trained 2D models. The idea of using unified queries to handle diverse 3D understanding tasks (semantic, instance, panoptic) within a single feed-forward pass is also an original and elegant contribution.\n\nWeaknesses:\n\n- The paper's core component for scene understanding, the Unified Query Decoder, is not sufficiently justified. It is a complex module, and it is not clear why it is preferable to a much simpler approach, such as running a state-of-the-art 2D segmentation model on each input view and lifting the results to 3D. The paper lacks a direct ablation study against such a baseline, making it difficult to assess the contribution of this specific design.\n- The \"Understanding Helps Reconstruction\" (U->R) module relies on a continuity loss (L_cont) that encourages adjacent pixels within a predicted mask to have similar depths. This assumes that object surfaces are locally smooth or planar. This heuristic is likely to be ineffective or even detrimental for common objects with complex or curved geometries (e.g., chairs, bottles, cups), calling into question the general applicability of this component.\n- Several key aspects of the method are not explained clearly. The process of Multi-View Mask Aggregation (R->U) is particularly opaque; Algorithm 1 is too high-level and does not explain how the model reaches a consensus on a semantic label for a 3D Gaussian when the input views provide conflicting information. Furthermore, the crucial segmentation loss L_mask is not defined in the paper, which simply refers to prior work, hindering the paper's self-containment and reproducibility.\n- Insufficient Experimental Validation: The ablation studies are not convincing enough to support the claims about the \"mutual benefit\" modules. The benefits of U->R and R->U are not well-isolated from the rest of the proposed architecture. More controlled experiments, as suggested in the questions below, are needed to truly validate the effectiveness of these individual components.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns. I have therefore updated my rating accordingly.", "rebuttal": "We thank reviewer Cqtb for recognizing our problem setting as well as the original and elegant contribution of our unified query design.\nOur detailed response to the feedback is as follows.\n\n**[W1,Q1] Stronger motivation for the Unified Query Decoder. Direct ablation study against a baseline that runs a state-of-the-art 2D segmentation model like Mask2Former and lifts the results to 3D.**\n\n**[Motivation]**\n\nThanks for raising this concern.\nOur Unified Query decoder is built upon the contributions of 2D segmentation giants such as Mask2Former and SAM.\nHowever, different from previous 2D methods that can only handle single-image inputs or specific segmentation tasks, our Unified Query Decoder enables our model to accommodate multi-view inputs while ensuring multi-view mask consistency, as well as multi-task understanding including instance, semantic, panoptic, and text-referred segmentation.\n\nWe note that these special designs **essentially serve the target of this work**: building a pixel-aligned representation-based simultaneous understanding and 3D reconstruction network, where the core is how to leverage pixel-aligned representation to combine multi-view masks and 3D Gaussians to achieve 3D-level understanding within a unified model.\n\n**Taking this into consideration, the motivation of Unified Query Decoder is twofold**: (1) Leverage unified queries shared across views to ensure multi-view mask consistency, which in turn ensures 3D-level segmentation accuracy. (2) Leverage unified queries shared across tasks to enable multi-task segmentation, which extends our approach to versatile 3D understanding applications.\nWe promise we will include the above discussions in our revision.\n\n**[Ablation]**\n\nThanks for the constructive suggestion.\nWe implement two models: (1) the proposed method trained with only Gaussian Decoder (Reconstruction Only), (2) Mask2Former trained with multi-view masks (Segmentation Only). Both models are re-trained under the same settings as ours. Following the suggestion, we **directly lift 2D masks from Mask2Former to the 3D Gaussians**. We refer to this approach as \"Mask2Former-GS\".\nThe comparison results are shown in the table below.\nWe can see that **both efficiency and performance of Mask2Former-GS degrade compared to the original SIU3R model**.\n\nAs for efficiency, the degradation of inference speed is caused by the extra computation from the additional image encoder of Mask2Former, highlighting the efficiency advantage of our unified model design.\n\nAs for performance, we attribute the degradation of both reconstruction quality and segmentation accuracy to the lack of encoder feature sharing between Gaussian and mask decoders, indicating an interesting potential correlation between the reconstruction and segmentation tasks, which we leave to future work for further exploration.\n\nThese observations further demonstrate our **\"idea of using unified queries to handle diverse 3D understanding tasks within a single feed-forward pass is also an original and elegant contribution\" as mentioned by the reviewer**.\n|Method|Efficiency|DepthEstimation|NovelViewSynthesis|Understanding(context-view)|Understanding(novel-view)|\n|:-|:-:|:-:|:-:|:-:|:-:|\n||Runtime↓|AbsRel↓RMSE↓|PSNR↑LPIPS↓|mIoU↑mAP↑PQ↑|mIoU↑mAP↑PQ↑|\n|Mask2Former-GS| 0.15s |0.075\\|0.209|25.74\\|0.194|0.552\\|0.250\\|0.606|0.552\\|0.249\\|0.603|\n|Ours| **0.07s** |**0.074**\\|**0.208**|**25.96**\\|**0.184**|**0.592**\\|**0.282**\\|**0.661**|**0.592**\\|**0.271**\\|**0.650**|\n\n\n**[W2,Q2] The smooth heuristic of continuity loss is likely to be ineffective for objects with complex or curved geometries. Controlled study where the continuity loss is applied to a vanilla 3D Gaussian Splatting framework using ground-truth semantic masks?**\n\n**[Effectiveness]**\n\nThanks for raising this concern. In our experiments, we only apply the mask-guided continuity loss to 3x3 neighbored pixels. We find **such small-range regularization rarely causes over-smoothness** in reconstructed 3D Gaussians, even for objects with complex or curved geometries.\nWe have also prepared some qualitative results for demonstration, and found the visualized depths are decent for objects with complex or curved geometries. However, due to the updated rebuttal policy, we cannot provide the images here. **We promise we will include the results in our paper and code repository in revision.**\n\n**[Controlled Study]**\n\nThanks for the constructive suggestion. We optimize vanilla 3D Gaussian splatting fields with and without mask-guided depth continuity loss for each scene in our validation set, and compare their depth and render quality in the table below.\nWe can see that, with such intra-instance depth continuity constraint, **the vanilla 3DGS also exhibits better depth and render quality**, further demonstrate the effectiveness of our mask-guided depth continuity loss for per-scene optimization methods, validating the universality of our insight that \"understanding helps reconstruction\".\n\n|Method|DepthEstimation|NovelViewSynthesis|\n|:-|:-:|:-:|\n||AbsRel↓RMSE↓|PSNR↑SSIM↑LPIPS↓|\n|3DGS w/o L_cont|0.155\\|0.359|28.56\\|0.872\\|0.207|\n|3DGS w/ L_cont|**0.101**\\|**0.261**|**28.81**\\|**0.885**\\|**0.191**|\n\n\n**[W3] Algorithm 1 does not explain how the model reaches a consensus on a semantic label for a 3D Gaussian when the input views provide conflicting information. Definition of the segmentation loss L_mask.**\n\n**[Algorithm 1]**\n\nAlgorithm 1 primarily illustrates, at a pseudocode level, how to utilize multi-view 3D Gaussians to lift multi-view mask logits from 2D to 3D and achieve 3D-level multi-view aggregation based on rasterization.\nAll operations correspond to PyTorch code and can be reproduced using the code in our supplementary materials.\n\nTo explain how this process reaches a consensus on mask label for a 3D Gaussian when the input views provide conflicting information, **we will include the following description in our revision.**\n\nSpecifically, before the \"multi-view mask aggregation step\" in Algorithm 1, different input views may exhibit conflicting information when decoding multi-view masks, which causes masks of the same object having different IDs across views. To address this, we reuse the predicted geometry-related 3D Gaussian parameters (i.e., mean, opacity, scale, rotation) to lift multi-view mask information (class-wise query probability maps) to the 3D space. **Through the alpha-blending mechanism of rasterization, pixels from different views but corresponding to the same object are explicitly forced to perform weighted averaging at the 3D level**, eliminating implicit 2D-level semantic inconsistencies before aggregation. Leveraging the aggregated mask information for mask decoding, we can further obtain multi-view consistent masks for 3D-level segmentation.\n\n**[L_mask]**\n\nThe definition of the segmentation loss L_mask is as follows:\n$L_{mask}=\\lambda_{bce} L_{bce} + \\lambda_{dice} L_{dice} + \\lambda_{ce} L_{ce} $. Specifically, $L_{bce}$ denotes binary cross-entropy loss that measures pixel-wise differences between the predicted foreground probability and ground truth. $L_{dice}$ denotes dice loss that measures the overlap ratio between the predicted foreground and ground truth. $L_{cls}$ denotes multi-class cross-entropy loss that measures differences between the predicted category probability and ground truth for each mask. In practice, we follow Mask2Former and set $\\lambda_{bce},\\lambda_{dice},\\lambda_{cls}=5.0,5.0,2.0$. We will include this in our revision.\n\n\n**[W4] More controlled experiments on the R$\\to$U and U$\\to$R modules. They are not well-isolated from the rest of the proposed architecture.**\n\nThanks for the constructive suggestion.\nWe acknowledge that our R$\\to$U and U$\\to$R modules can indeed be used as plugins and integrated into other network architectures.\nSpecifically, the R$\\to$U module is a training-free module that can work as long as multi-view 3D Gaussians and mask logits are provided. The U$\\to$R module is a model-free mask-guided loss function that can work as long as predicted / ground-truth multi-view masks are provided.\n\nTo verify their effectivenesses under different network architecture, we implement a variant model without other components of our method (i.e., Unified Query Decoder), and directly combine this Gaussian-only network with Mask2Former (\"Mask2Former-GS\") as mentioned in [W1,Q1].\nWe conduct ablation studies of R$\\to$U and U$\\to$R modules for this variant and show the results in the table below. We can see that **both of the two modules show similar improvements in reconstruction and understanding just as they do in the original architecture**.\n|Mask2Former-GS|DepthEstimation|NovelViewSynthesis|Understanding(context-view)|Understanding(novel-view)|\n|:-:|:-:|:-:|:-:|:-:|\n|R$\\to$U\\|U$\\to$R|AbsRel↓RMSE↓|PSNR↑SSIM↑LPIPS↓|mIoU↑mAP↑PQ↑|mIoU↑mAP↑PQ↑|\n|✘\\|✘|0.090\\|0.233|25.02\\|0.812\\|0.202|0.543\\|0.236\\|0.587|N/A\\|N/A\\|N/A|\n|✓\\|✘|0.090\\|0.233|25.02\\|0.812\\|0.202|0.551\\|0.247\\|0.595|0.550\\|0.243\\|0.592|\n|✘\\|✓|0.075\\|0.209|25.74\\|0.817\\|0.194|0.545\\|0.241\\|0.590|N/A\\|N/A\\|N/A|\n|✓\\|✓|**0.075**\\|**0.209**|**25.74**\\|**0.817**\\|**0.194**|**0.552**\\|**0.250**\\|**0.606**|**0.552**\\|**0.249**\\|**0.603**|\n\n\n**[Q3] Could you please provide more details on the implementation of Algorithm 1?**\nPlease refer to [W3] for details.\n\n\n**[Q4] Could you please explicitly define the L_mask loss function?**\nPlease refer to [W3] for details."}]

Author rebuttal:

Meta-review: This paper introduces SIU3R, the first alignment-free framework for simultaneous scene, instance understanding, and 3D reconstruction from unposed images. It addresses an important and timely problem by leveraging pixel-to-3D feature lifting. The proposed unified query design and mutual-benefit modules are effective, yielding consistent improvements across reconstruction and multi-task understanding. Reviewers initially raised concerns regarding generalization, fairness of comparisons, and limited evaluation, but the authors’ rebuttal provided additional experiments and clarifications that satisfactorily resolved these issues.   Overall, the work is technically sound, impactful, and clearly addresses a new problem. I recommend acceptance.
Decision: Accept (spotlight)