Large Spatial Model: End-to-end Unposed Images to Semantic 3D
Authors: Zhiwen Fan
Jian Zhang
Wenyan Cong
Peihao Wang
Renjie Li
Kairun Wen
Shijie Zhou
Achuta Kadambi
Zhangyang Wang
Danfei Xu
Boris Ivanovic
Marco Pavone
Yue Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Reconstruction
3D Scene Understanding
Gaussian Splatting
Primary area: machine_vision
TL;DR: We propose a method that utilizes two unposed and uncalibrated images as input, and reconstructs the explicit radiance field, encompassing geometry, appearance, and semantics in real-time.

Abstract: Reconstructing and understanding 3D structures from a limited number of images is a classical problem in computer vision. Traditional approaches typically decompose this task into multiple subtasks, involving several stages of complex mappings between different data representations. For example, dense reconstruction using Structure-from-Motion (SfM) requires transforming images into key points, optimizing camera parameters, and estimating structures. Following this, accurate sparse reconstructions are necessary for further dense modeling, which is then input into task-specific neural networks. This multi-stage paradigm leads to significant processing times and engineering complexity.

In this work, we introduce the Large Spatial Model (LSM), which directly processes unposed RGB images into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward pass and can synthesize versatile label maps by interacting through language at novel views. Built on a general Transformer-based framework, LSM predicts global geometry via pixel-aligned point maps. To improve spatial attribute regression, we adopt local context aggregation with multi-scale fusion, enhancing the accuracy of fine local details. To address the scarcity of labeled 3D semantic data and enable natural language-driven scene manipulation, we incorporate a pre-trained 2D language-based segmentation model into a 3D-consistent semantic feature field. An efficient decoder parameterizes a set of semantic anisotropic Gaussians, allowing supervised end-to-end learning. Comprehensive experiments on various tasks demonstrate that LSM unifies multiple 3D vision tasks directly from unposed images, achieving real-time semantic 3D reconstruction for the first time.

Large Spatial Model: End-to-end Unposed Images to Semantic 3D
Zhiwen Fan 1:2 Jian Zhang Wenyan Cong', Peihao Wang', Renjie Li*, Kairun Wen Shijie Zhou';
Achuta Kadambi', Zhangyang Wang'. Danfei Xu2.6, Boris Ivanovic" Marco Pavone Yue Wang"
UT Austin
2NVDDI Research 8XMU
"TAMU
SUCAA
6GaTech "Stanford University
USC
Project Website: hatps://largessatiaannnn htw.:aaaiiooee...ii....... github.io
Images
Feature Field Radiance Field
SAS
Unposed Uuseen
Large Spatial Model
M
100+
FPS
Depth
0.1 second
4
Seg.
Text
Figure 1: Large Spatial Model takes two unposed images input and reconstructs an explicit radi- ance field, capturing geometry, appearance, and semantics in real time. This yields high performance versatile tasks such as view synthesis, depth prediction, and open-vocabulary 3D segmentation.
Abstract
Reconstructing and understanding 3D structures from limited number of im- ages is classical problem in computer vision. Traditional approaches typically decompose this task into multiple subtasks, involving several stages of complex mappings between different data representations. For example, dense reconstruc- tion using
from-Motion (SfM) requires transforming images into key points, optimizing camera parameters, and estimating structures. Following this, accurate sparse reconstructions necessary for further dense modeling. which is then input into task-specific neural networks. This multi-stage paradigm leads to significant processing times and engincering complexity.
In this work, we introduce the Large Spatial Model (LSM),, which directly pro- cesses unposed RGB images into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward pass and can synthesize versatile label maps by interacting through language at novel views. Built on a general Transformer-based framework. LSM predicts global geometry via pixel-aligned point maps. To improve spatial attribute regression, we adopt local context aggregation with multi-scale fusion, enhancing the accuracy of fine local details. To address the scarcity of labeled 3D semantic data and enable natural language-drive driven scene manipulation, we incorporate pre-trained 2D language- based segmentation model into 3D-consistent semantic feature field. An efficient decoder parameterizes set of semantic anisotropic Gaussians, allowing supervised end-to-end learning. Comprehensive experiments on various tasks demonstrate that LSM unifies multiple 3D vision tasks directly from unposed images, achieving real-time semantic 3D reconstruction for the first time.
ZZ. Fan and Zhang contributed equally: Z. Fan the Project Lead
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The computer vision community has devoted considerable effort to recovering and understanding 3D information (e.g., depth and semantics) from 2D sensory data (e.g., images). This process aims to derive 3D representations that encapsulate both geometric and semantic details from cheap and widely available 2D data, facilitating further interaction, reasoning, and planning within 3D physical world. Traditional approaches [ tackle this by pipelining several distinct tasks: detecting, matching, and triangulating points for initial sparse reconstructions and the subsequent dense reconstruction, followed by the integration of specialized submodules for semantic 3D modeling.
Recent developments in this domain have markedly proceeded with a more powerful representa- tion using both sparse reconstruction, and subsequent dense 3D modeling via Multi- View Stereo (MVS) 2 3 Neural Radiance Field (NeRF) 4, and 3D Gaussian Splatting (3D-GS) [.) This trend influenced various industries, including autonomous driving [6], robotics [7]. digital twins (8), and virtual/augmented reality (VR/AR) 10). Due to the complexity of inferring 3D information from 2D images, previous methods have broken down the holistic task into distinct, manageable subproblems. However, this strategy propagates errors from stage to stage and downgrades the performance of subsequent tasks. For instance, the critical step of precomputing camera poses utiliziig Structure from Motion (SfM) D- has proven to be vulnerable and often fails in scenes covered by sparse number of views exhibiting low-textured surfaces [1]. Such inaccuracies in camera pose estimation can ultimately lead to imprecise interpretation of the 3D scene.
Furthermore, reasoning about and interacting with the environment would benefit from comprehen- sive 3D understanding. Open-vocabulary methods, which perform semantic segmentation without relying fixed set of labels, provide notable flexibility. However, unlike single-image under- standing, the absence of large-scale and diverse 3D scene data with accurate multiview language annotations complicates the challenge. Efforts have been made to integrate 2D features into frame- works such as NeRF 12 and 3D-GS 15 [6]. Yet, these methods, such as Feature-3DGS [15] typically require overfitting each 3D scene separately with extensive captured viewpoints and preprocessing camera poses using Structure- froeie--foo-Maae Motion..
To address the challenges outlined above, we propose for the first time novel unified framework for these key 3D vision subproblems: dense 3D reconstruction, open-vocabulary semantic segmen- lation, and novel view synthesis from unposed and uncalibrated images. Our approach leverages single Transformer- based model that learns the attributes of a 3D scene via semantic anisotropic Gaussians. Unlike previous methods that rely on epipolar Transformers with known camera parame- ters 17 or require extensive per-scene fitting [5] we employ a coarse-to ceurse-to-fine strategy. This strategy predicts dense 3D geometry using pixel-aligned point maps, progressively refining these points into anisotropic Gaussians in a single feed-forward pass.
Our framework, dubbed Large Spatial Model (LSM), begins with general Transformer architecture incorporating cross-view attention 20) which constructs pixel-aligned point maps at a normalized scale, enabling generalization across various datasets. LSM further enhances point-based representa- tions through multi-scale fusion and local context aggregation using a ViT encoder. Additionally, LSM performs hierarchical cross- mooss fusion, integrating features from a pre-trained 2D semantic model into a consistent 3D feature field. Through differentiable splatting of the regressed semantic anisotropic Gaussians, LSM enables end-to-end supervision and supports real-time scene-level 3D semantic reconstruction and rendering without needing explicit camera parameters. This allows for efficient, data-driven rendering of labels from novel viewpoints, demonstrated in Figure 1
Our contributions are summarized as follows:
We introduce a unified 3D representation and an end-to-end framework that addresses dense 3D reconstruction, 3D language- based segmentation, and novel-view synthesis directly from unposed images in a single forward pass.
Our method leverages a Transformer architecture with cross-view attention for multi-view ge- ometry prediction, combined with hierarchical cross modal attention to propagate geometry- rich features. We further integrate a pre-trained semantic segmentation model enhance 3D understanding. By aggregating local context at the point level, we achieve fine-grained fea- ture integration, enabling the prediction of anisotropic 3D Gaussians and efficient splatting for RGB, depth, and semantics.
2

---

Our model performs multiple tasks simultaneously with real-time reconstruction and render- ing on a single GPU. Experiments show that our unified approach scales effectively across different 3D vision tasks, surpassing many tfae-offthheart baselines without the need for additional SfM steps.
2
Related Work
SfM and Differentiable Neural Representation Structure-from-Motiie (SfM) aims to jointly esti- mate camera poses and reconstruct sparse 3D structures from multiple views. Traditional pipelines []] involve multiple stages, including descriptor extraction, correspondence estimation, and incremental bundle adjustment. Recent advances in learning-based techniques [21 have further improved the accuracy and efficiency of SIM. These methods are widely adopted in 3D vision tasks, where differentiable neural representations typically assume accurate camera poses provided by SIM. For instance, NeRF 4 and its successors [26] rely on poses estimated offline via COLMAP [ 27 Similarly, 3D Gaussian Splatting [5] uses SfM-generated 3D points for initialization and has been applied to robotics [28)30], healthcare 63), and many other domains 36]. Beyond novel view synthesis, lifting 2D features to 3D has gained traction in various editing tasks
End-to-End Image-to-3D 3D reconstruction is long- stagditnnding problem in computer vision, with
traditional approaches like SfM [38 39 I. Multi-view Stereo (MVS) B 2 40 41], and Signed
Distance Function (SDF) 42 43 More recent techniques utilize neural representations, including
implicit 4 and explicit 55] formats generate 3D models. Semantic understanding is often integrated
during the reconstruction process 44 or through additional optimization steps [12 13 However, most methods depend on a preprocessing step like SIM [ to estimate camera calibration, poses, and sparse point clouds before dense reconstruction, either through feed-forward prediction or test-time optimization. This reliance on calibration and pose estimation limits scalability with large- The latest pose-free,
scale data, contrasting the success seen with large foundation models 445
feedforward approaches, such as Scene Representation Transformers 46 48 have advanced the concept of representing multiple images view generation even in the presence of inaccurate camera poses or without any pose information. However, these methods struggle to produce explicit geometry. DUSt3R (0) addresses this limitation by predicting dense point clouds directly from unposed stereo image pairs, enabling pixel-aligned geometry prediction normalized scales. Practically, dense point prediction requires accurate multi-view RGB-D pairs, which significantly limits its scalability. InstantSplat [49 addresses this by utilizing novel-view synthesis with only posed image data and employing Gaussian Bundle Adjustment to jointly optimize camera and scene parameters for ultra-fast dense 3D reconstruction.
"set latent scene representation." allowing for novel
In contrast, framework offers holistic solution for dense 3D semantic reconstruction from unposed images. integrates dense 3D geometry reconstruction, and language-basee 3D interaction, task.
while minimizing the need for extensive data annotation by using novel view synthesis Since dense 3D annotations are often scarce in real-world scenarios, we propose semantic anisotropic Gaussians to lift 2D features map 3D semantic embeddings without additional annotation. Our approach addresses higher-level 3D tasks in perception and dense 3D reconstruction compared to DUS(3R (20). by utilizing lightweight annotations and solving these tasks jointly within unified framework.
3 Methods
Overview Figure illlstrrates the architecture for training the Large Spatial Model (LSM). During training, the input consists of stereo image pairs along with associated camera intrinsics and poses: {[I E 12 #x (x), (T, E 33x4), ), (K, E R J At inference, however, unposed images can be directly fed into the framework. The pixel-aligned geometry is predicted using a standard Transformer architecture (50) with cross-attention between input views. Dense prediction heads are employed to regress normalized point maps during training: (D, E #H 3 (see Sec. 3.1).
To support fine-grained semantic anisotropic 3D Gaussian regression, which represents the 3D scene and lifts generic feature fields from pre-trained 2D vision models, we apply point-based attention with learnable positional encoding in a local window. This propagates features from neighboring points (Sec. 3.2). effectively merging encoded features with rich semanties (Sec. (.2) at multiple scales using 2D pre-trained models (Sec. 3.3 New views from the semantic radiance fields can be
3

---

decoded using splitting [S] on the target poses (Sec. 3.4 During inference, semantic anisotropic Gaussians are directly predicted, and the renderer takes the camera parameters derived from the point maps. An overview of the model architecture is shown in Figure
Feature
Target Image
Cross-view Attention Block
Input Image
(Sec. 1.13
Geometry
x8
(Bec.
Patchify PE
Encoder
Decoder
Resterizer
Point-wise Aggregation (Sec 1:2)
Feature Lifting
Par
P. Fbl
(See. X.5)
Mb
Ob.
Sb.
rb.
Encoder
Decoder
Figure 2: Network Architecture. Our method utilizes input images from which pixel-aligned point maps regressed using a generic Transformer. A set of semantic anitrosopic 31) Gaussians incorporating geometry, appearance, and semantics are then predicted employing another point-based Transformer that facilitates local context aggregation and hierarchical fusion. It is supervised end- to-end, minimizing the loss function through comparisons against ground truth and rasterized label maps on new views. During the inference stage, our approach is capable of predicting the scene representation without requiring camera parameters, enabling real-time semantic 3D reconstruction.
3.1 Dense Geometry Prediction
Instead of adopting a conventional Transformer with Epipolar attention-whicl can be inefficient as pixel-wise prediction requires hundreds of queries sampled epipolar lines []7 [8 we implement an encoder-decoder structure for directly regressing view-specific point maps at normalized scales. Cross-view attention is utilized aggregate multi-vie information efficiently.
Direct Regression of Normalized Depth Map We employ Siamese ViT-based encoder [50] that processes stereo images using shared weights. It involves the patchification and tokenization of images, followed by the integration of sinusoidal positional embeddings. To directly regress the pixel-aligned point maps from the unposed images for view E {1,2]. cross-view attention also employed, enhancing the architecture's s capacity to infer spatial relationships and propagate information between views-an approach that has proven effective in prior research [51 20 52). The decoder block consists of interleaved self-attention which integrates tokens from both images. The inter-view decoder includes 12 attention blocks, akin to those utilized in previous multi-view stereo (MVS) studies [52 20). These blocks generate tokenized features for subsequent Dense Prediction Transformer head (DPT) (53), which estimates a pixel-wise point map in a normalized coordinate system along with confidence value:
each view and cross-attention across views,
Loonf
Σ
Σ M Ldepth (v, log M 2.
(1)
(E(1,2) TED
where M is pixel-aligned confidence map, same as DUS(3R, D indicates all valid points to the origin, M..1 denotes the confidence map obtained from view expressed in the coordinate frame of view a hyper-parameter that upply regularization, encouraging the network to perform robustly in challenging areas. The depth error is calculated by
Eeeeth
Σ vE(1.2)
p 0,
where the normalization factors (z and 2) indicate that the predicted and ground-truth pointmaps are processed by normalizing. For example, z obtained by: mor = D Eve( EieD. [P:]]-

---

3.2 Point-wise Feature Aggregation
Building on the foundational work in N and Multi-view Stereo which employ a coarse- to-fine strategy for high-quality radiance field and depth estimation, we also aggregate the initial predicted geometry by upplying a Transformer [54] at the point level, leveraging hierarchical repre- sentations to achieve more refined, point-based regression.
Point-wise Attribute Prediction Rather than relying solely on a single network to represent the scene, we employ two Transformer-basee networks optimized for distinct tasks: one for capturing "coarse" global geometry and another for "fine" local information aggregation. Initially, we integrate stereo point maps, including color information for each point primitive, formulated as {p: (Ii, Ves 24, Tis Gi, b,) [== ) to serve as input. Unlike tokenized image patches, point primitives carry distinct geometric significance within Euclidean space. Inspired by recent advancements in point- cloud processing [53/57], we employ a Transformer within a localized window to perform point-wise aggregation, selectively emphasizing key features from neighboring primitives. Point-wiss encoding and decoding are essential for refining scene representation, utilizing multiscale aggregation across five hierarchical levels.
After aggregating the point- features, we employ an additional layer of multilayer perceptron (MLP) to regress the parameters, representing the 3D scene through a set of anisotropic Gaussians [5]. The parameters include the opacity a, scale factor s, rotation r, and Spherical Harmonics coefficients
{c; E | = 1, 2..... } where k (K + 1)2 is the number of coefficients of SH with degree K. The Gaussian centers μ are regressed from geometry prediction backbone. The color e of direction d is then computed by summing up all SH basis as (d) Σi= c,B, (d), where B, is the 7th SH basis. The final pixel intensity e is calculated by blending n ordered Gaussians overlapping the pixels using the following render function:
C:O, (1=);;)
This equation efficiently models the contributions of each Gaussian to the pixel's final appearance, accounting for their transparency and layering order.
Cross-model Feature Aggregation To effectively combine multi-view image features with point- wise geometrie information, we implement cross-model attention between two sets of tokens. The attention block fuses tokens from different sources by first applying self-attention to the input P. allowing each token attend other tokens within the same sequence. This process helps capture internal relationships and enrich the representation of the input token. Next, cross-attention is used, where two sets of tokens (P and F) from the latent layers of two different models are fused, enabling the integration of external information into P. Finally, a feed-forward network (MLP) further processes the updated information following cross-model fusion.
The original point features contain explicit and precise spatial information, which is critical for accurate geometry reconstruction. In contrast, the image token features F, from image encoder (Sec.3.1) 1 are rich in semantic content, providing important contextual information that enhances general understanding of the scene. Cross-model fusion enables the integration of detailed spatial geometry with semantic richness:
Q = Proj(P),
V,K Proj(F).
QK
P = softmax
Vdle
where P and F were normalized with a linear layer before projection.
3.3 Learning Hierarchical Semantics
To facilitate semantic 3D representation, we augment the anisotropic 3D Gaussians with a learnable semantic feature embedding (a.k.a. semantic anisotropic Gaussians) and rasterize into the 2D image plane by blending Gaussians that overlap with each pixel using a feature rendering function.
8
I a4) j=1
(4)
5

---

indicates the final rasterized feature embedding on image plane, and s; is semantic embedding on anisotropic Gaussians.
3D Semantic Field from 2D Images After obtaining 8, we optimize s, by minimizing the difference between the rasterized feature map and the feature maps generated by pre-trained 2D model. Unlike the previous method [15] which requires test time optimization, transform the estimation of the feature field into a fully learnable process.
Feature maps (fS; E " 1i=1) from a pre-trained 2D multi-modal model [58] are inherently view-inconsistent due to the lack of spatial awareness during the model's training. To elevate multi- view feature embeddings into a coherent 3D feature field for holistic 3D understanding, we introduce a dynamic fusion strategy employing an attention-based correlation module. This module is specifically designed to learn blending weights for each token within Point Transformer [54) from the input
pixel-wise feature embedding (S,). We employ attention blocks described in 3. 3.2 synchronize in the latent spaces through a supplementary set cross-attention layers. The visual feature from (38), denoted as S, is utilized for this purpose. This loss function is minimized during training by utilizing rasterized feature maps on new views S and directly inferred feature maps using ground truth images on new views $ (LSeg (58)), thereby facilitating the learning of blending weights for consistent semantic field regression.
S.S Lasat = 11 - sim = ||||||||||||||||||||||||
(5)
Multi-scale Feature Fusion To improve model efficiency, we propagate information from VIT Encoder feature F and the frozen semantic feature S, to the 3D latent space (point feature P) which has fewer tokens, thereby enabling selective attention to critical features. We further refine feature fusion across multiple stages, optimizing information flow while minimizing additional computational overhead. Novel view synthesis serves as an effective task to encode the complete geometric and appearance features into a low-dimensional 3D latent space, while recovering a set of
semantic anisotropic Gaussians (G {g: E RRIx through learning from large-scale data and end-to-end training.
3.4 Training Objective
Putting all together, our model can be optimized end-to-end:
C(G. d) - c + D-SSIM(C(G,,) C)
(6)
Photometric
+ X2 (SG +
Σ
Ag - Leont (D,,1, Du,1,De.1. De,1)
(7)
Semantic
(1,2)
Geometric
where C and Ć are rasterized and GT pixel intensities, G denotes represented 3D scene using set of 3D semantic anisotropic Gaussians, S and S denotes rendered LSeg feature extractor and feature on the target image, d indicates the direction and position at new views. In our methodology, we leverage both photometric loss and semantic loss supervise the generation of rasterized new views. In order for geometry prediction and semantic feature lifting, we employ a confidence weiegheee depth loss applied to the input views. The parameters Å₁, A2, Ag are set 0.25, 0.3, and 1.5, respectively, as determined by the grid search.
4 Experiments
4.1
Implementation Details
For our architecture, employ ViT-Large as the encoder and ViT-Base as the decoder, complemented by a DPT head [53] for pixel-wise geometry regression. We initialize the geometry prediction layers using DUS13R (20). Point Transformer layers consists of encoder and decoder blocks with progressive downsampling and upsamping. The cross-model fusion strategy implemented at the output of the last encoder and the output of the first decoder. The entire system is optimized

---

Table Quantitative Comparison in 3D Tasks. We report novel-view synthesis. depth estimation quality, and open- -eccboccyyl segmentation accuracy. Our method eliminates the need for any preprocessing 3D tasks, while achieving performance comparable to other baselines that rely on SfM to obtain camera parameters and poses.
Reconstruction Timel
Source
View
Target View
SfM
Per-Seene
m
Acc.f
rel
mloU
Acc.1
PSNR
SSIM f
LPIPS
LSeg
N/A
N/A
0.5278
0.7654
0.5281
0.7612
NeRF-DFF
20.52s
I 1min2s
0.4540
0.7173
27.68
9.61
0.4037
0.6755
19.86
0.6650
0.3629
Feature-3DGS
20.52s
18mins36s
0.4453
0.7276
21.07
0.4223
0.7174
24.49
0.8132
0.2293
pixelSplat
20.52s
0.064s
24.89
0,8392
0.1641
Ours
0.108s
0.5034
0.7740
3.38
67.77
0.5078
0.7686
24.39
0.8072
0.2506
Input (Unposed Images)
Feature Field Rendered at New Views
Visualization of the 3D Feature Field. We present examples features rendered from novel viewpoints, illustrating how our method converts 2D features into a consistent 3D, facilitating versatile and efficient segmentation. Visualizations are generated using PCA [59]
Figure 3:
end-to-end using the loss function described in Eq. 6 The training of our model contains 100 epochs, leveraging a combined dataset of ScanNet++ and Scannet[6]], of 1565 scenes. Training is on 8 Nvidia A100 GPU lasts for days. We start with base learning rate of le-4 and incorporate 10-epoch warm-up period. AdamW employed as the optimizer for all experiments. Evaluation is conducted on 40 unseen scenes from ScanNet. Additionally, assess tasks: novel view synthesis, multi-view depth prediction, and 3D language-base based semantic segmentation.
4.2 Semantic 3D Reconstruction
Evaluation of Synthesized Images Quality Novel view synthesis is evaluated using NeRF- DFF [3] and Feature-3DGS (15), both of which are capable of predicting RGB values as well as features. In addition, we compared our approach with the state-of-the-arr, generalizable, pose- based 3D Gaussian Splatting method, pixelSplat [8]. which generates point-based representations through feed-forward pass. Unlike our method, these existing approaches rely on known camera intrinsics and poses prior evaluation. As indicated in Table NeRF-DFF and Feature-3DGS tend to overfit on each individual scene, requiring significantly more time than our method, yet performing comparably in terms of output quality. pixelSplat utilizes an Epipolar Transformer, searching along the epipolar line using GT camera parameters to regress Gaussian attributes, resulting in longer inference times. Visualizations in Figure demonstrate that our results are sharper and exhibit fewer artifacts than NeRF-DFF, and are comparable to Feature-3DDo 3DGS and pixelSplat in performance.
Evaluation of Open-vocabulary Semantic 3D Segmentation The semantic segmentation is evaluated by class-wise intersection over union (mloU) and average pixel accuracy (mAcc) on novel views metries. Following the approach of Feature-3DGS [15, we map thousands of category labels from diverse datasets into a set of common categories, including (Wall, Floor, Ceiling, Chair, Table, Bed, Sofa, Others). We compare our model against two oftee-of-the 3D baselines with the capacity for generating RGB, semantics and depth on any view: Feature-3DGS [1]] and NeRF-DFF [3] which are based on 3D-GS 5 and NeRF 4], respectively. Additionally, the model LSeg [88 used as 2D open-vocayulary segmenter for feature lifting, is included in our comparisons. We present statistics related the semantic annotations on the adopted the ScanNet datasets Table []] where LSM demonstrates competitive performance compared to baseline 3D methods that require
7

---

Ground-Trumd
NeRF-DFF
Feature-3DGS
pixelSplat
Figure 4: Novel-View Synthesis (NVS) Comparisons. We evaluate scene-level reconstruction by comparing our method to approaches that require per-scene optimization, such as NeRF-DFF and Feature-3DGS, which predicts both RGB and segmentation, and the generalizable 3D Gaussian Splatting method (pixelSplat). Notably, these methods require a pre- pre-processing step to obtain camera poses using off-the-shelf SfM. Through end-to-end, data-driven training, method achieves comparable visual quality to these approaches while reconstructing the 3D radiance field in a single pass.
Table 2: Ablation Study on Our Design Choices. We refer to the model that integrates cross- view attention for multi-view geometry with point-wise aggregation for future refinement as the baseline configuration (Exp #1). Implementing cross-modal attention to fuse geometry encoder features enhances both the rendering quality of new views and the segmentation accuracy (Exp #2). Additionally, incorporating features from frozen 2D semantic backbone into the fusion process (Exp #3) for consistent feature field amalgamation, and multi-scale fusion enhances hierarchical information flow (Exp #4), substantially improving language- based semantic 3D segmentation. Segmentation metrics use LSeg results as ground-truu in this table.
Exp ID
Model
mloUt
Acc.t
PSNR1
SSIM
[1]
Baseline
0,4562
0.6940
24.00
0.7981
[2]
[1] + Fuse Encoder Feat.
0,5410
0.8083
23.67
0.7876
[3]
[1] + Fuse LSeg Feat.
0.5586
0.8505
23.85
0.7902
[4]
[1] + [2] [3]+ Multi-scale Fusion
0.6042
0.8681
24.39
0.8072
ground-truth camera parameters and extensive per-scene optimization. The visualized results in Figure 5 illustrate that L.M can produce view-consistent semantic maps. In contrast, the 2D method LSeg yields detailed segmentation results but lacks cross-view consistency. To validate that LSM learns semantically meaningful features, we visualize the lifted feature field using PCA to reduce the high-dimensional features into three channels [13]. As shown in Figure LSM effectively generates a faithful semantic feature field through feed-forward inference using pair images.
Evaluation of Depth Accuracy We also evaluate the performance of our model on the task of multi-view stereo depth estimation. We utilize the Absolute Relative Error (rel) and Inlier Ratio (r) with threshold of 1.03 to assess each scene, similar to DUSt3R [20]. Since our approach does rely on any camera parameters for prediction, we align the scene scale between the predictions and the ground truth. Specifically, we normalize the predicted point maps using the median of the predicted depths and similarly normalize the ground truth depths, following procedures established in previous literature [22 20 to align the two sets of depth maps. We observe in Table. that LSM achieves wate-offthe--ee accuracy on ScanNet datasets than the per-scene wise methods, Our model is significantly faster than baseline methods, it only require forward-pass.

---

RGB
NeRF-DFF
Feature-3DGS
LSeg
Vaye
Wall
Floor
Ceiling
Chair
Table
Sofa
Bed
Others
None
Figure 5: Language-based 3D Segmentation Comparison. We visualize the segmentation results across four unseen scenes and observe that our method performs comparably NeRF-DFF and Feature-3DGS. This indicates that LSM effectively lifts 2D feature maps into high-quality 3D feature fields.
Table 3: Inference Time per Module. Breakdown of inference time for each module for analysis.
Module
Inference Time (seconds)
Dense Geometry Prediction (Sec. 3.1)
0.029
Point-wise Aggregation (Sec. 3.2)
Feature Lifting (See. 3.3)
0.019
Total
0.096
4.3
Ablation Studies
We conduct ablations to validate our desing effectiveness. Experiments are on both language- based segmentation and novel view synthesis. The quantitative results can be views Table 2
Cross-Modes Feature Aggregation Incorporating the geometry encoder feature from ViT into the hidden layer of the point-aggregation layer (Sec. 3.2) demonstrates that such cross-model information flow significantly benefits the segmentation task, improving the mean Intersection over Union (mloU) from 0.4562 to 0.5410 (Exp + 2).
Semantic Feature Fusion Multi-Scale Employing cross-model fusion, where latent features of the semantic model are integrated into the middle layers of point-based aggregation, also improves injection of semantically rich embeddings (0.4562 0.5586, Exp # 3). The decoded features confirm that the lifted feature field produces higher-quality feature maps, with the semantic mloU improving from 0.5586 to 0.6042 (Exp #3 - 4) through multi-scale fusion.
Module Timing. We analyze the computational cost of each module by running inference 1,000 times on the ScanNet test dataset with the model, as shown Table and calculating the average inference time for each module of the Large Spatial Model.

---

Table 4: Performance Comparison on Replica Dataset. LSM operates without ground -ruth camera parameters, achieving decent PSNR and low relative depth error, while also enabling semantic understanding within a unified framework.
Model
Offline SfM
mIoU t
rel L
PSNR
pixelSplat
Required Required Not Required
None
20.14
26.28
Replica
Splatter Image
None 0.51
None
12.37
Ours
4.91
23.10
Evaluation of Generalizable Methods on New Datasets. To avoid potential overfitting, we adopt the Replica dataset[63), a photorealistic simulated 3D dataset with accurate RGB, dense depth maps, and semantic annotations for comprehensive evaluation. We use the same data preparation with Feature-3DGS. LSM generalizes well to the simulated Replica test set, achieving the best depth estimation metrics and enabling 3D semantic segmentation, which is unique among generalizable methods. Splatter Image[64), an ultra-fas monocular 3D object reconstruction method using 3D-GS, performs well for object reconstruction with masked backgrounds but struggles with scene-wise reconstruction in complex backgrounds.
Conclusion, Limitation, and Broader Impact
We have introduced the Large Spatial Model (LSM), unified framework for holistic 3D semantic reconstruction from uncalibrated and unposed images, with the added capability of interaction through language. LSM leverages cross-view attention to aggregate multi-view cues and utilizes multi- scale cross-modal attention to integrate semantically rich features into point-based representation. Hierarchical point-wise aggregation layers further refine these representations and enhance the integration of cross-modal attention. By splatting regressed anisotropic 3D Gaussians, L.M enables the generation of novel views with versatile label maps. LSM is highly efficient, capable of real-time end-to-end 3D modeling, and supports various downstream applications.
While our method significantly accelerates semantic 3D scene reconstruction, relies on a pre-trained model for feature lifting, which can increase GPU memory requirements during training, especially when the integrated 2D model has a large number of parameters. Additionally, the need for ground- truth depth maps, although there are millions of multi-view datasets annotated with them, could limit its scalability for internet-scale video applications.
Our research enables efficient, real-time 3D scene-level reconstruction and understanding, which advantagcous for applications such as end-to-end robotic learning, AR/VR, and digital twins. However, there is potential for misuse, such as the arbitrary distribution of digital assets or privacy leakage related to building structures. These risks can be mitigated by embedding watermarks into the 3D assets 655]
References
[1] Johannes L Schonberger and Jan-Michael Frahm. Structure-from mrotimoooooooooooonnooooo revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[2] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pages 767-783, 2018.
[3] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi view stereo and stereo matching. In Proceedings the JEEE/CVF conference on computer vision and pattern recognition, pages 2495-2504, 2020.
[4] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-10, 2021.
10

---

[5] Bernhard Kerbl, Georgios Kopanas. Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4))1-14, 2023.
[6] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al. Mars: An instance-aware, modular and realistic simulator for autonomous driving, arXiv preprint arXiv:2307. 0tt::2307.11000 1058., 2023.
[7] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation. arXiv preprint urXiv:2308.0793/ 2023.
[8] Luca De Luigi, Damiano Bolognini, Federico Domeniconi, Daniele De Gregorio, Matteo Poggi, and Luigi Di Stefano. Scannerf: a scalable benchmark for neural radiance fields. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 816-825, 2023.
[9] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Sterco magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint arXiv: 098117 2018.
[10] Jianmei Dai, Zhilong Zhang, Shiwen Mao, and Dampu Liu. A view synthesis- 360 vr caching system over mec-enabled c-ran. IEEE Transactions on Circuits and Systems for Video Technology,
2019.
[11] Wenjing Bian, Zirui Wang, Kejie Li, lia-Wang Bian, and Victor Adrian Prisacariu. Nope- nerf: Optimising neural radiance field with no pose prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4160 4169, 2023.
[12] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes. In The Eleventh International Conference Learning Representations, 2023.
[13] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems, 35:23311 1--3330, 2022.
[14] Mukund Varma, Peihao Wang, Zhiwen Fan, Zhangyang Wang, Hao Su, and Ravi Ramamoorthi. Lift3d: Zero-shot lifting of any 2d vision model to 3d. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21367-21377. IEEE, 2024.
[15) Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21676-21685, 2024.
[16] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051-20060 2024.
[17] Mukund Varma, Peihao Wang. Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference on Learning Representations, 2023.
[18] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312. 12337, 2023.
[19] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Bortin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view magge-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[20] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. arXiv preprint arX1v:2312.14132.222 2023.

---

[21] Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21 21686-21697, 2024.
[22] Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint 2018.
[23] Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. arXiv preprint arXiv: 00..11.2.044555 2018.
[24] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d 2021.
cameras. Advances in neural information processing systems,
[25] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy llg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5038-5041 2017.
[26] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 1 15, 2022.
[27] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-perfect with featuremetric refinement. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 5987-5997, 2021.
[28] Timothy Chen, Ola Shorinwa, Weijia Zeng, Joseph Bruno, Philip Dames, and Mac Schwa- ger. Splat-nav: Safe real-time robot navigation in gaussian splatting maps. arXiv preprint arXiv:2403.02751, 2024.
[29] Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang, Bu Jin, Pengfei Li, Chengliang Zhong, Zengmao Wang. Lina Liu, et al. Gaussiangrasper: 3d language gaussian splatting for open-vocabulary robotic grasping. arXiv preprint arXiv:2403.099. 2024.
[30] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18039-18048, 2024.
[31] Shuojue Yang, Qian Li, Daiyun Shen, Bingchen Gong, Qi Dou, and Yueming Jin. Deform3dgs: Flexible deformation for fast surgical scene reconstruction with gaussian splatting. In Interna- tional Conference on Medical Image Computing and Computer-Assinn Intervention, pages 132-142. Springer, 2024.
[32] Yuanhao Cai, Yixun Liang, Jiahao Wang, Angtian Wang, Yulun Zhang, Xiaokang Yang. Zong- wei Zhou, and Alan Yuille. Radiative gaussian splatting for efficient x-ray novel view synthesis. In European Conference on Computer Vision, pages 283-299. Springer, 2025.
[33] Chenxin Li, Brandon Y Feng, Yifan Liu, Hengyu Liu, Cheng Wang, Weihao Yu, and Yixuan Yuan. Endosparse: Real-time sparse view synthesis of endoscopic scenes using gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 352-262.55 Springer, 2024.
(34] Yuanhao Cai, Zihao Xiao, Yixun Liang, Yulun Zhang, Xiaokang Yang, Yaoyao Liu, and Alan Yuille. Hdr-gs: Efficient high dynamic range novel view synthesis at 1000x speed via gaussian splatting. arXiv preprint arXiv:2405. 15/25, 2024.
[35] Lingzhe Zhao, Peng Wang, and Peidong Liu. Bad-gaussians: Bundle adjusted deblur gaussian splatting. arXiv preprint arXiv:2403. 1831, 2024.
[36] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming- Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
12

---

[37] Jianglong Ye. Naiyan Wang, and Xiaolong Wang. Featurenerf: Learning generalizable nerfs by distilling foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8962-8973, 2023.
[38]
Changchang Wu et al. Visualsfm: A visual structure from motion system, 2011. URL http://www. cs. washington. edu/honnes/ccww/vsfin, 14:2, 2011.
[39] Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Towards internet-scale multi-view stereo. In 2010 JEEE computer society conference on computer vision and pattern recognition, pages 1434-1441. IEEE, 2010.
[40] Maxime Lhuillier and Long Quan. quasi-dense approach surface reconstruction from uncalibrated images. IEEE transactions on pattern analysis and machine intelligence, 27(3):418- 433, 2005.
[41] Engin Tola, Christoph Strecha, and Pascal Fua. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Machine Vision and Applications, 23:903-920, 2012.
[42] Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pages 303-312, 1996.
[43] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[44] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838-15847, 2021.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8763. PMLR, 2021.
[46] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučic, Daniel Duckworth, Alexey Dosovitskiy, al. Scene representation transformer: Geometry-fire free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2022.
[47] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd Van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in neural information processing systems, 35:9512-9524, 2022.
[48] Mehdi SM Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučic, and Klaus Greff. Rust: Latent neural scene representations from unposed imagery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17297-17306, 2023.
[49] Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang. Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. arXiv preprint arXiv:2403.20300 2024.
[50] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929/ 2020.
[51] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8922-8931, 2021.
13

---

[52] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems, 35:3502-3516, 2022.
[53] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pages 12179- 12188, 2021.
[54] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4840-4851, 2024.
[55] Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1538-1547, 2019.
[56] Meng-Has Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
[57] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 16259-16268, 2021.
[58] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprint arXhv:2201.03546 2022.
[59] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2833 2011.
[60] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 12-22, 2023.
[61] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[62] Philipp Schröppel, Jan Bechtold, Artemij Amiranashvili, and Thomas Brox. A benchmark and a baseline for robust multi-view depth estimation. In 2022 International Conference on 3D Vision (3DV), pages 637-645. IEEE, 2022.
[63] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, al. The replica dataset: digital replica of indoor spaces. arXiv preprint arXiv: 1atv::9966 000..900007777 05797, 2019.
[64] Stanislaw Szymanowicz. Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10208-100 10217, 2024.
[65] Chenxin Li, Brandon Y Feng, Zhiwen Fan, Panwang Pan, and Zhangyang Wang. Steganerf: Embedding invisible information within neural radiance fields. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 441-453. 2023.
[66] F Plastria.
The weiszfeld algorithm: proof, amendments and extensions, ha eiselt and v. marianov (eds.) foundations of location analysis, international series in operations research and management science, 2011.
[67] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6))381-395 1981.
14

---

[68] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[69] Vincent Lepetit, Francese Moreno-Noguer, and Pascal Fua. Ep n p: An accurate o (n) solution to the p problem. International journal of computer vision, 81:155-166, 2009,
15

---

Technical Appendices
We have included visualizations of rendered new views, visualized features, and the final language- based segmentation videos can be seen from our webpage.
Training/Tes DentiinngTTesting Split. Similar to NeRF literatures, we select one image out of four as test images, and the rest ones used as training for Feauture-3DGG and NeRF-DFF For pixelSplat and ours, we directly use the rest ones as source-view images reconstruct the 3D representation. We use the last checkpoint for evaluation.
How Derive Camera Parameters from Normalized Point Maps. We obtain pixel-aligned point map at where we can build the mapping from 2D to the camera coordinate system. We can first solve the simple optimization problem based on the Weiszfeld algorithm (66 to calculate per-camera focal, the same as DUSt3R [20]
W
H o
(Pi50, p(..))
f arg min
ΣΣ
(1,5)
(8)
pi.22
==
where i' = --- W and j' = H denote centered pixel indices. Assuming a single-camera setup 2
similar to that used in COLMAP for single scene capture, we propose stabilizing the estimated focal length by averaging across all training views:
x Σn fr The resulting f represents the
computed focal length that is utilized in subsequent processes. Relative transformation {T [R]t]} can be computed by RANSAC [67] with PnP 68 69) for each image pair.
Additional Model Details. We utilize the initial geometry prediction from DUSt3 which provides pixel-aligned geometry as the starting point. The subsequent point-wise aggregation is implemented using Point Transformer V3 [54]. The 2D- 2D-trained model, LSeg, is employed to provide multi-moda feature embeddings through its tokenization module, using the feature from the second-to-las layer of the DPT head. Additionally, the last layer of the ViT encoder is integrated into the feature space of Point Transformer. The fusion is carried out by single standard attention block, facilitating cross- model information flow. We will release the code. The middle layers of the Point Aggregation Module are utilized for this fusion. Both Feature-3DGS and NeRF-DFF models are trained with 5,000 iterations to prevent overfitting on real-world outward-facing scenes, and they also lift features from LSeg for the creation of 3D feature field. The point-wise aggregation module consists of four encoder blocks with progressive downsampling, and four decoder blocks with upsampling operators. The depth of each block is configured as [1. 1, 1, 1) for the encoders and {1, 1, 1, 1) for the decoders, respectively.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly stated the contributions and scope in the Abstract and Intro-
duction.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We describe the limitations in the Section S
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have presented the information in experimental section.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We will release our code after our paper gets accepted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are specified in method section.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We use fixed seed for both model training and data shuffling, and can reproduce the reported metrics.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the type of resources used in the experiments in Section 4.1
9. Code Of Ethics
17

---

Question:
Does the research conducted in the paper conform, in every respect. with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discusses the broader impact in Section
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The released model does not have high risk for misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite corresponding papers for the asserts we use in Section 4.1
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not involve the introduction of new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
18


Reviews:
[{"review_id": "8FKKJkujx3", "reviewer": "Reviewer_K6pm", "summary": "This paper solves the sparse-view scene reconstruction problem by Large Scene Model, a unified scene reconstruction model via unposed RGB images. The model utilizes a ViT backbone for extracting the feature and uses cross-view attention to align the multi-pose feature for consistent features. The 3D scene is further rendered from the 3D semantic field derived by the multi-view features. The unified model is capable of multiple 3D-based tasks including novel view synthesis and 3D language-based segmentation. Experiments showed that the work achieves better results with limited performance sacrifices in the NVS task and higher performance in the multi-view language-based segmentation task.", "questions": "1. Did the authors try replacing the designed module with large-scale pretrained models, such as using pretrained monocular depth estimation model?\n2. The design philosophy is similar to multi-view image generation works. Can this model output high-quality and consistent multi-view images, as the fashion of Free3D? \n3. The term of “language-driven segmentation” is not quite clear to me. Does it mean semantic segmentation?", "limitations": "The authors claimed that the major drawback of the model is the VRAM consuming. The social impact of this work mainly related to potential misuse of 3D assets and can be solved via integrating watermarks into the generated result.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "1. The model is general and multi-purpose in sparse-view scene reconstruction.\n2. The model can achieve better results while still obtaining lighting-fast rendering speed and can be applied to real-time reconstruction.", "weaknesses": "1. The technical contribution is limited. The model is generally designed via multi-purpose modules glued attention and Transformers, which is a straightforward and widely applied idea. There is no significant new problem has arisen and novel solutions proposed.\n2. The performance comparison with NVS-related works is limited. Firstly, the authors train and run comparison experiments on the same dataset, which can be biased. Secondly, several popular scene datasets incorporated in similar works (such as RealEstate10k) are not utilized in this work. Thirdly, methods similar to pixelSplat such as Splatter Image[1] are not included in comparison.\n3. The presentation can still be improved. Firstly, the authors titled their work “Large Scene Model”, while the design is more similar to the idea of pixel-based Gaussian splatting (such as pixelSplat and GaussianImage). Secondly, each module's input and output data type cannot be directly recognized from the pipeline graph. \n4. The bibliography of this paper lacks some related works, such as Splatter Image[1], which is also an image-space Gaussian splatting method. \n\nReference: \n[1] Szymanowicz, Stanislaw, Chrisitian Rupprecht, and Andrea Vedaldi. \"Splatter image: Ultra-fast single-view 3d reconstruction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10208-10217. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 1 (K6pm) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.  \n\n**[W1]: No significant new problem has arisen and novel solutions proposed?**  \nWe acknowledge that our work builds upon the contributions of many giants, such as DUSt3R [1] (end-to-end point map regression model), pixelNeRF [2] (one of the first generalizable NeRFs), pixelSplat [3] (one of the first generalizable 3D-GSs), and the concept of large-scale training. However, we respectfully disagree with the assertion that \"no significant new problem has arisen and novel solutions proposed.\"  \nWe target a very practical yet under-explored scenario: end-to-end semantic 3D reconstruction directly from images. While previous literature reduces this problem to “Computing Camera Parameters” followed by “Training 3D Representation”. We, for the first time, unify these problems into one differentiable vision system.  \nTechnically, while there are many alternative design choices, our method successfully integrates these components into an end-to-end trainable system using a standard Transformer architecture, demonstrating the feasibility of future scalable 3D training.  \nWe kindly suggest that the methods and ideas presented in this paper carry significant research value that can impact future 3D deep learning studies.\n\n\n\n\n\n\n**[W2]: Evaluation of generalizable methods on new datasets.**  \nTo avoid any possibility of overfitting with existing generalizable methods and considering the suggestion by Reviewer #MCB6, we adopt the Replica dataset [3], which is a photorealistic simulated 3D dataset containing accurate annotations of RGB, dense depth maps, and semantic maps for thorough evaluation. Specifically, the LSM generalizes well from the real-world ScanNet test set (25.44dB) to the simulated Replica test set (23.10dB). LSM also produces the best depth estimation metrics and is the only generalizable method that can enable 3D semantic segmentation. Splatter Image is an ultra-fast monocular 3D object reconstruction method using 3DGS, and we utilize the provided checkpoint for evaluation. However, while Splatter Image addresses object 3D reconstruction well when the background is masked out, it cannot handle scene-wise reconstruction well with complex backgrounds (please see the attached Fig.1).  \n\n| Methods/Metrics    | GT Camera Parameters | mIoU ↑ | rel ↓ | PSNR ↑ |\n|--------------------|----------------------|--------|-------|--------|\n| pixelSplat         | Required             | None     | 20.14 | 26.28  |\n| Splatter Image     | Required             | None      | None      | 12.37  |\n| Ours               | Not Required         | 0.51 | 4.91 | 23.10  |\n\n\n\n\n**[W3]: Improve the presentation.**  \nWe will highlight the input and output formats in the methodology section in a future version.   \nAlthough our method architecture exhibits some similarity to generalizable Gaussian models, we emphasize that the investigated problem is very different. pixelSplat focuses on the standard NVS setting, where **calibrated images** are assumed and the emphasis is solely on image synthesis. In contrast, our overarching goal is to develop a versatile 3D foundation model that unifies scene reconstruction and understanding with **uncalibrated images**, which is more aligned with real-world applications to deploy a single model. We will adjust the title to emphasize our unification of 3D tasks according to Reviewer 8Lhg's suggestions. Thank you for the feedback.\n\n**[W4] Missing citation.**  \nSplatter Image [4] presents one of the first solutions for ultra-fast monocular 3D object reconstruction using 3D Gaussian Splatting as a representation. It takes a single or two object-wise images as input for reconstructing the 3D representation without test-time optimization, while demonstrating promising reconstruction accuracy. We will incorporate a detailed discussion in our revision.\n\n\n**[Q1] Replace encoder using monocular depth?**  \nThe “Dense Geometry Prediction” cannot be replaced with a monocular depth estimator. Our method takes unposed images as input to build global 3D point maps with versatile attributes for many 3D problems. It requires a stereo depth estimator to unify the point maps from two viewpoints into the same coordinate system.\n\n\n**[Q2] Can this model perform generation, like Free3D?**  \nOur work reconstructs 3D scenes based on unposed images, similar to 3D-GS, but without the need for COLMAP pre-computation. The supervision comes from the new view interpolated between the training views, meaning it is not capable of extrapolating viewpoints (generation). We will clarify this point and discuss the potential to extend the proposed framework into a generative approach in the future work section.\n\n**[Q3] What’s “language-driven segmentation”?**  \n\"language-driven segmentation\" means segmenting areas based on a set of language descriptions  (e.g., “wall,” “floor”). This allows us to associate each Gaussian with text to render the 2D semantic map. Please refer to [5] for more detailed explanations.\n\n\n**Reference:**  \n[1]. DUSt3R: Geometric 3D Vision Made Easy. CVPR 2024  \n[2]. pixelnerf: Neural radiance fields from one or few images. CVPR 2021  \n[3]. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR 2024  \n[4]. Splatter Image: Ultra-Fast Single-View 3D Reconstruction, CVPR 2024  \n[5].  Language-driven Semantic Segmentation, ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RUyjosT8U6", "reviewer": "Reviewer_fir7", "summary": "The paper aims to train a network that takes in a set of unposed images and directly produces a semantic radiance field.\n\nThe method utilizes a single Transformer-based model that learns the attributes\nof a 3D scene represented by a point-based radiance field.  A decoder produced 3D Gausians that can be splatted to make novel images, depth estimates, and semantic segmentations.", "questions": "It is interesting that the 3D methods, which have access to multiple views of the same scene do not perform as well as LSeg in Table 1.   This is counter-intuitive.  Can you please explain why?\n\nThe results on multiview depth accuracy are kinda amazing.   Why is the proposed method better than ones that take the camera parameters?   Is it due aligning the scene scale (do all the methods get the same method for scale alignment?\n\nThe novel view synthesis images look very good.   Can you please provide some info about how close the novel cameras are to the reference ones provided at inference time?   Is there a way for you to quantify and compare to PixelSplat the NVS results as the novel cameras deviate further and further from the reference ones?", "limitations": "Limitations are discussed briefly in the second paragraph of the conclusion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a transformer architecture for producing 3D gaussians with rich features from unposed images, which seem very valuable.   The design choices in the proposed system are well-chosen from methods available at this time, leading to a system that has a good combination of little-compute and competitive-accuracy on three different tasks (nvs, depth, semantics).", "weaknesses": "The paper shares goals and ideas with \"Scene Representation Transformers\" (Sajjadi et al., CVPR 2022) and its follow up work Object Scene Representation Transformer (NeurIPS 2022) and RUST: Really Unposed SRT (CVPR 2023).   This paper is different, because it ultimate produces a set of gaussians rather than a LLFF or NeRF volume, and it distills features from 2D foundation models.   However, it is similar in that a transform encoder and decoder produces a scene representation directly from a set of images, which is then used for novel view synthesis, depth estimation, and semantic segmentation.   In any case, those paper seem fairly similar in concept and so I think they should be discussed in the related work, and possibly approach sections.\n\nThe ablation study in table 4 suggests that the key methods in the paper have little impact on the results of NVS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 2 (fir7) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.  \n\n**[W1] Discussion w/ SRT and RUST.**  \nScene Representation Transformer (SRT)[1] and RUST[3] have pioneered the exploration of representing multiple images as a \"set latent scene representation\" (OSRT[2] utilized Slot Scene Representation) and generating novel views even in the presence of flawed camera poses or without any pose information. In contrast, our method targets a holistic model for semantic 3D reconstruction from unposed images, using point-based representation for efficient rendering speed. We will include this discussion in the revision.\n\n**[W2] Table 4 suggests little impact on the results of NVS.**  \nTo formulate the well-represented radiance field, the Point-wise Aggregation module takes the pixel-aligned (x, y, z, r, g, b) as input (line 153) for regressing point-wise attributes, which already contain the complete appearance observation within the input views for interpolation. Thus, additional incorporation of feature maps does not provide visual cues to enrich the radiance field. However, the Point-wise Aggregation module design is important as it enables the lifting of features from inconsistent stereo image features to consistent feature fields, and the Multi-scale Fusion further improves the lifting accuracy.\n\n**[Q1] Novel view semantic segmentation with LSeg?**  \nWe notice that LSeg fails to produce view-consistent segmentation results but generates reasonably good per-view segmentation results. Our hypothesis is that LSeg is a well-trained 2D model for language-driven semantic segmentation, serving as a foundation for our method to understand 3D. Our method lifts the view-inconsistent multi-view feature maps from LSeg into a consistent 3D feature field in a zero-shot manner, utilizing the Point-wise Aggregation module. See the figures in the attached PDF (Fig.2) to see the consistency difference.\n\n**[Q2] High multiview depth accuracy induced by aligning scene scale?**  \n\nYes, the improved geometry accuracy stems from the alignment and combination of scenes, and our training data consists of dense depth annotations of various scenes for supervision. \n\n**[Q3] How to define novel cameras? Can they deviate from the reference one?**  \nThank you for acknowledging the good rendering quality. The rendered novel camera positions and orientations are determined by interpolation between the first and second reference (or support) images. Specifically, orientation interpolation is conducted using Spherical Linear Interpolation [4] between the two sets of quaternions, while position interpolation is conducted using linear interpolation.  \nWe acknowledge that both our method and pixelSplat are reconstruction methods that build the radiance field using pixel-aligned Gaussians for interpolation, and thus cannot perform generation tasks that significantly deviate from the reference images.\n\n\n\n**Reference:**  \n[1]. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations, CVPR 2022  \n[2]. Object Scene Representation Transformer, NeurIPS 2022  \n[3]. Rust: Latent neural scene representations from unposed imagery, CVPR 2023  \n[4]. Slerp from wikipedia page  (we cannot provide the direct link in author response)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "236S6RR30f", "reviewer": "Reviewer_MCB6", "summary": "This paper presents the Large Scene Model (LSM), which generates semantic radiance fields from uncalibrated RGB images using a unified Transformer-based framework. LSM can infer geometry, appearance, and semantics simultaneously and synthesize label maps in real-time. The model integrates multi-scale fusion and features from 2D models to enhance accuracy and efficiency.", "questions": "1. Module Timing: I am curious about how the authors manage to use eight cross-attention modules and still achieve reconstruction in 0.1 seconds. Please provide the time consumption for each step.\n\n2. Image Resolution: What is the resolution of the images? More details regarding the inference process should be provided, especially concerning the time comparison.", "limitations": "The authors have discussed limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Unified Framework: LSM combines multiple 3D vision tasks into a single framework, streamlining the process and reducing complexity.\n\n2. Real-time Performance: The model achieves real-time 3D reconstruction and rendering, suitable for applications needing fast processing.\n\n3. Enhanced Feature Fusion: By incorporating 2D model features, LSM improves the quality of feature lifting and semantic understanding, enhancing overall performance.", "weaknesses": "1. Dataset: I recommend the authors organize the training and testing phases in alignment with previous methods (NeRF-DFF and Feature-3DGS) and provide results on the Replica Dataset. The authors have not sufficiently justified deviating from the baseline evaluation split. Furthermore, an explanation is needed for the significant performance discrepancy of the baselines between the Replica Dataset and the authors' setup. Additional training details may also be necessary.\n\n2. Writing: The paper's abstract, introduction, and methods sections require improvement.  Specifically, the methods section should introduce each module and their interconnections from a high-level perspective rather than presenting them as isolated components.\n\n3. Method Details: Do the authors use camera parameters? If so, why are camera parameters mentioned in line 117? If camera parameters are used, the model cannot be described as \"unposed.\"\n\n4. Visualization: In Figure 4, there are category colors that are not listed in the legend. Additionally, a more diverse set of results should be displayed, as the current experimental set predominantly features sofas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 3 (MCB6) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.    \n**[W1.1]: Reasons for deviation from Replica, and performance discrepancy.**  \nThe reasons for the deviation are threefold:\n\n1). The processed Replica datasets (by Feature-3DGS) lack ground-truth depth maps for geometric accuracy. Feature-3DGS uses COLMAP for SfM results, leading to misaligned coordinate systems.\n2). Feature-3DGS uses LSeg as GT for evaluating semantic quality, while we use annotated GT semantic labels.\n3). Replica datasets are simulated under ideal environments, whereas our draft focuses on real-world textures, lighting changes, and sensor noise.\n\nThe performance discrepancy of Feature-3DGS stems from resolution and semantic ground-truth differences. Our experiments follow the generalizable NVS method, pixelSplat, using 256x256 resolution and cropped boundaries, reducing the valid field-of-view. We also use annotated ScanNet for semantic segmentation, rather than LSeg as ground-truth.  \n\n**[W1.2]: Results on Replica Dataset with explanation.**  \nComparing our methods with NeRF-DFF and Feature-3DGS is challenging due to their assumptions of dense views, long optimization steps, and precomputed camera poses, while ours uses sparse and pose-free views. pixelSplat also assumes known camera parameters. This places our method at a disadvantage, but Tab. 1 in our submission demonstrates its strength.\n\nTo further validate, we compare with Feature-3DGS on the Replica dataset, using Semantic-NeRF [1] preprocessed data with annotated RGB-D-Semantics (256x256 resolution). We randomly sample 2,000 points from GT depth maps for initializing Feature-3DGS and use 70 images for training and 10 for testing, following their setup.\n\nLSM generalizes well from the real-world ScanNet test set (25.44dB) to the simulated Replica test set (23.10dB) in zero-shot, even without GT camera parameters. Our model builds the feature field in a feed-forward pass, achieving high quality. Notably, our method achieves the best geometric accuracy without using epipolar attention (pixelSplat, need pose) or requiring test-time optimization with dense views and GT camera parameters (Feature-3DGS, need pose).\n\n\n\n| Methods/Metrics  | Support View | GT Camera Parameters | Training Time | mIoU (Seg.) ↑ | rel (Depth) ↓ | PSNR (RGB) ↑ |\n|------------------|---------------|----------------------|---------------|---------------|---------------|--------------|\n| Feature-3DGS     | 70            | Required             | 18.5 min      | 0.62          | 8.25          | 31.89        |\n| Nerf-DFF         | 70            | Required             | 1.1 min       | 0.49          | 14.33         | 24.67        |\n| pixelSplat       | 2             | Required             | 0.06 sec      | None          | 20.14         | 26.28        |\n| Ours             | 2             | Not Required         | 0.09 sec      | 0.51          | 4.91          | 23.10        |\n\n\n\nTo be more aligned with our assumed setting, which are more practical in real applications, we proposed to benchmark the performance of investigated baselines under the sparse view setting. We reduce the support view number for all methods from 70 to 20, while preserving the same test sets. In the table below, Feature-3DGS drastically decrease the PSNR with degraded geometry and semantic accuracy.\n\n| Methods/Metrics  | Support View | GT Camera Parameters | Training Time | mIoU (Seg.) ↑ | rel (Depth) ↓ | PSNR (RGB) ↑ |\n|------------------|---------------|----------------------|---------------|---------------|---------------|--------------|\n| Feature-3DGS     | 20            | Required             | 18.5 min      | 0.46          | 32.44         | 19.61        |\n| Nerf-DFF         | 20            | Required             | 1.1 min       | 0.39          | 45.09         | 16.27        |\n| pixelSplat       | 2             | Required             | 0.06 sec      | None          | 27.13         | 20.62        |\n| Ours             | 2             | Not Required         | 0.09 sec      | 0.45          | 5.614         | 18.80        |\n\n**[W1.3]. Training details.**  \nWe included training details in Sec. 4.1. We use the default configuration to train Feature-3DGS and NeRF-DFF. For pixelSplat, we use its pretrained checkpoint and code without any modifications to perform the evaluation.  \n\n\n\n\n**[W2]:Detailed module design.**  \nWe will add a diagram to illustrate module interconnections in the revision. Additionally, we plan to release the source code for reproduction.\n\n\n**[W3]:Why use camera parameters in L117?**  \nThey are needed in training and evaluation to obtain the ground-truth point maps, and specify the target view. They are not required for inference.  \n**[Q1]: Additionally, a more diverse set of results should be displayed.**  \nPlease refer to supplementary webpage for ten free-view videos. We also incorporate additional comparisons in the Fig.3 of the attached PDF.\n\n\n**[Q2]:Module Timing? Image resolution?**  \nWe study the computational cost (time) of each module by inferring 1,000 times with the model and obtaining the average inference cost for each model.  \n| Module | Inference Time (seconds) |\n|--------|--------------------------|\n| Dense Geometry Prediction (Sec.3.1) | 0.0297               |\n| Point-wise Aggregation (Sec.3.2)   | 0.0464               |\n| Feature Lifting (Sec.3.3)   | 0.0199               |\n| **Total** | **0.096**        |\n\nThe tested resolution (256x256) is aligned with the generalizable 3D-GS method pixelSplat.  pixelSplat is slightly faster than our framework 0.06s while it requires additional steps to run Structure-from-Motion (e.g., 20.52s for each scene on ScanNet on average) to obtain the camera parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJLsnYID5d", "reviewer": "Reviewer_8Lhg", "summary": "The authors proposed the Large Scene Model (LSM), a novel 3D scene understanding framework that unifies multiple vision tasks within a single model. LSM represents a scene using pixel-aligned point maps, integrating geometric, appearance, and semantic information into a unified representation. By leveraging a Transformer architecture with cross-view and cross-modal attention, the model effectively incorporates multi-view cues and semantic knowledge from a 2D vision model.\n\nLSM's design enables efficient scene-level 3D semantic reconstruction and rendering in real time on a single GPU. The model's integration of a 2D semantic model allows for open-vocabulary understanding, extending its applicability to diverse real-world scenarios. Furthermore, by consolidating multiple tasks within a single model, LSM minimizes error propagation, leading to more robust and accurate results compared to state-of-the-art baselines.", "questions": "- For the quantitative comparisons given in Figure 3, are they predicted from the input view, or are they for a novel view?", "limitations": "- My understanding is that in this method, all the Gaussians being generated are pixel aligned with the original input images. Is that a limitation of the method, since that would make the model unable to model large pose divergences that require rendering regions not originally visible in the input view, for instance, the back side of a sofa etc.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The described technical approach in this work is sound and clearly presented. The contributions from the various proposed modules are well ablated and investigated in the experiments (Table 4).\n* The model demonstrates high inference efficiency compared to other approaches. with reconstruction time of 0.1s and rendering at 270 due to the underlying 3DGS representation that is being generated.\n* I like that the model reconstructs the underlying 3D representation in a single feedforward pass, as compared to multiview + test time optimization for fusion approaches. This improves the speed and efficiency for inference. It is good to see compelling quality based on the novel view synthesis.", "weaknesses": "* I think the main contribution of this paper is the unification of the various scene modeling tasks into the same model, including geometry, color and semantics. The authors further claimed in the abstract and introduction that multitask training end-to-end allows LSM to outperform state-of-the-art baselines. However the paper did not ablate the multi-task learning design choice. For instance, what if some of the tasks are removed (e.g., semantic feature prediction). How does that affect the performance of the other tasks?\n* A suggestion is that for Figure 5, it is unclear how much pose divergence there is between the input source view and the synthesized novel view. It would be helpful to also show the source view supplied as input to the model.\n* The paper is named Large Scene Model, which seems to suggest something to do with model parameter scaling, hence large. However the paper does seem to do much scaling on model size. So perhaps a more accurate terminology would be Multitask or Unified Scene Model? \n\nNits.\n* Line 153: Typo: to serve as input?\n* In Tables 1-4, I suggest highlighting the best (and possibly second-best result) for easier comparison of the various experiments.\n* In Table 4, why is + Multi-scale Fusion indented?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 4 (8Lhg) for recognizing the contribution of our paper and offering insightful comments. Please find our response to the feedback below.       \n\n**[W1]: Ablate the multi-task design choice?**  \nWe ablate the “novel view feature synthesis (Eq.4)” and “geometry prediction (Eq.1)” tasks in the following table. The row labeled “no Feat.” indicates the removal of the multi-scale fusion of LSeg features (green part in Point-wise Aggregation of Fig. 2). We observe that incorporating semantic task learning into point-wise aggregation enables the model to lift view-inconsistent feature maps to a view-consistent feature field, while it does not impact view synthesis and depth tasks a lot. We believe the input to “Point-wise Aggregation” contains pixel-wise point maps (x, y, z) and RGB (r, g, b) from the support images, which already provide the complete observation within the input views for interpolating novel views and depth prediction. Thus, additional incorporation of feature maps does not provide visual cues to enrich the radiance field.\nRemoving geometric supervision leads to the model diverging quickly, which means pixel-aligned depth regression helps the model recognize correspondences across input views, guiding the model's learning when it is unaware of the camera parameters. The experiment setting follows Table 4 of the main draft.\n\n\n| Methods/Metrics           | mIoU ↑ | Acc. ↑ | rel ↓  | &tau; ↑ | PSNR ↑ | SSIM ↑ |\n|---------------------------|--------|--------|--------|---------|--------|--------|\n| Full Model                | 0.599  | 0.8125 | 4.09 | 61.39 | 21.10  | 0.7752 |\n| Full Model (no Feat.)     | None     | None  | 4.18 | 60.84 | 20.97  | 0.7745 |\n\n\n**[W2]: Update Figure 5, and the pose divergence for the novel views?**  \nWe will include the source (support) views in the revision; thank you for the suggestion. We specify the rendered novel camera positions and orientations by interpolating between the first and second reference (or support) images for free-view exploration (see the supplementary webpage). Specifically, the orientation interpolation is conducted using Spherical Linear Interpolation [1] between the two sets of quaternions, while the position interpolation is conducted using linear interpolation.  \n\n**[W3]: More accurate terminology?**  \nWe will take your suggestion into account and revise it accordingly.  \n\n\n\n**[Q1]: Typos and highlighting the table?**  \nWe will fix it.\n\n\n**[Q2]: Why is “+ Multi-scale Fusion” indented in Table 4?**  \nIndented row (Multi-scale Fusion) means it is based on the rows above (Fuse LSeg Feat. and Fuse Encoder Feat.), indicating that we apply multi-scale fusion to both. We have clarified this part in the revision.  \n\n\n**[Q3]:  Input view or novel view in Figure 3?**  \nThey are novel views.  \n\n**[Limitation]**  \nThanks for pointing this out. Our method, along with all the adopted methods (pixelSplat, Feature-3DGS, NeRF-DFF), belongs to the category of reconstruction models, which focus on building the 3D based on input observations. A discussion about extending to generation for synthesizing new viewpoints with large pose divergences will be included in our revision.  \n\n\n**Reference:**  \n[1]. Slerp from wikipedia (we cannot provide the direct link in author response)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for acknowledging that the work is sound and clearly presented (8Lhg). The presented Transformer-based design is very valuable (fir7) and general (K6pm), running lighting-fast (K6pm, fir7, MCB6, 8Lhg) while achieving compelling quality (K6pm, 8Lhg). We have addressed all the questions posed by the reviewers with additional experimental results. We will carefully revise our main manuscript, following those suggestions.

Meta-review: This paper proposes the Large Scene Model (LSM) to process unposed RGB images directly into semantic radiance fields. Evaluation is performed on eight unseen datasets from ScanNet, and the results are impressive compared to related methods based on posed RGB images. The paper received unanimous praise from all reviewers for its low computational cost and competitive accuracy on three different tasks: NVS, depth estimation, and semantic segmentation. AC recommends the authors to include all information in the rebuttal into the final camera-ready version and congratulates the authors on the paper acceptance.
Decision: Accept (poster)