Large Spatial Model: End-to-end Unposed Images to Semantic 3D

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 01:27:25

Abstract: Reconstructing and understanding 3D structures from a limited number of images is a classical problem in computer vision. Traditional approaches typically decompose this task into multiple subtasks, involving several stages of complex mappings between different data representations. For example, dense reconstruction using Structure-from-Motion (SfM) requires transforming images into key points, optimizing camera parameters, and estimating structures. Following this, accurate sparse reconstructions are necessary for further dense modeling, which is then input into task-specific neural networks. This multi-stage paradigm leads to significant processing times and engineering complexity.

In this work, we introduce the Large Spatial Model (LSM), which directly processes unposed RGB images into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward pass and can synthesize versatile label maps by interacting through language at novel views. Built on a general Transformer-based framework, LSM predicts global geometry via pixel-aligned point maps. To improve spatial attribute regression, we adopt local context aggregation with multi-scale fusion, enhancing the accuracy of fine local details. To address the scarcity of labeled 3D semantic data and enable natural language-driven scene manipulation, we incorporate a pre-trained 2D language-based segmentation model into a 3D-consistent semantic feature field. An efficient decoder parameterizes a set of semantic anisotropic Gaussians, allowing supervised end-to-end learning. Comprehensive experiments on various tasks demonstrate that LSM unifies multiple 3D vision tasks directly from unposed images, achieving real-time semantic 3D reconstruction for the first time.

Main paper: 
 introduction: 
 The computer vision community has devoted considerable effort to recovering and understanding 3D information (e.g., depth and semantics) from 2D sensory data (e.g., images). This process aims to derive 3D representations that encapsulate both geometric and semantic details from cheap and widely available 2D data, facilitating further interaction, reasoning, and planning within 3D physical world. Traditional approaches [1] tackle this by pipelining several distinct tasks: detecting, matching, and triangulating points for initial sparse reconstructions and the subsequent dense reconstruction, followed by the integration of specialized submodules for semantic 3D modeling.Recent developments in this domain have markedly proceeded with a more powerful representation using both sparse reconstruction, and subsequent dense 3D modeling via Multi-View Stereo (MVS) [2, 3] , Neural Radiance Field (NeRF) [4] , and 3D Gaussian Splatting (3D-GS) [5] , This trend influenced various industries, including autonomous driving [6] , robotics [7] , digital twins [8] , and virtual/augmented reality (VR/AR) [9, 10] . Due to the complexity of inferring 3D information from 2D images, previous methods have broken down the holistic task into distinct, manageable subproblems. However, this strategy propagates errors from stage to stage and downgrades the performance of subsequent tasks. For instance, the critical step of precomputing camera poses -utilizing Structure from Motion (SfM) [1] -has proven to be vulnerable and often fails in scenes covered by a sparse number of views or exhibiting low-textured surfaces [11] . Such inaccuracies in camera pose estimation can ultimately lead to imprecise interpretation of the 3D scene.Furthermore, reasoning about and interacting with the environment would benefit from a comprehensive 3D understanding. Open-vocabulary methods, which perform semantic segmentation without relying on a fixed set of labels, provide notable flexibility. However, unlike single-image understanding, the absence of large-scale and diverse 3D scene data with accurate multiview language annotations complicates the challenge. Efforts have been made to integrate 2D features into frameworks such as NeRF [12] [13] [14] and 3D-GS [15, 16] . Yet, these methods, such as Feature-3DGS [15] , typically require overfitting each 3D scene separately with extensive captured viewpoints and preprocessing camera poses using Structure-from-Motion.To address the challenges outlined above, we propose for the first time a novel unified framework for these key 3D vision subproblems: dense 3D reconstruction, open-vocabulary semantic segmentation, and novel view synthesis from unposed and uncalibrated images. Our approach leverages a single Transformer-based model that learns the attributes of a 3D scene via semantic anisotropic Gaussians. Unlike previous methods that rely on epipolar Transformers with known camera parameters [17] [18] [19] or require extensive per-scene fitting [5, 15] , we employ a coarse-to-fine strategy. This strategy predicts dense 3D geometry using pixel-aligned point maps, progressively refining these points into anisotropic Gaussians in a single feed-forward pass.Our framework, dubbed Large Spatial Model (LSM), begins with a general Transformer architecture incorporating cross-view attention [20] , which constructs pixel-aligned point maps at a normalized scale, enabling generalization across various datasets. LSM further enhances point-based representations through multi-scale fusion and local context aggregation using a ViT encoder. Additionally, LSM performs hierarchical cross-modal fusion, integrating features from a pre-trained 2D semantic model into a consistent 3D feature field. Through differentiable splatting of the regressed semantic anisotropic Gaussians, LSM enables end-to-end supervision and supports real-time scene-level 3D semantic reconstruction and rendering without needing explicit camera parameters. This allows for efficient, data-driven rendering of labels from novel viewpoints, as demonstrated in Figure 1 .Our contributions are summarized as follows:• We introduce a unified 3D representation and an end-to-end framework that addresses dense 3D reconstruction, 3D language-based segmentation, and novel-view synthesis directly from unposed images in a single forward pass.• Our method leverages a Transformer architecture with cross-view attention for multi-view geometry prediction, combined with hierarchical cross-modal attention to propagate geometryrich features. We further integrate a pre-trained semantic segmentation model to enhance 3D understanding. By aggregating local context at the point level, we achieve fine-grained feature integration, enabling the prediction of anisotropic 3D Gaussians and efficient splatting for RGB, depth, and semantics.• Our model performs multiple tasks simultaneously with real-time reconstruction and rendering on a single GPU. Experiments show that our unified approach scales effectively across different 3D vision tasks, surpassing many state-of-the-art baselines without the need for additional SfM steps. 
related work: 
 SfM and Differentiable Neural Representation Structure-from-Motion (SfM) aims to jointly estimate camera poses and reconstruct sparse 3D structures from multiple views. Traditional pipelines [1] involve multiple stages, including descriptor extraction, correspondence estimation, and incremental bundle adjustment. Recent advances in learning-based techniques [21] [22] [23] [24] [25] have further improved the accuracy and efficiency of SfM. These methods are widely adopted in 3D vision tasks, where differentiable neural representations typically assume accurate camera poses provided by SfM. For instance, NeRF [4] and its successors [26] rely on poses estimated offline via COLMAP [1, 27] .Similarly, 3D Gaussian Splatting [5] uses SfM-generated 3D points for initialization and has been applied to robotics [28] [29] [30] , healthcare [31] [32] [33] , and many other domains [34] [35] [36] . Beyond novel view synthesis, lifting 2D features to 3D has gained traction in various editing tasks [13, 15, 37, 14] .End-to-End Image-to-3D 3D reconstruction is a long-standing problem in computer vision, with traditional approaches like SfM [38, 39, 1] , Multi-view Stereo (MVS) [3, 2, 40, 41] , and Signed Distance Function (SDF) [42, 43] . More recent techniques utilize neural representations, including implicit [4] and explicit [5] formats to generate 3D models. Semantic understanding is often integrated during the reconstruction process [44] , or through additional optimization steps [12, 13] . However, most methods depend on a preprocessing step like SfM [1] to estimate camera calibration, poses, and sparse point clouds before dense reconstruction, either through feed-forward prediction or test-time optimization. This reliance on calibration and pose estimation limits scalability with largescale data, contrasting the success seen with large foundation models [45] . The latest pose-free, feedforward approaches, such as Scene Representation Transformers [46] [47] [48] , have advanced the concept of representing multiple images as a "set latent scene representation," allowing for novel view generation even in the presence of inaccurate camera poses or without any pose information. However, these methods struggle to produce explicit geometry. DUSt3R [20] addresses this limitation by predicting dense point clouds directly from unposed stereo image pairs, enabling pixel-aligned geometry prediction at normalized scales. Practically, dense point prediction requires accurate multi-view RGB-D pairs, which significantly limits its scalability. InstantSplat [49] addresses this by utilizing novel-view synthesis with only posed image data and employing Gaussian Bundle Adjustment to jointly optimize camera and scene parameters for ultra-fast dense 3D reconstruction.In contrast, our framework offers a holistic solution for dense 3D semantic reconstruction from unposed images. It integrates dense 3D geometry reconstruction, and language-based 3D interaction, while minimizing the need for extensive data annotation by using novel view synthesis as a core task. Since dense 3D annotations are often scarce in real-world scenarios, we propose semantic anisotropic Gaussians to lift 2D features map to 3D semantic embeddings without additional annotation. Our approach addresses higher-level 3D tasks in perception and dense 3D reconstruction compared to DUSt3R [20] , by utilizing lightweight annotations and solving these tasks jointly within a unified framework. 
methods: 
 Overview Figure 2 illustrates the architecture for training the Large Spatial Model (LSM). During training, the input consists of stereo image pairs along with associated camera intrinsics and poses:{(I i ∈ R H×W ×3 ), (T i ∈ R 3×4 ), (K i ∈ R 3×3 )} 2 i=1. At inference, however, unposed images can be directly fed into the framework. The pixel-aligned geometry is predicted using a standard Transformer architecture [50] with cross-attention between input views. Dense prediction heads are employed to regress normalized point maps during training:{D i ∈ R H×W ×3 } 2i=1 (see Sec. 3.1). To support fine-grained semantic anisotropic 3D Gaussian regression, which represents the 3D scene and lifts generic feature fields from pre-trained 2D vision models, we apply point-based attention with learnable positional encoding in a local window. This propagates features from neighboring points (Sec. 3.2), effectively merging encoded features with rich semantics (Sec. 3.2) at multiple scales using 2D pre-trained models (Sec. 3.3). New views from the semantic radiance fields can be decoded using splitting [5] on the target poses (Sec. 3.4). During inference, semantic anisotropic Gaussians are directly predicted, and the renderer takes the camera parameters derived from the point maps. An overview of the model architecture is shown in Figure 2 . Instead of adopting a conventional Transformer with Epipolar attention-which can be inefficient as pixel-wise prediction requires hundreds of queries on sampled epipolar lines [17, 18] -we implement an encoder-decoder structure for directly regressing view-specific point maps at normalized scales. Cross-view attention is utilized to aggregate multi-view information efficiently.We employ a Siamese ViT-based encoder [50] that processes stereo images using shared weights. It involves the patchification and tokenization of images, followed by the integration of sinusoidal positional embeddings. To directly regress the pixel-aligned point maps from the unposed images for view v ∈ {1, 2}, cross-view attention is also employed, enhancing the architecture's capacity to infer spatial relationships and propagate information between views-an approach that has proven effective in prior research [51, 20, 52] . The decoder block consists of interleaved self-attention for each view and cross-attention across views, which integrates tokens from both images. The inter-view decoder includes 12 attention blocks, akin to those utilized in previous multi-view stereo (MVS) studies [52, 20] . These blocks generate tokenized features for a subsequent Dense Prediction Transformer head (DPT) [53] , which estimates a pixel-wise point map in a normalized coordinate system along with confidence value:EQUATIONwhere M is pixel-aligned confidence map, same as DUSt3R, D indicates all valid points to the origin, M v,1 denotes the confidence map obtained from view v, expressed in the coordinate frame of view 1, α is a hyper-parameter that apply regularization, encouraging the network to perform robustly in challenging areas. The depth error is calculated byEQUATIONwhere the normalization factors (z and ẑ) indicate that the predicted and ground-truth pointmaps are processed by normalizing. For example, z is obtained by: norm(P 1 , P 2 ) =1 |D 1 |+|D 2 | v∈{1,2} i∈D v ∥P i v ∥ .Building on the foundational work in NeRF [4] and Multi-view Stereo [3] , which employ a coarseto-fine strategy for high-quality radiance field and depth estimation, we also aggregate the initial predicted geometry by applying a Transformer [54] at the point level, leveraging hierarchical representations to achieve more refined, point-based regression.Point-wise Attribute Prediction Rather than relying solely on a single network to represent the scene, we employ two Transformer-based networks optimized for distinct tasks: one for capturing "coarse" global geometry and another for "fine" local information aggregation. Initially, we integrate stereo point maps, including color information for each point primitive, formulated as {p i = (x i , y i , z i , r i , g i , b i )} N i=1 to serve as input. Unlike tokenized image patches, point primitives carry distinct geometric significance within Euclidean space. Inspired by recent advancements in pointcloud processing [55] [56] [57] , we employ a Transformer within a localized window to perform point-wise aggregation, selectively emphasizing key features from neighboring primitives. Point-wise encoding and decoding are essential for refining scene representation, utilizing multiscale aggregation across five hierarchical levels.After aggregating the point-wise features, we employ an additional layer of multilayer perceptron (MLP) to regress the parameters, representing the 3D scene through a set of anisotropic Gaussians [5] . The parameters include the opacity α, scale factor s, rotation r, and Spherical Harmonics coefficientsc i ∈ R 3 |i = 1, 2, ..., kwhere k = (K + 1) 2 is the number of coefficients of SH with degree K. The Gaussian centers µ are regressed from geometry prediction backbone. The color c of direction d is then computed by summing up all SH basis as c(d) = n i=1 c i B i (d), where B i is the i th SH basis. The final pixel intensity c is calculated by blending n ordered Gaussians overlapping the pixels using the following render function:EQUATIONThis equation efficiently models the contributions of each Gaussian to the pixel's final appearance, accounting for their transparency and layering order.To effectively combine multi-view image features with pointwise geometric information, we implement cross-model attention between two sets of tokens. The attention block fuses tokens from different sources by first applying self-attention to the input P, allowing each token to attend to other tokens within the same sequence. This process helps capture internal relationships and enrich the representation of the input token. Next, cross-attention is used, where two sets of tokens (P and F) from the latent layers of two different models are fused, enabling the integration of external information into P. Finally, a feed-forward network (MLP) further processes the updated information following cross-model fusion.The original point features P contain explicit and precise spatial information, which is critical for accurate geometry reconstruction. In contrast, the image token features F, from image encoder (Sec. 3.1) are rich in semantic content, providing important contextual information that enhances general understanding of the scene. Cross-model fusion enables the integration of detailed spatial geometry with semantic richness:Q = Proj(P), V, K = Proj(F), P = softmax QK ⊤ √ d k Vwhere P and F were normalized with a linear layer before projection.To facilitate semantic 3D representation, we augment the anisotropic 3D Gaussians with a learnable semantic feature embedding (a.k.a. semantic anisotropic Gaussians) and rasterize into the 2D image plane by blending Gaussians that overlap with each pixel using a feature rendering function.EQUATIONs indicates the final rasterized feature embedding on image plane, and s i is semantic embedding on anisotropic Gaussians. 3D Semantic Field from 2D Images After obtaining s, we optimize s i by minimizing the difference between the rasterized feature map and the feature maps generated by a pre-trained 2D model. Unlike the previous method [15] which requires test time optimization, we transform the estimation of the feature field into a fully learnable process.i ∈ R H ′ ×W ′ ×N } 2 i=1) from a pre-trained 2D multi-modal model [58] are inherently view-inconsistent due to the lack of spatial awareness during the model's training. To elevate multiview feature embeddings into a coherent 3D feature field for holistic 3D understanding, we introduce a dynamic fusion strategy employing an attention-based correlation module. This module is specifically designed to learn blending weights for each token within Point Transformer [54] from the input pixel-wise feature embedding (S i ). We employ attention blocks as described in Eq.3.2 to synchronize in the latent spaces through a supplementary set of cross-attention layers. The visual feature from LSeg [58] , denoted as Ŝ, is utilized for this purpose. This loss function is minimized during training by utilizing rasterized feature maps on new views S and directly inferred feature maps using ground truth images on new views Ŝ (LSeg [58] ), thereby facilitating the learning of blending weights for consistent semantic field regression.EQUATIONMulti-scale Feature Fusion To improve model efficiency, we propagate information from ViT Encoder feature F and the frozen semantic feature S, to the 3D latent space (point feature P) which has fewer tokens, thereby enabling selective attention to critical features. We further refine feature fusion across multiple stages, optimizing information flow while minimizing additional computational overhead. Novel view synthesis serves as an effective task to encode the complete geometric and appearance features into a low-dimensional 3D latent space, while recovering a set of semantic anisotropic Gaussians (G ∈ g i ∈ R 1×C N i=1 ) through learning from large-scale data and end-to-end training.Putting all together, our model can be optimized end-to-end:L = C(G, d) -Ĉ + λ 1 • D-SSIM(C(G, d), Ĉ) Photometric (6) + λ 2 • L dist (S(G, d), Ŝ) Semantic + v∈{1,2} λ 3 • L conf (D v,1 , Dv,1 )Geometric (7) where C and Ĉ are rasterized and GT pixel intensities, G denotes represented 3D scene using a set of 3D semantic anisotropic Gaussians, S and Ŝ denotes rendered LSeg feature extractor and feature on the target image, d indicates the direction and position at new views. In our methodology, we leverage both photometric loss and semantic loss to supervise the generation of rasterized new views. In order for geometry prediction and semantic feature lifting, we employ a confidence-weighted depth loss applied to the input views. The parameters λ 1 , λ 2 , λ 3 are set to 0.25, 0.3, and 1.5, respectively, as determined by the grid search. 
implementation details: 
 For our architecture, we employ ViT-Large as the encoder and ViT-Base as the decoder, complemented by a DPT head [53] for pixel-wise geometry regression. We initialize the geometry prediction layers using DUSt3R [20] . Point Transformer layers consists of 5 encoder and 4 decoder blocks with progressive downsampling and upsamping. The cross-model fusion strategy is implemented at the output of the last encoder and the output of the first decoder. The entire system is optimized Table 1 : Quantitative Comparison in 3D Tasks. We report novel-view synthesis, depth estimation quality, and open-vocabulary segmentation accuracy. Our method eliminates the need for any preprocessing in 3D tasks, while achieving performance comparable to other baselines that rely on SfM to obtain camera parameters and poses. We present examples of features rendered from novel viewpoints, illustrating how our method converts 2D features into a consistent 3D, facilitating versatile and efficient segmentation. Visualizations are generated using PCA [59] .end-to-end using the loss function described in Eq. 6. The training of our model contains 100 epochs, leveraging a combined dataset of ScanNet++ [60] and Scannet [61] , of 1565 scenes. Training is on 8 Nvidia A100 GPU lasts for 3 days. We start with a base learning rate of 1e-4 and incorporate a 10-epoch warm-up period. AdamW is employed as the optimizer for all experiments. Evaluation is conducted on 40 unseen scenes from ScanNet. Additionally, we assess on tasks: novel view synthesis, multi-view depth prediction, and 3D language-based semantic segmentation.Evaluation of Synthesized Images Quality Novel view synthesis is evaluated using NeRF-DFF [13] and Feature-3DGS [15] , both of which are capable of predicting RGB values as well as features. In addition, we compared our approach with the state-of-the-art, generalizable, posebased 3D Gaussian Splatting method, pixelSplat [18] , which generates point-based representations through a feed-forward pass. Unlike our method, these existing approaches rely on known camera intrinsics and poses prior to evaluation. As indicated in Table 1 , NeRF-DFF and Feature-3DGS tend to overfit on each individual scene, requiring significantly more time than our method, yet performing comparably in terms of output quality. pixelSplat utilizes an Epipolar Transformer, searching along the epipolar line using GT camera parameters to regress Gaussian attributes, resulting in longer inference times. Visualizations in Figure 4 demonstrate that our results are sharper and exhibit fewer artifacts than NeRF-DFF, and are comparable to Feature-3DGS and pixelSplat in performance.The semantic segmentation is evaluated by class-wise intersection over union (mIoU) and average pixel accuracy (mAcc) on novel views as metrics. Following the approach of Feature-3DGS [15] , we map thousands of category labels from diverse datasets into a set of common categories, including {Wall, Floor, Ceiling, Chair, Table , Bed, Sofa, Others}. We compare our model against two state-of-the-art 3D baselines with the capacity for generating RGB, semantics and depth on any view: Feature-3DGS [15] and NeRF-DFF [13] , which are based on 3D-GS [5] and NeRF [4] , respectively. Additionally, the model LSeg [58] , used as a 2D open-vocabulary segmenter for feature lifting, is included in our comparisons. We present statistics related to the semantic annotations on the adopted the ScanNet datasets in Table 1 , where LSM demonstrates competitive performance compared to baseline 3D methods that require We evaluate scene-level reconstruction by comparing our method to approaches that require per-scene optimization, such as NeRF-DFF and Feature-3DGS, which predicts both RGB and segmentation, and the generalizable 3D Gaussian Splatting method (pixelSplat). Notably, these methods require a pre-processing step to obtain camera poses using off-the-shelf SfM. Through end-to-end, data-driven training, our method achieves comparable visual quality to these approaches while reconstructing the 3D radiance field in a single feed-forward pass. ground-truth camera parameters and extensive per-scene optimization. The visualized results in Figure 5 illustrate that LSM can produce view-consistent semantic maps. In contrast, the 2D method LSeg yields detailed segmentation results but lacks cross-view consistency. To validate that LSM learns semantically meaningful features, we visualize the lifted feature field using PCA to reduce the high-dimensional features into three channels [13] . As shown in Figure 3 , LSM effectively generates a faithful semantic feature field through feed-forward inference using pair images.We also evaluate the performance of our model on the task of multi-view stereo depth estimation. We utilize the Absolute Relative Error (rel) and Inlier Ratio (τ ) with a threshold of 1.03 to assess each scene, similar to DUSt3R [20] . Since our approach does not rely on any camera parameters for prediction, we align the scene scale between the predictions and the ground truth. Specifically, we normalize the predicted point maps using the median of the predicted depths and similarly normalize the ground truth depths, following procedures established in previous literature [62, 20] to align the two sets of depth maps. We observe in Table . 1 that LSM achieves state-of-the-art accuracy on ScanNet datasets than the per-scene wise methods. Our model is significantly faster than baseline methods, as it only require a forward-pass. We conduct ablations to validate our desing effectiveness. Experiments are on both language-based segmentation and novel view synthesis. The quantitative results can be views at Table 2 .Cross-Model Feature Aggregation Incorporating the geometry encoder feature from ViT into the hidden layer of the point-aggregation layer (Sec. 3.2) demonstrates that such cross-model information flow significantly benefits the segmentation task, improving the mean Intersection over Union (mIoU) from 0.4562 to 0.5410 (Exp #1 → 2).Semantic Feature Fusion at Multi-Scale Employing cross-model fusion, where latent features of the semantic model are integrated into the middle layers of point-based aggregation, also improves injection of semantically rich embeddings (0.4562 to 0.5586, Exp # 1 → 3). The decoded features confirm that the lifted feature field produces higher-quality feature maps, with the semantic mIoU improving from 0.5586 to 0.6042 (Exp #3 → 4) through multi-scale fusion.Module Timing. We analyze the computational cost of each module by running inference 1,000 times on the ScanNet test dataset with the model, as shown in Table 3 , and calculating the average inference time for each module of the Large Spatial Model. Evaluation of Generalizable Methods on New Datasets. To avoid potential overfitting, we adopt the Replica dataset [63] , a photorealistic simulated 3D dataset with accurate RGB, dense depth maps, and semantic annotations for comprehensive evaluation. We use the same data preparation with Feature-3DGS. LSM generalizes well to the simulated Replica test set, achieving the best depth estimation metrics and enabling 3D semantic segmentation, which is unique among generalizable methods. Splatter Image [64] , an ultra-fast monocular 3D object reconstruction method using 3D-GS, performs well for object reconstruction with masked backgrounds but struggles with scene-wise reconstruction in complex backgrounds. 
conclusion, limitation, and broader impact: 
 We have introduced the Large Spatial Model (LSM), a unified framework for holistic 3D semantic reconstruction from uncalibrated and unposed images, with the added capability of interaction through language. LSM leverages cross-view attention to aggregate multi-view cues and utilizes multiscale cross-modal attention to integrate semantically rich features into a point-based representation. Hierarchical point-wise aggregation layers further refine these representations and enhance the integration of cross-modal attention. By splatting regressed anisotropic 3D Gaussians, LSM enables the generation of novel views with versatile label maps. LSM is highly efficient, capable of real-time end-to-end 3D modeling, and supports various downstream applications.While our method significantly accelerates semantic 3D scene reconstruction, it relies on a pre-trained model for feature lifting, which can increase GPU memory requirements during training, especially when the integrated 2D model has a large number of parameters. Additionally, the need for groundtruth depth maps, although there are millions of multi-view datasets annotated with them, could limit its scalability for internet-scale video applications.Our research enables efficient, real-time 3D scene-level reconstruction and understanding, which is advantageous for applications such as end-to-end robotic learning, AR/VR, and digital twins. However, there is potential for misuse, such as the arbitrary distribution of digital assets or privacy leakage related to building structures. These risks can be mitigated by embedding watermarks into the 3D assets [65] . 
 
 Appendix:  


Decision: This paper proposes the Large Scene Model (LSM) to process unposed RGB images directly into semantic radiance fields. Evaluation is performed on eight unseen datasets from ScanNet, and the results are impressive compared to related methods based on posed RGB images. The paper received unanimous praise from all reviewers for its low computational cost and competitive accuracy on three different tasks: NVS, depth estimation, and semantic segmentation. AC recommends the authors to include all information in the rebuttal into the final camera-ready version and congratulates the authors on the paper acceptance.

Reviews: review: paper_summary: paper_summary: This paper solves the sparse-view scene reconstruction problem by Large Scene Model, a unified scene reconstruction model via unposed RGB images. The model utilizes a ViT backbone for extracting the feature and uses cross-view attention to align the multi-pose feature for consistent features. The 3D scene is further rendered from the 3D semantic field derived by the multi-view features. The unified model is capable of multiple 3D-based tasks including novel view synthesis and 3D language-based segmentation. Experiments showed that the work achieves better results with limited performance sacrifices in the NVS task and higher performance in the multi-view language-based segmentation task.
strength_weakness: strength_weakness: 1. The technical contribution is limited. The model is generally designed via multi-purpose modules glued attention and Transformers, which is a straightforward and widely applied idea. There is no significant new problem has arisen and novel solutions proposed.
2. The performance comparison with NVS-related works is limited. Firstly, the authors train and run comparison experiments on the same dataset, which can be biased. Secondly, several popular scene datasets incorporated in similar works (such as RealEstate10k) are not utilized in this work. Thirdly, methods similar to pixelSplat such as Splatter Image[1] are not included in comparison.
3. The presentation can still be improved. Firstly, the authors titled their work “Large Scene Model”, while the design is more similar to the idea of pixel-based Gaussian splatting (such as pixelSplat and GaussianImage). Secondly, each module's input and output data type cannot be directly recognized from the pipeline graph. 
4. The bibliography of this paper lacks some related works, such as Splatter Image[1], which is also an image-space Gaussian splatting method. 

Reference: 
[1] Szymanowicz, Stanislaw, Chrisitian Rupprecht, and Andrea Vedaldi. "Splatter image: Ultra-fast single-view 3d reconstruction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10208-10217. 2024.
questions: questions: 1. Did the authors try replacing the designed module with large-scale pretrained models, such as using pretrained monocular depth estimation model?
2. The design philosophy is similar to multi-view image generation works. Can this model output high-quality and consistent multi-view images, as the fashion of Free3D? 
3. The term of “language-driven segmentation” is not quite clear to me. Does it mean semantic segmentation?
limitations: limitations: The authors claimed that the major drawback of the model is the VRAM consuming. The social impact of this work mainly related to potential misuse of 3D assets and can be solved via integrating watermarks into the generated result.
paper_summary: paper_summary: The paper aims to train a network that takes in a set of unposed images and directly produces a semantic radiance field.

The method utilizes a single Transformer-based model that learns the attributes
of a 3D scene represented by a point-based radiance field.  A decoder produced 3D Gausians that can be splatted to make novel images, depth estimates, and semantic segmentations.
strength_weakness: strength_weakness: The paper shares goals and ideas with "Scene Representation Transformers" (Sajjadi et al., CVPR 2022) and its follow up work Object Scene Representation Transformer (NeurIPS 2022) and RUST: Really Unposed SRT (CVPR 2023).   This paper is different, because it ultimate produces a set of gaussians rather than a LLFF or NeRF volume, and it distills features from 2D foundation models.   However, it is similar in that a transform encoder and decoder produces a scene representation directly from a set of images, which is then used for novel view synthesis, depth estimation, and semantic segmentation.   In any case, those paper seem fairly similar in concept and so I think they should be discussed in the related work, and possibly approach sections.

The ablation study in table 4 suggests that the key methods in the paper have little impact on the results of NVS.
questions: questions: It is interesting that the 3D methods, which have access to multiple views of the same scene do not perform as well as LSeg in Table 1.   This is counter-intuitive.  Can you please explain why?

The results on multiview depth accuracy are kinda amazing.   Why is the proposed method better than ones that take the camera parameters?   Is it due aligning the scene scale (do all the methods get the same method for scale alignment?

The novel view synthesis images look very good.   Can you please provide some info about how close the novel cameras are to the reference ones provided at inference time?   Is there a way for you to quantify and compare to PixelSplat the NVS results as the novel cameras deviate further and further from the reference ones?
limitations: limitations: Limitations are discussed briefly in the second paragraph of the conclusion.
paper_summary: paper_summary: This paper presents the Large Scene Model (LSM), which generates semantic radiance fields from uncalibrated RGB images using a unified Transformer-based framework. LSM can infer geometry, appearance, and semantics simultaneously and synthesize label maps in real-time. The model integrates multi-scale fusion and features from 2D models to enhance accuracy and efficiency.
strength_weakness: strength_weakness: 1. Dataset: I recommend the authors organize the training and testing phases in alignment with previous methods (NeRF-DFF and Feature-3DGS) and provide results on the Replica Dataset. The authors have not sufficiently justified deviating from the baseline evaluation split. Furthermore, an explanation is needed for the significant performance discrepancy of the baselines between the Replica Dataset and the authors' setup. Additional training details may also be necessary.

2. Writing: The paper's abstract, introduction, and methods sections require improvement.  Specifically, the methods section should introduce each module and their interconnections from a high-level perspective rather than presenting them as isolated components.

3. Method Details: Do the authors use camera parameters? If so, why are camera parameters mentioned in line 117? If camera parameters are used, the model cannot be described as "unposed."

4. Visualization: In Figure 4, there are category colors that are not listed in the legend. Additionally, a more diverse set of results should be displayed, as the current experimental set predominantly features sofas.
questions: questions: 1. Module Timing: I am curious about how the authors manage to use eight cross-attention modules and still achieve reconstruction in 0.1 seconds. Please provide the time consumption for each step.

2. Image Resolution: What is the resolution of the images? More details regarding the inference process should be provided, especially concerning the time comparison.
limitations: limitations: The authors have discussed limitations.
paper_summary: paper_summary: The authors proposed the Large Scene Model (LSM), a novel 3D scene understanding framework that unifies multiple vision tasks within a single model. LSM represents a scene using pixel-aligned point maps, integrating geometric, appearance, and semantic information into a unified representation. By leveraging a Transformer architecture with cross-view and cross-modal attention, the model effectively incorporates multi-view cues and semantic knowledge from a 2D vision model.

LSM's design enables efficient scene-level 3D semantic reconstruction and rendering in real time on a single GPU. The model's integration of a 2D semantic model allows for open-vocabulary understanding, extending its applicability to diverse real-world scenarios. Furthermore, by consolidating multiple tasks within a single model, LSM minimizes error propagation, leading to more robust and accurate results compared to state-of-the-art baselines.
strength_weakness: strength_weakness: * I think the main contribution of this paper is the unification of the various scene modeling tasks into the same model, including geometry, color and semantics. The authors further claimed in the abstract and introduction that multitask training end-to-end allows LSM to outperform state-of-the-art baselines. However the paper did not ablate the multi-task learning design choice. For instance, what if some of the tasks are removed (e.g., semantic feature prediction). How does that affect the performance of the other tasks?
* A suggestion is that for Figure 5, it is unclear how much pose divergence there is between the input source view and the synthesized novel view. It would be helpful to also show the source view supplied as input to the model.
* The paper is named Large Scene Model, which seems to suggest something to do with model parameter scaling, hence large. However the paper does seem to do much scaling on model size. So perhaps a more accurate terminology would be Multitask or Unified Scene Model? 

Nits.
* Line 153: Typo: to serve as input?
* In Tables 1-4, I suggest highlighting the best (and possibly second-best result) for easier comparison of the various experiments.
* In Table 4, why is + Multi-scale Fusion indented?
questions: questions: - For the quantitative comparisons given in Figure 3, are they predicted from the input view, or are they for a novel view?
limitations: limitations: - My understanding is that in this method, all the Gaussians being generated are pixel aligned with the original input images. Is that a limitation of the method, since that would make the model unable to model large pose divergences that require rendering regions not originally visible in the input view, for instance, the back side of a sofa etc.

Comments: title: Respectfully Requesting Comments from the Reviewer
Responses to Reviewer's Comment
Responses to Reviewer's Comment
Responses to Reviewer's Comment
Response to rebuttal
comment: Dear Reviewer K6pm,

Thank you once again for your review. As the deadline for the author-reviewer discussion approaches, we noticed that we haven't received any further comments from you.

We have addressed all your questions with additional experiments and clarifications:
- We have demonstrated the importance and novelty of the problem: "Semantic 3D reconstruction directly from unposed images using a **single differentiable model**."
- We provided comparisons showing that our method is not only pose-free but also enables **more 3D tasks** than other methods.
- We plan to adjust the title to better emphasize the unification of 3D tasks that our approach offers.
- We clarified the necessity of a model that maps global 3D points for input images.
- We provided further clarification on how our method compares with Free3D.
- We explained the terminology you requested.

As the discussion period is coming to an end, we would greatly appreciate any additional feedback you might have. If our responses have clarified your understanding of our paper, we sincerely hope you might consider raising the rating.

Thank you again for your effort in reviewing our paper.

Best regards,   
Authors of Paper 3523
We are grateful for your positive recommendation and the supportive feedback on our paper.
We are grateful for your positive recommendation and the supportive feedback on our paper.
Thank you very much for your positive feedback and for recognizing our efforts in addressing the major concerns. We are glad that our clarifications have been helpful, and we will certainly incorporate them into the main paper as you suggested.

We are always **open to further suggestions or feedback** that could help us improve the paper even more. If there are no additional concerns, we would greatly appreciate it if you could kindly consider **raising the rating**.

Thank you again for your valuable input.
Thanks for your great efforts! After reading the response, some major issues have been addressed well, so I still lean towards positive for the submission. I encourage the author to add these clarifications to the main paper.
Thanks for responding to my questions in the rebuttal.   The answers are pretty-much as expected.
I will leave my review score unchanged.
Thank you for your clarifications and further ablations on LSeg features and thanks for considering my suggestions for adding the source view and revise the "Large Scene Model" terminology. Overall I will maintain my rating after checking notes with the other reviewers.