Title: MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild
Venue: NeurIPS 2025
Accepted: yes
Keywords: novel view synthesis

scene reconstruction
Abstract: In-the-wild photo collections often contain limited volumes of imagery and exhibit multiple appearances, e.g., taken at different times of day or seasons, posing significant challenges to scene reconstruction and novel view synthesis. Although recent adaptations of Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) have improved in these areas, they tend to oversmooth and are prone to overfitting. In this paper, we present MS-GS, a novel framework designed with \textbf{M}ulti-appearance capabilities in \textbf{S}parse-view scenarios using 3D\textbf{GS}. To address the lack of support due to sparse initializations, our approach is built on the geometric priors elicited from monocular depth estimations. The key lies in extracting and utilizing local semantic regions with a Structure-from-Motion (SfM) points anchored algorithm for reliable alignment and geometry cues. Then, to introduce multi-view constraints, we propose a series of geometry-guided supervision steps at virtual views in pixel and feature levels to encourage 3D consistency and reduce overfitting. We also introduce a dataset and an in-the-wild experiment setting to set up more realistic benchmarks. We demonstrate that MS-GS achieves photorealistic renderings under various challenging sparse-view and multi-appearance conditions, and outperforms existing approaches significantly across different datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a collection of improvements to the standard Gaussian Splatting pipeline, particularly when the number of available input frames is small. The work focuses on (i) producing denser initialisation point clouds and (ii) generating and using virtual views as regularisers within the splat optimisation.\n\nSTRENGTHS AND WEAKNESSES:\nPositive:\n- The paper is well written and easy to understand.\n- The approach improves results consistently in all comparisons included by the authors.\n- The approach is quite quick, or at least significantly quicker than competing approaches, like Wild-GS.\n- I found the virtual view idea to be quite novel and interesting.\n\nNegative:\n- While many of the results demonstrate clear improvements, in some other tables the numbers are quite close. An example is Table 1 (ie the ablation section of the results). It would be useful if the authors included variance in these cases. \n- I could not find an ablation for the semantic scaling (i.e. as opposed for a single scalar for entire depth images). \n- Also related to the semantic scaling, I am not convinced that having scale parameters for each semantically coherent region of an image is very intuitive.\n\nQUESTIONS:\nMy main questions / requests focus on the semantic scaling. I do not see why one would necessarily expect different regions of a single view depth output image to have different scaling factors, or for these scaling factors to necessarily be semantically-coherent. It would be great if the authors added some validation on this. Could could be done relatively simply -- take some images for which they have GT depth estimates and demonstrate that a single per-frame estimate of scale is less accurate than per-semantic-region estimates.\n\nWhile I very much like the virtual view regularisation, it is not entirely obvious to me how reprojecting pixels from one real view into a virtual view, and then asking the splat to re-generate this (via the splat optimisation) is not already enforced by the fact that the splat optimisation aimed to generate those very same pixels for the real view. \n\nOverall, given the good overall results and the two improvements proposed by the paper, I lean towards acceptance.\n\nLIMITATIONS:\nThe authors do not appear to discuss the limitations and impact. The submission pdf appears to say that this is discussed in \"section 6\", yet I can only find sections up to 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MS-GS, a novel method that extends 3DGS for novel view synthesis under multi-appearance sparse-view conditions. The method offers two core contributions: (1) the Semantic Depth Alignment strategy that produces dense point cloud initialization using monocular depth aligned with SfM, guided by region-wise masks obtained from SAM; and (2) the Multi-view Geometry-guided Supervision that introduces virtual views and enforces pixel- and feature-level consistency via 3D warping. The authors also introduce a new benchmark dataset and an in-the-wild evaluation setting. Extensive experiments across multiple challenging datasets demonstrate the superior performance of MS-GS.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The proposed approach is straightforward and effective, yielding clear improvements over baselines.\n2. The Semantic Depth Alignment approach is novel and compelling. Its region-wise alignment strategy yields semantically meaningful point clouds that significantly outperform both sparse SfM and global alignment baselines.\n3. The use of virtual views for multi-view supervision is clever and enhances consistency under sparse inputs.\n4. The experiments are comprehensive, and the ablation studies are informative. The results consistently support the claimed improvements.\n\n**Weaknesses:**\n\n1. Although the authors claim to discuss limitations in Section 6, no such section exists. Only a few sentences in Section 5 briefly mention this topic.\n2. As acknowledged, MS-GS does not currently handle transient or dynamic scene elements (e.g., vehicles and pedestrians), which limits its applicability in unconstrained real-world environments.\n3. The effectiveness of geometry-guided supervision may degrade in cases of occlusion, reflections, or view-dependent effects (e.g., shadows or specularities). Further clarification on how the method handles these cases would be valuable.\n4. Section A.5 could benefit from a clearer explanation. For instance, Equation (9) defines $P_{cam}$ as a $4 \\times 3$ matrix but appears to represent a set of four 3D points. Clarifying the coordinate alignment would improve accessibility.\n5. There are minor errors (e.g., an incorrect citation for FSGS in Table 2) that merit proofreading.\n\nQUESTIONS:\n1. The method relies on monocular depth estimation for initialization. How well does the proposed Semantic Depth Alignment generalize across different depth models (e.g., Depth Anything v2)? Does the method consistently outperform image-level alignment regardless of the chosen model?\n2. The geometry-guided supervision may not generalize well to non-Lambertian scenes with reflections, shadows, or occlusions. Could the authors elaborate on how MS-GS handles such cases? Are there any specific failure cases where the 3D warping introduces artifacts or degrades performance?\n3. Could the authors provide qualitative or quantitative examples where MS-GS struggles or fails (e.g., due to noisy segmentation or limited view overlap)? Such discussion would help readers better understand the practical boundaries of the method. \n\nAdditionally, I recommend including more visualization results in the main paper or supplementary material, such as qualitative comparisons across more scenes and a visual overview of the proposed dataset, to help readers gain a more intuitive understanding of the method and its effectiveness.\n\nLIMITATIONS:\nYes. The paper does acknowledge some limitations in Section 5, particularly regarding transient scene modeling. However, the discussion is brief and could be expanded. Potential limitations (e.g., sensitivity to segmentation quality or reflective materials) are not discussed and would be worth including in future revisions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MS-GS, an approach for multi-appearance sparse-view 3DGS. MS-GS mainly consists of two components: semantic depth alignment and multi-view geometry-guided supervision. The former is proposed to address the problem of sparse SfM points in initialization, where the mono-depth estimates from an off-the-shelf model are aligned to the depths from SfM points. To perform more accurate alignment, MS-GS leverages SAM to identify semantically consistent regions and performs local alignment. Multi-view geometry-guided supervision is designed to provide additional training signals for the in-the-wild 3DGS setup. Specifically, training views are warped to novel views to create pseudo ground truth, which offers supervision for both rendered pixels and image features. Given 20 input views, the proposed method outperforms previous in-the-wild 3DGS methods, and ablation studies verify the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Leveraging input views and the learned geometry to create virtual views to mitigate overfitting in the in-the-wild setup is a clever idea. \n2. The proposed method outperforms baseline methods on multiple benchmarks. The experiments in this paper are comprehensive.\n3. The ablation studies clearly verify the effectiveness of the proposed method.\n\nWeaknesses:\n1. Lack of novelty: Densifying initial SfM points for 3DGS and using semantic segmentation to identify semantically consistent regions for alignment have been explored in [1]. Although the technical details differ (for example, [1] uses MASt3R 3D points), the motivation and high-level idea of leveraging semantic segmentation are similar. \n2. The authors use 20 input images for training, but 20 is not truly a “sparse” setting. Could the authors provide results (comparisons and ablations) with fewer input views?\n\n[1] Tang et al., SPARS3R: Semantic Prior Alignment and Regularization for Sparse 3D Reconstruction, CVPR 2025.\n\nQUESTIONS:\n1. The reviewer does not fully understand the proposed “in-the-wild” setup and the coordinate alignment approach. In this setup, is pose estimation performed separately for the training and test views? It would be helpful if the authors could further elaborate on the motivation and technical details of this design.\n\nLIMITATIONS:\nThe authors briefly mention that the proposed method cannot handle transient objects but do not address the potential societal impact of this limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Paper Summary\nThis paper addresses an important problem: novel view synthesis (NVS) in the wild from a sparse set of input images. It introduces two key technical contributions along with a new dataset. \n\nThe two technical contributions are:\n1. Semantic depth alignment: This step leverages Segment Anything (SAM) and structure-from-motion (SfM) points to correct distortions in monocular depth predictions. These aligned points are then used to initialize the proposed NVS-in-the-wild model.\n2. Multi-view geometry-guided supervision: This component enforces both photometric and feature consistency. The losses are computed using rendered virtual views and point-projected virtual views.\n\nThe dataset contribution is a large-scale, unbounded drone dataset captured under diverse weather and lighting conditions.\n\nEvaluation: The method is evaluated on three datasets: two standard benchmarks (Mip360 and Phototourism) and the newly proposed drone dataset.\n\n## Review Summary\nThe technical contributions are solid and directly address the challenge of sparse-view NVS in the wild. The evaluation is thorough and demonstrates the effectiveness of the proposed approach. The primary weakness lies in the limited details provided about the newly introduced dataset.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper is very well written and easy to follow.\n- It addresses a valuable and challenging problem: NVS in the wild from sparse input views.\n- The proposed technical contributions are well motivated and technically sound.\n\n## Weaknesses\n1. The main concern is the lack of clarity around the newly introduced dataset. Lines 78–86 describe the motivation for a new dataset to enable NVS-in-the-wild evaluation without requiring access to half of the test images, as was common in previous benchmarks. However, the dataset is only briefly described in Lines 232–237. It remains unclear how the proposed dataset addresses the limitations of prior setups, specifically the need to reserve half of the test images.\n2. The paper lacks a runtime and memory analysis. Given that the proposed method is based on 3D Gaussian Splatting (3DGS) and uses dense initialization points, it is important to quantify the additional memory usage and training time compared to a more traditional sparse-point initialization.\n\nQUESTIONS:\n1. Details of the proposed dataset? And how does it address the limitation of requiring half of a test image in the previous evaluation pipeline?\n2. Runtime and memory analysis?\n\nLIMITATIONS:\nLimitations are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}