MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views

Authors: yuedong  chen
chuanxia  zheng
haofei  xu
bohan  zhuang
andrea  vedaldi
tat-jen  cham
jianfei  cai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 04:27:20

Abstract: We introduce MVSplat360, a feed-forward approach for 360° novel view synthesis (NVS) of diverse real-world scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided, making it challenging for conventional methods to achieve high-quality results. Our MVSplat360 addresses this by effectively combining geometry-aware 3D reconstruction with temporally consistent video generation. Specifically, it refactors a feed-forward 3D Gaussian Splatting (3DGS) model to render features directly into the latent space of a pre-trained Stable Video Diffusion (SVD) model, where these features then act as pose and visual cues to guide the denoising process and produce photorealistic 3D-consistent views. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views. To evaluate MVSplat360's performance, we introduce a new benchmark using the challenging DL3DV-10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-art methods on wide-sweeping or even 360° NVS tasks. Experiments on the existing benchmark RealEstate10K also confirm the effectiveness of our model. Readers are highly recommended to view the video results at [donydchen.github.io/mvsplat360](https://donydchen.github.io/mvsplat360).

Main paper: 
 introduction: 
 The rapid advancement in 3D reconstruction and NVS has been facilitated by the emergence of differentiable rendering [29, 31, 41, 40, 21] . These methods, while fundamental and impressive, are primarily tailored for per-scene optimization, requiring hundreds or even thousands of images to comprehensively capture every aspect of the scene. Consequently, the optimization process for each scene can be time-consuming, and collecting thousands of images is impractical for casual users.In contrast, we consider the problem of novel view synthesis in diverse real-world scenes using a limited number of source views through a feed-forward network. In particular, this work investigates the feasibility of rendering wide-sweeping or even 360°novel views using extremely sparse observations, like fewer than 5 images. This task is inherently challenging due to the complexity of scenes, where the limited views do not contain sufficient information to recover the whole 3D scene. Consequently, there is a necessity to ensemble visible information under minimal overlap accurately and generate missing details reasonably. This represents a new problem setting in sparse-view feed-forward NVS. Existing feed-forward methods typically focus on two distinct scenarios: 360°NVS with extremely sparse observations, but only at object-level [20, 52, 66, 59, 27, 72, 63, 56, 62, 50, 46, 47, 18] , or generating reasonable results for scene-level synthesis, but only for nearby viewpoints [53, 7, 19, 43, 9, 13, 6, 60, 10, 70, 42, 61] . In contrast, we argue that the time is ripe to unify these previously distinct research directions. Our goal should be to develop systems capable of synthesizing wide-sweeping or even 360°novel views of large, real-world scenes with complex geometry and significant occlusion. Specifically, this work explores synthesising 360°novel views from fewer than 5 input images. We show that in this challenging setting, existing feed-forward scene synthesis approaches [9, 13, 6, 60, 10, 57] struggle to succeed. This failure arises from two main factors: i) the limited overlap among input views causes many contents to appear in only a few views or even a single one, posing significant challenges for 3D reconstruction; ii) the extremely sparse observations lack sufficient information to capture the comprehensive details of the whole scene, resulting in regions unobserved from novel viewpoints.In this paper, we propose a simple yet effective framework to address these limitations and introduce the first benchmark for feed-forward 360°scene synthesis from sparse input views. Our key idea is to leverage prior knowledge from a large-scale pre-trained latent diffusion model (LDM) [35] to "imagine" plausible unobserved and disoccluded regions in novel views, which are inherently highly ambiguous. Unlike existing 360°object-level NVS approaches [27, 63, 49, 56, 25, 72, 50] , large-scale real-world scenes comprise multiple 3D assets with complex arrangements, heavy occlusions, and varying rendering trajectories, which makes it particularly challenging to condition solely on camera poses, as also verified by concurrent work ViewCrafter [68] .To develop a performant framework for scene-level synthesis, we opt to treat the LDM as a refinement module, while relying on a 3D reconstruction model to process the complex geometric information. Broadly, we build upon the feed-forward 3DGS [21] model, MVSplat [10] , to obtain coarse novel views by matching and fusing multi-view information with the cross-view transformer and cost volume. Although these results are imperfect, exhibiting visual artifacts and missing regions (see Fig. 1 ), they represent the reasonable geometric structure of the scene, as they are rendered from 3D representation. Furthermore, we choose Stable Video Diffusion (SVD) [3] over other image-based LDM as the refinement module, since its strong temporal consistency capabilities align better with the view-consistent requirement of the NVS task, as also observed by concurrent work 3DGS-Enhancer [28] . Conditioning SVD with the 3DGS rendered outputs, our MVSplat360 produces visually appealing novel views that are multi-view consistent and geometrically accurate (see Fig. 1 ).Importantly, the original MVSplat outputs only RGB images, which is not the optimal condition for the generator, and is difficult to optimize jointly with the SVD denoising module. To tackle this, we propose a simple Gaussian feature rendering with multi-channels, supervised with an introduced latent space alignment loss. Despite a seemingly minor change, the additional feature condition for SVD leads to a significant impact: It bypasses the SVD's frozen image encoder, allowing the gradients from SVD to backpropagate to enhance the geometry backbone and lead to improved visual quality, especially on the new challenging DL3DV-10K dataset. While related work Reconfusion [58] , CAT3D [15] and latentSplat [57] also combine the 3D representation with 2D generators, the former two focus more on per-scene optimisation, while the latter only shows 360°NVS at the object level.We conduct a series of experiments, mainly on two datasets. First, we establish a new benchmark on DL3DV-10K dataset [23] , creating a new training and testing split for feed-forward wide-sweeping and 360°NVS. In this challenging setting, our MVSplat360 achieves photorealistic 360°NVS from sparse observations and demonstrates significantly better visual quality, where the previous scene-level feed-forward methods [9, 6, 60, 10] fail to achieve plausible results. Second, we deploy MVSplat360 on the existing RealEstate10K [74] benchmark. Following latentSplat [57] , we estimate both interpolation and extrapolation NVS, and report state-of-the-art performance.Our main contributions can be summarized as follows. 1) We introduce a crucial and pressing problem for novel view synthesis, i.e., how to do wide-sweeping or even 360°NVS from sparse and widely-displaced observations of diverse in-the-wild scenes (not objects) in a feed-forward manner (no any per-scene optimization). 2) We propose an effective solution that nicely integrates the latest feed-forward 3DGS and the pre-trained Stable Video Diffusion (SVD) model with meticulous integration designs, where the former is for reconstructing coarse geometry and the latter is for refining the noisy and incomplete coarse reconstruction. 3) Extensive results on the challenging DL3DV-10K and RealEstate10K datasets demonstrate the superior performance of our MVSplat360. 
related work: 
 Sparse view per-scene reconstruction and synthesis. Differentiable rendering methods, such as NeRF [31] and 3DGS [21] , are mainly designed for very dense views (e.g., 100) as inputs for per-scene optimization, which is impractical to collect for casual users in real applications. To bypass the requirement for dense views, various regularization terms have been proposed in per-scene optimization [33, 11, 69, 48] . Recently, ZeroNVS [38] , Reconfusion [58] and concurrent submissions, including CAT3D [15] , ReconX [24] , ViewCrafter [68] , LM-Gaussian [67] , 3DGS-Enhancer [28] , have leveraged large-scale diffusion models for generating pseudo dense views of a 3D scene, which are then input into a per-scene reconstruction pipeline. However, these methods are inherently slow for reconstructing unseen scenes due to the necessity of per-scene optimisation.Feed-forward scene reconstruction and synthesis. To mitigate these limitations, early approaches like Light Field Networks [40] use ray querying to predict novel views. Subsequent methods [44, 43] employ epipolar attention for multi-view geometry estimation. Later, pixelNeRF [66] devised pixel-aligned features for NeRF reconstruction [31] , leading to a range of subsequent methods that incorporate feature matching fusion [7, 9] , Transformers [36, 13, 32] and 3D volume representation [7, 60] . Recently, 3D Gaussian Splatting [21] has been implemented into feed-forward networks, such as pixelSplat [6] , MVSplat [10] , Splatter Image [46] , Flash3D [45] and latentSplat [57] . While these methods were successful in novel view synthesis from sparse views, they fail to achieve this in a wide-sweeping or 360°setting. Concurrent yet unpublished submissions, DepthSplat [61] and Long-LRM [75] , also show promising results in the 360°setting, but their frameworks have limited generation capabilities, necessitating the use of denser inputs, e.g., 12 or 32 views.Camera trajectory controllable synthesis. Generative models have achieved remarkable results for image/video synthesis [14, 73, 8, 35, 5, 17, 39, 3] , but they lack precise control over the viewpoint of generated images. To address this, several approaches fine-tune large-scale pre-trained diffusion models with explicit image and pose conditions [27, 26, 25, 4, 72, 64, 56, 50] . However, these methods mainly show 360°NVS results on single objects, leaving the complex scene synthesis problem unsolved. Natural scenes comprise multiple objects with intricate occlusion relationships, presenting greater challenges that are not easily addressed by these single-object NVS models. Besides, camera trajectories can be highly irregular and varied when roving around such complex scenes. Although related works [55, 22, 38, 15] have explored training or fine-tuning diffusion models with camera control for scene synthesis, they often struggle with precise camera pose control [55, 22] , and still rely on per-scene optimization for 3D reconstruction [38, 15] . (c) Considering such coarse reconstruction is imperfect, we further adapt a pre-trained SVD, using features rendered from the 3DGS representation as conditions to achieve 360°novel view synthesis.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in the supplementary material.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 Given N sparse views I = {I i } N i=1 and the corresponding camera poses P = {P i } N i=1 , with P i = (K i , R i , T i ) comprising intrinsic K i , rotation R i and translation T i , our goal is to learn a model Φ that synthesizes wide-sweeping or even 360°novel view synthesis (NVS).We opt to go beyond per-scene optimisation [1, 2, 58, 15] , and to deal with a more general feedforward network capable of achieving 360°NVS for unseen scenes, yet without the need of additional per-scene training. This requires effectively matching information between sparse views in 3D space, as well as generating sufficient content based on only partial observations. To achieve that, our MVSplat360 framework, illustrated in Fig. 2 , comprises two main components: a multi-view geometry reconstruction module (Section 3.1) and a multi-frame consistent appearance refinement network (Section 3.2). The former is responsible for matching and fusing multi-view information from sparse observations to create a coarse geometry reconstruction, whereas the latter is designed to refine the appearance with a pre-trained latent video diffusion model. While similar two-step approaches have been explored in recent related works, e.g., [4, 58, 38, 15] , we are the first (to the best of our knowledge) to explore it on wide-sweeping or even 360°NVS for large-scale scenes from sparse views (as few as 5), in a feed-forward manner.The first module is built upon a feed-forward 3DGS reconstruction model, i.e., MVSplat [10] as in our implementation. Specifically, given sparse-view observations I = {I i } N i=1 and their corresponding camera poses P = {P i } N i=1 , the model learns to predict 3D Gaussian parameters{(µ i , α i , Σ i , c i )} H×W ×N i=1, which can then be splatted to obtain a set of RGB images Ĩtgt using the target camera poses P tgt . To ensure better integration with the following diffusion module, we predict an additional Gaussian feature fi , in parallel with other parameters, which can be rasterized to the corresponding latent features Ftgt . Furthermore, we also improve the view selection strategy to improve the model's robustness in handling widely displaced inputs.Coarse geometry reconstruction. Our backbone comprises multi-view feature extraction, cost volume construction, depth estimation, and 3D Gaussian parameter predictions. First, a cross-view transformer encoder is applied to fuse multi-view information and obtain cross-view aware features F = {F i } N i=1 . Then, N cost volumes C = {C i } N i=1 are constructed by matching feature correlations between cross-views. Specifically, it uniformly divides the depth into L layers in the near and far depth ranges, i.e. D = {D m } L m=1 , and then warps the features from one view j to another view i via F j→i Dm = W(F j , P i , P j , D m ). The cost volumeC i = [C i D1 , C i D2 , . . . , C i D L ]is then collected by L correlations, where each correlation is expressed asC i Dm = F j→i Dm •F i √ C, with C denoting channel dimension. Finally, the per-view estimated depth d is obtained by applying the softmax operation on the cost volumes in the depth dimension. After that, the Gaussian mean is computed by µ = K -1 ud + ∆, where K is the camera intrinsic, u = (u x , u y , 1) denotes each pixel, and ∆ ∈ R 3 is the predicted offset, along with opacity a ∈ [0, 1], covariance Σ ∈ R 3×3 , and colorc ∈ R 3(S+1) 2 ,where S is the order of the spherical harmonics [10] . Once the model predicts a set of 3D Gaussian parameters{(µ i , α i , Σ i , c i )} H×W ×N i=1, the target view Ĩtgt can be rendered through rasterization.Gaussian feature rendering. Given sparse-view observations, MVSplat [10] tends to render images with noticeable artifacts in wide-sweeping novel viewpoints (see Fig. 1 ), resulting in suboptimal conditioning for the subsequent SVD. Since the rendered images must first be encoded into the latent space using a frozen encoder (see Section 3.2), enhancing the backbone with gradients from SVD would be computationally expensive. To address this issue, we propose directly rasterising features F into the latent space of SVD, by predicting an additional parameter fi for each 3D Gaussian. This operation offers two advantages: (i) The latent feature includes multi-channel information, providing a more comprehensive representation of the scene; (ii) The entire framework is end-to-end connected by conditioning SVD on the rendered latent features instead of the image-encoded ones. It enables the SVD loss to optimize the Gaussian features, further enhancing the reconstruction backbone.Observed and novel viewpoints selection. To enable 360°scene synthesis, it is crucial to choose the correct camera viewpoints, so that they can cover most contents in diverse and complex scenes [15] . It is impractical to assume a circular orbital camera trajectory like those object-level 360°view synthesis [27, 46, 72, 47] , whereas it is suboptimal to randomly choose a video sequence like existing scene-level nearby viewpoint synthesis [46, 6, 10] . To this end, we propose to choose views evenly distributed within a set of targeted viewpoints as input. Specifically, for a given set of candidate views, we apply farthest point sampling over the camera locations to identify the input views and randomly choose from the rest as target views. The number of candidate views gradually increases throughout the training, stably improving the model's capability toward handling 360°scene synthesis.View interaction within the local group. Recalling that the 3D reconstruction backbone MVSplat is primarily designed for nearby viewpoints, with key components like multi-view transformers and cost volume assuming sufficient overlap among input views. However, in the more challenging 360°settings, the widely displaced input views lead to minimal overlap between specific view pairs, hindering the effectiveness of the backbone. To mitigate this limitation, we refactor our backbone to use cross-view attention and construct the cost volume only within a local group of input views based on camera locations, reducing memory consumption and ensuring stable model convergence.Video diffusion model. MVSplat360 utilizes an off-the-shelf multi-frame diffusion model, i.e.Stable Video Diffusion (SVD) [3] , to refine the visual appearance of the aforementioned coarse reconstruction. SVD is pre-trained on large-scale video datasets and has strong prior knowledge of temporal consistency. It adheres to the original formulation used by Stable Diffusion (SD) [35] that conducts the denoising processing in the latent space. In particular, given a target sequence of x 1:M with M images, they are initially embedded into the latent space by a frozen encoder E, yielding z 1:M 0 = E(x 1:M ), and then perturbed by adding Gaussian noise ϵ ∼ N (0, I) in a Markov process:EQUATIONwhile ᾱ0 , . . . , ᾱT is a pre-defined noise schedule within T steps. Given noise input z 1:M t , the denoiser ϵ θ is then trained by optimizing the following objective function:EQUATIONwhere x 1:M is the target images, and y is the conditional inputs. After ϵ θ is trained, the model can generate a video by performing iterative denoising from pure Gaussians z 1:M T conditioned on y.Note that SVD is trained with the latest v-prediction formulation [37] , instead of the original ϵprediction in SD. Hence, the final loss is calculated in latent space using the mean squared error (MSE) between the ground truth and its prediction ||z1:M 0 -ẑ1:M t || 2 2, where ẑ1:M t is obtained by translating the velocity v = Φ(z 1:M t , t, y) to latent space, i.e., ẑ1:M t = α t z 1:M t -σ t v.Multi-view hybrid conditions. To ensure an accurate understanding of the scene, the model requires the integration of both low-level perception (e.g., depth and texture) and high-level understanding (e.g., semantics and geometry). Following [27, 72, 50] , we adopt a hybrid conditioning mechanism to fine-tune the SVD model for wide-sweeping NVS with sparse observations. In one stream, a CLIP [34] image embedding token of the original visible views I is used as a global type and text prompt. At each UNet block, a cross-attention operation is applied to capture high-level semantics of the input images to the model. Since we have sparse views, we average these tokens to become one global token. In the other stream, the spatial conditions from the coarse geometry rendered features Ftgt = { Fi } M i=1 is channel-concatenated with the noised latent z1:M t . These spatially conditional features assist the model to capture the view information, and learn low-level perception to maintain the texture of the scenes. Compared to the concurrent work CAT3D [15] , this coarse feature conditioning not only provides accurate pose information from the 3DGS rendering, but also offers reasonable visual information.Color adjustment. While our MVSplat360 can achieve photorealistic NVS, the synthesized videos sometimes exhibit oversaturated colors (detailed in Appendix C). This may be visually acceptable for video generation, but it can decrease performance when evaluated on NVS task. To mitigate this, we apply post-processing by matching the color histogram between the SVD refined views Îtgt and our 3DGS rendered views Ĩtgt before performing pixel-aligned measurements, i.e., PSNR and SSIM.Our MVSplat360 predicts two sets of images, including the coarse one Ĩtgt from the 3DGS module and the refined one Îtgt from the SVD module, where the former is mainly rendered to help supervise the geometry backbone. The entire model is end-to-end trainable, using three groups of loss functions, namely reconstruction loss, diffusion loss and latent space alignment loss.In particular, the reconstruction loss is a linear combination of ℓ 2 and LPIPS [71] , applied between the coarse outputs Ĩtgt and the corresponding ground truth I tgt . The other two loss functions are applied to the following SVD module, whose gradients will backpropagate to the 3DGS module but will not update those structural parameters, i.e., µ, α, Σ. This is achieved by stopping the gradients from the structural parameters when rendering the latent features Ftgt , since keeping those gradient flows will lead to unstable training, as also observed by latentSplat [57] . We use the standard v-prediction formulation (detailed in Section 3.2) as the diffusion loss to fine-tune the denoising network of the SVD, keeping the first-stage encoder and decoder frozen. Since the SVD's released model is conditioned on image-encoded features while our diffusion module is on 3DGS-rendered ones, we find it beneficial to align these two spaces by regularizing with a latent space alignment loss, min g θ E F ∼g(I) ∥E(I tgt ) -Ftgt ∥ 2 2 , where g refers to the geometry backbone with trainable parameters θ and E is the frozen SVD encoder.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental details: 
 Datasets. To verify the effectiveness of MVSplat360 in synthesizing wide-sweeping and 360°novel views, we have established a challenging benchmark derived from DL3DV-10K [23] . It comprises 51.3 million frames from 10,510 real-world scenes, adhering to 65 point-of-interest (POI) [65] categories. For training, we use a subset in subfolders "3K" and "4K", resulting in ~2,000 scenes. We tested on the 140 benchmark scenes and filtered them out from the training set to ensure correctness. For each scene, we selected 5 input views using farthest point sampling based on camera locations and evaluated 56 views by equally sampling from the remaining, yielding a total of 7,840 test views. Additionally, since most DL3DV-10K scenes contain a two-round trajectory, we also report another setting by focusing only on half of the sequence, intending to cover the camera trajectory of one round. We denote the two settings as n = 300 and n = 150, where n refers to the frame distance span across all test views, as most scenes contain roughly 300 frames. We also assess our model on RealEstate10K [74] , which contains real estate videos downloaded from YouTube. Consistent with existing works [6, 10] , we train MVSplat360 on 67,477 scenes and test it on 7,289 scenes. Metrics. To measure models from different perspectives, we follow [57] to report both the pixel-align metrics, i.e., PSNR and SSIM [54] , and the perceptual metrics, i.e., LPIPS [71] and DISTS [12] . Since MVSplat360 aims to generate plausible contents for unobserved and disoccluded regions, we also reported the distribution metric, i.e., Fréchet Inception Distance (FID) [16] 1 , which measures the similarity between distributions of the generated images and the real ones.Table 1 : Comparison with SoTA methods on DL3DV-10K. Below, n is the frame distance span across all the tested novel views within each scene, which is set to 300 by default as most DL3DV-10K scenes contain roughly 300 extracted frames. Since most DL3DV-10K scenes contain a two-round trajectory, we also report another setting of n = 150 aiming for coverage of one round. Implementation details. MVSplat360 is implemented with PyTorch and a CUDA-implemented 3DGS renderer. For coarse geometry reconstruction (Section 3.1), we set hyperparameters following MVSplat [10] , except that we apply cross-view attention and build each cost volume within the nearest 2 views rather than all other views. For multi-frame appearance refinement (Section 3.2), we fine-tune from the 14-frame SVD [3] pre-trained model, but using rendered Gaussian features as conditions. We also remove the original "motion value" and "fps" conditions, since they are unrelated to our NVS task. We rescale the rendered feature to have a similar shape as the original image-encoded latent feature in the pre-trained model, which is critical for getting better details as it affects the decoder (more discussions are in Appendix A). We train SVD using 14 frames sampled along natural camera trajectories, captured by the initial videos. At inference, we directly feed 56 views to SVD but change all related temporal attention blocks to local attention with a window size of 14 to better align with the training. More implementation details can be found in Appendix B, and the codes are publicly available at https://github.com/donydchen/mvsplat360.We first assess the ability of MVSplat360 and baselines to synthesize wide-sweeping and 360°NVS in the newly constructed challenging benchmark with diverse scene categories.Baselines. We perform a thorough comparison of MVSplat360 to the latest state-of-the-art (SoTA) 3DGS-based models, including pixelSplat [6] , MVSplat [10] and latentSplat [57] . All models are trained on the same training split and evaluated on the publicly available 140 scenes.Quantitative results. All models are trained to 100K steps and reported at Table 1 , except for the latentSplat, which suffers from unstable training due to its GAN-based architecture. Hence, we report its best performance at around 60K training steps before the subsequent collapse. Our MVSplat360 outperforms all existing SoTA models in all metrics on the two settings n = 300 and n = 150. All models generally perform better on the n = 150 setting than on the n = 300 one since the latter spans larger viewpoints. It can be seen that the two generative models (latentSpalt and MVSplat360) generally perform better than the other two regression models, suggesting the importance of additional refinement in addressing feed-forward scene reconstruction.Although our improvement on pixel-aligned metrics appears minor, this is expected since refinement via either interpolation (for disoccluded regions) or extrapolation (for unobserved regions) does not guarantee matching the ground truth at the pixel level. It mainly aims to provide a reasonable solution to refine the images and ensure they align with real-world image distribution. This is verified by the fact that our improvements on perceptual metrics are larger, and it is even more apparent on FID, which measures the distribution deviation. The superiority of our MVSplat360 stands out more from the qualitative results presented below.Qualitative results. The qualitative comparisons are visualized in Fig. 3 . MVSplat360 achieves remarkable visual results even under challenging conditions. pixelSplat [6] and MVSplat [10] exhibit obvious artifacts due to the issue of floating Gaussians. latentSplat [57] improves the results with an additional decoder and adversarial training. However, its resulting object geometry and image quality are still far from satisfactory, suggesting that the GAN-based framework cannot provide enough prior knowledge for refining 360°NVS in diverse real-world scenes. Readers are referred to our project page for video results with more comprehensive comparisons, where our MVSplat360 shows Table 2 : Comparison with SoTA methods on RealEstate10K. We report interpolation scores using the settings of [6, 10] , where we retrain latentSplat [57] to maintain fair comparison (indicates with * ). We report extrapolation scores by following [57] . multi-view consistent, high-quality visual results along complex camera trajectories, while others suffer from apparent artifacts.We also assess MVSplat360 on the existing benchmark, following the existing "Interpolation" setting [6, 10] and "Extrapolation" setting [57] . We retrained latentSplat on Interpolation and MVSplat on Extrapolation for fair comparisons. Quantitative results. Table 2 shows quantitative comparisons on RealEstate10K [74] of MVSplat360 and other approaches. Our MVSplat360 surpasses all previous state-of-the-art methods, mainly in terms of the perceptual metrics and the distribution metric. The former implies that our rendered views are more aligned with human perception, while the latter shows that our refined images correspond better to the dataset distribution. These observations can be further confirmed by visual assessment.Qualitative results. The qualitative comparisons of the top four best models are in Fig. 4 . Pixel-Splat [6] and MVSplat [10] fail to render any content for the unobserved regions due to the lack of generative capability. In contrast, latentSplat can perform extrapolation via its GAN-based decoder, improving the overall visual quality. However, we observed that the content generated by latentSplat is not visually reasonable. Our MVSplat360 generates more plausible content (see the "window" in 1st row and "chair" in 2nd row), thanks to the stronger generative capability of the diffusion model.We conduct ablations on DL3DV-10K [23] to analyze MVSplat360 further. Results are reported in Table 3 , and discussed in detail next.Accessing model components. The baseline refers to MVSplat [10] since our model is built on top of it. (i) A natural extension is to render novel views from MVSplat and use them directly as conditions in the SVD denoising process. However, this straightforward approach performs slightly worse than the original MVSplat, likely because SVD struggles to infer pose and visual cues from the noisy image-encoded features. (ii) To better utilize the input context views, we average CLIP-embedded tokens from all views instead of just the first. This provides SVD with richer scene information via the cross-attention blocks, leading to a noticeable improvement. (iii) Lastly, we render high-dimensional features via the 3DGS rasterizer and concatenate them directly into the diffusion latent space, enabling gradients from the denoising UNet to backpropagate through the geometry backbone. This end-to-end training improves performance significantly and is used in our default model.Accessing the number of input views. As shown in Table 4b , while our model is only trained with 5 input views, the performance can be gradually improved by adding more input views at testing. This is reasonable as more input views can provide more observable areas. On the contrary, reducing input views will inevitably result in worse performance. Surprisingly, even with 3 sparse views, our MVSplat360 still outperforms regression models (pixelSplat and MVSplat) that use 5 views.Assessing the geometry accuracy. Our MVSplat360 builds on the video diffusion model SVD, which does ensure strong temporal/multi-frame consistency but does not inherently guarantee geometric accuracy. To confirm that our MVSplat360 produces geometrically accurate outputs, we run structurefrom-motion (SfM) on both the input source views and the rendered novel views using VGGSfM [51] .As shown in Fig. 5 , VGGSfM recovers reasonable camera poses and 3D point clouds, confirming that our novel views are both multi-view consistent and geometrically correct. This highlights how the 3DGS backbone's latent features provide essential geometric cues, enhancing 3D consistency in the final SVD-based outputs.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide all the information needed to reproduce the main experimental results of the paper. 
conclusion: 
 We present MVSplat360, a feed-forward model that synthesises 360°novel views of diverse realworld scenes from sparse input views. Our MVSplat360 leverages a feed-forward 3DGS model for recovering the coarse geometry and appearance from sparse observations of a 3D scene, which are then used to render latent features as the pose and visual cues to guide the following SVD in generating 3D consistent 360°novel views. To demonstrate its effectiveness, we construct a challenging benchmark for 360°NVS of real-world scenes. Experimental results show that MVSplat360 achieves superior visual quality compared to other SoTA feed-forward approaches.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We will release our code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification:We elaborate the implementation details of our paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: This paper does not include experiments reporting error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the positive and negative social impact in the supplementary document.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Assessing the robustness of SVD's first-stage autoencoder to input resolution. Since our MVS-plat360 fine-tunes the denoising process in latent space, it is essential to ensure the autoencoder accurately maps between pixel and latent spaces. We observe that when the input resolution (256 × 480) differs significantly from the autoencoder's pre-trained resolution (768 × 1280), the autoencoding process causes noticeable information loss, leading to missing details (see Fig. Following MVSplat [10] , we set the near and far depth range used in the cost volume construction as 1 and 100, respectively. We render features from the 3D Guassians rasterizer with shape 4 × h × w, where h and w refer to the image height and width, respectively, and the channel dimension is set to 4 to align with that of the SVD initial conditional vector. We bilinearly interpolate the latent features to a resolution of 1/4h × 1/4w, matching the encoded features from images of size 2h × 2w, as the encoder downsamples inputs by a factor of 8. This design ensures proper projection into the initial latent space, as detailed in Appendix A. The SVD decoder outputs are then bilinearly interpolated from 2h × 2w back to h × w. Due to resource limitations, we mainly experiment on the "images_8" branch of the DL3DV-10K dataset, which contains images with resolution 256 × 480. Our default model is trained with the Adam optimizer, and the learning rate is set to 1.e -5 and decayed with the one-cycle strategy. All models are trained for 100,000 steps with an effective batch size of 8 on 1 to 8 A100 GPUs, and we apply the gradient accumulation technique whenever needed. During training, we sampled 5 views as input views and another 14 views as targeted rendering views, with the intention of better aligning with the following SVD module, which is trained on videos with 14 frames.Although MVSplat360 achieves plausibly consistent 360°NVS and significantly outperforms previous works in visual quality by leveraging SVD, it also inherits several limitations from SVD. For instance, the results may exhibit oversaturated colors (see Fig. B , left), limiting improvements on pixel-aligned metrics like PSNR and SSIM. This is likely because SVD is primarily trained on artistic videos with vibrant colors, and fine-tuning from its pre-trained weight can bias the outputs toward the original training data distribution [50, 30] . Additionally, the outputs might exhibit hallucinations and contain contents not existing in the input views (see Fig. B , right). Lastly, the inference is slow due to the multiple sampling steps in the diffusion process. We expect these limitations to be mitigated as better video diffusion models and pre-trained weights become available in the future.Our MVSplat360 renders 360°novel views from sparse observations, making it a valuable tool for augmented reality applications. It can enhance immersive experiences in entertainment and media, including 3D videos and video games, and support historical reconstruction for educational purposes. However, MVSplat360 's powerful generative capabilities could be misused to create fake videos. Additionally, while the rendered views are high quality, they may not fully capture real-world details. Therefore, precautions are necessary when using the generated data in safety-critical applications, such as training autonomous driving models.E More Visual Comparisons.Below, we provide more visual comparisons with existing state-of-the-art models.Input Views pixelSplat MVSplat latentSplat MVSplat360 Ground Truth Updated • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/mseitzer/pytorch-fid 


Decision: The paper received borderline reviews initially, stating a large variety of concerns about the clarity, paper framing and technical novelty of the approach. At the same time, all reviewers acknowledged the high quality results of the presented method in a very challenging setup.

The authors provided exhaustive answers, leading to the scores converging to 4x borderline accept. While some concerns regarding clarity and paper framing have been resolved, some others, especially technical novelty, remain unresolved.

After reading the paper, reviews and rebuttal, I can draw the following conclusions:
- The paper indeed shows impressive results in 360° novel view synthesis, qualitatively and quantitatively.
- The tackled setting is a very challenging one and still heavily underexplored. Methods that solve it are very welcome in the community.
- The technical novelty is limited, as the method essentially combines two existing methods (MVSplat + SVD) into a single pipeline in a similar way as Reconfusion did with pixelNeRF + an image diffusion model.
- The paper could do a better job in highlighting specific aspects that surfaced during the discussion phase, such as the effect of using a pre-trained video diffusion model instead of image diffusion models. Such analysis could lead to more conceptual insight transported to the reader, strengthening the paper.

This paper is clearly in a borderline state. Since all reviewers are agreeing on an accept score and I am also impressed by the qualitative results of the work, I follow the reviewers and recommend acceptance.

I heavily encourage the authors to adjust the paper by incorporating the different aspects that surfaced in the discussion phase.

Reviews: review: paper_summary: paper_summary: The paper proposes MVSplat360, a method for wide-sweepign or 360-degree novel view synthesis on general scenes from sparse input views. It extends an existing state-of-the-art approach MVSplat to render 3D feature Gaussians as conditioning for a refinement network in form of a pre-trained video diffusion model, which is fine-tuned jointly. The approach is evaluated on RealEstate10k, an existing benchmark of home walkthrough videos, and on a newly proposed benchmark for 360-degree novel view synthesis, leveraging an existing dataset of diverse scenes. Quantitative and qualitative results show improvements over regression-based and generative, splatting-based baselines and plausable completions in case of incomplete observations.
strength_weakness: strength_weakness: - Confusing title: Given the contributions, the title should rather focus on the proposed method.
  - The title of the paper suggests a focus on benchmarking existing methods.
  - The paper still proposes a new approach MVSplat360.
  - Evaluation is done on one existing benchmark (RealEstate10k) and one newly proposed one leveraging the already existing DL3DV dataset.
  - The problem of generalizable novel view synthesis given sparse views is not novel (see baselines [1], [2], [3]) such that the benchmark creation only consists of the definition of input and target views.

- Insufficient contextualization relative to prior work:
  - Inaccurate use of the term 'concurrent work':
    - In line 60, ReconFusion [4] and latentSplat [3] (again in line 244) are referred to as concurrent work.
    - This is not really the case anymore, because
      - ReconFusion [4] appeared on arxiv in the beginning of December 2023 and was presented at CVPR 2024.
      - latentSplat [3] appeared on arxiv end of March 2024, 3 days after MVSplat [2], which is one of the building blocks of the proposed method.
  - Missing references to related work regarding conditioning via feature rendering and CLIP embeddings:
    - In lines 163-168 and 196-201, the authors propose to render latent features as spatial conditions for a generative model (Stable Video Diffusion), which is later ablated in lines 296-299.
       - This idea is not novel and the paper is missing important references in this context:
         - GeNVS [5] proposed to use rendered pixelNeRF features as conditoning for a 2D diffusion model.
         - ReconFusion [4] adopts this to condition a text-to-image latent diffusion model.
         - latentSplat[3] also renders feature maps that are decoded to a novel view in a GAN setup.
    - In lines 193, the authors describe the use of CLIP image embeddings of the input views as global conditioning.
      - ReconFusion [4] desribes a very similar procedure. 

- Lack of clarity regarding the proposed method:
  - From the paper and supplement, it is not clear, for which loss terms the architecture is optimized during training.
    - Since the paragraph about "Multi-frame diffusion model" starting in line 174 mainly describes SVD [6], it is not clear, whether the loss in equation (2) is also the only loss for MVSplat360.
  - In line 167f., the authors claim that joint training of MVSplat and SVD can further enhance geometry through the feature conditioning.
    - Is that the only source of gradient signals to the structural parameters of the 3D Gaussians (location, scale, rotation...)?
    - ReconFusion [4] and latentSplat [3] both additionally optimize direct RGB renderings to regress the ground truth for better geometry gradients.
  - The role of the video diffusion model compared to a single-view diffusion model is not clear.
    - The paper misses to explain, whether and if so how novel views are refined jointly (see questions).
  
- The proposed approach is incremental.
  - Compared to ReconFusion, MVSplat360 uses MVSplat instead of pixelNeRF and a video instead of an image diffusion model.
  - Compared to latentSplat, it replaces the pixelSplat encoder with the improved MVSplat and a small CNN-based decoder with SVD [6].

- Unclear conclusion of benchmarking:
  - latentSplat builds upon pixelSplat, which is shown to be worse in depth estimation than MVSplat.
    - Regarding this baseline, it is difficult to conclude, how much the reconstruction and the refinement modules contribute to the performance.
  - If the goal of the paper is benchmarking, a component-wise evaluation (backbone feature extractors, depth estimation, refinement networks)  would be more insightful.

1. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction, CVPR 2024
2. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024
3. latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction, ECCV 2024
4. ReconFusion: 3D Reconstruction with Diffusion Priors, CVPR 2024
5. GeNVS: Generative Novel View Synthesis with 3D-Aware Diffusion Models, ICCV 2023
6. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, arxiv 2023
questions: questions: - Are the structural parameters of the Gaussians purely trained via the SVD loss through the features?
  - Does the for features extended 3D Gaussians rasterizer compute gradients through the features to the structural parameters of the Gaussians?
  - If so, what is the intuition why this apparently works, while baselines (latentSplat and ReconFusion) use auxiliary losses?
- Are novel views refined jointly by the video diffusion model?
  - If yes, how do you deal with long videos?
  - Is temporal ordering leveraged or is the model agnostic to temporal permutations of target camera poses?
  - What are implications regarding 3D consistency?
- How is the training done on RealEstate10k?
  - Do you train two different models (also for baselines) for inter- and extrapolation?
limitations: limitations: The authors addressed limitations and societal impacts in the appendix.
paper_summary: paper_summary: This paper proposes MVSplat360, a generalized sparse-view novel view synthesis method. MVSplat360 utilizes the Stable Video Diffusion model to guess the novel views besides input sparse views.
strength_weakness: strength_weakness: 1. Most of the success is due to the stable video diffusion, and this paper provides mainly engineering works.
questions: questions: 1. I would like to see what the contributions are besides introducing the diffusion model.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper aims to advance 360° novel view synthesis from sparse observations in wild scene scenarios. The key idea is to utilize the improved MVSplat for coarse geometry, refined by a stable video diffusion model to enhance appearance. This differs from prior work that, due to sparse viewpoint inputs, resulted in the ambiguous rendering of wide-sweeping or even 360° novel views. The proposed MVSplat360 method has been evaluated on challenging datasets, showing a marked enhancement over existing state-of-the-art techniques.
strength_weakness: strength_weakness: 1. From L193-201 and the provided demo, it becomes apparent that for complex scenes with occluded and invisible parts, influenced by the generative model, MVSplat360 may struggle to achieve stable multi-view consistency solely through "Multi-view conditions."
2. The article addresses the issue of oversaturated colors through a simple post-processing method; however, the effectiveness of this solution is not validated in the experimental and ablation sections.
questions: questions: 1. In L193-201, the "Multi-view conditions" exhibit limitations in maintaining multi-view consistency across scenes. Firstly, can CLIP embedding tokens of the original visible views guarantee a scene-level global description that guides an accurate denoising process? Additionally, for occluded and invisible parts (as depicted by the black pixels in the rendered results in Fig. 4), are the GS-rendered features filled with background values for these areas? How is multi-view consistency ensured in these regions? It is advisable for the authors to include visual results or analyses of rendered views for invisible parts from multiple perspectives of the same scene in the experiments or appendices.
2. The process of “Appearance Refinement” appears to be influenced solely by the Gaussian spherical harmonics parameters. In cases where the geometry inferred by MVS is inaccurate, what impact would this have on the denoising process of SVD?
3. The distinction between a) and b) in Fig.5 is unclear; it is advisable to revise the figure captions accordingly.
limitations: limitations: The authors have thoroughly outlined the limitations of their work as well as the potential negative societal impacts.
paper_summary: paper_summary: This paper proposes a framework of generalizable 3DGS for 360 degree of NVS. The framework comprises of two main components, one is similar to MVSplat that combines multi-view context image information, the other uses a pre-trained diffusion model for post-processing, which is conditioned on the rendered features from the MVS model and outputs detailed images. They also condition the diffusion model on the CLIP features of input images to integrate high-level semantics. Experiments show improvements on both DL3DV and RealEstate10K datasets, especially in qualitative results where the model can generate plausible contents that are not observed in the context images.
strength_weakness: strength_weakness: 1. Contribution: One limitation of this paper is the technical contribution. The combination of generative model and genalizable 3DGS, and conditioning it on rendered features, are similar to latentSplat [1], and the generative process is directly taken from video diffusion models [2]. And the multi-view image feature fusion is taken from MVSplat [3]. The key contribution is somewhat ambiguous.
2. Experiments: the reported quantitative improvements over previous SOTA methods are limited on both datasets. PSNR of $17.00\sim 18.21$ and SSIM of $0.496\sim 0.553$ is far from satisfactory. The authors mentioned that the refinement module does not guantee improvements on pixel-wise metrics, but the improvements on LPIPS is also minor.
3. Efficiency: another key limitation is on the rendering efficiency. Comparing to latentSplat, integrating a heavy diffusion model for post-processing leads to significantly slower rendering speed ($>100FPS\rightarrow\sim 1.75FPS$). The major problem is that applying similar post-processing as NeRF-based works like ReconFusion [4] largely hinders the real-time rendering of 3DGS.



[1] Wewer, Christopher, et al. "latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction." arXiv preprint arXiv:2403.16292 (2024).

[2] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023)

[3] Chen, Yuedong, et al. "MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images." arXiv preprint arXiv:2403.14627 (2024).
questions: questions: 1. To compare with latentSplat more thoroughly, is it possible to evaluate the performance using MVSplat+latentSplat Decoder? The results could better evaluate the necessity of applying a heavy diffusion model for decoding.
2. PSNR results are missing in Table 3(b), and the performance is still not satisfying when using 7 views. Is it possible to provide ablation study results with PSNR using more input views ($>7$) during inference? The results can evaluate the generalization of the model to credibly reconstruct (nearly) fully observed scenes, and also strengthens the claimed 360$^{\circ}$ NVS that can possibly be applied to large scene reconstruction.
3. In Table 2, the extrapolation results of pixelSplat and latentSplat on re10k seem to be adopted from Table 2 of latentSplat, but latentSplat reported interpolation PSNR of 24.32dB for pixeSplat (instead of 25.89dB) due to a different training strategy (supervising on both interpolation and extrapolation views). Is it possible to report latentSplat results and pixelSplat extrapolation results on re10k using the same training strategy as the other methods? The results would form more fair comparisons.
4. In Table 2, MVSplat360 improves only 0.12dB PSNR on extrapolation results over MVSplat. However, there should be major cases where the non-generative methods render nothing in the unobserved regions in extrapolation views (eg. Figure 4 pixelSplat and MVSplat results) due to lack of post-processing. Theoretically, MVSplat360 should improves more than only 0.12dB over MVSplat.
limitations: limitations: See the weaknesses and questions above.

Comments: title: Further response to follow-up comments (2/2)
Further response to follow-up comments (1/2)
Rebuttal Questions
comment: We thank the reviewer for the thoughtful discussions. We are more than grateful for your recognition of our key contributions: our strong performance on the under-explored and challenging 360 scene synthesis setting. 

We will make the writing clearer following all of our discussions. We will also start working on your follow-up suggestions regarding highlighting and further evaluating SVD in terms of 3D consistency, the comparison with Plücker coordinates conditions and mesh reconstruction. Thanks again for your insightful suggestions for making this work more solid.
Thanks again for the discussion. The rebuttal addresses most of my concerns such that I would like to increase my rating to 5: borderline accept.
Reasons for that are:
- I agree with the significance and difficulty of the 360° NVS setting from sparse views on scene level and the strong performance of the proposed method on the DL3DV benchmark.
- The discussion resolved the lack of clarity in the paper.

Additionally, I would like to give the following suggestions for a final version:
- The joint generation of novel views using a video (SVD) instead of an image diffusion model (e.g. SD) conditioned on features rendered from a 3D representation and the resulting 3D consistency is a very important part of the method and should be highlighted and further evaluated.
  - Regarding multi-view diffusion models as a recent trend, 3D consistency of the generated views seems to be the main bottleneck.
  - It would be very interesting to see how view-conditioning via features rendered from a 3D representation compete against alternatives like Plücker coordinates used in multi-view diffusion models.
- For a more convincing evaluation of 3D consistency (compared to the SfM results in the rebuttal PDF), I would recommend a similar approach of mesh reconstruction from generated novel views as done in latentSplat. 

The incremental nature of the proposed method compared to previous works is the main reason that prevents me from giving an even higher rating.
We thank the reviewer for the follow-up comments. We are very grateful for your recognition of our key contribution: good visual quality in an under-explored setting of 360 scene synthesis.

Below, we summarise information to justify the running efficiency, in case you, other reviewers and AC want to refer to it during the internal discussion.

&nbsp;

* **The task of feed-forward 360 scene synthesis from sparse views is very challenging.**
  * Sparse views (e.g., 5 views in our experiments) provide only limited coverage of a scene, resulting in many unseen or occluded regions when rendering from 360-degree viewpoints. This causes significant challenges to previous feed-forward methods like pixelSplat, MVSplat and latentSplat. Prominent failures can be observed in all these methods, as demonstrated in our supplementary video. 
  * Another line of research, like ReconFusion and CAT3D, relies on additional per-scene optimization to achieve plausible results, which is an order of magnitude slower than our method (minutes vs. seconds (Ours)). 
  * In this paper, we demonstrate that high-quality feed-forward 360 scene synthesis can be achieved from sparse views, without any additional per-scene optimization.

* **The efficiency of video diffusion models can be potentially improved with the latest techniques since video diffusion models are actively developed and advanced.**
  * The main efficiency bottleneck of our method lies in the video diffusion model, which is slow since it requires the multi-step denoising process. 
  * However, we note that speeding up video diffusion models is an active topic and advanced techniques are emerging. For example, the recent SF-V model speeds up SVD with improved *one-step* denoising. Our model can benefit from such advancements.

&nbsp;

Overall, we provide a viable solution to the under-explored task of feed-forward 360 scene synthesis from sparse views, we believe our efficiency can be further improved with more advanced video diffusion models.

Feel free to let us know if you would like us to clarify anything further. Many thanks.
Thank you for your detailed rebuttal, which addressed my concerns on the experiments. I think overall this is a technically solid paper on an under-explored direction with good qualitative results. Since I still have concerns about the efficiency of leveraging a video diffusion model on the rendered Gaussian features, I will first raise my rating to 5-borderline accept, and discuss further with the other reviewers and AC in the later phase.
*(see previous comments for more response, thanks.)*

&nbsp;
&nbsp;

> **A7: More details of the refinement module SVD**

**Temporal convolution**

We do not apply similar operations to temporal convolution. We observe that applying temporal attention to all 56 frames leads to oversmoothness, so we change it to local attention with a window size of 14. In contrast, convolution is operated in local regions and has no such issues.

**Discussions with CAT3D**

Sorry for the unintentional confusion regarding “permutation.” Our initial intention was to emphasize that our refinement module takes *unconstrained* camera trajectories. We assume that the trajectory is plausible, as detailed in the above clarification regarding A4 (Test time trajectories), but we do not require it to be a meticulous design such as an orbital trajectory.    

In contrast, CAT3D does assume that its camera trajectory should meet several strict requirements, as detailed in its paper Sec. 3.2 and Appendix C.

&nbsp;

*Feel free to let us know if you would like us to clarify anything further. Many thanks.*
We are grateful to see the follow-up comments. We address the additional concerns below. Feel free to let us know if you would like us to clarify anything further.

> **A2: Discussions of related “concurrent” work**

**Description of GeNVS**   

We agree that the Matterport3D dataset used by GeNVS is similar to RealEstate10K, and we will correct its description to “GeNVS mainly works on 360-degree object-centric scenes *or nearby viewpoint scene-level datasets*”. Nonetheless, it does not alter the previous conclusion as it still belongs to the two existing scenarios summarized in L28-L30 and Fig. I of the one-page PDF. The key contribution of MVSplat360 is that it focuses on an *unexplored* setting: feed-forward *360-degree* scene synthesis. The *majority of experiments are conducted on DL3DV* (rather than RealEstate10K), which significantly verifies MVSplat360’s effectiveness in handling 360-degree NVS from sparse views. 

**Discussions with ReconFusion**   

* We refer the reviewer to the project page of ReconFusion for the results from the diffusion model. In particular, as shown in the last video entitled “ReconFusion distils a consistent 3D model from inconsistent samples” on ReconFusion’s project page, views sampled purely from its diffusion module are far from consistent, showing obvious jittering from frame to frame. Our supplementary video shows that MVSplat360 renders multi-view novel views with much higher consistency. 
* Note that those inconsistent demos provided by ReconFusion mainly consist of forward-facing or *constraint* orbital camera trajectories, while our consistent demo contains way more challenging rendering trajectories, including different types of *unconstrained* trajectories. 
* It is necessary for ReconFusion to apply per-scene optimization in order to get satisfactory consistent novel views, while our MVSplat360 renders 3D consistent novel views in a feed-forward manner thanks to our effective design summarised in “Contributions of MVSplat360 (Method)” in the global response to all reviewers.

&nbsp;

> **A3: Training objectives**

We will update this potentially ambiguous claim regarding "enhancing geometry" in the paper.

* It is correct that the reconstruction loss is the only source of gradients for the structural parameters of the Gaussians.
* Our initial intention is that joint training can help “enhance the backbone features”. Since the features and other Gaussian parameters are from different heads but share the same backbone, we assume that enhancing the backbone feature will lead to a better reconstruction module and, hence better reconstructed coarse geometry. To avoid unintentional overclaim, we will tone down L168 to “the SVD loss can further optimize the Gaussian features, further *enhancing the reconstruction backbone*”.

&nbsp;

> **A4: SD vs SVD**

**Ablation of SD**

In the updated version, we will make it clearer that we denoise all novel views jointly. We will also add visual comparisons with the SD-based ablation model, which shows inconsistent novel views compared to our default design, as observed in our experiments.

**When novel views are far away from each other**

We did not observe obvious limitations regarding this issue. This is probably because although the viewpoints range from 180 to 360 degrees for each scene in DL3DV, it still belongs to one scene and might not contain significant “far away” novel views. 

**Test time trajectories**

* In our experiments, the set of novel views is always a plausible trajectory. In particular, for quantitative evaluation, we use the camera trajectories captured by the initial video. For the video demo, we apply a Gaussian filter to the initial captured camera path to obtain 6DoF stabilization results.
* Each trajectory contains 56 frames, which are uniformly sampled from the initial video that contains around 300 frames.

**Cross-view attention vs. concatenation**

In typical single-view or multi-view diffusion-based NVS models, their input features come from the *source/observed viewpoints*. In this case, they are required to align/correct those source viewpoint features to match with the novel view cameras, hence it might be better to achieve via cross-view attention. In contrast, in MVSplat360, the features provided to the SVD are those rendered from *target/novel viewpoints*, containing coarse but correct geometry information for the target viewpoint, hence it is reasonable to achieve via concatenation. 

It might be helpful to add *additional source/observed view* features to the SVD refinement module via cross-view attention, keeping the current concatenation for *target/novel views*. We will explore this strategy in our further experiments. Thanks for the insightful suggestion!

&nbsp;
&nbsp;

*(see next comments for more response, thanks.)*
I thank the authors for their clarifications regarding my concerns and providing additional results. Follwing the rebuttal, I still have some questions and concerns:

> **A2: Discussions of related “concurrent” work**

The discussion regarding related work is not completely accurate. GeNVS is not limited to object-centric scenes, but the paper includes results for the Matterport3D dataset, which is very similar to RealEstate10k. Moreover, ZeroNVS, ReconFusion and CAT3D do not necessarily require per-scene optimization of a NeRF, but can also be used directly for possibly inconsistent novel view synthesis. However, this is exactly comparable to the proposed MVSplat360. For obtaining a consistent 3D representation that allows fast rendering, the output novel views would need to be fused via optimization, e.g., of 3D Gaussians or a NeRF.

> **A3: Training objectives**

Could you elaborate more on my first question (and sub-points) in the questions section of my review?
In line 167f. of the paper, you claim that joint training of MVSplat and SVD can further enhance geometry through the feature conditioning. Is that really the case, if the auxiliary reconstruction loss is the only source of gradients for the structural parameters of the Gaussians?

> **A3: SV vs SVD**

To me, the fact that you denoise novel views jointly was not clear from the paper. The comparison with SD sounds like an interesting insight that would be very nice to have in the paper as an ablation. However, I have some follow-up questions regarding this:
- As you use concatenation along the channel dimension processed by a 3D UNet of SVD, I would expect problems if novel views are too far away from each other w.r.t. the camera pose. Is that the case?
- How do you handle this at test time? 
  - Does a set of novel views always have to be a plausible camera trajectory? 
  - How long are these trajectories?
- Would cross-view attention as done in multi-view diffusion models be a more suitable alternative than concatenation in channel dimension?

> **A7: More details of the refinement module SVD**

1. SVD does not only consist of temporal attention, but also temporal convolutions. Is temporal consistency in the camera trajectory considered for this fact?
2. I am confused about this. If I am not mistaken, CAT3D conditions on individual camera poses, but does not assume any consistent temporal camera trajectory for this. SVD however is build for videos, e.g., by using temporal convolutions. Therefore, the architecture is tailored for temporally consistent camera poses, making it not agnostic to temporal permutations. Please correct me, if I am wrong.
Dear Reviewer 3TXW,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,   
Authors of #2348
Dear Reviewer pb1U,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,   
Authors of #2348
Dear Reviewer RhQM,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,   
Authors of #2348
Dear Reviewer 4RiL,

Did we satisfactorily answer your questions? Would you like us to clarify anything further? Feel free to let us know, many thanks.

Best regards,   
Authors of #2348