MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views
Authors: Yuedong Chen
Chuanxia Zheng
Haofei Xu
Bohan Zhuang
Andrea Vedaldi
Tat-Jen Cham
Jianfei Cai
Venue: NeurIPS 2024 poster (2024)
Keywords: novel view synthesis
feed-forward 3DGS
3D gaussians splatting
latent video diffusion model
Primary area: generative_models
TL;DR: MVSplat360 is a feed-forward approach for 360° novel view synthesis of diverse real-world scenes, using only sparse observations.

Abstract: We introduce MVSplat360, a feed-forward approach for 360° novel view synthesis (NVS) of diverse real-world scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided, making it challenging for conventional methods to achieve high-quality results. Our MVSplat360 addresses this by effectively combining geometry-aware 3D reconstruction with temporally consistent video generation. Specifically, it refactors a feed-forward 3D Gaussian Splatting (3DGS) model to render features directly into the latent space of a pre-trained Stable Video Diffusion (SVD) model, where these features then act as pose and visual cues to guide the denoising process and produce photorealistic 3D-consistent views. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views. To evaluate MVSplat360's performance, we introduce a new benchmark using the challenging DL3DV-10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-art methods on wide-sweeping or even 360° NVS tasks. Experiments on the existing benchmark RealEstate10K also confirm the effectiveness of our model. Readers are highly recommended to view the video results at [donydchen.github.io/mvsplat360](https://donydchen.github.io/mvsplat360).

MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views
Yuedong Chen'
Chuanxia Zheng²
Haofei Xu
Bohan Zhuang'
Andrea V
Tat-Jen Cham
Jianfei Cail
Monash University VGG, University of Oxford
3ETH Zurich
University of Tübingen, Tübingen Al Center
*Nanyang Technological University
Abstract
We introduce MVSplac360, a feed-forward approach for 360 novel view synthesis (NVS) of diverse real- worldworld scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided, making challenging for conventional methods to achieve high-quality results. Our MVSplat360 addresses this by effectively combining geometry-aware 3D reconstruction with temporally consistent video generation. Specifically, it refactors feed-forward 3D Gaussian Splatting (3DGS) model to render features directly into the latent space of pre-trained Stable Video Diffusion (SVD) model, where these features then act as pose and visual cues to guide the denoising process and produce photorealistic 3D-consistent views. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views. To evaluate MVSpla3360's performance, we introduce
benchmark using the challenging DL3DV- 10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-ar methods wide-sweeping or even 360 NVS tasks. Experiments the existing benchmark RealEstate 10K also confirm the effectiveness of our model. Readers are highly recommended to view the video results at donydchen. github........... 10 1.0...o/mvsse ho//sssaa
Introduction
The rapid advancement in 3D reconstruction and NVS has been facilitated by the emergence of
40 21 These methods, while fundamental and impressive, are primarily tailored for per-scene optimization, requiring hundreds or even thousands of images to comprehensively capture every aspect of the scene. Consequently, the optimization process for each scene can be time-consuming, and collecting thousands of images is impractical for casual users.
differentiable rendering 29 31
In contrast, consider the problem of novel view synthesis in diverse real-world scenes using a lim- ited number of source views through feasibility of rendering wide-sweeping or even 360 novel views using extremely sparse observations, like fewer than 5 images. This task is inherently challenging due to the complexity of scenes, where the limited views do not contain sufficient information recover the whole 3D scene. Consequently, there is necessity to ensemble visible information under minimal overlap accurately and generate missing details reasonably.
network. In particular, this work investigates the
This represents a new problem setting in sparse-view feed-forward NVS. Existing feed-forward methods typically focus on two distinct scenarios: 360 NVS with extremely sparse observations, but
only at object-level [20] [22 666, 59 22 63 56 for scene-level synthesis, but only for nearby viewpoints
62
50
46
47
18] or generating reasonable results
53
6 60
10,
70
42 61].
In contrast, we argue that the time is ripe to unify these previously distinct research directions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MVSpla Myspia
MVSpp
MVSplar
369
Figure 1: Examples of our MVSplat360. Given sparse and in-the-wild scenes, MVSplat360 can directly render 360" novel views (inward outward facing) other natural camera trajectory views in a feed-forward manner, without any per-scene optimization.
observations of diverse
Our goal should be to develop systems capable of synthesizing wide-sweeping or even 360 novel views of large, real-world scenes with complex geometry and significant occlusion. Specifically, this work explores synthesising 360 novel views from fewer than 5 input images. We show that in this
challenging setting, existing ferw-fd scene synthesis approaches [ 6 60
10, [7] struggle
succeed. This failure arises from two main factors: the limited overlap among input views causes many contents to appear in only few views or even a single one, posing significant challenges for 3D reconstruction; ii) the extremely sparse observations lack sufficient information to capture the comprehensive details of the whole scene, resulting in regions unobserved from novel viewpoints.
In this paper, propose simple yet effective framework to address these limitations and introduce the first benchmark for feed-forward 360 scene synthesis from sparse input views. Our key idea to leverage prior knowledge from large-scale pre- pre-trained latent diffusion model (LDM) [35 to "imagine" plausible unobserved and disoccluded regions novel views, which are inherently highly ambiguous. Unlike existing 360 object-level NVS approaches 27
63 49 56 25 50), large-scale real-world scenes comprise multiple 3D assets with complex arrangements, heavy occlusions, and varying rendering trajectories, which makes particularly challenging to condition solely camera poses, as also verified by concurrent work ViewCrafter [68]
To develop performant framework for scene-level synthesis, opt to treat the LDM as refinement module, while relying on a 3D reconstruction model to process the complex geometric informa- tion. Broadly, we build upon the feed-forward 3DGS [21] model, MVSplat [10]. to obtain coarse novel views by matching and fusing multi-view information with the cross-view transformer and cost volume. Although these results imperfect, exhibiting visual artifacts and missing regions (see Fig. M. they represent the reasonable geometric structure of the scene, as they are rendered from 3D representation. Furthermore, we choose Stable Video Diffussinn (SVD) 3 over other image-based LDM as the refinement module, since its strong temporal consistency capabilities align better with the view-consistent requirement of the NVS task, as also observed by concurrent work 3DGS-Enhancer (28. Conditioning SVD with the 3DGS rendered outputs, our MVSplat360 produces visually appealing novel views that multi-view consistent and geometrically accurate (see Fig. [T].
Importantly, the original MVSplat outputs only RGB images, which not optimal condition for the generator, and is difficult to optimize jointly with the SVD denoising module. To tackle this,

---

propose simple Gaussian feature rendering with multi-channels, supervised with an introduced latent space alignment loss. Despite seemingly minor change, the additional feature condition for SVD leads to significant impact: It bypasses the SVD's frozen image encoder, allowing the gradients from SVD to backpropagate to enhance the geometry backbone and lead to improved visual quality, especially on the new challenging DL3DV-10K dataset. While related work Reconfusion [58], CAT3D [15] and latentSplat [57 also combine the 3D representation with 2D generators, the former two focus more on per-scene optimisation, while the latter only shows 360 NVS at the object level.
We conduct a series of experiments, mainly on two datasets. First, we establish a new benchmark on DL3DV-10F dataset (23), creating a new training and testing split for feed- forward wide-sweeping and 360 NVS. In this challenging setting, our MVSplat360 achieves photorealistic 360 NVS from sparse observations and demonstrates significantly better visual quality, where the previous scene-level feed-forward methods [ 6
60 fail to achieve plausible results. Second, we deploy
MVSplat360 on the existing Estate 10K
74) benchmark. Following latentSplat [57]. we estimate both interpolation and extrapolation NVS, and report state-of-the-ar performance.
Our main contributions can be summarized as follows. 1) We introduce a crucial and pressing problem for novel view synthesis, i.e., how to do wide-sweeping or even 360 NVS from sparse and widely-displacee
observations of diverse in-the-wild scenes (not objects) in a feed-forward manner (no any per-scene optimization). 2) We propose an effective solution that nicely integrates the latest feed-forward 3DGS and the pre-trained Stable Video Diffusion (SVD) model with meticulous integration designs, where the former is for reconstructing coarse geometry and the latter for refining the noisy and incomplete coarse reconstruction. 3) Extensive results on the challenging DL3DV-10K and RealEstate 10K datasets demonstrate the superior performance of our MVSplat360.
2
Related Work
Sparse view per-scene reconstruction and synthesis. Differentiable rendering methods, such as NeRF 31 and 3DGS (2I), are mainly designed for very dense views (e.g., 100) as inputs for per-scene optimization, which is impractical to collect for casual users in real applications. To bypass the requirement for dense views. various regularization terms have been proposed in per-scene optimization [33] 1 69 48 Recently, ZeroNVS 38) Reconfusion (58) and concurrent submissions, (28), have leveraged large-scale diffusion models for generating pseudo dense views of a 3D scene, which are then input into per-scene reconstruction pipeline. However, these methods are inherently slow for reconstructing unseen scenes due to the necessity of per-scene optimisation.
including CAT3D [5], ReconX 24. ViewCrafter 68 LM-Gaussian 677, 3DGS-Enhance
Feed-forward scene reconstruction and synthesis. To mitigate these limitations, early approaches like Light Field Networks 400 use ray querying to predict novel views. Subsequent methods [4] 4%)
employ epipolar attention for multi-view geometry estimation. Later, pixeINeRF [66] devised pixel-aligned features for NeRF reconstruction [31), leading to range of subsequent methods that
incorporate feature matching fusion 97, Transformers [36] 13 32] and 3D volume representation 7
60). Recently, 3D Gaussian Splatting 21 has been implemented into feed-forward networks, such as pixelSplat 66) MVSplat [1). Splatter Image [46.. Flash3D
45 and latentSplat [57]. While these methods were successful in novel view synthesis from sparse views, they fail to achieve this in wide-sweeping or 360 setting. Concurrent yet unpublished submissions, DepthSplat [6]] and
Long-LRM (75), also show promising results in the 360 setting, but their frameworks have limited generation capabilities, necessitating the use of denser inputs, e.g., 12 or 32 views.
Camera trajectory controllable synthesis. Generative models have achieved remarkable results for 17 39 3, but they lack precise control over the viewpoint of generated images. To address this, several approaches fine-tune large-scale pre-trained diffusion
image/video synthesis [14 73 8 35
5
models with explicit image and pose conditions 27 26 25 4
64 36, [5]. However, these
methods mainly show 360 NVS results on single objects, leaving the complex scene synthesis problem unsolved. Natural scenes comprise multiple objects with intricate occlusion relationships. presenting greater challenges that are not easily addressed by these single-object NVS models. Besides, camera trajectories can be highly irregular and varied when roving around such complex scenes. Although related works 55 22 38 15) have explored training or fine-tuning diffusion models with camera control for scene synthesis, they often struggle with precise camera pose control 55 22
and still rely on per-scene optimization for 3D reconstruction [38]]]] [5].
3

---

CLIP
Latent Video Diffusion
Multi-View Transformer
Decoder
Cost Volume
Input Views
MVSplat
3D Gaussians
Rendered Features
Output Views
Figure 2: Overview of our MVSplat360. (a) Given sparse posed images as input, we first match and fuse the multi-view information using multi-view Transformer and cost volume-based encoder. (b) Next, 3DGS representation is constructed to represent the coarse geometry of the entire scene. (c) Considering such coarse reconstruction is imperfect, we further adapt pre-trained SVD, using features rendered from the 3DGS representation as conditions to achieve 360 novel view synthesis.
Methodology
Given N sparse views I [ and the corresponding camera poses P [PP)) with pi= (K', R',T) T comprising intrinsic K', rotation R and translation T our goal is to learn a model & that synthesizes wide-sweeping even 360 novel view synthesis (NVS).
58 [5], and to deal with a more general feed- forward network capable of achieving 360 NVS for unseen scenes, yet without the need of additional per-scene training. This requires effectively matching information between sparse views in 3D space, well as generating sufficient content based on only partial observations. To achieve that, our MVSplat360 framework, illustrated in Fig. 2 comprises two main components: multi-view geometry reconstruction module (Section 3.1 and multi-frame consistent appearance refinement network (Section] 3.22.. The former is responsible for matching and fusing multi-vien view information from sparse observations to create coarse geometry reconstruction, whereas the latter is designed to refine the appearance with a pre-trained latent video diffusion model. While similar two-step approaches have been explored in recent related works, e.g., 4 58 38 [5],
We opt to go beyond per-scene optimisation
the first (to the best of our knowledge) to explore it on wide-sweeping or even 360 NVS for large-scale scenes from sparse views (as few as 5), in feed-forward manner.
3.1 Multi- Coarse Geometry Reconstruction
The first module is built upon a feed-forward 3DGS reconstruction model, i.e., MVSplat [[O] in implementation. Specifically, given sparse-view observations I [ corresponding camera poses P {PiJ.N H the model learns to predict 3D Gaussian parameters {Ms, as, Σ, </ N which can then be splatted obtain set of RGB images I using the target camera poses pigt To ensure better integration with the following diffusion module, we predict an additional Gaussian feature fú, in parallel with other parameters, which can be rasterized to the corresponding latent features Figt Furthermore, we also improve the view selection strategy improve the model's robustness handling widely displaced inputs.
J and their
Coarse geometry reconstruction. Our backbone comprises multi-view feature extraction, cost volume construction, depth estimation, and 3D Gaussian parameter predictions. First, a cross-view transformer encoder is applied to fuse multi-view information and obtain cross-view aware features constructed by matching feature correlations
F F=FF |== Then, N cost volumes C ) between cross-views. Specifically, it uniformly divides the depth into L layers in the near and far depth ranges, i.e. D (Dm) 1=- and then warps the features from one view j to another view via F W(Fi, P Pi, D.). The cost volume C (C,, D....., Cb.) then
collected by correlations, where each correlation is expressed as CD... pi-iF
VC with C denoting channel dimension. Finally, the per-view estimated depth d is obtained by applying the softmax operation on the cost volumes in the depth dimension. After that, the Gaussian mean is computed by = --lud uddd + A, where K the camera intrinsic, u = (ug, uy 1) denotes each pixel, and E R is the predicted offset, along with opacity E [0, 1), covariance E E 33x3 and color E 33((+1)²

---

where S is the order of the spherical harmonics [10]. Once the model predicts set of 3D Gaussian parameters {(H;, 0;, En / the target view Itst can be rendered through rasterization.
Gaussian feature rendering. Given sparse-view observations, MVSplat [T0] tends render images with noticeable artifacts in wide-sweeping novel viewpoints (see Fig. D. resulting in suboptimal conditioning for the subsequent SVD. Since the rendered images must first be encoded into the latent space using a frozen encoder (see Section 3.2). enhancing the backbone with gradients from SVD would be computationally expensive. To address this issue, we propose directly rasterising features F into the latent space of SVD, by predicting an additional parameter f. for each 3D Gaussian. This operation offers two advantages: (i) The latent feature includes multi-channel information, providing a more comprehensive representation of the scene; (ii) The entire framework is end-to-end connected by conditioning SVD on the rendered latent features instead of the image-encoded ones. It enables the SVD loss to optimize the Gaussian features, further enhancing the reconstruction backbone.
Observed and novel viewpoints selection. To enable 360 scene synthesis, it is crucial to choose the correct camera viewpoints, so that they can cover most contents in diverse and complex scenes (15) It is impractical to assume a circular orbital camera trajectory like those object-level 360 view synthesis (27 46 72 47,, whereas it suboptimal randomly choose video sequence like existing scene-level nearby viewpoint synthesis [46 6 [0]. To this end, we propose to choose views evenly distributed within set of targeted viewpoints as input. Specifically, for given set of candidate views, we apply farthest point sampling over the camera locations to identify the input views and randomly choose from the rest as target views. The number of candidate views gradually increases throughout the training, stably improving the model's capability toward handling 360 scene synthesis.
View interaction within the local group. Recalling that the 3D reconstruction backbone MVSplat is primarily designed for nearby viewpoints, with key components like multi-view transformers and cost volume assuming sufficient overlap among input views. However, in the more challenging 360" settings, the widely displaced input views lead to minimal overlap between specific view pairs, hindering the effectiveness of the backbone. To mitigate this limitation, we refactor our backbone to use ciow attention and construct the cost volume only within local group of input views based on camera locations, reducing memory consumption and ensuring stable model convergence.
3.2 Multi-Frame Appearance Refinement
Video diffusion model. MVSplat360 utilizes an off-the-shelf multi-frame diffusion model, i.e. Stable Video Diffusion (SVD) 33. to refine the visual appearance of the aforementioned coarse reconstruction. SVD is pre-trained on large-scale video datasets and has strong prior knowledge of temporal consistency. It adheres to the original formulation used by Stable Diffusion (SD) [35] that conducts the denoising processing in the latent space. In particular, given a target sequence of x I:M with M images, they are initially embedded into the latent space by frozen encoder &, yielding J:M = E(x IMM, and then perturbed by adding Gaussian noise ~~~ ~N,0,, I) in a Markov process: 20
2t
J:M vā I.M + - āge, vi-āce I:M
(1)
while āg,... ..., aT a pre-defined noise schedule within steps. Given noise input 27 21:M, the denoiser eg is then trained by optimizing the following objective function:
min Eggua w [lle
co(2) .t. 1.
where xz I:M is the target images, and y is the conditional inputs. After eg is trained, the model can generate a video by performing iterative denoising from pure Gaussians 2Y:M conditioned on y.
Note that SVD is trained with the latest v-prediction formulation (37. instead of the original 6- prediction in SD. Hence, the final loss is calculated in latent space using the mean squared error (MSE) between the ground truth and its prediction [|z] 2. 2l:M
1|2 where 2) sI:M is obtained by translating the velocity v = (=! .t, y) to latent space, i.e., aI:Ad a, 2 o.V.
Multi-view hybrid conditions. To ensure an accurate understanding of the scene, the model requires the integration of both low-level perception (e.g.. depth and texture) and high-level understanding (e.g., semantics and geometry). Following (27 72 [50], we adopt a hybrid conditioning mechanism to fine-tune the SVD model for wide-sweeping NVS with sparse observations.
In one stream, a CLIP 34 image embedding token of the original visible views is used as a global type and text prompt. At each UNet block, cross-attention operation is applied to capture high-level
5

---

semantics of the input images to the model. Since we have sparse views, we average these tokens to become one global token. In the other stream. the spatial conditions from the coarse geometry rendered features Ffg { [ M is channel-concatenated with the noised latent 2t .. These spatially conditional features assist the model to capture the view information, and learn low-level perception to maintain the texture of the scenes. Compared to the concurrent work CAT3D [SS. this coarse feature conditioning not only provides accurate pose information from the 3DGS rendering. but also offers reasonable visual information.
Color adjustment. While our MVSplat360 can achieve photorealistic NVS, the synthesized videos sometimes exhibit oversaturated colors (detailed Appendix C). This may be visually acceptable for video generation, but it can decrease performance when evaluated on NVS task. To mitigate this, we apply post-processing by matching the color histogram between the SVD refined views I'st and our 3DGS rendered views Z before performing pixel-aligned measurements, ie., PSNR and SSIM.
3.3 Training Objectives
Our MVSplat360 predicts two sets of images, including the coarse one Ztst from the 3DGS module and the refined one z from the SVD module, where the former is mainly rendered to help supervise the geometry backbone. The entire model is end-to-end trainable, using three groups of loss functions, namely reconstruction loss, diffusion loss and latent space alignment loss.
In particular, the reconstruction loss is a linear combination of & and LPIPS [71,, applied between the coarse outputs Zu and the corresponding ground truth Zugt The other two loss functions are applied to the following SVD module, whose gradients will backpropagate to the 3DGS module but will not update those structural parameters, i..,, Ja, a, Σ. This is achieved by stopping the gradients from the structural parameters when rendering the latent features figtt since keeping those gradient flows will lead to unstable training, as also observed by latentSplat [57]. We use the standard v- v-prediction formulation (detailed Section 3.2) as the diffusion loss to fine-tune the denoising network of the SVD, keeping the first-stage encoder and decoder frozen. Since the SVD's released model is conditioned on image-cncoded features while our diffusion module is on 3DGS-rendered ones, we find it beneficial to align these two spaces by regularizing with latent space alignment loss, mingo E E I| f 1|2 where g refers to the geometry backbone with trainable parameters 0 and E is the frozen SVD encoder.
Experiments
4.1 Experimental Details
Datasets. To verify the effectiveness of MVSplat360 in synthesizing wide-sweeping and 360 novel views, we have established a challenging benchmark derived from DL3DV-10K [23] comprises 51.3 million frames from 10,510 real-world scenes, adhering to 65 point- ontinterest (POI) [6S] categories. For training, we use subset in subfolders "3K" and "4K", resulting in -2,000 scenes. We tested on the 140 benchmark scenes and filtered them out from the training set to ensure correctness. For each scene, we selected 5 input views using farthest point sampling based on camera locations and evaluated 56 views by equally sampling from the remaining, yielding a total of 7,840 test views. Additionally, since most DI DL.3DV11KK scenes contain a two-round trajectory, we also report another setting by focusing only on half of the sequence, intending to cover the camera trajectory of one round. We denote the two settings as n = 300 and n = 150, where n refers to the frame distance span across all test views, most scenes contain roughly 300 frames. We also assess our model on RealEstate 10K 74), which contains real estate videos downloaded from Tuueube. Consistent with we train MVSplat360 67,477 scenes and test it on 7,289 scenes.
existing works 6 10],
Metrics. To measure models from different perspectives, we follow [57] to report both the pixel-align metrics, i.e., PSNR and SSIM [54], and the perceptual metrics, iee,, LPIPS 77] and DISTS [1]] Since MVSplat360 aims to generate plausible contents for unobserved and disoccluded regions, we also reported the distribution metric, i.e., Fréchet Inception Distance (FID) [16] which measures the similarity between distributions of the generated images and the real ones.

---

Table I: Comparison with SoTA methods on D13DV-10K, Below, is the frame distance span across all the tested novel views within each scene, which is set to 300 by default as most DL3DV-1OK scenes contain roughly 300 extracted frames. Since most DL3DV-10K scenes contain two-round trajectory, we also report another setting of 150 aiming for coverage of one round.
300
1 150
Method
PSNRÍ
SSIM
LPIPS
DISTS!
FID
PSNR
SSIMF
LPIPS
DISTS!
FID
PixelSplat 6
14.83
0,401
0.576
0.383
142.83
16.05
0.453
0.521
0.348
134.70
MVSplat
10
15.72
0.433
0.501
0.291
78.95
17.05
0.499
0.435
0.247
61.92
latentSplat [57
16.68
0,469
0.439
0.234
37.68
17.79
0.527
0.391
0.206
34.55
MVSplat360
16.81
0.514
0.418
0.175
17.01
17.81
0.562
0.352
0.151
18.89
Implementation details. MVSplat360 is implemented with PyTorch and -UDA-implemented 3DGS renderer. For coarse geometry reconstruction (Section (.1). we set hyperparameters following MVSplat [TO), except that we apply cross-view attention and build each cost volume within the nearest views rather than all other views. For multi-frame appearance refinement (Section (.2). we fine-tune from the 14-frame SVD 33 pre-trained model, but using rendered Gaussian features as conditions. We also remove the original "motion value" and "fps" conditions, since they are unrelated to our NVS task. We rescale the rendered feature to have a similar shape as the original image-encode latent feature in the pre-trained model, which is critical for getting better details as it affects the decoder (more discussions are in Appendix A).. We train SVD using 14 frames sampled along natural camera trajectories, captured by the initial videos. At inference, we directly feed 56 views to SVD but change all related temporal attention blocks local attention with a window size of 14 to better align with the training. More implementation details can be found in Appendix B and the codes are publicly available at pottts://githhn hwtp:///ooodcccce////p.....o.c.
4.2 Results on the New DL3DV-10K Benchmark
We first assess the ability of MVSplat360 and baselines synthesize wide-sweeping and 360 NVS in the newly constructed challenging benchmark with diverse scene categories.
Baselines. We perform thorough comparison of MVSplat360 the latest state-of-the-an (SoTA)
3DGS-based models, including pixelSplat (6). MVSppltt [0] and latentSplat (57). All models are trained on the same training split and evaluated the publicly available 140 scenes.
Quantitative results. All models are trained to 100K steps and reported at Table [ except for the Splan, which suffers from unstable training due to its GAN-based architecture. Hence, we report best performance at around 60K training steps before the subsequent collapse. Our MVSplat360 outperforms all existing SoTA models in all metrics on the two settings n = 150. All models generally perform better on the n = 150 setting than on the n = 300 one since the latter spans larger viewpoints. It can be seen that the two generative models (latentSpalt and MVSplat360) generally perform better than the other two regression models, suggesting the importance of additional refinement in addressing feed-forwer forward scene reconstruction.
300 and
Although our improvement on pixel-aligned metrics appears minor, this expected since refinement via either interpolation (for disoccluded regions) or extrapolation (for unobserved regions) does not guarantee matching the ground truth at the pixel level. mainly aims to provide reasonable solution to refine the images and ensure they align with real-world image distribution. This is verified by the fact that our improvements on perceptual metrics are larger, and it is even more apparent on FID, which measures the distribution deviation. The superiority of our MVSplat360 stands out more from the qualitative results presented below.
Qualitative results. The qualitative comparisons are visualized in Fig. MVSplat360 achieves remarkable visual results even under challenging conditions. pixelSplat 6 and MVSplat [0O] exhibit obvious artifacts due to the issue of floating Gaussians. latentSplat [57] improves the results with an additional decoder and adversarial training. However, its resulting object geometry and image quality are still far from satisfactory, suggesting that the GAN-based framework cannot provide enough prior knowledge for refining 360" NVS in diverse real-world scenes. Readers are referred to our project page for video results with more comprehensive comparisons, where our MVSplat360 shows
7

---

Input
pixelSplat e
MVSplal
LD
IatemiSplat
MVSpla1360
Truth
Figure 3: Qualitative comparisons on DL3DV-10K. MVSplat360 shows significant improvement compared to existing SoTA models. Here, we showcase with a rich mix of diversity and complexity, including indoor (bounded) vs. outdoor (unbounded), high vs. low texture frequency, more vs. less reflection, and more vs. less transparency. More results are provided in Appendix E
Table 2: Comparison with SoTA methods on CealEstate10K: We report interpolation scores using settings of [0), where we retrain latentSplat [57] to maintain fair comparison (indicates with
"). We report extrapolation scores by following 57
Interpolation SSIM1
Extrapolation
Method
PSNR
LPIPS
PSNR
SSIM
LPIPS
DISTS,
FID.
PixeINeRF [66]
20.43
0.589
0.550
20.05
0.575
0.567
0.371
160.77
Du al. [13
24.78
0.820
0.213
21.83
0.790
0.242
0.144
11.34
pixelSplat [6
25.89
0,858
0.142
21.84
0.777
0.216
0.130
5.78
latentSplat'
[57
25.53
0.851
0.139
22.62
0.777
0.196
0.109
2.79
MVSplat [10
26.39
0.869
0.128
23.04
0.812
0.185
0.110
3.83
MVSplat360
26.41
0.869
0.126
23.16
0.810
0.176
0.104
1.79
multi-view consistent, high-quality visual results along complex camera trajectories, while others
suffer from apparent artifacts.
4.3 Results the Existing RealEstate10K Benchmark
We also assess MVSplat360 on the existing benchmark, following the existing "Interpolation" set- ting 6 10 and "Extrapolation" setting [57]. We retrained latentSplat on Interpolation and MVSplat
on Extrapolation for fair comparisons.

---

Input Views
pixelSplat
MVSplat
IatemtSplat
MVSpla1360
Ground Truth
Figure 4: Qualitative comparisons on RealEstate MVSplat360 shows reasonable generations
for disoccluded and unobserved regions, while latentSplat [57 fills in content with artifacts.
Table 3: MVSplat360 ablations. All models are trained and evaluated on the DL3DV-10K dataset.
Views
SSIM
LPIPS
DISTS
FID
Models
SSIMt
LPIPS
DISTS
FID
3 views
0.432
0.485
0.203
21.40
Baseline
0,433
0.501
0.291
78.95
views
0,464
0.448
0.187
18.79
+ SVD
0.399
0.556
0.248
38.05
default 0.514
0.418
0.175
17.01
ctx-attn
0.467
0.451
0.185
22.78
+ GS-feet.
0.514
0.418
0.175
17.01
6 views
0,514
0.401
0.169
16.54
views
0.526
0.390
0.166
16.26
(a) Model components. The baseline is the original MVSplat [1O]. 'ctx-attn' refers to using multiple con- text views to enhance the cross attention, while "GS- feat.' is the default model where SVD is conditioned with the 3DGS rendered features.
(b) Number of input views. The 'default' model is trained and tested with 5 views, while the others directly evaluated with different numbers of input views during testing.
Quantitative results. Table shows quantitative comparisons RealEEstate 10K 74 of MVSplat360 and other approaches. Our M VSplat360 surpasses all previous state-of-the-al methods, mainly in terms of the perceptual metrics and the distribution metric. The former implies that rendered views are more aligned with human perception, while the latter shows that our refined images correspond better to the dataset distribution. These observations can be further confirmed by visual assessment.
Qualitative results. The qualitative comparisons of the top four best models are in Fig. 4 Pixel- Splat (6) and MVSplat [10] fail render any content for the unobserved regions due the lack of generative capability. In contrast, latentSplat can perform extrapolation via its GAN-based decoder, improving the overall visual quality. However, observed that the content generated by latentSplat not visually reasonable. Our MVSplat360 generates more plausible content (see the "window" in 1st row and "chair" in 2nd row), thanks to the stronger generative capability of the diffusion model.
4.4 Ablations and Analysis
We conduct ablations on DL3DV-1OK 10K 23 to analyze MVSplat360 further. Results are reported in Table and discussed in detail next.
ccessing model components. The baseline refers to MVSplat 10] since our model is built on top of it. (i A natural extension is to render novel views from MVSplat and use them directly as conditions

---

Figure 5: SfM on input and rendered views. Images with red borders are the input views, while others are rendered by our MVSplat360. The reasonably recovered camera poses and 3D point clouds via VGGSfM imply that our outputs are multi-view consistent and geometrically correct.
in the SVD denoising process. However, this straightforward approach performs slightly worse than the original MVSplat, likely because SVD struggles to infer pose and visual cues from the noisy image-encoded features. (ii) To better utilize the input context views, average CLIPP-ebeddeed tokens from all views instead of just the first. This provides SVD with richer scene information via the cross-attention blocks, leading a noticeable improvement. (iii) Lastly, we render high-dimensional features via the 3DGS rasterizer and concatenate them directly into the diffusion latent space, enabling gradients from the denoising UNet to backpropagate through the geometry backbone. This end-to-end training improves performance significantly and is used in our default model.
Accessing the number of input views. As shown in Table 4b) while model is only trained with 5 input views, the performance can be gradually improved by adding more input views at testing. This is reasonable more input views provide more observable areas. On the contrary, reducing input views will inevitably result in worse performance. Surprisingly, even with 3 sparse views, our MVSplat360 still outperforms regression models (pixelSplat and MVSplat) that use 5 views.
Assessing the geometry accuracy. Our MVSplat360 builds on the video diffusion model SVD, which does ensure strong accuracy. To confirm that our MVSplat360 produces geometrically accurate outputs, we run structure- (SfM) both the input source views and the rendered novel views using VGGSIM (5I). As shown in Fig. S VGGSfM recovers reasonable camera poses and 3D point clouds, confirming that novel views both multi-view view consistent and geometrically correct. This highlights how the 3DGS backbone's latent features provide essential geometric cues, enhancing 3D) consistency in the final SVD-based outputs.
consistency but does not inherently guarantee geometric
Conclusion
present MVSplat360, a feed-forward model that synthesises 360 novel views diverse real- world scenes from sparse input views. Our MVSplat360 leverages feed-forward 3DGS model for recovering coarse geometry and appearance from sparse observations a 3D scene, which are then used to render latent features the pose and visual cues to guide the following SVD in generating 3D consistent 360 novel views. To demonstrate its effectiveness, we construct a challenging benchmark for 360 NVS of real-world scenes. Experimental results show that MVSplat360 achieves superior visual quality compared to other SoTA feed-forward approaches.
A Acknowledgements This research supported by the Monash FIT Start-up Grant, Dr. Chuanxia Zheng is supported by EPSRC SYN3D EP/Z001811/1.
10

---

References
[1] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: ICCV. pp. 5855-5864 (2021)
[2] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P.. Hedman, P:: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: CVPR. pp. 5470-5479 (2022)
[3] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311. 11.15127 (2023)
[4] Chan, E.R., Nagano, K., Chan, M.A., Bergman, W., Park, J.J., Levy, A., Aittala, M., Mello, S.D., Karras, T., Wetzstein, G.: GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In: Proceedings of the International Conference on Computer Vision (ICCV) (2023)
[5] Chang, H., Zhang, H., Jiang. L., Liu, C.,, Freeman, W.T.: Maskgit: Masked generative image transformer. In: CVPR. pp. 11315-11325 (2022)
[6] Charatan, D., Li, S., Tagliasacchi, A., Sitzmann, V:: pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint 1rXiv:2312.1233) (2023)
[7] Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, Yu, J., Su, H.: Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In: ICCV. pp. 14124-14133 (2021)
[8] Chen, Y., Wu, Q., Zheng, C., Cham, T.J., Cai, J.: Sem2nerf: Converting single-view semantic masks to neural radiance fields. In: ECCV, pp. 730-748. Springer (2022)
[9] Chen, Y., Xu, H., Wu, Q. Zheng, C., Cham, T.J., Cai, J.: Explicit correspondence matching for generalizable neural radiance fields. arXiv preprint arXiv:2304. 12294 (2023)
[10] Chen, Y., Xu, H., Zheng, C.,, Zhuang, B., Pollefeys, M., Geiger, A., Cham, T.J., Cai, J.: Mvsplat: Efficient gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.1465 14627 (2024)
[11] Deng, K., Liu, A., Zhu, J.Y., Ramanan, D.: Depth-supervised nerf: Fewer views and faster training for free. In: CVPR (2022)
[12] Ding, K., Ma, K., Wang. S., Simoncelli, E.P.: Image quality assessment: Unifying structure and texture similarity. TPAMI 44(5), 2567-2581 (2020)
[13] Du, Y., Smith, C., Tewari, A., Sitzmann, V.: Learning to render novel views from iide-baseline stereo pairs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4970-4980 (2023)
[14] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: CVPR. pp. 12873-12883 (2021)
[15] Gao, R., Holynski, A., Henzler, P., Brussee, A., Martin-Brualla, R., Srinivasan, P., Barron, J.T.,
Poole, B.: Cat3d: Create anything in 3d with multi-view diffusion models. arXiv (2024)
[16] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge local nash equilibrium. In: NeurIPS, pp. 6626-6666 6227-6637222 (2017)
[17] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models, arXiv preprint arXiv:2210.02303 (2022)
[18] Jin, H., Jiang, H., Tan, H., Zhang, K., Bi, S., Zhang, T., Luan, F., Snavely, N., Xu, Z.: Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint arXiv:2410.177 (2024)

---

[19] Johari, M.M., Lepoittevin, Y., Fleuret, F.: Geonerf: Generalizing nerf with geometry priors. In: CVPR. pp. 18365-18375 (2022)
Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specifie mesh reconstruc- [20]
tion from image collections. In: ECCV. pp. 371-386 (2018)
[21] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics (ToG) 42(4), 1-14 (2023)
[22] Kumari, N., Su, G., Zhang, R., Park, T., Shechtman, E., Zhu, J.Y.: Customizing text-to-image diffusion with camera viewpoint control. arXiv preprint arXiv:2404. 12333 (2024)
[23] Ling, L., Sheng, Y., Tu, Z., Zhao, W., Xin, C., Wan, K. Yu, L., Guo, Q.. Yu, Z., Lu, Y., et al.: D13dv-10k, A lcage scene dataset for deep learning- basd 3d vision. arXiv preprint arXiv:2312. 16256 (2023)
[24] Liu, F,, Sun, W. Wang, H., Wang, Y., Sun, H., Ye, J., Zhang, J., Duan, Y:: Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408. 16767 (2024)
[25] Liu, M., Shi, R., Chen, L., Zhang, Z., Xu, C., Wei, X., Chen, H., Zeng, C., Gu, J., Su, H.: One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion, arXiv preprint arXiv:2311.07885 (2023)
[26] Liu, M., Xu, C., Jin, H., Chen, L., Xu, Z., Su, H., et al.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.16928 (2023)
[27] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-I-to-3: Zero-shot one image to 3d object. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp.
(2023)
[28] Liu, X., Zhou, C., Huang, S.: adgs-enhancerr Enhancing unbounded 3d gaussian splatting with View-consistent 2d diffusion priors. arXiv preprint arXiv:2410. 16266 (2024)
[29] Lombardi, S., Simon, T., Saragih, J.. Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: learning dynamic renderable volumes from images. TOG 38(4), 1-14 (2019)
[30] Melas-Kyriazi, L.., Laina, I., Rupprecht, C., Neverova, N., Vedaldi, A., Gafni, O., Kokkinos, F:: Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. arXiv preprint arXiv:2402.08682 (2024)
[31] Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J., Ramamoorthi, R., Ng. R.: Nerf: Repre- senting scenes as neural radiance fields for view synthesis. In: ECCV (2020)
[32] Miyato, T., Jaeger, B., Welling, M., Geiger, A.: Gta: A geometry-aware attention mechanism for multi-view transformers. In: ICLR (2024)
[33] Niemeyer, M., Barron, J.T., Mildenhall, B., Sajjadi, M.S., Geiger, A., Radwan, N.: Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In: CVPR (2022)
[34] Radford, A., Kim, J. J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748-8763, PMLR (2021)
[35] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684-10695 (2022)
[36] Sajjadi, M.S., Meyer, H., Pot, E., Bergmann. U., Greff, K., Radwan, N., Vora, S., Lučić, M., Duckworth, D., Dosovitskiy, A., et al.: Scene representation transformer: Geometry- novel view synthesis through set-latent scene representations. In: CVPR. pp. 6229-6238 (2022)
[37] Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models, arXiv preprint arXiv:2202.005 12 (2022)
12

---

[38] Sargent, K., Li, Z., Shah, T., Herrmann, C., Yu, H.X., Zhang, Y., Chan, E.R., Lagun, D., Fei-Fei, L.. Sun, D., Wu, J.: ZeroNVS: Zero-shot 360-degree view synthesis from single real image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
[39] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al.: Make-a-video: Text-to-video generation without text-video data. In: ICLR (2022)
[40] Sitzmann, V., Rezchikov, S., Freeman, B., Tenenbaum, J., Durand, F: Light field networks: Neural scene representations with single-evaluation rendering. NeurlPS 34, 19313-19325 (2021)
[41] Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: Continuous 3d- structure-aware neural scene representations. NeurlPS 32 (2019)
[42] Smart, B., Zheng, C.,, Laina, I., Prisacariu, V.A.: Splatt3r: Zero-shot gaussian splatting from uncalibarated image pairs. arXiv preprint arXiv:2408. 13912 (2024)
[43] Suhail, M., Esteves, C., Sigal, L., Makadia, A.: Generalizable patch-based neural rendering. In: ECCV. pp. 156-174. Springer (2022)
[44] Suhail, M., Esteves, C., Sigal, L.,, Makadia, A.: Light field neural rendering. In: CVPR (2023)
[45] Szymanowicz, S., Insafutdinov, E., Zheng, C., Campbell, D., Henriques, J.F., Rupprecht, C., Vedaldi, A.: Flash3d: Feed-forward generalisable 3d scene reconstruction from single image. arXiv preprint arXiv:2406.06s 043443 (2024)
[46] Szymanowicz, S., Rupprecht, C., Vedaldi, A.: Splatter image: Ultra-fast single-view 3d
reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[47] Tang, J., Chen, Z., Chen, X., Wang. T., Zeng, G., Liu, Z.: LGM: Large multi-view Gaussian model for high-resolution 3D content creation. arXiv preprint arXiv:2402.05054 (2024)
[48] Truong, P., Rakotosaona, M.J., Manhardt, F., Tombari, F:: Sparf: Neural radiance fields from sparse and noisy poses. In: CVPR (2023)
[49] Tseng, H.Y., Li, Q., Kim, C., Alsisan, S., Huang, J.B., Kopf, J.: Consistent view synthesis with pose-guided diffusion models. In: Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16773-16783 (2023)
[50] Voleti, V., Yao, C.H., Boss, M., Letts, A., Pankratz, D., Tochilkin, D., Laforte, C., Rombach, R., Jampani, V. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403. 12008 (2024)
[51] Wang, J., Karaev, N., Rupprecht, C., Novotny, D.: Vggsfm: Visual geometry grounded deep structure from motion. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21686-21699 (2024)
[52] Wang, N., Zhang, Y., Li, 7., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: Generating 3d mesh models from single rgb images. In: ECCV. pp. 52-67 (2018)
[53] Wang, Q.. Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin-Brualla, R., Snavely, N., Funkhouser, T.: Ibrnet: Learning multi-view image-based rendering. In: CVPR. pp. 4690-4699 (2021)
[54] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. TIP 13(4) (2004)
[55] Wang, Z., Yuan, Z., Wang, X., Chen, T., Xia, M., Luo, P., Shan, Y.: Motionetrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.0364) (2023)
[56) Weng, H., Yang, T., Wang, J., Li, Y., Zhang, T., Chen, C., Zhang, I..: Consistent123: 23: Improve consistency for one image to 3d object synthesis. arXiv preprint 10.00892 (2023)
13

---

[57] Wewer, C., Raj. K., llg, E.., Schiele, B., Lenssen, J.E.; latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction. arXiv preprint arXiv:2403.1222 (2024)
[58] Wu, R., Mildenhall, B., Henzler, P., Park, K., Gao, R., Watson, D., Srinivasan, P.P., Verbin, D., Barron, J.T., Poole, B., Holynski, A.: Reconfusion: 3d reconstruction with diffusion priors. arXiv (2023)
[59] Wu, S., Li, R., Jakab, T., Rupprecht, C., Vedaldi, A.: Magicpony: Learning articulated 3d animals in the wild. In: CVPR. pp. 8792-8802 (2023)
[60] Xu, H., Chen, A., Chen, Y., Sakaridis, C., Zhang, Y., Pollefeys, M., Geiger, A., Yu, F. Murf: Multi-baseline radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[61] Xu, H., Peng, S., Wang. F., Blum, H., Barath, D., Geiger, A., Pollefeys, M.: Depthsplat: Connecting gaussian splatting and depth. arXiv preprint ar arXiv:2410.13862 (2024)
[62] Xu, Y., Shi, Z., Yifan, W., Chen, H., Yang, C., Peng, S., Shen, Y., Wetzstein, G.: Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint (xXiv:240311462) (2024)
[63] Yang, J., Cheng, Z., Duan, Y., Ji, P., Li, H.: Consistnet: Enforcing 3d consistency for multi-view images diffusion. arXiv preprint 1XX1:23300110444 (2023)
[64] Ye, J., Wang, P., Li, K., Shi, Y., Wang, H.: Consistent-l-to-33 Consistent image to 3d view synthesis via geometry-aware diffusion models, In: Proceedings of the International Conference on 3D Vision (3DV) (2024)
[65] Ye, M., Yin, P., Lee, W.C., Lee, D.L.: Exploiting geographical influence for collaborative point- of-interest recommendation. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. pp. 325-334 (2011)
[66] Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one few images. In: CVPR. pp. 4578-4581 (2021)
[67] Yu, H., Long, X., Tan, P.: Lm-gaussian: Boost sparse-view 3d gaussian splatting with large model priors. arXiv preprint arXiv:2409.03455 (2024)
[68] Yu, W., Xing, J., Yuan, L., Hu, W., Li, X., Huang, Z., Gao, X., Wong, T.T., Shan, Y., Tian, Y.: Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409. 00000 (2024)
[69] Yu, Z., Peng, S., Niemeyer, M., Sattler, T., Geiger, A.: Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. NeurlPS (2022)
[70] Zhang, K., Bi, S., Tan, H., Xiangli, Y., Zhao, N., Sunkavalli, K., Xu, Z.: GS-LRM: large reconstruction model for 3D Gaussian splatting. arXiv preprint arXiv:2404.19700 (2024)
[71] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features perceptual metrie. In: CVPR. pp. 586-595 (2018)
[72] Zheng, C., Vedaldi, A.: Free3d: Consistent novel view synthesis without 3d representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[73] Zheng, C., Vuong, T.L., Cai, J., Phung, D.: Movq: Modulating quantized vectors for high-fidelity image generation. NeurIPS 35, 23412-2344 (2022)
[74] Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph 37 (2018)
[75] Ziwen, C., Tan, H., Zhang, K., Bi, S., Luan, F., Hong, Y., Fuxin, L., Xu, Z.: Long-Irm: Long-sequence large reconstruction model for wide-coverage gaussian splats, arXiv preprint arXiv:2410.1 12781 (2024)
14

---

A More Experiment Results
Assessing the robustness of SVD's first-stage autoencoder to input resolution. Since our MVS- plat360 fine-tunes the denoising process in latent space, it is essential to ensure the autoencoder accu- rately maps between pixel and latent spaces. We observe that when the input resolution (256 480) differs significantly from the autoencoder's pre-trained resolution (768 x 1280), the autoencoding process causes noticeable information loss, leading to missing details (see Fig. A 2nd column) with an average PSNR of 26. 76.77dB on the test set. Although fine-tuning the autoencoder might address this limitation, it risks overfitting due to our relatively small-scale training set (around 2000 scenes). Instead, we find that simply upscaling the input 2x via bilinear interpolation before feeding them the encoder helps preserve details effectively (see Fig. A column) and improves the PSNR to 32.52dB, ensuring the latent space remains accurate.
WHON FOOD MARKE
MARCEY
Original Images
Auto-encoded Images (wlo upscale)
Auto-encoded Images (w/ upscale)
Figure A: Autoencoder using inputs with diffferent resolutions. The SVD's ' first-stage encoder and decoder are sensitive to image resolution. Hence, to ensure that images are encoded to the expected latent space, we upscale them 2 times via bilinear interpolation before feeding them to SVD.
B More Implementation Details
Following MVSplat [T], set the near and far depth range used in the cost volume construction as and 100, respectively. We render features from the 3D Guassians rasterizer with shape 4 x h x w, where lu and w refer the image height and width, respectively, and the channel dimension is set to 4 to align with that of the SVD initial conditional vector. We bilinearly interpolate the latent features resolution of 1 /4h 1/Au, matching the encoded features from images of size 2/ 2w, the encoder downsamples inputs by factor of 8. This design ensures proper projection into the initial latent space, as detailed Appendix A The SVD decoder outputs are then bilinearly interpolated from 2h x 2w back to h X W. Due to resource limitations, we mainly experiment on the "images 8" branch of the DL3DV-10K dataset, which contains images with resolution 256 x 480. Our default model is trained with the Adam optimizer, and the learning rate is set to 1.e 5 and decayed with the one-cycle strategy. All models are trained for 100,000 steps with an effective batch size of on 1 to 8 A100 GPUs, and we apply the gradient accumulation technique whenever needed. During training, we sampled 5 views as input views and another views as targeted rendering views, with the intention of better aligning with the following SVD module, which is trained on videos with 14 frames.
C Limitations and Discussions
Although MVSplat360 achieves plausibly consistent 360 NVS and significantly outperforms previous works in visual quality by leveraging SVD, also inherits several limitations from SVD. For instance, the results may exhibit oversaturated colors (see Fig. B left), limiting improvements pixel-aligned metrics like PSNR and SSIM. This is likely because SVD is primarily trained on artistic videos with vibrant colors, and fine-tuning from its pre-trained weight can bias the outputs toward the original
15

---

Oversturtte oulput
Ground trahh
3DGS backbose outgut
SVD redine wi "tulllacinaioos"
Ground buch
Figure B: Limitations. Left: Views may appear oversaturated; Right: Refined views can contain
over-hallucinated contents. Both limitations stem from the diffusion module's prior knowledge.
training data distribution 50 30 Additionally, the outputs might exhibit hallucinations and contain contents not existing in the input views (see Fig. [ right). Lastly, the inference is slow due the multiple sampling steps the diffusion process. We expect these limitations to be mitigated as better video diffusion models and pre-trained weights become available in the future.
D Broader Social Impacts
Our MVSplat360 renders 360 novel views from sparse observations, making it valuable tool for augmented reality applications. can enhance immersive experiences entertainment and media, including 3D videos and video games, and support historical reconstruction for educational purposes. However, MVSplat360 's powerful generative capabilities could be misused create fake videos. Additionally, while the rendered views are high quality, they may not fully capture real-world details. Therefore, precautions are necessary when using the generated data in safety-criticll applications, such training autonomous driving models.
E
More Visual Comparisons.
Below, we provide more visual comparisons with existing state-of- of-te---- models.
Input Views
pixelSplat
MVSplat
latentSplat
MVSplat360
Ground Truth
Figure C: More qualitative comparisons DL3DV-10K. Our MVSplat360 significantly outper- forms all existing models in various complex scenes.
16

---

Input Views
PixclSplat
MVSplat
latentSplat
MVSplat360
Ground Truth
Figure D: More qualitative comparisons on DL3DV-10K. Our MVSplat360 significantly outper- forms all existing models in various complex scenes.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions are explicitly stated in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
17

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the supplementary material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
18

---

Justification: We provide all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce)
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips cc/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
19

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We elaborate the implementation details of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This paper does not include experiments reporting error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report it in the implementation details.
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
iness
Answer: [Yes]
Justification: This paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive and negative social impact in the supplementary document.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
21

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have correctly cited and credited assets used by our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
22

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "da0YnomrSc", "reviewer": "Reviewer_4RiL", "summary": "The paper proposes MVSplat360, a method for wide-sweepign or 360-degree novel view synthesis on general scenes from sparse input views. It extends an existing state-of-the-art approach MVSplat to render 3D feature Gaussians as conditioning for a refinement network in form of a pre-trained video diffusion model, which is fine-tuned jointly. The approach is evaluated on RealEstate10k, an existing benchmark of home walkthrough videos, and on a newly proposed benchmark for 360-degree novel view synthesis, leveraging an existing dataset of diverse scenes. Quantitative and qualitative results show improvements over regression-based and generative, splatting-based baselines and plausable completions in case of incomplete observations.", "questions": "- Are the structural parameters of the Gaussians purely trained via the SVD loss through the features?\n  - Does the for features extended 3D Gaussians rasterizer compute gradients through the features to the structural parameters of the Gaussians?\n  - If so, what is the intuition why this apparently works, while baselines (latentSplat and ReconFusion) use auxiliary losses?\n- Are novel views refined jointly by the video diffusion model?\n  - If yes, how do you deal with long videos?\n  - Is temporal ordering leveraged or is the model agnostic to temporal permutations of target camera poses?\n  - What are implications regarding 3D consistency?\n- How is the training done on RealEstate10k?\n  - Do you train two different models (also for baselines) for inter- and extrapolation?", "limitations": "The authors addressed limitations and societal impacts in the appendix.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper tackles a challenging and interesting task of generalizable 360-degree novel view synthesis from sparse views on diverse real-world scenes.\n- The proposed method combines the strengths of splatting-based generalizable 3D reconstruction and large-scale pre-trained video diffusion models as a genrative refinement network:\n  - pixelSplat [1] and MVSplat [2] have shown strong performance in view interpolation.\n  - latentSplat [3] has shown the advantages of using a decoder generative network.\n  - Video diffusion models like Stable Video Diffusion [4] have been trained on massive data and learn a strong prior useful for plausible view extrapolation and scene completion.\n- The problem definition and main approach explained well.\n- The evaluation validates the effectiveness of MVSplat360\n  - The proposed benchmark on the DL3DV dataset [5] is challenging and supports the claims of wide-sweeping or 360-degree novel view synthesis on large real-world scenes.\n  - The approach outperforms all baselines in almost all metrics quantitatively on DL3DV and RealEstate10k.\n  - Qualitative results show high-quality novel views with less artifacts than baselines for both datasets with plausible completions for extrapolation on RealEstate10k.\n  - The supplement provides more convincing qualitative results including a video.\n- The paper includes an ablation study validating all the proposed design choices and showing desired performance scaling w.r.t. increasing numbers of input views.\n\n1. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction, CVPR 2024\n2. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024\n3. latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction, ECCV 2024\n4. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, arxiv 2023\n5. DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision, CVPR 2024", "weaknesses": "- Confusing title: Given the contributions, the title should rather focus on the proposed method.\n  - The title of the paper suggests a focus on benchmarking existing methods.\n  - The paper still proposes a new approach MVSplat360.\n  - Evaluation is done on one existing benchmark (RealEstate10k) and one newly proposed one leveraging the already existing DL3DV dataset.\n  - The problem of generalizable novel view synthesis given sparse views is not novel (see baselines [1], [2], [3]) such that the benchmark creation only consists of the definition of input and target views.\n\n- Insufficient contextualization relative to prior work:\n  - Inaccurate use of the term 'concurrent work':\n    - In line 60, ReconFusion [4] and latentSplat [3] (again in line 244) are referred to as concurrent work.\n    - This is not really the case anymore, because\n      - ReconFusion [4] appeared on arxiv in the beginning of December 2023 and was presented at CVPR 2024.\n      - latentSplat [3] appeared on arxiv end of March 2024, 3 days after MVSplat [2], which is one of the building blocks of the proposed method.\n  - Missing references to related work regarding conditioning via feature rendering and CLIP embeddings:\n    - In lines 163-168 and 196-201, the authors propose to render latent features as spatial conditions for a generative model (Stable Video Diffusion), which is later ablated in lines 296-299.\n       - This idea is not novel and the paper is missing important references in this context:\n         - GeNVS [5] proposed to use rendered pixelNeRF features as conditoning for a 2D diffusion model.\n         - ReconFusion [4] adopts this to condition a text-to-image latent diffusion model.\n         - latentSplat[3] also renders feature maps that are decoded to a novel view in a GAN setup.\n    - In lines 193, the authors describe the use of CLIP image embeddings of the input views as global conditioning.\n      - ReconFusion [4] desribes a very similar procedure. \n\n- Lack of clarity regarding the proposed method:\n  - From the paper and supplement, it is not clear, for which loss terms the architecture is optimized during training.\n    - Since the paragraph about \"Multi-frame diffusion model\" starting in line 174 mainly describes SVD [6], it is not clear, whether the loss in equation (2) is also the only loss for MVSplat360.\n  - In line 167f., the authors claim that joint training of MVSplat and SVD can further enhance geometry through the feature conditioning.\n    - Is that the only source of gradient signals to the structural parameters of the 3D Gaussians (location, scale, rotation...)?\n    - ReconFusion [4] and latentSplat [3] both additionally optimize direct RGB renderings to regress the ground truth for better geometry gradients.\n  - The role of the video diffusion model compared to a single-view diffusion model is not clear.\n    - The paper misses to explain, whether and if so how novel views are refined jointly (see questions).\n  \n- The proposed approach is incremental.\n  - Compared to ReconFusion, MVSplat360 uses MVSplat instead of pixelNeRF and a video instead of an image diffusion model.\n  - Compared to latentSplat, it replaces the pixelSplat encoder with the improved MVSplat and a small CNN-based decoder with SVD [6].\n\n- Unclear conclusion of benchmarking:\n  - latentSplat builds upon pixelSplat, which is shown to be worse in depth estimation than MVSplat.\n    - Regarding this baseline, it is difficult to conclude, how much the reconstruction and the refinement modules contribute to the performance.\n  - If the goal of the paper is benchmarking, a component-wise evaluation (backbone feature extractors, depth estimation, refinement networks)  would be more insightful.\n\n1. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction, CVPR 2024\n2. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024\n3. latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction, ECCV 2024\n4. ReconFusion: 3D Reconstruction with Diffusion Priors, CVPR 2024\n5. GeNVS: Generative Novel View Synthesis with 3D-Aware Diffusion Models, ICCV 2023\n6. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, arxiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer 4RiL (R1)**\n\n&nbsp;\n\n### **Q1: “Benchmarking” in the paper title is inaccurate.**\nA1: To make the title reflect our contributions more accurately, we decided to change the title to “MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views”.   \nOur initial intention is to emphasise that MVSplat360 is the first to explore a new setting: 360-degree feed-forward NVS from sparse views for large-scale scenes, and we demonstrate how to benchmark the models’ effectiveness in this setting. Although it differs from the existing two dominated settings (360-degree NVS for objects or limited NVS for scenes) only in source and target view selection, none of the existing approaches can achieve plausible outcomes in this new setting in a feed-forward way (Fig.3, Fig. 6 and the supplementary video). \n\n&nbsp;\n\n### **Q2: Discussions of related “concurrent” work**\nA2: We acknowledge that MVSplat360 is built on top of existing components, similar to several related works (L120, L191-192), but the key focus of this work is to verify that MVSplat360 renders remarkably higher-quality images in an unexplored new setting: 360-degree feed-forward NVS from sparse views for large-scale scenes.   \nExisting approaches, such as ZeroNVS, Reconfusion and CAT3D, still need to train a NeRF with the per-scene optimization approach (L61). GeNVS is tailored for object-centric scenes. LatentSplat fails to render plausible views in the new setting (Fig. 3 and Fig. 6), and its GAN-based decoder easily leads to model collapsing (L248).  We will add detailed discussions in the updated version and remove the term “concurrent work” to avoid unintentional misunderstandings.\n\n&nbsp;\n\n### **Q3: More details of MVSplat360**\nA3: \n* **Training objectives:** Similar to latentSplat, our MVSplat360 contains two objective functions: reconstruction loss and diffusion loss. We modify the MVSplat backbone to render 1) images and 2) features. 1) The images are supervised by the reconstruction loss (L1 and LPIPS loss) against the ground truth RGB images. 2) The features are used as conditions in the following SVD module supervised by the diffusion loss (MSE in latent space as detailed in L185-188). We will detail the training objective functions in the updated version, and will also release our code to ensure better understanding.\n\n* **SD v.s. SVD**: In our experiments, we observe that refining using the single-view diffusion model leads to inconsistency among different novel views, which is similar to the findings of Reconfusion (see Fig. 4 in Reconfusion’s paper). On the other hand, we do not want to train additional 3D NeRF/GS for each scene (like Reconfusion did), which is expensive and fussy. To achieve consistent novel view synthesis without additional per-scene training, we therefore opt for the video diffusion model. The integration is achieved by channel-wise concatenating a sequence of the rendered features with the sampled noise (L196-L198). \n\n&nbsp;\n\n### **Q4: The approach is incremental.**\nA4: Kindly refer to the global response to all reviewers for more discussions regarding our contributions.\n\n&nbsp;\n\n### **Q5: Conclusion of “benchmarking”: ablations on reconstruction and refinement modules**\nA5: Regarding the unintentional misuse of the term “benchmarking,” kindly refer to A1.   \n1. We provide comparisons with the “MVSplat Encoder + latentSplat GAN Decoder” as suggested by R4Q4, which further confirms the superiority of our design.   \n2. We refer the readers to MVSplat’s paper for the ablations of backbone feature extractors and depth estimation. The ablations of the refinement module are provided in Tab. 3.   \n\n&nbsp;\n\n### **Q6: Training objectives of the 3DGS reconstruction module**\nA6: Kindly refer to \"Training objectives\" in A3.\n\n&nbsp;\n\n### **Q7: More details of the refinement module SVD.**\nA7: \n1. The SVD module can, by default, generate an arbitrary number of views. In our experiments, during training, we fixed the target rendering view number to 14 to better align with the SVD pre-trained model (L502-L503). At inference, we feed 56 views to the SVD module and change all related temporal attention blocks to local attention with a window size of 14 to better align with our training.\n2. Our trained refinement module is agnostic to temporal permutations. The main reason is that we condition the diffusion model with features rendered by the 3DGS from arbitrary viewpoints, while related work such as SV3D and CAT3D conditions on camera trajectory, leading to their requirements of meticulously designed order (L135-L140).   \n3. 3D consistency implies that a) our model is 3D-aware due to the usage of 3DGS; b) the rendered novel views are multi-view consistent thanks to the video diffusion module. In addition, we run structure-from-motion on our refined novel views, whose results further confirmed that our outputs are 3D consistent (see Fig. III in the one-page PDF).\n\n&nbsp;\n\n### **Q8: Inter- and extrapolation experiments on RE10K.**\n\nA8: Kindly refer to R4Q6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oBUsVO2yeh", "reviewer": "Reviewer_RhQM", "summary": "This paper proposes MVSplat360, a generalized sparse-view novel view synthesis method. MVSplat360 utilizes the Stable Video Diffusion model to guess the novel views besides input sparse views.", "questions": "1. I would like to see what the contributions are besides introducing the diffusion model.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MVSplat360 achieves better novel view synthesis with sparse input views by introducing the stable video diffusion.\n2. The paper is well-written.", "weaknesses": "1. Most of the success is due to the stable video diffusion, and this paper provides mainly engineering works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer RhQM (R2)**\n\n&nbsp;\n\n### **Q1: Limited contributions: engineering via adding SVD**\nA1: Please refer to the global response to all reviewers for more detailed discussions of our main contributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1wJsNUIov", "reviewer": "Reviewer_pb1U", "summary": "This paper aims to advance 360° novel view synthesis from sparse observations in wild scene scenarios. The key idea is to utilize the improved MVSplat for coarse geometry, refined by a stable video diffusion model to enhance appearance. This differs from prior work that, due to sparse viewpoint inputs, resulted in the ambiguous rendering of wide-sweeping or even 360° novel views. The proposed MVSplat360 method has been evaluated on challenging datasets, showing a marked enhancement over existing state-of-the-art techniques.", "questions": "1. In L193-201, the \"Multi-view conditions\" exhibit limitations in maintaining multi-view consistency across scenes. Firstly, can CLIP embedding tokens of the original visible views guarantee a scene-level global description that guides an accurate denoising process? Additionally, for occluded and invisible parts (as depicted by the black pixels in the rendered results in Fig. 4), are the GS-rendered features filled with background values for these areas? How is multi-view consistency ensured in these regions? It is advisable for the authors to include visual results or analyses of rendered views for invisible parts from multiple perspectives of the same scene in the experiments or appendices.\n2. The process of “Appearance Refinement” appears to be influenced solely by the Gaussian spherical harmonics parameters. In cases where the geometry inferred by MVS is inaccurate, what impact would this have on the denoising process of SVD?\n3. The distinction between a) and b) in Fig.5 is unclear; it is advisable to revise the figure captions accordingly.", "limitations": "The authors have thoroughly outlined the limitations of their work as well as the potential negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Originality and Significance\nThe concept of combining the generalizable GS with the SVD model is eminently logical; this innovative method effectively addresses the current limitations of feed-forward neural novel view synthesis techniques in 360° scene scenarios. By establishing a new benchmark for 360° scene reconstruction from sparse views, this paper proposes a novel direction for future research in the field.\n2. Experimentation and Evaluation\nThe authors constructed a new benchmark using the challenging DL3DV dataset and performed extensive comparisons on the RealEstate10K dataset. The results consistently show that MVSplat360 outperforms existing state-of-the-art methods in both qualitative and quantitative assessments. \n3. Presentation\nThe paper is overall well-written and the limitations have been sufficiently discussed in the supplementary.", "weaknesses": "1. From L193-201 and the provided demo, it becomes apparent that for complex scenes with occluded and invisible parts, influenced by the generative model, MVSplat360 may struggle to achieve stable multi-view consistency solely through \"Multi-view conditions.\"\n2. The article addresses the issue of oversaturated colors through a simple post-processing method; however, the effectiveness of this solution is not validated in the experimental and ablation sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer pb1U (R3)**\n\n&nbsp;\n\n### **Q1: Unsatisfactory multi-view consistency in complex scenes.**\nA1: \n1) The setting is extremely challenging: as verified by all provided visual results, MVSplat360 is the only approach that can provide reasonably good results on occluded and invisible regions compared to other state-of-the-art feed-forward models. \nAlthough concurrent work CAT3D might have a similar demo, it differs from our setting as it relies on per-scene optimization. Please refer to the global response for more discussions between feed-forward and per-scene tuning settings.\n\n2) Long-sequence multi-view consistency for the diffusion model is still under exploration: built on top of the video diffusion model SVD, MVSplat360 maintains good multi-view consistency in typical sequence lengths such as 14 frames. It only becomes less satisfactory when the sequence gets significantly longer such as 56 frames in our testing. This shortcoming originates from our component SVD, and it can be addressed by replacing SVD with other released more powerful video diffusion models in the near future.\n\n3) Our rendered novel views maintain reasonable multi-view consistency as verified by the SoTA reconstruction model: we run structure-from-motion on our rendered novel views, and the results confirm that they are multi-view consistent (see Fig. III in the one-page PDF).\n\n&nbsp;\n\n### **Q2: Ablation of the post-processing operation.**\nA2: Below is the comparison on DL3DV regarding with and without (w/o) post-processing operation, using the same setting as Tab.3.\n\n|                                 |  FID$\\downarrow$  | DISTS$\\downarrow$ | LPIPS$\\downarrow$ |  SSIM$\\uparrow$ |  PSNR$\\uparrow$ |\n|---------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|\n| MVSplat360  w/o post-processing | 20.25 | 0.174 | 0.427 | 0.474 | 16.70 |\n| MVSplat360                      | 20.17 | 0.172 | 0.425 | 0.496 | 17.00 |\n\nAs reported, the post-processing (histogram matching) helps align the output color space with the input one, hence relieving the oversaturated issues caused by SVD and improving the quantitative performance. The improvements are more obvious in those image space metrics (PSNR and SSIM) since other metrics are measured in the feature domain and are more robust to image space distortion. \n\n&nbsp;\n\n### **Q3: More details of multi-view conditions**\nA3: \n1) The CLIP embedding extracted from the context views (observed viewpoints) mainly aims to provide a high-level semantic understanding of the scene (L193-L195). But more importantly, the diffusion module is also conditioned with the 3DGS rendered features of the novel viewpoints (L197-L199). These features contain pose and texture information, guaranteeing a successful denoising process.\n\n2) For invisible parts, the 3DGS rendered features are filled with background values since GS is a regression model with no generative power. Based on the visible parts, information in these invisible regions can be hallucinated by the diffusion model, which has strong prior knowledge gained from large-scale data. The multi-view consistency in the in-painted/out-painted areas is ensured by the strong prior of the video diffusion model. Related novel views containing invisible regions are provided in Fig. II of the one-page PDF.\n\n&nbsp;\n\n### **Q4: The impact of the reconstructed geometry on SVD.**\nA4: The SVD refinement module is influenced by the rendered features, whose rendering/rasterization process takes as input all Gaussian attributes, including both geometry and texture factors, not solely the SH coefficients. In case the geometry is not accurate, the SVD model can help refine to get better visual results using its powerful prior knowledge.\n\n&nbsp;\n\n### **Q5: Unclear illustration of Fig. 5**\nA5: In the updated version, we will indicate (a) and (b) in Fig. 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1wEikt6wHg", "reviewer": "Reviewer_3TXW", "summary": "This paper proposes a framework of generalizable 3DGS for 360 degree of NVS. The framework comprises of two main components, one is similar to MVSplat that combines multi-view context image information, the other uses a pre-trained diffusion model for post-processing, which is conditioned on the rendered features from the MVS model and outputs detailed images. They also condition the diffusion model on the CLIP features of input images to integrate high-level semantics. Experiments show improvements on both DL3DV and RealEstate10K datasets, especially in qualitative results where the model can generate plausible contents that are not observed in the context images.", "questions": "1. To compare with latentSplat more thoroughly, is it possible to evaluate the performance using MVSplat+latentSplat Decoder? The results could better evaluate the necessity of applying a heavy diffusion model for decoding.\n2. PSNR results are missing in Table 3(b), and the performance is still not satisfying when using 7 views. Is it possible to provide ablation study results with PSNR using more input views ($>7$) during inference? The results can evaluate the generalization of the model to credibly reconstruct (nearly) fully observed scenes, and also strengthens the claimed 360$^{\\circ}$ NVS that can possibly be applied to large scene reconstruction.\n3. In Table 2, the extrapolation results of pixelSplat and latentSplat on re10k seem to be adopted from Table 2 of latentSplat, but latentSplat reported interpolation PSNR of 24.32dB for pixeSplat (instead of 25.89dB) due to a different training strategy (supervising on both interpolation and extrapolation views). Is it possible to report latentSplat results and pixelSplat extrapolation results on re10k using the same training strategy as the other methods? The results would form more fair comparisons.\n4. In Table 2, MVSplat360 improves only 0.12dB PSNR on extrapolation results over MVSplat. However, there should be major cases where the non-generative methods render nothing in the unobserved regions in extrapolation views (eg. Figure 4 pixelSplat and MVSplat results) due to lack of post-processing. Theoretically, MVSplat360 should improves more than only 0.12dB over MVSplat.", "limitations": "See the weaknesses and questions above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Originality: the paper proposes a novel setting of scene-level wide-sweeping representation through feed-forward. The setting is practical and with potential. \n\n2. Clarity: The paper is well-written and readers can easily understand the proposed method.", "weaknesses": "1. Contribution: One limitation of this paper is the technical contribution. The combination of generative model and genalizable 3DGS, and conditioning it on rendered features, are similar to latentSplat [1], and the generative process is directly taken from video diffusion models [2]. And the multi-view image feature fusion is taken from MVSplat [3]. The key contribution is somewhat ambiguous.\n2. Experiments: the reported quantitative improvements over previous SOTA methods are limited on both datasets. PSNR of $17.00\\sim 18.21$ and SSIM of $0.496\\sim 0.553$ is far from satisfactory. The authors mentioned that the refinement module does not guantee improvements on pixel-wise metrics, but the improvements on LPIPS is also minor.\n3. Efficiency: another key limitation is on the rendering efficiency. Comparing to latentSplat, integrating a heavy diffusion model for post-processing leads to significantly slower rendering speed ($>100FPS\\rightarrow\\sim 1.75FPS$). The major problem is that applying similar post-processing as NeRF-based works like ReconFusion [4] largely hinders the real-time rendering of 3DGS.\n\n\n\n[1] Wewer, Christopher, et al. \"latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction.\" arXiv preprint arXiv:2403.16292 (2024).\n\n[2] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023)\n\n[3] Chen, Yuedong, et al. \"MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images.\" arXiv preprint arXiv:2403.14627 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Reviewer 3TXW (R4)**\n\n&nbsp;\n\n### **Q1: Limited technical contributions.**\nA1: Kindly refer to the global response to all reviewers for more discussions regarding our contributions.\n\n&nbsp;\n\n### **Q2: The scores of quantitative results on DL3DV are smaller than expected.**\nA2: These values are reasonable since the latest generative 3D synthesis is different to the traditional 3D reconstruction. The PSNR, SSIM, and LPIPS are all pixel-aligned metrics, which are not fully correct for the generated new content. Our MVSplat360 is able to generate multiple and diverse results for the occluded and invisible regions, which are consistent with high quality, but may not match the original ground truth. Note that our generated 360 video is consistent (supplementary video), and the results match well with the inputs’ distribution (FID score improves around 13.4)\n\nFor better reference, we refer to the scores of Reconfusion (CVPR24) and CAT3D (arXiv24) tested on a similar dataset mip-NeRF 360. As reported in their paper, Reconfusion scores (PSNR=16.93, SSIM=0.401, LPIPS=0.544) and CAT3D scores (PSNR=17.72, SSIM=0.425, LPIPS=0.482). Note that even with per-scene optimisation, their scores are not impressive by using 6 observed views and averaging over only 9 scenes, while our scores are reported with a more challenging feed-forward manner, using only 5 observed views and averaging over 140 scenes.\n\nNonetheless, our MVSplat360 outperforms all existing state-of-the-art models, as reported in Tab. 1. The extensive visual results in both the paper and the supplementary video also verify its superiority.\n\n&nbsp;\n\n### **Q3: The rendering speed is slow.**\nA3: Unlike the latest related works, such as ZeroNVS, ReconFusion and CAT3D, that trained additional 3D NeRF for each scene, our MVSplat360 can directly generate consistent novel view video in a feed-forward architecture, which is much faster than these related models (see further discussions in the global response). As discussed in the limitation section (L517-L520), we are aware that integrating with a diffusion model will lead to slower rendering speed. This is caused by the multi-step denoising process, while it can be sped up by those improved one-step SVD models such as SF-V. Since this is beyond the main focus of our work, we leave it for future improvement. Besides, although latentSplat runs faster, its rendering quality on complex scenes is far from satisfactory, as demonstrated in the supplementary video, and latentSplat’s GAN-based decoder is unstable and can cause the model to collapse on complex scenes (L248-L249). \n\n* SF-V: Zhang, Zhixing, et al. \"SF-V: Single Forward Video Generation Model.\" arXiv preprint arXiv:2406.04324 (2024).\n\n&nbsp;\n\n### **Q4: Comparisons with “MVSplat Encoder + latentSplat Decoder”.**\nA4: We replace latentSplat’s encoder with the MVSplat’s encoder, and its performance is reported below.\n\n|  | FID↓ | DISTS↓ | LPIPS↓ | SSIM↑ | PSNR↑ |\n|---|:---:|:---:|:---:|:---:|:---:|\n| latentSplat | 37.68 | 0.234 | 0.439 | 0.469 | 16.68 |\n| MVSplat Encoder + latentSplat Decoder | 35.16 | 0.229 | 0.436 | 0.471 | 16.68 |\n| MVSplat360 | **20.17** | **0.172** | **0.425** | **0.496** | **17.00** |\n\nAs reported, replacing latentSplat’s encoder with the MVSplat’s slightly improves the performance since its backbone features are enhanced. However, latentSplat’s GAN-based decoder still limits its performance in modelling complex scenes. Our MVSplat360 again outperforms this upgraded latentSplat across all metrics, assuring the superiority of our design.\n\n&nbsp;\n\n### **Q5: Testing with more than 7 views.**\nA5: This work mainly targets the sparse view scenario, which is more challenging yet practical for casual users. Testing the feed-forward model (refer to the difference between feed-forward and per-scene tuning models in the global response) with a large number of views is not our primary focus, which is also quite challenging. The main reasons are: 1) Our backbone model MVSplat, similar to pixelSplat, predicts 3D Gaussians in a pixel-align manner, it becomes more difficult to correctly align Gaussians from different views as the view number increases, leading to more artifacts. The unreliable reconstruction results will then impact our refinement module, potentially leading to unexpectedly worse quality. 2) Our model is trained on 5 views, testing it with a significantly different number of views might lead to less effective performance. Hence, we only verify our model within a reasonable range (from 3 to 7 views) to ensure the correctness of our conclusions.\n\n&nbsp;\n\n### **Q6: Inter- and extrapolation experiments on RE10K.**\nA6: We train two separate models for the RE10K experiments, one for interpolation and one for extrapolation. To maintain a fairer comparison, we retrain latentSplat’s model using the interpolation-based training strategy, and its performance is reported in the table below. Our MVSplat360 remains the best. We will update it to Tab. 2 in the updated version.\n\n|  | LPIPS↓ | SSIM↑ | PSNR↑ |\n|---|:---:|:---:|:---:|\n| pixelSplat | 0.142 | 0.858 | 25.89 |\n| latentSplat (retrained with interpolation strategy) | 0.139 | 0.851 | 25.53 |\n| MVSplat360 | **0.126** | **0.869** | **26.41** |\n\n&nbsp;\n\n### **Q7: The PSNR improvement on RE10K is smaller than expected.**\nA7: As replied in A2, the generative 3D synthesis is different from the traditional “accurate” 3D reconstruction. MVSplat360 has a more obvious improvement in the feature-level metrics (L274-L277), although the gain is smaller in the pixel-aligned metric. This might be caused by 1) the oversaturated issues of the SVD model, as detailed in the limitation section (L509-L511); 2) the generative model only provides a reasonable guess for the unseen regions, which is not guaranteed for a pixel-align match with the original ground truth.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Global Response to All Reviewers**

&nbsp;

We thank all reviewers for their constructive comments. We are encouraged by the appraising comments "the problem definition and main approach explained well", "the evaluation validate the effectiveness of MVSplat360" (**4RiL**), "achieves better NVS with sparse input views" (**RhQM**), "a novel direction", "better performance" and "well written" (**pb1U**, **3TXW**). 

&nbsp;

Please kindly check the **attached one-page PDF** for more information. In particular, the PDF contains

* **Figure I: Taxonomy of existing sparse view novel view synthesis.** Our MVSplat360 is the first to address an unexplored task: feed-forward 360 scene synthesis from sparse views.
* **Figure II: Visualization of our MVSplat360 generates multi-view consistent content for invisible regions.**
* **Figure III: 3D reconstruction using input and rendered views from our MVSplat360.** We obtain reasonably good 3D reconstructions of a 360-degree scene, indicating that the rendered views from our model are multi-view consistent and geometrically correct.

&nbsp;

We provide more detailed discussions below.

### **1: Taxonomy of sparse view novel view synthesis** 
A1: Although sparse view novel view synthesis has been extensively explored in recent years, our MVSplat360 targets an unexplored new task. Below, we present the taxonomy regarding our related work (readers are recommended to see Fig. I in the one-page PDF for a more expressive diagram version).

```
|---- Sparse view novel view synthesis
      |---- Per-scene optimization (Reconfusion, CAT3D, ZeroNVS, etc.)
      |---- Feed-forward / Generalizable
            |---- Object-centric
                  |---- 360-degree viewpoints (Zero123, Free3D, SV3D, LGM, latentSplat, etc.)
            |---- Scene-level
                  |---- Nearby viewpoints (pixelSplat, MVSplat, latentSplat, etc.)
                  |---- 360-degree viewpoints (MVSplat360)
```

Note that both Reconfusion and CAT3D require per-scene optimization in a two-step pipeline, where they first generate dense views from a multi-view diffusion model and then optimize a NeRF for each specific scene. In contrast, our MVSplat360 directly outputs 360-degree views in a single feed-forward inference without any test-time optimization. Since Reconfusion and CAT3D require optimizing a NeRF for every unseen scene, it leads to significant time (10+ mins/scene) and additional storage (100+ M/scene) consumption. In contrast, our MVSplat360 uses only a single model, and it can be directly tested on unseen scenes, which is more efficient in terms of inference time (~32 secs/scene) and storage consumption (0 additional storage/scene).

&nbsp;

### **2: Contributions of MVSplat360**

A2: Although existing works, such as GeNVS, ZeroNVS, Reconfusion, and CAT3D, also train/fine-tune a diffusion generator for novel view synthesis, the research goal and technical difference are significant:

1) **(Task) an unexplored yet more practical setting:** 360-degree feed-forward NVS from sparse views for large-scale scenes is an unexplored setting. Most highly related works, such as ZeroNVS, Reconfusion and CAT3D, need to train a NeRF for each scene (which are *not* feed-forward), while other 360-degree NVS works mainly focus on object-centric scenes (L27-L30).   
In contrast, our main goal is to build a feed-forward 360-degree NVS model for large-scale scenes without per-scene optimization. To the best of our knowledge, we are the first to explore this challenging new setting on the new dataset, which will shed new light on how to advance the area of sparse-view NVS.
2) **(Method) an effective feed-forward model for the new setting:** Although combining conditional NeRF with an image/video generator has been explored in recent works, coming up with our design and making it work for such a challenging setting are non-trivial. All existing SoTA approaches fail to achieve satisfying results on the new setting. In particular, we 1) build a GS feature rendering as coarse geometry for multiview SVD generation; 2) extend the single-view-to-video SVD model to a multi-view conditioned refinement model; 3) propose a color adjustment mechanism to relieve the oversaturated issue in SVD.   
Furthermore, to achieve the goal, it needs suitable training data and training strategy. We build a new training and testing split from the latest DL3DV dataset and design the nearest view warping strategy to train the model.   
Note that many SoTA impactful methods are also simple at the concept level, e.g., Splatter Image and pixelSplat replace the NeRF in pixelNeRF with 3DGS, and MVSplat replaces the NeRF in MVSNeRF with 3DGS, but similar to ours, a lot of detailed designs are needed to make them work.
3) **(Results) remarkably higher quality visual results.** As demonstrated in the paper and the supplementary video, MVSplat360 renders much better novel views than all existing SoTA feed-forward models. We highly recommend the readers to view the supplementary video.

Meta-review: The paper received borderline reviews initially, stating a large variety of concerns about the clarity, paper framing and technical novelty of the approach. At the same time, all reviewers acknowledged the high quality results of the presented method in a very challenging setup.

The authors provided exhaustive answers, leading to the scores converging to 4x borderline accept. While some concerns regarding clarity and paper framing have been resolved, some others, especially technical novelty, remain unresolved.

After reading the paper, reviews and rebuttal, I can draw the following conclusions:
- The paper indeed shows impressive results in 360° novel view synthesis, qualitatively and quantitatively.
- The tackled setting is a very challenging one and still heavily underexplored. Methods that solve it are very welcome in the community.
- The technical novelty is limited, as the method essentially combines two existing methods (MVSplat + SVD) into a single pipeline in a similar way as Reconfusion did with pixelNeRF + an image diffusion model.
- The paper could do a better job in highlighting specific aspects that surfaced during the discussion phase, such as the effect of using a pre-trained video diffusion model instead of image diffusion models. Such analysis could lead to more conceptual insight transported to the reader, strengthening the paper.

This paper is clearly in a borderline state. Since all reviewers are agreeing on an accept score and I am also impressed by the qualitative results of the work, I follow the reviewers and recommend acceptance.

I heavily encourage the authors to adjust the paper by incorporating the different aspects that surfaced in the discussion phase.
Decision: Accept (poster)