3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors
Authors: Xi Liu
Chaoyi Zhou
Siyu Huang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: 3D model enhancement
3D Guassian splatting
novel view synthesis
diffusion model
image restoration
Primary area: machine_vision
TL;DR: We propose a method that exploits view-consistent 2D generative priors, i.e., a video diffusion model, to enhance 3D Gaussian splatting rendering quality.

Abstract: Novel-view synthesis aims to generate novel views of a scene from multiple input
images or videos, and recent advancements like 3D Gaussian splatting (3DGS)
have achieved notable success in producing photorealistic renderings with efficient
pipelines. However, generating high-quality novel views under challenging settings,
such as sparse input views, remains difficult due to insufficient information in
under-sampled areas, often resulting in noticeable artifacts. This paper presents
3DGS-Enhancer, a novel pipeline for enhancing the representation quality of
3DGS representations. We leverage 2D video diffusion priors to address the
challenging 3D view consistency problem, reformulating it as achieving temporal
consistency within a video generation process. 3DGS-Enhancer restores view-
consistent latent features of rendered novel views and integrates them with the
input views through a spatial-temporal decoder. The enhanced views are then
used to fine-tune the initial 3DGS model, significantly improving its rendering
performance. Extensive experiments on large-scale datasets of unbounded scenes
demonstrate that 3DGS-Enhancer yields superior reconstruction performance and
high-fidelity rendering results compared to state-of-the-art methods. The project
webpage is https://xiliu8006.github.io/3DGS-Enhancer-project.

3DGS-Enhancer:
Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors
Xi Liu
Chaoyi Zhou
Siyu Huang
Visual Computing Division
School of Computing Clemson University
{xi9, chaoyiz, siyuh)@clemson.co edu
3DGS (PSNR: 16.29)
Ours (PSNR: 26.04)
Ground-truth
3DGS (PSNR: 12.83)
Ours (PSNR: 22.26)
Ground-truth
Figure 1: The 3DGS-Enhama Enhancer improves 3D Gaussian splatting representations on unbounded scenes with sparse input views.
Abstract
Novel-view synthesis aims to generate novel views of scene from multiple input images or videos, and recent advancements like 3D Gaussian splatting (3DGS) have achieved notable success in producing photorealistic renderings with efficient pipelines. However, generating high-quality novel views under challenging settings, such sparse input views, remains difficult due insufficient information in under-sampled areas, often resulting in noticeable artifacts. This paper presents 3DGGSFEnancee,, novel pipeline for enhancing the representation quality of 3DGS representations. We leverage 2D video diffusion priors to address the challenging 3D view consistency problem, reformulating it as achieving temporal consistency within video generation process. consistent latent features of rendered novel views and integrates them with the input views through spatial-tempore decoder. The enhanced views are then used fine-tune the initial 3DGS model, significantly improving its rendering performance. Extensive experiments on large-scale datasets of unbounded scenes demonstrate that 3DGG-Ennance yields superior reconstruction performance and high-fidelity rendering results compared to tfate-of-the-as art methods. The project webpage is https://xiliu8006 github.io/3DGr CaCG-----neer---- Enhancer-project.
restores view-
Corresponding author: Siyu Huang
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Novel-view synthesis (NVS) has decades of history in computer vision and graphics communities, aiming to generate views of scene from multiple input images or videos. Recently, 3D Gaussian splatting (3DGS) [18] has excelled in producing photorealistic renderings with a highly efficient rendering pipeline. However, rendering high-quality novel views far from existing viewpoints remains very challenging, as often encountered in sparse-view settings, due to insufficient information in under-sampled areas. As shown in Figure noticeable ellipsoid- like and hollow artifacts manifest when there are only three input views. Due to these common low-quality rendering results in practice, it is essential to enhance 3DGS ensure its viability for real-world applications.
To our knowledge, few prior studies have specifically focused on enhancement methods aimed at improving the rendering quality of NVS. Most existing enhancement work for NVS [19, 44] focuses on incorporating additional geometric constraints such as depth and normal into the 3D reconstruction process to fulfill the gap between the observed and unobserved regions. For example, DNGaussian [19] applies a hard-and-soft depth regularization to the geometry of radiance fields. However, these methods heavily rely the effectiveness of additional constraint and are often sensitive to noises. Another line of work leverages generative priors to regularize the NVS pipeline. For instance, ReconFusion [41] enhances Neural Radiance Fields (NeRFs) [26] by synthesising the geometry and texture for the unobserved regions. Although it can generate photo-realistic novel views, the view consistency is still challenging when the generated views are far away from the input ones.
In this work, we exploit the 2D generative priors, e.g., the latent diffusion models (LDMs) [32], for 3DGS representation enhancement. LDM has demonstrated powerful and robust generation capabilities in various image generation [32] and restoration tasks [43]. Nevertheless, the main challenge lies in the poor 3D view consistency among generated 2D images, which significantly hinders the 3DGS training process that requires highly precise view consistency. Although some efforts have been made, such the Score Distillation Sampling (SDS) loss [30] that distills the optimization objective of a pre-trained diffusion model, it fails to generate the 3D representation allowing rendering high-fidelity images
Motivated by the analogy of the visual consistency between multi-view images and the temporal consistency between video frames, we propose to reformulate the challenging 3D consistency problem as an casier task of achieving temporal consistency within video generation, so we can leverage the powerful video diffusion models for restoring high-quality and view-consistent images. We propose novel 3DGS enhancement pipeline, dubbed JDGS-Ennancer The core of LDM consisting of image encoder that encodes latent features of rendered views, iideo-based diffusion model that restores temporally consistent latent features, and a ppatial-temporal decoder that effectively integrates the high-quality information in original rendered images with the restored latent features. The initial 3DGS model will be finetuned by these enhanced views to improve its rendering performance. The proposed 3DGS-Enhanen unbound scenes from sparse views and generate the natural 3D representation for the invisible area between two known views. A cocurrent work V3D [7] also leverages latent video diffusion models [4] for generating object-level 3DGS models from single images. In contrast, our -nnnnnEhaneer focuses on enhancing any existing 3DGS models and thus can be applied to more generalized scenes, e.g., the unbounded outdoor scenes.
is a video
can be trajectory-free
reconstruct the
In experiments, generate large-scale datasets with pairs of low-quality and high-quality images on hundreds of unbounded scenes, based on DL3DV [21], for comprehensively evaluating the novelly investigated 3DGS enhancement problem. Empirical results demonstrate that the proposed 3DGS-Fnhance method achieves superior reconstruction performance on various challenging scenes, yielding more distinct and vivid rendering results. The code and the generated dataset will be publicly available. The contributions of this paper are summarized as follows.
1. To the best of our knowledge, this is the first work to tackle the problem of enhancing low- quality 3DGS rendering results, an issue that widely exists in practical 3DGS applications.
2. We propose a novel pipeline
that addresses the 3DGS enhancement prob- lem. 3DGS-Enhancer reformulates the 3D-consistent image restoration task as temporally consistent video generation, such that powerful video LDMs can be leveraged for gener- ating both high-quality and 3D-consistent images. Novel 3DGS fine-tuning strategies are

---

achieving superior enhancement performance for also devised for an effective integration of the enhanced views with the original 3DGS representation.
3. We conduct extensive experiments on large-scale datasets of unbounded scenes to demon- strate the effectiveness of the proposed methods over existing Sfate-of-the-a few-shot NVS methods.
Related Work
Radiance fields for novel view synthesis. Novel view synthesis (NVS) aims to generate unseen viewpoints from a set of input images and camera information. Radiance fields methods, like NeRFs [26], encode 3D scenes as radiance fields and use volume rendering for novel views, achieving high-fidelity results but at the cost of lengthy training and inference times. Improvements such as Mip-NeRF [1. 2] enhance rendering quality through anti-aliasing, while others [6, 9, 47, 27] focus on speeding up the processes. Recently, 3D Gaussian splatting (3DGS) [18] has emerged, offering competitive rendering quality and significantly higher efficiency by representing scenes as Gaussian spheres and using fast differentiable splatring pipeline [50]. However, 3DGS still requires high-quality and numerous input views for optimal reconstruction, which often impractical.
Few-shot novel view synthesis. Leveraging additional information is essential for generating novel views from sparse input images. Various approaches incorporate different regularization techniques to prevent 3D geometry from overfitting to the training views. [19, 10, 28, 23] introduce extra geometric information, such as depth maps or coarse mesh, enhance the robustness and performance of 3D reconstruction from sparse views. [5, 8] leverage the learned priors from multi-view stereo datasets as general priors to improve performance sparse view reconstruction tasks. FreeNeRF [44] integrates frequency and occlusion regularization during training to mitigate overfitting issues in few-shot neural rendering. Similarly, DietPixelNeRF [16] employs a semantic view consistency loss to ensure that all views share consistent semantics, thereby alleviating overfitting. However, these methods are highly sensitive to the network's performance, where incorrect depth estimations or inaccurate mesh reconstructions can significantly degrade the final output.
Diffusion priors for novel view synthesis. Recently, utilizing diffusion models as priors for few-shot novel view synthesis has proven to be an effective approach. DreamFusion [30] employs Score Distillation Sampling (SDS) with a pre-trainne pre-trained diffusion model to guide 3D object generation from text prompts [36, 33, 46]. Some works [22, 34, 35] embed 3D awareness into 2D diffusion models to generate multi-view images, though these methods typically require large datasets [49] and significant training resources [16, 28]. ReconFusion [41] leavarge the 2D diffusion priors to recover high- fidelity NeRF from sparse input views. More advanced approaches leverage video diffusion models [4, 12, 13, 24] for few-shot NVS. For instance, AnimateDiff [11] fine-tunes diffusion models with additional camera motions using LoRA [14], while methods like SVD-MV [4], V3D [37] and IM-3D [24] propose camera-controlled video diffusion models for object-level 3D generation. In contrast, our approach offers greater generalizability for unbounded outdoor scenes.
Radiance fields enhancement. Several existing studies focus on enhancing NeRFs by addressing the limited detail preservation issue caused by insufficient or low-quality input data. NeRF-SR [38] and Refsr-nerf [15] use a super-resolution network to upscale the training view images, allowing novel views to be synthesized higher resolutions with appropriate details. Alignerf [17] introduce optical-flow network to solve the misalignment problem to enhance the performance. Some other approaches incorporate 2D diffusion priors into 3D reconstructions. For instance, DiffusionNeRF [42] leverages diffusion model to learn gradients of logarithms of RGBD patch priors, serving as regularized geometry and color for a scene. Nerfbusters [40] use diffusion priors to remove ghostly artifacts in the 3D gaussians. Our work aim to addresses the radiance fields enhancement problem by proposing a novel framework low-quality unbounded 3DGS representations.
3 Preliminary of 3D Gaussian Splatting
Here, we briefly review the formulation and rendering process of 3DGS [18]. 3DGS represents scene as a set of anisotropic 3D Gaussian spheres, allowing high-fidelity NVS with extremely low rendering latency. A 3D Gaussian sphere includes a center position μ e gg scaling factor E 123
3

---

Video diffusion prior for novel view enhancement
3D representation fine-tuning
Temponii Denosing U-Nen
VAE Enc
CLIP
Fine: luned
S
Frozen
Ingul rendered
ResBlock
Spatio-tempral atteetion
Enhanned novel views
An overview of the proposed ment. We learn video diffusion priors on a large-scale novel view synthesis dataset to enhance the novel views rendered from the 3DGS model on novel scene. Then, the enhanced views and input views jointly fine-tune the 3DGS model.
Figure 2:
framework for 3DGS representation enhance-
and rotation quaternion q R4, such that the Gaussian distribution is
G(x) (x-n)) (
where Σ RRST RT, S the scaling matrix determined by s and is the rotation matrix determined by q. To additionally model the view-dependent appearance, the Gaussian sphere also includes spherical harmonics (SH) coefficients C E RR*, where is the number of SH functions, and a E IR for opacity. The color and opacity are also calculated by the Gaussian distribution illustrated in Eq. 1.
For rendering. all the 3D Gaussian spheres projected onto the 2D camera planes via differentiable Gaussian splatting pipeline [50]. Given the viewing transform matrix W and Jacobian matrix the affine approximation of the projective transformation, the covariance matrix E' in camera coordinates calculated as
E' J"
The differentiable splatting method efficiently projects the 3D Gaussian spheres to 2D Gaussian distributions, ensuring fast a-blending for rendering and color supervision. For each pixel, the color rendered by M Gaussian spheres that overlap with the pixel on the 2D camera planes, sorted the depth distance as
C= Σ Cass II oz)- jEM
f=1
4 Method
4.1
3DGS-Enhancer: An Overview
This work studies the 3DGS enhancement problem. More specifically, given a 3DGS model trained on scene consisting of input views { l ....,,,,... and corresponding camera poses {pter , .......p' . the goal of this work is to enhance a set of low-quality novel views [11,2,133 Iz...., I.....,,IN.... IN...l rendered by the 3DGS model. The enhanced images further fine-tune the 3DGS model to improve its reconstruction and rendering quality.
This work novelly reformulates the challenging task of 3D-consistent image restoration as the task of video restoration, in light of the analogy between the multi-view consistency and the video temporal consistency. We propose a novel framework named 3DGS -nhhancr that employs a video LDM comprising an image encoder, video-based diffusion model, and spatial-tempora. decoder to enhance the rendered images while preserving a high 3D consistency. DDGGS-Enhance also adopts novel fine-tuning strategies to selectively integrate the views enhanced by the video LDM into the 3DGS fine-tuning process. An illustration of the 3DGS-Fnnanceer framework shown in Figure We discuss more details of the framework in the following.

---

4.2 Video Diffusion Prior for Temporal Interpolation
In this section, we introduce the video diffusion model for achieving co-csssitt 2D image restoration. To lift the consistency between the generated 2D video frames and the high-quality reference views, we further propose formulate the video restoration task as a video interpo- lation task, where the first frame and the last frame of inputs to the video diffusion model are two reference views. This formulation provides stronger guidance for the video restoration pro- cess. Let {p.-1- Pi, P P.. tween two reference views, the images rendered accordingly RR( serves as the input to the video diffusion model, e-g., a pre-trained image- guided stable video diffusion (SVD) model [4, 20] that adopts cross-frame spatio-temporal attention module and 3D residual convolution in the diffusion U-Net. Unlike SVD, which repeats the single input image feature extracted by CLIP [31] for T times as the conditional inputs, we input to the CLIP encoder to get a sequence conditional inputs Celip and add to the video diffusion model through cross attention. Meanwhile, we input to the VAE encoder to get latent feature Cvae and add it into the diffusion model through a classifier free guidance strategy to incorporate richer color information. The diffusion U-Net eg predicts the noise for each diffusion step t, and the training objective is
pt*) P!) be the camera poses sampled from the trajectory fitted be-
{ [1. .2..... Ir. I,
Ldituusinn E |||c co co(2), L, Celip: Crae
where 2 = 0,2 + TyE *.. where z is the gt latent, E No I), ar and o, define noise at timestep t. The learned video diffusion model generates sequence of enhanced image latents 2, corresponding to the rendered low-quality views v.
4.3 Spatial-Temports Decoder
Although the video diffusion model can generate enhanced image latents 2g, observe that there are artifacts such as temporal inconsistency, blurring, and color shift in outputs of the original decoder of video LDM. To address this issue, we propose modified spatial-temporal decoder (STD). STD makes the following improvements over the original VAE Decoder: 1) Temporal decoding manner. STD adopts additional temporal convolution layers ensure the temporal consistency between decoded outputs. Similar to our video diffusion model, the first and the last input frames are the reference view images, and the intermediate inputs are the generated views; Effective integration of rendered views. STD adopts additional conditional inputs, same as those of the video diffusion model, allowing the decoder to better leverage the original rendered images. Inspired by [45, 48], these conditional inputs are fed into STD through Controllable Feature Warping (CFW) modules [39], such that their nighffreuencc patterns are better preserved. 3) Color correction. To address the color shift issue, we apply color normalization to the decoded images by following StableSR [39]. However, we observe that highly blurred and low-quality images in the conditional inputs can undermine the color correction effects. To mitigate this, we use the first reference view to calculate the mean and variance, and then align all the other decoded images with this reference view. Let I be the i-th decoded images with a mean Usg and a variance igg. Ig the reference view with a mean H and a variance ry,, the corrected image 18 is computed by:
1
P1
1
+ fgg-
(5)
0T
The optimization objective of STD consists of an LI reconstruction loss and an LPIPS perceptual loss between 1 and fround-truth 19, and an adversarial loss, as
LSTD = Eree(19 fo) + (T fo) + (
(6)
where Lut is the adversarial loss that discriminates between real image 1' and fake image 19.
4.4 Fine-tuning Strategies of 3D Gaussian Splatting
Confidence-aware 3D Gaussian splatting. Unlike existing sparse-view NVS methods, our ap- proach does not rely on depth estimation networks for depth regularization. Instead, we take a purely 2D visual method by utilizing video diffusion model enhance images rendered from low-quality 3DGS model. Despite this significant enhancement in the quality of the rendered views, propose
5

---

Rendered images
Confidence Map
GT
Figure 3: The red circle indicates the area with high confidence, meaning the generated videos can contribute more information. Conversely, the green quadrilateral highlights the area with low confidence, suggesting that the generated video should not tend to optimize this area.
to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive slight inaceuracies in the restored views. These inaccuracies could be amplified during the fine-tuning process.
To minimize the negative impact of generated images Gaussian training, propose confidence- aware 3D Gaussian splatting. This strategy involves two levels of confidence, image level and pixel level. For the image level, the generated images that closer to real images have lower confidence. For pixel level, the larger the mean covariance of all the Gaussians used to render this pixel, the higher its confidence.
Image level confidence. In the task of novel view synthesis, if noise exists two image views, a close distance between them increases the likelihood of generating conflicts and disrupting the 3D consistency of the scene. Therefore, for novel views that close the reference view, it is crucial to carefully optimize the 3D Gaussians to mitigate the adverse effects of noise. Conversely, when novel view far from all known views, it has smaller likelihood of disturbing already well-reconstrueted areas. Based on this reasoning, we normalize the distance from novel views to reference views between 0 and The farther viewpoint is from the reference view, the higher its confidence.
Pixel level confidence. Inspired by ActiveNeRF [29], which uses Gaussian distributions in NeRF estimate uncertainty and identify views with the highest information gain, we aim to find the pixels that can provide the highest information gain from the generated images. As shown in Fig 3, we observed that well-reconstruutee areas are typically represented by Gaussians with very small volumes, calculated using the scaling vector E IR. Based on this observation, we propose method to calculate pixel-level confidence.
The unique representation of 3D Gaussians allows us to render an HxW x3 image using a process similar rendering colors, where each channel corresponds to one of the three components of the scaling vector s. In 3DGS-EN SinS-Enhancer, we multiply these three channels of the scale map to obtain pixel-level confidence. For each pixel the generated images, higher confidence results in greater weight in supervising the training of the 3DGS model.
Given set of 3D Gaussian, the 3-channel Ccanf confidence map is rendered same colour rendering, and the formula is defined as follows
Coonf
Σ s;0; I
(1 as). 1=1
(7)
REM

---

Table 1: A quantitative comparison of few-shot 3D reconstruction. Experiments DL3DV and
LLFF follow the setting of [44]. Experiments on Mip-NeRF 360 follow the setting of [41].
3 views
6 views
9 views
Method
PSNR SSIM [PIPS||
PSNR
SSIM LPIPS
PSNRT
SSIM
LPIPS
DL3DV (130 training scenes, 20 test scenes)
Mip-NeRF [1]
10.92
0.191
0.618
11.56
0.199
0.608
12.42
0.218
0.600
RegNeRF [28]
11.46
0.214
0.600
12.69
0.236
0.579
12.33
0.219
0.598
FreeNeRF [44]
10.91
0.211
0.595
12.13
0.230
0.576
12.85
0.241
0.573
3DGS [18]
10.97
0.248
0.567
13.34
0.332
0.498
14.99
0.403
0.446
DNGaussian [19]
11.10
0.273
0.579
12.67
0.329
0.547
13.44
0.365
0.539
3DGS-Enhancer (ours)
14.33
0.424
0.464
16.94
0.565
0.356
18.50
0.630
0.305
And the I channel pixel level confidence map P. /CConn [0] x Ceon/(L] x Coons [2]. Overall, in
our training process for 3D Gaussians, the loss functions were defined as
LyDaS = Ie (Pe @ ||C - Cl11 SSIM(C,
(8)
where SSIM is the Structural Similarity Index and @ is Hadamard's 's product, Ie is the image-level confidence map and C' is the real pixel value.
5 Experiments
5.1
3DGS-Enhamne Dataset
Given that the enhancement of 3DGS representations a new task, we create dataset to simulate various artifacts of the 3DGS representations. This dataset also serves benchmark for evaluating the performance of few-shot NVS methods. Existing few-shot NVS algorithms [44, 19] primarily focus on face-forward evaluations [25], where the test views have significant overlap with the input views. However, this evaluation method is not suitable for large- scale unbounded outdoor scenes. Therefore, we propose dataset processing strategy that allows us to post-procced any existing multi-vies dataset to generate a large number of training image pairs that include typical artifacts caused by few-shot NVS.
comprehensive
More specifically, for each scene, we have n views Tuain = (1, I2,..... In which serve as the input for high-quality 3DGS model. We uniformly sample small number m of views Jow from Auain which serve as the input for the low-quality 3DGS model. By linearly fitting the high- quality camera poses p!rain {p P2 p2....... ................................ P pifiin) we randomly sample camera trajectory pf pfender = [ppender prender ...... low-quality 3DGS models. This creates a set of high-quall hugh-tualtty and low-quality image pairs used for the training of our video diffusion model.
... prnder)) on p. and render the image pairs using both high-quality and
We apply this dataset processing strategy to DL3DD [21], large-scale outdoor dataset containing 10K scenes. We randomly select 130 scenes from the original DL3DV dataset and form more than 150,000 image pairs. We randomly select another 20 scenes from DL3DV form the test sets, evaluating the corss-scene capability of our method. More implementation details of the method can be found in the supplementary material.
5.2 Comparison with State-of-the-Arts
The quantitative and gualitative results the DL3DV test set with 3 6 and input views shown in Table and Figure 4. Our approach outperforms all the other baselines in PSNR, SSIM, and LPIPS scores. NeRF-based methods including Mip-NeRF [1] and FreeNeRF [44] produce blurry novel views due to smoothing inconsistencies. In contrast, 3DGS [18] generates elongated elliptical artifacts due to local minima convergence. DNGuassian [19] reduces artifacts with depth regularization but results in blurry and noisy novel views.
The first example in Figure 4 demonstrates preserving view consistency. By interpolating input views using video diffusion model, we incorporate more information while enrusing high view consistency, enabling high-quality novel
capability to remove artifacts while

---

MipNerf [1]
FreeNert [44]
3DGS (18]
DNGaussian [9]
Ours
Ground -ruuth
MipNerf [1]
FreeNerf (44)
3DGS (18)
DNGaussian [19]
Ours
Ground-trut
MipNerf [1]
FreeNerf [44]
3DGS [18]
DNGaussian [19]
Ours
Ground-trin truth
Figure 4: A visual comparison of rendered images scenes from DL3DV [21] test set with the 3-view setting.
views and avoiding local minima. The second example highlights recovering high-frequency details. Our dataset processing strategy and video diffusion model enable an understand of strong multi-view prior across various scenes. As a result, very challenging cases such the trees can be restored with sharp details. In summary, comparisons with baseline methods demonstrate our approach's 's potential to significantly improve the unbounded 3DGS representations, synthesizing high-fidelity novel views for open environments.
's advantage in
To demonstrate the generalizability of our method for out-of-distributiin dataset, we train the methods on the DL3DV-10K dataset [21] and test them on the Mip-NcRF360 dataset [2]. The results, as

---

3DGS (PSNR: 16.15)
Ours (PSNR: 22.32)
Ground-truie
3DGS (PSNR: 13.90)
Ours (PSNR: 22.32)
Ground-truth
Figure 5: A visual comparison of cross-dataset generalization ability, where the methods trained on the DL3DV-IOK dataset [21] and tested on the Mip-NeRF360 dataset [2].
summarized in Table and Fig 5, show that our method outperforms the baseline approaches, highlighting its remarkable generalization capabilities in unbounded environments.
Table 2: A quantitative comparison of methods the unseen Mip-NeRF360 dataset [2].
views
views SSIM
Method
PSNR f SSIM
LPIPS
PSNR t
LPIPS
Mip-NeRF360 (all test scenes)
Mip-NeRF
13.08
0.159
0.637
13.73
0.189
0.628
RegNeRF
12.69
0.175
0.660
13.73
0.193
0.629
FreeNeRF
12.56
0.182
0.646
13.20
0.198
0.635
3DGS
11.53
0.144
0.651
12.65
0.187
0.607
DNGaussian
11.81
0.208
0.689
12.51
0.228
0.683
SaGG--Ehancer (ours)
13.96
0.260
0.570
16.22
0.399
0.454
5.3
Ablation Study
Real image reference views. Table 3 shows the quantitative comparisons of different compo- nents in 3DGS-Enhancer framework. The video diffusion model provides strong multi-view priors. However, due to its native restrictions, we directly feed the original input views into the 3DGS fine-tuning process. This results in more reliable and niewcconsistent information from the input domain to facilitate 3DGS fine-ture mne-tuning, demonstrated by the "Real image" in Table 3.
Confidence aware reweighting. Distant views are less likely to cause artifacts, so we normalize their distance reference views between [0, 1], giving higher confidence of video diffusion results to farther viewpoints. This strategy is denoted by "Image confidence" in Table 3. Pixel-level confidence, denoted by "Pixel confidence" in Table is based on the density of small-volume Gaussians in well-reconstructed areas, using a color rendering pipeline to calculate volumes. Both pixel and image-level confidence strategies improve results individually, and their combination yields the best performance.
Video diffusion and STD. Figure 6 visualizes the effects of video diffusion and STD module, respectively. Video diffusion removes most of the artifacts, and STD module enhances fine-grained and high-frequency textures, resulting in more vivid novel view renderings, which are closer the ground truth. Table 4 shows the improvment for each modules.

---

Table 3: An ablation study of the four modules of our 3DGS-Enhance framework, where all results
are averaged across 3, 6, 9, and 12 input views on DL3DV dataset [21].
Video diffusion
Real image
Image confidence
Pixel confidence
PSNR1
SSIM
LPIPS
14.33
0.476
0.422
17.01
0.553
0.361
17.29
0.570
0.354
/
17.16
0.564
0.351
17.34
0.574
0.351
Table 4: ablation study of STD (temporal layers) and color correction module on the DL3DV test dataset with 9-view setting.
Video diffusion
STD (temporal layers)
color correction
PSNR
SSIM
I.IPS $
18.11
0.312 18.44
0.591
0.625
0.306
18.50
0.630
0.305
Input
Diffusion
STD
Ground-
Figure 6: An ablation study of the video diffusion model components in 3DGS-Enhancer framework.
6 Conclusions, Limitations, and Future Work
This paper has introduced prior from video diffusion and use trajectory interpolation method enhance unbounded 3DGS representations. By combining image and pixel-level confidence with 3DGS fine-tuning, we have achieved ofate-of-the-art performance NVS enhancement. However, our approach relies on adjacent views for continuous interpolation, it cannot be easily adapted to single-view 3D model generation. Moreover, the confidence-aware 3DGS fine-tumin to-inn strategies are relatively simple and straightforward. In the future, it is interesting to integrate confidence maps directly with the video generation model, enabling the generation of images that are more line with the real 3D world without the need for post-processing. Meanwhile, utilizing the efficient data generation capability of 3DGS construct massively scaled dataset for our video generation model presents prime opportunity to enhance the model's ' 3D consistency. This approach also facilitates the 2D models to understand the 3D world directly from 2D images without additional geometric constraints. Regarding the social impact, the goal of this work is to advance the fields of 3D reconstruction and NVS. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
unified framework that applies view- coo-coonssttency
Acknowledgement
The authors gratefully acknowledge the Clemson University Palmetto Cluster for providing the nigh-performance computing resources that supported the computations of this work.
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Bartin-ruualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5855-5864, 2021.
[2] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
[3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arX1v:2311.55127, 2023,
[5] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19457-19467, 2024.
[6] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference computer vision, pages 333-350. Springer, 2022.
[7] Zilong Chen, Yikai Wang, Feng Wang. Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.0677 06440000678,, 2024.
[8] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes. JEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2021.
[9] Sara Fridovich- Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[10] Guangcong, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[11] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference Learning Representations, 2024.
[12] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[13] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[15] Xudong Huang, Wei Li, Jie Hu, Hanting Chen, and Yunhe Wang. Refsr-nerf: Towards high fidelity and super resolution view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 8244 8253, June 2023.
[16] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEFF//VV International Conference on Computer Vision (ICCV), pages 5885-5894, October 2021.

---

[17] Yifan Jiang, Peter Hedman, Ben Mildenhall, Dejia Xu, Jonathan T. Barron, Zhangyang Wang. and Tianfan Xue. Alignerf: High-fidelity neural radiance fields via alignment aware training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 46-55, June 2023.
[18] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[19] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-vine seare--view/. 3d gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20775-20785, 2024.
[20] Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Lanqing Hong, Guo Zhou, Hua Yao, Dit- Yan Yeung, Huchuan Lu, and Xu Jia. Trackdiffusion: Tracklet-conditioned video generation via diffusion models, 2024.
[21] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning- based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160-221690 2024.
[22] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1ero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2023.
[23] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang, Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision, pages 210-227. Springer, 2022.
[24] Luke Helas-Kyriazz Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. International Conference Machine Learning, 2024, 2024.
[25] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[27] Muhammad Husnain Mubarik, Ramakrishna Kanungo, Tobias Zirr, and Rakesh Kumar, Hard- ware acceleration of neural graphics. In Proceedings of the 50th Annual International Sympo- sium on Computer Architecture, pages 1-12, 2023.
[28] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall. Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2022.
Recognition, pages
[29] Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: Learning where to see with uncertainty estimation. In European Conference on Computer Vision, pages 230-246. Springer, 2022.
[30] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
12

---

[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[33] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. CVPR, 2024, 2023.
[34] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. base model, 2023.
a single image to consistent multi-view diffusion
[35] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations, 2024.
[36] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 2.0.//,330/6663, 2023.
[37] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitrii Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from single image using latent video diffusion. arXiv, 2024.
[38] Chen Wang, Xian Wu, Yuan-Chen Guo, Song-Hai Zhang, Yu- Yu-Wing Tai, and Shi-Min Hu. Nerf-sr: High quality neural radiance fields using supersampling. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6445-6454, 2022.
[39] Jinny Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. In arXiv preprint arXiv:2305.07015 2023.
[40] Frederik Warburg, Ethan Weber, Matthew Tancik, Aleksander Holynski, and Angjoo Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 18120-18130, October 2023.
[41] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan Barron, Ben Poole, al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21551-21561, 2024.
[42] Jamie Wynn and Daniyar Turmukhambetov. DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models. In CVPR, 2023.
[43] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang. Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. ICCV, 2023.
[44] Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2023.
[45] Xi Yang, Chenhang He, Jianqi Ma, and Lei Zhang. Motion-guided latent diffusion for temporally consistent real-world video super-resolution. In European Conference on Computer Vision, pages 224-242. Springer, 2025.
[46] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In CVPR, 2024.
[47] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In arXiv, 2021.
13

---

[48] Shangchen Zhou, Peiqing Yang. Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscale- a-video: Temporal-consisten diffusion model for real-world video super-resolution. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2535 2545, 2024.
[49] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
[50] Matthias Zwicker, Hanspeter Pfister, Jeroen Baar, and Markus Gross. Surface splatting. Pro- ceedings of the ACM SIGGRAPH Conference on Computer Graphics, 2001, 08 2001.
14

---

8 Appendix
8.1 Details of 3DGS Enhancement Dataset
For our 3DGS Enhancement Dataset, constructed based on DL3DV, we randomly select 120 scenes to create the training set for our video diffusion model and 30 scenes as the test set. By following previous works, we use the standard train/test split, selecting every 8th frame of the remaining frames for evaluation.
To create image pairs simulating the artifacts due to the lack of input views in novel view synthesis problem, we render the image pairs from pairs of low-high quality 3DGS models. Specifically, the input views for the high-quality model consist of all images in the original dataset, while the inputs for the low-quality model are a subset uniformly sampled from the original dataset. To add more complexity. we sample the subset according to a certain number (e.g., 3, 6, 9) or certain ratio (e.g., 5%). With the aim to fully capture the distribution of artifacts created by the sparse input views and train the video diffusion model with smoother inputs, we propose a heuristic trajectory fitting algorithm, as shown Figure 7, proving a sequence of cameras by interpolating the low or high-quality model's 's input views. Specifically, if the original camera trajectories are smooth and simple, such those of DL3DV, we use the high-quality input views as the reference to fit the trajectories. For complex trajectories, such those in Mip-NeRF 360, we use the low-quality input to avoid significantly poor rendering results, which would lead to unreasonable artifact distributions. As result, we render large number of image pairs with and without artifacts, as shown in Figure 8, at a resolution of 512 x 512, leading to powerful vidco diffusion priors with high view consistency and photo-realism.
Input:
Input: 6
Input: 9
Figure 7: The fitting trajectories under different number of input views.
8.1.1 Training details
Our video diffusion model includes a pre-trained VAE to encode an image sequence into latent sequence and decode the latent sequence back into the image sequence. It also includes U-Net with learnable temporal layers, which employs cross fross-frame attention modules and 3D CNN modules to ensure framseconsistent outputs. The input of video diffusion model is image sequence segment that includes 25 images with different sample steps from the image sequences rendered from the low-quality 3DGS model. The first and the last frames in this segment are replaced with images rendered from the high-quality 3DGS model. During fine-tuning, our video diffusion model is conditioned on these image sequence segments and trained to synthesize the corresponding segments rendered from the high-quality 3DGS model.
Our video diffusion model is fine-tuned with learning rate 0.0001, incorporating 500 steps for warm-up, followed by total of 80,000 training steps. The batch size is set to I in each GPU, where each batch consisted of 25 images at 512x512 resolution. To optimize the training process, the Adam optimizer is employed. Additionally. a dropout rate of 0.1 is applied to the conditions between the first and last frames and the training process utilize CFG (classifier-fre free guidance) train the diffusion model. The training is conducted on 2 NVIDIA A A100-80G GPUs over 3 days. The STD fine-tuned with a learning rate 0.0005 and 50,000 training steps. The batch size set to I in each GPU, where each batch consists of 5 images at 51 312x112 resolution, but for inference, it was increased 25. The fine-tuning process conducted on 2 NVIDIA 1100-8G GPUs days. The entire pipeline's inference and training speeds were evaluated and are presented in Table 5.
15

---

Input: 3
Ground-trutt
Input:
Ground-trut -rund
Input: 9
Ground-truth
Input:
Ground-truth
Figure 8: The low and high quality image pairs created in our 3DGS Enhancement dataset.
Table 5: A comparison of per-scene training time and rendering FPS between methods. For our method, the LQ-3DGS reconstruction takes 10.5 minutes, stable video diffusion inference for 50 novel views requires 2.0 minutes, and the HQ-3DGS reconstruction takes 12.0 minutes.
Method
Per-scene training time Rendering FPS 1
Mip-NeRF
10.7h
0.09
RegNeRF
2.5h
0.09
FreeNeRF
3,8h
0.09
3DGS
10.5min
100
DNGaussian
3.3min
100
3DGS-Ent -eeennen (ours)
24.5min
100
8.2 Details of Comparison Baselines
For the evaluation datasets, we compare against the standard 3D Gaussian Splatting [18] (which also the reconstruction pipeline used in our work), and the state-of-the-ar few-view NVS regularization
methods, including Mip-NeRF [1], FreeNeRF [44], Zip-NeRF [3], and RegNeRF [28]. We also compare to some few-shot NVS methods using generative priors including ZeroNVS [33], and ReconFusion [41].
For the evaluation of MipNeRF, FreeNeRF, RegNeRF, and DNGaussian on DL3DV and Mip-NeRF 360 dataset, we follow the original configurations and code shared by the authors. Additionally, we use random point cloud as the initialization for 3DGS, following the implementations from DNGaussian. We also decrease the batch size for RegNeRF from 4096 to 512 according to the limited computation resource.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We ensured that the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussions on limitations of this work can be found in the last section of the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have discussed all implementation details of the proposed method in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The project page has been online. The code and generated datasets will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
Moolli//giies/Codeeeee
wwwassoooollii imPolccy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpp.ccpppblliie wwtpccccpccccceeccccccccccccttlt cooollccguidess.
ceabbbmmsssoonPe ssininolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have discussed implementation details of the proposed method in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments on large-scale 3D reconstruction datasets are computationally expensive. Existing literature also did not provide error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of compute resources can found in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neurt
cttpp//ccurppp..ppppiiiccccc//ct /tc//uulcc//eee csGuidelines!
Answer:
[Yes]
Justification:
We have ensured that this research followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impacts in the last section of main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have such risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original papers that produced the code package and datasets. The licenses for re- re-packaged datasets will be provided on the dataset website.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "DVUbF3F2NC", "reviewer": "Reviewer_19Ng", "summary": "This paper is working on the problem of novel view synthesis with sparse input views. The authors present 3DGS-Enhancer to enhance the rendering quality and address 3D view consistency problem using 2D video diffusion priors. The experimental results show that this work has achieved state-of-the-art performance in novel view synthesis enhancement.", "questions": "The experiments are conducted almost on the self-introduced dataset. Can 3DGS-Enhancer also surpasses the other baselines on more commonly used outdoor unbounded datasets (e.g. Mip-NeRF 360 dataset)?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Writing quality is great, clean, and easy to follow.\n2. To tackle the sparse view reconstruction problem, the idea of fine-tuning the original 3DGS representation instead of incorporating additional geometric constraints is novel and promising.\n3. A new dataset is introduced in this work, which is useful for this research direction and the community.\n4. The idea of leveraging video LDM for sparse view reconstruction in unbounded outdoor scenes is novel and effective.", "weaknesses": "Runtime is not discussed in this paper. I’m wondering how long it will take for one scene compared to the other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. Runtime is not discussed in this paper. I’m wondering how long it will take for one scene compared to the other baselines.**\n\nAs shown in the below table, we estimate the per-scene runtime and rendering FPS of different methods on the DL3DV test set (3 views) with one NVIDIA A100 GPU. Our method takes 24.5 minutes for one scene, including 10.5 minutes for low-quality 3DGS training, 2.0 minutes for SVD inference, and 12.0 minutes for high-quality 3DGS training. Thank you for the helpful suggestion, we will include the efficiency analysis in the supplementary material.\n\n| Method           | Per-scene runtime &darr; | Rendering FPS &uarr; |\n|------------------|--------|:------:|\n| Mip-NeRF         | 10.7h         | 0.09       |\n| RegNeRF          | 2.5h           | 0.09       |\n| FreeNeRF         | 3.8h           | 0.09       |\n| 3DGS             | 10.5min        | 100      |\n| DNGaussian       | 3.3min        | 100      |\n| 3DGS-Enhancer (ours) |24.5min (LQ-3DGS 10.5min, SVD 2.0min, HQ-3DGS 12.0min)            |100            |\n|\n\n**Q1. The experiments are conducted almost on the self-introduced dataset. Can 3DGS-Enhancer also surpasses the other baselines on more commonly used outdoor unbounded datasets (e.g. Mip-NeRF 360 dataset)?**\n\nThank you for the constructive feedback, we conducted experiments on the Mip-NeRF 360 dataset, and the results are detailed below and in the attached PDF file in Author Rebuttal section. Our model is trained on the DL3DV dataset, and evaluated on all 9 scenes of Mip-NeRF 360 dataset. Our method significantly outperforms the baselines, although it has not been trained on this complex dataset. This indicates that our method has remarkable *cross-dataset generalization capability* for unbounded environments, thanks to the powerful 2D video diffusion priors.\n\n| Method           | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |\n|------------------|-------------|-------------|--------------|-------------|-------------|--------------|\n|                  |                          | 6 views                   |             |             | 9 views  |                 |\n| Mip-NeRF         | 13.08       | 0.159       | 0.637        | 13.73       | 0.189       | 0.628        |\n| RegNeRF          | 12.69       | 0.175       | 0.660        | 13.73       | 0.193       | 0.629        |\n| FreeNeRF         | 12.56       | 0.182       | 0.646        | 13.20       | 0.198       | 0.635        |\n| 3DGS             | 11.53       |0.144        |0.651         |12.65        |0.187        |0.607        |\n| DNGaussian       | 11.81       | 0.208       | 0.689        | 12.51       | 0.228       | 0.683        |\n| 3DGS-Enhancer (ours) |**13.96**         |**0.260**         |**0.570**          |**16.22**         |**0.399**        |**0.454**       |\n|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1exH7gWXUs", "reviewer": "Reviewer_KbVQ", "summary": "This paper presents a novel pipeline aimed at enhancing the quality of 3D Gaussian splatting (3DGS) representations, especially in scenarios with sparse input views. They propose a novel framework, 3DGS-Enhancer, that leverages video LDMs for generating high-quality and 3D-consistent images. Moreover, to mitigate the artifacts caused by the temporal inconsistency, they introduce a spatial-temporal decoder and fine-tuning strategy for the 3DGS optimization process. extensive experiments are conducted to demonstrate their superior reconstruction performance on large-scale datasets of unbounded scenes.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Applying video models to scene reconstruction is a novel idea and has achieved excellent results.\n2. To mitigate the negative impact of inconsistent generated images, this paper proposes improvements to the decoder and defines a confidence measure to enhance the corresponding 3D Gaussian Splatting optimization process.", "weaknesses": "1. The writing of this paper is not very clear, and some typo errors exist in the submission, for instance, line147, it should be $ \\{ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i \\}$; line216: it should be $I_c$, consistent to the equation (8).\n\n2. Training details are missing in the paper, more details about the video diffusion model and fine-tuning process should be provided.\n\n3. More datasets, such as MipNerf360, should be evaluated to verify the effectiveness and robustness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.The writing of this paper is not very clear, and some typo errors exist in the submission, for instance, line147, it should be ${ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i }$; line216: it should be $I_c$, consistent to the equation (8).**\n\nThank you for the helpful comment. You are correct, line147 should be ${ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i }$. In line216, $I_w$ should be $I_c$. We will carefully revise the writing and typos in the camera ready version. \n\n**2. Training details are missing in the paper, more details about the video diffusion model and fine-tuning process should be provided.**\n\nOur video diffusion model includes a pre-trained VAE to encode an image sequence into a latent sequence and decode the latent sequence back into the image sequence. It also includes a U-Net with learnable temporal layers, which employs cross-frame attention modules and 3D CNN modules to ensure frame-consistent outputs.\nThe input of video diffusion model is a image sequence segment that includes 25 images with different sample steps from the image sequences rendered from the low-quality 3DGS model. The first and the last frames in this segment are replaced with images rendered from the high-quality 3DGS model. During fine-tuning, our video diffusion model is conditioned on these image sequence segments and trained to synthesize the corresponding segments rendered from the high-quality 3DGS model. \n\nOur video diffusion model is fine-tuned with a learning rate of 0.0001, incorporating 500 steps for warm-up, followed by a total of 80,000 training steps. The batch size is set to 1 in each GPU, where each batch consisted of 25 images at 512x512 resolution. To optimize the training process, the Adam optimizer is employed. Additionally, a dropout rate of 0.1 is applied to the conditions between the first and last frames and the training process utilize CFG (classifier-free guidance) to train the diffusion model. The training is conducted on 2 NVIDIA A100-80G GPUs over 3 days. The STD is fine-tuned with a learning rate of 0.0005 and 50,000 training steps. The batch size is set to 1 in each GPU, where each batch consists of 5 images at 512x512 resolution. The fine-tuning process is conducted on 2 NVIDIA A100-80G GPUs in 2 days. Thank you for the valuable feedback, more details of the models and fine-tuning process will be included in the supplementary material.\n\n\n**3. More datasets, such as MipNerf360, should be evaluated to verify the effectiveness and robustness of the proposed method.**\n\nThank you for the constructive feedback, we conducted experiments on the Mip-NeRF 360 dataset, and the results are detailed below and in the attached PDF file in Author Rebuttal section. Our model is trained on the DL3DV dataset, and evaluated on all 9 scenes of Mip-NeRF 360 dataset. Our method significantly outperforms the baselines, although it has not been trained on this complex dataset. This indicates that our method has remarkable *cross-dataset generalization capability* for unbounded environments, thanks to the powerful 2D video diffusion priors.\n\n\n| Method           | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |\n|------------------|-------------|-------------|--------------|-------------|-------------|--------------|\n|                  |                         | 6 views                   |             |             | 9 views                   ||\n| Mip-NeRF         | 13.08       | 0.159       | 0.637        | 13.73       | 0.189       | 0.628        |\n| RegNeRF          | 12.69       | 0.175       | 0.660        | 13.73       | 0.193       | 0.629        |\n| FreeNeRF         | 12.56       | 0.182       | 0.646        | 13.20       | 0.198       | 0.635        |\n| 3DGS             | 11.53       |0.144        |0.651         |12.65        |0.187        |0.607        |\n| DNGaussian       | 11.81       | 0.208       | 0.689        | 12.51       | 0.228       | 0.683        |\n| 3DGS-Enhancer (ours) |**13.96**         |**0.260**         |**0.570**          |**16.22**         |**0.399**        |**0.454**       |\n|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FMdHdiGavz", "reviewer": "Reviewer_RSAr", "summary": "This paper utilizes a video diffusion model to enhance 3DGS rendering results. It proposes a 3DGS-Enhancer pipeline to reformulate 3D-consistent image restoration tasks and leverage it to generate high-quality and 3D-consistent images. They have enough experiments to prove the soundness of their method.", "questions": "Video Diffusion prior may have motion information of moving objects in the scene. How do we eliminate that effect in enhancing 3DGS?", "limitations": "The limitation is definitely from the pre-trained video diffusion model whether the data is more like novel view synthesis data (static scene) or a sequence of video frames (moving objects).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "They didn't use depth information but video diffusion prior to do the task, which is interesting.\nThey also have the spatial-temporal decoder which solves the temporal consistency problem.\nFor fine-tuning, they add confidence constraints on both pixel level and image level to enhance the result.\nExtensive comparison with state-of-the-art is done, and the results seem robust.", "weaknesses": "They can provide more comparisons on ablating the temporal consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. They can provide more comparisons on ablating the temporal consistency.**\n\n\nAs shown in Fig. 1 of the paper, the images rendered by 3DGS model trained on sparse view inputs often contain significant artifacts or blank areas. We observed that a image diffusion model trained solely on these images is unable to restore high quality images without additional information, and the synthesized multi-view images are often inconsistent even for adjacent views. \nIn addition, for the temporal layers in STD, we conducted an ablation study to verify their effectiveness on DL3DV dataset with 9 views, and the result is detailed below. Temporal layers help STD produce more consistent image outputs to improve final reconstruction performance. Thank you for the helpful suggestion, we will provide more ablation studies on the temporal consistency in the supplementary material.\n\n| Temporal layers in STD | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |\n|:------:|---------|-------|----------|\n| $\\times$ | 18.11 | 0.591 | 0.312 |\n| $\\checkmark$ | **18.50** | **0.630** | **0.305** |\n|\n\n\n**Q1. Video Diffusion prior may have motion information of moving objects in the scene. How do we eliminate that effect in enhancing 3DGS?**\n\nWe took two steps to mitigate this effect of moving objects in video diffusion prior. First, we created a paired novel view synthesis dataset from DL3DV to fine-tune the video diffusion model from an image-to-video model to a video-to-video model, ensuring that it outputs images more akin to the novel view synthesis of static scenes. Second, we introduced images rendered from a low-quality 3D Gaussian model as the conditional inputs for video diffusion. Using these images as guidance helps the video diffusion model generate images suitable for 3D static scene reconstruction. \n\n**L1. The limitation is definitely from the pre-trained video diffusion model whether the data is more like novel view synthesis data (static scene) or a sequence of video frames (moving objects).**\n\nYour statement of the limitation of this work is correct. Since our approach is based on video diffusion, it is indeed limited by the performance of the pre-trained video diffusion model (e.g., SVD). However, our method will also benefit from advances in video diffusion models, since better pre-trained video diffusion models or larger datasets can enhance the potential of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYP3P6Nzyu", "reviewer": "Reviewer_LSGq", "summary": "The paper tackles the problem of enhancing 3GS novel views which are far from existing viewpoints (encountered in sparse-view settings), due to insufficient information in under-sampled areas. This work enhances the low quality views using a video diffusion model to maintain multi view and temporal consistency. The authors reformulate the task of 3D-consistent image restoration as video restoration which is again reformulated as a video interpolation tasks between frames. The high quality frames along with the restored images are then used to fine tune the 3DGS. The authors also propose a modified spatial temporal decoder (STD) to reduce artifacts and introduce confidence awareness at both image level (based on distance of novel view from reference) and pixel level (information gain) to minimize the negative impact of generated images.", "questions": "1. Authors describe in L184-186 \"Despite this significant enhancement in the quality of the rendered views, we propose to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive to slight inaccuracies in the restored views.\" I would appreciate a more thorough discussion on this.\n\n2. \"modified baselines denoted with ∗ taken from [36]\". However, no baseline seems to be marked with * in table 1. This is possibly an oversight and the authors should correct it. \n\n3. What's the adversarial loss described in L179 for STD? \n\n4. On what data is the Temporal Denosing U-Net fine tuned on?\n\nSuggestions:\nFigure 2 can be more interpretable if the right part shows the output of the enhanced views (input from left part); currently there is no 1:1 parity", "limitations": "Limitations are adequately addressed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well motivated- enhancing low quality 3DGS rendering results for camera viewpoints which are far away\n* Reformulates the 3D-consistent image restoration task in 3DGS as temporally consistent video generation to leverage video diffusion models \n* The results look quite impressive and have been tested on a variety of scenes\n* can be applied to any existing 3DGS models and more generalized scenes\n* introduces a dataset for evaluation of artifacts in 3DGS (will this dataset be open sourced?)\n* Sound experimental setup and adequate comparisons with other methods", "weaknesses": "* Related work appears to be thin and some of these paragraphs can be more descriptive\n* Method relies on adjacent views for continuous interpolation\n* Choice for STD should be quantitatively ablated. Individual parts in STD like color correction should be ablated as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**S1. Will this dataset be open sourced?**\n\nOur 3DGS Enhancement dataset is generated based on the publicly available DL3DV dataset. We will provide the complete dataset generation code to ensure the community can quickly reproduce our results and explore new research opportunities.\n\n\n**W1. Related work appears to be thin and some of these paragraphs can be more descriptive.**\n\nThank you for the constructive feedback. We will refine our Related Work section in the camera ready version. Specifically, NVS methods dealing with sparse-view inputs will be categorized under \"Few-shot novel view synthesis\", while more general enhancement efforts, such as super-resolution and artifact removal, will be categorized under \"Radiance fields enhancement\". More literature under these two categories will be added. These revisions would make the Related Work more descriptive and organized.\n\n\n**W2. Method relies on adjacent views for continuous interpolation.**\n\nOur method performs well when there is a significant overlap between two views. Since we use images rendered from a low-quality 3D Gaussian model as conditions for video diffusion model, even if two views are not close to each other, our method can still achieve decent results when these conditional images contain certain structure or texture information. On the other hand, our method can generate free-trajectory conditional images for continuous interpolation, offering more flexibility in addition to linear interpolation between adjacent views.\n\n\n**W3. Choice for STD should be quantitatively ablated. Individual parts in STD like color correction should be ablated as well.**\n\nThank you for the valuable suggestion, we conducted an ablation study on two STD components, the color correction and the temporal layers, on the DL3DV dataset (9 views). Results are detailed below. Both two STD components contribute to performance improvement, while temporal layers contribute more as they ensure STD produces consistent image outputs to improve final reconstruction performance. We will include the ablation study on STD components in the supplementary material.\n\n| Video diffusion | STD (temporal layers) | STD (color correction) | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |\n|:------:|:------:|:------:|-|-|-|\n| $\\checkmark$ | $\\times$ | $\\times$ | 18.11 | 0.591 | 0.312 |\n| $\\checkmark$ | $\\checkmark$ | $\\times$ | 18.44 | 0.625 | 0.306 |\n| $\\checkmark$ | $\\checkmark$ | $\\checkmark$ | **18.50** | **0.630** | **0.305** |\n|\n\n**Q1. Authors describe in L184-186 \"Despite this significant enhancement in the quality of the rendered views, we propose to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive to slight inaccuracies in the restored views.\" I would appreciate a more thorough discussion on this.**\n\nIn the 3DGS training process, we observed that the 3DGS model can fit the training images very well, even if the training images contain noises. Unfortunately, images generated by the diffusion model inevitably introduce various noises and inconsistencies, resulting in a performance decrease. \n\nTo mitigate the negative impact of these generated images for the 3DGS model, a straightforward approach is assigning larger weights to real images and generated images of higher quality. From our observations, the generated images that are closer to the reference views tend to have higher quality. Motivated by this, we introduce an image-level weighting scheme. Additionally, we observed that smaller 3D Gaussian volumes are more likely to be well-reconstructed. To prevent well-reconstructed regions from being adversely affected by the generated images, we introduce pixel-level confidence scores which are calculated based on the volume of 3D Gaussians. From Table 2 of the paper, we can see the improvement after incorporating image-level confidence and pixel-level confidence. Thank you for the insightful feedback, we will add more discussions in the camera ready version. \n\n\n**Q2. \"modified baselines denoted with ∗ taken from [36]\". However, no baseline seems to be marked with * in table 1. This is possibly an oversight and the authors should correct it.**\n\nThank you for the helpful comment. This is an oversight, and we will remove it in the camera ready version. \n\n**Q3. What's the adversarial loss described in L179 for STD?**\n\nThe adversarial loss employed in Eq. 6 is the standard adversarial loss, which trains a discriminator $D$ to discriminate between real image $\\hat{I^g}$ and fake image $I^g$, as \n$\nL_{GAN} = E_{\\hat{I^g}} \\left[\\log D(\\hat{I^g})\\right] + E_{I^g} \\left[1 - \\log D(I^g)\\right].\n$\nThank you for the valuable comment, we will add it into the camera ready version.\n\n\n\n**Q4. On what data is the Temporal Denosing U-Net fine tuned on?**\n\nOur Temporal Denosing U-Net is fine-tuned on the 3DGS Enhancement Dataset created in this work, which includes a large number of image pairs generated from corresponding high-quality and low-quality 3DGS models. More details of the dataset can be found in Section 5.1 of the paper. \n\n\n**Suggestions: Figure 2 can be more interpretable if the right part shows the output of the enhanced views (input from left part); currently there is no 1:1 parity**\n\nThank you for the helpful suggestion, we will revise Figure 2 of the paper to ensure its right part shows output images corresponding to the input images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
The authors thank all reviewers for the careful review and constructive feedback. We are encouraged that all four reviewers appreciate the novel idea and excellent experimental results of this work. We address all the raised concerns in corresponding reviewer's rebuttal section. The code and dataset will be publicly available.

**Mip-NeRF 360 Result (Cross-dataset Generalization Experiment):**

Thanks to constructive feedback from Reviewers 19Ng and KbVQ, we conducted a cross-dataset generalization experiment on the Mip-NeRF 360 dataset, and the results are shown in the below table and attached PDF file. Our video diffusion and STD models are trained on the DL3DV dataset, and evaluated on all 9 scenes of Mip-NeRF 360 dataset. Results show that our method significantly outperforms the baselines, indicating it has remarkable cross-dataset generalization capability for unbounded environments, thanks to the powerful 2D video diffusion priors.

| Method           | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |
|------------------|-------------|-------------|--------------|-------------|-------------|--------------|
|                  |             |              6 views                   |             |             | 9 views                   ||
| Mip-NeRF         | 13.08       | 0.159       | 0.637        | 13.73       | 0.189       | 0.628        |
| RegNeRF          | 12.69       | 0.175       | 0.660        | 13.73       | 0.193       | 0.629        |
| FreeNeRF         | 12.56       | 0.182       | 0.646        | 13.20       | 0.198       | 0.635        |
| 3DGS             | 11.53       |0.144        |0.651         |12.65        |0.187        |0.607        |
| DNGaussian       | 11.81       | 0.208       | 0.689        | 12.51       | 0.228       | 0.683        |
| 3DGS-Enhancer (ours) |**13.96**         |**0.260**         |**0.570**          |**16.22**         |**0.399**        |**0.454**       |
|

Meta-review: All the reviewers agree to accept this paper. It presents a novel method to enhance the quality of 3DGS representations by leveraging 2D video diffusion priors. Additionally, the paper introduces a spatial-temporal decoder that effectively addresses the issue of temporal consistency. The experimental results both qualitative and quantitive show significant improvements compared to the baselines. All the reviewers' concerns have been addressed.
Decision: Accept (spotlight)