Title: 3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D model enhancement

3D Guassian splatting

novel view synthesis

diffusion model

image restoration
Abstract: Novel-view synthesis aims to generate novel views of a scene from multiple input
images or videos, and recent advancements like 3D Gaussian splatting (3DGS)
have achieved notable success in producing photorealistic renderings with efficient
pipelines. However, generating high-quality novel views under challenging settings,
such as sparse input views, remains difficult due to insufficient information in
under-sampled areas, often resulting in noticeable artifacts. This paper presents
3DGS-Enhancer, a novel pipeline for enhancing the representation quality of
3DGS representations. We leverage 2D video diffusion priors to address the
challenging 3D view consistency problem, reformulating it as achieving temporal
consistency within a video generation process. 3DGS-Enhancer restores view-
consistent latent features of rendered novel views and integrates them with the
input views through a spatial-temporal decoder. The enhanced views are then
used to fine-tune the initial 3DGS model, significantly improving its rendering
performance. Extensive experiments on large-scale datasets of unbounded scenes
demonstrate that 3DGS-Enhancer yields superior reconstruction performance and
high-fidelity rendering results compared to state-of-the-art methods. The project
webpage is https://xiliu8006.github.io/3DGS-Enhancer-project.

3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

Abstract

Novel-view synthesis aims to generate novel views of a scene from multiple input images or videos, and recent advancements like 3D Gaussian splatting (3DGS) have achieved notable success in producing photorealistic renderings with efficient pipelines. However, generating high-quality novel views under challenging settings, such as sparse input views, remains difficult due to insufficient information in under-sampled areas, often resulting in noticeable artifacts. This paper presents 3DGS-Enhancer, a novel pipeline for enhancing the representation quality of 3DGS representations. We leverage 2D video diffusion priors to address the challenging 3D view consistency problem, reformulating it as achieving temporal consistency within a video generation process. 3DGS-Enhancer restores viewconsistent latent features of rendered novel views and integrates them with the input views through a spatial-temporal decoder. The enhanced views are then used to fine-tune the initial 3DGS model, significantly improving its rendering performance. Extensive experiments on large-scale datasets of unbounded scenes demonstrate that 3DGS-Enhancer yields superior reconstruction performance and high-fidelity rendering results compared to state-of-the-art methods. The project webpage is https://xiliu8006.github.io/3DGS-Enhancer-project.

1 Introduction

Novel-view synthesis (NVS) has decades of history in computer vision and graphics communities, aiming to generate views of a scene from multiple input images or videos. Recently, 3D Gaussian splatting (3DGS) [18] has excelled in producing photorealistic renderings with a highly efficient rendering pipeline. However, rendering high-quality novel views far from existing viewpoints remains very challenging, as often encountered in sparse-view settings, due to insufficient information in under-sampled areas. As shown in Figure 1, noticeable ellipsoid-like and hollow artifacts manifest when there are only three input views. Due to these common low-quality rendering results in practice, it is essential to enhance 3DGS to ensure its viability for real-world applications.

To our knowledge, few prior studies have specifically focused on enhancement methods aimed at improving the rendering quality of NVS. Most existing enhancement work for NVS [19, 44] focuses on incorporating additional geometric constraints such as depth and normal into the 3D reconstruction process to fulfill the gap between the observed and unobserved regions. For example, DNGaussian [19] applies a hard-and-soft depth regularization to the geometry of radiance fields. However, these methods heavily rely on the effectiveness of additional constraint and are often sensitive to noises. Another line of work leverages generative priors to regularize the NVS pipeline. For instance, ReconFusion [41] enhances Neural Radiance Fields (NeRFs) [26] by synthesising the geometry and texture for the unobserved regions. Although it can generate photo-realistic novel views, the view consistency is still challenging when the generated views are far away from the input ones.

In this work, we exploit the 2D generative priors, e.g. , the latent diffusion models (LDMs) [32], for 3DGS representation enhancement. LDM has demonstrated powerful and robust generation capabilities in various image generation [32] and restoration tasks [43]. Nevertheless, the main challenge lies in the poor 3D view consistency among generated 2D images, which significantly hinders the 3DGS training process that requires highly precise view consistency. Although some efforts have been made, such as the Score Distillation Sampling (SDS) loss [30] that distills the optimization objective of a pre-trained diffusion model, it fails to generate the 3D representation allowing rendering high-fidelity images

Motivated by the analogy of the visual consistency between multi-view images and the temporal consistency between video frames, we propose to reformulate the challenging 3D consistency problem as an easier task of achieving temporal consistency within video generation, so we can leverage the powerful video diffusion models for restoring high-quality and view-consistent images. We propose a novel 3DGS enhancement pipeline, dubbed 3DGS-Enhancer. The core of 3DGS-Enhancer is a video LDM consisting of an image encoder that encodes latent features of rendered views, a video-based diffusion model that restores temporally consistent latent features, and a spatial-temporal decoder that effectively integrates the high-quality information in original rendered images with the restored latent features. The initial 3DGS model will be finetuned by these enhanced views to improve its rendering performance. The proposed 3DGS-Enhancer can be trajectory-free to reconstruct the unbound scenes from sparse views and generate the natural 3D representation for the invisible area between two known views. A cocurrent work V3D [7] also leverages latent video diffusion models [4] for generating object-level 3DGS models from single images. In contrast, our 3DGS-Enhancer focuses on enhancing any existing 3DGS models and thus can be applied to more generalized scenes, e.g. , the unbounded outdoor scenes.

In experiments, we generate large-scale datasets with pairs of low-quality and high-quality images on hundreds of unbounded scenes, based on DL3DV [21], for comprehensively evaluating the novelly investigated 3DGS enhancement problem. Empirical results demonstrate that the proposed 3DGS-Enhancer method achieves superior reconstruction performance on various challenging scenes, yielding more distinct and vivid rendering results. The code and the generated dataset will be publicly available. The contributions of this paper are summarized as follows.

To the best of our knowledge, this is the first work to tackle the problem of enhancing lowquality 3DGS rendering results, an issue that widely exists in practical 3DGS applications.

We propose a novel pipeline 3DGS-Enhancer that addresses the 3DGS enhancement problem. 3DGS-Enhancer reformulates the 3D-consistent image restoration task as temporally consistent video generation, such that powerful video LDMs can be leveraged for generating both high-quality and 3D-consistent images. Novel 3DGS fine-tuning strategies are

also devised for an effective integration of the enhanced views with the original 3DGS representation.

We conduct extensive experiments on large-scale datasets of unbounded scenes to demonstrate the effectiveness of the proposed methods over existing state-of-the-art few-shot NVS methods.

2 Related Work

Radiance fields for novel view synthesis. Novel view synthesis (NVS) aims to generate unseen viewpoints from a set of input images and camera information. Radiance fields methods, like NeRFs [26], encode 3D scenes as radiance fields and use volume rendering for novel views, achieving high-fidelity results but at the cost of lengthy training and inference times. Improvements such as Mip-NeRF [1, 2] enhance rendering quality through anti-aliasing, while others [6, 9, 47, 27] focus on speeding up the processes. Recently, 3D Gaussian splatting (3DGS) [18] has emerged, offering competitive rendering quality and significantly higher efficiency by representing scenes as 3D Gaussian spheres and using a fast differentiable splatting pipeline [50]. However, 3DGS still requires high-quality and numerous input views for optimal reconstruction, which is often impractical.

Few-shot novel view synthesis. Leveraging additional information is essential for generating novel views from sparse input images. Various approaches incorporate different regularization techniques to prevent 3D geometry from overfitting to the training views. [19, 10, 28, 23] introduce extra geometric information, such as depth maps or coarse mesh, to enhance the robustness and performance of 3D reconstruction from sparse views. [5, 8] leverage the learned priors from multi-view stereo datasets as general priors to improve performance in sparse view reconstruction tasks. FreeNeRF [44] integrates frequency and occlusion regularization during training to mitigate overfitting issues in few-shot neural rendering. Similarly, DietPixelNeRF [16] employs a semantic view consistency loss to ensure that all views share consistent semantics, thereby alleviating overfitting. However, these methods are highly sensitive to the network's performance, where incorrect depth estimations or inaccurate mesh reconstructions can significantly degrade the final output.

Diffusion priors for novel view synthesis. Recently, utilizing diffusion models as priors for few-shot novel view synthesis has proven to be an effective approach. DreamFusion [30] employs Score Distillation Sampling (SDS) with a pre-trained diffusion model to guide 3D object generation from text prompts [36, 33, 46]. Some works [22, 34, 35] embed 3D awareness into 2D diffusion models to generate multi-view images, though these methods typically require large datasets [49] and significant training resources [16, 28]. ReconFusion [41] leavarge the 2D diffusion priors to recover a highfidelity NeRF from sparse input views. More advanced approaches leverage video diffusion models [4, 12, 13, 24] for few-shot NVS. For instance, AnimateDiff [11] fine-tunes diffusion models with additional camera motions using LoRA [14], while methods like SVD-MV [4], V3D [37] and IM-3D [24] propose camera-controlled video diffusion models for object-level 3D generation. In contrast, our approach offers greater generalizability for unbounded outdoor scenes.

Radiance fields enhancement. Several existing studies focus on enhancing NeRFs by addressing the limited detail preservation issue caused by insufficient or low-quality input data. NeRF-SR [38] and Refsr-nerf [15] use a super-resolution network to upscale the training view images, allowing novel views to be synthesized at higher resolutions with appropriate details. Alignerf [17] introduce optical-flow network to solve the misalignment problem to enhance the performance. Some other approaches incorporate 2D diffusion priors into 3D reconstructions. For instance, DiffusionNeRF [42] leverages a diffusion model to learn gradients of logarithms of RGBD patch priors, serving as regularized geometry and color for a scene. Nerfbusters [40] use diffusion priors to remove ghostly artifacts in the 3D gaussians. Our work aim to addresses the radiance fields enhancement problem by proposing a novel framework 3DGS-Enhancer, achieving superior enhancement performance for low-quality unbounded 3DGS representations.

3 Preliminary of 3D Gaussian Splatting

Here, we briefly review the formulation and rendering process of 3DGS [18]. 3DGS represents a scene as a set of anisotropic 3D Gaussian spheres, allowing high-fidelity NVS with extremely low rendering latency. A 3D Gaussian sphere includes a center position µ ∈ R 3 , a scaling factor s ∈ R 3 ,

Input views and rendered novel views

VAE Enc

Temporal

CLIP

ResBlock-3D

Spatio-temporal attention

Video diffusion prior for

novel view enhancement

Fine-tuned

Frozen

3D representation

fine-tuning

Enhanced novel views

and a rotation quaternion q ∈ R 4 , such that the Gaussian distribution is

where Σ = RSS T R T , S is the scaling matrix determined by s and R is the rotation matrix determined by q . To additionally model the view-dependent appearance, the Gaussian sphere also includes spherical harmonics (SH) coefficients C ∈ R k , where k is the number of SH functions, and an α ∈ R for opacity. The color and opacity are also calculated by the Gaussian distribution illustrated in Eq. 1.

For rendering, all the 3D Gaussian spheres are projected onto the 2D camera planes via a differentiable Gaussian splatting pipeline [50]. Given the viewing transform matrix W and Jacobian matrix J of the affine approximation of the projective transformation, the covariance matrix Σ ′ in camera coordinates is calculated as

The differentiable splatting method efficiently projects the 3D Gaussian spheres to 2D Gaussian distributions, ensuring fast α -blending for rendering and color supervision. For each pixel, the color is rendered by M Gaussian spheres that overlap with the pixel on the 2D camera planes, sorted in the depth distance as

4 Method

4.1 3DGS-Enhancer: An Overview

This work studies the 3DGS enhancement problem. More specifically, given a 3DGS model trained on a scene consisting of input views { I ref 1 , I ref 2 , . . . , I ref N ref } and corresponding camera poses { p ref 1 , p ref 2 , . . . , p ref N ref } , the goal of this work is to enhance a set of low-quality novel views { I 1 , I 2 , I 3 , . . . , I N new } rendered by the 3DGS model. The enhanced images further fine-tune the 3DGS model to improve its reconstruction and rendering quality.

This work novelly reformulates the challenging task of 3D-consistent image restoration as the task of video restoration, in light of the analogy between the multi-view consistency and the video temporal consistency. We propose a novel framework named 3DGS-Enhancer that employs a video LDM comprising an image encoder, a video-based diffusion model, and a spatial-temporal decoder to enhance the rendered images while preserving a high 3D consistency. 3DGS-Enhancer also adopts novel fine-tuning strategies to selectively integrate the views enhanced by the video LDM into the 3DGS fine-tuning process. An illustration of the 3DGS-Enhancer framework is shown in Figure 2. We discuss more details of the framework in the following.

4.2 Video Diffusion Prior for Temporal Interpolation

In this section, we introduce the video diffusion model for achieving 3D-consistent 2D image restoration. To lift the consistency between the generated 2D video frames and the high-quality reference views, we further propose to formulate the video restoration task as a video interpolation task, where the first frame and the last frame of inputs to the video diffusion model are two reference views. This formulation provides stronger guidance for the video restoration process. Let { p ref i -1 , p s 1 , p s 2 , . . . , p s T , p ref i } be the camera poses sampled from the trajectory fitted between two reference views, the images rendered accordingly are v = { I ref i -1 , I 1 , I 2 , . . . , I T , I ref i } . v ∈ R ( T +2) × 3 × H × W serves as the input to the video diffusion model, e.g. , a pre-trained imageguided stable video diffusion (SVD) model [4, 20] that adopts cross-frame spatio-temporal attention module and 3D residual convolution in the diffusion U-Net. Unlike SVD, which repeats the single input image feature extracted by CLIP [31] for T times as the conditional inputs, we input v to the CLIP encoder to get a sequence of conditional inputs c clip and add it to the video diffusion model through cross attention. Meanwhile, we input v to the VAE encoder to get latent feature c vae and add it into the diffusion model through a classifier-free guidance strategy to incorporate richer color information. The diffusion U-Net ϵ θ predicts the noise ϵ for each diffusion step t , and the training objective is

where z t = α t z + σ t ϵ ∗ .. where z is the gt latent, ϵ ∈ N (0 , I ) , α t and σ t define a noise at timestep t . The learned video diffusion model generates a sequence of enhanced image latents z v corresponding to the rendered low-quality views v .

4.3 Spatial-Temporal Decoder

Although the video diffusion model can generate enhanced image latents z v , we observe that there are artifacts such as temporal inconsistency, blurring, and color shift in outputs of the original decoder of video LDM. To address this issue, we propose a modified spatial-temporal decoder (STD). STD makes the following improvements over the original VAE Decoder: 1) Temporal decoding manner. STD adopts additional temporal convolution layers to ensure the temporal consistency between decoded outputs. Similar to our video diffusion model, the first and the last input frames are the reference view images, and the intermediate inputs are the generated views; 2) Effective integration of rendered views. STD adopts additional conditional inputs, same as those of the video diffusion model, allowing the decoder to better leverage the original rendered images. Inspired by [45, 48], these conditional inputs are fed into STD through Controllable Feature Warping (CFW) modules [39], such that their high-frequency patterns are better preserved. 3) Color correction. To address the color shift issue, we apply color normalization to the decoded images by following StableSR [39]. However, we observe that highly blurred and low-quality images in the conditional inputs can undermine the color correction effects. To mitigate this, we use the first reference view to calculate the mean and variance, and then align all the other decoded images with this reference view. Let I g i be the i -th decoded images with a mean u ˆ I g 0 and a variance σ ˆ I g 0 , ˆ I g 0 be the reference view with a mean µ I g i and a variance σ I g i , the corrected image I c i is computed by:

The optimization objective of STD consists of an L1 reconstruction loss and an LPIPS perceptual loss between I g and ground-truth ˆ I g , and an adversarial loss, as

where L adv is the adversarial loss that discriminates between real image ˆ I g and fake image I g .

4.4 Fine-tuning Strategies of 3D Gaussian Splatting

Confidence-aware 3D Gaussian splatting. Unlike existing sparse-view NVS methods, our approach does not rely on depth estimation networks for depth regularization. Instead, we take a purely 2D visual method by utilizing a video diffusion model to enhance images rendered from a low-quality 3DGS model. Despite this significant enhancement in the quality of the rendered views, we propose

Rendered images

Confidence Map

GT

to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive to slight inaccuracies in the restored views. These inaccuracies could be amplified during the fine-tuning process.

To minimize the negative impact of generated images on Gaussian training, we propose confidenceaware 3D Gaussian splatting. This strategy involves two levels of confidence, image level and pixel level. For the image level, the generated images that are closer to real images have lower confidence. For pixel level, the larger the mean covariance of all the Gaussians used to render this pixel, the higher its confidence.

Image level confidence. In the task of novel view synthesis, if noise exists in two image views, a close distance between them increases the likelihood of generating conflicts and disrupting the 3D consistency of the scene. Therefore, for novel views that are close to the reference view, it is crucial to carefully optimize the 3D Gaussians to mitigate the adverse effects of noise. Conversely, when a novel view is far from all known views, it has a smaller likelihood of disturbing already well-reconstructed areas. Based on this reasoning, we normalize the distance from novel views to reference views between 0 and 1. The farther a viewpoint is from the reference view, the higher its confidence.

Pixel level confidence. Inspired by ActiveNeRF [29], which uses Gaussian distributions in NeRF to estimate uncertainty and identify views with the highest information gain, we aim to find the pixels that can provide the highest information gain from the generated images. As shown in Fig 3, we observed that well-reconstructed areas are typically represented by Gaussians with very small volumes, calculated using the scaling vector s ∈ R . Based on this observation, we propose a method to calculate pixel-level confidence.

The unique representation of 3D Gaussians allows us to render an H × W × 3 image using a process similar to rendering colors, where each channel corresponds to one of the three components of the scaling vector s . In 3DGS-Enhancer, we multiply these three channels of the scale map to obtain pixel-level confidence. For each pixel in the generated images, higher confidence results in greater weight in supervising the training of the 3DGS model.

Given a set of 3D Gaussian, the 3-channel C conf confidence map is rendered as same as colour rendering, and the formula is defined as follows

And the 1 channel pixel level confidence map P c = 3 √ C conf [0] × C conf [1] × C conf [2] . Overall, in our training process for 3D Gaussians, the loss functions were defined as

where SSIM is the Structural Similarity Index and ⊙ is Hadamard's product, I c is the image-level confidence map and ˆ C is the real pixel value.

5 Experiments

5.1 3DGS-Enhance Dataset

Given that the enhancement of 3DGS representations is a new task, we create a dataset to simulate various artifacts of the 3DGS representations. This dataset also serves as a more comprehensive benchmark for evaluating the performance of few-shot NVS methods. Existing few-shot NVS algorithms [44, 19] primarily focus on face-forward evaluations [25], where the test views have significant overlap with the input views. However, this evaluation method is not suitable for largescale unbounded outdoor scenes. Therefore, we propose a dataset processing strategy that allows us to post-process any existing multi-view dataset to generate a large number of training image pairs that include typical artifacts caused by few-shot NVS.

More specifically, for each scene, we have n views I train = { I 1 , I 2 , . . . , I n } , which serve as the input for a high-quality 3DGS model. We uniformly sample a small number m of views I low from I train, which serve as the input for the low-quality 3DGS model. By linearly fitting the highquality camera poses p train i = { p train 1 , p train 2 , . . . , p train n ∗ } , we randomly sample a camera trajectory p render i = { p render 1 , p render 2 , . . . , p render n ∗ } on p train i and render the image pairs using both high-quality and low-quality 3DGS models. This creates a set of high-quality and low-quality image pairs used for the training of our video diffusion model.

We apply this dataset processing strategy to DL3DV [21], a large-scale outdoor dataset containing 10K scenes. We randomly select 130 scenes from the original DL3DV dataset and form more than 150,000 image pairs. We randomly select another 20 scenes from DL3DV to form the test sets, evaluating the corss-scene capability of our method. More implementation details of the method can be found in the supplementary material.

5.2 Comparison with State-of-the-Arts

The quantitative and qualitative results on the DL3DV test set with 3 6 and 9 input views are shown in Table 1 and Figure 4. Our approach outperforms all the other baselines in PSNR, SSIM, and LPIPS scores. NeRF-based methods including Mip-NeRF [1] and FreeNeRF [44] produce blurry novel views due to smoothing inconsistencies. In contrast, 3DGS [18] generates elongated elliptical artifacts due to local minima convergence. DNGuassian [19] reduces artifacts with depth regularization but results in blurry and noisy novel views.

The first example in Figure 4 demonstrates 3DGS-Enhacer's capability to remove artifacts while preserving view consistency. By interpolating input views using a video diffusion model, we incorporate more information while enrusing a high view consistency, enabling high-quality novel

MipNerf [1]

FreeNerf [44]

3DGS [18]

DNGaussian [19]

Ours

Ground-truth

MipNerf [1]

FreeNerf [44]

3DGS [18]

DNGaussian [19]

Ours

Ground-truth

MipNerf [1]

FreeNerf [44]

3DGS [18]

DNGaussian [19]

Ours

Ground-truth

views and avoiding local minima. The second example highlights 3DGS-Enhancer's advantage in recovering high-frequency details. Our dataset processing strategy and video diffusion model enable an understand of strong multi-view prior across various scenes. As a result, very challenging cases such as the trees can be restored with sharp details. In summary, comparisons with baseline methods demonstrate our approach's potential to significantly improve the unbounded 3DGS representations, synthesizing high-fidelity novel views for open environments.

To demonstrate the generalizability of our method for out-of-distribution dataset, we train the methods on the DL3DV-10K dataset [21] and test them on the Mip-NeRF360 dataset [2]. The results, as

3DGS (PSNR: 16.15)

Ours (PSNR: 22.32)

3DGS (PSNR: 13.90)

Ours (PSNR: 22.32)

Ground-truth

summarized in Table 2 and Fig 5, show that our method outperforms the baseline approaches, highlighting its remarkable generalization capabilities in unbounded environments.

5.3 Ablation Study

Real image as reference views. Table 3 shows the quantitative comparisons of different components in 3DGS-Enhancer framework. The video diffusion model provides strong multi-view priors. However, due to its native restrictions, we directly feed the original input views into the 3DGS fine-tuning process. This results in more reliable and view-consistent information from the input domain to facilitate 3DGS fine-tuning, as demonstrated by the "Real image" in Table 3.

Confidence aware reweighting. Distant views are less likely to cause artifacts, so we normalize their distance to reference views between [0, 1], giving higher confidence of video diffusion results to farther viewpoints. This strategy is denoted by "Image confidence" in Table 3. Pixel-level confidence, as denoted by "Pixel confidence" in Table 3, is based on the density of small-volume Gaussians in well-reconstructed areas, using a color rendering pipeline to calculate volumes. Both pixel and image-level confidence strategies improve results individually, and their combination yields the best performance.

Video diffusion and STD. Figure 6 visualizes the effects of video diffusion and STD module, respectively. Video diffusion removes most of the artifacts, and STD module enhances fine-grained and high-frequency textures, resulting in more vivid novel view renderings, which are closer to the ground truth. Table 4 shows the improvment for each modules.

Input

Diffusion

STD

Ground-truth

6 Conclusions, Limitations, and Future Work

This paper has introduced 3DGS-Enhancer, a unified framework that applies view-consistency prior from video diffusion and use trajectory interpolation method to enhance unbounded 3DGS representations. By combining image and pixel-level confidence with 3DGS fine-tuning, we have achieved state-of-the-art performance in NVS enhancement. However, our approach relies on adjacent views for continuous interpolation, it cannot be easily adapted to single-view 3D model generation. Moreover, the confidence-aware 3DGS fine-tuning strategies are relatively simple and straightforward. In the future, it is interesting to integrate confidence maps directly with the video generation model, enabling the generation of images that are more in line with the real 3D world without the need for post-processing. Meanwhile, utilizing the efficient data generation capability of 3DGS to construct a massively scaled dataset for our video generation model presents a prime opportunity to enhance the model's 3D consistency. This approach also facilitates the 2D models to understand the 3D world directly from 2D images without additional geometric constraints. Regarding the social impact, the goal of this work is to advance the fields of 3D reconstruction and NVS. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

References

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5855-5864, 2021.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR , 2022.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV , 2023.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19457-19467, 2024.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision , pages 333-350. Springer, 2022.

Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06738 , 2024.

Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, jun 2021.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5501-5510, 2022.

Guangcong, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations , 2024.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Xudong Huang, Wei Li, Jie Hu, Hanting Chen, and Yunhe Wang. Refsr-nerf: Towards high fidelity and super resolution view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8244-8253, June 2023.

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5885-5894, October 2021.

Yifan Jiang, Peter Hedman, Ben Mildenhall, Dejia Xu, Jonathan T. Barron, Zhangyang Wang, and Tianfan Xue. Alignerf: High-fidelity neural radiance fields via alignment-aware training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 46-55, June 2023.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), July 2023.

Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20775-20785, 2024.

Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Lanqing Hong, Guo Zhou, Hua Yao, Dit-Yan Yeung, Huchuan Lu, and Xu Jia. Trackdiffusion: Tracklet-conditioned video generation via diffusion models, 2024.

Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22160-22169, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9298-9309, 2023.

Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision , pages 210-227. Springer, 2022.

Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. International Conference on Machine Learning, 2024 , 2024.

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) , 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Muhammad Husnain Mubarik, Ramakrishna Kanungo, Tobias Zirr, and Rakesh Kumar. Hardware acceleration of neural graphics. In Proceedings of the 50th Annual International Symposium on Computer Architecture , pages 1-12, 2023.

Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5480-5490, 2022.

Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: Learning where to see with uncertainty estimation. In European Conference on Computer Vision , pages 230-246. Springer, 2022.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. CVPR, 2024 , 2023.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.

Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations , 2024.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitrii Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from a single image using latent video diffusion. arXiv , 2024.

Chen Wang, Xian Wu, Yuan-Chen Guo, Song-Hai Zhang, Yu-Wing Tai, and Shi-Min Hu. Nerf-sr: High quality neural radiance fields using supersampling. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6445-6454, 2022.

Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. In arXiv preprint arXiv:2305.07015 , 2023.

Frederik Warburg, Ethan Weber, Matthew Tancik, Aleksander Holynski, and Angjoo Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 18120-18130, October 2023.

Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21551-21561, 2024.

Jamie Wynn and Daniyar Turmukhambetov. DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models. In CVPR , 2023.

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. ICCV , 2023.

Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2023.

Xi Yang, Chenhang He, Jianqi Ma, and Lei Zhang. Motion-guided latent diffusion for temporally consistent real-world video super-resolution. In European Conference on Computer Vision , pages 224-242. Springer, 2025.

Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In CVPR , 2024.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In arXiv , 2021.

Shangchen Zhou, Peiqing Yang, Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscalea-video: Temporal-consistent diffusion model for real-world video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2535-2545, 2024.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH , 2018.

Matthias Zwicker, Hanspeter Pfister, Jeroen Baar, and Markus Gross. Surface splatting. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics , 2001, 08 2001.

8 Appendix

8.1 Details of 3DGS Enhancement Dataset

For our 3DGS Enhancement Dataset, constructed based on DL3DV, we randomly select 120 scenes to create the training set for our video diffusion model and 30 scenes as the test set. By following previous works, we use the standard train/test split, selecting every 8th frame of the remaining frames for evaluation.

To create image pairs simulating the artifacts due to the lack of input views in novel view synthesis problem, we render the image pairs from pairs of low-high quality 3DGS models. Specifically, the input views for the high-quality model consist of all images in the original dataset, while the inputs for the low-quality model are a subset uniformly sampled from the original dataset. To add more complexity, we sample the subset according to a certain number (e.g., 3, 6, 9) or a certain ratio (e.g., 5%). With the aim to fully capture the distribution of artifacts created by the sparse input views and train the video diffusion model with smoother inputs, we propose a heuristic trajectory fitting algorithm, as shown in Figure 7, proving a sequence of cameras by interpolating the low or high-quality model's input views. Specifically, if the original camera trajectories are smooth and simple, such as those of DL3DV, we use the high-quality input views as the reference to fit the trajectories. For complex trajectories, such as those in Mip-NeRF 360, we use the low-quality input to avoid significantly poor rendering results, which would lead to unreasonable artifact distributions. As a result, we render a large number of image pairs with and without artifacts, as shown in Figure 8, at a resolution of 512 × 512, leading to powerful video diffusion priors with high view consistency and photo-realism.

Input: 3

Input: 6

Input: 9

8.1.1 Training details

Our video diffusion model includes a pre-trained VAE to encode an image sequence into a latent sequence and decode the latent sequence back into the image sequence. It also includes a U-Net with learnable temporal layers, which employs cross-frame attention modules and 3D CNN modules to ensure frame-consistent outputs. The input of video diffusion model is a image sequence segment that includes 25 images with different sample steps from the image sequences rendered from the low-quality 3DGS model. The first and the last frames in this segment are replaced with images rendered from the high-quality 3DGS model. During fine-tuning, our video diffusion model is conditioned on these image sequence segments and trained to synthesize the corresponding segments rendered from the high-quality 3DGS model.

Our video diffusion model is fine-tuned with a learning rate of 0.0001, incorporating 500 steps for warm-up, followed by a total of 80,000 training steps. The batch size is set to 1 in each GPU, where each batch consisted of 25 images at 512x512 resolution. To optimize the training process, the Adam optimizer is employed. Additionally, a dropout rate of 0.1 is applied to the conditions between the first and last frames and the training process utilize CFG (classifier-free guidance) to train the diffusion model. The training is conducted on 2 NVIDIA A100-80G GPUs over 3 days. The STD is fine-tuned with a learning rate of 0.0005 and 50,000 training steps. The batch size is set to 1 in each GPU, where each batch consists of 5 images at 512x512 resolution, but for inference, it was increased to 25. The fine-tuning process is conducted on 2 NVIDIA A100-80G GPUs in 2 days. The entire pipeline's inference and training speeds were evaluated and are presented in Table 5.

Input: 3

Ground-truth

Input: 6

Input: 9

Ground-truth

Input: 5%

Ground-truth

8.2 Details of Comparison Baselines

For the evaluation datasets, we compare against the standard 3D Gaussian Splatting [18] (which is also the reconstruction pipeline used in our work), and the state-of-the-art few-view NVS regularization methods, including Mip-NeRF [1], FreeNeRF [44], Zip-NeRF [3], and RegNeRF [28]. We also compare to some few-shot NVS methods using generative priors including ZeroNVS [33], and ReconFusion [41].

For the evaluation of MipNeRF, FreeNeRF, RegNeRF, and DNGaussian on DL3DV and Mip-NeRF 360 dataset, we follow the original configurations and code shared by the authors. Additionally, we use random point cloud as the initialization for 3DGS, following the implementations from DNGaussian. We also decrease the batch size for RegNeRF from 4096 to 512 according to the limited computation resource.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We ensured that the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The discussions on limitations of this work can be found in the last section of the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have discussed all implementation details of the proposed method in Section 4 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The project page has been online. The code and generated datasets will be publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have discussed all implementation details of the proposed method in Section 4 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Experiments on large-scale 3D reconstruction datasets are computationally expensive. Existing literature also did not provide error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details of compute resources can be found in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have ensured that this research followed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the societal impacts in the last section of main paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not have such risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original papers that produced the code package and datasets. The licenses for re-packaged datasets will be provided on the dataset website.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of enhancing 3GS novel views which are far from existing viewpoints (encountered in sparse-view settings), due to insufficient information in under-sampled areas. This work enhances the low quality views using a video diffusion model to maintain multi view and temporal consistency. The authors reformulate the task of 3D-consistent image restoration as video restoration which is again reformulated as a video interpolation tasks between frames. The high quality frames along with the restored images are then used to fine tune the 3DGS. The authors also propose a modified spatial temporal decoder (STD) to reduce artifacts and introduce confidence awareness at both image level (based on distance of novel view from reference) and pixel level (information gain) to minimize the negative impact of generated images.\n\nSTRENGTHS:\n* The paper is well motivated- enhancing low quality 3DGS rendering results for camera viewpoints which are far away\n* Reformulates the 3D-consistent image restoration task in 3DGS as temporally consistent video generation to leverage video diffusion models \n* The results look quite impressive and have been tested on a variety of scenes\n* can be applied to any existing 3DGS models and more generalized scenes\n* introduces a dataset for evaluation of artifacts in 3DGS (will this dataset be open sourced?)\n* Sound experimental setup and adequate comparisons with other methods\n\nWEAKNESSES:\n* Related work appears to be thin and some of these paragraphs can be more descriptive\n* Method relies on adjacent views for continuous interpolation\n* Choice for STD should be quantitatively ablated. Individual parts in STD like color correction should be ablated as well.\n\nQUESTIONS:\n1. Authors describe in L184-186 \"Despite this significant enhancement in the quality of the rendered views, we propose to rely more on the reference views rather than the restored novel views when fine-tuning the 3DGS model, since the 3DGS model is highly sensitive to slight inaccuracies in the restored views.\" I would appreciate a more thorough discussion on this.\n\n2. \"modified baselines denoted with ∗ taken from [36]\". However, no baseline seems to be marked with * in table 1. This is possibly an oversight and the authors should correct it. \n\n3. What's the adversarial loss described in L179 for STD? \n\n4. On what data is the Temporal Denosing U-Net fine tuned on?\n\nSuggestions:\nFigure 2 can be more interpretable if the right part shows the output of the enhanced views (input from left part); currently there is no 1:1 parity\n\nLIMITATIONS:\nLimitations are adequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper utilizes a video diffusion model to enhance 3DGS rendering results. It proposes a 3DGS-Enhancer pipeline to reformulate 3D-consistent image restoration tasks and leverage it to generate high-quality and 3D-consistent images. They have enough experiments to prove the soundness of their method.

STRENGTHS:
They didn't use depth information but video diffusion prior to do the task, which is interesting.
They also have the spatial-temporal decoder which solves the temporal consistency problem.
For fine-tuning, they add confidence constraints on both pixel level and image level to enhance the result.
Extensive comparison with state-of-the-art is done, and the results seem robust.

WEAKNESSES:
They can provide more comparisons on ablating the temporal consistency.

QUESTIONS:
Video Diffusion prior may have motion information of moving objects in the scene. How do we eliminate that effect in enhancing 3DGS?

LIMITATIONS:
The limitation is definitely from the pre-trained video diffusion model whether the data is more like novel view synthesis data (static scene) or a sequence of video frames (moving objects).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper presents a novel pipeline aimed at enhancing the quality of 3D Gaussian splatting (3DGS) representations, especially in scenarios with sparse input views. They propose a novel framework, 3DGS-Enhancer, that leverages video LDMs for generating high-quality and 3D-consistent images. Moreover, to mitigate the artifacts caused by the temporal inconsistency, they introduce a spatial-temporal decoder and fine-tuning strategy for the 3DGS optimization process. extensive experiments are conducted to demonstrate their superior reconstruction performance on large-scale datasets of unbounded scenes.

STRENGTHS:
1. Applying video models to scene reconstruction is a novel idea and has achieved excellent results.
2. To mitigate the negative impact of inconsistent generated images, this paper proposes improvements to the decoder and defines a confidence measure to enhance the corresponding 3D Gaussian Splatting optimization process.

WEAKNESSES:
1. The writing of this paper is not very clear, and some typo errors exist in the submission, for instance, line147, it should be $ \{ I^{ref}_{i-1}, I_1, I_2, ..., I_T, I^{ref}_i \}$; line216: it should be $I_c$, consistent to the equation (8).

2. Training details are missing in the paper, more details about the video diffusion model and fine-tuning process should be provided.

3. More datasets, such as MipNerf360, should be evaluated to verify the effectiveness and robustness of the proposed method.

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
This paper is working on the problem of novel view synthesis with sparse input views. The authors present 3DGS-Enhancer to enhance the rendering quality and address 3D view consistency problem using 2D video diffusion priors. The experimental results show that this work has achieved state-of-the-art performance in novel view synthesis enhancement.

STRENGTHS:
1. Writing quality is great, clean, and easy to follow.
2. To tackle the sparse view reconstruction problem, the idea of fine-tuning the original 3DGS representation instead of incorporating additional geometric constraints is novel and promising.
3. A new dataset is introduced in this work, which is useful for this research direction and the community.
4. The idea of leveraging video LDM for sparse view reconstruction in unbounded outdoor scenes is novel and effective.

WEAKNESSES:
Runtime is not discussed in this paper. I’m wondering how long it will take for one scene compared to the other baselines.

QUESTIONS:
The experiments are conducted almost on the self-introduced dataset. Can 3DGS-Enhancer also surpasses the other baselines on more commonly used outdoor unbounded datasets (e.g. Mip-NeRF 360 dataset)?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-14